JP6426215B2 - Endoscope apparatus and program - Google Patents

Description

  The present invention relates to an endoscope apparatus having a measurement function. The present invention also relates to a program for realizing a measurement function by a computer.

  BACKGROUND In recent years, industrial endoscopes are widely used for observation and inspection of flaws, corrosion and the like inside boilers, turbines, engines, chemical plants and the like. In recent industrial endoscopes, it has become possible to attach a stereo optical adapter for imaging an object from different viewpoints to the tip of the endoscope, and measure various spatial characteristics of the object according to the principle of triangulation (stereo Measurement) (see, for example, Patent Document 1). In stereo measurement, for example, a pair of left and right subject images is acquired, and matching processing is performed in which a point on the other subject image corresponding to a point designated on one subject image is detected by image processing. Furthermore, based on the coordinates of the corresponding left and right points, the length, depth, area and the like of the subject are calculated according to the principle of triangulation.

  The result of the above matching process affects the accuracy of the stereo measurement. In particular, in the case where a mismatch in matching occurs where the positions of corresponding left and right points are greatly different, the measurement results will be completely different, and it will be necessary to re-specify the points and to retake the image. As a technique for reducing the false correspondence of matching, for example, a method of using the highest reliability result by using a plurality of templates (refer to Patent Document 2), evaluation of reliability with a low resolution image, and response There is known a method of determining the window size of a high resolution image for point calculation (see Patent Document 3). Further, as a method of prompting the user to confirm whether or not a matching error has occurred, a method of emphasizing and displaying a region related to matching in a subject image is also known (see Patent Document 4).

JP 2004-33487 A JP, 2001-109879, A JP, 2009-92551, A JP 2011-170276 A

  In an inspection using an industrial endoscope, an image recorded at an observation site may be inspected by stereo measurement outside the observation site.

  However, when a matching error occurs in the inspection image recorded at the observation site, it is inconvenient because it is necessary to reacquire the inspection image at the observation site.

  According to the technology described in Patent Document 4, even if it is known that a matching error occurs, it takes time for the user to manually correct the corresponding point, etc.

  The present invention has been made in view of the problems described above, and it is an object of the present invention to provide an endoscope apparatus and a program that can reduce the time and effort of the user.

The present invention has been made to solve the above-described problems, and an imaging unit for capturing a first subject image and a second subject image corresponding to the same subject to generate image data, and the image data A display unit for displaying an image based on the second image, and candidates for corresponding points that are points on the second object image corresponding to the measurement points set on the first object image using a plurality of matching methods. A plurality of corresponding point candidate calculation units for calculating a plurality of points, a reliability calculation unit for calculating the degree of matching reliability based on the corresponding point candidates respectively corresponding to the plurality of matching methods, and the degree of reliability is less than a predetermined value A display processing unit that performs processing to display the measurement point and a plurality of corresponding point candidates corresponding to the plurality of matching methods on the image, and the reliability is less than a predetermined value To the A plurality of said of said corresponding points specified among the corresponding point candidate candidates corresponding to each of the matching method, and corresponding point determination unit for determining the corresponding points, and the measuring point, and was determined the corresponding point And a measurement processing unit configured to measure the size of the subject based on the above.

  In the endoscope apparatus of the present invention, the endoscope apparatus further includes an operation unit for instructing the user to increase the number of corresponding point candidates to be displayed in the image.

  In the endoscope apparatus of the present invention, the plurality of corresponding point candidates are candidates for corresponding points to be added by an instruction to increase the number of corresponding point candidates input by the user via the operation unit. It is characterized by including.

  Further, in the endoscope apparatus according to the present invention, the corresponding point candidate calculation unit is a candidate for the corresponding point to be added after an instruction from the user to increase the number of the corresponding point candidates to be displayed in the image is given. It is characterized by calculating.

  Further, in the endoscope apparatus of the present invention, the corresponding point candidate calculation unit is a candidate for the corresponding point to be added in advance before the user instructs to increase the number of the corresponding point candidates to be displayed in the image. To calculate.

  Further, in the endoscope apparatus according to the present invention, the display processing unit is characterized in that the display processing unit displays a candidate of the overlapping corresponding points among the plurality of corresponding points as a single mark on the image. I assume.

  Further, in the endoscope apparatus according to the present invention, the display processing unit is characterized in that the display processing unit displays the number of overlapping corresponding point candidates in the vicinity of the corresponding point candidates.

  Further, in the endoscope apparatus according to the present invention, each of the plurality of corresponding point candidates has a priority coefficient, and the display processing unit is configured to use the priority coefficient corresponding to each of the plurality of corresponding point candidates. It is characterized in that processing for displaying on the image in a display form according to the request is performed.

  Further, in the endoscope apparatus according to the present invention, the display processing unit may further select corresponding point candidates to be additionally displayed so that the number of corresponding point candidates displayed in the image is kept constant. When displaying on an image, it is characterized in that some of the corresponding point candidates already displayed are not displayed.

Further, according to the present invention, a measurement point set on the first subject image of an image based on image data generated by capturing a first subject image and a second subject image corresponding to the same subject Calculating a plurality of corresponding point candidates that are points on the corresponding second subject image using a plurality of matching methods, and based on the corresponding point candidates corresponding to each of the plurality of matching methods The step of calculating the reliability of matching, and when the reliability is less than a predetermined value, the measurement point and a plurality of corresponding point candidates respectively corresponding to the plurality of matching methods are displayed on the image and performing a process of, wherein when the reliability is less than the predetermined value, the corresponding points are specified among the plurality of candidates of the corresponding point corresponding to each of the plurality of matching technique candidate Determining the corresponding points, and the measuring point, a program for executing the steps of measuring the size of the object based on the determined the corresponding point, to the computer.

  According to the present invention, the time and effort of the user are reduced as compared with the manual correction of the corresponding points.

  Therefore, it is possible to reduce the error in matching and reduce the time and effort of the user.

1 is a perspective view showing an entire configuration of an endoscope apparatus according to an embodiment of the present invention. It is a block diagram showing an internal configuration of an endoscope apparatus by one embodiment of the present invention. It is a block diagram showing functional composition of CPU which an endoscope apparatus by one embodiment of the present invention has. It is a flowchart which shows the procedure of operation | movement of the endoscope apparatus by one Embodiment of this invention. It is a flowchart which shows the procedure of operation | movement of the endoscope apparatus by one Embodiment of this invention. It is a reference drawing showing the picture which the endoscope apparatus by one embodiment of the present invention acquired. It is a reference drawing showing the picture which the endoscope apparatus by one embodiment of the present invention acquired. It is a reference drawing showing the picture which the endoscope apparatus by one embodiment of the present invention acquired. It is a reference drawing showing the picture which the endoscope apparatus by one embodiment of the present invention acquired. It is a reference drawing showing the picture which the endoscope apparatus by one embodiment of the present invention acquired. It is a reference drawing showing the picture which the endoscope apparatus by one embodiment of the present invention acquired. It is a reference drawing showing the picture which the endoscope apparatus by one embodiment of the present invention acquired. It is a reference drawing for explaining how to obtain the three-dimensional coordinates of the measurement point by stereo measurement.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 shows the overall configuration of an endoscope apparatus according to an embodiment of the present invention. As shown in FIG. 1, the endoscope apparatus 1 includes an endoscope 2 and an apparatus main body 3 connected to the endoscope 2. The endoscope 2 includes an elongated insertion unit 20, and an operation unit 6 for performing operations necessary to execute various operation control of the entire apparatus. The apparatus main body 3 includes a monitor 4 for displaying an image of an object imaged by the endoscope 2, an operation control content (for example, a processing menu), and the like, and a case 5 having a control unit 10 (see FIG. 2) inside. Have.

  The insertion portion 20 is configured by continuously arranging, from the tip side, a hard tip portion 21, a bending portion 22 that can be bent in the vertical and horizontal directions, and a flexible tube portion 23 having flexibility. Various types of optical adapters such as a stereo optical adapter having two observation fields of view, a normal observation optical adapter having one observation field of view, and the like are detachably attached to the distal end portion 21.

  As shown in FIG. 2, in the housing 5, an endoscope unit 8, a CCU 9 (camera control unit), and a control unit 10 are provided. The proximal end of the insertion portion 20 is connected to the endoscope unit 8. The endoscope unit 8 is configured to include a light source device (not shown) for supplying illumination light necessary for observation, and a bending device (not shown) for bending the bending portion 22 constituting the insertion portion 20. . The CCU 9 is configured to include a drive device (not shown) that drives the imaging device 28.

  An imaging element 28 is built in the tip end portion 21. The imaging device 28 photoelectrically converts an object image formed through the optical adapter to generate an imaging signal. The imaging signal is converted into a video signal (image data) such as an NTSC signal in the CCU 9 and supplied to the control unit 10.

  In the control unit 10, a video signal processing circuit 12 to which a video signal is input, a ROM 13, a RAM 14, a card I / F 15 (card interface), a USB I / F 16 (USB interface), and an RS-232C I / F 17 (RS- 232C interface, and a CPU 18 are provided.

  A CCU 9 and an endoscope unit 8 are connected to the RS-232C I / F 17, and an operation unit 6 for controlling the CCU 9 and the endoscope unit 8 and instructing operation is connected. When the user operates the operation unit 6, communication necessary for controlling the operation of the CCU 9 and the endoscope unit 8 is performed based on the content of the operation.

  The USB I / F 16 is an interface for electrically connecting the control unit 10 and the personal computer 31. By connecting the control unit 10 and the personal computer 31 via the USB I / F 16, control based on various instructions such as an instruction to display an endoscope image and image processing at the time of measurement is performed on the personal computer 31 side. While being able to do, it becomes possible to input and output control information, data, etc. required for various processes between the control unit 10 and the personal computer 31.

  Further, the memory card 32 can be freely attached to and detached from the card I / F 15. By loading the memory card 32 into the card I / F 15, the control processing information stored in the memory card 32, data such as image information, etc. is taken into the control unit 10 under the control of the CPU 18, or the control processing information It becomes possible to record data such as image information on the memory card 32.

  The video signal processing circuit 12 is a graphic image signal based on the operation menu generated by the CPU 18 to display a composite image obtained by combining the endoscopic image based on the video signal supplied from the CCU 9 and the operation menu by graphic. And CCU 9 and processes necessary for displaying on the screen of the monitor 4 and the like, and supplies a display signal to the monitor 4. In addition, the video signal processing circuit 12 can also perform processing for solely displaying an endoscopic image or an image such as an operation menu. Therefore, on the screen of the monitor 4, an endoscope image, an operation menu image, a composite image of the endoscope image and the operation menu image, and the like are displayed.

  The CPU 18 executes the program stored in the ROM 13 to control various circuit units and the like so as to perform processing according to the purpose, and performs operation control of the entire endoscope apparatus 1. The RAM 14 is used by the CPU 18 as a work area for temporarily storing data.

  The program executed by the CPU 18 may be recorded in a computer readable recording medium, and the program recorded in the recording medium may be read by a computer other than the endoscope apparatus 1 and executed. For example, the personal computer 31 reads and executes a program, transmits control information for controlling the endoscope apparatus 1 to the endoscope apparatus 1 according to the program, and controls the endoscope apparatus 1 to perform an endoscope A video signal may be acquired from the device 1 and measurement may be performed using the acquired video signal.

  Here, the "computer" also includes a homepage providing environment (or display environment) if the WWW system is used. The term "computer-readable recording medium" means portable media such as flexible disks, magneto-optical disks, ROMs, CD-ROMs, DVD-ROMs, flash memories, and storage devices such as hard disks incorporated in computers. Say Furthermore, the "computer-readable recording medium" is a volatile memory (RAM) in a computer system serving as a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. In addition, those that hold the program for a certain period of time are also included.

  Also, the above-described program may be transmitted from a computer in which the program is stored in a storage device or the like to another computer via a transmission medium or by transmission waves in the transmission medium. Here, the “transmission medium” for transmitting the program is a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. Also, the program described above may be for realizing a part of the functions described above. Furthermore, it may be a so-called difference file (difference program) that can realize the above-described functions in combination with a program already recorded in the computer.

  In the present embodiment, an image is acquired in a state in which the stereo optical adapter having the two left and right observation fields of view is mounted on the distal end portion 21. The light from the subject is incident on the stereo optical adapter, and the stereo optical adapter forms two left and right subject images (first subject image and second subject image) corresponding to the same subject. The two subject images are incident on the imaging device 28 and captured by the imaging device 28. The acquired image is an image including two subject images. In the present embodiment, among the acquired images, images of regions corresponding to two subject images are defined as a left image and a right image.

  FIG. 3 shows a functional configuration of a part of the CPU 18 that is the center of the description of the present embodiment. The control unit 180, the measurement point setting unit 181, the corresponding point candidate calculation unit 182, the reliability calculation unit 183, the display processing unit 184, the corresponding point determination unit 185, and the measurement processing unit 186 constitute the function of the CPU 18.

  The control unit 180 controls processing performed by each unit. The measurement point setting unit 181 sets the measurement point at the measurement position designated on the left image based on the image data by the user operating the operation unit 6. The corresponding point candidate calculation unit 182 uses a plurality of matching methods to select corresponding point candidates (hereinafter referred to as corresponding point candidates) corresponding to the measurement points set on the left image in the right image based on the image data. calculate. The reliability calculation unit 183 calculates the reliability of matching based on corresponding point candidates corresponding to each of the plurality of matching methods.

  The display processing unit 184 causes the user to confirm the position of the corresponding point candidate, so that for the image based on the image data, the graphic (mark) of the measurement point and the corresponding point candidate corresponding to each of the plurality of matching methods Perform processing for superposition. The corresponding point determination unit 185 determines corresponding points to be used for measurement processing based on the calculated corresponding point candidates. The measurement processing unit 186 calculates three-dimensional coordinates in space based on two-dimensional coordinates on the image of the measurement point and the determined corresponding point, and based on three-dimensional coordinates corresponding to the plurality of measurement points, Measure the size of the subject.

Next, the principle of stereo measurement in the present embodiment will be described. By using the stereo optical adapter and determining the three-dimensional space coordinates of the object using the principle of triangulation based on the coordinates of the left and right optical distance measuring points when the object image is captured by the left and right optical systems, Various measurements (stereo measurement) are possible. Hereinafter, how to obtain three-dimensional coordinates of measurement points by stereo measurement will be described with reference to FIG. For images taken by the left and right optical systems, the middle point of the line connecting the left and right optical centers 63 and 64 with the method of triangulation is the origin O and the right direction is exactly x-axis, down The three-dimensional coordinates (X, Y, Z) of the measurement point 60 when the direction is defined as y axis and the direction away from the optical system in parallel with the optical axis are defined as (X, Y, Z) 3) Calculated by the equation.
However, with the two-dimensional coordinates of the measurement point 61 and the corresponding point 62 on the left and right image planes subjected to distortion correction, the intersection OL between the optical axis of the left and right optical systems and the image plane is taken as the origin (XL , YL), (XR, YR), the distance between the left and right optical centers 63, 64 is D, the focal length is F, and t = D / (XR-XL).
X = t × XR + D / 2 (1)
Y = −t × YR (2)
Z = t × F (3)

When the coordinates of the measurement point 61 and the corresponding point 62 on the image plane on the original image are determined as described above, the three-dimensional coordinates of the measurement point 60 are obtained using the parameters D and F. By obtaining the three-dimensional coordinates of several points, various measurements such as the distance between two points, the distance between a line connecting two points, one point, area, depth, surface shape, etc. are possible. Also, the left optical center 63 or the right optical center 6
It is also possible to obtain the distance from 4 to the subject (object distance). Object distance is the tip 2
The distance from 1 to the subject, for example, the distance from the imaging device 28 or the observation optical system to the subject. In order to perform the above-described stereo measurement, optical data indicating the characteristics of the optical system including the tip 21 and the stereo optical adapter is required. The details of the matching process and the optical data are described in, for example, Japanese Patent Application Laid-Open No. 2004-49638, and thus the description thereof is omitted.

  Next, details of the measurement process in the present embodiment will be described. FIG. 4 shows the procedure of the operation of the endoscope apparatus 1 at the time of measurement. After the image (still image) acquired by the endoscope 2 is displayed on the monitor 4 and the user designates a measurement position on the left image based on the image data through the operation unit 6, the measurement point setting unit 181 The two-dimensional coordinate of the designated measurement position is calculated, and the measurement point is set at that position (step S1). Subsequently, a matching process (step S2) is performed to determine corresponding points on the right image based on the image data. Details of the matching process will be described later.

  After the matching process ends, the measurement processing unit 186 calculates three-dimensional coordinates in space based on two-dimensional coordinates of the measurement point and the corresponding point on the image (step S3). Subsequently, the control unit 180 determines whether the setting of the measurement point is completed (step S4). When designation of the measurement position by the user continues, the next measurement point is set in step S1. Further, when designation of the measurement position by the user is completed, the measurement processing unit 186 performs three-dimensional measurement to calculate a distance between two points etc. based on the three-dimensional coordinates calculated in step S3 (step S5) . Subsequently, the display processing unit 184 performs a process of causing the monitor 4 to display an image on which the measurement result calculated by the measurement processing unit 186 is superimposed on a graphic (character) on the image based on the image data (step S6). . As a result, the measurement result is displayed on the monitor 4. When the measurement result is displayed, the measurement process ends.

  FIG. 5 shows the procedure of the matching process (step S2). First, the outline of the matching process will be described. In the matching process, a plurality of corresponding point candidates are calculated by a plurality of matching methods. When the reliability of the plurality of corresponding point candidates is high, the corresponding points are automatically determined in accordance with a predetermined standard that is not related to the user's instruction. In addition, when the reliability of a plurality of corresponding point candidates is low, the user selects a corresponding point candidate suitable for the corresponding point from among the plurality of corresponding point candidates, and the selected corresponding point candidate is determined as a corresponding point. Ru.

  The details of the matching process will be described below. The control unit 180 sets the value of the variable i corresponding to the matching method used to calculate the corresponding point to 0 (step S200). Subsequently, the corresponding point candidate calculation unit 182 calculates two-dimensional coordinates of corresponding point candidates on the right image corresponding to the measurement points set on the left image, using a matching method corresponding to the value of the variable i. (Step S205).

In the present embodiment, three types of matching methods are used as an example. For example, a matching method A for performing template matching, a matching method B for performing matching at feature points such as light and dark edge portions, and a phase only correlation method (POC: Phase Only Correlation) for calculating the correlation of phase components obtained by Fourier transforming an image. The matching method C used is used. In addition to the above matching method, even if a correlation function used in template matching is different (for example, NCC (Normalized Cross-Correlation), SSD (Sum of Squared Difference), SAD (Sum of Absolute Difference), etc.) Good. In addition, methods with different template sizes and shapes may be used. Furthermore, a method may be used in which the resolution of the image to be subjected to template matching is different, and the image processing (binarization, noise removal, various filtering) is different, or a method using color information as well as luminance information of the image. May be used.
The matching method used to calculate the corresponding point candidates is set in advance, but the user may be able to change the matching method.

  In the present embodiment, priority coefficients corresponding to the respective matching methods are provided. For example, the priority coefficient of matching method A is 1.0, the priority coefficient of matching method B is 0.8, and the priority coefficient of matching method C is 0.6. The priority coefficient indicates the magnitude (priority) of the influence exerted by each corresponding point candidate on the process of calculating the reliability of matching and the process of determining the corresponding point among a plurality of corresponding point candidates. . The higher the priority coefficient, the higher the priority. Although the priority coefficient of each matching method is previously set, the user may change the value.

  After the corresponding point candidates are calculated, the control unit 180 determines whether the corresponding point candidates have been calculated using all the matching methods by determining whether the value of the variable i is equal to a predetermined value. (Step S210). When three types of matching methods are used, the predetermined value is two. If there is a matching method that has not calculated corresponding point candidates, the control unit 180 increases the value of the variable i by 1 (step S215). Subsequently, at step S205, calculation of corresponding point candidates is performed again.

  When corresponding point candidates are calculated using all the matching methods, the reliability calculation unit 183 calculates the matching reliability based on the two-dimensional coordinates of the corresponding point candidates corresponding to each of the plurality of matching methods (step S215). The reliability does not indicate the reliability of matching for each matching method but indicates the reliability of the entire matching performed using a plurality of matching methods.

  In step S215, the reliability of matching is calculated as follows. For example, the reliability of matching is calculated based on the distance on the image between the corresponding point candidates. The closer the distance between the corresponding point candidates on the image, the higher the reliability of matching tends to be. More specifically, among the calculated corresponding point candidates, the matching reliability is a corresponding point within a predetermined distance R (for example, 3 pixels) centered on the corresponding point candidate whose parallax amount is a median value. Calculated as the sum of candidate priority coefficients. Assuming that the two-dimensional coordinates of the measurement point are (XL, YL) and the two-dimensional coordinates of the corresponding point candidate are (XR, YR), the parallax amount is XR-XL. Although the value of the predetermined distance R is set in advance, the user may change the value.

  By calculating the reliability of matching as the sum of priority coefficients, when making a determination by setting a threshold to the reliability of matching, it is possible to perform the determination according to the priority of the corresponding point candidate, and more accurately A decision can be made. For example, the sum of priority coefficients of corresponding point candidates calculated using matching methods A, B and C is 2.4, and the sum of priority coefficients of corresponding point candidates calculated using matching methods A and B is 1.8 The sum of priority coefficients of corresponding point candidates calculated using matching methods A and C is 1.6, and the sum of priority coefficients of corresponding point candidates calculated using matching methods B and C is 1.4. When the threshold is 1.5, the sum of the priority coefficients when using matching methods A, B, C, when using matching methods A, B, or when using matching methods A, C is the threshold. The sum of the priority coefficients in the case of using the matching methods B and C having the lower priority coefficient but exceeding the threshold does not exceed the threshold.

  Among the calculated corresponding point candidates, the sum of the priority coefficients of the corresponding point candidates contained within a predetermined distance R centered on the corresponding point candidate having the highest priority coefficient may be used as the matching reliability. As described above, by calculating matching reliability using only corresponding point candidates that are close in distance, the influence of corresponding point candidates that are far in distance, that is, corresponding point candidates that are not likely to be true corresponding points It can be eliminated.

  The priority coefficient may not be used to calculate the matching reliability. For example, the ratio n / of the number n of corresponding point candidates contained within a predetermined distance R centered on the corresponding point candidate whose parallax amount is a median value to the number N of matching methods used for calculation of the corresponding point candidates N may be the matching reliability.

  When the reliability of matching is equal to or more than a predetermined value, the corresponding point determination unit 185 determines a corresponding point based on a plurality of corresponding point candidates according to a predetermined criterion (step S225). In step S225, corresponding points are determined as follows. For example, the two-dimensional coordinates of all corresponding point candidates contained within a predetermined distance R centered on the corresponding point candidate whose parallax amount is the center value obtained when the matching reliability is calculated in step S215 match In this case, the two-dimensional coordinate point is determined as the corresponding point.

  In addition, when the two-dimensional coordinates of all corresponding point candidates within a predetermined distance R centered on the corresponding point candidate whose parallax amount is a median value do not completely match, 2 of those corresponding point candidates A point of two-dimensional coordinates obtained by averaging the dimensional coordinates is determined as a corresponding point. By averaging the two-dimensional coordinates of corresponding point candidates, the position of the corresponding point can be obtained with an accuracy of 1 pixel or less, and a result with higher measurement accuracy can be obtained.

  The two-dimensional coordinates of all the corresponding point candidates contained within the predetermined distance R centered on the corresponding point candidate whose median is the parallax amount are weighted by the priority coefficient, and the weighted two-dimensional coordinates are calculated. A point of averaged two-dimensional coordinates may be used as the corresponding point. Alternatively, the above weighting may be performed using a value indicating the reliability of the matching result using each matching method (for example, a correlation coefficient in the case of template matching) instead of the priority coefficient.

  The corresponding point candidate obtained using the matching method with the highest priority coefficient may be used as the corresponding point. Alternatively, a corresponding point candidate having the highest value (for example, a correlation coefficient in the case of template matching) indicating the reliability of the result of matching performed using each matching method may be used as the corresponding point.

  If the reliability of matching is less than a predetermined value, the display processing unit 184 generates a graphic image signal for displaying the measurement point and the plurality of corresponding point candidates, and outputs the graphic image signal to the video signal processing circuit 12. The video signal processing circuit 12 outputs to the monitor 4 a display signal obtained by combining the graphic image signal and the video signal from the CCU 9. The monitor 4 displays an image based on the display signal to display an image in which the measurement point and the plurality of corresponding point candidates are superimposed (step S235).

  FIG. 6 shows an example of an image in which measurement points and a plurality of corresponding point candidates are superimposed. In the image 600 displayed on the screen of the monitor 4, the measurement point PL specified by the user is displayed in the left image 600L, and in the right image 600R, the corresponding point candidate PR1 corresponding to the matching method A and the matching method B Corresponding corresponding point candidate PR2 and corresponding point candidate PR3 corresponding to the matching method C are displayed. Each corresponding point candidate is displayed in a different display form for each matching method used to calculate the corresponding point candidate. In the example shown in FIG. 6, each corresponding point candidate is displayed in a different shape for each matching method. Each corresponding point candidate may be displayed in a different color for each matching method used to calculate the corresponding point candidate.

  In addition, each corresponding point candidate may be displayed in a different display form for each priority coefficient of the matching method used to calculate the corresponding point candidate. For example, the corresponding point candidates may be displayed in a size corresponding to the priority coefficient so that the corresponding point candidate is displayed larger as the priority coefficient is larger. Alternatively, only corresponding point candidates having large priority coefficients may be displayed in different colors, or only corresponding point candidates having large priority coefficients may be highlighted to blink.

  When corresponding point candidates are calculated using template matching, each corresponding point candidate may be displayed in a display form according to the correlation coefficient. Alternatively, each corresponding point candidate may be displayed in a display form according to the amount of deviation from each corresponding point candidate's epipolar line.

  Each corresponding point candidate may be displayed in a display form according to the characteristics of the matching method. For example, in template matching, there is a characteristic that the reliability of matching for an object having a step in a direction parallel to the optical axis of the optical system is relatively low, and the reliability of matching for an object having a pattern with low periodicity is relatively high. There is. Also, in matching based on feature points such as dark and light edge portions, the reliability of matching for an object having a step in the direction parallel to the optical axis of the optical system is relatively high and does not have a distinctive feature pattern There is a characteristic that the reliability of matching for the subject is relatively low. For example, according to the user's instruction, only corresponding point candidates calculated using template matching are emphasized and displayed, or only corresponding point candidates calculated using matching by feature points are emphasized and displayed. It is also good.

  After the plurality of corresponding point candidates are displayed, the user selects, through the operation unit 6, corresponding point candidates considered to be correct. When one of the corresponding point candidates is selected, the corresponding point determination unit 185 determines the selected corresponding point candidate as a corresponding point (step S240). As described above, by displaying the corresponding point candidates in various display forms, the user can confirm the certainty of the corresponding points with the multi-directional index, and can select the corresponding point candidates more accurately.

  After the corresponding points are determined in steps S225 and S240, the display processing unit 184 generates a graphic image signal for displaying the determined corresponding points, and outputs the graphic image signal to the video signal processing circuit 12. The video signal processing circuit 12 outputs to the monitor 4 a display signal obtained by combining the graphic image signal and the video signal from the CCU 9. The monitor 4 displays an image based on the display signal to display an image in which the measurement point and the corresponding point are superimposed (step S230).

  FIG. 7 shows an example of an image in which measurement points and corresponding points are superimposed. In the image 600 displayed on the screen of the monitor 4, the measurement point PL specified by the user is displayed in the left image 600L, and the determined corresponding point PR is displayed in the right image 600R. When the corresponding points are displayed, the matching process ends.

  In the present embodiment, by using a plurality of matching methods, it becomes easy to obtain matching results that do not cause incorrect matching of matching. Depending on the matching method, even when an image in which a matching error is likely to occur is acquired, if the matching error is not generated using a matching method suitable for the image, the image does not have to be reworked. In addition, when the matching reliability is high, the corresponding point is automatically determined, so that the time and effort of the user does not increase. In addition, when the reliability of matching is low, corresponding point candidates specified by the user among corresponding point candidates corresponding to each of a plurality of matching methods are determined as corresponding points. Time and effort of the user are reduced. Therefore, it is possible to reduce the error in matching and reduce the time and effort of the user.

  Next, a modification of this embodiment will be described.

(First modification)
The first modification relates to a method of displaying a plurality of corresponding point candidates in step S235. In the first modification, three corresponding point candidates are calculated using three types of matching methods, and the process after displaying the three corresponding point candidates in step S235 will be described. FIG. 8 shows an example of the image displayed in step S235. Measurement points PL and three corresponding point candidates PR1, PR2 and PR3 are displayed. None of the corresponding point candidates is at a position exactly corresponding to the position of the measurement point PL. The user can increase the corresponding point candidates by inputting an instruction via the operation unit 6.

  When an instruction to calculate a new corresponding point candidate is input, the corresponding point candidate calculation unit 182 calculates a new corresponding point candidate using a matching method different from the matching method already used. At this time, a plurality of matching point candidates may be newly calculated using a plurality of matching methods. The display processing unit 184 generates a graphic image signal for displaying the newly calculated corresponding point candidate, and outputs the graphic image signal to the video signal processing circuit 12. Thereafter, the image is displayed on the screen of the monitor 4 in the same manner as the process described above.

  FIG. 9 shows an example of an image including newly calculated corresponding point candidates. Compared with the image shown in FIG. 8, corresponding point candidates PR4, PR5 and PR6 are added. It can be seen that the newly calculated corresponding point candidate PR4 is at a position corresponding to the measurement point PL. In this case, when the user selects the corresponding point candidate PR4, the corresponding point determination unit 185 determines the selected corresponding point candidate PR4 as a corresponding point.

  New corresponding point candidates may be calculated as follows. In the matching method for performing template matching, the point with the highest correlation coefficient is the corresponding point candidate. When the corresponding point candidate is calculated by template matching in step S205, the corresponding point candidate calculation unit 182 sets one or more high-order correlation coefficients calculated at that time as a new corresponding point candidate.

  By increasing the corresponding point candidates as described above, the corresponding point candidate at the position exactly corresponding to the position of the measurement point is more likely to be calculated, and the user can recalculate the image or manually correct the corresponding point. Is less likely to occur. After displaying a plurality of corresponding point candidates in step S235, the corresponding point candidates may be newly calculated when the user instructs to calculate a new corresponding point candidate, or in step S205, display is performed in step S235. The corresponding point candidates other than the corresponding point candidates are also calculated in advance, and after displaying a plurality of corresponding point candidates in step S235, a new corresponding point candidate is displayed when the display of a new corresponding point candidate is instructed by the user. You may display it. Further, calculation and display of new corresponding point candidates may be repeatedly performed based on the user's instruction.

  As a result of increasing the number of corresponding point candidates as described above, if the number of corresponding point candidates increases too much, it becomes troublesome for the user to select corresponding point candidates. Therefore, the following may be performed.

  When a plurality of corresponding point candidates overlap (when the two-dimensional coordinates of a plurality of corresponding point candidates match or when the distance on the image is equal to or less than a predetermined distance), the user indicates that the corresponding point candidates have high reliability In order to inform them, overlapping corresponding point candidates may be highlighted. For example, the corresponding point candidate calculation unit 182 calculates the distances between all the corresponding point candidates including the plurality of corresponding point candidates displayed in step S235 and the new corresponding point candidate, and whether or not there are overlapping corresponding point candidates. Determine if When a plurality of corresponding point candidates overlap, the display processing unit 184 generates a graphic image signal for emphasizing and displaying the overlapping corresponding point candidates, and outputs the graphic image signal to the video signal processing circuit 12. Thereafter, the image is displayed on the screen of the monitor 4 in the same manner as the process described above.

  FIG. 10 shows an example of an image including newly calculated corresponding point candidates. Compared with the image shown in FIG. 8, corresponding point candidates PR4, PR5 and PR6 are added, corresponding point candidates PR4 and PR5 overlap, and are displayed larger than other corresponding point candidates. A method of emphasizing and displaying a plurality of corresponding point candidates overlapped may be another method, and a method of displaying in different colors, blinking, or the like may be used.

  As described above, since the overlapping corresponding point candidates are emphasized, the user can easily identify highly reliable corresponding point candidates. Therefore, even if the number of corresponding point candidates increases, the user can easily select corresponding point candidates.

  Among the corresponding point candidates newly calculated as described above, the corresponding point candidates (two-dimensional coordinates are the same) overlap with any of the plurality of corresponding point candidates displayed in step S235 and the other newly calculated corresponding point candidates. Only matching corresponding point candidates or matching point candidates whose distance on the image is equal to or less than a predetermined distance may be displayed, and other corresponding point candidates may not be displayed. That is, although the plurality of corresponding point candidates displayed in step S235 continue to be displayed as they are, the newly calculated corresponding point candidate is displayed only when overlapping with any corresponding point candidate.

  For example, the corresponding point candidate calculation unit 182 calculates the distance between the newly calculated corresponding point candidate and the plurality of corresponding point candidates displayed in step S235 and the other newly calculated corresponding point candidates. It is determined whether the newly calculated corresponding point candidate overlaps with any corresponding point candidate. If the newly calculated corresponding point candidate overlaps any corresponding point candidate, the display processing unit 184 generates a graphic image signal for displaying the overlapping corresponding point candidate, and outputs the graphic image signal to the video signal processing circuit 12 Do. Thereafter, the image is displayed on the screen of the monitor 4 in the same manner as the process described above.

  As described above, by displaying the newly calculated corresponding point candidate only when overlapping with any corresponding point candidate, the user can easily select the corresponding point candidate even if the number of corresponding point candidates increases. Become. When a new corresponding point candidate is repeatedly calculated and displayed, when the newly calculated corresponding point candidate is displayed, the newly calculated corresponding point candidate, the corresponding point candidate being displayed, and the newly calculated corresponding point candidate are displayed. It is sufficient to calculate the distance between the other corresponding point candidate and to determine whether the newly calculated corresponding point candidate overlaps any corresponding point candidate.

  An upper limit may be set to the number of corresponding point candidates to be displayed, and a predetermined number of corresponding point candidates may be displayed. For example, the corresponding point candidate calculation unit 182 determines whether the total number of the plurality of corresponding point candidates displayed in step S235 and the newly calculated corresponding point candidates is equal to or more than a predetermined number. If the sum is less than the predetermined number, all corresponding point candidates including the newly calculated corresponding point candidate are displayed. If the total is equal to or greater than the predetermined number, the display processing unit 184 selects, from among the plurality of corresponding point candidates displayed in step S235, the number of corresponding point candidates equal to the difference between the above sum and the predetermined number. Is not displayed, a graphic image signal for displaying the newly calculated corresponding point candidate is generated, and is output to the video signal processing circuit 12. Thereafter, the image is displayed on the screen of the monitor 4 in the same manner as the process described above. The upper limit of the number of corresponding point candidates to be displayed may be set by the user.

  As described above, by keeping the number of corresponding point candidates displayed constant, even if the number of corresponding point candidates increases, it is possible to prevent a decrease in image visibility, and the user selects a corresponding point candidate. It becomes easy to do. When new corresponding point candidates are repeatedly calculated and displayed, the corresponding point candidates being displayed are displayed so that the number of corresponding point candidates displayed is constant when displaying the newly calculated corresponding point candidates. (For example, the oldest corresponding point candidate, the corresponding point candidate with the smallest correlation coefficient, the candidate with a large amount of deviation from the epipolar line, or the like) may be hidden.

(Second modification)
The second modification relates to a method of displaying a plurality of corresponding point candidates in step S235. In the second modification, five corresponding point candidates are calculated using five types of matching methods, and processing after displaying five corresponding point candidates in step S235 will be described. FIG. 11 shows an example of the image displayed in step S235. A measurement point PL and five corresponding point candidates PR1, PR2, PR3, PR4, and PR5 are displayed. The three corresponding point candidates PR1, PR2, PR4 are at the same position. In FIG. 11, since three corresponding point candidates overlap, the visibility of the image is reduced. For this reason, when a plurality of corresponding point candidates overlap (when the two-dimensional coordinates of a plurality of corresponding point candidates match or when the distance on the image is equal to or less than a predetermined distance), the corresponding display candidates have the same display form It may be displayed by.

  For example, the corresponding point candidate calculation unit 182 calculates the distance on the image between the plurality of corresponding point candidates, and determines whether the corresponding point candidates overlap. When there is a plurality of overlapping corresponding point candidates, the display processing unit 184 generates a graphic image signal for displaying the overlapping corresponding point candidates with the same graphic (mark), and outputs the graphic image signal to the video signal processing circuit 12. Thereafter, the image is displayed on the screen of the monitor 4 in the same manner as the process described above.

  FIG. 12 shows an example of an image including overlapping corresponding point candidates. In FIG. 12, a mark M1 indicating three overlapping corresponding point candidates, a mark M2 indicating corresponding point candidate PR3 of FIG. 11, and a mark M3 indicating corresponding point candidate PR5 of FIG. 11 are displayed. The three overlapping corresponding point candidates are displayed as the same mark M1, and the mark M1 is displayed and emphasized as a thicker line than the lines constituting the marks M2 and M3. Further, in the vicinity of each mark, the number of corresponding point candidates corresponding to the mark is displayed. For example, "3" is displayed in the vicinity of the mark M1 indicating three corresponding point candidates, and "1" is displayed in the vicinity of the marks M2 and M3 indicating one corresponding point candidate.

  As described above, since the overlapping corresponding point candidates are displayed with the same mark, it is possible to reduce the reduction in the visibility of the image. In addition, since the number of corresponding point candidates is displayed in the vicinity of the mark, it is possible to make it easy for the user to understand whether corresponding point candidates overlap or not, that is, which corresponding point candidate has high reliability.

(Third modification)
The third modification relates to a method of calculating the priority coefficient of the matching method. The priority coefficient may be calculated based on the calculation result of past corresponding point candidates. For example, when the reliability calculation unit 183 calculates corresponding point candidates using a plurality of matching methods, among the calculated corresponding point candidates, a predetermined distance R centered on the corresponding point candidate whose parallax amount is a median value It is considered that matching is successful for corresponding point candidates contained in (for example, 3 pixels), and that matching is failed for other corresponding point candidates. The reliability calculation unit 183 holds the result of success or failure of matching for each of the plurality of matching methods, and based on the calculation result of the corresponding point candidates for the plurality of measurement points, the plurality of matching methods Calculate each priority coefficient.

  For example, when corresponding point candidates for measurement points of ten or more points are calculated, the reliability calculation unit 183 calculates a priority coefficient for each matching method. The priority coefficient is calculated as a matching success rate, which is a ratio of the number of successful matchings to the number of corresponding point candidates calculated.

  As described above, by calculating the priority coefficient based on the calculation result of the past corresponding point candidate, the past matching result is reflected in the calculation of the reliability and the determination of the corresponding point, and the reliability of the matching is improved. be able to.

  The priority coefficient may be calculated for each condition that affects the matching success rate using the above method. Conditions that affect the success rate of matching are the measurement object, conditions based on image data, and the like. The measurement target may be a specific subject or a specific area in the subject. Conditions based on image data are an object distance, a texture value, a luminance value, and the like. The condition based on the image data may be a condition as to whether or not halation is included in the image. The reliability calculation unit 183 calculates and manages a priority coefficient for each of the plurality of matching methods for each of the conditions described above. That is, for each of the plurality of matching methods, the priority coefficient for each measurement object, the priority coefficient for each object distance, the priority coefficient for each texture value, and the priority coefficient for each luminance value are calculated.

  For example, when the corresponding point candidate is calculated using a plurality of matching methods, the reliability calculation unit 183 determines whether the matching has succeeded or failed for each matching method as described above. The reliability calculation unit 183 holds, for each matching method, the result of success / failure of the matching in association with the information of the measurement object at that time. Alternatively, the reliability calculation unit 183 calculates one of the object distance, the texture value, and the luminance value based on the image data used to calculate the corresponding point candidate at that time, and calculates the calculated value (or calculation) for each matching method. The result of matching success / failure is held in association with a predetermined range (including a predetermined value). In the case of calculating the object distance, since it is necessary to calculate three-dimensional coordinates, the process performed in step S3 of FIG. 4 is performed in the matching process in step S2.

  When calculating the priority coefficient of each matching method, the reliability calculation unit 183 performs matching for the same measurement target, or for the same object distance, texture value, and luminance value (or for the same range regarding them). A priority coefficient is calculated for each condition based on the result of success / failure. For example, when performing measurement of a specific measurement target, the user repeatedly sets measurement points for the same measurement target in advance to calculate corresponding point candidates, and the respective priority coefficients of the plurality of matching methods Calculate the

  As described above, by providing the priority coefficient for each condition, the reliability of matching can be improved.

  As described above, according to this embodiment, a plurality of matching point candidates are calculated using a plurality of matching methods, and when the matching reliability is low, the corresponding point candidates selected by the user are determined as the corresponding points. If the matching reliability is high, by determining corresponding point candidates automatically selected from a plurality of corresponding point candidates as corresponding points, it is possible to reduce erroneous matching of the matching and reduce the time and effort of the user. .

  In addition, by calculating the reliability of matching or determining the corresponding point using the priority coefficient of the matching method, the result of the matching by the specific matching method is made larger with respect to the reliability or the corresponding point of matching. It can be reflected. For example, in the case of using an image obtained by capturing a specific measurement target, the matching result suitable for the specific measurement target can be more largely reflected with respect to the matching reliability or the corresponding point.

  In addition, by calculating the priority coefficient of the matching method based on the calculation result of the past corresponding point candidate, the past matching result is reflected in the calculation of the matching reliability and the determination of the corresponding point, and the matching reliability Can be improved. Furthermore, the reliability of matching can be improved by providing a priority coefficient for each condition such as measurement target, object distance, texture value, and luminance value when corresponding point candidates are calculated.

  Moreover, when displaying the several corresponding point candidate calculated using the several matching method, the fall of the visibility of an image can be reduced by displaying the overlapping corresponding point candidate by the same display form. Furthermore, by displaying the number of overlapping corresponding point candidates in the vicinity of the corresponding point candidates, it is possible to make it easy for the user to understand whether corresponding point candidates overlap or not, that is, which corresponding point candidate has high reliability. Can.

  Further, when displaying a plurality of corresponding point candidates calculated using a plurality of matching methods, displaying corresponding point candidates in a display form corresponding to the priority coefficient corresponding to each of the plurality of matching methods is displayed to the user. On the other hand, it is possible to more clearly indicate the matching result by a specific matching method. For example, in the case of using an image obtained by photographing a specific measurement target, the user can obtain a corresponding point with higher reliability by displaying larger corresponding point candidates calculated using a matching method suitable for the specific measurement target. It becomes easy to select a candidate.

  In addition, after calculation of a plurality of corresponding point candidates using a plurality of matching methods, when calculation of a new corresponding point candidate is instructed by the user, a new matching method using a different matching method from the plurality of matching methods is used. By calculating corresponding point candidates, it is possible to reduce the time and effort of the user, such as retaking an image or manually correcting the corresponding points.

  Further, when displaying the newly calculated corresponding point candidate, the number of corresponding point candidates can be obtained by displaying only corresponding point candidates overlapping with other corresponding point candidates among the newly calculated corresponding point candidates. Even if the number is increased, the user can easily select corresponding point candidates.

  In addition, when displaying the newly calculated corresponding point candidate, by keeping the number of displayed corresponding point candidates constant, it is possible to prevent the decrease in the visibility of the image, and the user selects the corresponding point candidate. It becomes easy to do.

  As mentioned above, although the embodiment of the present invention has been described in detail with reference to the drawings, the specific configuration is not limited to the above embodiment, and design changes within the scope of the present invention are also included. .

  DESCRIPTION OF SYMBOLS 1 endoscope apparatus, 2 endoscope, 3 apparatus main body, 4 monitors (display part), 5 housings, 6 operation parts, 8 endoscope units, 9 CCU, 10 control units, 12 video signal processing circuit, 13 ROM, 14 RAM, 18 CPU, 28 imaging device (imaging unit), 180 control unit, 181 measurement point setting unit, 182 correspondence point candidate calculation unit, 183 reliability calculation unit, 184 display processing unit, 185 correspondence point determination unit, 186 Measurement processing unit

Claims (10)

An imaging unit configured to capture a first subject image and a second subject image corresponding to the same subject to generate image data;
A display unit for displaying an image based on the image data;
A corresponding point candidate calculation unit that calculates a plurality of corresponding point candidates that are points on the second object image corresponding to the measurement points set on the first object image using a plurality of matching methods ;
A reliability calculation unit that calculates the reliability of matching based on the corresponding point candidate corresponding to each of the plurality of matching methods;
A display processing unit configured to display, on the image, the measurement point and a plurality of corresponding point candidates corresponding to the plurality of matching methods when the reliability is less than a predetermined value ;
A corresponding point determination unit which determines, as a corresponding point, a candidate for the corresponding point designated among the plurality of corresponding point candidates corresponding to the plurality of matching methods when the reliability is less than a predetermined value When,
A measurement processing unit that measures the size of the subject based on the measurement point and the determined corresponding point;
An endoscope apparatus comprising:   The endoscope apparatus according to claim 1, further comprising an operation unit for instructing a user to increase the number of corresponding point candidates displayed on the image. The plurality of candidates for the corresponding points are
The endoscope apparatus according to claim 2, further comprising a candidate of corresponding points added by an instruction to increase the number of corresponding point candidates input by the user via the operation unit. The corresponding point candidate calculation unit
The endoscope apparatus according to claim 3, wherein a candidate for the corresponding point to be added is calculated after an instruction from the user to increase the number of the corresponding point candidates to be displayed on the image. The corresponding point candidate calculation unit
The endoscope apparatus according to claim 3, wherein a candidate for the corresponding point to be added is calculated in advance before the user instructs to increase the number of the corresponding point candidates to be displayed on the image. The endoscope according to claim 1, wherein the display processing unit performs a process of displaying the overlapping corresponding point candidate among the plurality of corresponding point candidates as one mark on the image. apparatus.   The endoscope apparatus according to claim 6, wherein the display processing unit performs a process of displaying the number of overlapping corresponding point candidates in the vicinity of the corresponding point candidates. Each of the plurality of corresponding point candidates has a priority coefficient,
The endoscope according to claim 1, wherein the display processing unit performs a process of displaying on the image in a display form according to the priority coefficient corresponding to each of a plurality of candidates for the corresponding points. apparatus. In order to keep the number of corresponding point candidates displayed in the image constant,
4. The display processing unit according to claim 3, wherein, when displaying the candidate for the corresponding point to be additionally displayed on the image, some of the candidates for the corresponding point already displayed are not displayed. Endoscope apparatus as described. The second corresponding to the measurement point set on the first subject image of the image based on the image data generated by imaging the first subject image and the second subject image corresponding to the same subject Calculating a plurality of corresponding point candidates that are points on the subject image using a plurality of matching methods ;
Calculating the reliability of matching based on the corresponding point candidate corresponding to each of the plurality of matching methods;
Performing a process of displaying the measurement point and a plurality of corresponding point candidates corresponding to each of the plurality of matching methods on the image when the reliability is less than a predetermined value ;
Determining the corresponding point candidate designated among the plurality of corresponding point candidates corresponding to each of the plurality of matching methods as the corresponding point, when the reliability degree is less than a predetermined value ;
Measuring the size of the subject based on the measurement point and the determined corresponding point;
A program to make a computer run.