JP6752468B2 - 3D shape measuring device and 3D shape measuring method - Google Patents

Description

The present invention relates to a three-dimensional shape measuring device and a three-dimensional shape measuring method.

Conventionally, an endoscope capable of inserting a long endoscope insertion portion into a lumen in a body cavity to perform diagnosis and processing of a measurement target has been used (see, for example, Patent Document 1). In endoscopic diagnosis, it is particularly important to measure the shape and size of a lesion such as a tumor as a measurement target in selecting a treatment method. Endoscopic measures and visual measurements can be time consuming and can lead to estimation errors due to human factors.

Therefore, in order to measure the shape and size of the observation site, a three-dimensional endoscope system based on the active stereo method for measuring the three-dimensional shape by the principle of triangulation has been developed (for example, Non-Patent Document 1). reference). In this system, the three-dimensional shape of the observation site and the like are measured based on the imaging result of the pattern image formed on the measurement target.

Japanese Unexamined Patent Publication No. 2011-240341

H. Aoki, R. Furukawa, M. Aoyama, S. Hiura, N. Asada, R. Sagawa, H. Kawasaki, S. Tanaka, S. Yoshida, and Y. Sanomura, Proposal on 3-d endoscope by using grid -based active stereo ”in EMBC, 2013, pp. 5694-5697.

In the above system, there is an inconvenience that the measurement range of depth is limited due to blurring due to defocus and aberration of the imaging optical system that forms a pattern image on the measurement target. In addition, strong subsurface scattering common to internal living tissues may occur, which not only blurs the projected pattern image but also weakens the brightness of the pattern light. As a result, the measurement may become unstable.

The depth of focus of a projector is generally deeper than the depth of focus of a camera. For example, at a distance of about 40 mm from the projector, the range that can be projected in a focused state is about 8 mm. Further, since the sensitivity of an endoscope camera is limited due to the size of the lens aperture, insufficient brightness of the pattern light becomes a serious problem in stable measurement of the measurement target.

The present invention has been made in view of the above circumstances, and provides a three-dimensional shape measuring device and a three-dimensional shape measuring method capable of stably measuring a three-dimensional shape of a measurement target or its size. With the goal.

In order to achieve the above object, the three-dimensional shape measuring device according to the first aspect of the present invention is
A light projecting portion for projecting a two-dimensional pattern formed on the measurement object from the graph consisting of nodes and edges with thus distinguishable features the line connection state,
An imaging unit that captures the two-dimensional pattern projected onto the measurement target, and
For the two-dimensional pattern and an image captured by the imaging unit, select a candidate of the solution of the node corresponding to each other on the basis of the epipolar constraint, based on the connection state in the graph of candidates among the solutions, the A correspondence calculation unit that eliminates the error of the two-dimensional pattern included in the image and obtains the correct correspondence solution from the solution candidates.
A shape output unit that calculates and outputs the three-dimensional shape of the measurement target by stereo processing based on the corresponding solution, and
Equipped with a,
The corresponding calculation unit
By performing pattern matching in the subgraph of the graph of the two-dimensional pattern and associating the node included in the image with the node in the two-dimensional pattern, the candidates for the corresponding solution are reduced.
The two-dimensional pattern is a pattern of a graph structure composed of vertical and horizontal line segments, and each node is characterized by a step of left and right horizontal line segments connected to a vertical straight line .

The corresponding calculation unit
Wherein the degree of matching of the pattern matching and data sections by the connection relationship of the graph and smoothing term, and calculates the corresponding points by Markov random field model (MRF) optimization,
It may be that.

The corresponding calculation unit
For said nodes of said subgraph, it said that the degree of coincidence of the pattern matching to vote is higher than the threshold value, obtaining a correct corresponding points from the corresponding candidate solutions,
It may be that.

The corresponding calculation unit
The subgraphs preparing a plurality to determine a solution of the corresponding by the total number of votes repeated votes in each of the subgraphs,
It may be that.

The shape output unit
The three-dimensional shape is calculated by using the optical cutting method from the correspondence relationship of the line segments based on the corresponding solution.
It may be that.

The series of line segments included in the two-dimensional pattern is
Arranged in a curved line,
It may be that.

The two-dimensional pattern is
Formed on the measurement target by diffraction of laser light by a diffraction optical element,
It may be that.

The light projecting unit is incorporated in the endoscope.
It may be that.

If it is measured from different directions, data of a plurality of three-dimensional shape of the measurement object is obtained,
Of the plurality of three-dimensional shape data, the first line segment constituting the first data and the first line segment constituting the second data different from the first data are that minimizes the deviation between the direction different second segment, a calculation unit for calculating a correction amount of the rotation and parallel proceeds,
By the correction amount of the rotational and translational calculated by the calculation unit, and an output unit for outputting the corrected shape wherein the first data second displacement data,
To prepare
It may be that.

The three-dimensional shape measuring method according to the second aspect of the present invention is
A three-dimensional shape measuring method executed by a three-dimensional shape measuring device.
The two-dimensional pattern consisting of the graph consisting of nodes and edges with thus distinguishable features the line connection state is projected onto the target object, imaging the two-dimensional pattern which is projected to the measurement target Pattern imaging step and
Wherein for said 2-dimensional pattern image captured by the pattern imaging step, selects a candidate solution of the node corresponding to each other on the basis of the epipolar constraint, based on the connection state of the graph of candidates among the solutions, the A correspondence calculation step that eliminates the error of the two-dimensional pattern included in the image and obtains the correct correspondence solution from the solution candidates.
A shape output step that calculates and outputs the three-dimensional shape of the measurement target by stereo processing based on the corresponding solution, and
Only including,
In the corresponding calculation step,
By performing pattern matching in the subgraph of the graph of the two-dimensional pattern and associating the node included in the image with the node in the two-dimensional pattern, the candidates for the corresponding solution are reduced.
The two-dimensional pattern is a pattern of a graph structure composed of vertical and horizontal line segments, and each node is characterized by a step of left and right horizontal line segments connected to a vertical straight line .

Measured from different directions, when the data of a plurality of three-dimensional shape of the measurement object is obtained, among the data of the plurality of three-dimensional shape, the first segment constituting the first data, constituting the second data different from said first data, said that the first line segment to minimize the deviation between the second line segments of different directions, rotation and parallel -ary Calculation steps to calculate the correction amount and
By the correction amount of the rotation and translation calculated in the calculation step, and outputting the corrected shape wherein the first data second displacement data,
including,
It may be that.

According to the present invention, the two-dimensional pattern projected onto the measurement target is composed of a graph consisting of nodes and edges having characteristics that can be distinguished by the connection state of line segments or the symmetry of the pattern. Even if scattering or a decrease in brightness occurs, the position information of the node can be easily read. As a result, the three-dimensional shape of the measurement target or its size can be stably measured.

It is a schematic diagram which shows the structure of the 3D shape measuring apparatus which concerns on embodiment of this invention. It is an enlarged cross-sectional view around the forceps hole of the insertion end of an endoscope. FIG. 3A is a diagram showing an example of a measurement pattern. FIG. 3B is a diagram showing an example of a code label assigned to each node of the measurement pattern. It is a figure which shows the state that the image of the measurement pattern is projected on the measurement target. It is a block diagram which shows the whole structure of the 3D shape measuring apparatus. 6 (A) and 6 (B) are schematic views showing a light cutting method. It is a flowchart of 3D shape measurement operation of 3D shape measurement apparatus. It is a flowchart of a subroutine of line detection. It is a flowchart of a subroutine for constructing a grid graph. It is a figure which shows an example of a grid graph. It is a schematic diagram which shows the state of performing the correspondence by the subgraph pattern. It is a schematic diagram which shows the voting operation (the 1) by a partial graph pattern. It is a schematic diagram which shows the voting operation (the 2) by a partial graph pattern. It is a schematic diagram which shows the mode of associating with different subgraph patterns. It is a figure which shows the measurement result of the biological specimen excised from the human stomach. It is a schematic diagram which shows the flow of the synthesis processing of a three-dimensional shape. It is a figure which shows the structure which performs the synthesis processing of a three-dimensional shape. It is a figure which shows an example of the measurement pattern which expressed the grid by the image which consists of a rectangular point. It is a figure which shows an example of a grid point and a grid line in the measurement pattern of FIG. It is a figure which shows an example of the image photographed by projecting the measurement pattern of FIG. It is an enlarged view of the part of the white frame of FIG. It is the figure which obtained the autocorrelation by the rotation of 180 degrees at each point of FIG. 21, and obtained the 180 degree rotation symmetry of each point by this, and imaged. It is the figure which detected the lattice point by finding the peak of the rotational symmetry of FIG. 22, and imaged the lattice point.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In each figure, the same elements are designated by the same reference numerals.

As shown in FIG. 1, the three-dimensional shape measuring device 100 according to the present embodiment includes a laser light source 10 and a pattern projector 20 as a light projecting unit that projects a measurement pattern 21 (see FIG. 3) on a measurement target. And. The three-dimensional shape measuring device 100 is used together with the endoscope 1. A forceps hole 2 is formed in the endoscope 1. Normally, a forceps for collecting an observation site is inserted through the forceps hole 2. However, in the present embodiment, the pattern projector 20 is inserted into the forceps hole 2 and used instead of the forceps.

The pattern projector 20 is connected to the laser light source 10. The laser light source 10 includes a laser module 10A, a diffuser plate 10B, an illumination diaphragm 10C, and a collimator lens 10D.

The laser module 10A oscillates and outputs a green laser beam IL (for example, a laser beam having a wavelength of 517 mm). The diffuser plate 10B diffuses the laser light IL emitted from the laser module 10A to make the intensity in the cross section of the laser light IL uniform. The collimator lens 10D converts the incident laser light IL into parallel light. The laser beam IL converted into parallel light by the collimator lens 10D is incident on the optical fiber 20A constituting the pattern projector 20.

As shown in FIG. 2, the pattern projector 20 includes an optical fiber 20A, a green lens 20B, a pattern chip 20C, and a cylindrical portion 20D. The size of the pattern projector 20 is, for example, 2.8 mm in diameter and 12 mm in length.

The optical fiber 20A is a single-mode optical fiber made of plastic or glass. The optical fiber 20A incidents the green laser light IL emitted from the collimator lens 10D from one end thereof and emits it from the other end.

The green lens 20B collects the laser beam emitted from the optical fiber 20A and projects it onto the biological tissue inside the target human body. The pattern chip 20C is an optical element called a DOE (Diffractive Optional Element). The DOE forms a pattern by diffracting the passing laser light. Therefore, the light loss can be reduced as compared with the mask pattern by the film, and the loss rate of light energy is, for example, 5% or less. Since the DOE can project a clear pattern regardless of the distance, the measurement range can be wider than that of the imaging optical system. Further, since the light efficiency of DOE is 90% or more, the decrease in the brightness of the pattern projected on the measurement target is suppressed. The size of the projected pattern is about 30 mm ² at a distance of 30 mm.

This optical element is installed on the optical path of the laser beam IL emitted from the green lens 20B. Therefore, the laser beam IL projected on the living tissue is the projected light including the image of the measurement pattern 21 (two-dimensional pattern) shown in FIG. As shown below, the measurement pattern 21 is composed of a graph consisting of nodes and edges having characteristics that can be distinguished by the connection state of the line segment.

The reflection characteristics of the surface of living tissue are significantly different from those of ordinary objects. In particular, the influence of subsurface scattering is large. Conventionally, a corrugated grid pattern has been used as the measurement pattern 21, but the curvature of wavy lines at grid points may be difficult to detect due to the loss of high-frequency components due to subsurface scattering, or information may be lost. May be done. In order to avoid such loss of detailed information, a pattern composed of a lower frequency image signal is adopted as the measurement pattern 21.

Conventionally, low-frequency patterns include low-density dot patterns and patterns that combine sparse lines, but in the dot pattern, the baseline length L of the camera of the endoscope 1 and the pattern projector 20 (see FIG. 6). ) Is long and the amount of deformation of the pattern is large, it becomes difficult to correspond the pattern with the observed image. Further, if the lines become sparse, the number of embedded information for measuring the three-dimensional shape decreases, and the measured three-dimensional shape has to be roughened.

Therefore, in the present embodiment, the measurement pattern 21 is a pattern including a line pattern having a large interval, which is relatively robust against blurring due to surface scattering. As shown in FIG. 3, the measurement pattern 21 is a grid pattern composed of vertical and horizontal lines. In the measurement pattern 21, all the line segments (vertical line segments) in the vertical direction are continuous and arranged in a straight line, and are vertically arranged in parallel at equal intervals as a whole. On the other hand, in the horizontal line segment (horizontal line segment), a small step is added between the adjacent horizontal line segments (left and right horizontal line segments connected to the vertical straight line) at each grid point. The size of the step includes 0. Since the position of the line segment longer than the size of the blur is not easily lost due to subsurface scattering, the step information can be stably acquired.

Under such a configuration, as shown in FIG. 3B, three types of code elements (code labels) of S, L, and R are assigned to each grid point. S means that the end points of the left and right horizontal lines are continuous. Further, L means that the end point of the horizontal line on the left side is higher than the end point of the horizontal line on the right side. On the contrary, R means that the end point of the horizontal line on the right side is higher than the end point of the horizontal line on the left side. FIG. 3B shows an array of these code elements (code labels). In this embodiment, each node is coded by a step of a horizontal line on the left and right connected to a vertical straight line. The measurement pattern 21 is a pattern having a graph structure in which the nodes are characterized by the steps of the horizontal lines on the left and right.

In the present embodiment, it is assumed that the measurement pattern 21 is limited in terms of the positional relationship between the camera of the endoscope 1 and the pattern projector 20. The limitation is that when the epipolar line of the intersection observed by the camera of the endoscope 1 is drawn on the pattern image, the direction of the line does not match the direction of the vertical straight line of the lattice pattern. This is because the detected vertical line shape (three-dimensional shape) is restored by the optical cutting method. If the epipolar lines on the pattern image match the vertical lines of the grid pattern, the three-dimensional shape cannot be restored. From the viewpoint of restoration accuracy, the direction of the epipolar line on the pattern image should be close to the horizontal direction, but restoration is possible even in the oblique direction without any problem.

In the present embodiment, as shown in FIG. 3B, the horizontal lines extending in the lateral direction are tilted at different angles in each row. A series of a series of horizontal lines connected (connected) in the horizontal direction are sinusoidal wavy lines as a whole when there is no step. That is, the overlap of the horizontal lines included in the measurement pattern 21 is arranged in a curved line. This is because if the horizontal lines are connected in a straight line, when the epipolar line of the intersection detected in the camera image matches the straight line, a large number of corresponding point candidates may occur, which may lead to a decrease in restoration accuracy. Is.

Returning to FIG. 2, the cylindrical portion 20D is a cylindrical member. The cylindrical portion 20D protrudes from the insertion end 3 of the endoscope 1. The outer diameter of the cylindrical portion 20D is large enough to fit in the forceps hole 2. The inner diameter of the cylindrical portion 20D is such that the optical fiber 20A, the green lens 20B, and the pattern chip 20C can be mounted.

The laser beam IL emitted from the green lens 20B is emitted to the outside from the open end of the cylindrical portion 20D. A mark M is formed on the outer surface of the cylindrical portion 20D. In the present embodiment, the mark M is used to perform calibration for calibrating the relative position information between the pattern projector 20 and the camera of the endoscope 1 (the imaging optical system 30 described later) in the active stereo method.

In this way, the cylindrical portion 20D protrudes from the insertion end 3 of the endoscope 1. A green laser beam IL is emitted from the open end of the cylindrical portion 20D. Since this laser light IL is diffracted by the pattern chip 20C to form a projected image of the measurement pattern 21, the measurement target (target T) irradiated with the laser light IL has a measurement pattern as shown in FIG. 21 is projected.

In addition to the forceps hole 2, an objective lens 30A and an illumination lens 40A are attached to the insertion end 3 of the endoscope 1. The objective lens 30A is a lens constituting the imaging optical system 30 (see FIG. 5) that images the target T, and the illumination lens 40A is a lens of the illumination optical system 40 (see FIG. 5) that irradiates the target T with illumination light. Is. The endoscope 1 has a function of irradiating the target T with illumination light via the illumination lens 40A and imaging the irradiated target T through the objective lens 30A.

FIG. 5 shows the overall configuration of the three-dimensional shape measuring device 100. As shown in FIG. 5, the three-dimensional shape measuring device 100 further includes a controller 50 in addition to the laser light source 10 and the pattern projector 20 described above. The controller 50 is a computer including a man-machine interface such as a CPU (Central Processing Unit), a storage device, an input / output device, a pointing device, or a display. The function of the controller 50 is realized by the CPU executing the program stored in the storage device according to the operation information of the operator input from the man-machine interface. That is, according to the program to be executed, the controller 50 controls the laser light source 10 via the input / output device, controls the illumination optical system 40 of the endoscope 1, and the endoscope 1 via the input / output device. The imaging data input from the imaging optical system 30 of the above is input.

Further, the endoscope 1 includes an imaging optical system 30 for photographing the measurement target and an illumination optical system 40 for illuminating the measurement target. The objective lens 30A shown in FIG. 4 is a lens constituting the imaging optical system 30 of the endoscope 1, and the illumination lens 40A constitutes the illumination optical system 40 of the endoscope 1 and externally (measures) the illumination light thereof. It is a lens that emits light to the target). Specifically, the imaging optical system 30 images the measurement target and the measurement pattern 21 projected on the measurement target.

The controller 50 includes an operation unit 50A, a control unit 50B, and a display unit 50C. The operation unit 50A outputs an operation signal corresponding to the operation of the operator to the control unit 50B.

The control unit 50B inputs the imaging data of the measurement target obtained from the imaging optical system 30 of the endoscope 1 according to the operation signal from the operation unit 50A, performs image processing on the input imaging data, and performs image processing on the input imaging data to perform the measurement target. Measure the three-dimensional shape. The control unit 50B includes a command unit 51 and an image processing unit 52. The command unit 51 outputs a control command to the laser light source 10 and the endoscope 1 according to the operation signal from the operation unit 50A. The image processing unit 52 performs image processing using the active stereo method based on the image of the measurement pattern 21 projected on the measurement target to obtain the three-dimensional shape of the measurement target.

The display unit 50C displays various images such as imaging data of the measurement target obtained from the imaging optical system 30 of the endoscope 1.

The three-dimensional shape measurement process performed by the image processing unit 52 will be described. As shown in FIG. 6A, in the present embodiment, the image processing unit 52 measures the three-dimensional shape of the measurement target based on the optical cutting method. The optical cutting method is one of the active stereo methods and is a method of applying the principle of triangulation. In the optical cutting method, the three-dimensional measurement target is based on the parallax (baseline length) L between the projection center P1 of the measurement target by the pattern projector 20 and the image pickup center P2 of the camera (imaging optical system 30) that images the measurement target. The shape is measured.

The projected light including the measurement pattern 21 is projected from the projection center P1 to the target T (measurement target), and the image of the measurement pattern 21 is also projected on the target T. The measurement pattern 21 includes a vertical straight line V.

Here, attention is paid to a specific point P on the target T on which the measurement pattern 21 is projected. When the point P on the target T changes in the direction along the projection direction of the projected image, as shown in FIG. 6B, the pattern on the point P is generated from the imaging optical system 30 at the position P2. It looks like it's off. In the optical cutting method, the depth d of each point of the target T is measured based on the positional deviation of this pattern.

The configuration of the image processing unit 52 will be described in more detail. As shown in FIG. 5, the image processing unit 52 includes a corresponding calculation unit 52A and a shape output unit 52B. The correspondence calculation unit 52A selects a node (see FIGS. 3A and 3B) that is a candidate for the correspondence solution based on the epipolar constraint for the captured image and the measurement pattern 21, and solves the solution. Based on the connection state of the graph between the candidates (nodes), the error of the pattern included in the image is eliminated and the correct corresponding solution (corresponding node) is obtained from the solution candidates. The shape output unit 52B is measured from the corresponding solution (corresponding node) by stereo processing (optical cutting method) as described above (as shown in FIGS. 6A and 6B). Calculates and outputs the dimensional shape.

The amount of deviation of the point P in the imaging result of the imaging optical system 30 when the depth of the target T changes by d changes according to the parallax (baseline length) L between the pattern projector 20 and the imaging optical system 30. In the embodiment, it is assumed that the parallax (baseline length) L is accurately obtained by calibration.

Next, a three-dimensional shape measuring operation using the three-dimensional shape measuring device 100 according to the present embodiment will be described.

As shown in FIG. 7, first, the controller 50 (command unit 51) projects the measurement pattern 21 onto the measurement target (target T) by the pattern projector 20, and uses the imaging optical system 30 of the endoscope 1. The measurement target (target T) is imaged (step S1).

(Line detection)
Next, the image processing unit 52 (corresponding calculation unit 52A) acquires a raw image from the imaging optical system 30 and detects a line from the raw image (step S2). In step S2, as shown in FIG. 8, the image processing unit 52 (corresponding calculation unit 52A) removes the fisheye distortion from the raw image (step S11). Subsequently, the image processing unit 52 (corresponding calculation unit 52A) reduces noise in the image by using a Gaussian filter or a median filter (step S12). Subsequently, the image processing unit 52 (corresponding calculation unit 52A) detects the vertical straight lines of the grid in the image (step S13).

Next, the image processing unit 52 (corresponding calculation unit 52A) detects a horizontal line portion that connects adjacent vertical straight lines in the horizontal direction (step S14). In this process, the luminance value is traced along the vertical straight line on each side of the detected vertical straight line, and the peak of the luminance value is detected. The positions of these peaks are candidates for the end points of the horizontal line. Among the vertical straight lines connecting the end point candidates, those within the range in which the length is determined are set as the initial candidates for the horizontal line.

There is a small error in the end point position of the initial candidate for the horizontal line. In order to correct this, the corresponding calculation unit 52A shifts the end point positions by a small amount along the vertical straight line for each initial candidate of the horizontal line segment, and maximizes the average value of the brightness values on the line segment connecting them. Search for the end point position so that The line segment at the searched end point position is used as a candidate for the optimized horizontal line segment. Some of the candidates for the horizontal line are overlapped, and some are detected by local optimization at a position where there is actually no projection pattern. In order to remove such horizontal line segment candidates, if there are horizontal line segment candidates that are close to each other (less than half the distance predicted from the projection pattern), the one with the smaller average luminance value on the line segment is removed. ..

After the end of step S14, the image processing unit 52 shifts to step S3 of FIG.

(Construction of grid graph)
Next, the image processing unit 52 (corresponding calculation unit 52A) constructs a grid graph (step S3). As shown in FIG. 9, the image processing unit 52 (corresponding calculation unit 52A) constructs a structure of a grid graph (lattice graph) based on the detected vertical lines and horizontal lines (steps S21 and S22). The structure of the grid graph in the measurement pattern 21 is shown in FIG. At this point, a group of vertical straight lines including vertical straight lines (for example, v ₁ , v ₂ ) has been detected, and horizontal edges (for example, e ₁ , e ₂ ) between them have been detected.

In order to construct a grid graph, when horizontal edges are selected one by one from the left and right of a certain vertical straight line, it is necessary to determine whether these horizontal edges are "continuous" or "discontinuous". Here, these lateral edges and a continuous, horizontal edge e ₃ and does not mean only that a geometrical continuous as horizontal edge e _4. For example, as in the horizontal edge e ₅ and the lateral edge e _6, the horizontal edge is geometrically a level difference, as the grid graph assumes that are connected by a single node n1, these "continuous" Defined as. Classification of "continuous" and "discontinuous" between horizontal edges is an important process. The code label of the node with no step between the horizontal edges is S.

In FIG. 10, the node indicated by the broken line circle is the node whose code label is S. Further, the node indicated by the solid line circle is a node whose code label is any of S, L, and R. Solid circle nodes and dashed circle nodes alternate on a vertical straight line.

For example, the vertical line v ₂ has a solid line (code label R) node n ₂ , a broken line (code label S) node n ₃ , a solid line (code label R) node n ₄ , and a broken line (code label R) in this order from the bottom. node _{n 5} of S) has appeared. In consideration of this arrangement, as shown in FIG. 9, the image processing unit 52 (corresponding calculation unit 52A) searches for the left and right horizontal edges of the vertical line where the distance between the end points is smaller than the threshold T _s. , The pair of these two end points is detected as a node of the code label S as a group (step S21).

Next, the image processing unit 52 (corresponding calculation unit 52A) searches for a pair of end points having a distance equal to or less than the threshold T _l larger than the threshold T _s for the remaining end points, and further edges above or below the edges. There already if it is marked as a continuous edge, the pair of end points one group, namely code label L, then detected as a node n _i in R (step S22).

In Figure 10, for example, horizontal edge e ₃ and the lateral edge e ₄ code label S is marked on the node as a continuous edge, subsequently, referring to the continuity of their edges, horizontal component e ₅ and horizontal min e ₆ code label L is marked to the node n ₁ as being continuous. Each node (for example, n ₆ ) is connected to adjacent nodes on the top, bottom, left, and right. Depending on the horizontal edge, the horizontal edge to be connected may not be found due to detection omission or the like. In such a case, there are adjacent nodes in the horizontal direction on only one side, right or left.

After executing step S22, the image processing unit 52 proceeds to step S4 of FIG.

(Association by subgraph pattern)
Next, the image processing unit 52 (corresponding calculation unit 52A) performs the association by the subgraph pattern (step S4). Specifically, the corresponding calculation unit 52A performs pattern matching in the subgraph of the graph of the measurement pattern 21 to associate the node included in the image with the node in the measurement pattern 21 as described below. As a result, the candidates for the corresponding solution (node) are reduced, and the correct corresponding solution (corresponding node) is obtained from the solution candidates. Further, the image processing unit 52 (shape output unit 52B) calculates and outputs the three-dimensional shape of the measurement target from the obtained corresponding solution by stereo processing (optical cutting method) (step S5).

Here, in step S3, the detection grid graph created from the image captured by the camera of the endoscope 1 is referred to as G, and the projection grid graph of the measurement pattern 21 shown in FIG. 3A is referred to as P. As shown in FIG. 11, the detection grid graph G may have missing edges, extra edges, or incorrect node labels S, L, and R. The image processing unit 52 collates the subgraphs of these graphs in order to allow an error in the graph topology and collate the detection grid graph G with the projection grid graph P.

The subgraph for collation is called a local sub-graph pattern (hereinafter, LSGP). As shown in FIG. 11, the LSGP (s) is a graph that serves as a template for collating the local subgraphs common to the detection grid graph G and the projection grid graph P. By giving the LSGP, the detection grid graph G and the projection grid graph P can be collated while allowing edge defects and false detections. Further, as shown in FIG. 11, by preparing a plurality of types of LSGPs and collating the detection grid graph G and the projection grid graph P with each LSGP, flexible collation processing becomes possible.

In the image processing unit 52, each LSGP is implemented as a path that passes through all the nodes. Comprehensive judgment from the collation results of the plurality of LSGP is done by voting for each node n _i.

The collation process is performed in the following procedure. First, the image processing unit 52, from the node n _i of detector grid graph G, to track the path of a lsgp, if there is no loss of edge, its subgraph (image graph pattern) and G _0. Next, the image processing unit 52 searches the projection grid graph P for a partial graph (partial graph pattern) P ₀ that satisfies all of the following conditions.

The conditions are as follows.
(1) G ₀ and P ₀ are graphs of the same topology (2) All corresponding nodes of G ₀ and P ₀ satisfy the epipolar constraint condition (3) Corresponding nodes of G ₀ and P ₀ The code labels S, L, and R of are matched at a rate equal to or higher than a certain threshold value (the degree of matching of pattern matching is higher than the threshold value).
The condition of (2) corresponds to the process of selecting the corresponding solution candidate based on the epipolar constraint.

In the measurement pattern 21, the number of nodes (solution candidates) in the projection grid graph P that satisfies the epipolar constraint for a node with the detection grid graph G is relatively small (the maximum is about 10). When P ₀ satisfying the condition is found, as shown in FIGS. 12 and 13, all the nodes of P ₀ are voted as corresponding nodes for each node of G ₀ . The correspondence calculation unit 52A eliminates the error of the pattern included in the image by voting when the matching degree of the pattern matching is higher than the threshold value for the node of the subgraph (eliminates the ambiguity from the candidates of the correspondence solution). ) Get the correct correspondence. As shown in FIG. 14, the corresponding calculation unit 52A prepares a plurality of local subgraph patterns. This process is performed on all the nodes of the detection grid graph G, the local subgraph pattern is changed, matching is repeated for all the LSGPs given in advance, and the total number of votes is obtained. When the iterative processing is completed, the image processing unit 52 (corresponding calculation unit 52A) determines the nodes of the projection grid graph in which the total number of votes obtained is the maximum for each node included in the detection grid graph G. When the number of votes exceeds the threshold value, the node of the projection grid graph P matches the node of the detection grid graph G, that is, the corresponding solution is determined and associated.

(Three-dimensional shape restoration by optical cutting method)
Returning to FIG. 7, the image processing unit 52 (shape output unit 52B) is detected based on the correspondence between the line segments after determining the correspondence between the captured image and the measurement pattern 21. The three-dimensional shape of the vertical straight line is restored by the optical cutting method (step S5). Here, the image processing unit 52 (shape output unit 52B) estimates the normal direction from the sparse point cloud restored by the optical cutting method by using two-dimensional regression. Then, the shape between the lines is restored by interpolation with a weight of the Radial Basis Function while considering the normal direction. As a result, the three-dimensional shape of the measurement target is obtained.

In order to evaluate the measurement by the three-dimensional shape measuring device 100 in a realistic situation, a biological specimen excised from the human stomach was measured. The target specimens were observed at different distances of about 25 mm and 15 mm. As shown in FIG. 15, by the pattern projector 20 using DOE, images of both clear patterns were obtained even if the patterns were observed at different distances. By the three-dimensional measurement method according to the present embodiment, each image could be restored almost normally. The correspondence between the grid pattern obtained from the measured image and the projected grid pattern was normally acquired, and the grid-like three-dimensional shape of the vertical and horizontal lines of the biological specimen was acquired.

Returning to FIG. 7, the control unit 50B then determines whether or not all imaging has been completed (step S6). When all the imaging is not completed (step S6; No), the control unit 50B takes steps S1 (projection and imaging) → S2 (line detection) → S3 (construction of grid graph) → S4 (correspondence by subgraph pattern). (Addition) → S5 (three-dimensional shape restoration by the optical cutting method) is performed, and the end determination of imaging is performed again (step S6). In this way, several images are taken and the three-dimensional shape is restored. During this period, the position of the tip of the endoscope 1 changes each time an image is taken, so that each image and the three-dimensional shape measured from the image become the three-dimensional shape of the measurement target measured from different viewpoints.

When all the imaging is completed (step S6; Yes), the image processing unit 52 synthesizes the obtained plurality of three-dimensional shapes (grid data) (step S7). This three-dimensional shape synthesis is performed using an ICP (Iterative Closest Point) algorithm. The ICP algorithm will be described below.

ここで、図１６に示すように、ｐ_ｉは、合成元の格子データの点群であり、ｑ_ｉは、合成相手の格子データの点群である。 In the conventional ICP algorithm, the lattice data is synthesized by calculating the following equation (1), that is, the parameter R of the rigid body transformation (rotation) that minimizes the cost function and the translation parameter t.
Equation (1)

Here, as shown in FIG. 16, _pi is a point group of the grid data of the synthesis source, and q _i is a point group of the grid data of the synthesis partner.

ここで、ｐ_ｉ ^ｈは、合成元の格子データの横線分上の点群であり、ｐ_ｉ ^ｖは、合成元の格子データの縦直線上の点群である。ｑ_ｉ ^ｖは、合成元の格子の縦直線上の点群であり、ｑ_ｉ ^ｈは、合成元の格子の横線分上の点群である。このようにすれば、図１６に示されるように、２つの３次元形状データを、より正確に合成することが可能となる。 However, in the above equation, the connection between the point groups on the two vertical lines and the point groups on the horizontal lines became too strong, and it was difficult to accurately synthesize the grid data (see FIG. 16). Therefore, in the present embodiment, the image processing unit 52 defines the cost function by the following equation (2), calculates the rigid transformation parameter R that minimizes the cost function, and the translation parameter t, and calculates the rigid transformation parameter. Two lattice data are combined using R and the translation parameter t.
Equation (2)

Here, p _i ^h is the point group of the horizontal component of the composite source grid data, p _i ^v is the vertical straight line point group of synthetic origin of the grid data. q _i ^v is a point group on the vertical straight line of the grid of the synthesis source, and q _i ^h is a point group on the horizontal line of the grid of the synthesis source. In this way, as shown in FIG. 16, it is possible to more accurately synthesize the two three-dimensional shape data.

In summary, the operation of the image processing unit 52 in step S17 is executed by the procedure shown in FIG. First, the image processing unit 52 (calculation unit), a plurality of three-dimensional shape data (p _{i, q i)} of the measurement object obtained from different directions of the vertical line segments constituting the first data p _i ( a point p _i ^v of the first segment) on, the first data p _i constitute another second data q _i, direction different second and horizontal component (the first segment between the points _q ^{i h} on the line segment), or between the vertical line segment on _q ^{i v} constituting the on horizontal component _p ^{i h} and the second data _{q i} constituting the first data _{p i} The correction amount of rotation and translation (rotation parameter (rotational deviation) R, translational parameter (translational deviation) t) that minimizes the deviation is calculated by performing regression analysis (step S31).

Subsequently, the image processing unit 52 (output unit) synthesizes the first data _pi and the second data q _i in a state where the calculated rotation parameter R and the translation parameter (translation deviation) t are cancelled. , Output as a three-dimensional shape corrected for deviation of the synthesized three-dimensional shape (step S32).

ここで、ｇ_ｉ（ｘ）は、サンプル点ｘ_ｉ近傍でのｄ（ｘ）の線形近似値であり、φ｜｜ｘ−ｘ_ｉ｜｜は、重みｐ_ｉ（ｘ）の値である。重みｐ_ｉ（ｘ）は、ｘ_ｉで最大で、ｘ_ｉから離れたｘで最小となる。重み関数φは、以下の式のものを用いることができる。

ここで、Ｒ_ｗは、近似半径である。 In order to realize a smoother surface, the image processing unit 52 calculates the depth function d (x) by interpolating the sample depth data s (x) using the following calculation formula. Can be done.

Here, g _i (x) is a linear approximation value of d (x) in the vicinity of the sample point x _i , and φ || x−x _i || is a value of the weight p _i (x). Weight _p i (x) is the largest at _{x i,} the minimum x away from _{x i.} As the weighting function φ, the following equation can be used.

Here, R _w is an approximate radius.

を用いたフィッティング（Ｒ_ｐは、半径の閾値）を行って、以下の式の係数ａを決定することにより、算出することができる。
ｐｉ（ｘ）＝ａ・（ｘ−ｘ_ｉ）＋ｓ（ｘ_ｉ） In addition, the weight _p i (x), the following sample set

It can be calculated by performing fitting using (R _p is a threshold value of radius) and determining the coefficient a of the following equation.
pi (x) = a · (x-x _i ) + s (x _i )

The image processing unit 52 (shape output unit 52B) synthesizes the three-dimensional shape data as described above to generate the three-dimensional shape of the entire measurement target (step S7). The generated three-dimensional shape is displayed on the display unit 50C and stored in a storage device or the like. After that, the control unit 50B ends the process.

As described in detail above, according to the present embodiment, the two-dimensional pattern projected onto the measurement target is composed of a grid graph P composed of nodes and edges having characteristics that can be distinguished by the connection state of the line segments. Therefore, even if subsurface scattering or a decrease in brightness occurs, the position information of the node can be easily read. As a result, the three-dimensional shape of the measurement target or its size can be stably measured.

That is, according to the three-dimensional shape measuring device 100, the shape of the living tissue can be stably restored with high accuracy over a wide working distance by using the linear grid step pattern and the new small pattern projector 20 by DOE. be able to.

More specifically, in the present embodiment, the pattern projector 20 by DOE and the three-dimensional shape measuring device 100 by the active stereo method (optical cutting method) using the grid pattern in which the code by the step is embedded are disclosed. According to the three-dimensional shape measuring device 100, by using DOE, clear pattern projection can be realized in a wide working range, and the utilization efficiency of light energy is 90% or more. Furthermore, the grid pattern of straight lines with embedded codes due to steps made it possible to correctly detect the pattern and recognize the code even if it was affected by blurring due to surface scattering. In addition, we have disclosed a new detection process that allows detection errors and collates the lattice structure of the step pattern according to the present embodiment. According to the three-dimensional shape measuring device 100 according to the present embodiment, it has succeeded in restoring the biological tissue excised from a human with different working distances.

In the above embodiment, the grid pattern by DOE is composed of line segments, but it may be a grid pattern in which dots are formed on the vertical straight line and the horizontal straight line. In addition, various other methods can be applied as long as the grid points can be encoded. For example, it is possible to code a node by the number of dots in each node.

In the above embodiment, the measurement pattern 21 is composed of nodes and edges having characteristics that can be distinguished by the connection state of the line segment, but the nodes and edges that have characteristics that can be distinguished by the symmetry of the pattern. It may consist of. For example, the measurement pattern 21 may have features that are locally rotationally symmetric or line-symmetrical, and the nodes or edges of the graph may be represented by the arrangement of the features. In this case, when forming the grid pattern, the grid pattern can be generated from the symmetry of the pattern on the image, instead of directly converting the grid lines into the image pattern. FIG. 18 is an example of a measurement pattern 22 in which a grid is represented by an image consisting of rectangular points. FIG. 19 shows an example of a grid point and a grid line in the measurement pattern 22 of FIG. The rectangular point D in FIG. 19 is a center point in which the pattern is rotationally symmetric by 180 degrees in the measurement pattern 22 of FIG. 18, and in the measurement pattern 22, such a point D is arranged on a grid. ing. Further, the vertical line L1 and the horizontal line L2 in FIG. 19 are center lines whose surrounding patterns are mirrored in the image of FIG. 18, and such center lines can represent the lines of the grid pattern.

FIG. 20 is an example of an image taken by projecting such a pattern. FIG. 21 is an enlarged view of the white frame portion of FIG. 20. FIG. 22 is an image obtained by obtaining the autocorrelation by rotating 180 degrees at each point in FIG. 21 and thereby obtaining the 180 degree rotational symmetry of each point. FIG. 23 shows a grid point detected by obtaining the peak of rotational symmetry of FIG. 22 and an image of the grid point. After that, the line connecting the grid points can be extracted by tentatively defining a grid block that is distorted into a parallel quadrilateral shape by projection and detecting line symmetry while considering the distortion due to projection.

In such a measurement pattern 22, there are a plurality of methods for adding features to grid points and grid lines. For example, even if the pixel at the location of the horizontal line L2 is changed in FIG. 19, the horizontal line L2 maintains the above-mentioned specularity. By utilizing this property, the pixels on the horizontal line L2 can be changed to give a feature to the horizontal line L2. The point symmetry on the grid points is not always established by the change of the pixels on the horizontal line L2, but if the amount of change in the entire pixel is adjusted, the point symmetry of the grid points is strong, so its detection. Is easy. If the characteristics of the grid points and grid lines can be detected, the shape restoration can be performed by collation with a partial bluff in the same manner as in the above embodiment. Alternatively, a point that can be distinguished from other points is placed inside the block that forms the grid (a single rectangular grid), and the position of that point is changed inside the block. May be attached.

Further, the correspondence calculation unit 52A calculates the correspondence point by optimizing the Markov random field model (MRF, Markov Random Field) by using the matching degree of the pattern matching as a data term and the connection relationship of the graph as a smoothing term. You may try to do it. The Markov field probability model is a method of defining the probability for label placement when a node in the graph structure is labeled, which allows "natural label placement" to be defined as the probability. .. Therefore, a plurality of algorithms have been proposed that realize labeling of nodes while removing noise by using a Markov field probability model.

を最大化するようなラベルを求める問題である。上記の式の項のうち、第一項は、ラベルとデータ（ノード特徴）の一致度を表すのでデータ項と呼ばれ、第二項はラベリングの平滑性（グラフの接続関係）を表すので平滑化項と呼ばれる。 In the models of these algorithms, the label probabilities taken by a graph node are limited to the local properties of that node (detected node characteristics) and the labels of the nodes connected to that node at the edge, that is, the labels of adjacent nodes. Dependent. Label placement with a high probability as a whole is expressed in the form of optimization of a function that represents the log-likelihood of labeling the entire graph. this is,

Is the problem of finding a label that maximizes. Of the terms in the above formula, the first term is called the data term because it represents the degree of matching between the label and the data (node features), and the second term is smooth because it represents the smoothness of labeling (connection relationship of graphs). It is called a chemical term.

In the above embodiment, in each node of the graph structure detected in the image captured by the camera, the ID of the node of the corresponding pattern corresponds to the label. In the data term, pattern matching and satisfaction of epipolar constraints are examined in the subgraph including that node, and a high log-likelihood is assigned to the corresponding node with a high degree of matching or the corresponding node with a high vote. In the smoothing term, the connectivity of the nodes is examined, and the connectivity of the nodes of the graph detected in the image captured by the camera and the connectivity of the nodes of the graph of the projection pattern are highly log-likelihood at the matching edge. Degrees are assigned. As an algorithm for obtaining such a label, a graphcut algorithm, a belief propagation algorithm, and the like are known.

The three-dimensional shape measuring device 100 according to the present embodiment can be used, for example, for endoscopic diagnosis and treatment in the gastrointestinal tract. In early gastric tumors, the size of the tumor measured using the 3D shape measuring device 100 is important information when selecting a treatment method.

In the present embodiment, the three-dimensional shape measuring device 100 used for the endoscope 1 has been described, but the present invention is not limited to this. For example, it can be applied to capture the three-dimensional shape of a dynamic scene for human movements, robots for fluid analysis, and visual information for analysis and inspection purposes. That is, it can be applied to the measurement of the three-dimensional shape of any object.

The present invention allows for various embodiments and modifications without departing from the broad spirit and scope of the invention. Moreover, the above-described embodiment is for explaining the present invention, and does not limit the scope of the present invention. That is, the scope of the present invention is indicated by the scope of claims, not by the embodiment. Then, various modifications made within the scope of the claims and the equivalent meaning of the invention are considered to be within the scope of the present invention.

The present invention can be applied to the measurement of the three-dimensional shape of the measurement target.

1 Endoscope, 2 Force hole, 3 Insertion end, 10 Laser light source, 10A laser module, 10B diffuser, 10C illumination aperture, 10D collimator lens, 20 pattern projector, 20A optical fiber, 20B green lens, 20C pattern chip, 20D Cylindrical part, 21 and 22 measurement patterns, 30 imaging optical system, 30A objective lens, 40 illumination optical system, 40A illumination lens, 50 controller, 50A operation unit, 50B control unit, 50C display unit, 51 command unit, 52 image processing Unit, 52A compatible calculation unit, 52B shape output unit, 100 3D shape measuring device, point D, IL laser beam, L baseline length, L1 vertical line, L2 horizontal line, M mark, T target.

Claims (11)

A light projecting portion for projecting a two-dimensional pattern formed on the measurement object from the graph consisting of nodes and edges with thus distinguishable features the line connection state,
An imaging unit that captures the two-dimensional pattern projected onto the measurement target, and
For the two-dimensional pattern and an image captured by the imaging unit, select a candidate of the solution of the node corresponding to each other on the basis of the epipolar constraint, based on the connection state in the graph of candidates among the solutions, the A correspondence calculation unit that eliminates the error of the two-dimensional pattern included in the image and obtains the correct correspondence solution from the solution candidates.
A shape output unit that calculates and outputs the three-dimensional shape of the measurement target by stereo processing based on the corresponding solution, and
Equipped with a,
The corresponding calculation unit
By performing pattern matching in the subgraph of the graph of the two-dimensional pattern and associating the node included in the image with the node in the two-dimensional pattern, the candidates for the corresponding solution are reduced.
The two-dimensional pattern is a pattern of a graph structure composed of vertical and horizontal line segments, and each node is characterized by a step of left and right horizontal line segments connected to a vertical straight line.
Three- dimensional shape measuring device. The corresponding calculation unit
Wherein the degree of matching of the pattern matching and data sections by the connection relationship of the graph and smoothing term, and calculates the corresponding points by Markov random field model (MRF) optimization,
The three-dimensional shape measuring device according to claim 1 . The corresponding calculation unit
For said nodes of said subgraph, it said that the degree of coincidence of the pattern matching to vote is higher than the threshold value, obtaining a correct corresponding points from the corresponding candidate solutions,
The three-dimensional shape measuring device according to claim 1 . The corresponding calculation unit
The subgraphs preparing a plurality to determine a solution of the corresponding by the total number of votes repeated votes in each of the subgraphs,
The three-dimensional shape measuring device according to claim 3 . The shape output unit
The three-dimensional shape is calculated by using the optical cutting method from the correspondence relationship of the line segments based on the corresponding solution.
The three-dimensional shape measuring device according to any one of claims 1 to 4 . The series of line segments included in the two-dimensional pattern is
Arranged in a curved line,
The three-dimensional shape measuring device according to any one of claims 1 to 5 . The two-dimensional pattern is
Formed on the measurement target by diffraction of laser light by a diffraction optical element,
The three-dimensional shape measuring device according to any one of claims 1 to 6 . The light projecting unit is incorporated in the endoscope.
The three-dimensional shape measuring device according to any one of claims 1 to 7 . If it is measured from different directions, data of a plurality of three-dimensional shape of the measurement object is obtained,
Of the plurality of three-dimensional shape data, the first line segment constituting the first data and the first line segment constituting the second data different from the first data are that minimizes the deviation between the direction different second segment, a calculation unit for calculating a correction amount of the rotation and parallel proceeds,
By the correction amount of the rotational and translational calculated by the calculation unit, and an output unit for outputting the corrected shape wherein the first data second displacement data,
To prepare
The three-dimensional shape measuring device according to claim 1. A three-dimensional shape measuring method executed by a three-dimensional shape measuring device.
The two-dimensional pattern consisting of the graph consisting of nodes and edges with thus distinguishable features the line connection state is projected onto the target object, imaging the two-dimensional pattern which is projected to the measurement target Pattern imaging step and
Wherein for said 2-dimensional pattern image captured by the pattern imaging step, selects a candidate solution of the node corresponding to each other on the basis of the epipolar constraint, based on the connection state of the graph of candidates among the solutions, the A correspondence calculation step that eliminates the error of the two-dimensional pattern included in the image and obtains the correct correspondence solution from the solution candidates.
A shape output step that calculates and outputs the three-dimensional shape of the measurement target by stereo processing based on the corresponding solution, and
Only including,
In the corresponding calculation step,
By performing pattern matching in the subgraph of the graph of the two-dimensional pattern and associating the node included in the image with the node in the two-dimensional pattern, the candidates for the corresponding solution are reduced.
The two-dimensional pattern is a pattern of a graph structure composed of vertical and horizontal line segments, and each node is characterized by a step of left and right horizontal line segments connected to a vertical straight line.
Three-dimensional shape measurement method. Measured from different directions, when the data of a plurality of three-dimensional shape of the measurement object is obtained, among the data of the plurality of three-dimensional shape, the first segment constituting the first data, constituting the second data different from said first data, said that the first line segment to minimize the deviation between the second line segments of different directions, rotation and parallel -ary Calculation steps to calculate the correction amount and
By the correction amount of the rotation and translation calculated in the calculation step, and outputting the corrected shape wherein the first data second displacement data,
10. The three-dimensional shape measuring method according to claim 10 .

Images