JP2019150574A - Three-dimensional shape data creation method and three-dimensional shape data creation system - Google Patents

Abstract

To provide a three-dimensional shape creation method and a three-dimensional shape creation system, capable of acquiring and reproducing a shape of a tube's inner side in a non-contact manner.SOLUTION: A three-dimensional shape creation method of the present invention includes: a step of creating a plurality of pieces of two-dimensional image data on the basis of a signal from imaging means, capable of moving on an inner side of a tube, for imaging the inner side of the tube; a step of acquiring space information on the imaging means 1 at the time of imaging, the space information being acquired by a motion sensor 2 installed at the imaging means; and a step of associating the two-dimensional image data with the space information and creating three-dimensional shape data at the tube's inner side on the basis of the two-dimensional image data and the space information.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a three-dimensional shape data creation method and a three-dimensional shape data creation system, and in particular, a method capable of obtaining and reproducing a three-dimensional shape inside a tubular body such as an external auditory canal without contact. And the system.

  For hearing aids, in order to be inserted into the external auditory canal and used, custom-made ear hearing aids and ear molds manufactured according to the user's external auditory canal are known. For example, in order to produce an ear mold, it is necessary to collect the inner shape of the user's external auditory canal. For this reason, conventionally, an impression material is injected into the ear canal with a syringe, and the cured impression material is taken out to remove the user's ear mold. Ear type sampling is performed (see, for example, Patent Document 1).

JP 2000-210327 A

  By the way, when performing ear type collection, for example, when the curve in the middle of the external auditory canal is abrupt, a rarely cured impression material may not be taken out. Elderly people may have sagging (has wrinkles) in the ear canal skin, and in this case, when the impression material is injected, the skin also moves, so the collected ear shape may differ from the original shape. is there. In addition, since the accuracy of the ear mold depends on the skill of the person who collects the ear mold, the shape of the ear mold may change depending on the person who collects the ear mold. For this reason, the technique which can know the inner shape of an ear canal irrespective of an impression material is calculated | required.

  On the other hand, in recent years, techniques for measuring the shape of an object in a non-contact manner have also advanced. However, most of the conventional ones measure the outer shape of an object, and it is not yet common to obtain three-dimensional shape data inside a tubular body such as the ear canal.

  An object of the present invention is to solve such problems, and a three-dimensional shape data creation method capable of acquiring and reproducing the shape inside a tubular body such as the ear canal in a non-contact manner, It is another object of the present invention to provide a three-dimensional shape data creation system.

  The three-dimensional shape data creation method according to one aspect of the present invention includes a step of generating a plurality of two-dimensional image data based on a signal from an imaging unit that is movable inside the tube and images the inside of the tube. And acquiring the spatial information of the imaging means at the time of imaging based on a signal from a motion sensor installed in the imaging means, associating the two-dimensional image data and the spatial information, and the two-dimensional image data And generating three-dimensional shape data inside the tubular body based on the spatial information.

  In this three-dimensional shape data creation method, the imaging means includes an illumination, an objective optical system, an imaging element that is disposed at an imaging position by the objective optical system and receives reflected light from the inside of the tubular body, The step of generating the three-dimensional shape data includes calculating a movement locus of the principal point of the objective optical system accompanying movement of the imaging means based on the spatial information, and using the movement locus as a central axis Setting cylindrical coordinates, discretizing the surrounding space of the imaging means based on the cylindrical coordinates, and setting the surrounding space to be composed of a plurality of minute surfaces according to the cylindrical coordinates; and the imaging A step of setting a straight line connecting each light receiving element of the element and the principal point; a step of determining whether or not the minute surface exists on the straight line; and reflected light from the minute surface Enter each photo detector A step of determining whether or not it is possible, and a reflection of a pixel value at each pixel of the two-dimensional image data from each of the minute surfaces that are on the straight line and can enter the light receiving elements. A step of calculating a reflected light intensity at each of the minute surfaces existing on the straight line based on the pixel value, and a reflected light of the minute surface existing on the straight line. It is preferable that the method includes estimating a minute surface inside the tubular body that actually exists based on an intensity distribution, and constructing three-dimensional shape data inside the tubular body based on the estimated minute surface.

  Further, in this three-dimensional shape data creation method, the imaging means includes an illumination, an objective optical system, and an imaging element that is disposed at an image forming position by the objective optical system and receives reflected light from the inside of the tubular body. The step of generating the three-dimensional shape data includes acquiring reference data relating to pixel values according to the distance between the principal point of the objective optical system and the inside of the tube in advance, and based on the spatial information The movement locus of the principal point of the objective optical system accompanying the movement of the imaging means is calculated, cylindrical coordinates with the movement locus as a central axis are set, and a predetermined configuration composed of a plurality of minute surfaces according to the cylinder coordinates Setting a cylinder with a radius, setting a straight line connecting each light receiving element of the image sensor and the principal point, determining whether or not the minute surface exists on the straight line, Two dimensions A step of extracting the minute surface corresponding to the image data; and a principal point and the minute surface at a position where the two-dimensional image data is acquired based on the reference data from the pixel value corresponding to the extracted minute surface And a step of calculating the shape of the tube according to the distance.

  It is preferable that the method further includes a step of correcting the spatial information of the imaging unit obtained by the motion sensor based on the spatial information of the tubular body at the time of imaging obtained by another motion sensor.

  The three-dimensional shape data creation system according to one aspect of the present invention generates a plurality of two-dimensional image data based on a signal from an imaging unit that can move inside the tubular body and images the inside of the tubular body. Two-dimensional image data generation means, spatial information acquisition means for acquiring spatial information of the imaging means at the time of imaging based on a signal from a motion sensor installed in the imaging means, the two-dimensional image data and the spatial information And 3D shape data generating means for generating 3D shape data inside the tubular body based on the 2D image data and the spatial information.

  In this three-dimensional shape data creation system, the imaging means is provided so as to be able to be bent while supporting the distal end portion including illumination, an objective optical system on which reflected light from the inside of the tubular body is incident. It is preferable that a bending portion and an image sensor that receives reflected light from the inside of the tubular body are disposed at an imaging position by the objective optical system, and the motion sensor is provided at the distal end portion.

  Such a three-dimensional shape data creation system has storage means for acquiring and storing reference data relating to pixel values according to the distance between the principal point of the objective optical system and the inside of the tube in advance, and the three-dimensional shape data The data generation means calculates a movement locus of the principal point of the objective optical system accompanying the movement of the imaging means based on the spatial information, sets cylindrical coordinates with the movement locus as a central axis, and sets the cylinder Whether a cylinder having a predetermined radius composed of a plurality of minute surfaces according to coordinates is set, a straight line connecting each light receiving element of the image sensor and the principal point is set, and whether or not the minute surface exists on the straight line And extracting the minute surface corresponding to the two-dimensional image data, and from the pixel value corresponding to the extracted minute surface, based on the reference data, the main surface at the position where the two-dimensional image data is acquired. A point and the microfacet It is preferable to determine the distance.

  Moreover, it is preferable to further comprise holding means for holding the tube body directly or indirectly.

  According to the present invention, the three-dimensional shape data inside the tubular body is generated from the plurality of two-dimensional image data inside the tubular body obtained by the imaging means and the spatial information of the imaging means obtained by the motion sensor. Can do. That is, since the shape inside the tube can be acquired and reproduced without contact, by generating three-dimensional shape data inside the ear canal according to the present invention, an ear shape is collected using an impression material. You can solve the problem when you were.

It is the figure which showed roughly one Embodiment of the three-dimensional shape data creation system according to this invention. It is a figure showing one embodiment of an image sensor. It is the figure shown about cylindrical coordinates. It is a figure explaining the straight line which connects an image pick-up element and the principal point of an objective optical system. It is a figure explaining the intersection of the cylindrical surface in a cylindrical coordinate, and the straight line shown in FIG. It is a figure explaining the method for discriminating whether the intersection shown in FIG. 5 exists inside a micro surface.

  Hereinafter, an embodiment of a three-dimensional shape data creation method and a three-dimensional shape data creation system inside a tubular body according to the present invention will be described with reference to the drawings. In the present embodiment, a case where three-dimensional shape data of the ear canal is created will be described.

  As shown in FIG. 1, the three-dimensional shape data creation system according to this embodiment includes an imaging unit 1, a motion sensor 2, and a computer 3.

  The imaging means 1 is movable inside the ear canal and images the inside of the ear canal. Examples of such a device include an otoscope 10 shown in FIG. 1A and a video scope 11 shown in FIG. 1B (an objective optical system 12 and an image sensor at the end inserted into the ear canal). 13). Although not shown in the drawings, an imaging device provided with an objective optical system at the end inserted into the ear canal, guiding light incident from the objective optical system with a fiber cable, and arranging it at the opposite end A fiberscope that converts light from the objective optical system into an electrical signal may be employed.

  The imaging unit 1 is provided with an objective optical system 12 as shown in FIG. The objective optical system 12 may be composed of a single lens or a combination of a plurality of lenses. At the imaging position of the objective optical system 12, an image sensor 13 that receives reflected light from the inside of the ear canal that enters from the objective optical system 12 and converts it into an electrical signal is disposed. Here, the imaging element 13 is composed of a plurality of light receiving elements 13a as shown in FIG. Although illustration is omitted, since the imaging means 1 is provided with illumination means for irradiating light toward the object to be imaged, it is possible to perform good imaging even inside the ear canal where it is difficult for light to enter from the outside. Can do.

  The motion sensor 2 acquires spatial information (information regarding position and orientation) of the imaging unit 1. Specifically, it includes a triaxial acceleration sensor and a triaxial angular velocity sensor, and can detect the position and orientation of the imaging means 1 in orthogonal coordinates. In the case where the imaging unit 1 is restricted to move only in one axis direction with a fixed posture, the spatial information of the imaging unit 1 detected by the motion sensor 2 is only in the position in the one axis direction. It is possible to select various motion sensors 2 in consideration of the positions and postures that the imaging means 1 can take.

  Such a motion sensor 2 is preferably provided in a place where the position and orientation thereof do not change with respect to the objective optical system 12 and the imaging element 13 in the imaging means 1. For example, when it is provided in the otoscope 10, it may be a location close to the site inserted into the ear canal as shown in FIG. 1 (a) or a location on the opposite side. In addition, as shown in FIG. 1B, the video scope 11 includes a distal end portion 11a including the objective optical system 12 and the imaging element 13, and a bent portion 11b that supports the distal end portion 11a and is provided so as to be bendable. The motion sensor 2 is preferably provided at the distal end portion 11a where the position and orientation of the objective optical system 12 and the image sensor 13 do not change. When the motion sensor 2 is provided at the distal end portion 11a, another motion sensor 2a is provided for the outer portion 11c located outside the external auditory canal when the video scope 11 is inserted into the external auditory canal. Also good. In this case, if the motion sensor 2a includes a triaxial angular velocity sensor, the posture of the motion sensor 2 can be estimated from the information. When a plurality of motion sensors are used in this way, a triaxial angular velocity sensor is not necessarily required for the motion sensor 2 provided at the distal end portion 11a. In addition, also when providing the motion sensor 2 in a fiberscope, providing in the front-end | tip part which provided the objective optical system is preferable.

  The computer 3 is connected to the imaging unit 1 and the motion sensor 2 and acquires an electrical signal output from the imaging unit 1 and the motion sensor 2. The computer 3 of this embodiment functions as a two-dimensional image data generation unit 30, a spatial information acquisition unit 31, and a three-dimensional shape data generation unit 32. As will be described later, the two-dimensional image data generating unit 30 generates two-dimensional image data inside the ear canal based on the electrical signal output from the imaging unit 1. The spatial information acquisition unit 31 acquires spatial information of the imaging unit 1 at the time of imaging based on the electrical signal output from the motion sensor 2. The three-dimensional shape data generating unit 32 associates the two-dimensional image data with the spatial information and generates three-dimensional shape data inside the ear canal based on the associated two-dimensional image data and the spatial information. Although not shown, the computer 3 can be connected to various peripheral devices (input means such as a mouse and keyboard, output means such as a monitor and a printer).

  Although not shown, the three-dimensional shape data creation system of the present embodiment is a motion sensor different from the motion sensor 2 and used for acquiring spatial information of the ear canal. May be provided. The motion sensor for the external auditory canal need not be directly attached to the inside of the external auditory canal. If the person to be imaged moves during imaging, the spatial information of the imaging means 1 obtained by the motion sensor 2 changes even if the relative position and orientation of the imaging means 1 and the ear canal actually do not change. However, in the case where a motion sensor for the external auditory canal is provided, the amount of change in the position and posture of the external auditory canal is known. The effect of the movement of the In place of the motion sensor for the external auditory canal (may be used in combination), a holding unit that directly or indirectly holds the tubular body such as the external auditory canal may be used. Examples of such holding means include a chin rest that prevents the head from moving by placing the chin of the person to be imaged (to prevent the external auditory canal from moving).

  Next, a 3D shape data creation method using such a 3D shape data creation system will be described.

First, the range that can be imaged by the image sensor 13 will be described. As shown in FIGS. 1B and 2, when the image sensor 13 is arranged in parallel to the xy plane, the imageable range of the image sensor 13 is the inner side of the two-dot chain line shown in FIG. It becomes an area. As illustrated, the width (length in the x direction) of the image sensor 13 is x _w , the height (length in the y direction) of the image sensor 13 is y _h , the angle of view (diagonal angle of view) is θ, and the focal length is When f is assumed, these are in the relationship of the following formula (Equation 1).

  Accordingly, when generating the three-dimensional shape data inside the ear canal, imaging is performed while changing the position and orientation of the image sensor 13 based on the range that can be imaged by the image sensor 13, and all expected portions are captured. To be. In the fiberscope, since the image plane of the image sensor 13 is in the fiberscope, when the fiberscope is used, the three-dimensional shape data generation unit 32 uses the image plane position instead of the light receiving element position of the image sensor 13. .

  As described above, the image pickup device 13 is composed of a plurality of light receiving elements 13a as shown in FIG. 2, and converts it into a voltage corresponding to the intensity of light incident on each light receiving element 13a, and outputs an electric signal. Further, the two-dimensional image data generating means 30 generates pixels converted into pixel values corresponding to the voltages of the respective light receiving elements 13a, and arranges these pixels in correspondence with the arrangement of the light receiving elements 13a, thereby enabling the inside of the ear canal to be formed. Two-dimensional image data is generated.

Incidentally, x in the center of the light receiving elements 13a, y position _{(x ix, y iy),} the width of the image pickup device 13 (x-direction length) _{x w,} the height of the image pickup device 13 (y-direction length) y _h , the resolution of the image sensor 13 in the x direction is N _x , the resolution of the image sensor 13 in the y direction is N _y , the pitch of each light receiving element 13a in the x direction is dx, and the pitch of each light receiving element 13a in the y direction is dy. When the coordinates of the center position of the image sensor 13 are (x _c , y _c , z _c0 ) and the variables i _x and i _y are used, the following equations (Equation 2) and (Equation 3) are used. Can be expressed as

  And the spatial information acquisition means 31 acquires the spatial information of the imaging means 1 at the time of imaging based on the electrical signal output from the motion sensor 2. Specifically, there is a method of acquiring spatial information of the imaging unit 1 by outputting an electrical signal from the motion sensor 2 and capturing the electrical signal of the motion sensor 2 at the timing when the imaging unit 1 performs imaging. Can be mentioned. It should be noted that while the electrical signal from the motion sensor 2 is stopped in normal times, the electrical signal is output from the motion sensor 2 with the electrical signal output when the imaging unit 1 performs imaging as a trigger. It may be.

  The two-dimensional image data inside the ear canal obtained as described above and the spatial information of the imaging unit 1 are associated by the three-dimensional shape data generation unit 32. Thereby, it can be understood which part of the inner side of the ear canal corresponds to the obtained two-dimensional image data.

  Further, the three-dimensional shape data generating means 32 generates three-dimensional shape data inside the ear canal based on the associated two-dimensional image data and spatial information. Specifically, the following procedure is performed.

  First, the imaging means 1 is moved to a place where three-dimensional shape data of the ear canal is necessary, and two-dimensional image data and spatial information are acquired at a predetermined interval (for example, 0.1 mm). The acquired two-dimensional image data and spatial information are stored in a memory (not shown) built in the computer 3. Based on the acquired spatial information, the movement trajectory of the principal point of the objective optical system 12 accompanying the movement of the imaging means 1 is calculated, cylindrical coordinates having the movement trajectory as the central axis are set, and the ear canal is assumed to be a tubular body. To do. Further, a plurality of minute surfaces are set on the inner surface of the tubular body based on the set cylindrical coordinates. Further, the computer 3 has a storage unit (not shown), and pixel value reference data corresponding to the distance l between the principal point and the minute surface when the reflection intensity of the inner wall surface of the tube is assumed to be a uniform predetermined value. Save in the storage unit in advance.

This point will be specifically described with reference to the drawings. FIG. 3 shows the relationship between the above-described cylindrical coordinates and minute surfaces. In practice, the external auditory canal space has a curved tubular shape, but first, in order to simplify the explanation of the concept, here, the imaging means 1 shown in FIG. 1B has moved along the z-axis ( In the following description, the movement locus of the principal point of the objective optical system 12 coincides with the z axis. In this state, the cylindrical coordinates having the central axis of the movement locus of the principal point of the objective optical system 12 can be expressed as shown in FIG. Here, when it is assumed that the inner surface of the ear canal is discretized based on the cylindrical coordinates and is configured by a plurality of minute surfaces according to the cylindrical coordinates, the minute surface has a certain radius as shown in FIG. It can be shown as a rectangular surface present on the cylindrical surface. In FIG. 3, it is assumed that the minute surface is on a cylindrical surface having a radius r ₀ , the azimuth angle of one vertex (point A) on the minute surface is φ ₀ , and the z position is z ₀ . In this case, the coordinates of the point A can be expressed as (r ₀ , φ ₀ , z ₀ ). The coordinates of the vertex (point B) existing at z ₁ whose z position is slightly different from point A can be expressed as (r ₀ , φ ₀ , z ₁ ), and φ ₁ whose azimuth is slightly different from point A The coordinates of the vertex (point C) existing in can be expressed as (r ₀ , φ ₁ , z ₀ ).

Here, when the planes passing through the points A, B, and C shown in FIG. 3 are expressed using parameters s ₁ and s ₂ , the parameters a _x1 , a _x2 , s _x0 , a _y1 , a _y2 , s _y0. , A _z1 , a _z2 , and s _z0 can be expressed as the following formula (Equation 4).

Further, using the coordinates of point A, point B, and point C, Equation (Equation 4) is used, and at point A, s ₁ = 0, s ₂ = 0, at point B, s ₁ = 1, s ₂ = 0, point C In the case of s ₁ = 0 and s ₂ = 1, it can be expressed as the following formula (Formula 5).

Then, the parameters a _x1 , a _x2 , s _x0 , a _y1 , a _y2 , s _y0 , a _z1 , a _z2 , s _z0 can be obtained by solving the simultaneous equations of the above equation (Equation 5).

Next, a straight line connecting the image sensor 13 and the principal point of the objective optical system 12 is set. The imaging element 13 in the present embodiment is composed of a plurality of light receiving elements 13a. Here, as shown in FIG. 4A, each center of the light receiving element 13a and the principal point of the objective optical system 12 are passed. A straight line shall be set. Here, when the straight line is expressed using the parameter l, it can be expressed as the following equation (Equation 6) using the parameters a _x , l _x0 , a _y , l _y0 , a _z , and l _z0. .

The coordinates at the center of the light receiving element 13a are (x _p , y _p , z _p ), the coordinates of the principal point of the objective optical system 12 are (x _f , y _f , z _f ). When passing through the center and the principal point of the objective optical system 12, the center of the light receiving element 13a is defined as the start point (l = 0) and the principal point of the objective optical system 12 is defined as the end point (l = 1). 6) can be expressed as the following equation (Formula 7).

The parameters a _x , l _x0 , a _y , l _y0 , a _z , and l _z0 can be obtained by solving the simultaneous equations of the above formula (Equation 7), and the main elements of the light receiving element 13a and the objective optical system 12 can be obtained. A straight line passing through the points can be defined.

  Next, the correspondence between each light receiving element and each minute surface is determined. That is, it is determined whether or not the above-described minute surface exists on this straight line. Specifically, an intersection of a certain plane represented by the equation (Equation 4) and a certain straight line represented by the equation (Equation 6) is obtained, and whether or not the intersection is inside the assumed minute surface and Determine if they intersect from the inside of the cylinder. That is, if it can be determined that the intersection is inside the minute surface, it can be said that the minute surface exists on the straight line, and if it can be determined that the intersection is outside, it can be said that the minute surface does not exist on the straight line. Furthermore, since the external auditory canal is actually bent, it is assumed that the straight line intersects the inner wall of the external auditory canal again. Therefore, even when there are a plurality of minute surfaces on the straight line, only when the inner surface faces the imaging means 1. Consider. Thereby, the micro surface image | photographed with the imaging means 1 can be specified.

  In the present embodiment, as shown in FIG. 5, the intersection of the cylindrical surface and the straight line is calculated in a state where the center of the light receiving element 13a is on the z axis. From the equations (Equation 4) and (Equation 6), the intersection of the cylindrical surface and the straight line can be expressed as the following equation (Equation 8).

Here, the parameters a _x1 , a _x2 , s _x0 , a _y1 , a _y2 , s _y0 , a _z1 , a _z2 , s _z0 of the cylindrical surface are known from the equation (Equation 5), and the linear parameters a _x , Since l _x0 , a _y , l _y0 , a _z , and l _z0 are known from the equation (Equation 7), the equation (Equation 8) described above is a ternary primary with s ₁ , s ₂ , and l as variables. It can be said that it is a simultaneous equation. Accordingly, since the values of s ₁ , s ₂ , and l are determined, the cylindrical surface is obtained by substituting the obtained s ₁ and s ₂ into the equation (Equation 4) or by substituting the obtained l into the equation (Equation 6). And the point of intersection with the straight line. Note that light from a minute surface existing on the cylindrical surface may be incident on the light receiving element 13a when it is far from the principal point when viewed from the light receiving element 13a, that is, when l> 1.

Then, it is determined whether or not the derived intersection point (hereinafter referred to as determination point P) is inside the minute surface. FIG. 6 shows the determination method. In this determination method, attention is first paid to the determination point P and two adjacent vertices A and B on the minute surface. Here, FIG. 6A illustrates a state where the determination point P is inside the minute surface, and FIG. 6B illustrates a state where the determination point P is outside the minute surface. Note that the PA vector and the PB vector shown in FIGS. 6A and 6B are both relative position vectors of two points A and B with reference to the determination point P, and n is a normal vector of the surface. It has become. Further, the angle (signed angle) θ ₁ formed by the PA vector and the PB vector shown in FIG. Here, PA · PB is an inner product of the PA vector and the PB vector, and PA × PB · n is an inner product of the outer product of the PA vector and the PB vector and the normal vector n. The signed angle θ ₁ ′ shown in FIG. 6B can also be expressed by the same expression as Expression (Equation 9).

  The signed angle is calculated for all combinations of adjacent vertices of the minute surface. Here, as shown in FIG. 6A, the determination point P exists inside the minute surface, and the surface inside the minute surface as viewed from the image pickup means 1 faces the image pickup means 1 side (inward). In this case, the sum of the calculated four signed angles is theoretically 2π. Further, when the outer surface of the minute surface faces the imaging means 1 side (outward), the sum of the calculated four signed angles is theoretically −2π. On the other hand, when the determination point P exists outside the minute surface as shown in FIG. 6B, the total of the four calculated signed angles is theoretically zero. Note that when the signed angle is calculated based on the data actually captured by the imaging unit 1, a value as the theoretical value including an error or the like may not be obtained. If it is larger than π), it may be determined that the determination point P exists inside the minute surface. Next, when there are a plurality of minute surfaces on which the determination point P exists inside, it is determined whether l is inward or outward when viewed from the image capturing unit 1 in the order of decreasing l (the order close to the image capturing unit 1). To do. When the sum of the signed angles is negative, it is determined to be outward and it is determined that the reflected light cannot be received. If the sum of the signed angles is positive, it is determined that it is inward and light can be received. This determination is repeated in ascending order of l, but if it is once determined to be outward, it is determined that no more images have been captured thereafter.

When the light receiving element corresponding to each minute surface is determined in this way, the pixel value corresponding to each minute surface is determined from the pixel value of the two-dimensional image data. For each microfacet, based on the above-mentioned reference data, the distance l which acquired pixel value corresponding to the l _P, calculating a radius r _P of the tubular body when the distance between the micro-surface and the principal point is l _P To do. Here radius r _P is a distance from the main point trajectory in perpendicular tube cross section principal point trajectory to the inner surface. For all of the extracted micro-area, calculates the radius r _P.

For all of the two-dimensional image data for calculating the radius r _P corresponding to each micro surface as described above. Each microfacet because it is captured in a plurality of 2-dimensional image data, the radius r _P corresponding to each micro surface will be more calculated. In this case, the average value of the radius r _P may radius r _P of the fine surface. Radius r _P of the micro surface, since the distance of each minute surface and principal point trajectory, by connecting the small surface sequentially, it is possible to construct a three-dimensional shape data inside the tube.

Further, after determining whether or not the minute surface exists on the straight line and can receive light, the pixel value in each pixel of the two-dimensional image data is obtained from each of the minute surfaces existing on the straight line. The sum of the reflected light intensities is set, and the reflected light intensities on each of the minute surfaces that exist on the straight line and can receive light are calculated based on the pixel values. For example, a pixel value of a certain pixel m and v _m, the reflected light intensity from the microscopic surface present on the cylindrical surface at position n in a cylindrical coordinate to u _n. Here, using the coefficient a _mn , it is determined that a minute surface that exists on the above-described straight line and that can be received is a _mn ≠ 0 (for example, a _mn = 1) and does not exist. _Assuming that a _mn = 0, the relationship between the pixel value of each pixel of the two-dimensional image data and the sum of the reflected light intensities from each of the minute surfaces existing on the straight line is It can be expressed as (Equation 10).

  When this relationship is considered for all the pixels, it can be expressed by a matrix such as the following equation (Equation 11). Here, M in the equation (Equation 11) indicates the total number of pixels (the number obtained by adding the number of effective pixels of each two-dimensional image data in a plurality of acquired two-dimensional image data). However, if there is no imaging surface that can be received by a certain light receiving element in a certain photographing, the light receiving element is excluded. Further, an imaging surface that is not received by any light receiving element is excluded in all photographing.

Here, the matrix composed of v ₁ , v ₂ ,... V _M in the equation (Equation 11) is represented by v, a ₁₁ , a ₁₂ ,... A _MN is represented by A, u ₁ , If u is a matrix composed of u ₂ ,..., u _N , equation (Equation 11) can be written as v = Au. Since this equation can be converted as the following equation (Equation 12), the matrix u can be obtained. Thus, based on the equation (Equation 12), the reflected light intensity from each of the minute surfaces existing on the unknown straight line can be calculated.

  Thereafter, the minute surface inside the tube is estimated based on the distribution of reflected light intensity of the minute surface existing on the straight line, and the three-dimensional shape inside the tube is estimated based on the estimated minute surface. Build data. For example, when the minute surfaces existing on this straight line are arranged in the order of distance from the light receiving element 13a, if the distribution of the reflected light intensity at this time is a normal distribution, the minute surface that is the extreme value actually exists. It can be estimated that it is a minute surface inside the tube. Various methods for estimating the minute surface from the distribution of the reflection intensity can be selected. Then, three-dimensional shape data inside the tubular body can be constructed by sequentially connecting the estimated minute surfaces.

In the above-described cylindrical coordinates, the central axis extends on a straight line. However, when the imaging unit 1 is bent and moved (the movement locus of the principal point of the objective optical system 12 is a curve), A minute surface may be set based on the curved cylindrical coordinates with the curve as the central axis. The intensity of the reflected light from the minute surface varies depending on the inner product of the direction vector of the straight line connecting the principal point of the light receiving element 13a and the objective optical system 12 and the inward normal vector of the minute surface of the curved cylindrical coordinate system and the illumination condition. Therefore, the reference data and the value of a _mn may be determined accordingly.

  Further, the present invention is not limited to the above-described three-dimensional shape data creation inside the external auditory canal. For example, the one that reproduces the inner shape of various tubular bodies such as digestive tract such as intestine, airway, and water pipe, It can also be applied as a means of automatic identification of pathological conditions and damaged parts by applying.

1: imaging means 2: motion sensor 3: computer 10: otoscope 11: video scope 11a: tip portion 11b: bent portion 12: objective optical system 13: imaging element 13a: light receiving element 30: two-dimensional image data generating means 31: Spatial information acquisition means 32: three-dimensional shape data generation means

Claims (8)

Generating a plurality of two-dimensional image data based on a signal from an imaging means that is movable inside the tube and images the inside of the tube;
Obtaining spatial information of the imaging means at the time of imaging based on a signal from a motion sensor installed in the imaging means;
A step of associating the two-dimensional image data with the spatial information, and generating three-dimensional shape data inside the tubular body based on the two-dimensional image data and the spatial information. 3D shape data creation method. The imaging means includes
Illumination, an objective optical system, and an image sensor that receives reflected light from the inside of the tubular body arranged at an image forming position by the objective optical system,
The step of generating the three-dimensional shape data includes:
Based on the spatial information, the movement locus of the principal point of the objective optical system accompanying the movement of the imaging means is calculated, and the cylindrical coordinates with the movement locus as the central axis are set. Discretizing the space around the imaging means and setting the surrounding space to be composed of a plurality of minute surfaces according to the cylindrical coordinates;
Setting a straight line connecting each light receiving element of the image sensor and the principal point;
Determining whether the minute surface is present on the straight line; and
Determining whether reflected light from the minute surface can be incident on each light receiving element of the imaging element;
The pixel value in each pixel of the two-dimensional image data is set to be the sum of the reflected light intensity from each of the minute surfaces that exist on the straight line and can enter the light receiving elements, Calculating a reflected light intensity at each of the minute surfaces existing on the straight line based on a pixel value;
3D shape data inside the tubular body is estimated based on the estimated minute surface by estimating the minute surface inside the tubular body based on the distribution of reflected light intensity of the minute surface existing on the straight line. The method for creating three-dimensional shape data according to claim 1, further comprising: The imaging means includes
Illumination, an objective optical system, and an image sensor that receives reflected light from the inside of the tubular body arranged at an image forming position by the objective optical system,
The step of generating the three-dimensional shape data includes:
Obtaining reference data relating to pixel values according to the distance between the principal point of the objective optical system and the inside of the tube in advance;
Based on the spatial information, a movement trajectory of the principal point of the objective optical system accompanying the movement of the imaging means is calculated, cylindrical coordinates with the movement trajectory as a central axis are set, and a plurality of minute coordinates according to the cylindrical coordinates are set. Setting a cylinder of a predetermined radius composed of faces;
Setting a straight line connecting each light receiving element of the image sensor and the principal point;
Determining whether the minute surface is present on the straight line; and
Extracting the minute surface corresponding to the two-dimensional image data;
Obtaining a distance between the principal point at the position where the two-dimensional image data is acquired and the minute surface from the extracted pixel value corresponding to the minute surface based on the reference data;
The method for generating three-dimensional shape data according to claim 1, further comprising: calculating a shape of the tube according to the distance.   The method further comprises a step of correcting the spatial information of the imaging unit obtained by the motion sensor based on spatial information of the tubular body at the time of imaging obtained by another motion sensor. The three-dimensional shape data creation method according to any one of the above. Two-dimensional image data generating means that is movable inside the tubular body and that generates a plurality of two-dimensional image data based on a signal from an imaging means that images the inside of the tubular body;
Spatial information acquisition means for acquiring spatial information of the imaging means at the time of imaging based on a signal from a motion sensor installed in the imaging means;
3D shape data generation means for associating the 2D image data with the spatial information and generating 3D shape data inside the tubular body based on the 2D image data and the spatial information. 3D shape data creation system characterized by The imaging means includes
Lighting,
A tip having an objective optical system on which reflected light from the inside of the tubular body is incident;
A bent portion that supports the distal end portion and bendable;
An image sensor that is disposed at an imaging position by the objective optical system and receives reflected light from the inside of the tubular body, and
The three-dimensional shape data creation system according to claim 5, wherein the motion sensor is provided at the tip portion. Storage means for acquiring and storing reference data relating to pixel values according to the distance between the principal point of the objective optical system and the inside of the tube in advance;
The three-dimensional shape data generation means calculates a movement locus of the principal point of the objective optical system as the imaging means moves based on the spatial information, and sets cylindrical coordinates with the movement locus as a central axis. In addition, a cylinder having a predetermined radius composed of a plurality of minute surfaces according to the cylindrical coordinates is set, a straight line connecting each light receiving element of the imaging element and the principal point is set, and the minute surface exists on the straight line Determining whether or not to extract the minute surface corresponding to the two-dimensional image data, and obtaining the two-dimensional image data based on the reference data from the extracted pixel value corresponding to the minute surface The three-dimensional shape data creation system according to claim 6, wherein a distance between a principal point at a position and the minute surface is obtained. The three-dimensional shape data creation system according to any one of claims 5 to 7, further comprising holding means for directly or indirectly holding the tubular body.

Images