JP2022130354A - Data generation method and data generation system - Google Patents

Abstract

To provide a technique suitable for shape measurement of a tube body internal surface.SOLUTION: A data generation system 100 comprises: a spatial light modulator 140 and the like that continuously create images having a periodic pattern projected on a measurement target surface while shifting the phase for every predetermined time; an image fiber 110 that can project the created images toward the measurement target surface; an image pickup device 150 that can photograph the images projected on the measurement target surface through the image fiber 110; and an analysis and reconfiguration unit 164 that generates three-dimensional data according to the shape of the measurement target surface based on three-dimensional Fourier transformation on time and a two-dimensional space obtained from a plurality of photographed images photographed by the image pickup device 150 through a process in which a photographing position of the image fiber 110 relatively stops and changes, and a plurality of photographing positions at which the photographed images are obtained.SELECTED DRAWING: Figure 1

Description

The present invention relates to a data generation method and a data generation system for reproducing the three-dimensional shape of a surface to be measured on data.

This type of data generation technology has been used in various fields in the past. For example, known non-contact three-dimensional shape measurement methods include a stereo method using a plurality of cameras, a laser scanning method, and a fringe pattern projection method (see, for example, Patent Document 1). These methods mainly measure the surface shape of the object to be measured from the outside, but a method different from the above is used for measuring the shape of the inner wall surface of the tube.

For example, in order to create a shell for custom-made hearing aids, etc., it is necessary to measure the shape of the inner wall surface of the human ear canal. In this case, the conventional method is to collect the user's ear shape using an impression material in advance and capture the ear shape as data using a 3D scanner or the like. A method for measuring the three-dimensional shape of the ear canal in a non-contact manner is already known (see, for example, Patent Documents 2 and 3).

JP 2004-317495 A JP-A-9-10254 JP 2019-150574 A

When the purpose is to measure the shape of the inner wall surface of a narrow tube such as the human ear canal, the stereo method using multiple cameras is difficult to apply due to the large scale of the apparatus. In addition, when laser scanning is performed, the method of irradiating the laser light is limited in a narrow tube, and in order to scan the entire wall surface that needs to be measured without leaving anything, the measurer must be highly skilled. It is difficult to stably measure the shape.

In addition, the stripe pattern projection method of the prior art (Patent Document 1, etc.) mentioned above can only measure the shape in the range where an image can be obtained at the shooting position (visible from the outside), so the principle is not known. However, there is no known method for analyzing patterns projected onto a measurement surface such as the inner wall of a tube.

On the other hand, the prior art mentioned later (Patent Documents 2, 3, etc.) is specialized for measuring the shape of the inner wall surface of the tubular body, and some of them have been put into practical use. There is still much room for improvement, such as the long time required for image data processing in order to measure the shape of .

Accordingly, the present invention provides a technique suitable for measuring the shape of the inner wall surface of a tubular body.

The present invention provides a data generation method and data generation system. The data generation method has the following steps.

[First step]
In this step, an image (projection image) is taken by shifting the phase of an image having a periodic pattern projected onto the surface to be measured at every predetermined time. Further, such an operation is performed while relatively stopping and changing the image projection and photographing positions on the surface to be measured. Here, the image captured in this step is "projected" as a result of "projecting" the modulated light or the like onto the surface to be measured. It is an "image" in meaning. Therefore, in this step, as a result, the "image having a periodic pattern" is projected onto the "surface to be measured", and the "projection image" is performed in order to make it ready for photographing as a "projected image (projection image)". ”. In addition, the “projection position” of the “projection and photographing positions” is the position where the latent image (modulated light, etc.) is emitted toward the surface to be measured as described above, and the light that is the source of the image is irradiated. It can also be said that it is a position to do. The “shooting position” is the light receiving position when “shooting” the surface to be measured on which the image is projected.
[Second step]
In this step, three-dimensional data corresponding to the shape of the surface to be measured is obtained based on three-dimensional Fourier analysis in time and two-dimensional space obtained from multiple projection and imaging positions in the first step and multiple captured images. to generate

The data generation method of the present invention is effective for a surface to be measured, such as the wall surface of the ear canal, for which the entire image cannot be obtained in one shot. That is, by changing the projection of the image pattern that appears as an image on the surface to be measured and the shooting position, multiple shots are taken, and images of the hidden surface, etc., which could not be obtained in one shot, can also be acquired. . Changes in the projection and shooting positions appear in the image pattern as periodic characteristics different from the period obtained from the image pattern at rest, so multiple results obtained at each shooting position by three-dimensional Fourier analysis of time and two-dimensional space. By connecting the , the entire surface to be measured can be reconstructed (reproduced) on the data.

Also, the data generation method of the present invention acquires elements used for three-dimensional Fourier analysis of time and two-dimensional space by a characteristic method.

[First method]
That is, when the light modulated by the periodic pattern of the image as the light intensity distribution is projected from the end face of the image fiber onto the surface to be measured, the measured light passes through a pinhole virtually arranged at a position away from the end face in the projection direction. Suppose that the return light from the measurement surface is made incident on the end surface to form an image on a virtual imaging surface. In this case, by defining the coordinates on the virtual imaging plane on an extension line connecting the position of the surface to be measured that emits the return light and the position of the virtual pinhole, and performing the photographing (first step). , the coordinates representing the position of the surface to be measured can be obtained based on the time of the image formed on the virtual imaging surface and the three-dimensional Fourier analysis in the two-dimensional space (second step). Here, the end face of the image fiber is the above-mentioned "projection and imaging position".

[Second method]
Alternatively, separate image fibers can be provided for light projection (illumination) and return light incidence (observation). That is, a first optical system for projecting light modulated by a periodic image pattern as a light intensity distribution from the end surface of a first image fiber onto a surface to be measured; A second optical system is used for photographing by making the return light from the surface to be measured enter the end face of the second image fiber (first step). Then, the coordinates representing the position of the surface to be measured can be obtained based on the time of the image captured using the second optical system and the three-dimensional Fourier analysis in the two-dimensional space (second step). In this case, the end face of the first image fiber is the "projection position" of the image, and the end face of the second image fiber is the "photographing position".

In the second method, photographing is performed by arranging a rod lens on the end face of the second image fiber. The rod lens causes return light from the surface to be measured to enter the end face of the second image fiber through a principal point at a position shifted in the projection direction from the end face.

The data generation method of the present invention also discloses a method for knowing the projection and shooting positions of an image.

[First method]
That is, a predetermined reference plane is known, and by continuously projecting and photographing images onto the surface to be measured starting from the photographing position with respect to this reference plane, subsequent projection and photographing positions of images onto the surface to be measured are obtained. be able to.

[Second method]
Alternatively, a plurality of second image fibers arranged in parallel are used to perform photographing using a plurality of second optical systems, and a plurality of images photographed by the plurality of second optical systems are used by a stereo method. By continuously projecting and photographing an image onto the surface to be measured from the obtained photographing position as the starting point, the projection and photographing positions can also be obtained.

As a result, even when measuring while moving in a narrow and complex space such as the ear canal, analysis results can be obtained by knowing the shooting position when the captured image is acquired by continuity from the reference plane. It becomes possible to obtain with high precision.

A data generation system of the present invention has a configuration for realizing the above data generation method. That is, the system comprises an image generating means, an image fiber, an imaging means and a generating means. The image generating means continuously generates an image having a periodic pattern to be projected onto the surface to be measured by shifting the phase every predetermined time. Further, the image fiber can project an image generated by the image generating means (a latent image at the stage of generation) toward the surface to be measured. The imaging means can capture an image projected on the surface to be measured through an image fiber.

Then, the generation means generates three-dimensional data in time and two-dimensional space obtained from a plurality of captured images captured by the imaging means through the process of relatively stopping and changing the projection of the image of the image fiber on the surface to be measured and the capturing position. Three-dimensional data corresponding to the shape of the surface to be measured is generated based on the original Fourier analysis and the plurality of projections and imaging positions from which each captured image was obtained. Thereby, a system capable of executing the data generation method of the present invention is constructed.

Note that the image fibers may be composed separately of a first image fiber for projecting an image (for illumination) and a second image fiber for photographing (for observation). In addition, the image generation means can generate an image by splitting the laser light into two polarized components and causing them to interfere on the second image fiber, or can polarize the non-polarized light into two components, These two polarization components can also be separated with varying phase differences and merged again on a second image fiber to produce an image. Alternatively, an image generated by a dedicated microdisplay can be launched into the second image fiber.

ADVANTAGE OF THE INVENTION According to this invention, the technique suitable for the shape measurement of the wall surface in a tubular body can be provided.

It is a figure showing roughly an example of composition of data generation system 100 of one embodiment. FIG. 4 is a diagram showing an outline of operations when performing measurement using the data generation system 100; FIG. 4 is a series of diagrams showing a fringe pattern projected from an image fiber 110 and its phase change; It is the figure which showed the relationship between the x direction position of a striped pattern image, and a pixel value by contrasting still photography data and close-up photography data. It is the figure which showed the relationship of the y direction position of a striped pattern image, and a pixel value by contrasting still photography data and close-up photography data. FIG. 4 is a diagram showing static photographing data and close-up photographing data in comparison with each other in the time direction; 2 is a perspective view showing an optical system model of the data generation system 100 alone; FIG. FIG. 8 is a plan view of the optical system model of FIG. 7; 3 is an enlarged view of the vicinity of an end face 110A' of the image fiber 110; FIG. FIG. 4 is an explanatory diagram of a position and orientation detection method using a magnetic marker 170; FIG. 4 is an explanatory diagram of a position and orientation detection method using a magnetic marker 170; FIG. 4 is an explanatory diagram of a position and orientation detection method using a magnetic marker 170; FIG. 4 is an explanatory diagram of a position and orientation detection method using a magnetic marker 170; FIG. 4 is a series of diagrams showing simulation results using the technique of the embodiment; FIG. 4 is a series of diagrams showing simulation results using the technique of the embodiment; FIG. 4 is a series of diagrams showing simulation results using the technique of the embodiment; It is a figure which shows the result of reconstructing the shape of the tubular body wall surface EE on three-dimensional data from the simulation result. It is a figure showing the example of the 1st modification composition of an optical system model. It is the figure which showed the 2nd modified structural example of an optical system model. FIG. 11 is a diagram showing a third modified configuration example of the optical system model; It is the figure which showed the 4th modification structural example of an optical system model.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following description, while citing the configuration of the data generation system, a data generation method that can be implemented using the system will also be disclosed.

〔System configuration〕
FIG. 1 is a diagram schematically showing a configuration example including a surrounding environment of a data generation system 100 of one embodiment. Therefore, even if shown in FIG. 1, there are elements that are not essential components of the data generation system 100, and elements that are not shown in FIG. 1 may constitute the data generation system 100.

The data generation system 100 of the present embodiment has a configuration suitable for generating shape data of the inner wall surface of the tube by taking the external auditory canal EE of the human body as an example of the tubular body to be measured and using the wall surface of the external auditory canal EE as the surface to be measured. . In addition, such a use is only an example, and the tubular body to be measured is not limited to the external auditory canal EE, nasal cavity, larynx, digestive tract, and even the human body, various pipes, ducts, fluid ducts, etc. A tubular body can also be measured.

The data generation system 100 comprises at least the following optical system elements. That is, an image fiber 110, a laser light source 120, lenses 130, 131, 132, 133 and a polarizing beam splitter 134. FIG. Also, a quarter-wave plate 135 is appropriately arranged between the polarizing beam splitter 134 and the lens 130 . It should be noted that it is also possible to realize a configuration using a non-polarizing beam splitter, but in that case the quarter-wave plate 135 becomes unnecessary.

The data generation system 100 includes a spatial light modulator 140 and an imaging device 150 as electronic devices. The spatial light modulator 140 is used in combination with the polarizing beamsplitter 134 of the optical system. That is, the light from the laser light source 120 is reflected by the polarizing beam splitter 134 toward the spatial light modulator 140, and only the pixel light modulated by the spatial light modulator 140 is transmitted through the polarizing beam splitter 134, and the lens 130 , 131 , 133 to enter the end face 110 A of the image fiber 110 . Here, the quarter-wave plate 135 arranged after the polarizing beam splitter 134 is used to rotate the polarization state by 90 degrees between the forward and backward directions. As a result, the return light that has entered the image fiber 110 from the ear canal EE is reflected by the polarization beam splitter 134 and travels toward the imaging device 150 . Note that the spatial light modulator 140 also includes those used within the projector.

The image fiber 110 of the optical system is used as an objective instrument to be inserted inside the tubular body. 4 mm tube geometry) is selected. In addition, since the length of the ear canal EE is generally said to be about 3.5 cm (average for adults), considering that it applies to all people, the length of the image fiber 110 should be at least about It can be about 60 mm. However, the total length of the image fiber 110 can be extended as appropriate in consideration of handling (operability) of the image fiber 110 . In addition, when the data generation system 100 is used for other tubular bodies as described above, the size and length are appropriately selected in accordance with the objects to be measured.

As the characteristics of the laser light source 120 (light emitting element), for example, a wavelength of 532 nm and a light intensity of about 100 mW are preferably selected. A light-emitting element having such characteristics has the advantage of being relatively inexpensive and readily available with high light intensity. Another advantage of using a laser light source as a light emitting element is that it is easy to obtain characteristics close to the ideal with a simple lens system. However, the characteristics of the light-emitting element are not limited to this example, and can be appropriately selected as desired. For example, LEDs or other light sources may be used as long as they meet the required light intensity. However, if a light source other than a laser is used, an advanced optical system (lens) design that is not limited to the optical system shown in FIG. Enough possibilities.

A characteristic of the spatial light modulator 140 is that it can convert an input laser beam into a striped pattern necessary for data generation (shape measurement) by the system 100 . An example of light modulation (stripe pattern conversion) using the spatial light modulator 140 will be described later with reference to another drawing.

Regarding the imaging element 150, in this embodiment, considering that the inner wall of a narrow tubular body such as the external auditory canal EE of the human body is illuminated and photographed, the brightness of the returned light obtained is extremely limited. It is necessary to select the resolution and the number of pixels sufficient to capture the desired imaging range. The minimum number of pixels (resolution) that satisfies the principle of measurement (data generation) is 3 × 3, but high accuracy cannot be expected from results obtained with this degree of resolution, and the shape can be reproduced at once. Since the configurable range is also very narrow, it is desirable to set the resolution to have some degree of margin (redundancy) practically.

In addition, optical elements such as the lenses 130, 131, 132, 133, the polarization beam splitter 134, and the quarter wave plate 135 are selected with characteristics that can obtain necessary and sufficient data for the inner wall image of the ear canal EE. , shall be configured by calibrating their arrangement.

The data generation system 100 also comprises an electronic unit 160 . The electronic unit 160 can be configured by hardware such as computer equipment. The electronic unit 160 can include various functional elements using software processing in its interior. It has functional elements.

The image processing unit 162 controls the driving of the spatial light modulator 140, for example, and modulates the incident laser light into a necessary fringe pattern. Also, the image processing unit 162 controls the driving of the imaging device 150 , data-processes the imaging signal of each pixel, and sends the data to the analysis/reconstruction unit 164 . The analysis/reconstruction unit 164 executes three-dimensional analysis processing based on the data of the imaging signal to generate desired data. The shape of the wall surface of the ear canal EE, which is the object of measurement, is reconstructed (reproduced) on the generated data.

The data generation system 100 can include, for example, an image display 168 as a peripheral device. The image display device 168 is driven by the image processing unit 162 to display an image captured by the imaging device 150 on its screen. This allows the user/operator to insert the image fiber 110 while viewing the internal image of the ear canal EE.

Further, the data generation system 100 can have a configuration in which a driving mechanism 112 is added to the image fiber 110, for example. The drive mechanism 112 mechanically drives the image fiber 110 to mechanically perform an operation to enter the external auditory canal EE and an operation to exit from the inside. In addition, an operation unit 114 may be attached to the drive mechanism 112 as appropriate, and the user/operator may operate the operation unit 114 to control the operation of the drive mechanism 112 . The driving mechanism 112 can be given a precision capable of moving, for example, the object-side end surface 110A' of the image fiber 110 in minute steps. For example, in the case of manual operation, the moving step of the image fiber 110 is manually adjusted while the user/operator visually recognizes the screen of the image display device 168. However, in the case of automating the drive mechanism 112, In order to avoid collision with the ear canal EE wall, it is desirable to be able to control the movement step, for example, on the order of 0.1 mm.

Further, in this embodiment, the image fiber 110 can be bent as desired by wire manipulation or the like. This makes it possible to insert the image fiber 110 deep into the ear canal EE without contact while deforming the image fiber 110 according to the shape of the ear canal EE. The configuration necessary for wire manipulation of the image fiber 110 is provided in the drive mechanism 112, and the configuration and operation thereof can be applied to those known in the field of medical equipment (endoscopes) and the like. Detailed description is omitted.

In addition, the data generation system 100 can include detection devices as peripheral equipment (auxiliaries). Specifically, a magnetic marker 170 is installed at the insertion end of the image fiber 110, and a plurality of magnetic sensors 170a to 170f are arranged around the external auditory canal EE to be measured. The magnetic sensors 170 a to 170 f detect the magnetic field intensity (magnetic flux density) of the magnetic marker 170 at respective positions and input the detection signals to the electronic unit 160 . The detection processing unit 166 processes the detection signal and converts the position and orientation (angle) of the magnetic marker 170 into known data. A detection process using such a detection device will also be described later using another figure.

[Operation overview]
FIG. 2 is a diagram showing an outline of operations when performing measurement using the data generation system 100. As shown in FIG. FIG. 2 also shows a form example in which a part of the configuration of the data generation system 100 is embodied in the hand unit HU.

FIG. 2A: The hand unit HU on which the components of the data generation system 100 are mounted (mounted) is embodied in a form that can be held by a user/operator with one hand, for example. As an example, the hand unit HU can be made into a gun shape, and the user/operator can carry it by using the grip portion as a handle. An image fiber 110 extends from the front end of the hand unit HU, and the user/operator inserts the tip portion into the human ear canal EE while holding the image fiber 110 with the other hand. At this time, the driving mechanism 112 can be used as described above to automate the process, and the image fiber 110 can be bent as desired. Also, the screen of the image display device 168 may be provided on the rear end surface of the hand unit HU or the like. Note that the magnetic marker 170 may be omitted.

Alternatively, the function of the electronic unit 160 may be placed in an external device (not shown), and the hand unit HU may incorporate a communication device to enable wireless or wired communication with the external device. In this case, the image display device 168 and the analysis/reconstruction unit 164 may be placed inside the hand unit HU or may be placed in an external device. When they are placed in an external device, they are transmitted by wired connection or wireless connection using a protocol having a sufficient transmission capacity for sequentially transmitting the imaging data of the imaging element 150 .

The laser light source 120 described above may be built in the hand unit HU, or a separate laser oscillator or the like may be used to generate laser light externally, and a fiber or the like (not shown) may be used to introduce the laser light to a required portion. . In this case, the hand unit HU is provided with an optical connector, a ferrule, etc. for introducing light from the external laser light source 120 .

The power supplied to the hand unit HU may be supplied from the outside, or may be driven by an internal battery. When the hand unit HU is driven by a battery, the hand unit HU is held in the hand of the user/operator in order to delicately measure the wall shape of the external auditory canal EE. is required.

The shape of the ear canal EE is measured by projecting stripe pattern light from the tip of the image fiber 110 on the object side. Such striped pattern light is projected onto the surface to be measured to form a striped pattern image. At this time, as shown in FIG. 2A, a small sticker S is affixed to the concha E1. This is to obtain the imaged data of the reference plane leading to the external auditory canal EE by utilizing the fact that the human ear concha E1 generally has a planar shape. In the technique of this embodiment, when reconstructing the shape from the photographed image data, the position and orientation (projection direction) of the end surface 110A' of the image fiber 110 must be known, and the position and orientation are also available in a plurality of images. It is supposed to analyze from the image data of For example, by taking an image so that a portion known to be a flat surface is captured in the image, the position and orientation of the end surface 110A' are estimated based on the flat surface. As one specific method for this purpose, a small sticker S is attached to the concha E1. A method for estimating the position and orientation from the imaging data of the reference plane will be described later in detail.

FIG. 2(B): An example of an image captured outside the ear canal EE (outside the entrance). It should be noted that examples of images are conceptually shown here (the same applies hereinafter). A fringe pattern FG is projected as an image on the external auditory canal EE and the ear concha cavity E1 around it. In addition, the sticker S attached to the concha E1 is also reflected in the image. The fringe pattern FG is a parallel straight line pattern in the original state modulated by the spatial light modulator 140, but in the image, it appears deformed according to the shape of the projection plane. Although the striped pattern FG is considered to appear on the entire screen, only a portion of the striped pattern FG is shown here, and most of it is omitted for convenience of illustration and to prevent complication (the same applies hereinafter). .).

FIG. 2(C): An image when the end surface 110A' of the image fiber 110 enters the ear canal EE. Since the shape of the ear canal EE is complicated, it can be seen that there is a hidden surface that cannot be fully captured in the image at this photographing position and angle (orientation). In this embodiment, by inserting the image fiber 110 into the ear canal EE and taking multiple shots at different shooting positions and angles, even a shadowed surface is shot in one shot.

FIG. 2(D): An image when the end face 110A' of the image fiber 110 is inserted deep into the ear canal EE. In order to create a custom-made hearing aid shell, it is necessary to measure the shape further into the ear canal EE. It can be seen that it is difficult to measure the shape of such a narrow tube inner wall surface only by the conventionally known stereo method or fringe projection method on the outer surface of the object.

[Stripe pattern]
FIG. 3 is a series of diagrams showing the fringe pattern projected from the image fiber 110 and its phase change. Among them, (A) to (D) in the upper part of FIG. 3 show the phase change of the fringe pattern in the original data state modulated by the spatial light modulator 140, and (I) to (IV) in the lower part of FIG. 8 shows the phase change (simulation result) of the fringe pattern of images obtained by projecting the upper fringe pattern onto the surface to be measured while moving the image fiber 110 and continuously photographing the fringe pattern.

In this embodiment, the phase of the projected fringe pattern is shifted at regular time intervals, and an image is acquired in synchronization with that time. 3A is the first stripe pattern FGS01, and the number of pixels of the original data modulated by the spatial light modulator 140 is 100×100, the horizontal direction (x direction) and the vertical direction (y direction) are 100×100. A periodic pattern is generated in the two-dimensional space (address space) of the direction). The shading in the drawing represents the modulated light intensity distribution. Although the pixel address notation is (1, 1) to (100, 100) here, it may be (0, 0) to (99, 99).

[Still photography data of striped pattern]
The fringe pattern whose phase is shifted at regular time intervals changes in phase as shown in FIG. 3(A)→(B)→(C)→(D). Assuming that the number of images changes at regular intervals, the image in FIG. 3B is the 23rd striped pattern FGS23, the image in FIG. 3C is the 46th striped pattern FGS46, and The image of D) is the 92nd striped pattern FGS92. As can be seen by comparing the same one pixel (for example, the pixel of address (100, 100)) in FIGS. 3A to 3D, it can be seen that the pixel value changes due to the phase change of the fringe pattern. Such an image is the result of photographing the fringe pattern projected onto the surface to be measured at a still (relatively stationary) photographing position.

[Close-up data of striped pattern]
3A to 3D are the original data of the striped pattern (at the time of static photography), and such a striped pattern is changed (approached) relatively to the surface to be measured. The results of photographing at progressive photographing positions are represented by images in FIGS. 3(I) to (IV). Photographing is performed in synchronization with the timing of shifting the phase of the fringe pattern at regular intervals, and the 1st, 23rd, 46th, and 92nd images in FIGS. 3(A) to (D) are photographed respectively. The results are the stripe patterns FGM01, FGM23, FGM46, and FGM92 of FIGS. 3(I) to (IV). As can be seen by comparing the same one pixel (for example, the pixel of address (100, 100)) in FIGS. It can be seen that the pixel value changes due to the phase change, and in addition to this, the pixel value also changes due to the influence of the movement of the shooting position. Therefore, comparing the same one pixel (for example, the pixel at the address (100, 100)) between (A) to (D) in the upper part of FIG. 3 and (I) to (IV) in the lower part of FIG. It can be seen that there is a difference in the characteristic of the periodic fringe pattern between the photographed data (during proximity) and the photographed data (during close proximity).

[Two-dimensional space]
FIG. 4 is a diagram showing the relationship between the x-direction position of the striped pattern image and the pixel value by comparing the static photographed data and the close-up photographed data. FIG. 5 is a diagram showing the relationship between the y-direction position of the striped pattern image and the pixel value by comparing the static photographed data and the close-up photographed data. In FIGS. 4 and 5, the horizontal axis represents the position of the pixel (what number the pixel is), and the vertical axis represents the pixel value (for example, maximum 255). 4 plots pixels with addresses (1, 1) to (100, 1) in the x direction, and FIG. 5 plots pixels with addresses (1, 1) to (1, 100) in the y direction. is doing. As is clear from FIGS. 4 and 5, in both the x direction and the y direction of the image, the close-up data changes in a periodic pattern in the two-dimensional space with respect to the still image data. I understand.

[Time direction]
FIG. 6 is a diagram showing static photographing data and close-up photographing data in comparison in the time direction. Here, one pixel with a photographed image (for example, the pixel at the address (100, 100) shown in the upper and lower rows) is focused, and the horizontal axis is time (which image was photographed). , the vertical axis represents the pixel value at that time. As is clear from FIG. 6, the pixel data appears as a periodic pattern also in the time direction, and the close-up data changes in the periodic pattern also in the time direction with respect to the still image data. (for example, the difference between the still image data pixel value PVS23 in the 23rd image and the close-up image data pixel value PVM23, the difference between the still image data pixel value PVS46 in the 46th image and the close-up image data pixel value PVM46, the 92nd image difference between the still photographing data pixel value PVS92 and the close-up photographing data pixel value PVM92, the difference between the still photographing data pixel value PVS100 of the 100th image and the close-up photographing data pixel value PVM100, etc.).

Thus, in this embodiment, when the phase of the fringe pattern to be projected is shifted at regular time intervals and images are acquired in synchronization with that time, a periodic pattern is generated in the time direction and in the two-dimensional space. . As a result, it is possible to reconstruct the shape of the inner wall of the tube on data from three-dimensional Fourier analysis of time and two-dimensional space. The analysis/reconstruction will be further described below.

[Generated stripe pattern]
As described above, data generation system 100 generates fringe patterns with spatial light modulator 140 . Here, if the coordinates of the pixel position of the spatial light modulator 140 are defined as (x ^(M) , y ^(M) ), the fringe patterns generated by the spatial light modulator 140 are the following four types (equation (1) (4)) or a combination thereof.

Here, a ^(M) , b ^(M) , K _x , K _y , Ω, and φ ₀ in the above equations (1) to (4) are constants, and a ^(M) ≧b ^(M) , K _x >0, K _y >0, and Ω>0. φ ₀ can be arbitrary.
K _x is set so that K _x Δx ^(M) < π, where Δx ^(M) is the pixel pitch in the x direction of the spatial light modulator 140 . Here, for example, K _x Δx ^(M) = π/2.
K _y is set so that K _y Δy ^(M) < π, where Δy ^(M) is the pixel pitch in the y direction of the spatial light modulator 140 . Here, for example, K _y Δy ^(M) =π/2.
t is time, and the fringe pattern of the spatial light modulator 140 is changed every constant time Δt. This is changed under the condition of ΩΔt<π. Here, for example, ΩΔt=π/2.

In order to project the fringe pattern generated under such conditions onto the inner wall of the tube such as the external auditory canal EE, the fringe pattern must be widened according to the distance. Therefore, the data generation system 100 uses an optical system as described above.

[Optical system model]
FIG. 7 is a perspective view showing an optical system model of the data generation system 100 alone. 8 is a plan view of the optical system model of FIG. 7. FIG. 9 is an enlarged view of the vicinity of the end face 110A' of the image fiber 110. FIG. Since each component of the optical system has already been described, the same components as those shown in FIG. 1 are denoted by the same reference numerals, and redundant description is omitted. For the sake of simplification, it is assumed here that the object to be measured is represented as a tubular body PT having a straight tubular shape, and that the interior thereof is the external auditory canal EE. Also, the concha cavity E1, which is a known plane, is defined as a reference plane E1' around the inlet of the tubular body PT.

で表される。つまり、端面１１０Ａ’と光軸の交点を原点とした投影位置を（ｘ'^(S)，ｙ'^(S)，ｚ'^(S)）とすると、次式（５）（６）

の関係になる。 8A and 8B: If the focal point of the system of lenses 130, 131, 133 is adjusted to the end face 110A of the image fiber 110, the pattern of the focal plane is transferred to the other end face 110A' on the objective side. The other end face 110A' can be considered the focal plane. At this time, if the focal length of the lens system 130, 131, 133 is F, and the distance in the normal direction from the end surface 110A' is z' ^(S) , the magnification of the fringe pattern is given by the following equation (Equation 2)

is represented by That is, if the projection position with the origin being the intersection of the end face 110A' and the optical axis is (x' ^(S) , y' ^(S) , z' ^(S) ), the following equations (5) and (6)

become a relationship.

The optical system model acquires the return light from the projection position (x' ^(S) , y' ^(S) , z' ^(S) ) with the image sensor 150. At this time, the image of the return light Assuming that a virtual point 152' (FIG. 7(B)) is placed on a plane away from the end face 110A' of the fiber 110 in the projection direction of the fringe pattern, the imaging element 150 has The physical configuration is such that an image is formed by the lens 132 on the front surface in the incident direction. For example, by placing a physical pinhole 152 (spatial filter) between the imaging element 150 and the lens 132, a virtual pinhole is placed at a point 152' away from the end face 110A' of the image fiber 110. An optical system equivalent to the situation is constructed.

This point 152' is called a virtual pinhole 152' in the optical system model. By creating a situation in which the virtual pinhole 152' is arranged on the end face 110A' side of the image fiber 110, the projection position (x' ^(S) , y' ^(S) , z' ^(S) ) and the virtual pinhole 152' A virtual imaging element plane 150' is placed on an extension line connecting the positions of (see FIG. 9). At this time, return light from the projection positions (x' ^(S) , y' ^(S) , z' ^(S) ) enters the pixels of the virtual image sensor plane 150' (actually the pixels of the image sensor 150). It will be.

In FIG. 9, the virtual imaging element surface 150′ is imaginarily placed. ' corresponds to the imaginary end surface 110A'' in front of the imaging device 150 shown in FIGS. 7A and 8A.

という関係が成り立つ。ここで、仮想ピンホール１５２’の位置を（ｘ^(p)，ｙ^(p)，ｚ^(p)）としている。また、各画素における（ｘ'^(S)，ｙ'^(S)，ｚ'^(S)）は，その画素とピンホール１５２を結ぶ直線の延長線上のどこかの点になるが、この直線上を動く点の位置を表現する媒介変数がｌとなる。そうすると、上記４種類の縞パターンは、撮像素子１５０面上では、次式（１０）

と表すことができ、上式のφが次の４種類となる（式（１１）～（１４））。

FIG. 8A: Assuming that the position of the pinhole 152 is the origin and the coordinates on the surface of the imaging element 150 are (x, y, z), the following equations (7) to (9) are obtained.

A relationship is established. Here, the position of the virtual pinhole 152' is (x ^(p) , y ^(p) , z ^(p) ). In addition, (x' ^(S) , y' ^(S) , z' ^(S) ) at each pixel is somewhere on the extension line of the straight line connecting the pixel and the pinhole 152. On this straight line A parameter representing the position of the point moving is l. Then, the above four stripe patterns are expressed by the following equation (10) on the surface of the imaging element 150.

φ in the above equation has the following four types (equations (11) to (14)).

Also, a and b in the above equation depend on the intensity of the return light, and have a relationship of a≧b. Here, if φ can be obtained by some means, it is considered that l can be obtained using any one of equations (11) to (14). Once l is obtained, the wall surface coordinates (x' ^(S) , y' ^(S) , z' ^(S) ) with the center of the focal plane as the origin are obtained from equations (7) to (9).

のように求まる。上式のａ_ｉｊは、変換行列であるアフィン行列の要素である。アフィン行列は、回転行列や並進移動を表す行列を掛け合わせて構成され、任意の座標変換を表現できる。そして、φが得られ、アフィン行列が既知であれば、上式の壁面座標（ｘ^(S)，ｙ^(S)，ｚ^(S)）が得られることになる。 Here, considering the position and attitude of the end surface 110A' of the image fiber 110, the wall surface coordinates (x' ^(S) , y' ^(S) , z' ^(S) ) are expressed by the following equation (Equation 7)

It is found like _aij in the above equation is an element of an affine matrix, which is a transformation matrix. Affine matrices are constructed by multiplying rotation matrices and translation matrices, and can express any coordinate transformation. Then, if φ is obtained and the affine matrix is known, the wall surface coordinates (x ^(S) , y ^(S) , z ^(S) ) of the above equation will be obtained.

のように変形できるが、これをｘ，ｙ，ｔで三次元フーリエ変換すると、次式（１６）

のようになり、３つの項は（ν_ｘ，ν_ｙ，ν_ｔ）空間上で分離される。ここで、ＦＴはフーリエ変換を表す演算子を意味する。 [Three-dimensional Fourier analysis method]
Here, a three-dimensional Fourier analysis method can be used as a method of obtaining φ from the time-varying fringe pattern image in the form of Equation (10). Equation (14) is obtained by the following equation (15) from Euler's formula

When this is three-dimensionally Fourier-transformed with x, y, and t, the following equation (16)

and the three terms are separated on the (ν _x , ν _y , ν _t ) space. Here, FT means an operator representing Fourier transform.

とする。ここでは大きさは重要ではないため、１／２という定数は省略している。これを逆フーリエ変換すると、次式（１８）

が求まる。さらに、ν_ｘＧ_１，ν_ｙＧ_１，ν_ｔＧ_１の逆フーリエ変換を求めると、それぞれ次式（１９）～（２１）

となる。ここで、ｉＦＴは逆フーリエ変換の演算子を意味する。上式（１８）～（２１）より、次式（２２）～（２４）

のように、φの勾配が求まる。Ｒｅは実部を表す。これを（ｘ，ｙ，ｔ）空間における任意の経路Ｃで座標（ｘ，ｙ，ｔ）まで線積分すると、φが求まることになる（次式（２５））。

ここで、ｄｌは線素ベクトルである。ただし、線積分の始点におけるφが既知でなければならないが、従来これを既知とする方法が知られていなかった。 The three-dimensional space consists of eight quadrants. The first term in Equation (16) is separated around the origin, and the second and third terms are separated into quadrants that are symmetrical with respect to the origin. At this time, one striped pattern occupies two quadrants, but the four striped patterns occupy different quadrants. , which can be separated here. The second or third term of formula (16) may be used, but here, the second term is taken out, and the following formula (17)

and The constant 1/2 has been omitted, since size is not important here. Inverse Fourier transform of this gives the following equation (18)

is sought. Further, the inverse Fourier transforms of ν _x G ₁ , ν _y G ₁ , and ν _t G ₁ are obtained by the following equations (19) to (21), respectively.

becomes. Here, iFT means an inverse Fourier transform operator. From the above formulas (18) to (21), the following formulas (22) to (24)

The gradient of φ is obtained as follows. Re represents the real part. By line-integrating this to coordinates (x, y, t) along an arbitrary path C in the (x, y, t) space, φ can be obtained (equation (25) below).

where dl is the line element vector. However, although φ at the starting point of the line integral must be known, there has been no known method of making this known.

のように表すことができる。ここで、（ｘ^(S)，ｙ^(S)）は平面上の座標であり、ａ_ｘ，ａ_ｙは平面の傾きを表すパラメータであるとする。
上式（２６）～（２８）と式（７）～（９）より、次式（２９）～（３１）

という連立方程式を立てることができる。これを解くと、次式（３２）～（３４）

となる。上式（３４）のｌを用いて４種類の縞パターンに対応する∂φ／∂ｘ，∂φ／∂ｙを表現すると、次式（３５）～（４２）

となる。ここから上式（３５）～（４２）を使って、画像の基準平面Ｅ１’が映っている部分、例えば画像の枠線上における∂φ／∂ｘ，∂φ／∂ｙを用いて関数フィッティングを行えば、ａ_ｘ，ａ_ｙが求まることになる。ａ_ｘ，ａ_ｙが求まれば、次に式（３２）～（３４）より、平面上の座標（ｘ^(S)，ｙ^(S)）とｌが求められる。そして、ｌが求まれば、式（７）～（９）より、（ｘ’^(S)，ｙ’^(S)，ｚ’^(S)）が求められる。また、式（２８）において、ｘ^(S)＝０，ｙ^(S)＝０としたときのｚ’^(S)の値をｚ^(S)とする。これにより、傾きがない場合の平面上の座標（ｘ^(S)，ｙ^(S)，ｚ^(S)）を求めることができる。 Therefore, the inventor of the present invention presents the following as a method for obtaining the starting point.
First, the first image is captured so that a plane (which is known to be a plane in advance) appears around the tubular body PT. At this time, the coordinates on the plane are given by the following equations (26) to (28)

can be expressed as Here, (x ^(S) , y ^(S) ) are coordinates on the plane, and a _x , a _y are parameters representing the inclination of the plane.
From the above formulas (26) to (28) and formulas (7) to (9), the following formulas (29) to (31)

A system of equations can be established. Solving this, the following equations (32) to (34)

becomes. When ∂φ/∂x and ∂φ/∂y corresponding to the four stripe patterns are expressed using l in the above equation (34), the following equations (35) to (42) are obtained.

becomes. From this, using the above equations (35) to (42), function fitting is performed using ∂φ/∂x and ∂φ/∂y on the portion where the reference plane E1′ of the image is reflected, for example, on the frame line of the image. By doing so, a _x and a _y are obtained. Once a _x and a _y have been obtained, coordinates (x ^(S) , y ^(S) ) and l on the plane are obtained from equations (32) to (34). Then, if l is found, (x' ^(S) , y' ^(S) , z' ^(S) ) can be found from equations (7) to (9). In equation (28), let z ⁽ S) be the value of z' ^(S) when x ^(S) =0 and y ^(S) =0. As a result, the coordinates (x ^(S) , y ^(S) , z ^(S) ) on the plane when there is no tilt can be obtained.

As a result, an arbitrary position on the reference plane E1' that appears in the first photographed image can be obtained. That is, select three pixels that are known to be imaging data from the reference plane E1′, and (x ^(S) , y ^(S) , z ^(S) ) and (x′ ^{( S)} , y' ^(S) , z' ^(S) ) can be used to obtain an affine matrix representing the initial orientation. Further, by using this data as the starting point of line integration and performing line integration within the xy plane, the shape of the vicinity of the inlet of the tubular body PT can be obtained.

The first three pixels to be selected are, for example, the three vertices of the frame (rectangle) of the image. By linearly integrating these three pixels in the time direction, it is possible to obtain the temporal change of (x' ^(S) , y' ^(S) , z' ^(S) ) of the wall surface reflected in the pixel. (x' ^(S) , y' ^(S) , z' ^(S) ) in the second photographed image can be considered to indicate the position of the wall found in the first photographed image. . Therefore, with the elements of the affine matrix representing translational movement and rotation as unknown parameters, an objective function φ containing unknown parameters is considered using the wall surface position obtained from the first photographed image, and three pixels (x' ^{(S )} , y' ^(S) , z' ^(S) ) to obtain an affine matrix. Thus, the position of the wall surface in the second photographed image is obtained.

Next, the wall surface position obtained in the second photographed image is used to obtain the wall surface position in the third photographed image. Thereafter, the wall surfaces can be sequentially connected in this manner. However, since the minimum number of 3 may not be practically accurate, we use 4 vertices of the frame of the image, as well as multiple points on the frame, for each time position and orientation information. is more preferable.

In order to perform function fitting, φ needs to be continuous and differentiable at all the points used. When line integration was performed in the time direction, the intensity of the light returned from the wall surface was weak and the photographed image could not be obtained in some cases. If there is a moment, the φ of that pixel is treated as something that cannot be used thereafter.

Further, in the same image, φ in other pixels can be obtained by performing line integration in the xy plane from a pixel having a known φ. φ obtained by line integration from the starting point of does not match. By this, it is determined whether or not φ can be acquired normally. If the number of photographed pixels that can be normally acquired decreases, the photographed image is acquired again from the beginning, or the position and orientation of the end surface 110A' of the image fiber 110 are obtained by the method using the magnetic marker 170 described above. , the position of the outer wall reflected in each pixel can be obtained from the shape of the outer wall obtained at that time.

[Method using a magnetic marker]
10 to 13 are explanatory diagrams of the position and orientation detection method using the magnetic marker 170. FIG. A description will be given below in order.

The shape of the magnetic marker 170 is, for example, cylindrical (disk-like), the diameter in the xy direction is 4 mm, and the height (thickness) in the z direction is 1 mm, for example. The material of the magnetic marker 170 is, for example, NC52 (JIS), and the residual magnetic flux density is about 1.45T. At this time, FIG. 10 shows the magnetic field direction of the magnetic marker 170, FIG. 10(A) is plotted in the xyz space, FIG. C) is plotted in the yz plane. In both cases, simulation results are shown for reference.

The magnetic marker 170 has a north pole in the y half 170N and a south pole in the other half 170S. As shown in FIG. 10, the magnetic field directions of the magnetic markers 170 are distributed in the radial direction, thereby facilitating detection of the position and orientation as follows.

[Coordinate system definition]
First, as shown in FIG. 11, a coordinate system whose origin is the initial position of a ring-shaped magnetic marker 170 attached to the tip of the image fiber 110 is defined. That is, the correspondence relationship between the position of an arbitrary point around the magnetic marker 170 and the magnetic flux density is formulated, and the obtained values of the magnetic sensors 170a to 170d are fitted using the equation. Thereby, the positional relationship between the magnetic marker 170 and the magnetic sensors 170a to 170d and the like is obtained. In the example of FIG. 11, magnetic field distribution curves are obtained with the initial position as the origin in each of the x, y, and z directions.

[Least-squares method]
Next, to estimate the position and orientation of the magnetic marker 170, it is necessary to obtain a position vector (three-dimensional), which is an unknown quantity, and a rotation angle vector (three-dimensional) with respect to the initial position of the magnetic marker 170. FIG. Therefore, in order to obtain these six variables, magnetic sensors 170a-170f (not shown in FIG. 11) are arranged at six known different locations as shown in FIG. For example, the magnetic sensors 170a to 170f are fixed to a device whose position is relatively fixed with respect to the ear canal EE (human head), and the coordinate system is kept stationary even if the head moves during measurement. do. Then, the least-squares method is applied again to the equation after fitting, and the position and orientation are estimated by finding six variables. Since each of the magnetic sensors 170a to 170f has sensitivities in three axial directions, three data can be obtained from each one. Therefore, 9 data are obtained even at 3 locations, and it is possible to obtain 6 variables from them. However, according to simulations conducted by the inventors of the present invention, it has been found that a preferable result can be obtained when there are five or more positions.

[Position estimation]
As shown in FIGS. 12 and 13, moving the image fiber 110 also moves the magnetic marker 170 accordingly. From this, the estimated position coordinates (x ₁ , y ₁ , z ₁ ) and (x ₃ , y ₃ , z ₃ ) of the magnetic marker 170 can be obtained by the above method. Similarly, the posture can also be obtained.

〔simulation result〕
FIGS. 14 to 16 are continuous diagrams showing the simulation results when the fringe pattern is projected with a phase shift using the above method of the present embodiment and photographed by the image fiber 110 while moving inside the tubular body PT. be.

In FIG. 14A, the fringe pattern FG1 is projected onto the reference plane E1' and the tube wall surface EE. The starting point is estimated as described above by using at least three photographed image data of such a reference plane E1'. At this time, it can be seen that the intensity of the return light from the tubular body wall surface EE is weaker than that on the reference plane E1', and the image is dark. In addition, no stripe pattern image is obtained at the deepest part of the tubular body PT.

In FIG. 14B, the image fiber 110 has moved from the starting position and is close to the tubular body wall surface EE. More of the fringe pattern FG2 is projected onto the tubular body wall surface EE than on the reference plane E1'. Moreover, periodic pattern changes in two-dimensional space appear in the stripe pattern FG2.

In FIG. 15C, the image fiber 110 has moved further and is very close to the tube wall surface EE. Most of the fringe pattern FG3 is projected onto the tube wall surface EE.
In FIG. 15(D), the image fiber 110 is inserted inside the tube. Therefore, the stripe pattern FG4 is projected only on the tube wall surface EE.

In FIG. 16(E), the image fiber 110 is inserted further inside the tubular body. Similarly, the striped pattern FG5 is projected only on the tubular body wall surface EE.
In FIG. 16F, the image fiber 110 is inserted to the deepest position inside the tubular body. The fringe pattern FG6 is projected only on the tube wall surface EE.

FIG. 17 is a diagram showing a result of reconstructing the shape of the tubular body wall surface EE on three-dimensional data from the above simulation results. As described above, according to the present embodiment, it is possible to project and photograph a fringe pattern while inserting the image fiber 110 into the tubular body, and generate data DT corresponding to the shape of the surface to be measured therefrom.

[Summary of data generation method]
Moreover, when the above is put together as a data generation method, it becomes the following.
(1) A fringe pattern whose phase changes with time is projected (irradiated) onto the inner wall of a tube such as the external auditory canal EE through the image fiber 110, and an image of the projected fringes is captured. Also, a photographed image of the inner wall of the tube is acquired through the same image fiber 110 .
(2) The fringe pattern is projected and photographed while changing the phase by 90 degrees as described above. Although there is no condition for the starting point of the phase of the fringe pattern, it is necessary to express one cycle using at least three pixels in both the horizontal and vertical directions. For this reason, depending on the location of the surface to be measured, some locations that do not satisfy the conditions will appear, but such locations are excluded when data conversion and reconstruction are performed by the analysis/reconstruction unit 164 .

(3) The reason for changing the phase of the fringe pattern is to identify whether the image fiber 110 is stationary or moving. When the image fiber 110 is moving, the imaging position is moving, and when it is not moving, the imaging position is stopped.
(4) The number of images to be shot at the same shooting position can be determined by the user/operator while viewing the monitor such as the image display 168, for example. At this time, if the wall surface of the ear canal EE is approached or moved away from the wall surface of the ear canal EE, the phase change of the fringe pattern deviates from 90 degrees. Shooting is required.
(5) Furthermore, when converting and reconstructing data from an image of the inner wall shape of a curved tubular body such as the external auditory canal EE, there are hidden surfaces that cannot be captured in a single shot, so there is a point ( In order to reconstruct the shape of a certain range), the shape is reconstructed from images obtained by taking multiple shots at different shooting positions and angles (viewpoints) from the image fiber 110 and converted into data.

(6) The phase of the projected fringe pattern is shifted at regular time intervals as described above, and an image is photographed/obtained in synchronization with the time. The fixed time of the photographing interval can be arbitrarily set, but the condition is that the timing of photographing and the timing of phase change of the fringe pattern match.
(7) It is desirable that the movement step of the end face 110A' of the image fiber 110 is such that the phase change does not change by 180 degrees or more, including the 90-degree phase change.
(8) As for the temporal change of the fringe pattern as described above, in the state where the image fiber 110 is stationary and the same image is captured, one example is to set the phase so that it changes by 90 degrees for each photographing. .
(9) As for the spatial period, the finer the period, the higher the accuracy of the shape converted into data, but the more sensitive it changes to the movement of the image fiber 110 . In other words, unless the image fiber 110 is finely moved in space, the phase change becomes excessively large, which becomes a problem during data generation.
(10) Ultimately, when it is desired to obtain data on the shape of the ear canal for creating a custom-made shell for a hearing aid, a precision of about 0.1 mm is required for the manufactured product, so repetition of the projection of the fringe pattern is required. The period should be at most less than 1 mm. In addition, since the period of the fringe pattern varies depending on the distance from the end face 110A' of the image fiber 110, it is necessary to set the optical system so that the period is less than 1 mm over the entire photographing range.
Although it is assumed that no objective lens is provided at the tip of the image fiber 110, a configuration with a lens is also possible. The rear side from the end surface 110A' of the can also be photographed.

The data generation system 100 of the embodiment described above can also be implemented by partially modifying some configuration examples given below.

[First modified configuration example]
FIG. 18 is a diagram showing a first modified configuration example of the optical system model of the data generation system 100. As shown in FIG. Here, the description will focus on the components that are different from the previous embodiment due to the modification, and the components that are the same as those already shown in FIGS. omitted.

[Configuration example with separate image fibers for projecting stripe pattern light and for photographing]
FIG. 18(A): In the first modified configuration example, the image fiber 110 is used for projecting the fringe pattern light to form a single irradiation optical system, and a photographing image fiber 210 is added. The imaging image fiber 210 has an object-side end face 210A′ arranged in parallel with the object-side end face 110A′ of the projection image fiber 110, and the tip portion inserted into the tubular body is parallel to the projection image fiber 110. bundled in The photographing image fiber 210 causes the return light from the ear canal EE to be incident on the end surface 210A′ on the object side, and from the opposite end surface 210A to the imaging element 150 through an optical system independent from the projection image fiber 110. can be imaged.

In the optical system model of the previous embodiment (FIGS. 7 to 9), one image fiber 110 has a common configuration for projecting the striped pattern light and photographing the returned light, so the light reflected at the end surface 110A is imaged. It is conceivable that a noise component may be generated toward the element surface of the element 150 . Even in this case, the image fiber 110 can be designed to prevent reflection on the end face 110A, so the noise problem can be sufficiently resolved.

On the other hand, in the case of the modified configuration example, since the image fiber for projection 110 and the image fiber for photographing 210 are provided in separate paths, it is possible to prevent the influence of reflection on the end surface 110A. Although the imaging image fiber 210 is shown bent here to emphasize the independent optical system, it is not necessary to bend the image fiber 210 and the like unless interference between the optical systems occurs.

[Configuration example using the stereo method]
FIGS. 18A and 18B: In the first modified configuration example, a plurality of (here, two) imaging image fibers 210 are arranged. The two imaging image fibers 210 are bundled in parallel with the projection image fiber 110 so that they can be inserted together into the external auditory canal EE. Then, if a stereo image is acquired using two imaging elements 150 by separately constructing two systems of optical systems through which two imaging image fibers 210 are communicated, the image processing unit 162 or the like obtains the starting point by the stereo method. be able to.

In the previous embodiment, the method of determining the starting point from the image of the fringe pattern FG projected onto the reference plane E1' is used. It is not necessary to attach the sticker S to the concha E1 for photographing. If the stereo method is not used, one imaging fiber 210 may be used. Alternatively, the number of imaging image fibers 210 may be three or more.

[Configuration example in which a rod lens is attached to the tip of the imaging fiber]
FIG. 18C: In a modified configuration, for example, a GRIN (GRadient Index) rod lens 212 may be arranged on the end surface 210A′ of the imaging image fiber 210 on the object side. The photographing image fiber 210 is configured to pass the principal point of the GRIN rod lens 212 and cause the returning light to enter the end face 210A'. In this case, a virtual plane passing through the principal point of the GRIN rod lens 212 corresponds to the virtual imaging element plane 150' (see FIG. 9).

The performance of the GRIN rod lens 212 is adjusted so that the position of the virtual plane passing through the principal point is shifted from the position of the end face 110A' of the projection image fiber 110. FIG. Although the striped pattern light spreads in the projection direction with a point on the end face 110A' as the origin, it is not essential to spread the striped pattern light. By adopting a configuration in which the GRIN rod lens 212 shifts the virtual plane from the end surface 110A', it is possible to easily obtain a sense of perspective when photographing the striped pattern light.

In this respect, the optical system model of the previous embodiment (FIGS. 7 to 9) arranges the pinhole 152 between the imaging device 150 and the lens 132, and the virtual pinhole 152 on the end surface 110A' side of the image fiber 110. ' is placed, but the pinhole 152 is not used in the configuration example in which the GRIN rod lens 212 is placed. The pinhole 152 is necessary when the optical path is shared for projection and imaging. By attaching the GRIN rod lens 212 to the end face 210A′, an image can be formed on the device surface of the imaging device 150. FIG. In addition, since the lens 133 mentioned in the previous embodiment is a configuration necessary for forming an image of the returned light of the striped pattern on the virtual image pickup device surface 150′ together with the lens 132, an optical system for photographing is separately provided. The lens 133 is not necessarily required in the system modified configuration example, and is omitted here.

[Configuration example in which a rod lens is attached to the tip of the image fiber for projection]
FIG. 18(D): In place of the lens 133 used in the embodiment (FIGS. 1, 7, etc.), a GRIN rod lens 133' may be arranged on the end surface 110A' of the projection image fiber 110. . When the GRIN rod lens 133' is arranged, a lens having a principal point position different from that of the GRIN rod lens 212 is used, or the end surfaces A' and 110 of the image fiber 110 for projection and the end surface 210A' of the image fiber 210 for photographing are arranged in the projection direction. It is necessary to shift the positions of the principal points of the GRIN rod lens 133' and the GRIN rod lens 212 in the projection direction.

[Second modified configuration example]
FIG. 19 is a diagram showing a second modified configuration example of the optical system model of the data generation system 100. As shown in FIG. Here, similarly, the explanation will focus on the constituent elements that are different from the previous embodiment due to the modification, and the constituent elements that are the same as those already shown in FIG. do. Further, the following modified configuration example is based on a configuration example in which the image fiber for projection 110 and the image fiber for photographing 210 are separate systems (same hereafter).

In the second modified configuration example, stripe pattern light is generated without using the spatial light modulator 140 . Specifically, a half-wave plate 240 and a quarter-wave plate 244 are arranged in order at the incident side position of the polarizing beam splitter 234 to polarize the laser light into two components. Also, the half-wave plate 240 is configured to be rotationally driven around the optical axis by a drive mechanism 242 .

Quarter wavelength plates 246, 250 and mirrors 248, 252 are arranged at the two branching positions of the polarizing beam splitter 234, respectively. is possible.

A polarizer 235 is arranged on the output side of the polarizing beam splitter 234, and lenses 230 and 231 are arranged facing the end face 110A of the image fiber 110 for projection.

The two polarized light components of the laser light are split into two directions by the polarized beam splitter 234, reflected by the mirrors 248 and 252 respectively, pass through the polarized beam splitter 234, pass through the lenses 230 and 231, and reach the end face of the projection image fiber 110. Superimposed at 110A. At this time, by making the angles of the two mirrors 248 and 252 different, the two polarized light components generate fringe interference and become fringe pattern light. The angles of mirrors 248 and 252 are adjusted by driving mechanisms 254 and 256 so that a suitable fringe pattern of light is obtained. By rotating the half-wave plate 240 on the incident side, the phase difference between the two polarization components is changed. This makes it possible to change the phase of the fringe pattern light (position of light and dark) as in the previous embodiment.

Also, the quarter-wave plates 246 and 250 on the front side of the mirrors 248 and 252 are elements for directing the reflected light toward the optical path of the image fiber 110, thereby reflecting the light from the polarizing beam splitter 234 to the incident side. Light is prevented from returning.

The polarizer 235 is an element for extracting a common direction from two orthogonal polarization components and superimposing them. For example, if the directions of two polarizations are defined as 0° and 90°, the 45° components from each are extracted and superimposed. Also, two lenses 230 and 231 located behind the polarizer 235 are used for condensing light onto the end face 110A, and are the same as the lenses 130 and 131 in the previous embodiment.

The phase difference of the fringe pattern light can be changed by shifting the positions of the two mirrors 248 and 252 without rotating the half-wave plate 240 to change the distance the light travels. However, since there are structural restrictions on the distance over which the mirrors 248 and 252 can be moved in the optical path direction, it is practically more convenient to use the infinitely rotatable half-wave plate 240. be.

[Third modified configuration example]
FIG. 20 is a diagram showing a third modified configuration example of the optical system model of the data generation system 100. As shown in FIG. The third modified configuration example is obtained by further simplifying the optical system model of the second modified configuration example.

In the third modified configuration example, for example, an LED 258 is used as the light source. Since the light from the LED 258 is non-polarized unlike the laser light, it is polarized by the polarizer 262 . In addition, the polarizer 262 is configured to rotate around the optical axis by using a drive mechanism 264, and the light polarized by the polarizer 262 changes the phase difference due to the rotation. The wavelength plate 250 is configured to generate a phase difference between the two polarization components.

Then, the two polarized light components are made incident on the Wollaston prism 260, and the Wollaston prism 260 separates them into two light beams at a minute angle. The two separated lights are polarized by a polarizer 235 and condensed on the end surface 110A of the image fiber 110 using two lenses 230 and 231 . The Wollaston prism 260 is an example of a polarization splitting polarizer, and other polarizers having the same function may be used.

In this way, the technique of recombining the light separated by the Wollaston prism 260 is suitable for the light of the LED 258 . That is, in the case of light other than laser light, the coherence is low and it is usually difficult to obtain interference fringes. Therefore, even light with low coherence can easily interfere with each other. Therefore, it is possible to realize time-varying interference fringes without using the relatively expensive laser light source 120 . Moreover, when laser light is not used, the high coherence (speckle pattern) of laser light also disappears, which is useful in terms of noise reduction.

[Fourth Modified Configuration Example]
FIG. 21 is a diagram showing a fourth modified configuration example of the optical system model of the data generation system 100. As shown in FIG.

A fourth modified configuration example is a configuration using a microdisplay 266 as a technique for generating stripe pattern light. Then, the fringe pattern light generated by the microdisplay 266 is focused on the end surface 110A of the projection image fiber 110 by the lenses 230 and 231, and projected from the end surface 110A' on the object side to project the fringe pattern FG onto the inner wall of the external auditory canal EE. .

Thereby, the optical system of the data generation system 100 can be configured with a very simple configuration. The microdisplay 266 must be specially designed as a device for outputting patterned light that can be imaged on the end face 110A of the image fiber 110 according to the purpose of use.

As described above, according to the data generation system 100 of this embodiment including various modified configuration examples and the data generation method using the same, the following advantages are obtained.
(1) Since non-contact optical measurement of the shape of the ear canal is possible, it is possible to easily measure the shape of the ear canal for custom-made hearing aids.
(2) In addition, by installing an ear canal shape measuring device to which the data generation system 100 is applied at each hearing aid sales site, there is no need to take the ear mold of the hearing aid purchaser and send it to the factory, and the factory can measure the shape of the ear canal. Just send the data. As a result, the number of days required for custom-made hearing aids can be shortened.
(3) Furthermore, since an ear mold is not taken, the impression material is not used on the human body, and accidents related to the impression material (damage to the eardrum, inner ear, etc.) can be eliminated.
(4-1) In the case of one embodiment, illumination (projection) and imaging can be performed with a single image fiber, so the entire apparatus can be made compact.
(4-2) Even in the case of various modified configuration examples, it is possible to configure with a small number of image fibers bundled in parallel, so it is possible to suppress an increase in the size of the entire device.
(5) By freely moving the tip of the image fiber within the external auditory canal, the shape of the portion that is hidden from the surface of the auricle can also be converted into data.
(6) By three-dimensional Fourier analysis of time and two-dimensional space, it is possible to accurately reproduce the three-dimensional shape of even the complicated shape of the inner wall surface of the tube such as the external auditory canal.

The present invention can be modified in various ways without being restricted by the above-described embodiments.
The configuration of the data generation system 100 is not limited to that illustrated in FIG. Also, the form of the hand unit HU is merely an example, and any suitable form may be adopted upon implementation.

The period and phase of the fringe pattern can be appropriately adjusted according to the object to be measured (surface to be measured), the size of the image fiber to be used, the characteristics of the optical system, and the like.
In one embodiment and modified configuration example, the image fiber 110 or the like is inserted into the external auditory canal EE without moving the head to relatively change the imaging position, but the image fiber 110 or the like is inserted while moving the head. Alternatively, the imaging position may be changed by moving the head (external auditory canal EE) with respect to the stationary image fiber 110 or the like and relatively inserting the image fiber 110 or the like into the external auditory canal EE. You may let

100 Data generation system 110, 210 Image fiber 120 Laser light source 130 Lens 134 Polarization beam splitter 140 Spatial light modulator 150 Image sensor 152 Pinhole 162 Image processing unit 164 Analysis/reconstruction unit 170 Magnetic marker 170a Magnetic sensor 212 GRIN rod lens 266 projector

Claims (18)

The phase of an image having a periodic pattern projected onto the surface to be measured is shifted every predetermined time, and the image is captured while the projection and shooting positions of the image on the surface to be measured are relatively stopped and changed. a first step to
Three-dimensional data corresponding to the shape of the surface to be measured is generated based on three-dimensional Fourier analysis in time and two-dimensional space obtained from the plurality of projection and photographing positions in the first step and the plurality of photographed images. and a second step. In the data generation method according to claim 1,
In the first step,
When light modulated with a periodic pattern of an image as a light intensity distribution is projected from the end surface of the image fiber onto the surface to be measured, the surface to be measured passes through a pinhole virtually arranged at a position away from the end surface in the projection direction. If the return light from the Photographing is performed by defining coordinates on a virtual imaging plane,
In the second step,
Data generation characterized by obtaining coordinates representing the position of the surface to be measured based on three-dimensional Fourier analysis in time and two-dimensional space of the image formed on the virtual imaging surface in the first step. Method. In the data generation method according to claim 1,
In the first step,
a first optical system for projecting light modulated by a periodic pattern of an image as a light intensity distribution from the end face of a first image fiber onto a surface to be measured; and projecting the light from the end face of the first image fiber. and a second optical system for photographing by causing the return light from the surface to be measured to enter the end face of the second image fiber,
In the second step,
A data generation method, wherein coordinates representing the position of a surface to be measured are obtained based on time and three-dimensional Fourier analysis in a two-dimensional space of an image captured using the second optical system. In the data generation method according to claim 3,
In the first step,
Photographing is performed by arranging a rod lens on the end face of the second image fiber so that return light from the surface to be measured is incident on the end face through a principal point at a position shifted from the end face in the projection direction. data generation method. In the data generation method according to any one of claims 1 to 4,
In the second step,
1. A data generation method characterized by obtaining projected and photographed positions by continuously projecting and photographing images onto a surface to be measured starting from a photographing position on a reference plane. In the data generation method according to claim 3 or 4,
In the first step,
photographing using a plurality of the second optical systems using a plurality of the second image fibers arranged in parallel;
In the second step,
By continuously projecting and photographing images onto the surface to be measured from a photographing position which is a starting point determined by a stereo method using a plurality of images photographed by a plurality of the second optical systems, the projection and photographing positions are determined. A data generation method characterized by: In the data generation method according to any one of claims 1 to 4,
In the second step,
A data generating method, comprising determining the projection and imaging positions of an image on a surface to be measured based on the magnetic field distribution of a magnetic marker that moves following the movement of the projection and imaging positions. an image generating means for continuously generating an image having a periodic pattern to be projected onto the surface to be measured, with the phase shifted every predetermined time;
an image fiber capable of projecting an image generated by the image generating means toward a surface to be measured;
an imaging means capable of capturing an image projected on the surface to be measured through the image fiber;
Three-dimensional Fourier analysis in time and two-dimensional space obtained from a plurality of captured images captured by the imaging means through the process of relatively stopping and changing the projection and capturing positions of the image of the image fiber on the surface to be measured; A data generating system comprising a generating means for generating three-dimensional data corresponding to the shape of a surface to be measured based on a plurality of projections and photographing positions from which each photographed image was obtained. In the data generation system according to claim 8,
The imaging means is
an imaging device having an imaging surface;
a pinhole arranged on an image plane on the incident side of the imaging surface of the imaging device;
By having a lens that forms an image on the incident side surface from the imaging surface of the imaging element through the pinhole,
When an image is projected onto the surface to be measured from the end surface of the image fiber, the return light from the surface to be measured is incident on the end surface through a pinhole that is virtually arranged at a position away from the end surface in the projection direction. A data generation system capable of generating a virtual situation in which an image is formed on an imaging plane. In the data generation system according to claim 9,
The generating means is
By defining the coordinates on the virtual imaging plane on the extension line connecting the position of the surface to be measured that emits the return light and the position of the virtually arranged pinhole, A data generation system that obtains coordinates representing the position of a surface to be measured based on time and three-dimensional Fourier analysis in a two-dimensional space of a formed image. In the data generation system according to any one of claims 8 to 10,
The generating means is
A data generation system characterized in that, starting from a photographing position of the image fiber with respect to a reference plane, the image projection and photographing positions of the image on the surface to be measured are obtained continuously. In the data generation system according to any one of claims 8 to 10,
a magnetic marker installed on the image fiber and following a change in an image capturing position of the image fiber;
a plurality of magnetic sensors arranged around the surface to be measured and detecting the magnetic field distribution of the magnetic marker at a plurality of different locations;
A data generation system further comprising a detection means for obtaining projection and photographing positions of an image on the surface to be measured of the image fiber based on detection results from the plurality of magnetic sensors. In the data generation system according to claim 8,
The image fiber is
a first image fiber that projects an image generated by the image generating means through a first optical system;
The imaging means is
photographing an image through a second image fiber constituting a second optical system different from the first optical system;
The generating means is
Obtained from a plurality of photographed images photographed by the photographing means through a process in which the projection position of the image of the first image fiber and the photographing position of the image of the second image fiber with respect to the surface to be measured are relatively stopped and changed. It is characterized by generating three-dimensional data according to the shape of the surface to be measured based on three-dimensional Fourier analysis in time and two-dimensional space, and a plurality of projections and shooting positions from which each shot image was obtained. Data generation system. In the data generation system of claim 13,
A rod lens is further provided which is attached to the end face of the second image fiber and causes return light from the surface to be measured to enter the end face through a principal point located at a position shifted from the end face in the projection direction. Data generation system. In the data generation system according to claim 13 or 14,
The imaging means is
including a plurality of the second image fibers arranged in parallel,
The generating means is
By obtaining the starting point of the photographing position by the stereo method using a plurality of images photographed using the plurality of second image fibers, the projection and photographing positions of the images on the surface to be measured are obtained continuously from the starting point. A data generation system characterized by: In the data generation system according to any one of claims 13 to 15,
The image generation means is
While changing the phase difference between the two polarized light components of the laser beam, the two polarized light components are branched into different optical paths and then interfered with each other at the end face of the first image fiber on the incident side. A data generation system characterized by continuously generating images having a periodic pattern with a phase shift at predetermined time intervals. In the data generation system according to any one of claims 13 to 15,
The image generation means is
a polarizer that polarizes unpolarized light into two components;
a modulator that changes the phase difference between the two polarization components polarized by the polarizer;
a polarization separating polarizer that separates the two polarization components;
and an optical device for combining the two polarized light components separated by the polarization separation type polarizer on the second image fiber. In the data generation system according to any one of claims 13 to 15,
The image generation means is
A data generation system comprising a microdisplay that generates an image that is projected onto a surface to be measured by the second image fiber and that is incident on the second image fiber.

Images