JP2020067592A - Movement amount determining device, movement amount determining method, and movement amount determining program - Google Patents

Abstract

To realize the determination of a movement amount of an image acquisition unit by a configuration unaffected by costs and size.SOLUTION: A movement amount determining device for determining the amount of relative movement of an image acquisition unit that is used for acquiring the image of a measurement object, comprises: an overlap determination unit for determining, each time second and subsequent images are acquired using an image acquisition unit after a specific timing, overlapping portions of the acquired image and a previously acquired image; a synthesized image generation unit for superimposing the plurality of images acquired using the image acquisition unit after the specific timing in the overlapping portions determined by the overlap determination unit to generate a synthesized image; and a movement amount determination unit for determining the amount of movement relative to the measurement object of the image acquisition unit on the basis of the synthesized image generated by the synthesized image generation unit.SELECTED DRAWING: Figure 2

Description

  INDUSTRIAL APPLICABILITY The present invention is executed by a movement amount determination device that determines a movement amount of an image acquisition unit used to acquire an image, a movement amount determination method executed by the movement amount determination device, and a computer of the movement amount determination device. It relates to a movement amount determination program.

Conventionally, a device for measuring a distance is known.
For example, in Patent Document 1, in an ultrasonic diagnostic apparatus that inserts an ultrasonic probe into a body to perform ultrasonic diagnosis, ultrasonic waves are transmitted and received using an ultrasonic transducer of the ultrasonic probe, It is disclosed to measure the distance to the object on the front side in the insertion direction.

JP, 2007-82624, A

In recent years, there is a demand for realizing a measurement of a movement amount when an image acquisition unit used for acquiring an image moves by a configuration that is not limited by cost or size.
In view of the above circumstances, the present invention is a movement amount determination device, a movement amount determination method, and a movement amount that can realize the determination of the movement amount of an image acquisition unit by a configuration that is not restricted by cost or size. The purpose is to provide a judgment program.

  A first aspect of the present invention is a movement amount determination device that determines a relative movement amount of an image acquisition unit used to acquire an image of a measurement target, and the image acquisition unit is provided after a specific timing. In the image acquired by using the overlap determination unit that determines the overlap portion between the acquired image and the previously acquired image every time the second and subsequent images are acquired, and the specific timing and thereafter. A plurality of images acquired by using the image acquisition unit, a composite image generation unit that generates a composite image by overlapping at the overlap portion determined by the overlap determination unit, and a composite image generated by the composite image generation unit. And a movement amount determination unit that determines a relative movement amount of the image acquisition unit with respect to the measurement object.

A second aspect of the present invention further includes a display unit for displaying the movement amount determined by the movement amount determination unit in the first aspect.
According to a third aspect of the present invention, in the first or second aspect, the overlap determination unit determines the overlap portion using an inference model.

  According to a fourth aspect of the present invention, in the inference model according to the third aspect, the inference model teaches at least one input / output pair for a relationship between a plurality of images taken with a known shift amount along an object and an overlapping portion thereof. It is obtained by machine learning performed as data.

  In a fifth aspect of the present invention based on the third aspect, the inference model has at least one relationship between two images captured with a known shift amount along an object and a width of an overlapping portion of these two images. It is obtained by machine learning with one input / output pair as teacher data.

  According to a sixth aspect of the present invention, in the third aspect, the inference model includes a plurality of images captured with a known shift amount along an object, and an image acquisition unit when an image of the object is acquired. The relative movement amount is obtained by machine learning using at least one input / output pair as teacher data.

A seventh aspect of the present invention is any one of the third to sixth aspects, further comprising a communication unit that communicates with an external device, and the inference model is acquired from the external device via the communication unit. To be done.
An eighth aspect of the present invention is the image capturing section according to any one of the first to seventh aspects, wherein the image acquisition section is used to acquire an image.

  According to a ninth aspect of the present invention, in the eighth aspect, when the image capturing unit moves in one direction, the composite image generated by the composite image generating unit uses the image capturing unit after the specific timing. The plurality of images acquired by the above are images arranged in a direction orthogonal to the one direction.

  According to a tenth aspect of the present invention, in the ninth aspect, the movement amount determination unit converts the length of the composite image generated by the composite image generation unit into the movement amount of the imaging unit, The amount of movement of the imaging unit is determined.

  In an eleventh aspect of the present invention based on the ninth aspect, the movement amount determination unit determines the movement amount of the imaging unit based on a composite image generated by the composite image generation unit and a reference composite image. However, the reference composite image is acquired every time the second and subsequent images are acquired in the image acquired by using the imaging unit while the imaging unit is moving the reference movement amount in the one direction. A plurality of images acquired by using the image capturing unit while the image capturing unit moves the reference movement amount in the one direction. Is generated by the composite image generation unit overlapping the overlapping portion to generate a composite image.

A twelfth aspect of the present invention is the ultrasonic probe according to any one of the first to seventh aspects, in which the image acquisition unit is used for acquiring an ultrasonic image.
A thirteenth aspect of the present invention is the twelfth aspect, wherein the movement amount of the ultrasonic probe determined by the movement amount determination unit is equal to that of the ultrasonic probe with respect to a subject having a hollow cylindrical shape. It is the rotation angle.

  According to a fourteenth aspect of the present invention, in the thirteenth aspect, the movement amount determination unit includes the length of the composite image generated by the composite image generation unit and the ultrasonic wave for the subject included in the composite image. The rotation angle of the ultrasonic probe with respect to the subject is determined based on the length of the partial composite image for one rotation of the probe.

  A fifteenth aspect of the present invention is a movement amount determination method executed in a movement amount determination device that determines a relative movement amount of an image acquisition unit used to acquire an image of an object to be measured. In the images acquired using the image acquisition unit after the timing of, every time the second and subsequent images are acquired, the overlapping portion between the acquired image and the previously acquired image is determined, and A plurality of images acquired by using the image acquisition unit after a specific timing, a composite image is generated by overlapping the overlapping portion, and based on the composite image, relative to the measurement object of the image acquisition unit. Determine the amount of movement.

  A sixteenth aspect of the present invention provides a computer of a movement amount determination device that determines a relative movement amount of an image acquisition unit used to acquire an image of a measurement target, the image acquisition unit being after a specific timing. In the image acquired using, every time the second and subsequent images are acquired, the overlapping portion between the acquired image and the previously acquired image is determined, and the image acquisition is performed after the specific timing. A plurality of images acquired by using the image acquisition unit are overlapped at the overlapping portion to generate a combined image, and a relative movement amount of the image acquisition unit with respect to the measurement object is determined based on the combined image. Is a movement amount determination program for executing.

  According to the present invention, it is possible to determine the amount of movement of the image acquisition unit used to acquire an image with a configuration that is not limited by cost or size.

It is a figure which shows the external appearance structural example of the industrial endoscope apparatus which is the movement amount determination apparatus which concerns on 1st Embodiment. It is a figure which shows the structural example of an endoscope apparatus and the external device which can communicate with an endoscope apparatus. It is a figure which shows typically the process in which a machine learning part produces | generates an input / output model. It is a flow chart which shows an example of image pick-up access control processing performed in an endoscope apparatus. FIG. 8 is a diagram showing an example of a composite image generated when the insertion section is inserted into the subject in one direction during the access mode period in the imaging access control process. It is a flowchart which shows an example of the process performed in an external device. It is a flowchart which shows the other example of the imaging access control process performed in an endoscope apparatus. FIG. 9 is a diagram (No. 1) showing an example of a screen displayed on the display unit when the calibration mode is set in the imaging access control process shown in FIG. 7. FIG. 9 is a diagram (No. 2) showing an example of a screen displayed on the display unit when the calibration mode is set in the imaging access control process shown in FIG. 7. It is a figure which shows the structural example of the ultrasonic inspection apparatus which is the movement amount determination apparatus which concerns on 2nd Embodiment. It is a figure which shows typically an example of the ultrasonic wave (transmitted wave) transmitted to the test object and the ultrasonic wave (reflected wave) reflected by the test object. It is a figure which shows an example of a transition of the reflected wave detection time when a test object is rotated. It is a flow chart which shows an example of flaw detection access control processing performed in an ultrasonic inspection device. It is a figure which shows an example of a hardware configuration.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
<First Embodiment>
FIG. 1 is a diagram showing an external configuration example of an industrial endoscope apparatus (hereinafter simply referred to as “endoscope apparatus”) which is a movement amount determination apparatus according to the first embodiment.

As shown in FIG. 1, the endoscope apparatus 1 includes an endoscope insertion portion (hereinafter simply referred to as “insertion portion”) 101, a display portion 102, and a bending operation portion 103.
The insertion portion 101 is a portion to be inserted into a structure as a subject, and has an imaging portion 104 described below at its tip. The insertion section 101 is also an insertion section of a so-called flexible endoscope.

The display unit 102 displays images and information. For example, an image acquired by using the image capturing unit 104 or an insertion distance of the inserting unit 101 (which is also a moving amount of the image capturing unit 104) described below is displayed.
The bending operation unit 103 receives an operation for bending the distal end portion of the insertion unit 101. As a result, the user can freely bend the distal end portion of the insertion portion 101 by operating the bending operation portion 103, and can change the subject region imaged by the imaging portion 104.

FIG. 2 is a diagram showing a configuration example of the endoscope apparatus 1 and an external device capable of communicating with the endoscope apparatus 1.
As shown in FIG. 2, the endoscope device 1 includes an imaging unit 104, a clock unit 105, a control unit 106, an inference engine 107, a communication unit 108, a display unit 102, an operation unit 109, and a recording unit 110. The imaging unit 104 is an example of an image acquisition unit used to acquire an image.

The imaging unit 104 is provided at the distal end portion of the insertion unit 101 as described above, and includes the optical system 104a, the imaging element 104b, and the bending portion 104c.
The optical system 104a forms an optical image of a subject (subject image) on the image sensor 104b.

  The image sensor 104b converts the subject image formed by the optical system 104a into an electric signal. An image is generated (acquired) in the imaging unit 104 based on this electric signal. The image sensor 104b is, for example, a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor).

The bending portion 104c is a mechanism that bends the distal end portion of the insertion portion 101 according to an operation received by the bending operation portion 103.
The clock unit 105 generates time data or date / time data.

  The control unit 106 controls the entire endoscope device 1. The control unit 106 includes an overlap determination unit 106a, a combination processing unit 106b, a drive unit 106c, a communication control unit 106d, a recording control unit 106e, a movement amount determination unit 106f, and a display control unit 106g.

  The overlap determination unit 106a acquires the image acquired last time (one before) with the acquired image every time the second and subsequent images are acquired in the image acquired using the imaging unit 104 after a specific timing. The overlapping portion with the image is judged. In this embodiment, this determination is made in cooperation with the inference engine 107.

  The combining processing unit 106b overlaps a plurality of images acquired by the imaging unit 104 after a specific timing with the overlapping portion determined by the overlapping determination unit 106a to generate a combined image. For example, the composite image generated when the image capturing unit 104 is inserted into the subject in one direction after a specific timing includes a plurality of images acquired using the image capturing unit 104 after the specific timing. , Images are arranged in a direction orthogonal to the one direction.

The driving unit 106c drives the imaging unit 104.
The communication control unit 106d controls communication with the external device 2 and the like performed via the communication unit 108.

The recording control unit 106e controls recording of image files and the like in the recording unit 110.
The movement amount determination unit 106f determines the movement amount of the imaging unit 104 based on the composite image generated by the composition processing unit 106b. For example, the movement amount determination unit 106f may determine the movement amount of the imaging unit 104 by converting the length of the combined image generated by the combination processing unit 106b into the movement amount of the imaging unit 104. Alternatively, for example, the movement amount determination unit 106f may determine the movement amount of the imaging unit 104 based on the combined image generated by the combination processing unit 106b and the reference combined image. In this case, the reference composite image is acquired, for example, every time the second and subsequent images are acquired in the image acquired by using the imaging unit 104 while the imaging unit 104 moves in the reference movement amount in one direction. The overlapping determination unit 106a determines the overlapping portion between the captured image and the previously captured image, and a plurality of images captured by the imaging unit 104 while the imaging unit 104 moves in the reference movement amount in one direction are displayed. It is created by the synthesizing processing unit 106b overlapping to generate a synthetic image at the overlapping portion.

The display control unit 106g controls the display of images and information on the display unit 102.
The inference engine 107 uses the inference model to infer the overlapping portion of the two images with respect to the input of the two images and output the overlapped portion. In the present embodiment, in the images acquired by using the image capturing unit 104 after a specific timing, every time the second and subsequent images are acquired, two images, the acquired image and the previously acquired image, are acquired. With respect to the input of, the overlapping portion of the two images is inferred and output to the overlapping determination unit 106a. Here, the inference of the overlapping portion may be performed based on, for example, the degree of similarity of the graininess of the two images.

  The inference engine 107 also includes a storage unit 107b that stores the inference model 107a. The storage unit 107b may store a plurality of inference models 107a. In this case, the plurality of inference models 107a may be, for example, an inference model for each structure to be inspected or an inference model for each material inside the structure to be inspected. Further, in this case, the user may select the inference model 107a to be used according to the type of the structure to be inspected and the material inside the structure to be inspected. This selection can be performed by operating the operation unit 109.

  The communication unit 108 is a communication interface that communicates with the external device 2 and the like. The communication unit 108 receives, for example, the inference model stored in the storage unit 107b from the external device 2. In recent years, attempts have been made to perform such learning in a terminal such as the endoscope device 1, and the part for performing machine learning does not necessarily have to be external. It goes without saying that the inference engine 107 can be designed externally.

  Further, a built-in learning device and an external learning device may cooperate with each other. An external learning device can operate at high speed without worrying about power consumption, so large-scale machine learning is performed with abundant computing resources here, and a small-scale built-in learning device is based on the information obtained there. It is also possible to use for efficient learning. For example, by adjusting the layers of the neural network or pruning information having a small contribution, it is possible to create a practical and highly accurate inference model even on a small scale.

  The display unit 102 displays images and information. The display unit 102 is, for example, an LCD (Liquid Crystal Display). Alternatively, the display unit 102 may be a display unit with a touch panel.

  The operation unit 109 includes the bending operation unit 103 described above, and receives various input operations on the endoscope apparatus 1. When the display unit 102 is a display unit with a touch panel, the touch panel is a part of the operation unit 109.

  The recording unit 110 is a non-volatile memory such as an HDD (hard disk drive), and records, for example, an image file of a captured image (still image or moving image) acquired by using the imaging unit 104. The image file has image data representing a captured image and auxiliary data regarding the captured image. The auxiliary data includes date and time data when the captured image was captured.

  In the endoscope device 1, the clock unit 105, the control unit 106, the inference engine 107, and the like are configured by circuits such as ASIC (Application Specific Integrated Circuit) and FPGA (Field-Programmable Gate Array). May be.

The external device 2 includes an external image DB (database) 201 and a machine learning unit 202. The external image DB 201 and the machine learning unit 202 may be configured as separate devices.
The external image DB 201 includes an image recording unit 201a and a communication unit 201b.

  The image recording unit 201a is a non-volatile memory such as an HDD, and records a plurality of continuously acquired image groups. The continuously acquired image group is a plurality of images that are continuously acquired, and is, for example, a plurality of continuously shot images or a moving image having a plurality of frame images. Further, the continuously acquired image group has accompanying information. The incidental information includes information about the subject such as the type and material of the subject represented in the continuously acquired image group.

  The communication unit 201b is a communication interface that communicates with the machine learning unit 202 and the like. The communication unit 201b transmits, for example, the continuously acquired image group recorded in the image recording unit 201a to the machine learning unit 202.

The machine learning unit 202 includes a population generation unit 202a, an output setting unit 202b, an input / output modeling unit 202c, and a communication unit 202d.
The population creating unit 202a creates a plurality of input / output pairs as teacher data. In this embodiment, a plurality of input / output pairs are created from the continuously acquired image group acquired from the external image DB 201. Here, the input / output pair is an input / output pair in which two images are input and an overlapping portion of the two images is output. The overlapping portion to be output is obtained, for example, by searching for a similar portion of image graininess while overlapping one of two images to be input with respect to the other and slowly moving them.

  The output setting unit 202b sets the output of the input / output model to be generated by the input / output modeling unit 202c. In the present embodiment, the overlapping portion of two input images (two image pair common portion) is set as its output.

  The input / output modeling unit 202c generates an input / output model based on the plurality of input / output pairs created by the population creating unit 202a and the output set by the output setting unit 202b. According to this, in the present embodiment, an input / output model is generated in which, for two input images, the overlapping portion of the two images is output. The input / output model generated here is also an inference model.

  The communication unit 202d is a communication interface that communicates with the external image DB 201, the endoscope device 1, and the like. The communication unit 202d receives, for example, the continuously acquired image group from the external image DB 201, and also transmits the generated input / output model (inference model) to the endoscope device 1.

  In addition, in the external device 2, the population creating unit 202a, the output setting unit 202b, the input / output modeling unit 202c, and the like may be configured by circuits such as ASIC and FPGA, for example.

FIG. 3 is a diagram schematically showing a process in which the machine learning unit 202 generates an input / output model.
As shown in FIG. 3, the machine learning unit 202 generates an input / output model based on the plurality of input / output pairs created by the population creating unit 202a and the output set by the output setting unit 202b. Specifically, the machine learning unit 202 machine-learns the relationship between inputs and outputs from a plurality of input / output pairs, and further outputs the overlapping portion of the two images with respect to the inputs of the two images. I do. Then, as a result of the machine learning, a machine-learned input / output model is obtained. The input / output model obtained at this time can be incorporated into a device (for example, the endoscope apparatus 1) as a circuit or a program. In recent years, it is known that machine learning is used in combination with SLAM (Simultaneous Localization and Mapping).

  When the non-specimen is manufactured, a surface state peculiar to the non-specimen may appear depending on the material, design, process and environment at the time of manufacturing. Such characteristics of an object (structure) to be an object are often a monotonous pattern even with human eyes, and the amount of repeated information is too large to be discriminated. Of course, other organisms other than those produced have similar individual differences under the conditions of growth, and the same can be said. Even with such a minute difference, it is possible to detect the difference according to the measurement location by pattern analysis, which is a strength of the image recognition technology by machine learning that handles a large amount of data in recent years. As described above, if there are a plurality of images obtained by previously capturing the same image of the same object from a known shifted position, the two images are used as the teaching data as the input / output pair with the relationship between the known shifted amount and the overlapping portion of the images. Machine learning such as outputting the overlapping portion of the two images with respect to the input of is possible. That is, the inference model obtained here is obtained by machine learning in which the relationship between a plurality of images taken with a known shift amount along the object and the overlapping portion thereof is used as teacher data as an input / output pair. In addition, it is possible to easily infer the movement distance based on the grain size of the surface pattern peculiar to the material. Of course, the moving distance may be determined by considering that the distance to the object is substantially constant and converting it from the angle of view to convert it into the information of the width of the object or the information of the overlap. If the overlapping portion can be determined, the determination of the movement distance can be inferred from that. The movement distance can be converted depending on how many patterns having specific graininess can be detected in the obtained image. In addition, the above-mentioned relationship between the known shift amount and the overlapping portion of the images is used as teacher data as an input / output pair, and the distance (width) of the overlapping portion is input instead of the overlapping portion of the two images for the input of the two images. Machine learning such as outputting, and machine learning for inferring the amount of progress by it are also possible. There is a learning method in which such data are simultaneously output, and it is also possible to obtain an inference engine that infers the overlapping position from a plurality of position-shifted images and infers the overlapping width and the movement amount of the device. In other words, this inference model consists of a plurality of images taken along a target object with a known shift amount and at least one relative movement amount of the image acquisition unit when the image of the target object is input / output. It can be expressed as what is obtained by machine learning performed as the teacher data.

  The object to be the teacher data was obtained by a process in which a structure similar to the non-specimen or an organ of a living body was prepared in advance, and the object was imaged and imaged at imaging positions shifted along the surface. Teacher data may be prepared so that the overlapping part of a plurality of captured images is correct. Further, at the measurement site, the non-specimen may be imaged a plurality of times with a prescribed shift amount, and the overlapping image portion may be searched for by pattern matching or the like. Since the amount of shift is a prescribed amount, it is possible to search for a characteristic portion in a monotonous pattern and determine the overlapping portion with high accuracy. Until now, as in panorama composition, there is a method that determines the overlap between the images obtained along the changing direction of the image pickup position on the changing direction side of the previous image and on the opposite side of the changing direction of the subsequent image. However, this is because, for example, the process of determining the matching portion is simple because it suffices to search for similar portions on one side of the rectangular image, but as in this embodiment, the image capturing direction and the traveling direction are the same. In determining the common part (overlap) of the peripheral portion of the image, the distance between the target parts changes rapidly, so that the common part of the image does not simply overlap, and it is difficult to determine the traveling distance. By inferring from the result of machine learning, it is possible to easily determine overlap and traveling distance. Furthermore, as the distance to the target object approaches in a biased manner, there are too many variable factors to make a judgment using simple logic, so machine learning was used as an effective means even in such a situation. The configuration of the present invention is advantageous.

The "deep learning" is a multi-layered structure of the "machine learning" process using a neural network.
A "forward-propagation neural network" that sends information from the front to the back to make a judgment is typical.

  In the simplest case, this consists of an input layer consisting of N1 neurons, an intermediate layer consisting of N2 neurons given by parameters, and N3 neurons corresponding to the number of classes to be discriminated. It is sufficient if there are three output layers.

Then, the neurons of the input layer and the intermediate layer, and the neurons of the intermediate layer and the output layer are connected by connection weighting, respectively, and a bias value is applied to the intermediate layer and the output layer, which facilitates formation of the logic gate.
For simple determination, three layers may be used, but if the number of intermediate layers is increased, it becomes possible to learn how to combine a plurality of feature amounts in the machine learning process.

In recent years, 9 to 152 layers have become practical in view of the time required for learning, determination accuracy, and energy consumption.
In addition, it involves a process called "convolution" that compresses the feature amount of the image, moves with a minimum amount of processing, is a "convolutional neural network" that is strong in pattern recognition, and can handle more complicated information, meaning according to order and order. A "recursive neural network" (fully connected recurrent neural network) that flows information in both directions may be used in response to the information analysis that changes.

  In addition, as a pattern recognition model using supervised learning, there are methods such as support vector machine and support vector regression. The learning here is to calculate the weight of the discriminator, the filter coefficient, and the offset, and there is another method using logistic regression processing.

  In order to realize these technologies, it is possible to use conventional general-purpose arithmetic processing circuits such as CPUs and FPGAs, but since most neural network processing is matrix multiplication, matrix calculation A specialized GPU (Graphic Processing Unit) or Tensor Processing Unit (TPU) may be used. In recent years, such artificial intelligence (AI) dedicated hardware "neural network processing unit (NPU)" is designed so that it can be integrated and integrated with other circuits such as a CPU, and may be a part of a processing circuit. is there.

Here, reference will be made to a device for making the same condition as the teacher data. For example, here is an example of “inspection of the same material from the same direction”.
Since the results of inference may be improved by aligning the conditions, the teacher data during learning may be devised by aligning these conditions. By making inferences, correct measurements can be made. It is effective to devise to input or judge the information of the material or the measured object or the information of the measurement environment before the inference using the inference model, and to correct the information of this condition and the environmental factors. It may have a sensor. Since the specification and performance of the inference engine change depending on whether or not such a constraint is added during learning, such trial and error may be performed or the trial and error may be displayed while advancing the annotation work.

Here, a small inference engine for a camera will be further described.
The compact type inference engine installed in information terminal products such as cameras and mobile devices is difficult to learn for high-precision judgment with a small number of layers, and it takes time, so devise an accurate annotation and learning method. Is desired. When the inference model is generated, the specification of the inference model changes depending on the image used for learning, so that efficient learning may be performed in cooperation with the information at the time of learning. Therefore, information indicating what kind of learning has been performed may be set during annotation work, and this information may be recorded in the recording unit of the information acquisition device as part of the inference information.

  Further, as already described, it is possible to perform more efficient learning by coordinating external and internal learning units or by coordinating a plurality of learning units. There is also a method that enables you to pick up important information and convert it into a smaller inference model by using the information that created inference models with a large amount of teacher data using a large-scale learning resource with abundant computational resources. The present application is supposed, and at this time, a small inference model is created using a small learning device. What is expressed as inference using teacher data may be performed in any number of stages as long as one of these is performed.

Here, reference is also made to no teaching data such as “unsupervised learning” and “reinforcement learning”, and others.
As explained so far, "supervised learning" is to learn "relationship between input and output" using teacher data whose output is defined by annotation, and highly reliable inference under specific conditions. On the other hand, an inference model capable of coping with more complicated situations may be acquired by using the method of “unsupervised learning” for learning the “data structure”.

  Further, a method of learning “behavior that maximizes value and effect” called “reinforcement learning” may be used. This is to learn to find a law that increases the state behavior value, and estimates the value of the next state instead of the current one and increases the value of trial and error until a specific reward is obtained. Reflect on learning. Teacher data may be used to verify the learning result. The output of the correct answer obtained by the annotation is not learned as it is, but it is learned so that a better answer can be obtained, and it is adapted to cope with an unknown situation.

  These may be used in combination with supervised learning, and inference by supervised learning may be performed after inference by unsupervised learning. The annotation data can also be used as verification data for such “unsupervised learning” and “reinforcement learning”.

  In order to make a machine judge something, it is necessary for a human to teach the machine how to make a judgment.Here, we adopted a method of deriving the judgment of the image by machine learning, but in addition, humans use empirical rules and heuristics. A rule-based method that adapts the acquired rules may be used.

FIG. 4 is a flowchart showing an example of the imaging access control processing executed in the endoscope apparatus 1.
As shown in FIG. 4, in the imaging access control processing, the control unit 106 first determines whether or not the access mode is set (S401). The access mode is a mode in which the user can be presented with the insertion distance of the insertion unit 101 (which is also the movement amount of the imaging unit 104) when the insertion unit 101 is inserted into the subject in one direction. The access mode can be set by the user operating the operation unit 109.

  If the determination result in S401 is YES, the control unit 106 performs control such as bending the distal end portion of the insertion unit 101 according to the operation of the bending operation unit 103 (S402). It should be noted that this S402 is not executed thereafter until the determination result of S401 becomes NO.

  Next, the control unit 106 uses the images acquired in time series using the imaging unit 104 from that time point until the determination result of S401 becomes NO (hereinafter, referred to as “access mode period”). Every time the second and subsequent images are acquired, S403 to S414 are performed. The access mode period may start at the timing when the user performs a predetermined operation on the operation unit 109 after S402.

  Specifically, when the second and subsequent images are acquired, the control unit 106 compares the acquired image with the previously acquired image (S403) and determines the overlapping portion (S404), It is determined whether or not there is the overlapping portion (S405).

When the determination result in S405 is NO, the control unit 106 issues a warning by displaying a warning message on the display unit 102 (S406).
On the other hand, if the determination result in S405 is YES, the control unit 106 overlaps the acquired image and the previously acquired image at the determined overlapping portion to generate a composite image (S407). However, in S407 performed after the second time in the access mode period, the acquired image and the composite image generated in the previous S407 are overlapped at the determined overlapping portion to generate a new composite image.

After S406 or S407, the control unit 106 determines whether or not the insertion distance of the insertion unit 101 after the start of the access mode period can be determined based on the generated composite image (S408).
If the determination result in S408 is NO, the control unit 106 issues a warning by displaying a warning message on the display unit 102 (S409).

  On the other hand, if the determination result in S408 is YES, the control unit 106 converts the length of the generated combined image into the insertion distance of the insertion unit 101 after the access mode period starts, and thus the insertion unit The insertion distance of 101 is determined, and the insertion distance is displayed on the display unit 102 (S410). The length of the composite image is, for example, the length in the arrangement direction of the plurality of images used to generate the composite image. Further, the conversion from the length of the composite image to the insertion distance may be performed, for example, based on the result of machine learning of the relationship between the length of the composite image and the insertion distance of the insertion unit 101.

  After S409 or S410, the control unit 106 determines whether or not there is an instruction to start shooting a moving image or a still image (S411). However, if the moving image is being captured at this determination timing, the determination result of S411 is YES. The user can give an instruction to start shooting by operating the operation unit 109.

If the decision result in S411 is NO, the process returns to S401.
On the other hand, when the determination result in S411 is YES, the control unit 106 records the image data representing the captured image acquired by using the imaging unit 104 in the recording unit 110 (S412).

  After S412, the control unit 106 determines whether the shooting process of the moving image or the still image is finished (S413). In the case of a moving image, the shooting process ends after the shooting end instruction is given. The user can give an instruction to end shooting by operating the operation unit 109.

If the decision result in S413 is NO, the process returns to S401.
On the other hand, if the determination result in S413 is YES, the control unit 106 creates a file of the image data representing the captured image recorded in the recording unit 110 in S412 and the auxiliary data regarding the captured image (S414). As a result, the image file including the image data and the auxiliary data is recorded in the recording unit 110.

After S414, the process returns to S401.
On the other hand, when the determination result in S401 is NO, the control unit 106 determines whether or not there is an inference model acquisition instruction (S415). The instruction to acquire the inference model can be issued by the user operating the operation unit 109.

If the decision result in S415 is NO, the process returns to S401.
On the other hand, if the determination result in S415 is YES, the control unit 106 sets the feature of the target object (structure) to be the subject, and requests the external device 2 to acquire the inference model including the feature. (S416). Here, the characteristics of the target object include the material and the like, and the setting of the characteristics can be performed by the user operating the operation unit 109.

After S416, the control unit 106 acquires the inference model corresponding to the acquisition request from the external device 2 and stores it in the storage unit 107b (S417).
After S417, the process returns to S401.

  FIG. 5 is a diagram showing an example of a composite image generated when the insertion unit 101 is inserted into the subject in one direction during the access mode period in the imaging access control process. In this example, the subject is an iron pipe.

  FIG. 5A shows the position of the insertion unit 101 inserted into the subject at the start timing of the access mode period and the image 121 acquired by using the imaging unit 104 at that time. At this time, since it is the start timing of the access mode period, the insertion distance D of the insertion unit 101 is set to zero.

  FIG. 5B shows the position of the insertion unit 101 when it is further inserted into the subject and the composite image 122 generated at that time. The insertion distance D at this time is determined to be D1, for example, by converting the length of the generated combined image 122 in the width direction into the insertion distance. Here, the width direction of the composite image 122 is also the arrangement direction of the plurality of images used to generate the composite image 122. Here, there is a unique structural non-uniformity that varies depending on the material, design, manufacturing method, and manufacturing environment, and it is detected on the surface of the object that appears as a pattern. Then, the inference is performed based on the concept described in FIG.

  FIG. 5C shows an example of the position of the insertion unit 101 when it is further inserted into the subject and the composite image 123 generated at that time. The insertion distance D at this time is determined to be 2 × D1 by converting the widthwise length of the generated combined image 123 into the insertion distance, for example. Here, the width direction of the composite image 123 is also the arrangement direction of the plurality of images used to generate the composite image 123.

  In addition, in FIG. 5, the composite images 122 and 123 can also be referred to as panoramic images. If the width of each image is known from the image such as the angle of view or the overlapping part can be inferred from such an image, the movement distance can be calculated by (image width + image width−overlap width). Even if the learning is not performed, it is possible to infer the moving distance directly from the change of the image depending on the learning method. In addition, it is also possible to repeat the movement by a known movement distance here and to learn as teacher data on the spot. When the known insertion amount (moving distance) and the inference result are different from each other, there may be a case where a condition change that the operator does not assume occurs, and therefore learning may be redone. It is also possible to determine from the image itself whether it is suitable for correct inference. It goes without saying that it is possible to devise whether or not the detected surface pattern, the size and shape of the tube, etc. are different from those used for learning, and give a warning first.

FIG. 6 is a flowchart showing an example of processing executed in the external device 2.
As shown in FIG. 6, in this process, the external device 2 first determines whether or not there is a learning request (S601). The learning request is also a request to acquire the inference model that the endoscope apparatus 1 makes to the external device 2 in S416 of FIG.

If the determination result in S601 is NO, this determination is repeated.
On the other hand, if the determination result in S601 is YES, the external device 2 sets the object type (S602). This setting is performed, for example, based on the characteristics of the target object included in the acquisition request for the inference model.

  After S602, the external device 2 sets the learning mother body based on the setting of the object type (S603). In setting the learning mother, for example, designation of a structure as an object is performed.

  After S603, the external device 2 searches for the overlapping portion of the images from the continuously acquired image group acquired from the image recording unit 201a based on the setting of the learning mother (S604), and a plurality of input / output pairs that are teacher data. Is created (S605). Here, the input / output pair is an input / output pair in which two images are input and an overlapping portion of the two images is output.

  After S605, the external device 2 generates an inference model that outputs the overlapping portion of the two images in response to the input of the two images from the created teacher data, and the reliability of the inference model is calculated. Is determined to be greater than or equal to a predetermined threshold value (S606).

If the determination result in S606 is NO, the external device 2 resets the learning mother body and the like (S607).
After S607, the external device 2 determines whether the learning mother has been reset a predetermined number of times or more between the determination result of S601 being YES and the determination result of S606 being YES. (S608).

If the decision result in S608 is NO, the process returns to S604.
On the other hand, when the determination result of S606 or S608 is YES, the external device 2 transmits the last generated inference model to the endoscope apparatus 1 (S609), and the process returns to S601.

FIG. 7 is a flowchart showing another example of the imaging access control processing executed in the endoscope apparatus 1.
As shown in FIG. 7, in the imaging access control process of another example, the control unit 106 first determines whether or not the calibration mode is set (S701). In the calibration mode, the endoscope apparatus 1 can be caused to create a reference composite image used for conversion of the insertion distance of the insertion unit 101 by the user inserting the insertion unit 101 into the subject at the reference distance. It is a mode that can. The calibration mode can be set by the user operating the operation unit 109.

  When the determination result of S701 is YES, the control unit 106 determines whether or not the reference distance insertion start is notified (S702). The notification of the start of the reference distance insertion can be performed by the user operating the operation unit 109.

If the decision result in S702 is NO, the process returns to S701.
On the other hand, if the decision result in S702 is YES, the control section 106 starts the continuous captured image synthesizing process (S703). When the continuous captured image synthesizing process starts, the control unit 106 acquires the second and subsequent images in the images acquired in time series using the image capturing unit 104 from the start to the end of the continuous captured image synthesizing process. The overlapping determination unit 106a determines the overlapping portion between the acquired image and the image acquired last time, and combines the plurality of images acquired after the start of the continuous captured image combining process with the combining processing unit 106b. Is repeated at the overlapping portion to generate a composite image (NO in S704 and S705). The continuous captured image synthesizing process ends when the determination result in S705 is YES. In S705, it is determined whether or not the reference distance insertion end is notified. The notification of the end of the reference distance insertion can be performed by the user operating the operation unit 109.

  It should be noted that, when the user notifies the start of the insertion of the reference distance, the user starts inserting the reference distance of the insertion unit 101 into the inside of the subject, and then inserts the reference distance into the insertion unit 101, and then the insertion of the reference distance is completed. Suppose you want to make a notification.

  Then, if the determination result in S705 is YES, the control unit 106 determines the combined image generated by the continuous captured image combining process as the reference combined image (S706), and the process returns to S701.

On the other hand, when the determination result in S701 is NO, the control unit 106 basically performs the same processes as S402 to S409 illustrated in FIG.
Then, if the determination result in S408 is YES, the control unit 106 compares the generated combined image and the reference combined image (S707), and determines the insertion distance of the insertion unit 101 from the length of each image and the reference distance. Based on the conversion, the insertion distance of the insertion unit 101 is determined, and the insertion distance is displayed on the display unit 102 (S708).

After S708 or S409, the control unit 106 determines whether or not there is a shooting start instruction for a still image (S709). If the determination result is NO, the process returns to S701.
On the other hand, if the determination result in S709 is YES, the control unit 106 converts the image data representing the captured image acquired by using the image capturing unit 104 and the auxiliary data related to the captured image into a file, and the recording unit 110 as an image file. Is recorded (step S710).

After S710, the process returns to S701.
8A and 8B are diagrams showing examples of screens displayed on the display unit 102 in the imaging access control process shown in FIG. 7 when the calibration mode is set.
The screen displayed on the display unit 102 shown in FIG. 8A is the screen displayed when S701 in FIG. 7 becomes YES. When this screen is displayed, the user can give notification of the start of reference distance insertion, which is the determination criterion of S702, by selecting "start" in the screen. For example, the user selects “start” when the insertion section 101 is ready for insertion into the subject.

  The screen displayed on the display unit 102 shown in FIG. 8B is the screen displayed when S702 of FIG. 7 becomes YES. When this screen is displayed, the user can notify the end of the reference distance insertion, which is the determination criterion in S705, by selecting "End" in the screen. For example, when the user inserts the insertion unit 101 into the subject at the reference distance, the user selects “end”.

  The image in the screen displayed on the display unit 102 shown in each of FIGS. 8A and 8B is the latest image acquired using the image capturing unit 104. Moreover, "10 cm" in the screen displayed on the display unit 102 shown in FIG. 8B indicates the reference distance.

As described above, according to the endoscopic device 1 according to the first embodiment, the determination of the insertion distance of the insertion section 101 when the insertion section 101 is inserted into the subject is limited to the cost and size. It can be realized by a configuration that does not receive the signal.
<Second Embodiment>

FIG. 9 is a diagram showing a configuration example of an ultrasonic inspection apparatus which is a movement amount determination apparatus according to the second embodiment.
As shown in FIG. 9, the ultrasonic inspection apparatus 3 includes an ultrasonic probe 301, a transmission / reception circuit 302, a transmission / reception control circuit 303, an image conversion circuit 304, a display control circuit 305, a display unit 306, a control circuit 307, and a recording circuit. 308, an operation unit 309, and a communication unit 310 are provided. The ultrasonic probe 301 is an example of an image acquisition unit used to acquire an image.

  The ultrasonic probe 301 is a piezoelectric vibration that transmits ultrasonic waves to the subject S, which is a hollow cylindrical structure such as an iron pipe, and receives the ultrasonic waves reflected by the subject S as echo signals. It has a configuration in which the children are arranged. A contact medium 311 such as glycerin is provided between the subject S and the ultrasonic probe 301 in order to efficiently transmit ultrasonic waves. For example, the contact medium 311 is applied to the outer peripheral surface of the subject S. Further, the inspection region of the subject S is changed by, for example, rotating the subject S.

  The transmission / reception circuit 302 supplies an electric signal to the ultrasonic probe 301 to generate an ultrasonic wave, receives an echo signal received by each piezoelectric transducer of the ultrasonic probe 301, and superimposes it on the harmonics. It has the function of separating wave components by using a filter.

The transmission / reception control circuit 303 controls transmission / reception of the transmission / reception circuit 302.
The image conversion circuit 304 converts the echo signal received by the transmission / reception circuit 302 into an ultrasonic image of the subject S.

The display control circuit 305 performs control for displaying the ultrasonic image and the like converted by the image conversion circuit 304 on the display unit 306.
The display unit 306 is, for example, an LCD, and displays an ultrasonic image and information. For example, an ultrasonic image acquired by using the ultrasonic probe 301 and a rotation angle of the ultrasonic probe 301 described later are displayed. The display unit 306 may be a display unit with a touch panel.

The control circuit 307 controls the entire ultrasonic inspection apparatus 3.
The control circuit 307 has the same functions as those of the overlap determination unit 106a, the combination processing unit 106b, the movement amount determination unit 106f, and the inference engine 107, which are included in the endoscope device 1 according to the first embodiment. Prepare

The recording circuit 308 includes a non-volatile memory and records the ultrasonic image and the like converted by the image conversion circuit 304.
The operation unit 309 receives various input operations to the ultrasonic inspection apparatus 3. When the display unit 306 is a display unit with a touch panel, the touch panel becomes a part of the operation unit 309.

The communication unit 310 is a communication interface that communicates with the external device 2 and the like.
FIG. 10 is a diagram schematically showing an example of ultrasonic waves (transmission waves) transmitted to the subject S and ultrasonic waves (reflected waves) reflected by the subject S. Here, it is assumed that the subject S is a rectangular parallelepiped structure.

  As shown on the left side of FIG. 10, when there is no scratch inside the subject S, the reflected wave from the bottom surface of the subject S returns after the transmitted wave. On the other hand, as shown on the right side of FIG. 10, when there is a scratch Sa inside the subject S, after the transmitted wave, before the reflected wave from the bottom surface of the subject S returns, The reflected wave of is generated. In the ultrasonic inspection apparatus 3, for example, the defect inside the subject S can be found by detecting the reflected wave from the flaw Sa.

FIG. 11 is a diagram showing an example of the transition of the reflected wave detection time when the subject S is rotated.
As shown on the left side of FIG. 11, here, the subject S is assumed to be a hollow cylindrical structure such as an iron pipe. However, in the subject S, it is supposed that the thickness between the inner peripheral surface and the outer peripheral surface should be uniform, but it is not uniform. The inner peripheral surface of the subject S is supposed to be originally smooth, but is not smooth due to corrosion or the like.

  In the reflected wave detection time when the subject S is rotated as shown on the right side of FIG. 11, the reflected wave detection time is the ultrasonic wave after the ultrasonic wave is transmitted from the ultrasonic probe 301 to the subject S. It is the time until the sound wave is reflected by the inner peripheral surface of the subject S and returns to the ultrasonic probe 301.

  As shown in FIG. 11, in the subject S, since the thickness between the inner peripheral surface and the outer peripheral surface is not uniform, the reflected wave detection time is reduced at the rotational position where the distance between the inner peripheral surface and the outer peripheral surface becomes short. Becomes shorter, and the reflected wave detection time becomes longer at the rotational position where the distance becomes longer. Moreover, since the inner peripheral surface of the subject S is not smooth, the transition of the reflected wave detection time is not smooth. On the other hand, the pattern of transition of the reflected wave detection time becomes the same every time the subject S makes one rotation. For example, the portions A and B in the transition pattern shown on the right side of FIG. 11 indicate the transition pattern at the same position in the subject S. Here, there is a unique structural non-uniformity that varies depending on the material, design, manufacturing method, and manufacturing environment as described in the embodiment as shown in FIG. Appears as a pattern on the surface of or inside. When this is detected, inference is performed according to the concept described in FIG.

  FIG. 12 is a flowchart showing an example of flaw detection access control processing executed in the ultrasonic inspection apparatus 3. This flowchart is also a flowchart corresponding to the imaging access control processing executed in the endoscope apparatus 1 according to the first embodiment.

  As shown in FIG. 12, in the flaw detection access control process, first, the control circuit 307 determines whether or not the access mode is set (S1201). In the ultrasonic inspection apparatus 3, in the access mode, the ultrasonic probe 301 is rotated when the ultrasonic probe 301 is rotated with respect to the subject S which is a hollow cylindrical structure such as an iron pipe. In this mode, the rotation angle can be presented to the user. Here, rotating the ultrasonic probe 301 with respect to the subject S also includes rotating the ultrasonic probe 301 relative to the subject S. Therefore, for example, as shown on the left side of FIG. 11, a case where the subject S rotates with respect to the ultrasonic probe 301 is also included. The access mode can be set by the user operating the operation unit 309.

  If the determination result in S1201 is YES, next, the ultrasonic probe 301 is applied to the subject S (S1202). This application may be performed mechanically under the control of the control circuit 307, for example. Further, the contact medium 311 such as glycerin is applied to the outer peripheral surface of the subject S in advance, and when the ultrasonic probe 301 is applied to the subject S, the subject S and the ultrasonic probe 301 are applied. A contact medium 311 is provided between and. Note that S1202 is not executed thereafter until the determination result of S1201 becomes NO.

  Next, the control circuit 307 is acquired in time series using the ultrasonic probe 301 from that time point until the determination result of S1201 becomes NO (hereinafter, referred to as “access mode period”). S1203 to S1214 are performed each time a second ultrasonic image and subsequent ultrasonic images are acquired. The access mode period may start at the timing when the user performs a predetermined operation on the operation unit 309 after S1202.

  Specifically, when the second and subsequent ultrasonic images are acquired, the control circuit 307 compares the acquired ultrasonic image with the previously acquired ultrasonic image (S1203), and determines the overlapping portion. A determination is made (S1204), and it is determined whether or not there is an overlapping portion (S1205).

When the determination result of S1205 is NO, the control circuit 307 issues a warning by displaying a warning message on the display unit 306 (S1206).
On the other hand, when the determination result of S1205 is YES, the control circuit 307 performs the one rotation determination process (S1207). In the one-rotation determination process, the acquired ultrasonic image and the previously acquired ultrasonic image are overlapped at the determined overlapping portion to generate a composite image. However, in S1207 performed after the second time in the access mode period, the acquired ultrasonic image and the composite image generated in the previous S1207 are overlapped at the determined overlapping portion to generate a new composite image. Then, it is determined from the generated composite image whether or not the ultrasonic probe 301 makes one rotation (360 ° rotation) with respect to the subject S. In this determination, for example, when the same partial image is detected at the opposite ends of the generated combined image, it is determined that one rotation has been performed. Here, the opposite ends of the combined image are, for example, opposite ends in the arrangement direction of the plurality of ultrasonic images used to generate the combined image.

  After S1206 or S1207, the control circuit 307 determines whether or not the rotation angle of the ultrasonic probe 301 with respect to the subject S after the start of the access mode period can be determined (S1208).

When the determination result of S1208 is NO, the control circuit 307 gives a warning by displaying a warning message on the display unit 102 (S1209).
On the other hand, if the determination result in S1208 is YES, the control circuit 307 specifies, for example, the length of the generated composite image in the specific direction and the partial composite image for one rotation (360 °) included in the composite image. The rotation angle of the ultrasonic probe 301 with respect to the subject S after the start of the access mode period is determined based on the length of the direction, and the rotation angle is displayed on the display unit 306 (S1210). Here, the specific direction is, for example, a direction in which a plurality of ultrasonic images used to generate the composite image are arranged.

  After S1209 or S1210, the control circuit 307 determines whether or not there is an instruction to record an ultrasonic image (S1211). This recording instruction can be issued by the user operating the operation unit 309.

If the decision result in S1211 is NO, the process returns to S1201.
On the other hand, when the determination result of S1211 is YES, the control circuit 307 causes the recording circuit 308 to record the ultrasonic image acquired using the ultrasonic probe 301 (S1212).

After S1212, the control circuit 307 determines whether or not the recording of the ultrasonic image is completed (S1213).
If the decision result in S1213 is NO, the process returns to S1201.

  On the other hand, if the determination result in S1213 is YES, the control circuit 307 creates a file of the ultrasonic image recorded in the recording circuit 308 in S1212 (S1214). That is, the image data representing the ultrasonic image and the auxiliary data relating to the ultrasonic image are filed and recorded in the recording circuit 308 as an image file. The auxiliary data may include the rotation angle determined in S1210.

After S1214, the process returns to S1201.
On the other hand, if the determination result in S1201 is NO, the control circuit 307 determines whether or not there is an inference model acquisition instruction (S1215). The instruction to acquire the inference model can be issued by the user operating the operation unit 309.

If the decision result in S1215 is NO, the process returns to S1201.
On the other hand, if the determination result in S1215 is YES, the control circuit 307 sets the characteristics of the target object (structure) to be inspected, and issues an inference model acquisition request including the characteristics to the external device 2. (S1216). Here, the characteristic of the target object includes the material and the like, and the setting of the characteristic can be performed by the user operating the operation unit 309.

After S1216, the control circuit 307 acquires the inference model corresponding to the acquisition request from the external device 2 and stores it in the internal nonvolatile memory (S1217).
After S1217, the process returns to S1201.

  As described above, according to the ultrasonic inspection apparatus 3 according to the second embodiment, the rotation angle of the ultrasonic probe 301 when the ultrasonic probe 301 is rotated with respect to the subject S is determined. Can be realized by a configuration that is not limited by cost or size.

  Although the first and second embodiments have been described above, each of the endoscope apparatus 1, the external device 2, and the ultrasonic inspection apparatus 3 described above is realized by, for example, the hardware configuration shown in FIG. 13. May be.

FIG. 13 is a diagram illustrating an example of the hardware configuration.
The hardware configuration shown in FIG. 13 is a portable recording in which a CPU (Central Processing Unit) 401, a memory 402, an input / output device 403, an input / output IF (interface) 404, a storage device 405, and a portable recording medium 408 are stored. A medium driving device 406 and a communication IF 407 are provided, and these are connected to each other via a bus 409.

  The CPU 401 is a computing device that executes a program for processing performed by the endoscope device 1, the external device 2, or the ultrasonic inspection device 3. The memory 402 is a RAM (Random Access Memory) and a ROM (Read Only Memory), the RAM is used as a work area of the CPU 401, and the ROM stores a program and information necessary for executing the program in a nonvolatile manner.

The input / output device 403 is an input device such as a touch panel and a keyboard, and an output device such as a display device.
The input / output IF 404 is an interface to which the insertion unit 101 (imaging unit 104), the ultrasonic probe 301, or the like is connected and which transmits / receives a signal to / from the connected unit.

  The storage device 405 is a storage that non-volatilely stores a program, information necessary for executing the program, information obtained by executing the program, and the like. The storage device 405 is, for example, an HDD. The portable recording medium drive device 406 drives the portable recording medium 408 to access the recorded contents. The portable recording medium 408 is a memory device, a flexible disk, an optical disk, a magneto-optical disk, or the like. The portable recording medium 408 also includes a CD-ROM (Compact Disk Read Only Memory), a DVD (Digital Versatile Disk), a USB (Universal Serial Bus) memory, and the like. Like the storage device 405, the portable recording medium 408 is also a storage that non-volatilely stores a program, information necessary for executing the program, information acquired by executing the program, and the like.

  The communication IF 407 is an interface that is connected to a network and communicates with the endoscope apparatus 1, the external device 2, the ultrasonic inspection apparatus 3, or the like via the network.

  Although the embodiment has been described above, the present invention is not limited to the above embodiment as it is, and constituent elements can be modified and embodied at the stage of implementation without departing from the scope of the invention. Further, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the above embodiments. For example, some of all the constituent elements shown in the embodiment may be deleted. Furthermore, the constituent elements of different embodiments may be combined appropriately.

DESCRIPTION OF SYMBOLS 1 Endoscope device 2 External device 3 Ultrasonic examination apparatus 101 Insertion part 102 Display part 103 Bending operation part 104 Imaging part 104a Optical system 104b Imaging element 104c Bending part 105 Clock part 106 Control part 106a Overlap determination part 106b Synthesis processing part 106c Drive unit 106d Communication control unit 106e Recording control unit 106f Movement amount determination unit 106g Display control unit 107 Inference engine 107a Inference model 107b Storage unit 108 Communication unit 109 Operation unit 110 Recording unit 121 Images 122, 123 Composite image 201 External image DB
201a Image recording unit 201b Communication unit 202 Machine learning unit 202a Population creation unit 202b Output setting unit 202c Input / output modeling unit 202d Communication unit 301 Ultrasonic probe 302 Transmission / reception circuit 303 Transmission / reception control circuit 304 Image conversion circuit 305 Display control circuit 305 306 display unit 307 control circuit 308 recording circuit 309 operation unit 310 communication unit 311 contact medium 401 CPU
402 memory 403 input / output device 404 input / output IF
405 storage device 406 portable recording medium drive device 407 communication IF
408 Portable recording medium 409 Bus

Claims (16)

A movement amount determination device for determining a relative movement amount of an image acquisition unit used for acquiring an image of a measurement object,
In the images acquired using the image acquisition unit after a specific timing, every time the second and subsequent images are acquired, the overlapping portion for determining the overlapping portion between the acquired image and the previously acquired image is overlapped. A judgment unit,
A plurality of images acquired by using the image acquisition unit after the specific timing, a composite image generation unit that generates a composite image by overlapping the overlapping portion determined by the overlap determination unit,
A movement amount determination unit that determines a relative movement amount of the image acquisition unit with respect to the measurement object based on the synthesis image generated by the synthesis image generation unit,
A movement amount determination device comprising: A display unit that displays the movement amount determined by the movement amount determination unit,
The movement amount determination device according to claim 1, further comprising: The overlap determination unit determines the overlap portion using an inference model,
The movement amount determination device according to claim 1 or 2, wherein. The inference model is obtained by machine learning in which a plurality of images taken with a known shift amount along the object and the relationship of the overlapping portions are used as at least one input / output pair as teacher data.
The movement amount determination device according to claim 3, wherein. The inference model is obtained by machine learning in which at least one input / output pair is used as teacher data for the relationship between two images captured with a known shift amount along the object and the width of the overlapping portion of these two images. ,
The movement amount determination device according to claim 3, wherein. The inference model is a plurality of images taken with a known shift amount along the object, and the relative movement amount of the image acquisition unit when the image of the object is acquired is at least one input / output pair teaching data. Obtained by machine learning performed as
The movement amount determination device according to claim 3, wherein. A communication unit for communicating with an external device is further provided,
The inference model is acquired from the external device via the communication unit,
The movement amount determination device according to any one of claims 3 to 6, characterized in that. The image acquisition unit is an imaging unit used to acquire an image,
The movement amount determination device according to any one of claims 1 to 7, characterized in that. When the image capturing unit moves in one direction, the composite image generated by the composite image generating unit is a direction in which a plurality of images acquired using the image capturing unit after the specific timing are orthogonal to the one direction. Is an image placed in
The movement amount determination device according to claim 8, wherein. The movement amount determination unit determines the movement amount of the imaging unit by converting the length of the composite image generated by the composite image generation unit into the movement amount of the imaging unit,
The movement amount determination device according to claim 9, wherein. The movement amount determination unit determines the movement amount of the imaging unit based on a composite image generated by the composite image generation unit and a reference composite image,
The reference composite image is an image acquired every time the second and subsequent images are acquired in the image acquired using the image capturing unit while the image capturing unit is moving the reference movement amount in the one direction. And a plurality of images acquired by using the image capturing unit while the image capturing unit determines the overlapping portion with the image previously acquired and the image capturing unit moves the reference movement amount in the one direction, It is created by the composite image generation unit overlapping to generate a composite image at the overlapping portion,
The movement amount determination device according to claim 9, wherein. The image acquisition unit is an ultrasonic probe used to acquire an ultrasonic image,
The movement amount determination device according to any one of claims 1 to 7, characterized in that. The movement amount of the ultrasonic probe determined by the movement amount determination unit is the rotation angle of the ultrasonic probe with respect to the subject having a hollow cylindrical shape,
The movement amount determination device according to claim 12, wherein. The movement amount determination unit determines the length of the combined image generated by the combined image generation unit and the length of the partial combined image included in the combined image for one rotation of the ultrasonic probe with respect to the subject. Based on, determine the rotation angle of the ultrasonic probe with respect to the subject,
14. The movement amount determination device according to claim 13, wherein: A movement amount determination method executed in a movement amount determination device that determines a relative movement amount of an image acquisition unit used to acquire an image of a measurement object,
In the image acquired using the image acquisition unit after a specific timing, every time the second and subsequent images are acquired, the overlapping portion between the acquired image and the previously acquired image is determined,
A plurality of images acquired by using the image acquisition unit after the specific timing are generated at the overlapping portion to generate a composite image,
Based on the composite image, determine the relative movement amount of the image acquisition unit with respect to the measurement object,
A movement amount determination method characterized by the above. In the computer of the movement amount determination device that determines the relative movement amount of the image acquisition unit used to acquire the image of the measurement object,
In the image acquired using the image acquisition unit after a specific timing, every time the second and subsequent images are acquired, the overlapping portion between the acquired image and the previously acquired image is determined,
A plurality of images acquired by using the image acquisition unit after the specific timing are generated at the overlapping portion to generate a composite image,
Based on the composite image, determine the relative movement amount of the image acquisition unit with respect to the measurement object,
A movement amount determination program, which is characterized by causing the process to be executed.