JP7443303B2 - Measuring device, measuring method and program - Google Patents

Description

Embodiments of the present invention relate to a measuring device, a measuring method, and a program.

For example, work is being carried out to image the inside of social infrastructure such as buildings and tunnels, or the inside of buildings using cameras, and to inspect using a large amount of image data obtained by the imaging. For example, the inside of social infrastructure, the inside of buildings, and the conditions of equipment installed therein are inspected using a large amount of image data. In such inspection work, the inspector has had to compare image data obtained by capturing an image of an inspection object with image data obtained by capturing images of the same inspection object at different times.

Patent No. 6788291

Masako Kashiwagi, Nao Mishima, Tatsuo Kozakaya, Shinsaku Hiura, “Deep Depth from Aberration Map”, 2019 IEEE/CVF International nal Conference on Computer Vision (ICCV), IEEE, 27 Oct. -2 Nov. 2019, pp. 4069-4078 Ryo Nakashima and Akihito Seki, “SIR-Net: Scene-Independent End-to-End Trainable Visual Relocalizer”, 2019 International C on 3D Imaging, Modeling, Processing, Visualization and Transmission (3DIMPVT), IEEE, 16-19 sept. 2019, pp. 472-481

With conventional technology, it has been difficult to detect three-dimensional changes in warpage and bending from image data.

The measuring device of the embodiment includes an acquisition section, a measurement control section, a storage section, and a comparison section. The acquisition unit acquires first target image data including the target object. The measurement control unit estimates a first three-dimensional position for each pixel of the first target image data, and defines a reference point defined on the first target image data and a reference point defined with the reference point as both ends. A first degree of distortion indicating the degree of distortion of the object on the reference line is measured from the reference line and the first three-dimensional position. The storage unit stores a second degree of distortion associated with the reference line. The comparison unit compares the first degree of distortion and the second degree of distortion, and outputs a comparison result.

FIG. 1 is a diagram illustrating an example of a functional configuration of a measuring device according to a first embodiment. 1 is a flowchart illustrating an example of a measurement method according to the first embodiment. FIG. 3 is a diagram showing example 1 of edges defined for the reconstructed three-dimensional shape of the first embodiment. FIG. 7 is a diagram illustrating a second example of edges defined for the reconstructed three-dimensional shape of the first embodiment. FIG. 3 is a diagram for explaining the degree of distortion in the first embodiment. FIG. 7 is a diagram illustrating an example of the functional configuration of a measuring device according to a second embodiment. 7 is a flowchart illustrating an example of a measurement method according to the second embodiment. FIG. 7 is a diagram illustrating an example of a functional configuration of a measuring device according to a third embodiment. The figure which shows the example of the functional structure of the measuring device of 4th Embodiment. The figure which shows the example of the measurement result image data of 4th Embodiment. 10 is a flowchart illustrating an example of a measurement method according to a fourth embodiment. The figure which shows the example of the functional structure of the measuring device of 5th Embodiment. 10 is a flowchart illustrating an example of a measurement method according to a fifth embodiment. FIG. 3 is a diagram illustrating an example of the hardware configuration of the measuring device according to the first to fifth embodiments.

Embodiments of a measuring device, a measuring method, and a program that make it possible to capture three-dimensional warping and bending (hereinafter referred to as "strain") from image data will be described in detail below with reference to the accompanying drawings.

(First embodiment)
First, the measuring device of the first embodiment will be explained.
[Example of functional configuration]
FIG. 1 is a diagram showing an example of the functional configuration of a measuring device 100 according to the first embodiment. The measuring device 100 of the first embodiment includes an acquisition section 1, a measurement control section 2, a storage section 3, and a comparison section 4. Each functional block is realized by a combination of one or more devices.

When each functional block is realized by a plurality of devices, each device is connected to be able to communicate directly or indirectly via a communication network. This allows each device to mutually transmit and receive shape data, etc., for example.

The type of communication network is arbitrary; for example, each device may be able to communicate with each other via a LAN (Local Area Network) installed within a building. Further, for example, each device may be configured to be able to communicate with each other via a network (cloud) such as the Internet.

Below, an overview of each functional block will be explained in order. Note that details of the operation of each functional block will be described later with reference to the flowchart of FIG. 2.

The acquisition unit 1 acquires target image data including an object (subject) to be measured.

The measurement control unit 2 estimates a three-dimensional position for each pixel of the target image data, and from the reference point defined on the target image data, the reference line defined with the reference point as both ends, and the three-dimensional position, The degree of distortion, which indicates the degree of distortion of the object on the reference line, is measured.

Specifically, the measurement control section 2 includes a first distance calculation section 21, a second distance calculation section 22, and a distortion degree calculation section 23.

The first distance calculation unit 21 estimates a three-dimensional position for each pixel of the target image data, and uses the three-dimensional position indicating the reference point to calculate the shortest distance between the reference points.

The second distance calculation unit 22 defines edges for a set of three-dimensional positions corresponding to pixels of the target image data, thereby generating a graph with three-dimensional positions as nodes, and between nodes indicating reference points. Calculate the minimum cost distance on the graph.

The distortion degree calculation unit 23 calculates the distortion degree based on the ratio between the shortest distance calculated by the first distance calculation unit 21 and the minimum cost distance on the graph calculated by the second distance calculation unit 22.

The storage unit 3 stores target image measurement information including target image data, one or more reference lines, and a degree of distortion corresponding to each reference line. Target image measurement information is managed for each target image data unit.

The comparison unit 4 compares the degree of distortion calculated by the degree of distortion calculation unit 23 and the degree of distortion in the storage unit 3, and outputs the comparison result. The comparison results include, for example, the presence or absence of a change in the degree of strain, a difference in the degree of strain, and the like.

[Example of measurement method]
FIG. 2 is a flowchart showing an example of the measurement method according to the first embodiment. First, the acquisition unit 1 acquires target image data in which a target object is captured from an external storage device or an imaging device such as a camera (step S1).

Next, the first distance calculation unit 21 calculates a three-dimensional position P for each pixel from the target image data acquired in step S1 (step S2).

The three-dimensional position P in this embodiment is determined by depth data and camera parameters. For example, the first distance calculation unit 21 estimates the distance to the object at each pixel position based on the intensity of blur in the target image data.

For example, the depth data can be estimated from the target image data by performing machine learning in advance on a model that associates the intensity of blur of an object included in the target image data with the distance to the object. Specifically, depth data can be calculated from target image data by using, for example, the deep neural network described in Non-Patent Document 1.

The set of three-dimensional positions P corresponding to each pixel can also be regarded as a set of points expressing the three-dimensional shape of the object included in the target image data. Hereinafter, the set of three-dimensional positions P will be referred to as a reconstructed three-dimensional shape.

Next, the first distance calculation unit 21 calculates the three-dimensional shortest distance D1 between two points specified in advance on the target image data (step S3).

The shortest distance D1 is obtained by calculating the distance between the three-dimensional positions Ps and Pe corresponding to the pixel at the designated position. Hereinafter, the line segment defined by Ps and Pe will be referred to as a reference line, and both end points of the reference line will be referred to as reference points. The information indicating the reference point includes information on a two-dimensional position in the image coordinate system and a three-dimensional position corresponding to the two-dimensional position. In the following explanation, processing for one reference line will be explained. Note that even if there are multiple reference lines, this can be handled by repeating the same process multiple times.

Note that in this embodiment, the three-dimensional position P is estimated from the depth based on the intensity of blur of the image, but the three-dimensional position P may be estimated from the context of the target image data or estimated from a stereo camera. Parallax may be used, or measurement may be performed using a distance measuring device such as LiDAR that uses laser irradiation.

Next, the second distance calculation unit 22 defines an edge for the three-dimensional position P corresponding to each pixel of the target image data, thereby generating a graph with the three-dimensional position P corresponding to the pixel as a node. (Step S4).

FIG. 3A is a diagram showing an example of an edge 201 defined for the reconstructed three-dimensional shape (set of three-dimensional positions P) according to the first embodiment. In FIG. 3A, an example will be described in which the target object 200 included in the target image data is a propeller. Each pixel of the target image data corresponds to a three-dimensional position P. Furthermore, in the example of FIG. 3A, a reference line 202 is defined between the three-dimensional positions Ps and Pe. The second distance calculation unit 22 generates an edge 201 between nodes corresponding to adjacent pixels (for example, between three-dimensional positions Pa and Pb).

Returning to FIG. 2, the second distance calculation unit 22 uses the three-dimensional position P calculated by the first distance calculation unit 21 and two reference points on the image specified in advance to determine the three-dimensional position P. The distance along the surface of the shape to be formed is determined by the minimum cost distance D2 on the graph (step S5). For example, Dijkstra's method can be used to calculate the shortest distance. In this case, the cost on the edge is determined by the following equation (1).

Cost _e = |Pa-Pb| ...(1)
Here, Pa and Pb are three-dimensional positions P (nodes) corresponding to adjacent pixels, respectively. The second distance calculation unit 22 sets the node corresponding to the three-dimensional position Ps of the reference point as the starting point, and the node corresponding to the three-dimensional position Pe of the reference point as the end point, on the graph so that Cost _e is minimized. Calculate the shortest distance.

Note that in FIG. 3A, the edge 201 is formed only between nodes corresponding to some adjacent pixels, but the edge 201 may be formed between nodes corresponding to all adjacent pixels. In that case, a maximum of eight edges will be constructed from one node, except for nodes corresponding to image boundaries. Further, if the range in which the target shape is captured is known, only nodes corresponding to pixels included in the range may be used in the graph configuration. For example, semantic segmentation technology, which is being researched in the field of image recognition, can be used to specify the range. Furthermore, as shown in FIG. 3B, a graph may be constructed using only nodes included within a certain range 210 from the reference line 202 on the image. The range can be specified, for example, as follows.

The following equation (2 ) A graph is composed of a set of nodes that satisfy ).

L(P(i))<=r...(2)

Alternatively, instead of using the range corresponding to the semicircular part of the boundary of the fixed range 210 in FIG. You may switch. The fixed range 210 may be any range specified by conditions based on the distance from the reference line 202.

Next, the distortion degree calculation unit 23 calculates the distortion degree indicating the degree of distortion of the object on the reference line from the shortest distance D1 and the minimum cost distance D2 (step S6). Specifically, the degree of distortion is calculated by D2/D1. This degree of distortion can be considered as a value corresponding to the amount of change with respect to the flat state.

FIG. 4 is a diagram for explaining the degree of distortion in the first embodiment. The shortest distance of the reference line 202 is assumed to be D1. Further, the route 203 has a minimum cost distance D′2 on the graph generated by the second distance calculation unit 22, and the route 204 has a minimum cost distance D′2 on the graph generated by the second distance calculation unit 22. Suppose that it has a distance D2. As shown in FIG. 4, when minimum cost distance D'2<minimum cost distance D2, distortion degree S' (=D'2/D1)<distortion degree S (=D2/D1).

Returning to FIG. 2, next, the comparison unit 4 compares the degree of distortion S' stored in the storage unit 3 and the degree of distortion S calculated in step S6 (step S7). Specifically, the comparison unit 4 determines that there is no change if the difference between the degrees of distortion is less than or equal to a predetermined threshold. For example, when the difference between the strain degrees is larger than a predetermined threshold value, the comparison unit 4 outputs the difference between the strain degrees to a display device or the like connected to the measuring device 100.

In this embodiment, a case has been described in which the degree of distortion S is measured and compared with respect to one reference line, but the process is similar when a plurality of reference lines are used as targets. For example, in the case of periodic inspections, the locations (reference lines) that require measurement are considered to have been determined in advance, so the degree of strain S' corresponding to each location is stored in the storage unit 3, and the locations that require measurement are determined in advance. Measurements and comparisons can be made in the order of inspection. For example, target image data including a target object and a plurality of reference lines defined for the target object is stored in the storage unit 3, a UI (User Interface) for selecting a reference line to be compared is prepared, and the UI may be used to select a reference line for measurement and comparison.

According to the measuring device 100 of the first embodiment, it is possible to detect three-dimensional changes in warpage and bending from target image data.

(Second embodiment)
Next, a second embodiment will be described. In the description of the second embodiment, descriptions similar to those in the first embodiment will be omitted, and points different from the first embodiment will be described. In the second embodiment, a case will be described in which the imaging position and imaging orientation of target image data are estimated, and a reference line to be compared is specified from the estimated imaging position and imaging orientation.

[Example of functional configuration]
FIG. 4 is a diagram showing an example of the functional configuration of the measuring device 100-2 of the second embodiment. The measuring device 100-2 of the second embodiment includes an acquisition section 1, a measurement control section 2, a storage section 3, a comparison section 4, and an estimation section 5. In this embodiment, an estimation unit 5 is newly provided.

The estimation unit 5 estimates imaging position data (for example, coordinates, etc.) indicating the imaging position of the imaging device and imaging posture data indicating the imaging attitude of the imaging device from the target image data captured by the imaging device.

In this embodiment, the storage unit 3 stores a database generated in advance, and the estimation unit 5 acquires a plurality of pieces of reference imaging information from the database. Each piece of reference imaging information is stored in association with reference image data, an imaging position when the reference image data is taken, and an imaging posture when the reference image data is taken. Then, the estimation unit 5 estimates the imaging position data and the imaging posture data by comparing the target image data with each of the plurality of reference imaging information.

Specifically, the estimating unit 5 searches for reference image data (hereinafter referred to as "proximity image data") captured at an imaging position and imaging orientation most similar to the target image data, from among the plurality of reference image data. . The estimation unit 5 then generates imaging position data and imaging orientation data based on the position difference and orientation difference between the target image data and the proximity image data, and the reference position data and reference orientation data associated with the proximity image data. Calculate.

Note that the estimation unit 5 can calculate the imaging position data and the imaging posture data using a deep neural network based on the target image data and a plurality of pieces of reference imaging information. For example, by using the deep neural network described in Non-Patent Document 2, it is possible to calculate imaging position data and imaging posture data.

The estimation unit 5 inputs the imaging position data and the imaging posture data to the comparison unit 4.

The comparison unit 4 identifies a reference line based on at least one of the imaging position data and the imaging posture data. Specifically, in this embodiment, the storage unit 3 stores target image measurement information further including imaging position data and imaging posture data. That is, in the present embodiment, the target image measurement information is stored in association with data indicating the degree of distortion, the target image data for which the degree of distortion was measured, and the imaging position and imaging posture of the imaging device that captured the target image data. be done. The comparison unit 4 acquires target image measurement information including imaging position data closest to the imaging position data estimated by the estimation unit 5, for example, from the storage unit 3.

Next, the comparing unit 4 calculates the distance between the imaging position data estimated by the estimating unit 5 and the imaging position data included in the target image measurement information acquired from the storage unit 3, and determines whether the distance is determined in advance. If the threshold value is exceeded, information indicating that there is no comparison target data is output to a display device or the like connected to the measuring device 100.

If it is below the threshold value, the comparing unit 4 performs the same comparison process as the comparing unit 4 of the first embodiment, and outputs the presence or absence of a change and the difference in the degree of distortion to a display device or the like connected to the measuring device 100.

Note that in the above description, the comparison unit 4 narrowed down the target image measurement information using only the imaging position data, but may further narrow down the target image measurement information using the imaging posture data. In that case, the comparison unit 4 determines whether the difference in posture is within the threshold value, similarly to the imaging position data, and if it is within the threshold value, performs the comparison process. By comparing the imaging posture data after the imaging position data, even if target image measurement information obtained from the results of taking multiple images at the same position with different postures exists in the storage unit 3, similar locations (comparison target locations) ) can be accurately compared.

It is also conceivable that the threshold value processing using the imaging position data is not performed. For example, in a case where a fixed position for photographing for inspection is determined, the positions are the same, so it is sufficient to perform threshold processing using only the photographing posture data. Alternatively, target image data may be directly compared. In that case, the comparison unit 4 may calculate the degree of similarity using various image recognition methods for calculating the degree of similarity between the two target image data, and perform threshold processing based on the degree of similarity.

[Example of measurement method]
FIG. 6 is a flowchart showing an example of the measurement method according to the second embodiment. The description of step S1 is the same as that of the measurement method of the first embodiment (see FIG. 2), so it will be omitted.

Next, the estimation unit 5 estimates the imaging position of the imaging device and the imaging posture of the imaging device from the target image data acquired in step S1 (step S1-2).

Descriptions of steps S2 to S7 are omitted because they are the same as the measurement method of the first embodiment (see FIG. 2).

Next, the comparison unit 4 reads the distortion degree S' of the comparison target from the storage unit 3 based on the imaging position and imaging posture estimated in step S1-2, and calculates the distortion degree S' and the distortion degree S' in step S6. The resulting strain degree S is compared (step S7).

According to the second embodiment, for example, even if a plurality of target image measurement information obtained from the results of photographing a plurality of images while changing the imaging position and the imaging posture exist in the storage unit 3, the degree of distortion of the comparison target can be correctly determined. You can choose.

(Third embodiment)
Next, a third embodiment will be described. In the description of the third embodiment, descriptions similar to those in the first embodiment will be omitted, and points different from the first embodiment will be described. In the first and second embodiments, a method for calculating the degree of distortion corresponding to an interval (reference line) defined as an arbitrary line segment on target image data, and a method for calculating the degree of distortion corresponding to an interval (reference line) defined as an arbitrary line segment on target image data, and a method for determining whether or not there is a change in shape of the target part using the degree of distortion are described. The detection method was explained. This embodiment differs from the first embodiment in that a reference line to be processed is automatically selected.

[Example of functional configuration]
FIG. 7 is a diagram showing an example of the functional configuration of a measuring device 100-3 according to the third embodiment. The measurement device 100-3 of the third embodiment includes an acquisition section 1, a measurement control section 2, a storage section 3, and a comparison section 4. The configuration of the third embodiment is the same as the first embodiment. Functional blocks that differ in processing from the first embodiment will be described below.

First, the storage unit 3 in the third embodiment further stores the above-mentioned reconstructed three-dimensional shape (set of three-dimensional positions P) and reference points associated with the reconstructed three-dimensional shape. This is different from the storage unit 3 of the first and second embodiments. In the following explanation, the reconstructed three-dimensional shape generated by the first distance calculation unit 21 will be referred to as reconstructed three-dimensional shape A, and the reconstructed three-dimensional shape stored in the storage unit 3 will be referred to as reconstructed three-dimensional shape B. do.

The first distance calculation unit 21 in the third embodiment automatically selects the reference line as follows. First, the first distance calculation unit 21 performs alignment between the reconstructed three-dimensional shape A and the reconstructed three-dimensional shape B, and obtains conversion information that allows the two to overlap to the maximum extent possible.

The transformation information includes, for example, a rotation matrix R and a translation vector t. The rotation matrix R and translation vector t are obtained by using, for example, ICP (Iterative Closest Point). Next, the first distance calculation unit 21 applies the rotation matrix R and the translation vector t to the three-dimensional position P indicating the position of the reference line included in the target image measurement information. Next, the first distance calculation unit 21 calculates the three-dimensional positions Ps and Pe included in the reconstructed three-dimensional shape A that are closest to both end points of the reference line after applying the rotation matrix R and the translation vector t. . Next, the first distance calculation unit 21 uses the line segment having the three-dimensional positions Ps and Pe as the reference line to be processed, and performs the subsequent measurement and comparison process in the same manner as in the first or second embodiment. I will do it.

According to the third embodiment, it is possible to automatically extract a measurement/comparison point (a reference line for measurement/comparison processing) of the target object 200 included in the target image data.

(Fourth embodiment)
Next, a fourth embodiment will be described. In the description of the fourth embodiment, descriptions similar to those in the first to third embodiments will be omitted, and portions that are different from the first to third embodiments will be described. The fourth embodiment differs from the first to third embodiments in the portion that visualizes not only the presence or absence of a change in the degree of strain and the magnitude of the difference, but also the presence or absence of a change in the degree of strain and the magnitude of the difference.

[Example of functional configuration]
FIG. 8 is a diagram showing an example of the functional configuration of a measuring device 100-4 according to the fourth embodiment. The measurement device 100-4 of the fourth embodiment includes an acquisition section 1, a measurement control section 2, a storage section 3, a comparison section 4, an estimation section 5, a generation section 6, and a display section 7. In the fourth embodiment, a generation section 6 and a display section 7 are newly provided.

The generation unit 6 generates an image (hereinafter referred to as "measurement result image data") that allows the degree of distortion or the difference in comparison results to be intuitively understood.

The display section 7 displays the measurement result image data generated by the generation section 6.

FIG. 9 is a diagram showing an example of measurement result image data according to the fourth embodiment. The generation unit 6 determines the display format of the reference line 202 based on the comparison result of the comparison unit 4, generates measurement result image data in which the reference line 202 is superimposed on the target image data, and displays the measurement result image data. output to section 7. Specifically, the generation unit 6 generates measurement result image data by superimposing a reference line 202 whose color, line thickness, etc. are changed depending on the degree of distortion, for example, on the target image data. do. This makes it possible to intuitively understand the degree of distortion from a two-dimensional image display.

Further, the generation unit 6 may superimpose the route 204 corresponding to the minimum cost distance D2 on the graph generated by the second distance calculation unit 22 on the target image data. In this case, the shortest path determined by Dijkstra's method in the second distance calculation unit 22 may be superimposed on the target image data. Since each point of the reconstructed three-dimensional shape has a one-to-one correspondence with a pixel, if the edge and node corresponding to the shortest path are known, the connection between the corresponding pixels can be found. By tracing this on the image, it can be superimposed on the target image data.

Furthermore, in order to intuitively indicate the degree of deviation between the reference line 202 corresponding to the shortest distance D1 and the route 204 corresponding to the minimum cost distance D2, the generation unit 6 creates a measurement in which the area surrounded by both lines is filled in. Result image data may also be generated.

Furthermore, the generation unit 6 may visually present the size by defining a symbol such as an arrow 205 from one side to the other. At this time, the generation unit 6 defines, for example, the foot of a perpendicular line from each node on the shortest path corresponding to the minimum cost distance D2 to the reference line corresponding to the shortest distance D1, and projects it onto the target image data. An arrow 205 can be defined. At this time, the generation unit 6 may change the color or thickness depending on the length of the perpendicular line, or may sample nodes used for calculating the perpendicular line at regular intervals. The former makes it possible to intuitively understand the size of the deviation between the three-dimensional shortest distance D1 and minimum cost distance D2, and the latter prevents the arrows 205 from becoming too dense and reducing visibility. Is possible.

[Example of measurement method]
FIG. 10 is a flowchart showing an example of a measurement method according to the fourth embodiment. The explanation of steps S1 to S7 is omitted because it is the same as the measurement method of the second embodiment (see FIG. 6).

The generation unit 6 generates the above-mentioned measurement result image data that allows the degree of distortion or the difference in comparison results to be intuitively understood, and displays the measurement result image data on the display unit 7 (step S8).

According to the fourth embodiment, it is possible to intuitively understand the degree of distortion or the difference in comparison results.

(Fifth embodiment)
Next, a fifth embodiment will be described. In the description of the fifth embodiment, descriptions similar to those in the first to fourth embodiments will be omitted, and portions that are different from the first to fourth embodiments will be described. The fifth embodiment differs in that it has a function of interactively accepting reference line input.

[Example of functional configuration]
FIG. 11 is a diagram showing an example of the functional configuration of a measuring device 100-5 according to the fifth embodiment. The measuring device 100-5 of the fifth embodiment includes an acquisition section 1, a measurement control section 2, a storage section 3, a comparison section 4, an estimation section 5, a generation section 6, a display section 7, and an input section 8. In the fourth embodiment, an input section 8 is newly provided.

The input unit 8 receives an input indicating a portion of the target image data presented on the display unit 7 whose degree of distortion is desired to be measured. The input unit 8 may take the form of any device capable of indicating a position, such as a mouse, keyboard, or trackball. Furthermore, the input unit 8 may be a device such as a touch panel that has an integrated input function and display function.

The display unit 7 displays at least target image data. The data to be displayed is not limited to the target image data, but any result related to the measurement/comparison process of this embodiment may be displayed. For example, the display unit 7 may display the measurement result image data generated by the generation unit 6 together with the target image data. Furthermore, the display section 7 can also display the presence or absence of a change and the difference in the degree of distortion, which are the output results from the comparison section 4.

Further, the generation unit 6 generates an input support image for supporting input of the reference line based on the target image measurement information stored in the storage unit 3, and the display unit 7 displays the input support image. You may. Specifically, the generation unit 6 performs alignment between the reconstructed three-dimensional shape A generated by the first distance calculation unit 21 and the reconstructed three-dimensional shape B stored in the storage unit 3. , find a rotation matrix R and a translation vector t such that the two overlap to the maximum extent possible. The rotation matrix R and the translation vector t can be obtained, for example, by using ICP as in the third embodiment. Next, the first distance calculation unit 21 applies the rotation matrix R and the translation vector t to the three-dimensional position P indicating the position of the reference line included in the target image measurement information. The generation unit 6 generates an input support image by projecting the reference line rotated and translated in accordance with the rotation matrix R and the translation vector t onto the target image data. Thereby, it is possible to intuitively understand where the reference line used in the previous measurement falls on the target image data that is currently being processed.

Note that the projected reference line may be easily distinguished visually from other information by changing its transparency or changing its color.

[Example of measurement method]
FIG. 12 is a flowchart showing an example of the measurement method according to the fifth embodiment. The explanation of steps S1 and S2 is the same as the measurement method of the fourth embodiment (see FIG. 10), so the explanation will be omitted.

Next, the input unit 8 receives an input indicating the reference line 202 via the input support image (step S2-2). The input unit 8 receives input indicating, for example, the position of a three-dimensional position Ps indicating the starting point of the reference line 202 and the position of a three-dimensional position Pe indicating the end point of the reference line 202.

The subsequent measurement/comparison process is performed with respect to the reference line 202 designated by the input process in step S2-2. The explanation of steps S3 to S8 is the same as the measurement method of the fourth embodiment (see FIG. 10), so the explanation will be omitted.

According to the fifth embodiment, it is possible to interactively receive an input of a measurement/comparison point (a reference line for measurement/comparison processing) of the target object 200 included in the target image data.

Finally, examples of the hardware configurations of the measuring devices 100 to 100-5 of the first to fifth embodiments will be described.

[Example of hardware configuration]
FIG. 13 is a diagram showing an example of the hardware configuration of the measuring devices 100 to 100-5 of the first to fifth embodiments.

The measuring devices 100 to 100-5 of the first to fifth embodiments include a control device 301, a main storage device 302, an auxiliary storage device 303, a display device 304, an input device 305, and a communication device 306. The control device 301, main storage device 302, auxiliary storage device 303, display device 304, input device 305, and communication device 306 are connected via a bus 310.

The control device 301 executes the program read from the auxiliary storage device 303 to the main storage device 302. The main storage device 302 is a memory such as ROM (Read Only Memory) and RAM (Random Access Memory). The auxiliary storage device 303 is an HDD (Hard Disk Drive), an SSD (Solid State Drive), a memory card, or the like.

Display device 304 displays display information. The display device 304 is, for example, a liquid crystal display. The input device 305 is an interface for operating the measuring device 100 (100-2 to 100-5). The input device 305 is, for example, a keyboard or a mouse. When the measuring device 100 (100-2 to 100-5) is a smart device such as a smartphone or a tablet terminal, the display device 304 and the input device 305 are, for example, touch panels. Communication device 306 is an interface for communicating with other devices.

The programs executed by the measuring device 100 (100-2 to 100-5) are files in an installable or executable format and are stored on CD-ROMs, memory cards, CD-Rs, DVDs (Digital Versatile Discs), etc. It is recorded on a computer-readable storage medium and provided as a computer program product.

Further, the program executed by the measuring device 100 (100-2 to 100-5) may be stored on a computer connected to a network such as the Internet, and may be provided by being downloaded via the network. . Further, the program executed by the measuring device 100 (100-2 to 100-5) may be provided via a network such as the Internet without being downloaded.

Further, the program to be executed by the measuring device 100 (100-2 to 100-5) may be provided in advance by being incorporated into a ROM or the like.

The program executed by the measuring device 100 (100-2 to 100-5) has a module configuration that includes functional blocks that can also be realized by a program among the functional blocks of the measuring device 100 (100-2 to 100-5). It has become. As actual hardware, each functional block is loaded onto the main storage device 302 when the control device 301 reads and executes a program from a storage medium. That is, each functional block described above is generated on the main storage device 302.

Note that some or all of the functional blocks described above may not be implemented by software, but may be implemented by hardware such as an IC (Integrated Circuit).

Further, when each functional block is implemented using a plurality of processors, each processor may implement one of each functional block, or may implement two or more of each functional block.

Further, the operating mode of the computer realizing the measuring device 100 (100-2 to 100-5) may be arbitrary. For example, the measuring device 100 (100-2 to 100-5) may be realized by one computer. Furthermore, for example, the measuring devices 100 (100-2 to 100-5) may be operated as a cloud system on a network.

Although several embodiments of the invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and changes can be made without departing from the gist of the invention. These embodiments and their modifications are included within the scope and gist of the invention, as well as within the scope of the invention described in the claims and its equivalents.

1 Acquisition unit 2 Measurement control unit 3 Storage unit 4 Comparison unit 5 Estimation unit 6 Generation unit 7 Display unit 8 Input unit 21 First distance calculation unit 22 Second distance calculation unit 23 Distortion degree calculation unit 100 Measuring device 200 Target object 301 Control Device 302 Main storage device 303 Auxiliary storage device 304 Display device 305 Input device 306 Communication device 310 Bus

Claims (21)

an acquisition unit that acquires first target image data including the target object;
A first three-dimensional position is estimated for each pixel of the first target image data, a reference point defined on the first target image data, and a reference line defined with the reference point as both ends; a measurement control unit that measures a first degree of distortion indicating a degree of distortion of the object on the reference line from the first three-dimensional position;
a storage unit that stores a second degree of distortion associated with the reference line;
a comparison unit that compares the first degree of distortion and the second degree of distortion and outputs a comparison result;
A measuring device comprising: further comprising an estimating unit that estimates, from the first target image data, an imaging position and an imaging posture of a first imaging device that captured the first target image data;
The storage unit stores the second degree of distortion in the second target image data in which the second degree of distortion was measured, the imaging position and imaging position of the second imaging device that captured the second target image data. It is associated with the posture and stored as target image measurement information,
The comparison unit includes the second degree of distortion included in the target image measurement information in which the imaging position and imaging orientation of the first imaging device are closer to the imaging position and imaging orientation of the second imaging device; Comparing with the first degree of distortion,
The measuring device according to claim 1. The measurement control section includes:
A first distance calculation that estimates a first three-dimensional position for each pixel of the first target image data and calculates the shortest distance between the reference points using the first three-dimensional position indicating the reference point. Department and
By defining edges for a set of first three-dimensional positions corresponding to pixels of the first target image data, a graph with the first three-dimensional positions as nodes is generated, and the reference point is a second distance calculation unit that calculates a minimum cost distance on the graph between nodes shown;
a distortion degree calculation unit that calculates the first distortion degree based on the ratio of the shortest distance and the minimum cost distance on the graph;
The measuring device according to claim 2. The storage unit further stores a second three-dimensional position estimated for each pixel of the second target image data and a reference point included in the second target image data,
The first distance calculation unit adjusts the second three-dimensional position to the first three-dimensional position so that the first three-dimensional position and the second three-dimensional position overlap more. defining a reference point on the first target image data by calculating conversion information to be converted and applying the conversion information to a reference point included in the second target image data;
The measuring device according to claim 3. By superimposing, on the first target image data, a reference point on the first target image data defined by applying the conversion information to a reference point included in the second target image data. , a generation unit that generates an input support image;
a display unit that displays the input support image;
an input unit that accepts designation of a reference line defined with the reference point as both ends via the input support image;
The measuring device according to claim 4, further comprising: The generation unit determines a display format of the reference line based on the comparison result of the comparison unit, generates measurement result image data in which the reference line is superimposed on the first target image data, and generates measurement result image data in which the reference line is superimposed on the first target image data. outputting image data to the display section;
The measuring device according to claim 5. The generation unit generates the measurement result image data in which a symbol indicating the degree of gap between the reference line and the path on the graph having the minimum cost distance is further superimposed on the first target image data. do,
The measuring device according to claim 6. acquiring first target image data including the target object;
A first three-dimensional position is estimated for each pixel of the first target image data, a reference point defined on the first target image data, and a reference line defined with the reference point as both ends; measuring a first degree of distortion indicating the degree of distortion of the object on the reference line from the first three-dimensional position;
storing a second degree of distortion associated with the reference line;
Comparing the first degree of distortion and the second degree of distortion and outputting a comparison result;
Measurement methods including. The method further includes the step of estimating, from the first target image data, an imaging position and an imaging posture of a first imaging device that captured the first target image data;
The storing step includes storing the second degree of distortion in the second target image data in which the second degree of distortion was measured, the imaging position of the second imaging device that captured the second target image data, and Store it as target image measurement information in association with the imaging posture,
In the step of comparing, the imaging position and imaging orientation of the first imaging device are closer to the imaging position and imaging orientation of the second imaging device with the second degree of distortion included in the target image measurement information; Comparing with the first degree of strain;
The measuring method according to claim 8. The measuring step includes:
estimating a first three-dimensional position for each pixel of the first target image data, and calculating the shortest distance between the reference points using the first three-dimensional position indicating the reference point;
By defining edges for a set of first three-dimensional positions corresponding to pixels of the first target image data, a graph with the first three-dimensional positions as nodes is generated, and the reference point is calculating a minimum cost distance on the graph between the nodes shown;
calculating the first degree of distortion based on the ratio of the shortest distance and the minimum cost distance on the graph;
The measuring method according to claim 9. The storing step further stores a second three-dimensional position estimated for each pixel of the second target image data and a reference point included in the second target image data,
The step of calculating the shortest distance includes changing the second three-dimensional position to the first three-dimensional position so that the overlap between the first three-dimensional position and the second three-dimensional position becomes larger. defining a reference point on the first target image data by calculating conversion information for converting into the second target image data and applying the conversion information to a reference point included in the second target image data;
The measuring method according to claim 10. By superimposing, on the first target image data, a reference point on the first target image data defined by applying the conversion information to a reference point included in the second target image data. , generating an input assistance image;
Displaying the input support image;
receiving, via the input support image, a designation of a reference line defined with the reference point as both ends;
The measuring method according to claim 11, further comprising: The generating step determines the display format of the reference line based on the comparison result of the comparing step, generates measurement result image data in which the reference line is superimposed on the first target image data, and the Outputs the measurement result image data to the display section,
The measuring method according to claim 12. The generating step includes generating the measurement result image data in which a symbol indicating the degree of gap between the reference line and the path on the graph having the minimum cost distance is further superimposed on the first target image data. generate,
The measuring method according to claim 13. computer,
an acquisition unit that acquires first target image data including the target object;
A first three-dimensional position is estimated for each pixel of the first target image data, a reference point defined on the first target image data, and a reference line defined with the reference point as both ends; a measurement control unit that measures a first degree of distortion indicating a degree of distortion of the object on the reference line from the first three-dimensional position;
a storage unit that stores a second degree of distortion associated with the reference line;
a comparison unit that compares the first degree of distortion and the second degree of distortion and outputs a comparison result;
A program to function as further comprising an estimating unit that estimates, from the first target image data, an imaging position and an imaging posture of a first imaging device that captured the first target image data;
The storage unit stores the second degree of distortion in the second target image data in which the second degree of distortion was measured, the imaging position and imaging position of the second imaging device that captured the second target image data. It is associated with the posture and stored as target image measurement information,
The comparison unit includes the second degree of distortion included in the target image measurement information in which the imaging position and imaging orientation of the first imaging device are closer to the imaging position and imaging orientation of the second imaging device; Comparing with the first degree of distortion,
The program according to claim 15. The measurement control section includes:
A first distance calculation that estimates a first three-dimensional position for each pixel of the first target image data and calculates the shortest distance between the reference points using the first three-dimensional position indicating the reference point. Department and
By defining edges for a set of first three-dimensional positions corresponding to pixels of the first target image data, a graph with the first three-dimensional positions as nodes is generated, and the reference point is a second distance calculation unit that calculates a minimum cost distance on the graph between nodes shown;
a distortion degree calculation unit that calculates the first distortion degree based on the ratio of the shortest distance and the minimum cost distance on the graph;
The program according to claim 16. The storage unit further stores a second three-dimensional position estimated for each pixel of the second target image data and a reference point included in the second target image data,
The first distance calculation unit adjusts the second three-dimensional position to the first three-dimensional position so that the first three-dimensional position and the second three-dimensional position overlap more. defining a reference point on the first target image data by calculating conversion information to be converted and applying the conversion information to a reference point included in the second target image data;
The program according to claim 17. By superimposing, on the first target image data, a reference point on the first target image data defined by applying the conversion information to a reference point included in the second target image data. , a generation unit that generates an input support image;
a display unit that displays the input support image;
an input unit that accepts designation of a reference line defined with the reference point as both ends via the input support image;
The program according to claim 18, further comprising: The generation unit determines a display format of the reference line based on the comparison result of the comparison unit, generates measurement result image data in which the reference line is superimposed on the first target image data, and generates measurement result image data in which the reference line is superimposed on the first target image data. outputting image data to the display section;
The program according to claim 19. The generation unit generates the measurement result image data in which a symbol indicating the degree of gap between the reference line and the path on the graph having the minimum cost distance is further superimposed on the first target image data. do,
The program according to claim 20.

Images