JP7477956B2 - Image processing device and control method thereof, and information processing system - Google Patents

Description

The present invention relates to image processing technology for orthorectification.

Traditionally, when inspecting infrastructure structures, cracks and other abnormalities in the structure were confirmed visually, and a report was created by noting the abnormalities on a front view. However, in recent years, reports have begun to be created based on photographed images. In this case, the wall surface of the structure is first photographed from a position directly facing the structure, and the photographed image is aligned with the front view. The report is then created by noting the abnormalities directly on the photographed image aligned with the front view. However, there are cases where it is not possible to photograph the structure from a position directly facing the structure due to factors such as topography.

Patent Document 1 discloses a technology that associates a map with a captured image and converts the captured image into an appearance as if it were captured from a position directly facing the ground. The image obtained by the conversion is called an orthoimage, and the conversion process to obtain an orthoimage is called orthorectification. By replacing the map in Patent Document 1 with a front view of a structure and using orthorectification, it is possible to obtain an image that can be used to inspect infrastructure structures.

Patent No. 4112077

However, when the surface of a structure is curved, such as the wall of an arch dam, the ortho-image obtained by the above-mentioned orthorectification process will contain distortion. In such cases, instead of applying the orthorectification process to the entire captured image, an image with less distortion can be obtained by applying the orthorectification process to each partial region within the captured image. However, it is necessary to repeatedly match the front view with the captured image for each partial region, making this a time-consuming process.

The present invention was made in consideration of such problems, and aims to provide a technology that enables the specification of a partial area to be processed with simpler operations.

In order to solve the above-mentioned problems, the image processing device according to the present invention has the following arrangement.
a first designation means for designating a first partial region in a first image in which a structure of an object is drawn according to a predetermined coordinate system;
a second designation means for designating a second partial region corresponding to the first partial region in a second image including an image obtained by photographing the object;
a conversion means for converting an image of the second partial region into an image conforming to the predetermined coordinate system based on coordinates of the first partial region in the first image and coordinates of the second partial region in the second image;
having
the first image is a design drawing of the object;
the second image is a photographed image obtained by photographing the object,
the first designation means is configured to further designate a third partial region in the first image, the third partial region being different from the first partial region;
The first designation means designates the third partial region based on the first partial region.
The image processing device includes:
a first designation means for designating a first partial region in a first image in which a structure of an object is drawn according to a predetermined coordinate system;
a second designation means for designating a second partial area corresponding to the first partial area in a second image including an image obtained by photographing the object;
a receiving means for receiving a correction from a user to at least one of the first partial region and the second partial region;
a conversion means for converting an image of the second partial region into an image conforming to the predetermined coordinate system based on coordinates of the first partial region in the first image and coordinates of the second partial region in the second image;
having
the first designation means is configured to further designate a third partial region in the first image, the third partial region being different from the first partial region;
The first designation means designates the third partial region based on the first partial region.
The image processing device includes:
a first designation means for designating a first partial region in a first image in which a structure of an object is drawn according to a predetermined coordinate system;
a second designation means for designating a second partial region corresponding to the first partial region in a second image including an image obtained by photographing the object;
a conversion means for converting an image of the second partial region into an image conforming to the predetermined coordinate system based on coordinates of the first partial region in the first image and coordinates of the second partial region in the second image;
having
the first partial region and the second partial region are each specified by an equal number of reference points;
the first designation means is configured to further designate a third partial region in the first image, the third partial region being different from the first partial region;
The first designation means designates the third partial region based on the first partial region.
Alternatively, the image processing device includes:
a first designation means for designating a first partial region in a first image in which a structure of an object is drawn according to a predetermined coordinate system;
a second designation means for designating a second partial region corresponding to the first partial region in a second image including an image obtained by photographing the object;
a conversion means for converting an image of the second partial region into an image conforming to the predetermined coordinate system based on coordinates of the first partial region in the first image and coordinates of the second partial region in the second image;
having
the first image is a design drawing of the object;
the second image is a photographed image obtained by photographing the object,
the first designation means is configured to sequentially designate first partial regions different from one another in the first image;
The first designation means designates a (k+1)th first partial region based on the kth first partial region designated.
The image processing device includes:
a first designation means for designating a first partial region in a first image in which a structure of an object is drawn according to a predetermined coordinate system;
a second designation means for designating a second partial region corresponding to the first partial region in a second image including an image obtained by photographing the object;
a receiving means for receiving a correction from a user to at least one of the first partial region and the second partial region;
a conversion means for converting an image of the second partial region into an image conforming to the predetermined coordinate system based on coordinates of the first partial region in the first image and coordinates of the second partial region in the second image;
having
the first designation means is configured to sequentially designate first partial regions different from one another in the first image;
The first designation means designates a (k+1)th first partial region based on the kth first partial region designated.
The image processing device includes:
a first designation means for designating a first partial region in a first image in which a structure of an object is drawn according to a predetermined coordinate system;
a second designation means for designating a second partial region corresponding to the first partial region in a second image including an image obtained by photographing the object;
a conversion means for converting an image of the second partial region into an image conforming to the predetermined coordinate system based on coordinates of the first partial region in the first image and coordinates of the second partial region in the second image;
having
the first partial region and the second partial region are each specified by an equal number of reference points;
the first designation means is configured to sequentially designate first partial regions different from one another in the first image;
The first designation means designates a (k+1)th first partial region based on the kth first partial region designated.

The present invention provides a technology that allows the specification of a partial area to be processed with simpler operations.

1 is a diagram illustrating a hardware configuration of an image processing apparatus according to a first embodiment. FIG. 10 is a diagram illustrating an example of a screen displayed on a display device. 1 is a diagram illustrating a functional configuration of an image processing apparatus according to a first embodiment. 11A and 11B are diagrams illustrating a sliding operation of corresponding points. 10A and 10B are diagrams illustrating an example of a reference point table and a corresponding point table. 1 is a flowchart of an orthorectification process in the first embodiment. 13 is a detailed flowchart of a corresponding point designation process (S602). 13 is another detailed flowchart of the corresponding point designation process (S602). FIG. 13 is a diagram illustrating generation of auxiliary lines. FIG. 11 is a diagram illustrating a functional configuration of an image processing device according to a second embodiment. 13 is a flowchart of an orthorectification process in the second embodiment. FIG. 11 is a diagram illustrating a functional configuration of an image processing device according to a third embodiment. 13 is a flowchart of an orthorectification process in the third embodiment. FIG. 10 is a diagram illustrating an example of a screen displayed on a display device.

The following embodiments are described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe multiple features, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicate explanations are omitted.

First Embodiment
As a first embodiment of the image processing device according to the present invention, an image processing device that inputs a drawing of a structure as an object and a photographed image of the structure and generates an orthoimage based on the photographed image will be described below. When a structure cannot be photographed from a position facing the structure, a photographing technique in which the optical axis of the imaging device is tilted with respect to the structure is sometimes called tilted photography. In addition, a process of converting an image of a subject photographed from a tilted angle into an appearance when photographed from a position facing the subject is sometimes called tilt correction. Hereinafter, the expression "generation of an orthoimage (orthorectification)" in this specification does not depend on a specific conversion technique, but includes a general process of converting an image photographed with the optical axis of the imaging device tilted with respect to the subject and reproducing the appearance from a position facing the subject.

<Device Configuration>
FIG. 1 is a diagram showing the hardware configuration of an image processing apparatus according to the first embodiment.

The central processing unit (CPU) 101 is responsible for controlling the computer system. Based on a control program, the CPU 101 performs calculations and processing of information and controls each piece of hardware to realize each functional configuration and process described below. The random access memory (RAM) 102 functions as the main memory of the CPU 101, and as a work memory required for loading and executing programs. The read-only memory (ROM) 103 records a control program that defines the operation processing procedure of the CPU 101. The ROM 103 includes a program ROM that records the operating system (OS), which is a system program that controls the equipment of the computer system, and a data ROM that records information required to operate the system. In some cases, the HDD 107 described below is used instead of the ROM 103.

The network interface (NETIF) 104 controls the input and output of data sent and received via the network. The display device 105 is, for example, a CRT display or a liquid crystal display. The input device 106 is, for example, a touch panel, a keyboard, a mouse, etc., for receiving operation instructions from the user. The hard disk drive (HDD) 107 is a storage device. The HDD 107 is used to store data such as application programs. The input/output bus 108 is a bus (address bus, data bus, and control bus) for connecting the above-mentioned units.

Figure 2 is a diagram showing an example of a screen displayed on a display device. The screen is a graphical user interface (GUI) that provides information to a user and accepts input from the user, and is displayed on the display device under display control by an image processing device.

The screen includes a drawing load button 201, an image load button 202, a drawing display area 203, a captured image display area 204, an orthorectification instruction button 207, and an application exit button 208.

The drawing load button 201 is a button for instructing the loading of a drawing. Here, a drawing is, for example, a design drawing (such as a front view) of a structure drawn according to a specified coordinate system. It may be vector image data including dimensional information of the structure as used in CAD software. Vector image data is image data in a format in which figures such as lines, circles, and polygons express an image using vector information. Note that raster image data may also be used as a drawing. Raster image data is image data in a format in which an image is expressed using an array of pixels that represent color and density. The loaded drawing is then displayed as an image 205 in the drawing display area 203.

The image load button 202 is a button for instructing the loading of a captured image. Here, a captured image is, for example, an image obtained by capturing an image of a structure corresponding to the above-mentioned drawing. Generally, a captured image is raster image data. The loaded captured image is then displayed as image 206 in the captured image display area 204.

In image 205, reference points 209 to 212 shown as squares (□) are reference points arranged on the drawing. In image 206, reference points 213 to 216 shown as squares are reference points arranged on the photographed image. Among reference points 209 to 212 and reference points 213 to 216, pairs of reference points with the same numbers in the squares represent the same position of the structure. Such pairs of reference points are hereinafter referred to as corresponding point pairs. That is, in FIG. 2, four corresponding point pairs are illustrated: reference points 209 and 213, reference points 210 and 214, reference points 211 and 215, and reference points 212 and 216. Note that here, reference points on the drawing or photographed image that represent the same position on the structure are represented by numbers in squares, but corresponding point pairs may be represented by assigning colors or symbols. Alternatively, if it is obvious to the user, the correspondence relationship between the pairs may not be clearly indicated on the GUI.

Figure 3 is a diagram showing the functional configuration of the image processing device according to the first embodiment. The drawing input unit 301 accepts input of a drawing. The drawing display unit 302 displays the drawing input by the drawing input unit 301 in the drawing display area 203. The captured image input unit 303 accepts input of a captured image. The captured image display unit 304 displays the image input by the captured image input unit 303 in the captured image display area 204.

The reference point input unit 305 accepts coordinate input of a reference point via the input device 106 for the drawing displayed in the drawing display area 203 and the captured image displayed in the captured image display area 204. The reference point management unit 306 holds the reference point input by the reference point input unit 305.

When inputting a reference point on a drawing, if the input reference point falls within a predetermined range centered on a feature point (a part that is easy to position, such as an intersection of line segments) included in the drawing, the coordinates of the input reference point may be replaced with the coordinates of the feature point. Also, when a pointer such as a mouse cursor displayed on the drawing enters a predetermined range centered on a feature point, the pointer may be moved to the feature point. Also, as a feature point, not only the intersection of line segments but also the outline of any figure included in the drawing or the intersection of the outlines of any figures may be used. When the coordinates of the reference point or the pointer are corrected, visual, auditory, or tactile feedback may be given to the user to clearly indicate that the coordinates have been corrected. As visual feedback, methods such as changing the color of the reference point or the pointer, blinking the reference point or the pointer, or changing the size of the reference point or the pointer when the coordinates are corrected are possible. As auditory feedback, methods such as sounding a sound when the coordinates are corrected are possible. As tactile feedback, methods such as vibrating the input device 106 when the coordinates are corrected are possible.

The orthorectification unit 307 performs orthorectification processing on the captured image. It receives reference points on the drawing and the captured image, specified by the reference point input unit 305, as input for correction parameters in the orthorectification processing. The orthoimage output unit 308 outputs an image that is the result of the orthorectification processing by the orthorectification unit 307. The output image is stored in the HDD 107. It may be displayed on the display device 105.

The corresponding point sliding unit 309 accepts a user operation on a partial area surrounded by multiple corresponding point pairs, and updates the partial area of interest (i.e., multiple corresponding point pairs). Specifically, it determines the direction and amount of movement of the multiple corresponding point pairs based on the user operation, and updates the information on the reference points that make up each of the multiple corresponding point pairs. A user operation is, for example, an operation of moving a group of reference points arranged on the image 205 of the drawing in an approximately translational manner by operating a directional key on the keyboard or dragging the mouse.

Figure 4 is a diagram for explaining the sliding operation of the corresponding points. In FIG. 4, an example is shown in which six rectangular areas surrounded by joints exist in a row. FIG. 4(a) is an image diagram before the sliding operation of the corresponding points, and four corresponding point pairs are illustrated. Meanwhile, FIG. 4(b) is an image diagram after the sliding operation of the corresponding points, and four updated corresponding point pairs are illustrated. Reference points 401 to 404 are a reference point group that defines a partial area in the drawing after the sliding operation, and reference points 405 to 408 are a reference point group that defines a partial area in the captured image after the sliding operation. In other words, it is shown that the four corresponding point pairs in FIG. 4(a) are updated to the positions of the four corresponding point pairs in FIG. 4(b) by the sliding operation of the corresponding points. In other words, in this embodiment, orthorectification is sequentially performed on partial areas formed by four or more reference points. In such processing, a partial area to be used in the (k+1)th processing is specified based on a partial area formed by four or more reference points specified by the user in the kth processing. Here, k is an integer of 1 or more.

Figure 5 shows an example of a reference point table and a corresponding point table. Figure 5(a) is a drawing reference point table that stores the coordinates of reference points in a drawing, and Figure 5(b) is a captured image reference point table that stores the coordinates of reference points in a captured image. Here, an example is shown in which four reference points are stored in each of the drawing and the captured image. That is, an example is shown in which the partial areas of interest in each of the drawing and the captured image are rectangular. Also, Figure 5(c) is a corresponding point table that stores corresponding point pairs (pairs of reference points in the drawing and reference points in the captured image).

For example, the drawing reference point table before the slide operation stores the coordinates of the four reference points 209 to 212 shown in FIG. 4(a) as the four reference points with IDs "1" to "4". The captured image reference point table before the slide operation stores the coordinates of the four reference points 213 to 216 shown in FIG. 4(a) as the four reference points with IDs "1" to "4". And the corresponding point table before the slide operation stores pairs of reference points with the same numbers in the squares in FIG. 4(a) as the four corresponding point pairs with IDs "1" to "4".

In FIG. 5, four IDs ("1" to "4") are assigned to corresponding point pairs in a partial region of interest, and when the partial region of interest is changed, the coordinates of the reference points for the four IDs are updated. However, when the partial region of interest is changed, different IDs may be added and stored in the table one by one. For example, the first partial region may be assigned IDs "1" to "4," and the second partial region may be assigned IDs "5" to "6."

<Device Operation>
6 is a flowchart of the orthorectification process in the first embodiment. The process is started by, for example, the CPU 101 executing an application stored in the HDD 107.

In S601, the drawing input unit 301 and the captured image input unit 303 accept the input of a drawing and a captured image, respectively. Then, the graphic display unit 302 and the captured image display unit 304 display the drawing and the captured image, respectively.

In S602, the reference point input unit 305 accepts input of multiple corresponding point pairs for the drawing and captured image input in S601. Details of S602 will be described later with reference to FIG. 7 and FIG. 8. In S603, the corresponding point sliding unit 309 determines whether a sliding operation of multiple corresponding point pairs has been input. Here, a sliding operation is intended to be an operation to change a partial area of interest. If input, the process proceeds to S604. If not, the process proceeds to S605. In S604, the corresponding point sliding unit 309 slides multiple corresponding point pairs. That is, it slides the reference point group in the drawing and the reference point group in the captured image, respectively.

Here, the direction in which the corresponding point pairs are slid is obtained from the input device 106. For example, when a keyboard is used as the input device, the direction of the input arrow key is set as the direction in which the corresponding point pairs are slid. The amount of movement in which the corresponding point pairs are slid is calculated as follows. First, a bounding box (circumscribed rectangle) is calculated for each of the reference point group in the drawing (reference points 209-211) and the reference point group in the captured image (reference points 213-216). Then, if the slide direction is vertical, the "height" of the bounding box is set as the amount of movement in which the corresponding point pairs are slid, and if the slide direction is horizontal, the "width" of the bounding box is set as the amount of movement in which the corresponding point pairs are slid. In other words, the area adjacent to the partial area currently focused on (in the kth time) is set as the partial area to be focused on next time (in the (k+1)th time).

In S605, the reference point input unit 305 determines whether or not a correction instruction has been received for the reference points included in the multiple corresponding point pairs input in S602 or slid in S604. This corresponds to a case where there is a misalignment between the reference points in the drawing and the reference points in the captured image included in the corresponding point pairs after sliding (they do not point to the same position in the structure) and the user manually corrects the misalignment. For example, a correction instruction for the reference point is received by dragging the reference point displayed on the drawing or the captured image. S605 allows the user to individually fine-correct the reference points included in the multiple corresponding point pairs slid in S604. If a correction instruction for the reference points included in the multiple corresponding point pairs has been received, proceed to S606. If not, proceed to S607. In S606, the reference point management unit 306 updates the corresponding point table when one or more reference points included in the multiple corresponding point pairs have been corrected.

In S607, the orthorectification unit 307 determines whether an orthorectification instruction has been input. If an orthorectification instruction has been input, the process proceeds to S608. If an orthorectification instruction has not been input, the process proceeds to S611. In S608, the orthorectification unit 307 calculates a coordinate transformation matrix for obtaining an orthoimage from the captured image input in S601. Here, it is assumed that a homography matrix is used as the coordinate transformation matrix. The homography matrix is a matrix used in homography transformation that maps (projects) an image coordinate system of one plane to an image coordinate system of another plane. A known algorithm such as the DLT (Direct Linear Transformation) method may be used to calculate the homography matrix. Note that the DLT method requires four or more corresponding point pairs to calculate the homography matrix.

In S609, the ortho-correction unit 307 performs conversion processing of the captured image input in S601 using the homography matrix obtained in S608. In S610, the ortho-image output unit 308 outputs the captured image converted in S609. In S611, the CPU 101 determines whether an instruction to end the application has been input. An instruction to end the application is input by pressing the application end button 208. If an instruction to end the application has been input, the processing ends. If not, the process returns to S603. The ortho-image generated by the above processing is used, for example, in a process to detect an abnormality such as a crack that has occurred on the surface of a structure. In this way, in this embodiment, the reference point group and partial area to be used in the (k+1)th processing are specified based on the user's specification operation in the kth processing, and the user can make corrections to the specified result. As a result, even if a user divides an image captured over a wide area into smaller parts and repeats orthorectification, after performing an operation to specify the coordinates of a specific reference point at least once, the user only needs to give a simple sliding instruction and make corrections if necessary. This reduces the operational burden on the user.

Figure 7 is a detailed flowchart of the corresponding point designation process (S602). Figure 7 explains a method for inputting four corresponding point pairs by designating four or more reference points on a drawing and then designating the same number of corresponding reference points on a captured image.

In S701, the reference point input unit 305 determines whether a reference point has been specified on the drawing. If a reference point has been specified, the process proceeds to S702. If not, the process proceeds to S703. In S702, the reference point input unit 305 adds the specified reference point to the drawing reference point table (FIG. 5A). In S703, the reference point input unit 305 determines whether there are four or more reference points stored in the drawing reference point table. If there are, the process proceeds to S704. If there are not, the process proceeds to S701.

In S704, the reference point input unit 305 determines whether a reference point has been specified on the captured image. If a reference point has been specified, the process proceeds to S705. If not, the process proceeds to S706. In S705, the reference point input unit 305 adds the specified reference point to the captured image reference point table (FIG. 5B). In S706, the reference point input unit 305 determines whether the number of reference points stored in the captured image reference point table is the same as the number of reference points stored in the drawing reference point table. If the number is the same, the process proceeds to S707. If they are different, the process proceeds to S704.

In S707, the reference point input unit 305 generates a corresponding point pair from the reference points stored in the drawing reference point table and the captured image reference point table. Specifically, first, reference points having the same ID are obtained from both tables. The obtained pair of reference points is then added to the corresponding point table ( FIG. 5( c )) as a corresponding point pair. Note that, although the reference points of the drawing are specified first in FIG. 7, the reference points of the captured image may be specified first.

Figure 8 is another detailed flowchart of the corresponding point designation process (S602). Figure 8 explains a method for inputting four corresponding point pairs by repeating the operation of designating a reference point on a drawing and then designating a corresponding reference point on a captured image four or more times.

In S801, the reference point input unit 305 determines whether a reference point has been specified on the drawing. If a reference point has been specified, the process proceeds to S802. If a reference point has not been specified, the process proceeds to S806. In S802, the reference point input unit 305 adds the reference point specified in S801 to the drawing reference point table (FIG. 5A). In S803, the reference point input unit 305 determines whether a reference point has been specified on the captured image. If a reference point has been specified, the process proceeds to S804. If a reference point has not been specified, the process proceeds to S803.

In S804, the reference point input unit 305 adds the reference point specified in S803 to the captured image reference point table (Figure 5 (b)).

In S805, the reference point input unit 305 generates corresponding point pairs from the reference points stored in the drawing reference point table and the captured image reference point table. Specifically, first, the last reference point in both tables is obtained. Next, the obtained pair of reference points is added to the corresponding point table (FIG. 5(c)) as a corresponding point pair. Note that, although the corresponding point pairs are associated in S707 of FIG. 7 and S805 of FIG. 8 based on the order (ID) in which the reference points were specified, this is not limiting. For example, once the same number of reference points have been specified on the drawing and the captured image, regardless of the order, processing may be performed to pair reference points that are close in relative position in each reference point group.

As described above, according to the first embodiment, by the operation described with reference to FIG. 6, multiple (e.g., four) corresponding point pairs that define a partial region are sequentially specified in a loop from S603 to S611, and an orthoimage of the partial region is generated. This makes it possible to obtain an orthoimage that corresponds to a desired region of the captured image. In particular, according to the first embodiment, multiple corresponding point pairs that correspond to a new partial region can be easily set by a sliding operation, making it possible to reduce the user's effort. In particular, when multiple rectangular regions surrounded by joints exist side by side in succession, as shown in FIG. 4, it is possible to significantly reduce the user's effort in setting a new partial region.

Here, the output process of the converted image in S610 will be further described. In this embodiment, the data of the converted captured image in S610 is stored in a storage area such as the RAM 102, and is presented to the user by being displayed on a display device.

Figure 14 shows one aspect of the graphical user interface shown in Figure 2. Elements already described in Figure 2 are numbered the same as in Figure 2, and detailed descriptions are omitted.

In FIG. 14(a), an image 1401 is displayed in the captured image display area 204. The image 1401 is a part of an image of a concrete wall surface to be inspected, and shows a continuous arrangement of rectangular areas surrounded by joints. However, in this embodiment, since the image is used to inspect the concrete wall surface, it is required that the image maintains a certain level of resolution after image processing such as orthorectification. In order to capture a wide area of the concrete wall surface at high resolution, a method may be used in which the concrete wall surface is divided into multiple parts, each divided part is photographed at high resolution, and a composite image is generated by stitching. In other words, the image 1401 may be an image obtained by stitching together captured images obtained by photographing the concrete wall surface in parts using an electronic camera platform and an imaging device.

In the drawing display area 203, drawing data corresponding to image 1401 is displayed as image 1402. FIG. 14(a) is an image diagram before a correction instruction is given, and reference points 209 to 216, which are four corresponding point pairs, are specified as the first operation (S602). When orthorectification is instructed by the user (Yes in S607), image conversion is executed (S608, S609). Then, the converted image is output (S610).

Figure 14(b) shows an example of a graphical user interface in which a newly converted image is displayed. However, in Figure 14(b), a sliding operation of the corresponding point pair has not yet been instructed. In the rectangular area of the drawing display area 203 surrounded by the designated points 209 to 212, the result of ortho-converting the image of the portion defined by the designated points 213 to 216 in the captured image display area 204 is superimposed and displayed. By displaying the converted image in this way, the user can check the result of the conversion. After checking the result, the user instructs a sliding operation of the corresponding points for the next process.

Figure 14(c) is a diagram showing the corresponding points after the slide operation, and shows the four updated corresponding point pairs. Reference points 1404 to 1407 are a group of reference points that define a partial area in the drawing after the slide operation, and reference points 1408 to 1411 are a group of reference points that define a partial area in the captured image after the slide operation.

In addition, in this embodiment, if the converted image is used as an input for the target image of the anomaly detection process by the detector, the position and width of anomalies (e.g. cracks) occurring in the structure over a wide area of the structure can be automatically estimated. A trained model based on machine learning is used as the detector. The trained model is generated by acquiring knowledge by machine learning using a number of sets of images of the structure as input data, pixels confirmed by humans to be anomalies (e.g. cracks) as output data, and the trained data. At this time, data in which the observer has corrected the detection results may be re-trained as teacher data. The trained model can be configured, for example, as a neural network model.

In this embodiment, cracks are detected using a neural network. The input to the neural network is 25 pixels x 25 pixels x RGB. There is one output, which is a value (a value between 0 and 1) that represents the likelihood of a crack in the center pixel of the 25 pixels x 25 pixels. Since neural networks are a well-known technology, a detailed explanation is omitted. Note that the abnormality detection process can be performed by the same image processing device as the orthorectification process described above, or by another information processing device that, together with the image processing device, constitutes an information processing system for abnormality detection.

In the first embodiment, the images that are output after coordinate transformation for each partial image specified by the corresponding point pair can also be synthesized based on their arrangement on the drawing display area 203 and the feature points extracted from each image.

Second Embodiment
In the second embodiment, a form will be described in which auxiliary lines are generated from characteristic lines such as joints captured in a captured image, and the auxiliary lines are used to set a plurality of corresponding point pairs corresponding to a new partial region.

Figure 9 is a diagram explaining the generation of auxiliary lines. Figure 9(a) is a diagram showing an example of a captured image before the generation of auxiliary lines. A characteristic line 901 of a joint (a gap or joint between components of a structure) is captured in an image 206 corresponding to the captured image. Figure 9(b) is a diagram showing the captured image on which the generated auxiliary line is superimposed. An auxiliary line 902 is an auxiliary line generated based on the characteristic line 901.

<Device Configuration>
10 is a diagram showing the functional configuration of an image processing apparatus according to the second embodiment. An auxiliary line generating unit 1001 generates auxiliary lines from characteristic lines such as joints captured in a captured image by image recognition or the like. Note that the configuration may also be such that the auxiliary lines are manually input by the user. As described above, the generated auxiliary lines are used to correct the reference points included in the corresponding point pair after the slide operation.

<Device Operation>
11 is a flowchart of the orthorectification process in the second embodiment. Each step of S601 to S611 is the same as that in FIG. 6, so the description will be omitted.

In S1101, the auxiliary line generating unit 1001 extracts characteristic lines such as joints captured in the captured image and generates auxiliary lines. The Canny method used for edge detection or the Hough transform used for feature extraction of straight lines can be used to extract characteristic lines such as joints. In S1102, the reference point input unit 305 corrects the reference points included in the new multiple corresponding point pairs provisionally specified by the slide operation based on the auxiliary lines and their intersections. For example, if the reference point included in the corresponding point pair is within a predetermined range centered on the intersection of the auxiliary lines, the coordinates of the reference point are replaced with the coordinates of the intersection. Also, if the reference point included in the corresponding point pair is within a predetermined range from the auxiliary lines, the coordinates of the reference point may be corrected based on the intersection of a perpendicular line dropped from the reference point to the auxiliary line with the auxiliary line.

As described above, according to the second embodiment, when setting the reference point to be used in the (k+1)th iteration based on the user's specification operation in the kth processing, an auxiliary line is generated and used based on the characteristic lines of joints and the like that appear in the captured image. This makes it possible to reduce the user's effort in correcting the misalignment between the reference point in the drawing included in the corresponding point pair and the reference point in the captured image.

Third Embodiment
In the third embodiment, a form will be described in which a plurality of corresponding point pairs in a new partial region are calculated using the transformation matrix used in the orthorectification process for a previously processed partial region.

<Device Configuration>
12 is a diagram showing the functional configuration of an image processing apparatus according to the third embodiment. A photographed image reference point calculation unit 1201 calculates a reference point group in a photographed image corresponding to a new partial region based on a reference point group in a drawing corresponding to the new partial region. Specifically, the reference point group in the photographed image is calculated from the reference point group in the drawing by using a transformation matrix used in the orthorectification process for the partial region previously processed.

<Device Operation>
13 is a flowchart of the orthorectification process in the third embodiment. Each step of S601 to S611 is the same as that in FIG. 6, so the description will be omitted.

In S1301, the corresponding point sliding unit 309 slides only the group of multiple reference points in the drawing. The direction and amount of sliding may be the same as in the first embodiment (S604). In S1302, the corresponding point sliding unit 309 corrects the group of reference points in the drawing calculated in S1301 based on the intersections of line segments included in the drawing. Note that the correction of the reference points can be performed not only for the intersections of line segments, but also for the contours of any shapes included in the drawing or the intersections of the contours of any shapes.

In S1303, the captured image reference point calculation unit 1201 applies the transformation matrix calculated in S608 for the previous partial region (the previous loop of S603 to S611) to the reference point group in the drawing obtained in S1301 and S1302. This allows the reference point group in the captured image to be obtained. For example, it is assumed that the transformation matrix calculated in S608 for the partial region in the previous (kth loop) was H _(k) . It is also assumed that the reference point group in the drawing of the partial region in this ((k+1)th loop) is _pdn(k+1) . In this case, the reference point group p _{in(k+1) in the captured image to be newly set this time ((k+1} )th loop) can be obtained by the following formula (1). Note that n is an ID, and here n=1, 2, 3, 4. It is also assumed that H _(k) ^-1 is the inverse matrix of H _(k) .
p _in(k+1) = H _(k) ^{- 1} p _dn(k+1) ... (1)

The reference point group in the captured image calculated by formula (1) may be further corrected using the auxiliary lines described in the second embodiment.

As described above, according to the third embodiment, multiple corresponding point pairs in a new partial region are calculated using the transformation matrix used in the orthorectification process for the previously processed partial region. This makes it possible to more accurately derive the coordinates of the reference point group in the captured image corresponding to the new partial region, reducing the user's effort in correcting the misalignment.

Other Examples
The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

The invention is not limited to the above-described embodiment, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to disclose the scope of the invention.

301 Drawing input section; 302 Drawing display section; 303 Photographed image input section; 304 Photographed image display section; 305 Reference point input section; 306 Reference point management section; 307 Ortho-correction section; 308 Ortho-image output section; 309 Corresponding point slide section

Claims (27)

a first designation means for designating a first partial region in a first image in which a structure of an object is drawn according to a predetermined coordinate system;
a second designation means for designating a second partial area corresponding to the first partial area in a second image including an image obtained by photographing the object;
a conversion means for converting an image of the second partial region into an image conforming to the predetermined coordinate system based on coordinates of the first partial region in the first image and coordinates of the second partial region in the second image;
having
the first image is a design drawing of the object;
the second image is a photographed image obtained by photographing the object,
the first designation means is configured to further designate a third partial region in the first image, the third partial region being different from the first partial region;
2. An image processing apparatus comprising: a first specifying unit for specifying the third partial area based on the first partial area. 2. The image processing apparatus according to claim 1 , further comprising a receiving unit for receiving a correction from a user to at least one of the first partial region and the second partial region. the object is a structure,
the second image is a captured image obtained by capturing an image of a wall surface of the structure,
3. The image processing apparatus according to claim 1 , wherein the image converted by the conversion means is used in a process for detecting abnormalities occurring on a wall surface of the structure. 4. The image processing apparatus according to claim 3 , wherein the second image is an image obtained by synthesizing a plurality of images obtained by photographing different areas of a wall surface of the structure. a first designation means for designating a first partial region in a first image in which a structure of an object is drawn according to a predetermined coordinate system;
a second designation means for designating a second partial area corresponding to the first partial area in a second image including an image obtained by photographing the object;
a receiving means for receiving a correction from a user to at least one of the first partial region and the second partial region;
a conversion means for converting an image of the second partial region into an image conforming to the predetermined coordinate system based on coordinates of the first partial region in the first image and coordinates of the second partial region in the second image;
having
the first designation means is configured to further designate a third partial region in the first image, the third partial region being different from the first partial region;
2. An image processing apparatus comprising: a first specifying unit for specifying the third partial area based on the first partial area. 6. The image processing apparatus according to claim 1, wherein the first partial area and the second partial area are each specified by an equal number of reference points. 7. The image processing device according to claim 6, wherein the conversion means calculates a homography matrix that performs projective transformation of an image coordinate system of the second image into the predetermined coordinate system based on the coordinates of the plurality of reference points, and converts the image of the second partial region into an image conforming to the predetermined coordinate system using the calculated homography matrix. a first designation means for designating a first partial region in a first image in which a structure of an object is drawn according to a predetermined coordinate system;
a second designation means for designating a second partial area corresponding to the first partial area in a second image including an image obtained by photographing the object;
a conversion means for converting an image of the second partial region into an image conforming to the predetermined coordinate system based on coordinates of the first partial region in the first image and coordinates of the second partial region in the second image;
having
the first partial region and the second partial region are each specified by an equal number of reference points;
the first designation means is configured to further designate a third partial region in the first image, the third partial region being different from the first partial region;
2. An image processing apparatus comprising: a first specifying unit for specifying the third partial area based on the first partial area. 9. The image processing device according to claim 8, wherein the conversion means calculates a homography matrix that performs projective transformation of an image coordinate system of the second image into the predetermined coordinate system based on the coordinates of the plurality of reference points, and converts the image of the second partial region into an image conforming to the predetermined coordinate system using the calculated homography matrix. a derivation means for deriving a coordinate transformation matrix based on coordinates of the first partial region in the first image and coordinates of the second partial region in the second image,
10. The image processing device according to claim 1, wherein the conversion means converts the image of the second partial region into an image conforming to the predetermined coordinate system based on the coordinate transformation matrix derived by the derivation means. 11. The image processing device according to claim 1, wherein the first designation means designates, as the third partial region, a region obtained by translating the first partial region and adjacent to the first partial region. the second designation means is configured to further designate a fourth partial region corresponding to the third partial region in the second image;
12. The image processing apparatus according to claim 1, wherein the second designation means designates the fourth partial region on the basis of the second partial region. The method further comprises: detecting means for detecting a characteristic line of the object in the second image;
13. The image processing device according to claim 12, wherein the second designation means provisionally designates the fourth partial area based on the coordinates of the second partial area, and corrects the provisionally designated fourth partial area based on the characteristic line detected by the detection means. 14. The image processing apparatus according to claim 13 , further comprising a providing unit that provides a visual, audio or tactile feedback to the user when the correction is made by the second designation unit. the second designation means is configured to further designate a fourth partial region corresponding to the third partial region in the second image;
The image processing device according to claim 12 when dependent on claim 10, characterized in that the second designation means designates the fourth partial area based on coordinates derived based on the inverse matrix of the coordinate transformation matrix derived by the derivation means and the coordinates of the third partial area. 16. The image processing apparatus according to claim 1, further comprising a display control unit that causes a display device to display the image of the converted second partial region superimposed on the first partial region. 17. The image processing apparatus according to claim 16 , wherein the display control means causes the display device to display the first image and the second image side by side. a first designation means for designating a first partial region in a first image in which a structure of an object is drawn according to a predetermined coordinate system;
a second designation means for designating a second partial region corresponding to the first partial region in a second image including an image obtained by photographing the object;
a conversion means for converting an image of the second partial region into an image conforming to the predetermined coordinate system based on coordinates of the first partial region in the first image and coordinates of the second partial region in the second image;
having
the first image is a design drawing of the object;
the second image is a photographed image obtained by photographing the object,
the first designation means is configured to sequentially designate first partial regions different from one another in the first image;
The image processing apparatus according to claim 1, wherein the first designation means designates a (k+1)th first partial area based on a kth first partial area designated. a first designation means for designating a first partial region in a first image in which a structure of an object is drawn according to a predetermined coordinate system;
a second designation means for designating a second partial area corresponding to the first partial area in a second image including an image obtained by photographing the object;
a receiving means for receiving a correction from a user to at least one of the first partial region and the second partial region;
a conversion means for converting an image of the second partial region into an image conforming to the predetermined coordinate system based on coordinates of the first partial region in the first image and coordinates of the second partial region in the second image;
having
the first designation means is configured to sequentially designate first partial regions different from one another in the first image;
The image processing apparatus according to claim 1, wherein the first designation means designates a (k+1)th first partial area based on a kth first partial area designated. a first designation means for designating a first partial region in a first image in which a structure of an object is drawn according to a predetermined coordinate system;
a second designation means for designating a second partial area corresponding to the first partial area in a second image including an image obtained by photographing the object;
a conversion means for converting an image of the second partial region into an image conforming to the predetermined coordinate system based on coordinates of the first partial region in the first image and coordinates of the second partial region in the second image;
having
the first partial region and the second partial region are each specified by an equal number of reference points;
the first designation means is configured to sequentially designate first partial regions different from one another in the first image;
The image processing apparatus according to claim 1, wherein the first designation means designates a (k+1)th first partial area based on a kth first partial area designated. A program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 20 . A method for controlling an image processing device that generates an image according to a predetermined coordinate system, comprising the steps of:
a first designation step of designating a first partial region in a first image in which a structure of an object is drawn according to the predetermined coordinate system;
a second designation step of designating a second partial region corresponding to the first partial region in a second image including an image obtained by photographing the object;
a conversion step of converting an image of the second partial region into an image conforming to the predetermined coordinate system based on coordinates of the first partial region in the first image and coordinates of the second partial region in the second image;
Including,
the first image is a design drawing of the object;
the second image is a photographed image obtained by photographing the object,
In the first designation step, a third partial region different from the first partial region is further designated in the first image,
2. A method for controlling an image processing apparatus, comprising: specifying the third partial area based on the first partial area in the first specifying step. A method for controlling an image processing device that generates an image according to a predetermined coordinate system, comprising the steps of:
a first designation step of designating a first partial region in a first image in which a structure of an object is drawn according to the predetermined coordinate system;
a second designation step of designating a second partial region corresponding to the first partial region in a second image including an image obtained by photographing the object;
a receiving step of receiving a correction from a user for at least one of the first partial region and the second partial region;
a conversion step of converting an image of the second partial region into an image conforming to the predetermined coordinate system based on coordinates of the first partial region in the first image and coordinates of the second partial region in the second image;
Including,
In the first designation step, a third partial region different from the first partial region is further designated in the first image,
2. A method for controlling an image processing apparatus, comprising: specifying the third partial area based on the first partial area in the first specifying step. A method for controlling an image processing device that generates an image according to a predetermined coordinate system, comprising the steps of:
a first designation step of designating a first partial region in a first image in which a structure of an object is drawn according to the predetermined coordinate system;
a second designation step of designating a second partial region corresponding to the first partial region in a second image including an image obtained by photographing the object;
a conversion step of converting an image of the second partial region into an image conforming to the predetermined coordinate system based on coordinates of the first partial region in the first image and coordinates of the second partial region in the second image;
Including,
the first partial region and the second partial region are each specified by an equal number of reference points;
In the first designation step, a third partial region different from the first partial region is further designated in the first image,
2. A method for controlling an image processing apparatus, comprising: specifying the third partial area based on the first partial area in the first specifying step. An information processing system for detecting a deformation occurring on a wall surface of a structure,
a first designation means for designating a first partial region in a first image in which the structure is rendered according to a predetermined coordinate system;
a second designation means for designating a second partial area corresponding to the first partial area in a second image including a captured image obtained by capturing an image of a wall surface of the structure;
a conversion means for converting an image of the second partial region into an image conforming to the predetermined coordinate system based on coordinates of the first partial region in the first image and coordinates of the second partial region in the second image;
A detection means for detecting a defect occurring on a wall surface of the structure from the image converted by the conversion means;
having
the first image is a design drawing of the structure;
the second image is a photographed image obtained by photographing the structure,
the first designation means is configured to further designate a third partial region in the first image, the third partial region being different from the first partial region;
An information processing system, wherein the first designation means designates the third partial region based on the first partial region. An information processing system for detecting a deformation occurring on a wall surface of a structure,
a first designation means for designating a first partial region in a first image in which the structure is rendered according to a predetermined coordinate system;
a second designation means for designating a second partial area corresponding to the first partial area in a second image including a captured image obtained by capturing an image of a wall surface of the structure;
a receiving means for receiving a correction from a user to at least one of the first partial region and the second partial region;
a conversion means for converting an image of the second partial region into an image conforming to the predetermined coordinate system based on coordinates of the first partial region in the first image and coordinates of the second partial region in the second image;
A detection means for detecting a defect occurring on a wall surface of the structure from the image converted by the conversion means;
having
the first designation means is configured to further designate a third partial region in the first image, the third partial region being different from the first partial region;
An information processing system, wherein the first designation means designates the third partial region based on the first partial region. An information processing system for detecting a deformation occurring on a wall surface of a structure,
a first designation means for designating a first partial region in a first image in which the structure is rendered according to a predetermined coordinate system;
a second designation means for designating a second partial area corresponding to the first partial area in a second image including a captured image obtained by capturing an image of a wall surface of the structure;
a conversion means for converting an image of the second partial region into an image conforming to the predetermined coordinate system based on coordinates of the first partial region in the first image and coordinates of the second partial region in the second image;
A detection means for detecting a defect occurring on a wall surface of the structure from the image converted by the conversion means;
having
the first partial region and the second partial region are each specified by an equal number of reference points;
the first designation means is configured to further designate a third partial region in the first image, the third partial region being different from the first partial region;
An information processing system, wherein the first designation means designates the third partial region based on the first partial region.

Images