JP2021196705A - Image processing system, image processing method and program - Google Patents

Abstract

To appropriately designate a position of a correction point on a photographic image corresponding to a reference point of a facing surface in tilt correction.SOLUTION: An image processing system includes: detection means of detecting a line segment from a photographic image; determination means of determining whether or not the line segment detected by the detection means is a line segment showing the construction of a structure which is a subject of the photographic image; display control means of superposing information which is for setting a correction point on the photographic image corresponding to a reference point of a facing surface in tilt correction facing the photographic image and which emphasizes the line segment determined by the determination means to be a line segment showing the structure on the photographic image so as to be displayed; and fixing means of fixing a position of the correction point after the information displayed by the display control means is displayed.SELECTED DRAWING: Figure 4

Description

The present invention relates to image processing.

Due to the need for maintenance of infrastructure structures, technology for inspecting changes over time in infrastructure structures is required. Patent Document 1 discloses a technique of automatically detecting cracks in an image of the appearance of an infrastructure structure and extracting the shape of the cracks as a line.

In infrastructure structure inspection using captured images, images of structures taken from the opposite position are required in order to accurately record deformations such as cracks. However, due to topographical factors, it may not be possible to photograph the structure from the opposite position. In such a case, if the inspection surface that appears diagonally in the image is deformed (tilt correction) so that it faces up, and the image with the inspection surface facing up and the drawing are overlapped, the crack width and length are appropriate. Can be evaluated. It should be noted that the deformation process of making the surface appearing diagonally in the image face-to-face is also called orthophoto correction. In the orthophoto correction process, four or more corresponding points that are a pair of a reference point facing each other and a correction point of the captured image are set, and the deformation process is performed based on the corresponding points.

Japanese Unexamined Patent Publication No. 2000-002523

When performing the above-mentioned orthophoto correction processing, the accuracy of the orthophoto correction processing may be insufficient due to the inability to appropriately specify the position of the correction point on the captured image corresponding to the reference point facing the front.

For example, it may be difficult to determine the exact position of the correction point due to deterioration of the structure or hiding by other objects such as leaves.

The present invention has been made in view of the above-mentioned problems, and an object of the present invention is to make it possible to appropriately specify the position of a correction point on a captured image corresponding to a reference point facing each other.

As one means for solving the above problems, the image processing apparatus according to the present invention has the following configuration.

That is, the image processing apparatus according to the present invention is
A detection means that detects line segments from captured images,
A determination means for determining whether the line segment detected by the detection means is a line segment indicating the structure of the structure that is the subject of the captured image.
According to the determination means, it is information for setting a correction point on the photographed image corresponding to a reference point on the facing surface in the tilt correction for making the photographed image face-to-face, and is a line segment showing the structure. A display control means for superimposing and displaying information for emphasizing the determined line segment on the captured image, and
It is characterized by having a determination means for determining the position of the correction point after the information displayed by the display control means is displayed.

According to the present invention, it is possible to display information for appropriately designating the position of the correction point on the captured image corresponding to the reference point facing the front.

It is a figure which shows the hardware configuration of the image processing apparatus which concerns on this embodiment. It is a figure which shows the screen to display the image processing apparatus which concerns on this embodiment. It is a figure which shows the functional structure of the image processing apparatus which concerns on this embodiment. It is a flowchart which shows the processing of the image processing apparatus which concerns on this embodiment. It is a figure which illustrates the line segment extraction result from the photographed image. It is a figure which illustrates the straight line fitting result. It is a figure which illustrates the classification result of the straight line based on the tendency of inclination. It is a figure which exemplifies the combination / deletion processing result of a straight line. It is a figure which shows the linear information table which shows the value of each importance index and the final importance for each straight line. It is a figure which shows the display example of the information for setting a correction point. It is a flowchart which shows the processing in the modification of the image processing apparatus which concerns on this embodiment. It is a figure which shows the display example of the candidate position of a correction point. It is a figure which shows the display example of the candidate position of a correction point, and the information for setting a correction point. It is a figure which shows the update result of the candidate position of an undetermined correction point.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings, based on its preferred embodiments. The configurations shown in the following embodiments are merely examples, and the present invention is not limited to the configurations shown.

In this actual embodiment, the infrastructure structure inspection system will be described. In the infrastructure structure inspection system according to the present embodiment, the infrastructure structure is photographed, and deformation such as cracks in the structure is confirmed from the photographed image. In the infrastructure structure inspection system, the photographed image of the infrastructure structure, the front view of the infrastructure structure, the photographed image, and the corresponding points are set, orthophoto correction (tilt correction) is performed on the photographed image, and the corrected image (the corrected image). Create an ortho image). Of the corresponding points, those set on the front view are called reference points, and those set on the captured image are called correction points. The front view is a design drawing including dimensional information and the like. The infrastructure structure inspection system detects deformation such as cracks from the ortho image.

When inputting a correction point on a captured image, it may be difficult to determine the correction point corresponding to the reference point on the front view at an accurate position due to deterioration of the structure or hiding by other objects such as leaves. .. When the orthophoto correction process is performed based on such an inaccurate correction point, the deformation accuracy is inferior to that when the correction point is set at the correct position. Therefore, the infrastructure structure inspection system according to the present embodiment draws an auxiliary line on the captured image to encourage the setting of the correction point at an accurate position. The infrastructure structure inspection system detects the boundaries and edges of the building unit (concrete block, etc.) shown in the drawing, and superimposes auxiliary lines on the captured image along the detected boundaries and edges of the building unit. do.

FIG. 1 is a diagram showing a hardware configuration of an image processing device related to an infrastructure structure inspection system. Reference numeral 101 is a central arithmetic unit (hereinafter referred to as CPU) that controls the control of the computer system. The CPU 101 realizes each function configuration and processing described later by executing information calculation and processing and control of each hardware based on the control program.

Reference numeral 102 denotes a random access memory (hereinafter referred to as RAM), which functions as a main memory of the CPU 101 and as a work memory necessary for loading an execution program and executing the program. Reference numeral 103 is a read-only memory (hereinafter referred to as ROM) for recording a control program that defines an operation processing procedure of the CPU 101.

The ROM 103 includes a program ROM in which basic software (OS), which is a system program for controlling equipment of a computer system, is recorded, and a data ROM in which information necessary for operating the system is recorded. HDD 107, which will be described later, may be used instead of ROM 103. Reference numeral 104 is a network interface (hereinafter referred to as NETIF), which controls input / output of data transmitted / received via the network.

Reference numeral 105 is a display device, for example, a CRT display, a liquid crystal display, or the like. Reference numeral 106 is an input device for receiving an operation instruction from the user, such as a touch panel, a keyboard, and a mouse. Reference numeral 107 is a hard disk drive (HDD) and a storage device. HDD 107 is used for storing data such as application programs. The HDD 107 may be configured by a storage medium (solid state drive) (SSD) using a semiconductor element memory. Reference numeral 108 denotes an input / output bus (address bus, data bus, and control bus) for connecting the above-mentioned units.

Although the case where the infrastructure structure inspection system is realized by a single image processing device is illustrated in FIG. 1, the present invention is not limited to this. For example, an infrastructure structure inspection system may be configured by distributed processing by a plurality of image processing devices.

FIG. 3 is a diagram illustrating a functional configuration of the image processing apparatus 300 according to the present embodiment. The functional configuration shown in FIG. 3 is realized by loading a program stored in the ROM 103 or the HDD 107 of the image processing apparatus 300 into the RAM 102 and executing the program by the CPU 101.

The input unit 301 inputs an instruction from the user to the image processing device via the input device 106. The line segment extraction unit 302 extracts a line segment from the image input by the input unit 301. The line segment extraction unit 02 extracts a straight line considered to be the boundary of the concrete block of the construction unit by extracting the straight line component from the image.

The importance setting unit 303 sets the importance indicating the priority of whether or not to display each line segment extracted by the line segment extraction unit 302. The importance is also a degree indicating whether each of the extracted line segments is a line segment indicating the structure of the structure that is the subject of the captured image. The importance setting unit 303 determines the structure of the infrastructure structure that is the subject of the input captured image based on the set importance. That is, the importance setting unit 303 determines whether the line segment extracted by the line segment extraction unit 302 is a line segment indicating the structure of a structure such as the boundary of a concrete block of a construction unit.

The correction information setting unit 304 performs settings and calculations necessary for performing orthophoto correction. The correction information setting unit 304 acquires the coordinates of the correction point on the captured image corresponding to the reference point in the front view. The correction information setting unit 304 sets the correction point based on the input of the user. The correction information setting unit 304 calculates and sets the image correction parameter from the reference point on the front view and the correction point on the captured image. The correction control unit 305 performs orthophoto correction on the captured image according to the image parameters set by the correction information setting unit 304. The display control unit 306 controls the display device 105 to display various information.

It should be noted that each of the above-mentioned functional configurations may be realized by one or a plurality of ASICs or FPGAs (such as ASICs. ASIC is an abbreviation for application specific integrated circuit. FPGA is a field-programmable gate array. It is an abbreviation.

The screen displayed on the display device 105 by the display control unit 306 of the image processing device 300 will be described with reference to FIG. In FIG. 2, 201 is a drawing read button. When the input to the drawing reading button 201 is detected, the image processing apparatus 300 reads the front view of the structure.

202 is an image reading button. When the input to the image reading button 202 is detected, the image processing device 300 reads the captured image obtained by capturing the structure. 203 is a drawing display area. The image processing device 300 causes the drawing display area 203 to display the drawing read in response to the input to the drawing reading button 201. 204 is a photographed image display area. The image processing device 300 displays the captured image read in response to the input to the image reading button 202 in the image display area 204.

In FIG. 2, the drawing 205 is displayed in the drawing display area 203, and the captured image 206 is displayed in the image display area 204. The drawing is a vector image including dimensional information of a structure as used in CAD software. A vector image is an image format in which an image is represented by a combination of figures such as lines, circles, and polygons. A raster image may be used as the drawing. A raster image is an image format in which an image is represented by an array of pixels representing colors and densities.

Reference numeral 207 is an orthophoto correction instruction button. When the input to the orthophoto correction instruction button 207 is detected, the image processing apparatus 300 performs orthophoto correction on the captured image displayed in the image display area 204. 208 is an application end button.

The image processing device 300 can set a corresponding point between the drawing 205 displayed in the drawing display area 203 and the captured image 206. The image processing apparatus 300 displays the reference points (reference points 209 to 211) set in the drawing 205 in the drawing display area 203. The image processing apparatus 300 sets the correction points 213 to 215 corresponding to the reference points 209 to 211 set in the drawing 205 in the captured image 206.

Of the reference points 209 to 212 and the correction points 213 to 216, the pair having the corresponding numbers represents the same position of the structure. This pair is called a correspondence point. FIG. 2 illustrates four corresponding points, that is, reference point 209 and correction point 213, reference point 210 and correction point 214, reference point 211 and correction point 215, and reference point 212 and correction point 216. ing. It should be noted that the reference point on the drawing representing the same position on the structure and the correction point on the captured image may be assigned the same color or symbol to indicate that they correspond to each other. Further, the image processing device 300 displays a pointer 217 indicating the coordinate position of the instruction input by the user on the screen.

Subsequently, the processing flow of the image processing apparatus 300 will be described with reference to the flowchart shown in FIG. The processing of each step in the flowchart shown in FIG. 4 is realized by a program stored in the ROM 103 or HDD 107 of the image processing apparatus 300, information calculation and processing by execution by the CPU 101, and control of each hardware. It should be noted that a configuration may be configured in which a part or all of the processing of each step in the flowchart shown in FIG. 4 is realized by hardware such as one or a plurality of ASICs and FPGAs.

In S401, the image processing device 300 generates a drawing list dialog by pressing the drawing reading button 201, and acquires a drawing by selecting a drawing to be read through the dialog. The method of specifying the drawing to be read is not limited to this, and for example, the file name of the drawing including the path name may be input. That is, it suffices if the drawing to be read can be appropriately specified.

Further, in S401, the image processing apparatus 300 acquires an image in the same manner as in the case of the drawing by pressing the image reading button 202 provided by the input unit 301. In this embodiment, the image is acquired after the drawing is input, but if the image and the drawing can be acquired appropriately, the order of acquisition may be reversed, or the images may be acquired at the same time. good.

After setting the drawing, in S402, the image processing device 300 receives the designation of the reference point from the user in the correction information setting unit 304 on the drawing. FIG. 2 shows an example in which reference points 209, 210, 211, and 212 are designated. The image processing device 300 sets the position information of the reference point designated by the user and also records it in the storage device 307.

In this embodiment, for the sake of explanation, the reading of drawings and images is configured to input / output with the storage device 307 (corresponding to HDD 107 in FIG. 1), but through a network (corresponding to NETIF 104 in FIG. 1). It may be configured to perform input / output. Further, for input / output from the user such as designation of a reference point, a dedicated device may be used.

Next, in S403, the image processing device 300 extracts a line segment from the input image in the line segment extraction unit 302. The line segment extraction unit 302 extracts a line segment by performing an edge extraction process on the input image. FIG. 5 is an image diagram when a line segment is extracted from the captured image 206, and the extracted line segment is shown by a black line. In FIG. 5, the boundary and the crack of the concrete block are detected by the edge extraction process.

After the line segment extraction is completed, the image processing apparatus 300 analyzes the line segment in S404. The line segment analysis is a preprocessing for calculating the importance in S406, such as applying a straight line to the extracted line segment, determining noise of the extracted line segment, and removing noise. The image processing device 300 first applies a straight line to the extracted line segment. After fitting the straight line, the fitted straight line is replaced with the line segment extracted by S403. Since the processing for the subsequent line segments is the processing for the replaced straight line, the expression "straight line" is used instead of "line segment" in the following description.

For the fitting of a straight line, for example, a straight line extraction based on the Hough transform or a Hough transform can be used. FIG. 6 shows an example of straight line fitting. In FIG. 6, a straight line replaced with the detected line segment is shown by a dotted line. In FIG. 6, the lower left of the screen is the origin, the x-axis is taken from the left to the right of the screen, the y-axis is taken from the bottom to the top of the screen, and the angle θ is 0 ° in the x-axis direction and the counterclockwise direction is the positive direction. The following explanation will be given using the coordinate system to be used.

Let ρ be the distance from the origin of this coordinate system to the straight line. The points at both ends of the straight line are the points where x is the minimum value (or the point where y is the minimum value) in the (x, y) coordinates from the points voted in (θ, ρ) used for the Hough transform. And the point where x is the maximum value (or the point where y is the maximum value) are selected. Then, the intersections of the straight line of interest, the straight line of interest passing through these two points, and the straight line orthogonal to each other may be obtained. The y-axis may be set from the top to the bottom of the screen with the upper left of the screen as the origin. In this case, the positive direction may be defined clockwise with the angle θ being 0 ° in the x-axis direction.

Next, the image processing apparatus 300 joins and removes the straight lines. In the combination of straight lines, two or more straight lines that are considered to be the same straight line are combined. The image processing device 300 combines straight lines whose slope differences, shortest distances between the end points of the straight lines, and distances between the straight lines are less than a predetermined value into one straight line, and joins the straight lines before joining into one straight line. Replace with a straight line. In this embodiment, straight lines are classified based on the tendency of inclination, and straight lines are combined based on the classification result.

In FIG. 7, the classification result based on the tendency of inclination is illustrated. In FIG. 7, a straight line having the same inclination is shown by a thick line. In FIG. 7A, a straight line having an inclination of 68.0 ° <θ≤113.5 ° is shown as a straight line 701-710 as a thick line. In FIG. 7B, a straight line having an inclination of 22.5 ° <θ≤68.0 ° is shown as a straight line 801 to 805. In FIG. 7 (c), a straight line having an inclination of -22.5 ° <θ≤22.5 ° is shown as a straight line 901 to 909. In FIG. 7D, a straight line having an inclination of −67.5 ° <θ≤-22.5 ° is shown as a straight line 1001 to 1003.

In the present embodiment, the image processing apparatus 300 combines straight lines having an inclination difference of less than 0.5 degrees, a shortest distance between endpoints, and a distance between straight lines of less than 10 pixels. After that, a straight line having a length less than 1/3 of that of the longest straight line is regarded as noise and deleted. In FIG. 7, the longest straight line is a straight line 704.

FIG. 8 shows an example of the result of the straight line joining / deleting process. In FIGS. 8A to 8E, the gray dotted line is the extracted line segment, the gray straight line is the deleted line segment, and the thick black straight line is the straight line remaining as a result of the processing. In FIG. 8, the straight line 705 and the straight line 707 shown in FIG. 7 satisfied the coupling condition and were combined to form 711, but the straight line after the coupling satisfied the deletion condition and was deleted. Further, in FIG. 8, the straight lines 702, 703, 706, 708, 709, 710, 801, 804, 805, 904, 907, 1001, 1002, and 1003 shown in FIG. 7 are also deleted because they satisfy the deletion conditions. rice field.

An example of the result of the analysis processing of the line segment of S404 is shown in FIG. 8 (e). The image processing device 300 stores the start point coordinates and the end point coordinates of the straight line that remains undeleted as a result of the analysis process in the storage device 307 as a straight line information table.

Next, in S405, the image processing device 300 determines in the importance setting unit 303 whether the importance setting for each straight line is completed. At this point, the importance setting has not been completed yet, so the process proceeds to S406. If the importance setting is completed by the process described later, the process proceeds to S410.

In S406, the image processing apparatus 300 selects one straight line for which the importance is not set and calculates the importance. In this embodiment, a method of using an error at the time of straight line fitting, a straight line length, and a straight line regularity as an index of importance will be described. A straight line with a high index is likely to be a straight line indicating the structure of the structure that is the subject, such as the boundary of a concrete block. Therefore, the image processing apparatus 300 can determine whether or not the straight line indicates the structure of the structure based on the importance of the detected straight line.

First, a method of calculating the importance index E based on the error at the time of linear fitting will be described. When setting the correction point, the "straight line" constituting the structure described in the drawing such as the boundary line of the concrete block can be used as an effective auxiliary line. Therefore, in the present embodiment, "straightness" indicating the linearity of the detected line segment is adopted as the importance index E. Here, the average e of the distances between each point voted in (θ, ρ) of the straight line of interest by the Hough transform and the straight line of interest is calculated. The importance index E is calculated by the equation (1) using e.
E = 1 / (1 + e) ・ ・ ・ (1)

When all the points voted in (θ, ρ) are on the straight line of interest, E = 1.0, and the farther each point is from the straight line of interest, the closer to E = 0. In the case of this embodiment, since a straight line is applied to the detected line segment, the value of E becomes large when the detected line segment is originally a "straight line", and when a straight line is applied to a line segment other than the straight line, E The value becomes smaller. That is, among the straight lines shown in FIG. 8 (e), the value of E in the straight lines 701, 704, 901, 902, 903, 905, 906, 908, 909 becomes large, and the values of E in the straight lines 802 and 803 become large. It gets smaller.

Next, a method of calculating the importance index L based on the length of the straight line of interest will be described. The straight lines that make up the structure described in the drawings, such as the boundaries of concrete blocks, often have a certain length or longer. Therefore, in this embodiment, the length of a straight line is adopted as the importance index L. Here, let L be the ratio to the length of the longest straight line among the straight lines fitted in S404. That is, if the length of the straight line of interest is l and the length of the longest straight line is l _max , L is calculated by the equation (2).
L = l / l _max ... (2)

The closer the length of the straight line of interest is to the longest straight line, the closer to L = 1, and the shorter the length, the closer to L = 0. In the case of this embodiment, l _max is the length of the straight line 704.

Finally, a method of calculating the importance index I based on the regularity of the straight line will be described. The straight lines that make up the structures described in the drawings generally have regularity. In this embodiment, attention is paid to the slope of the line segment as regularity. That is, the more straight lines parallel to the straight line of interest, the larger the importance index I of the straight line of interest is set. It should be noted that the more straight lines that are similar to the slope of the straight line of interest, even if they are not completely parallel, the larger the importance index I of the straight line of interest may be set.

For (θ, ρ) used for the Hough transform of each straight line, the maximum number of ρ (that is, the maximum number of parallel lines) when θ is constant is defined as R _max . Assuming that the number of ρ for each straight line θ is n _ρ , I is calculated by Eq. (3).
I = n _ρ / R _max ... (3)

That is, when the maximum number of straight lines constituting parallel lines _{is large, R max} of the denominator becomes large, so when the number of straight lines having the same slope as the straight line of interest (= ρ) is small, I = 0. On the contrary, when the maximum number of straight lines constituting the parallel lines is small and the number of straight lines having the same slope as the straight line of interest is large, I = 1.0 is approached. In this embodiment, since there is only one pair of straight lines 802 and 803 that can be regarded as parallel, R _max = 2. Further, only the straight lines 802 and 803 have I = 1.0, and the other straight lines have I = 0.5. As described above, in the present embodiment, the importance is set based on whether or not the straight line corresponding to the detected line segment is a straight line that regularly exists in the structure.

Finally, the importance W of the straight line of interest is calculated by Eq. (4).
W = E × L × I ・ ・ ・ (4)

In S407, the importance index calculated in S406 and the importance are set in the straight line of interest. Specifically, the importance index E, the importance index L, the importance index I, and the importance W calculated from each straight line are stored in the cells of the corresponding columns of the straight line information table.

FIG. 9 shows an example of a linear information table. In FIG. 9, 1600 is a linear information table, 1601 is an ID storage column, 1602 is an X coordinate storage column of a start point, 1603 is a Y coordinate storage column of a start point, 1604 is an X coordinate storage table of an end point, and 1605 is a Y coordinate storage of the end point. It's a table. Further, 1606 to 1609 are an importance index E storage column, an importance index L storage column, an importance index I storage column, and an importance W storage column, respectively, and the corresponding importance is stored. In the present embodiment, by using these importance, it is possible to determine whether the extracted line segment is a line segment indicating the structure of the structure.

Next, when the importance setting unit 303 determines that the importance setting for each straight line is completed by S405, the process proceeds to S410 executed by the correction control unit 305.

The image processing device 300 accepts the designation of the correction point after setting the importance of each straight line. In S410, the image processing device 300 detects the candidate position of the correction point in the correction control unit 305. In the present embodiment, the position of the mouse pointer 217 is detected as a candidate position for the correction point.

Next, in S411, the image processing device 300 displays information for setting a correction point on the captured image corresponding to the reference point of the facing surface in the tilt correction. Specifically, the image processing device 300 selects a line segment to be presented to the user. Then, the image processing apparatus 300 highlights the selected line segment as an auxiliary line when setting the correction point. In the present embodiment, a straight line to be displayed is selected from the straight lines whose importance has been set in S407.

Here, the distance from the candidate position of the correction point and the importance set in S407 are used for setting the selection criterion. Specifically, a straight line passing through a region within a predetermined distance (for example, a radius of 20 pixels) from the position of the mouse pointer 217 is selected based on the coordinate information of the columns 1602 to 1605 with reference to the straight line information table 1600.

Next, a predetermined number (for example, two) of the selected straight lines are selected in descending order of importance based on the value of the importance W storage column 1609. The display of the straight line in S411 highlights the line segment on the captured image. For example, a straight line having a thicker line width may be superimposed and displayed on the corresponding line segment on the captured image, or a colored straight line may be superimposed and displayed on the corresponding line segment on the captured image. May be good.

Further, although the example in which the candidate position of the correction point is used for selecting the straight line to be displayed is shown, the straight line to be displayed may be selected by using only the importance. In this case, for example, all the straight lines whose importance W exceeds the threshold value (for example, 0.3) may be displayed. That is, the image processing apparatus 300 may be configured to display all the line segments determined to be the line segments indicating the structure of the structure, such as the boundaries of concrete blocks, based on the importance.

In S412, the image processing apparatus 300 detects whether or not a correction point confirmation command has been input, and if it is detected, the correction point candidate position detected in S410 is used as the correction point confirmation position. In the present embodiment, the correction point is confirmed by double-clicking the left mouse button. Of course, the method of determining the correction point is not limited to double-clicking the mouse, and if the correction point can be determined, it may be configured to be determined by another method. For example, it can be configured to display a menu including "determine correction point" by right-clicking and select "determine correction point" from the menu to confirm the correction point.

FIG. 10 is a diagram showing an example of displaying information for setting a correction point. A display example of the selected line segment is shown in. In FIG. 10, the light gray solid line indicates the concrete block boundary and cracks in the photographed image. In the figure, the black dotted line indicates an area within a predetermined distance from the mouse pointer. In the present embodiment, the black dotted line indicates the region 1701 having a radius of 20 pixels from the mouse pointer 217. That is, a straight line passing through the inside of the region 1701 having a radius of 20 pixels from the mouse pointer 217, and the straight line to the top two in terms of importance is displayed.

By utilizing the displayed straight line as an auxiliary line, it becomes easy to set a correction point corresponding to the reference point set on the concrete block boundary.

In S413, the image processing device 300 determines in the correction control unit 305 whether the positions of all the correction points have been determined, and if not, returns the process to S410 to execute S410 to S412. A state in which S410 to S412 are repeatedly processed while the mouse pointer is moving will be described with reference to FIG.

FIG. 10A is a display example when a straight line does not exist within a radius of 20 pixels from the mouse pointer 217. In FIG. 10A, since there is no straight line whose importance is set within a radius of 20 pixels from the mouse pointer 217, the straight line is not displayed.

FIG. 10B is a display example when the mouse pointer 217 is moving to set the correction point 213. In FIG. 10 (B), since only two straight lines 701 and 901 pass through the radius of 20 pixels from the mouse pointer 217, both the straight lines 701 and 901 are displayed.

FIG. 10C is a display example when the mouse pointer 217 is moving to set the correction point 216. In FIG. 10C, straight lines 802 and 902 are drawn within a radius of 20 pixels from the mouse pointer 217. Since only two lines pass through, both straight lines 802 and 902 are displayed. However, when the position further approaches the set position of the correction point 216, the result is as shown in (d).

FIG. 10D is another display example when the mouse pointer 217 is moving to set the correction point 216. In FIG. 10D, three straight lines 704, 802, and 902 pass through a radius of 20 pixels from the mouse pointer 217. When the importance W of each straight line is confirmed, it is a straight line 704 (importance = 0.490), a straight line 802 (importance = 0.140), and a straight line 902 (importance = 0.328). The two lines in descending order of importance are straight lines 704 and 902, and these straight lines are displayed. In FIG. 10D, the invisible straight line 802 is shown by a dotted line.

After that, when the mouse pointer 217 is further moved to set the correction point 216, the straight line 704 and the straight line 901 are displayed as shown in FIG. 10 (e).

In this way, visibility is improved by displaying only the straight line required by the user to set the correction point as an auxiliary line, and the user can obtain the information necessary to set the correction point at an appropriate position. It will be possible to provide to.

The explanation is returned to the flowchart shown in FIG. In S413, the image processing device 300 confirms in the correction control unit 305 whether or not the positions of all the correction points have been determined. In the present embodiment, the number of correction points (corresponding points) is four, but other numbers may be used. When it is confirmed that the positions of all the correction points have been confirmed, the process is transferred to S408.

In S408, the image processing apparatus 300 uses the four reference points 209, 210, 211, and 212 and the four correction points 213, 214, 215, and 216 corresponding to the correction information setting unit 304, and the conversion parameter of the projection conversion. Is calculated. The projective transformation is sometimes called a homography transformation.

Here, projective transformation is represented by the formula (5), the parameters 9 becomes bracts of _{(H 11, H 12, H} 13, H 21, H 22, H 23, H 31, H 32, H 33) ( Parameters s is canceled during the expansion of the expression). However, even if the matrix is multiplied by a constant, the same conversion is obtained, so it can be expressed by eight parameters. The eight parameters are obtained from four different sets of correspondence points.

In S409, the image processing device 300 writes the conversion parameters calculated in S408 to the storage device 307. Further, when the input to the orthophoto correction instruction button 207 is detected, the image processing apparatus 300 may perform orthophoto correction on the captured image using the conversion parameter calculated in S408.

As described above, in the present embodiment, the structure of the structure described in the drawing such as the boundary line of the concrete block is determined from the photographed image by using the index of importance, and the assistance along the structure of the structure is determined. Changed to display the line. Further, in the present embodiment, a line segment is extracted from the input image, a straight line is applied to the line segment, the importance is set to the applied straight line, and the importance is set high with the operation of selecting the correction point. It was configured to display a straight line. As a result, even if there is a structural difference or a difference in the shooting angle in the subject on the shot image when performing the orthophoto correction processing of the shot image, the auxiliary line necessary for inputting the correction point is displayed. Is possible.

Further, according to the present embodiment, for a structure whose situation may change due to a difference in building conditions such as site environment, design circumstances, design circumstances, a difference in deterioration status after construction, or a difference in conditions at the time of shooting. Then, it becomes possible to display the auxiliary line required for inputting the correction point.

Further, according to the present embodiment, even if the shooting image of the infrastructure structure may have a different shooting angle from the facing position each time, the straight line constituting the infrastructure structure such as the boundary of the concrete block is emphasized as an auxiliary line. Can be displayed.

Although the line segment extraction is performed by performing the edge extraction process in the line segment extraction unit 302, the line segment may be extracted by any other method. For example, it is possible to extract a line segment from a captured image using a model generated by machine learning.

As a specific example, first, a model that has learned the concrete block boundary is prepared, and the detection score map of the concrete block boundary is acquired by applying the model to the captured image. It is possible to extract the line segments constituting the concrete block boundary by extracting the pixels holding the detection score of a predetermined value or more as the constituent elements of the concrete block boundary.

In this case, in the line segment analysis in S404, pixels holding a detection score equal to or higher than a predetermined value are extracted from the detection score map of the concrete block boundary, and a straight line is applied to the extracted line segment. After that, in S406, the average detection score of the concrete boundary of the constituent pixels may be calculated for the fitted straight line, and the calculated average value may be set as the importance in S407. As a result, the importance index can be set on a learning basis, and even for an image that cannot be handled on a rule basis, an appropriate auxiliary line can be selected and displayed in order to input a correction point. Further, the importance may be defined in combination with the importance by the detection score map using machine learning and the importance of the equation (4).

Further, in S404, one end or both ends of the straight line may be extended to the image end. For example, after performing straight line joining processing, both ends of all straight lines may be extended to the image edge, or extended to the image edge only when the distance from the straight line end to the image edge is less than a predetermined value. You may try to do it. This makes it possible to select and display an appropriate auxiliary line for inputting the correction point even if the correction point is hidden in the place where the correction point should be input.

Further, in the present embodiment, the slope of the line segment has been focused on as the regularity of the line segment for setting the importance, but if the regularity for setting the importance can be defined, only the slope of the line segment can be defined. Not exclusively. For example, numerical values based on the line segment spacing, length, thickness, and shape may be calculated from the drawing and used for setting the importance.

Further, in S404, a curved line may be applied instead of a straight line. For curve fitting, existing methods such as multiple regression analysis can be used. Further, in the present embodiment, the importance index I based on the regularity of the straight line is defined for the sake of explanation, but when a curve is applied, the importance index calculated in S406 may be changed. For example, the curve on the drawing may be superimposed on the fitted curve, and the overlap ratio of the constituent pixels when the overlap is the largest may be set as the importance index. As a result, even if the subject is a structure composed of curved lines, it is possible to select and display an appropriate auxiliary line for inputting a correction point.

Further, although S404 is configured to fit a straight line to all the extracted line segments, it may be configured so that the line segment to be fitted to the straight line can be selected by the user himself / herself. This allows the user to select the auxiliary line that suits his or her taste.

Further, in S404, in order to make the explanation easy to understand, the straight lines are first classified based on the tendency of inclination, but it is not always necessary to classify the straight lines first.

Further, although S405 is configured to determine whether the importance setting for each straight line is completed, it may be configured to have the user confirm or approve before or after the determination. At this time, it may be configured so that the user can further modify the importance. Further, the importance calculation method of S406 is not limited to the above, and any importance index that reflects the structure of the subject in the importance of each line segment can be arbitrarily defined. These make it possible to reflect the user's preference in the importance setting.

Although the line segment presented to the user is selected and displayed in S411, the line segment may not be selected in S411 and all the line segments obtained in S404 may be displayed. Further, the user may select a line segment from the line segments obtained in S404, and may be configured to switch the display / non-display of the selected line segment. The line segment can be selected by any method as long as the line segment intended by the user can be selected. For example, for a certain line segment, when a user's click motion is detected at a position within a predetermined distance from the line segment, the line segment may be regarded as selected.

(Modification example)
In the above-described embodiment, the line segment is extracted from the input image, a straight line is applied to the line segment, the importance is set to the applied straight line, and the importance is set to be high according to the operation of selecting the correction point. Explained how to configure to display.

Hereinafter, a method of displaying not only a more appropriate auxiliary line for inputting a correction point but also an input candidate position of a correction point by using the drawing information will be described using the flowchart shown in FIG. .. The processing of each step in the flowchart shown in FIG. 11 is realized by the calculation and processing of information by the CPU 101 executing the program stored in the ROM 103 or the HDD 107 of the image processing apparatus 300, and the control of each hardware. It should be noted that a configuration may be configured in which a part or all of the processing of each step in the flowchart shown in FIG. 11 is realized by hardware such as one or a plurality of ASICs and FPGAs.

In the flowchart shown in FIG. 11, the same reference numerals are given to the steps similar to those in the flowchart shown in FIG. 4, and detailed description thereof will be omitted. The flowchart shown in FIG. 11 is compared with the flowchart shown in FIG. 4, and the processes from S2201 to S2206 are added before the execution of S410, and the process of S2207 is added after the process of S412.

The image processing apparatus 300 executes the processing of S401 to S407, and when the importance setting is completed, the processing is transferred to S2201.

In S2201, the image processing apparatus 300 determines whether or not the number of confirmed candidate points is 2 or more, and if it is 2 or more, the process is transferred to S2202, and if it is less than 2, the process is transferred to S410.

At the time of first processing S2201, no candidate points have been set yet, so the processing is transferred to S410. The processes of S410 to S412 are the same as the steps in the flowchart shown in FIG.

In S2207, the image processing apparatus 300 determines whether or not the position of the current correction point has been determined. In the present embodiment, the number of fixed correction points by the user is counted, and if the count number is the same as that at the time of the previous determination in S2207, it is determined that the position of the current correction point has not been determined yet, and processing is performed. Is returned to S410. If the count number has increased by one or more since the previous determination in S2207, or if the count number has reached 4, it is determined that the position of the current correction point has been determined, and the process is transferred to S413.

In S413, the image processing device 300 determines whether the positions of all the correction points have been determined, returns the processing to S2201 if the positions have not been determined, and performs the processing if the positions of all the correction points have been determined. Move to S408. The processing after S408 is the same processing as the step in the flowchart shown in FIG.

The operation when it is determined in S2201 that the number of confirmed candidate points is 2 or more will be described below.

In S2202, the image processing apparatus 300 calculates the conversion parameter from the fixed correction point and the corresponding reference point in the correction information setting unit 304. However, the types of conversion parameters that can be calculated differ depending on the number of confirmed candidate points. That is, when the number of confirmed candidate points is two, the conversion parameter of the similarity transformation is calculated, and when the number is three, the conversion parameter of the affine transformation is calculated. When the number of confirmed candidate points is four, in S2202, the conversion parameter of the projective transformation is calculated in the same manner as in S408.

Here, the similarity transformation is expressed by the equation (6), and the parameters are (t _x , _ty , θ, s). The four parameters are obtained by solving four simultaneous equations obtained from two different sets of correspondence points.

The affine transformation is expressed by the equation (7), and there are six _{parameters (A 11} , A ₁₂ , A ₁₃ , A ₂₁ , A ₂₂ , A _23,). The six parameters are obtained by solving the six equations obtained from three different sets of correspondence points.

In S2203, the image processing apparatus 300 calculates the candidate position for the undetermined correction point in the correction control unit 305 by using the reference point on the drawing and the conversion parameter calculated in S2202. For example, when the correction points 213 and the correction points 214 have been determined as the corresponding points of the reference point 209 and the reference point 210, the correction points corresponding to the reference point 211 reference point 212 are candidates on the captured image of the respective correction points. Calculate the position.

FIG. 12 is a diagram showing a display example of candidate positions of correction points. FIG. 12 shows an example of displaying an undetermined correction point using a reference point and a calculated conversion parameter when two correction points are fixed. In FIG. 12, 2301 is a fixed correction point corresponding to the reference point 209, and 2302 is a fixed correction point corresponding to the reference point 210. 2303 is a conversion rectangle. In the generation of the conversion rectangle, first, the conversion parameters of the equation (6) are calculated using the reference points 209 and 210 on the drawing 205 and the fixed correction points 2301 and 2302 on the captured image 206. Next, the rectangle (reference point 209-210-211-212) on the design drawing is converted using the calculated conversion parameters.

2304 and 2305 are candidate positions of correction points corresponding to the reference points 211 and 212, respectively, and the positions are calculated by the reference points 211 and 212 and the conversion parameters. In FIG. 12, the confirmed correction point is represented by a symbol in which a white number is described on a black rectangle, and the unconfirmed correction point is represented by a symbol in which a black number is described on a white circle. When three correction points are fixed, the conversion parameter of the equation (7) is calculated, the rectangle on the design drawing is converted using the calculated conversion parameter, and the last remaining undetermined correction point is fixed. Determine the candidate position of.

In S2204, the image processing apparatus 300 selects a straight line to be displayed from the straight lines whose importance has been set in S407. Here, for each of the candidate positions of the correction points calculated in S2203, a predetermined number (for example, two) of straight lines passing through a region within a predetermined distance (for example, a radius of 20 pixels) from that position are selected in descending order of importance. In the present embodiment, the distance from the candidate position of the correction point to the straight line is set to 20 pixels for the sake of explanation, but the distance may be set to an appropriate distance according to the input image. For example, if the rotation of the input captured image in the depth direction is small, the distance can be set short, and if the rotation of the input captured image in the depth direction is large, the distance is set to 20 pixels or more. You may. Further, when the number of line segments extracted as noise is small, it is possible to configure the configuration without limiting by the distance or to set the distance condition to the maximum value that can be set.

In S2205, the image processing device 300 causes the display control unit 306 to display the candidate position of the correction point calculated in S2203 and the straight line selected in S2204.

FIG. 13 is a diagram showing a display example of candidate positions of correction points and information for setting correction points. FIG. 13 shows an example of straight line selection based on the candidate positions of undetermined correction points. In FIG. 13, 2401 and 2402 are straight lines selected with reference to the candidate position 2304 of the correction point corresponding to the reference point 211. Further, 2401 and 2403 are straight lines selected with reference to the candidate position 2305 of the correction point corresponding to the reference point 212 (straight line 2401 is selected in duplicate).

In S2206, the image processing device 300 further fine-tunes the calculated candidate position in the correction information setting unit 304. First, the importance of the straight line within a predetermined distance from the confirmed correction point is set to the maximum value. The straight lines within a predetermined distance with respect to the fixed correction point 2301 corresponding to the reference point 209 are the straight line 701 and the straight line 901. The straight line 901 is a straight line 2403 selected with reference to the candidate position 2305 of the correction point corresponding to the reference point 212 in FIG. 13 (the straight line 701 is not shown in FIG. 13). Similarly, the straight lines within a predetermined distance with respect to the fixed correction point 2302 corresponding to the reference point 210 are the straight line 702 and the straight line 902. In FIG. 13, the straight line 902 is a straight line 2402 selected with reference to the candidate position 2304 of the correction point corresponding to the reference point 211.

Next, the image processing apparatus 300 determines whether the distance between the intersection of the straight lines selected in S2204 and the candidate position 2304 of the correction point corresponding to the reference point 211 is within a predetermined distance. In the case of a predetermined distance, the candidate position 2304 of the correction point corresponding to the reference point 211 is updated to the position of the intersection in S2205 and displayed by the display control unit 306. The same applies to the candidate position 2305 corresponding to the reference point 212.

FIG. 14 is a diagram showing an example of updating candidate positions of undetermined correction points. In FIG. 14, 2501 is an update candidate position after updating the correction point candidate position 2304 corresponding to the reference point 211 to a straight line intersection, and 2502 is a correction point candidate position 2305 corresponding to the reference point 212 as a straight line intersection. This is the update candidate position after updating. The image processing device 300 sets the candidate position of the correction point on the auxiliary line. This makes it possible for the user to set the correction point accurately and easily.

After that, after processing S410, the line segment selected in S411 is displayed.

As described above, in the modified example, the input candidate positions of the correction points are displayed by using the drawing information. With the configuration in this embodiment, the correct straight line and the intersection of the straight lines are determined from the straight line candidates extracted each time the user determines the correction point, so that the correction point is not determined each time the user determines the correction point. The accuracy of estimating the candidate position of the correction point of is improved. Therefore, it becomes easy for the user to set the correction point at the correct position, and the accuracy of the orthophoto correction is lost.

In addition, it is configured to display a straight line set with high importance according to the input candidate position of the correction point. This makes it possible to select and display an appropriate auxiliary line for inputting a correction point even when a standardized image or an image close to the standardized image cannot be obtained.

<Other embodiments>
INDUSTRIAL APPLICABILITY The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

101 Central processing unit (CPU) that controls the control of computer systems
102 Random access memory (RAM)
103 Read-only memory (ROM)
104 Network interface 105 Display device 106 Input device 107 Hard disk drive (HDD)
108 I / O bus

Claims (19)

A detection means that detects line segments from captured images,
A determination means for determining whether the line segment detected by the detection means is a line segment indicating the structure of the structure that is the subject of the captured image.
According to the determination means, it is information for setting a correction point on the photographed image corresponding to a reference point on the facing surface in the tilt correction for making the photographed image face-to-face, and is a line segment showing the structure. A display control means for superimposing and displaying information for emphasizing the determined line segment on the captured image, and
An image processing apparatus comprising: a determination means for determining the position of the correction point after the information displayed by the display control means is displayed. The display control means is a straight line corresponding to a line segment determined by the determination means to be a line segment showing the structure, and a straight line having a line width thicker than the line segment is superimposed on the captured image. The image processing apparatus according to claim 1, wherein the image processing apparatus is displayed. The display control means is a straight line corresponding to a line segment determined by the determination means to be a line segment showing the structure, and a straight line longer than the line segment is superimposed and displayed on the captured image. The image processing apparatus according to claim 1 or 2. The display control means is a straight line corresponding to a line segment determined by the determination means to be a line segment showing the structure, and is characterized in that a colored straight line is superimposed and displayed on the captured image. The image processing apparatus according to any one of claims 1 to 3. The aspect according to any one of claims 1 to 4, wherein the display control means superimposes and displays a straight line based on a combination of a plurality of line segments detected by the detection means on the captured image. Image processing device. Further, the line segment detected by the detection means has a setting means for setting the importance indicating whether the line segment indicates the structure of the structure.
The determination means is characterized in that it determines whether the line segment detected by the detection means is a line segment indicating the structure of the structure based on the importance set by the setting means. Item 6. The image processing apparatus according to any one of Items 1 to 5. The image processing apparatus according to claim 6, wherein the setting means sets the importance of the line segment based on the linearity of the line segment detected by the detection means. The image processing according to claim 7, wherein the straight line-likeness is based on the average of the distances between each voted point and the straight line in the Hough transform of the straight line applied to the line segment detected by the detection means. Device. The image according to any one of claims 6 to 8, wherein the setting means sets the importance of the line segment based on the length of the line segment detected by the detection means. Processing equipment. The setting means sets the importance of the line segment based on whether or not the straight line corresponding to the line segment detected by the detection means is a straight line that regularly exists in the structure. The image processing apparatus according to any one of claims 6 to 9, wherein the image processing apparatus is characterized. The setting means is a straight line having a slope similar to that of a straight line corresponding to a line segment detected by the detection means, and is based on the number of straight lines corresponding to other line segments detected by the detection means. The image processing apparatus according to claim 10, wherein the importance of the line segment is set. Further having a means for detecting the candidate position of the correction point,
One of claims 6 to 11, wherein the display control means displays information that emphasizes a line segment detected by the detection means selected based on the candidate position and the importance. The image processing apparatus according to. 12. The display control means according to claim 12, wherein the display control means displays information that is a line segment existing at a predetermined distance from the candidate position and emphasizes a predetermined number of line segments in descending order of importance. Image processing device. 12. The display control means according to claim 12, wherein the display control means displays information that is a line segment existing at a predetermined distance from the candidate position and emphasizes a predetermined number of line segments in descending order of importance. Image processing device. The display control means has a third display control means when the position of the first correction point corresponding to the first reference point and the position of the second correction point corresponding to the second reference point are determined by the determination means. The image processing apparatus according to any one of claims 1 to 14, wherein information indicating a position on the captured image, which is a candidate for a third correction point corresponding to the reference point of the above, is displayed. The image according to claim 15, wherein the position of the candidate for the third correction point is on a straight line corresponding to the line segment determined by the determination means to be a line segment showing the structure. Processing equipment. The one according to any one of claims 1 to 16, wherein the determination means determines whether the line segment detected by the detection means is a line segment indicating a boundary of a unit concrete block. Image processing device. A detection process that detects line segments from captured images,
A determination step of determining whether the line segment detected in the detection step is a line segment indicating the structure of the structure that is the subject of the captured image.
In the determination step, it is information for setting a correction point on the photographed image corresponding to the reference point of the facing surface in the tilt correction for making the photographed image face-to-face, and it is a line segment showing the structure. A display control step of superimposing information for emphasizing the determined line segment on the captured image and displaying it.
An image processing method comprising: a display control step followed by a determination step of determining the position of the correction point. A program for operating a computer as the image processing device according to any one of claims 1 to 17.

Images