JP7207073B2 - Inspection work support device, inspection work support method and inspection work support program - Google Patents

Description

The present invention relates to an inspection work support device, an inspection work support method, and an inspection work support program.

Conventionally, structures such as bridges and tunnels are periodically inspected by close visual inspection. In the case of a close-up visual inspection, the inspector sets up scaffolding around the structure and directly accesses the structure to perform a visual inspection, measure the size of the detected damaged part, and inspect the inside of the structure. The position of the damaged part of the body and the image of the damaged part are recorded.

On the other hand, in order to reduce the workload of inspection workers in close-up visual inspections, we are developing an inspection work support device that continuously photographs structures using unmanned aircraft and uses the obtained image data to perform inspection work. is in progress. According to the inspection work support device, an inspection operator detects a damaged part from an orthoimage generated based on a three-dimensional model, calculates the size of the detected damaged part, or determines the damaged part in the three-dimensional model. It is possible to record orthoimages, etc., including the position of the wound and the damaged part.

JP 2017-049152 A

However, it is difficult to achieve high accuracy in inspection work using orthoimages, because the generation of orthoimages generally causes image degradation (blurring, discontinuous parts, etc.). is.

An object of one aspect is to provide an inspection work support device, an inspection work support method, and an inspection work support program that improve the accuracy of inspection work.

According to one aspect, the inspection work support device includes:
a first generating unit that generates and displays a three-dimensional model image by synthesizing partial images obtained by photographing each part of a structure;
a second generating unit that generates and displays an orthorectified image of a designated inspection area in the three-dimensional model image;
a display unit for displaying a partial image corresponding to a position pointed to in the orthorectified image being displayed, based on a correspondence relationship between each area of the orthorectified image and the partial images forming the respective areas. death,
When the first generation unit generates the three-dimensional model image, each patch forming a mesh, three-dimensional coordinates of each vertex of the mesh, each partial image related to generation of each patch, each part The apparatus further includes a storage unit that stores position and orientation information of the imaging device when the image was captured in association with the information .

It is possible to provide an inspection work support device, an inspection work support method, and an inspection work support program that improve the accuracy of inspection work.

1 is a first diagram showing an example of a system configuration of an inspection work support system; FIG. It is a figure which shows an example of the hardware constitutions of an inspection work assistance apparatus. FIG. 3 is a diagram for explaining an outline of various functions implemented in the inspection work support device; 1 is a first diagram showing an example of a functional configuration of an inspection work support device; FIG. FIG. 10 is a diagram showing a specific example of orthoimage display processing; FIG. 10 is a diagram showing a specific example of partial image and pointing position display processing; FIG. 10 is a diagram showing a specific example of trace processing; 4 is a first flowchart showing the flow of inspection work support processing; 9 is a second flowchart showing the flow of inspection work support processing; It is a figure which shows an example of the display screen of an inspection work assistance apparatus. FIG. 4 is a second diagram showing an example of the functional configuration of the inspection work support device; FIG. 10 is a diagram showing a specific example of orthoimage update processing; 9 is a third flowchart showing the flow of inspection work support processing; Fig. 2 is a second diagram showing an example of the system configuration of the inspection work support system;

Each embodiment will be described below with reference to the accompanying drawings. In the present specification and drawings, constituent elements having substantially the same functional configuration are denoted by the same reference numerals, thereby omitting redundant description.

[First embodiment]
<System configuration of inspection work support system>
First, the system configuration of the inspection work support system will be described. FIG. 1 is a first diagram showing an example of a system configuration of an inspection work support system. As shown in FIG. 1 , the inspection work support system 100 includes an inspection drone 110 and an inspection work support device 120 .

The inspection drone 110 is an unmanned aerial vehicle equipped with an imaging device. The inspection drone 110 flies at high speed to the position of the structure (to be inspected) such as the bridge 130, and then flies around the structure at slow speed to continuously photograph the structure over a wide area. After returning, the inspection drone 110 transmits image data obtained by continuous shooting to the inspection work support device 120 .

The inspection work support device 120 generates a three-dimensional model image based on the image data transmitted from the inspection drone 110 and displays it to the inspection operator. In addition, the inspection work support device 120 generates an orthorectified image and enlarges and displays the inspection area specified by the inspection operator in the displayed three-dimensional model image. The inspection work support device 120 also receives the position of the damaged part pointed by the inspection operator in the enlarged orthoimage. Furthermore, the inspection work support device 120 displays a high-quality partial image corresponding to the pointed position of the damaged part, and also displays a three-dimensional point corresponding to the pointed position of the damaged part on the high-quality partial image. Visualize on.

As a result, the inspection work support device 120 can create an inspection work record 140 including a three-dimensional model image, an orthoimage, and a partial image in which three-dimensional points corresponding to the position of the damaged part are visualized. . Thus, according to the inspection work support system 100, it is possible to create the inspection work report 140 based on highly accurate inspection work by the inspection worker.

<Hardware configuration of inspection work support device>
Next, the hardware configuration of the inspection work support device 120 will be described. FIG. 2 is a diagram illustrating an example of a hardware configuration of an inspection work support device; As shown in FIG. 2 , the inspection work support device 120 has a CPU (Central Processing Unit) 201 , a ROM (Read Only Memory) 202 and a RAM (Random Access Memory) 203 . The CPU 201, ROM 202, and RAM 203 form a so-called computer.

The inspection work support device 120 also has an auxiliary storage device 204 , a display device 205 , an operation device 206 , an I/F (Interface) device 207 and a drive device 208 . Each piece of hardware of the inspection work support device 120 is interconnected via a bus 209 .

The CPU 201 is an arithmetic device that executes various programs (for example, an inspection work support program, etc.) installed in the auxiliary storage device 204 .

ROM 202 is a non-volatile memory. The ROM 202 functions as a main storage device that stores various programs, data, etc. necessary for the CPU 201 to execute various programs installed in the auxiliary storage device 204 . Specifically, the ROM 202 functions as a main storage device that stores boot programs such as BIOS (Basic Input/Output System) and EFI (Extensible Firmware Interface).

A RAM 203 is a volatile memory such as a DRAM (Dynamic Random Access Memory) or an SRAM (Static Random Access Memory). The RAM 203 functions as a main storage device that provides a working area to be expanded when various programs installed in the auxiliary storage device 204 are executed by the CPU 201 .

The auxiliary storage device 204 is an auxiliary storage device that stores various programs and information used when various programs are executed. For example, a 3D model storage unit and a damage 3D position/damage image storage unit, which will be described later, are implemented in the auxiliary storage device 204 .

The display device 205 is a display device that displays various display screens including three-dimensional model images, orthorectified images, partial images, and the like. The operation device 206 is an input device for the inspection worker to give various instructions to the inspection work support device 120 (for example, an operation to specify an inspection area, an operation to point to the position of the damaged part, etc.).

The I/F device 207 is a communication device that is wirelessly connected to the inspection drone 110 and communicates with the inspection drone 110 .

A drive device 208 is a device for setting a recording medium 210 . The recording medium 210 here includes media such as CD-ROMs, flexible disks, magneto-optical disks, etc. that record information optically, electrically, or magnetically. The recording medium 210 may also include a semiconductor memory or the like that electrically records information, such as a ROM or a flash memory.

Various programs to be installed in the auxiliary storage device 204 are installed by, for example, setting the distributed recording medium 210 in the drive device 208 and reading the various programs recorded in the recording medium 210 by the drive device 208. be done. Alternatively, various programs installed in the auxiliary storage device 204 may be installed by being downloaded from a network (not shown).

<Functions implemented in the inspection work support device>
Next, an outline of main functions realized in the inspection work support device will be described. FIG. 3 is a diagram for explaining an outline of various functions implemented in the inspection work support device.

As shown in FIG. 3 , the inspection work support device 120 receives image data 310 transmitted from the inspection drone 110 . Each frame image included in the image data 310 is a partial image obtained by photographing each part of the structure.

The inspection work support device 120 performs three-dimensional synthesis processing using the partial images included in the image data 310 to generate a three-dimensional model image 320 that allows a bird's-eye view of a wide area of the structure. At this time, in the inspection work support device 120, the three-dimensional coordinates of each vertex of the mesh calculated when performing the three-dimensional synthesis processing, each patch forming the mesh, each partial image related to generation of each patch, Position and orientation information of the imaging device when each partial image was shot is associated with and managed.

The database 350 of FIG. 3 contains three-dimensional coordinates of each vertex of the mesh calculated when performing three-dimensional synthesis processing, index information of each patch forming the mesh, each partial image related to generation of each patch, It is a database that associates position and orientation information of an imaging device.

The inspection work support device 120 displays the generated three-dimensional model image 320 on the display device 205 . Accordingly, the inspection operator designates the inspection area 321 using the operation device 206 . The inspection work support device 120 orthographically projects the specified inspection area 321 to generate an orthorectified image 330 , and enlarges and displays the generated orthorectified image 330 on the display device 205 .

When the orthoimage 330 is enlarged and displayed, the inspection operator uses the operation device 206 to point to the position of the damaged portion (P). When the inspector points the position of the damaged part (P), the inspection work support device 120 refers to the patch index image 340 to identify the patch containing the pointed position of the damaged part (P). do.

In the example of FIG. 3, the inspection work support device 120 refers to the patch index image 340 to specify the patch with index information=“id3” as the patch containing the position of the pointed damaged part (P). showing the situation. The patch index image 340 is an image containing index information of patches forming each area of the orthorectified image 330 .

When the inspection work support device 120 identifies the patch containing the pointed position of the damaged portion (P), it refers to the database 350 and extracts the three-dimensional coordinates of each vertex of the mesh forming the identified patch. Then, the inspection work support device 120 calculates the three-dimensional position coordinates of the damaged portion (P) based on the three-dimensional coordinates of each vertex of the mesh.

The inspection work support device 120 also refers to the database 350 to extract each partial image related to generation of the specified patch and the position and orientation information of the imaging device when each partial image was captured. Note that the partial images related to patch generation refer to all partial images in which the patch is shown.

The inspection work support device 120 selects a partial image suitable for display from among the extracted partial images. Specifically, the inspection work support device 120 selects a partial image suitable for display based on the position and orientation information of the imaging device when each partial image was shot.

Further, the inspection work support device 120 projects the 3D positional coordinates of the damaged portion (P) onto the selected partial image, thereby visualizing the 3D point corresponding to the damaged portion (P) in the partial image.

In FIG. 3, a partial image 360 shows how a three-dimensional point (P') on the partial image 360 corresponding to the damaged portion (P) designated in the orthoimage 330 is displayed.

In this way, according to the inspection work support device 120, when the inspection operator points the position of the damaged part (P) in the orthoimage 330, the corresponding partial image 360 is displayed, and the corresponding three-dimensional point (P ') can be visualized. As a result, even if the orthoimage 330 is blurred or includes discontinuous parts, the inspector can visually recognize the damaged part on the high-quality partial image 360. . As a result, according to the inspection work support device 120, the inspection worker can perform highly accurate inspection work.

<Functional configuration of inspection work support device>
Next, the functional configuration of the inspection work support device 120 will be described. FIG. 4 is a first diagram showing an example of the functional configuration of the inspection work support device. As described above, the inspection work support program is installed in the inspection work support device 120, and by executing the program, the inspection work support device 120 becomes the 3D synthesis processing unit, which is the first generation unit. 401 and functions as a 3D model display unit 402 . The inspection work support device 120 also functions as an inspection area designating unit 403, an orthorectified image generating unit 404, and an orthorectified image display unit 405, which are second generation units. Furthermore, the inspection work support device 120 includes a display unit such as a 3D point cloud information management unit 406, a damage position designation unit 407, a patch index management unit 408, a damage 3D position calculation unit 409, a partial image selection unit 410, and a partial image display unit. It functions as the part 411 .

The 3D synthesis processing unit 401 calculates three-dimensional model information by performing SfM (Structure from Motion) processing, which is an example of synthesis processing, using a plurality of partial images. The 3D model information includes position and orientation information of the imaging device when each partial image was captured, 3D coordinates corresponding to the image feature points of the structure to be inspected, 3D coordinates of each vertex of the mesh, and each of the meshes. It contains patches (specifically, texture information mapped to each patch). Note that the 3D synthesis processing unit 401 stores the calculated 3D model information in the 3D model storage unit 421 .

The 3D model display unit 402 draws texture by projecting each patch in the view direction from the 3D model information stored in the 3D model storage unit 421 . Thereby, the 3D model display unit 402 generates a 3D model image and displays it on the display device 205 .

The inspection area specifying unit 403 receives an inspection area specified using the operation device 206 in the three-dimensional model image displayed on the display device 205 . Specifically, the inspection area designating unit 403 receives setting values for the center position of the inspection area and the size of the inspection area.

The orthoimage generation unit 404 extracts each patch corresponding to the designated inspection area from the three-dimensional model information, and determines the orthographic projection direction from the normal direction of each patch. The orthorectified image generation unit 404 determines the orthogonal projection direction based on the average value or mode of the normal direction of each patch.

Further, the orthoimage generation unit 404, based on the determined orthographic projection direction,
・ An image (ortho image) obtained by orthographically projecting each patch (specifically, texture information mapped to each patch),
・ An image (patch index image) in which the area obtained by orthographically projecting each patch is filled with the index information of each patch,
to generate

The orthoimage display unit 405 displays the generated orthoimage on the display device 205 .

The 3D point group information management unit 406 gives index information to each patch in the 3D model information calculated by the 3D synthesis processing unit 401 . In addition, the 3D point group information management unit 406 captures the three-dimensional coordinates of each vertex of the mesh corresponding to each patch, a partial image showing each patch, and a partial image in the index information given to each patch. and the position and orientation information of the imaging device at that time. Thereby, the 3D point cloud information management unit 406 generates and manages a database.

The damaged position specifying unit 407 receives the position of the damaged part (pointing position) pointed using the operation device 206 in the orthorectified image displayed on the display device 205 . The damaged position designation unit 407 identifies each mode of cursor, point designation, and trace based on the button event of the operation device 206, and accepts the pointing position.

The patch index management unit 408 identifies the patch corresponding to the pointing position on the orthorectified image by referring to the patch index image, and outputs index information.

The damage 3D position calculation unit 409 calculates and outputs 3D position coordinates on the 3D model image corresponding to the pointing position on the orthoimage received by the damage position designation unit 407 .

The partial image selection unit 410 selects one partial image. in particular,
The patch index management unit 408 identifies the patch corresponding to the pointing position on the orthoimage received by the damaged position designation unit 407,
The 3D point cloud information management unit 406 outputs the position and orientation information of the imaging device when each partial image was captured from each partial image in which the specified patch is captured,
In this case, the partial image selection unit 410 extracts partial images in which the deviation between the normal direction of the patch and the photographing direction of the image pickup device when the partial image was photographed is equal to or less than the set value. do. Further, the partial image selection unit 410 selects one partial image with the shortest shooting distance from among the extracted partial images. The partial image selection unit 410 outputs the selected partial image and the position and orientation information of the imaging device.

A partial image display unit 411 displays one partial image selected by the partial image selection unit 410 on the display device 205 . Also, the partial image display unit 411 displays the 3D point (or 3D point sequence) corresponding to the pointing position on the orthoimage calculated by the damage 3D position calculation unit 409 on the selected partial image. to display.

In the damage 3D position/damage image storage unit 422, when the mode set by the damage position designation unit 407 is point designation, the information of the 3D point corresponding to the pointing position on the orthorectified image and the corresponding partial image is stored. are stored as one record. In addition, in the damage 3D position/damage image storage unit 422, when the mode set by the damage position specifying unit 407 is trace, the information of the three-dimensional point sequence corresponding to the pointing position on the orthorectified image and the corresponding partial image is stored. are stored as one record.

<Specific example of processing executed by inspection work support device>
Next, among the various processes executed by each functional block of the inspection work support device 120,
・ Ortho image display processing,
・Partial image and pointing position display processing,
・Trace processing,
A specific example will be described.

(1) Specific example of ortho image display processing First, among the processing executed by each functional block of the inspection work support device 120
- 3D synthesis processing unit 401,
- 3D model display unit 402,
・Inspection area designation unit 403,
orthorectified image generator 404,
orthorectified image display unit 405,
- 3D point cloud information management unit 406,
A specific example of the orthorectified image display processing executed by will be described. FIG. 5 is a diagram showing a specific example of the orthoimage display processing.

As shown in FIG. 5 , the 3D synthesis processing unit 401 calculates 3D model information by performing SfM processing using a plurality of partial images included in the image data 310 . Also, the 3D point cloud information management unit 406 gives index information to each patch of the three-dimensional model information calculated by the 3D synthesis processing unit 401 . In addition, the 3D point cloud information management unit 406 adds, to the index information assigned to each patch, the three-dimensional coordinates of each vertex of the mesh corresponding to each patch, the partial image in which each patch is captured, and the position and orientation of the imaging device. and information to generate database 350 (see arrow 501).

The 3D model display unit 402 also generates a 3D model image 320 based on the 3D model information and displays it on the display device 205 (see arrow 502). Also, the inspection area specifying unit 403 receives an inspection area 321 specified via the operation device 206 in the three-dimensional model image 320 displayed on the display device 205 (see arrow 503).

The orthoimage generating unit 404 extracts each patch corresponding to the specified inspection area 321 from the three-dimensional model information, and generates an orthoimage 330 and patch index images 340 (see arrows 504 and 505). Also, the orthoimage display unit 405 displays the generated orthoimage 330 on the display device 205 .

(2) Specific example of partial image and pointing position display processing Next, among the processing executed by each functional block of the inspection work support device 120,
- 3D point cloud information management unit 406,
・Damage position specifying unit 407,
A patch index management unit 408,
- Damage 3D position calculation unit 409,
a partial image selection unit 410,
a partial image display unit 411,
A specific example of partial image and pointing position display processing executed by . FIG. 6 is a diagram showing a specific example of partial image and pointing position display processing.

As shown in FIG. 6, the damaged position specifying unit 407 receives the pointing position of the damaged part (P) in the orthorectified image 330 displayed on the display device 205 (see arrow 506). The patch index management unit 408 identifies the patch corresponding to the pointing position on the orthorectified image 330 by referring to the patch index image 340 (see arrow 507), and outputs index information (see arrow 508). The example of FIG. 6 shows how the patch index management unit 408 outputs index information=“id3”.

In addition, the damage 3D position calculation unit 409 calculates and outputs the 3D position coordinates of the 3D point on the 3D model image corresponding to the pointing position of the damaged part (P) on the orthoimage 330 ( See arrows 601, 603). The damage 3D position calculation unit 409 extracts the three-dimensional coordinates of each vertex of the mesh corresponding to the index information output by the patch index management unit 408 (see arrow 602), and based on the extracted three-dimensional coordinates , to calculate the 3D position coordinates of the 3D points.

Further, the partial image selection unit 410 selects each partial image (file name=img0, img1, . Extract a partial image that satisfies
the normal direction of the identified patch;
- The shooting direction of the imaging device when each partial image was shot,
is less than or equal to the set value.

Further, the partial image selection unit 410 selects the partial image with the shortest shooting distance (eg, file name=img0) from among the extracted partial images (eg, file name=img0, img1) (see arrow 604). Furthermore, the partial image selection unit 410 outputs the selected partial image (file name=img0) and the position and orientation information (C0) of the imaging device.

The partial image display unit 411 displays the partial image 360 (file name=img0) selected by the partial image selection unit 410 on the display device 205 (see arrow 605). Further, the partial image display unit 411 projects the 3D position coordinates calculated by the damage 3D position calculation unit 409 onto the partial image 360 (file name=img0) to display a 3D point (P′) on the display device 205. (see arrow 606).

(3) Specific example of trace processing Next, among the processing executed by each functional block of the inspection work support device 120,
- Damage 3D position calculation unit 409,
a partial image selection unit 410,
A specific example of trace processing executed by is described. FIG. 7 is a diagram showing a specific example of trace processing.

The example of FIG. 7 shows the case where the pointing position for the damaged portion (P) on the orthoimage 330 is included in the patch with index information=“id3” on the patch index image 340 . The example of FIG. 7 also shows how the partial image selection unit 410 selects the partial image 710 as the partial image corresponding to the patch with index information=“id3” (see arrow 701).

Furthermore, the example of FIG. 7 shows how the corresponding three-dimensional point is traced on the partial image 710 in response to the pointing position of the damaged area (P) being traced on the orthoimage 330. (See reference numeral 711). The damage 3D position calculation unit 409 stores the 3D point sequence traced on the partial image 710 in the damage 3D position/damage image storage unit 422 as a trace history (see arrow 702).

Here, it is assumed that the pointing position of the damaged portion (P) is traced on the orthoimage 330 and the pointing position is switched to the patch with index information=“id2” on the patch index image 340 . In this case, the partial image selection unit 410 selects the partial image 720 as the partial image corresponding to the patch with index information=“id2” as shown in FIG. 7 (see arrow 703).

At this time, the partial image selection unit 410 refers to the damage 3D position/damage image storage unit 422 and reads out the trace history (see arrow 704). Thereby, the partial image selection unit 410 can display the trace history on the partial image 720 (see reference numeral 721).

As described above, in the inspection work support device 120, even when the partial image to be selected is switched in response to the pointing position being traced on the orthorectified image 330, the trace history of the partial image 710 before switching is changed to the trace history of the partial image 710 after switching. is taken over to the partial image 720 of . This improves the visibility when tracing the pointing position of the damaged portion (P), and enables continuous execution of the pointing position tracing work.

<Flow of inspection work support processing>
Next, the flow of inspection work support processing by the inspection work support device 120 will be described with reference to FIGS. 8 and 9. FIG. 8 and 9 are first and second flowcharts showing the flow of inspection work support processing.

In step S<b>801 , the 3D synthesis processing unit 401 acquires partial images included in the image data transmitted from the inspection drone 110 .

In step S802, the 3D synthesis processing unit 401 executes 3D synthesis processing to generate three-dimensional model information.

In step S<b>803 , the 3D synthesis processing unit 401 stores the generated 3D model information in the 3D model storage unit 421 . Also, the 3D model display unit 402 generates a 3D model image based on the 3D model information and displays it on the display device 205 .

In step S804, the 3D point cloud information management unit 406 assigns index information to each patch in the 3D model information, associates the 3D coordinates of each vertex of the mesh, the partial image, and the position/orientation information, and stores the information in a database. to generate

In step S805, the inspection area specifying unit 403 receives an inspection area specified in the three-dimensional model image.

In step S806, the inspection area specifying unit 403 determines whether or not the inspection area has been specified. If it is determined in step S806 that the designation of the inspection area has ended (if Yes in step S806), the inspection work support processing ends.

On the other hand, if it is determined in step S806 that designation of the inspection area has not ended (No in step S806), the process proceeds to step S807. In step S807, the inspection area specifying unit 403 determines whether or not an inspection area has been specified. If it is determined that an inspection area has not been specified (No in step S807), the process returns to step S806. , waits until the inspection area is specified. On the other hand, if it is determined in step S807 that an inspection area has been specified, the process proceeds to step S808.

In step S808, the orthorectified image generation unit 404 sets the specified inspection area as an area for generating an orthorectified image, and extracts patches corresponding to the specified inspection area from the three-dimensional model information.

In step S809, the orthoimage generating unit 404 determines the orthographic projection direction from the normal direction of the extracted patch. Further, the orthoimage generation unit 404 generates an orthoimage by orthographically projecting each patch (specifically, texture information mapped to each patch) based on the determined orthographic projection direction. Also, the orthoimage generating unit 404 generates a patch index image based on the determined orthographic projection direction. Furthermore, the orthoimage display unit 405 displays the generated orthoimage on the display device 205 . After that, the process proceeds to step S901 in FIG.

In step S901, the damage position specifying unit 407 receives the position of the damaged part pointed in the displayed orthorectified image.

In step S902, the damaged position specifying unit 407 determines whether or not pointing to the position of the damaged part has been completed. If it is determined in step S902 that pointing to the position of the damaged portion has been completed (if Yes in step S902), the process returns to step S806 in FIG.

On the other hand, if it is determined in step S902 that pointing to the position of the damaged portion has not been completed (No in step S902), the process proceeds to step S903.

In step S903, the damaged position specifying unit 407 determines whether or not pointing to the position of the damaged part has been received. If it is determined in step S903 that pointing to the position of the damaged part has not been received (No in step S903), the process returns to step S902 and waits until pointing to the position of the damaged part is received.

On the other hand, if it is determined in step S903 that pointing to the position of the damaged portion has been received (Yes in step S903), the process proceeds to step S904. In step S904, the patch index management unit 408 refers to the patch index image and outputs patch index information corresponding to the pointing position on the orthorectified image. Also, the damage 3D position calculation unit 409 calculates and outputs three-dimensional position coordinates corresponding to the pointing position on the orthoimage.

In step S905, the partial image selection unit 410 selects one partial image from the partial images corresponding to the patch index information.

In step S<b>906 , the partial image display unit 411 displays the selected one partial image on the display device 205 . The partial image display unit 411 also displays a three-dimensional point corresponding to the pointing position on the orthoimage on the partial image.

In step S907, the partial image display unit 411 determines whether the displayed partial image has been updated. If it is determined in step S907 that the displayed partial image has not been updated (No in step S907), the process returns to step S902.

On the other hand, if it is determined in step S907 that the displayed partial image has been updated (Yes in step S907), the process proceeds to step S908. In step S908, the partial image display unit 411 displays the trace history of the pointing position of the damaged part in the updated partial image.

<Description of display screen>
Next, a display screen displayed on the inspection work support device 120 will be described. FIG. 10 is a diagram showing an example of a display screen of the inspection work support device. Among them, FIG. 10A is a diagram showing an example of a 3D model image display screen on which a 3D model image 1010 generated by the 3D model display unit 402 is displayed. The inspector visually recognizes the three-dimensional model image 1010 while rotating, moving, enlarging, and displaying the displayed three-dimensional model image 1010 . Also, the inspection operator designates an inspection area in the three-dimensional model image 1010 .

FIG. 10B is a diagram showing an example of an orthoimage display screen in which an orthoimage 1020 is displayed by the inspection operator specifying an inspection area 1011 on the three-dimensional model image 1010 . The inspector points the position of the damaged part on the orthoimage 1020 .

FIG. 10C is a diagram showing an example of a partial image display screen in which a partial image 1030 is displayed when the inspection operator points the position of the damaged part on the orthoimage 1020 . In this way, the inspection operator can visually recognize the damaged portion on the corresponding high-quality partial image 1030 by pointing the position of the damaged portion using the orthoimage 1020 (see FIG. 10). In the partial image 1030, the damaged part is emphasized). Also, the inspector can visually recognize the three-dimensional point corresponding to the pointing position on the orthorectified image 1020 in the high-quality partial image 1030 . As a result, the inspection operator can perform inspection work with high accuracy.

As is clear from the above description, the inspection work support device according to the first embodiment synthesizes partial images obtained by photographing each part of a structure to generate a three-dimensional model image. Also, the inspection work support device according to the first embodiment generates an orthoimage of the inspection area in the three-dimensional model image. In addition, when the position of the damaged part is pointed in the orthorectified image being displayed, the inspection work support device according to the first embodiment determines the portion corresponding to the pointing position based on the correspondence relationship between the orthorectified image and the partial image. Display an image.

As a result, according to the inspection work support device according to the first embodiment, the inspection operator designates the inspection area in the three-dimensional model image and points the position of the damaged part in the orthoimage to obtain the corresponding height. The damaged part can be visually recognized on the high-quality partial image. As a result, according to the inspection work support device according to the first embodiment, highly accurate inspection work can be realized.

[Second embodiment]
In the above-described first embodiment, an orthoimage is generated for an inspection region specified in a three-dimensional model image, and the position of the damaged portion is pointed in the generated orthoimage. On the other hand, if the designated inspection area is, for example, a part of the curved surface of a structure, it may be better to update the orthoimage being displayed depending on the pointing position.

Therefore, in the second embodiment, processing for appropriately updating the orthorectified image according to the pointed position of the damaged portion when the position of the damaged portion is pointed in the displayed orthorectified image will be described. The second embodiment will be described below, focusing on differences from the first embodiment.

<Functional configuration of inspection work support device>
First, the functional configuration of the inspection work support device according to the second embodiment will be described. FIG. 11 is a second diagram showing an example of the functional configuration of the inspection work support device. A difference from the functional configuration described with reference to FIG.

The orthoimage update determination unit 1101 uses the damage position designation unit 407 to determine the normal direction of the patch of the three-dimensional model information corresponding to the pointing position (or its vicinity) on the current orthoimage and the currently displayed patch. Compare with the orthographic projection direction of the orthoimage. If the orthorectified image update determination unit 1101 determines that the difference is large as a result of the comparison (if it determines that the difference is equal to or greater than the reference value), it uses a partial image centered at the current pointing position to update the orthorectified image. The orthorectified image generation unit 404 is instructed to generate the .

<Specific example of ortho image update processing>
Next, among the processes executed by each functional block of the inspection work support device 120,
orthorectified image generator 404,
orthorectified image display unit 405,
・Damage position specifying unit 407,
A patch index management unit 408,
Ortho image update determination unit 1101,
A specific example of the orthorectified image update process executed by will be described. FIG. 12 is a diagram showing a specific example of the orthoimage update process.

The orthorectified image update determination unit 1101 identifies the orthographic projection direction of the orthorectified image 330 currently being displayed by the orthorectified image display unit 405 (see arrow 1201). The orthorectified image update determination unit 1101 also identifies the normal direction of the patch at (or near) the current pointing position on the orthorectified image 330 (see arrows 1202 to 1204). Further, the orthoimage update determination unit 1101 determines whether or not the difference between the identified orthographic projection direction and the identified normal direction is equal to or greater than a reference value.

If the orthorectified image update determination unit 1101 determines that the value is equal to or greater than the reference value, it instructs the orthorectified image generation unit 404 to generate an orthorectified image centered on the current pointing position (see arrow 1205).

In this way, the orthorectified image update determination unit 1101 updates the orthorectified image according to the pointing position. , can display orthoimages with less distortion.

<Flow of inspection work support processing>
Next, the flow of inspection work support processing by the inspection work support device 120 will be described with reference to FIG. 13 . FIG. 13 is a third flowchart showing the flow of inspection work support processing. The difference from the second flowchart shown in FIG. 9 is steps S1301 to S1303.

In step S1301, the orthorectified image update determination unit 1101 compares the orthographic projection direction of the currently displayed orthorectified image with the normal direction of the patch corresponding to the current pointing position.

In step S1302, the orthorectified image update determination unit 1101 determines whether the difference is equal to or greater than the reference value as a result of the comparison. If it is determined in step S1302 that the difference is less than the reference value (No in step S1302), the process returns to step S904.

On the other hand, if it is determined in step S1302 that the difference is equal to or greater than the reference value (Yes in step S1302), the process proceeds to step S1303. In step S1303, the orthorectified image generation unit 404 generates an orthorectified image centered on the current pointing position. Further, the orthoimage display unit 405 displays the generated orthoimage.

As is clear from the above description, the inspection work support device according to the second embodiment synthesizes partial images obtained by photographing each part of a structure to generate a three-dimensional model image. Also, the inspection work support device according to the second embodiment generates an orthoimage of the inspection area in the three-dimensional model image. Furthermore, the inspection work support device according to the second embodiment updates the orthorectified image being displayed according to the pointing position.

As a result, according to the inspection work support device according to the second embodiment, the same effects as those of the first embodiment can be obtained, and an orthoimage with less distortion according to the pointing position can be displayed. can be done.

[Third Embodiment]
In the first and second embodiments, the inspection work support device 120 that receives image data from the inspection drone 110 is caused to execute the inspection work support program. However, the inspection work support program may be executed in another device.

FIG. 14 is a second diagram showing an example of the system configuration of the inspection work support system. As shown in FIG. 14, the inspection work support system 1400 includes inspection drones 1410_1 to 1410_n and information processing devices 1420_1 to 1420_n. The inspection work support system 1400 also has an image data storage unit 1430 , a server device 1440 , and an information processing device 1450 . In the inspection work support system 1400, the information processing devices 1420_1 to 1420_n, the image data storage unit 1430, the server device 1440, and the information processing device 1450 are connected via a network 1460 so as to be communicable.

The inspection drones 1410_1 to 1410_n transmit image data obtained by shooting to the information processing devices 1420_1 to 1420_n.

The information processing devices 1420_1 to 1420_n are devices used by owners of the inspection drones 1410_1 to 1410_n, respectively. The information processing devices 1420_1 to 1420_n store the received image data in the image data storage unit 1430. FIG.

Server device 1440 is an example of an inspection work support device. An inspection work support program is installed in the server device 1440 , and the server device 1440 functions as an inspection work support service unit 1441 by executing the program. Thereby, the server device 1440 can provide an inspection work support service. It is assumed that the inspection work support service unit 1441 has the same functional configuration as the functional configuration shown in FIG. 4 or the functional configuration shown in FIG.

The information processing device 1450 is, for example, a device used by an inspection operator. The information processing device 1450 creates an inspection work report 1451 by designating a predetermined structure and using the inspection work support service provided by the server device 1440 .

Thus, according to the inspection work support system 1400, the administrator of the image data storage unit 1430 requests the owners of the inspection drones 1410_1 to 1410_n to photograph each structure, and instructs the inspection worker to perform the inspection work. Request preparation of records. Then, the inspection worker uses the inspection work support service provided by the server device 1440 to create an inspection work record 1451 .

As a result, the manager of the image data storage unit 1430 can acquire the inspection work report.

[Other embodiments]
In the above-described first embodiment, when the pointing position is traced in the orthorectified image, the 3D point sequence is displayed in the partial image. You may

It should be noted that the technology disclosed herein can take forms such as those described below.
(Appendix 1)
a first generating unit that generates and displays a three-dimensional model image by synthesizing partial images obtained by photographing each part of a structure;
a second generating unit that generates and displays an orthorectified image of a designated inspection area in the three-dimensional model image;
a display unit that displays a partial image corresponding to a position pointed in the orthorectified image being displayed, based on the corresponding relationship between each area of the orthorectified image and the partial images that form the respective areas. Work support device.
(Appendix 2)
When the first generation unit generates the three-dimensional model image, each patch forming a mesh, three-dimensional coordinates of each vertex of the mesh, each partial image related to generation of each patch, each part 1. The inspection work support device according to appendix 1, further comprising a storage unit that stores position and orientation information of the imaging device when the image is captured in association with the information.
(Appendix 3)
The inspection work support device according to appendix 2, wherein the second generation unit generates a patch index image indicating a relationship between each region of the orthorectified image and each patch.
(Appendix 4)
The display unit identifies a patch corresponding to the pointed position in the orthoimage being displayed based on the patch index image, and identifies one partial image corresponding to the identified patch based on the correspondence relationship. The inspection work support device according to appendix 3, which is selected.
(Appendix 5)
In the partial image corresponding to the specified patch, the display unit displays a portion in which the deviation between the normal direction of the specified patch and the photographing direction of the image pickup device when the partial image is photographed is equal to or less than a predetermined set value. 5. The inspection work support device according to appendix 4, wherein the one partial image is selected by extracting the image.
(Appendix 6)
6. The inspection work support device according to Supplementary note 5, wherein the display unit selects, as the first partial image, a partial image with the shortest photographing distance among the partial images that are equal to or less than the predetermined set value.
(Appendix 7)
The display unit calculates three-dimensional position coordinates of a position pointed to in the displayed orthorectified image, and based on the calculated three-dimensional position coordinates, a portion corresponding to the pointed position in the displayed orthorectified image 3. The inspection work support device according to appendix 3, which displays three-dimensional points on the image.
(Appendix 8)
The display unit identifies a patch corresponding to the pointed position in the orthoimage being displayed based on the patch index image, and based on the three-dimensional coordinates of each vertex of the mesh in which the identified patch is formed. 8. The inspection work support device according to appendix 7, wherein the three-dimensional position coordinates of the pointed position in the displayed orthoimage are calculated.
(Appendix 9)
When the corresponding patch is changed by tracing the pointed position in the orthoimage being displayed and the partial image to be displayed is switched, the display unit switches the trace history in the partial image before switching. The inspection work support device according to appendix 8, displayed in a later partial image.
(Appendix 10)
The inspection according to appendix 9, wherein when the orthorectified image being displayed is pointed, the second generating unit generates an orthorectified image corresponding to the pointed position and updates the orthorectified image being displayed. Work support device.
(Appendix 11)
Generate and display a 3D model image by synthesizing partial images of each part of the structure,
generating and displaying an orthoimage of a designated inspection area in the three-dimensional model image;
displaying a partial image corresponding to a position pointed to in the displayed orthorectified image based on the corresponding relationship between each area of the orthorectified image and the partial images forming the respective areas;
An inspection work support method in which a computer executes processing.
(Appendix 12)
Generate and display a 3D model image by synthesizing partial images of each part of the structure,
generating and displaying an orthoimage of a designated inspection area in the three-dimensional model image;
displaying a partial image corresponding to a position pointed to in the displayed orthorectified image based on the corresponding relationship between each area of the orthorectified image and the partial images forming the respective areas;
An inspection work support program for causing a computer to execute processing.

It should be noted that the present invention is not limited to the configurations shown here, such as combinations with other elements, etc., in the configurations described in the above embodiments. These points can be changed without departing from the gist of the present invention, and can be determined appropriately according to the application form.

100: inspection work support system 110: inspection drone 120: inspection work support device 130: bridge 401: 3D synthesis processing unit 402: 3D model display unit 403: inspection area designation unit 404: orthoimage generation unit 405: orthoimage display unit 406: 3D point cloud information management unit 407: damage position designation unit 408: patch index management unit 409: damage 3D position calculation unit 410: partial image selection unit 411: partial image display unit 1101: orthoimage update determination unit 1430: image data Storage unit 1440: Server device 1441: Inspection work support service unit

Claims (11)

a first generating unit that generates and displays a three-dimensional model image by synthesizing partial images obtained by photographing each part of a structure;
a second generating unit that generates and displays an orthorectified image of a designated inspection area in the three-dimensional model image;
a display unit for displaying a partial image corresponding to a position pointed to in the orthorectified image being displayed, based on a correspondence relationship between each area of the orthorectified image and the partial images forming the respective areas. death,
When the first generation unit generates the three-dimensional model image, each patch forming a mesh, three-dimensional coordinates of each vertex of the mesh, each partial image related to generation of each patch, each part An inspection work support device further comprising a storage unit that stores position and orientation information of an imaging device when an image is taken in association with the information . 2. The inspection work support device according to claim 1 , wherein said second generation unit generates a patch index image indicating a relationship between each region of said orthorectified image and said each patch. The display unit identifies a patch corresponding to the pointed position in the orthoimage being displayed based on the patch index image, and identifies one partial image corresponding to the identified patch based on the correspondence relationship. 3. The inspection work support device according to claim 2 , which selects. In the partial image corresponding to the specified patch, the display unit displays a portion in which the deviation between the normal direction of the specified patch and the photographing direction of the image pickup device when the partial image is photographed is equal to or less than a predetermined set value. 4. The inspection work support device according to claim 3 , wherein the one partial image is selected by extracting the image. 5. The inspection work support device according to claim 4 , wherein the display unit selects, as the first partial image, a partial image with the shortest photographing distance among the partial images that are equal to or less than the predetermined set value. The display unit calculates three-dimensional position coordinates of a position pointed to in the displayed orthorectified image, and based on the calculated three-dimensional position coordinates, a portion corresponding to the pointed position in the displayed orthorectified image 3. The inspection work support device according to claim 2 , wherein three-dimensional points are displayed on the image. The display unit identifies a patch corresponding to the pointed position in the orthoimage being displayed based on the patch index image, and based on the three-dimensional coordinates of each vertex of the mesh in which the identified patch is formed. 7. The inspection work support device according to claim 6 , wherein said three-dimensional position coordinates of a position pointed in said orthorectified image being displayed are calculated. When the corresponding patch is changed by tracing the pointed position in the orthoimage being displayed and the partial image to be displayed is switched, the display unit switches the trace history in the partial image before switching. 8. The inspection work support device according to claim 7 , which is displayed in a later partial image. 9. The second generation unit according to claim 8 , wherein, when the orthorectified image being displayed is pointed, the orthorectified image is generated according to the pointed position, and the orthorectified image being displayed is updated. Inspection work support device. Generate and display a 3D model image by synthesizing partial images of each part of the structure,
generating and displaying an orthoimage of a designated inspection area in the three-dimensional model image;
displaying a partial image corresponding to a position pointed to in the displayed orthorectified image based on the corresponding relationship between each area of the orthorectified image and the partial images forming the respective areas;
A computer performs the processing,
When generating the 3D model image, each patch forming a mesh, the 3D coordinates of each vertex of the mesh, each partial image related to the generation of each patch, and an imaging device when each partial image is photographed The inspection work support method in which the computer further executes a process of correlating and storing the position and orientation information . Generate and display a 3D model image by synthesizing partial images of each part of the structure,
generating and displaying an orthoimage of a designated inspection area in the three-dimensional model image;
displaying a partial image corresponding to a position pointed to in the displayed orthorectified image based on the corresponding relationship between each area of the orthorectified image and the partial images forming the respective areas;
let the computer do the work ,
When generating the 3D model image, each patch forming a mesh, the 3D coordinates of each vertex of the mesh, each partial image related to the generation of each patch, and an imaging device when each partial image is photographed 1. An inspection work support program for causing the computer to further execute a process of correlating and storing the position and orientation information of .

Images