JP2019211257A - Inspection system - Google Patents

Abstract

To identify at which coordinate in a coordinate system of a three-dimensional model an object photographed in a two-dimensional image is located for the three-dimensional model created from a plurality of two-dimensional images.SOLUTION: An inspection system for inspecting an inspection object includes: a three-dimensional model generation section for generating a three-dimensional model of the inspection object on the basis of a plurality of images obtained by photographing the inspection object with a flight device having a camera; a photographing information acquisition section for acquiring a photographing position where an image in a three-dimensional coordinate system is photographed and a viewpoint axis direction of the camera for each of the plurality of images; an abnormality detection section for detecting an abnormality of the inspection object on the basis of the image for each of the plurality of images; an abnormal position identification section for identifying an abnormal position which is a position in the three-dimensional coordinate system according to the photographing position and the viewpoint axis direction for the detected abnormality; and a three-dimensional model display section for displaying the three-dimensional model in which the abnormal position is mapped.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an inspection system.

  In recent years, flying objects such as drones and unmanned aerial vehicles (UAVs) (hereinafter collectively referred to as “flight devices”) have begun to be used in industry. Patent Document 1 describes that a three-dimensional image is created and displayed from an image captured by a camera mounted on a drone.

JP 2018-10630 A

  A point group is extracted from a plurality of two-dimensional images to create a three-dimensional model. However, it is difficult to specify where in the three-dimensional model what is photographed in the two-dimensional image is located. Also in Patent Document 1, it is not specified at which coordinate in the three-dimensional image the degradation position is located.

  The present invention has been made in view of such a background, and for a three-dimensional model created from a plurality of two-dimensional images, what is captured in the two-dimensional image is any coordinate in the coordinate system of the three-dimensional model. One object is to provide a technique capable of specifying whether or not it is located.

  A main invention of the present invention for solving the above-described problem is an inspection system for inspecting an inspection object, wherein a flying device including a camera is configured to detect the inspection object based on a plurality of images obtained by capturing the inspection object. A three-dimensional model generation unit that generates a three-dimensional model, a shooting information acquisition unit that acquires a shooting position and a viewpoint axis direction of the camera in a three-dimensional coordinate system for each of the plurality of images; For each of a plurality of images, an abnormality detection unit that detects an abnormality of the inspection object based on the image, and the detected abnormality in the three-dimensional coordinate system according to the photographing position and the viewpoint axis direction An abnormal position specifying unit that specifies an abnormal position, and a three-dimensional model display unit that displays the three-dimensional model in which the abnormal position is mapped. And the.

  Other problems and solutions to be disclosed by the present application will be made clear by the embodiments of the present invention and the drawings.

  According to the present invention, for a three-dimensional model created from a plurality of two-dimensional images, it is possible to specify at which coordinate in the coordinate system of the three-dimensional model what is photographed in the two-dimensional image.

It is a figure showing the whole inspection system composition concerning one embodiment of the present invention. 2 is a diagram illustrating a hardware configuration example of a flying device 10. FIG. FIG. 3 is a diagram illustrating a software configuration example of a flight controller 11. 6 is a diagram illustrating a configuration example of position and orientation information stored in a position and orientation information storage unit 151. FIG. 6 is a diagram illustrating a configuration example of shooting information stored in a shooting information storage unit 152. FIG. 2 is a diagram illustrating a hardware configuration example of an inspection server 30. FIG. FIG. 3 is a diagram illustrating a software configuration example of an inspection server 30. 3 is a diagram illustrating a configuration example of a three-dimensional model storage unit 352. FIG. It is a figure which shows the structural example of the abnormality information registered into the abnormality information storage part 353. FIG. It is a figure explaining the flow of the process which image | photographs the test target object. It is a figure which shows the flow of the inspection process performed by the inspection server. It is a figure explaining the three-dimensional projection image which the three-dimensional model display part 315 displays. It is a figure which shows the flow of the process which displays the picked-up image by which the abnormal location of the designated position is image | photographed. It is another figure explaining the three-dimensional projection image which the three-dimensional model display part 315 displays.

  The contents of the embodiment of the present invention will be listed and described. A flying device according to an embodiment of the present invention has the following configuration.

[Item 1]
An inspection system for inspecting an inspection object,
A three-dimensional model generation unit that generates a three-dimensional model of the inspection object based on a plurality of images obtained by photographing a flying object with a camera;
For each of the plurality of images, a photographing information acquisition unit that acquires a photographing position where the image is photographed in a three-dimensional coordinate system and a viewpoint axis direction of the camera;
For each of the plurality of images, an abnormality detection unit that detects an abnormality of the inspection object based on the images;
An abnormal position specifying unit that specifies an abnormal position that is a position in the three-dimensional coordinate system according to the photographing position and the viewpoint axis direction with respect to the detected abnormality,
A three-dimensional model display unit that displays the three-dimensional model in which the abnormal position is mapped;
An inspection system comprising:
[Item 2]
The inspection system according to item 1, wherein
The abnormal position specifying unit specifies, as the abnormal position, coordinates at which a straight line from the shooting position toward the abnormal point intersects the three-dimensional model based on the shooting position and the viewpoint axis direction;
Inspection system characterized by
[Item 3]
The inspection system according to item 1, wherein
An abnormal position storage unit that stores the image in association with the abnormal position;
An input unit for receiving designation of a position on the three-dimensional model;
An image specifying unit for specifying the image corresponding to the specified position;
An inspection system comprising:
[Item 4]
An inspection system for inspecting an inspection object,
A three-dimensional model generation unit that generates a three-dimensional model of the inspection object based on a plurality of images obtained by photographing a flying object with a camera;
For each of the plurality of images, a photographing information acquisition unit that acquires a photographing position where the image is photographed in a three-dimensional coordinate system and a viewpoint axis direction of the camera;
For each of the plurality of images, an intersection position calculation unit that calculates an intersection position that intersects the three-dimensional model in the viewpoint axis direction from the photographing position;
An image specifying unit that receives a specified position on the three-dimensional model and specifies the image in which the specified position is captured according to a distance between the specified position and the intersection position;
An inspection system comprising:
[Item 5]
The inspection system according to item 4, wherein
For each of the plurality of images, an abnormality detection unit that detects an abnormality of the inspection object based on the images;
An abnormal position specifying unit that specifies an abnormal position that is a coordinate at which a straight line from the shooting position toward the abnormal point intersects the three-dimensional model based on the shooting position and the viewpoint axis direction;
An inspection system comprising:

== Overview / Overall structure ==
FIG. 1 is a diagram showing an overall configuration example of an inspection system according to an embodiment of the present invention. The inspection system of the present embodiment creates a three-dimensional model of the inspection object 1 based on a plurality of images related to the inspection object 1 photographed by the flying device 10. Further, in the inspection system of the present embodiment, a photographed image is analyzed to detect an abnormal part, and the detected abnormal part is mapped on a three-dimensional model. The inspection object 1 may be a concrete structure such as a slope or a dam, a steel structure such as a steel tower or an iron bridge, or a solar panel or a building. In the present embodiment, a building is assumed as the inspection target unit 1. The inspection object 1 may be anything as long as it can be photographed, and may be, for example, an animal or a car, or may be a disaster area at the time of a disaster.

  The inspection system of the present embodiment includes a flying device 10 that images the inspection object 1 and an inspection server 30 that analyzes an image captured by the flying device 10. The flying device 10 and the inspection server 30 are connected to each other via a communication network 50 so that they can communicate with each other. In the present embodiment, the communication network 50 is assumed to be the Internet. The communication network 50 is constructed by, for example, a wireless communication path, a cellular phone line network, a satellite communication path, a public telephone line network, a dedicated line network, Ethernet (registered trademark), or the like.

(Flying device 10)
FIG. 2 is a diagram illustrating a hardware configuration example of the flying device 10. The flying device 10 includes a propeller 18, a propulsion mechanism connected to the propeller 18 via an ESC (Electronic Speed Controller) 16 (a motor 17 is assumed in the present embodiment), and a flight controller 11 that controls them. Prepare.

  The flying device 10 includes a camera 12 that captures a part or all of the inspection object 1. In the present embodiment, the camera 12 is fixed to the body. In addition, the camera 12 includes a vertically downward lens and captures an image directly below in the vertical direction. Therefore, when the attitude of the flying device 10 is fixed, the viewpoint axis (optical axis) 121 of the camera 12 is also fixed. In the present embodiment, it is assumed that the camera 12 captures an RGB image capturing visible light, but a thermal image capturing infrared light may be captured, and both the RGB image and the thermal image may be captured simultaneously or You may make it image | photograph sequentially. Further, instead of the camera 12 or in addition to the camera 12, various sensors such as a human sensor may be provided.

  The flight controller 11 can include one or more processors 101 such as a programmable processor (in this embodiment, a central processing unit (CPU) is assumed). The flight controller 11 includes a memory 102 and can access the memory 102. Memory 102 stores logic, code, and / or program instructions that are executable by flight controller 11 to perform one or more steps. The memory 102 may include a separable medium such as an SD card or a random access memory (RAM) or an external storage device. Data acquired from the camera 12, sensor, or the like may be directly transmitted to and stored in the memory 102.

  The flight controller 11 also includes various sensors 103. In the present embodiment, the sensors 103 may include, for example, an inertial sensor (acceleration sensor, gyro sensor), a GPS (Global Positioning System) sensor, a proximity sensor (eg, a rider), or a vision / image sensor (eg, a camera). .

  The flight controller 11 is configured to send and / or receive data from one or more external devices (eg, a transceiver, a terminal, a display device, or other remote controller, etc.). It is possible to communicate with the transmitted / received unit 14. The transmission / reception unit 14 can use any appropriate communication means such as wired communication or wireless communication. In the present embodiment, the transmission / reception unit 14 mainly communicates with the inspection server 30. The transmission / reception unit 14 is, for example, a local area network (LAN), a wide area network (WAN), infrared, wireless, WiFi, point-to-point (P2P) network, telecommunication network, cloud communication, etc. One or more of these can be used. The transmission / reception unit 14 may transmit and / or receive one or more of data acquired by sensors, a processing result generated by the flight controller, predetermined control data, a user command from a terminal or a remote controller, and the like. it can.

  FIG. 3 is a diagram illustrating a software configuration example of the flight controller 11. The flight controller 11 includes an instruction receiving unit 111, a flight control unit 112, a position / orientation information acquisition unit 113, an imaging processing unit 114, an imaging information transmission unit 115, a position / orientation information storage unit 151, an imaging information storage unit 152, a GPS sensor 104, An atmospheric pressure sensor 105, a temperature sensor 106, and an acceleration sensor 107 are provided.

  The instruction receiving unit 111, the flight control unit 112, the position / orientation information acquisition unit 113, the imaging processing unit 114, and the imaging information transmission unit 115 are realized by the processor 101 executing a program stored in the memory 102. . In addition, the position / orientation information storage unit 151 and the shooting information storage unit 152 are realized as storage areas provided by the memory 102.

  The instruction receiving unit 111 receives various commands for instructing the operation of the flying device 10 (hereinafter referred to as a flight operation command). In the present embodiment, it is assumed that the instruction receiving unit 111 receives a flight operation command from the inspection server 30, but the flight operation command may be received from a transceiver such as a transmitter.

  The flight control unit 112 controls the operation of the flying device 10. For example, the flight control unit 112 adjusts the spatial arrangement, speed, and / or acceleration of the flying device 10 having six degrees of freedom (translational motions x, y, and z and rotational motions θx, θy, and θz). The motor 17 is controlled via the ESC 16. The propeller 18 is rotated by the motor 17 to generate lift of the flying device 10. Further, the flight control unit 112 can control one or more of the states of the mounting unit and sensors. In the present embodiment, the flight control unit 112 controls the operation of the flying device 10 in accordance with the flight operation command received by the instruction receiving unit 111. Further, the flight control unit 112 can perform various controls so that the flying device 10 continues to fly without depending on a command so as to enable autonomous flight.

  The position / orientation information acquisition unit 113 acquires information indicating the current position and orientation of the flying device 10 (hereinafter referred to as position / orientation information). In the present embodiment, the position and orientation information includes the position of the flying device 10 on the map (represented by latitude and longitude), the flight altitude of the flying device 10, and the x, y, and z axes of the flying device 10. The slope with respect to is assumed to be included. Note that the x, y, and z axes of the flying device 10 are coordinate axes orthogonal to each other, and can represent the plane coordinates and the vertical direction of the aircraft.

  The sensors 103 include the GPS sensor 104, and the position / orientation information acquisition unit 113 can calculate the position of the flying device 10 on the map from the radio wave received by the GPS sensor 104 from the GPS satellite.

  The sensors 103 include an atmospheric pressure sensor 105 and a temperature sensor 106, and the position / orientation information acquisition unit 113 detects the atmospheric pressure (hereinafter referred to as a reference atmospheric pressure) measured by the atmospheric pressure sensor 105 before the flight and during the flight. Based on the difference between the atmospheric pressure measured by the atmospheric pressure sensor 105 (hereinafter referred to as the current atmospheric pressure) and the temperature measured by the temperature sensor 106 during the flight, the flight altitude of the flying device 10 can be calculated.

  Further, the sensors 103 include a triaxial acceleration sensor 107, and the position / orientation information acquisition unit 113 can obtain the attitude of the flying device 10 based on the output from the acceleration sensor 107. Further, the optical axis 121 (viewpoint axis) of the camera 12 can be determined from the attitude of the flying device 10.

  The position and orientation information acquisition unit 113 includes a position (latitude and longitude) on the map of the flying device 10 acquired using the GPS sensor 104, a flight altitude of the flying device 10 acquired using the atmospheric pressure sensor 105 and the temperature sensor 106, The position and orientation information in which the attitude of the flying device 10 acquired using the acceleration sensor 107 is set is stored in the position and orientation information storage unit 151 of the memory 14.

  FIG. 4 is a diagram illustrating a configuration example of the position / orientation information stored in the position / orientation information storage unit 151. As shown in FIG. 4, the position / orientation information storage unit 151 includes the latitude / longitude indicating the current position of the flying device 10, the flight altitude calculated as described above, the reference atmospheric pressure measured before the flight, and measured during the flight. The current atmospheric pressure, the temperature measured during the flight, and the attitude of the flying device 10 acquired using the acceleration sensor 107 (the inclination of the optical axis 121 of the camera 12) are stored.

  The photographing processing unit 114 controls the camera 12 to photograph a part or all of the inspection object 1 and acquires a photographed image photographed by the camera 12. The imaging processing unit 114 performs imaging at a preset timing. For example, the imaging processing unit 114 can perform imaging every predetermined time (for example, any time such as 5 seconds or 30 seconds can be designated). Note that the imaging processing unit 114 may perform imaging in response to an instruction from the inspection server 30.

  The imaging processing unit 114 stores the acquired captured image in the imaging information storage unit 152. The photographing processing unit 114 is attached to the photographed image, and the photographing date and time, the latitude and longitude on the map of the flying device 10 at that time (imaging position), the flying altitude (imaging altitude) of the flying device 10 at that time, the flying device 10 (The inclination of the optical axis 121 of the camera 12) is also stored in the photographing information storage unit 152. An image with such information attached is referred to as shooting information.

  FIG. 5 is a diagram illustrating a configuration example of shooting information stored in the shooting information storage unit 152. The shooting information stored in the shooting information storage unit 152 includes shooting date and time, shooting position, shooting altitude, tilt, and image data. The shooting information can be stored as a file on a file system, for example. Further, the shooting date / time, shooting position, shooting altitude, and tilt may be stored as, for example, Exif (Exchangeable image file format) information of a JPEG (Joint Photographic Experts Group) image file, or image data as a file. A record may be stored in the system, and a record in which the shooting date / time, shooting position, shooting altitude / tilt, and file name are associated with each other is registered in the database. Further, the shooting information may include characteristics of the camera 12 such as a focal length and a shutter speed.

  The imaging information transmission unit 115 transmits the image captured by the camera 12 to the inspection server 30. In the present embodiment, the shooting information transmission unit 115 transmits shooting information in which a shooting date and time, a shooting position, a shooting altitude, and a tilt are added to a shooting image to the inspection server 30.

(Inspection server 30)
FIG. 6 is a diagram illustrating a hardware configuration example of the inspection server 30. The inspection server 30 includes a CPU 301, a memory 302, a storage device 303, a communication device 304, an input device 305, and an output device 306. The storage device 303 is, for example, a hard disk drive, a solid state drive, or a flash memory that stores various data and programs. The communication device 304 communicates with other devices via the communication network 50. The communication device 304 includes, for example, an adapter for connecting to Ethernet (registered trademark), a modem for connecting to a public telephone line network, a wireless communication device for performing wireless communication, a USB connector for serial communication, an RS232C connector, and the like It is comprised including. The input device 305 is, for example, a keyboard, mouse, touch panel, button, or microphone that inputs data. The output device 306 is, for example, a display, a printer, or a speaker that outputs data.

  FIG. 7 is a diagram illustrating a software configuration example of the inspection server 30. The inspection server 30 includes a flight control unit 311, an imaging information reception unit 312, a 3D model creation unit 313, an abnormality detection unit 314, a 3D model display unit 315, a captured image display unit 316, an imaging information storage unit 351, and a 3D model. A storage unit 352 and an abnormality information storage unit 353 are provided.

  Note that the flight control unit 311, the imaging information reception unit 312, the 3D model creation unit 313, the abnormality detection unit 314, the 3D model display unit 315, and the captured image display unit 316 are stored in the storage device 303 by the CPU 301 included in the inspection server 30. The imaging information storage unit 351, the three-dimensional model storage unit 352, and the abnormality information storage unit 353 are realized by reading the stored program into the memory 302, and the memory 302 and the storage device 303 included in the inspection server 30. This is realized as a part of the storage area to be provided.

  The flight control unit 311 controls the flight device 10 to fly. In the present embodiment, the flight control unit 311 operates the flying device 10 by transmitting a flight operation command to the flying device 10. The flight control unit 311 can receive, for example, designation of a flight path on a map from an operator and transmit a flight operation command so as to fly on the accepted flight path. Further, the flight control unit 311 continuously makes the flying device 10 fly around the inspection object 1 so that at least a part of the imaging region overlaps in order to create a three-dimensional model of the inspection object 1 as described later. The flying device 10 is controlled so that it can be photographed. In the present embodiment, the flying device 10 is controlled by the flight control unit 311 of the inspection server 30. For example, the flying device 10 is autonomous by software (data) for automatic flight stored in advance. It is possible to use a method that is controlled automatically, a method that is controlled (operated) by an operator's manual operation, or a combination thereof.

  The shooting information receiving unit 312 receives shooting information transmitted from the flying device 10. The shooting information receiving unit 312 stores the received shooting information in the shooting information storage unit 351. In the present embodiment, the configuration of the shooting information storage unit 351 is the same as that of the shooting information storage unit 152 of the flying device 10 shown in FIG. Note that the imaging information storage unit 351 included in the inspection server 30 may store imaging information in association with information specifying the flying device 10 that is the transmission source of imaging information.

  The three-dimensional model creation unit 313 creates a three-dimensional model that represents a three-dimensional structure from a plurality of captured images. The three-dimensional model creation unit 313 can create a three-dimensional model using a so-called photogrammetry (photogrammetry) technique. In the present embodiment, the world coordinate system of the three-dimensional model is represented by latitude, longitude, and altitude. The position and viewpoint direction of the camera 12 in the world coordinate system can be indicated by the shooting position, the shooting altitude, and the tilt of the optical axis 121 included in the shooting information. The three-dimensional model creation unit 313 extracts feature points from the image data included in the shooting information, and extracts feature points extracted from the plurality of image data based on the shooting position, shooting height, and inclination included in the shooting information. Association is performed, and a three-dimensional point group (also referred to as a point cloud) in the world coordinate system is acquired. In the present embodiment, the three-dimensional model is assumed to be composed of a three-dimensional point group, but the three-dimensional model creation unit 313 can also generate surface data such as a polygon based on the point group. Further, the three-dimensional model creation unit 313 can generate an ortho image based on the three-dimensional model and the captured image.

  The three-dimensional model creation unit 313 registers the created three-dimensional model (a three-dimensional point group in the present embodiment) in the three-dimensional model storage unit 352. FIG. 8 is a diagram illustrating a configuration example of the three-dimensional model storage unit 352. As shown in the figure, the coordinates for each point constituting the three-dimensional point group are registered in the three-dimensional model storage unit 352. When the 3D model creation unit 313 creates surface data such as a polygon based on the 3D point group, the 3D model storage unit 352 adds or replaces the point coordinates, The surface data may be stored.

  The abnormality detection unit 314 detects the abnormality of the inspection object 1 by analyzing the captured image from the flying device 10. For example, the abnormality detection unit 314 performs learning using machine learning such as a neural network using, as a teacher signal, an image obtained by photographing the inspection object 1 in which an abnormality has occurred in the past, and obtains a photographed image received from the flying device 10. Abnormality can be determined as an input signal. In addition, the abnormality detection unit 314 stores a normal image in the memory 302 in advance, compares the normal image with the captured image, and recognizes an area constituted by pixels having a difference greater than or equal to a predetermined value. However, it is possible to determine that the area of the region is equal to or greater than a predetermined value as abnormal. Note that the abnormality detection unit 314 can detect an abnormal part from the image using a known technique.

  The abnormality detection unit 314 specifies the position of the world coordinate system for the detected abnormal part on the captured image. For example, the anomaly detection unit 314 has, for each of the points included in the three-dimensional point group, in the direction indicated by the inclination included in the shooting information from the camera installed at the shooting position and shooting height included in the shooting information. Determine the position on the image at the time of shooting, determine whether the point on the image is included in the area detected as an abnormal location, whether the point constitutes an abnormal location, It is possible to specify the coordinates of the points constituting the abnormal part as the position of the abnormal part. The abnormality normal detection unit 315 registers information related to the detected abnormality (hereinafter referred to as abnormality information) in the abnormality information storage unit 353.

  FIG. 9 is a diagram illustrating a configuration example of abnormality information registered in the abnormality information storage unit 353. As shown in the figure, the abnormality information includes a coordinate position (abnormal position) on the world coordinate system indicating the abnormal part and information for specifying photographing information related to an image obtained by photographing the abnormal part (for example, a JPEG file). And a position where an abnormality is detected on the image (hereinafter referred to as an abnormal image position). In the example of FIG. 8, an example in which the coordinates of two vertices indicating a rectangle are included in the image abnormal position is shown, but the image abnormal position may include only one coordinate, for example. Three or more coordinates constituting a polygon may be included, or center coordinates representing an ellipse and two radii may be included. That is, the image abnormal position may be information representing a point or a geometric figure. In addition to the above-described example, for example, a portion showing the abnormality may be directly drawn at an abnormal position in the image. In this case, the coordinate position related to the abnormal position is not used, and for example, the information related to the captured image may be directly associated with the abnormal position.

  The 3D model display unit 315 displays an image obtained by projecting the 3D model created by the 3D model creation unit 313 onto a plane (hereinafter referred to as a 3D projection image). The three-dimensional model display unit 315 may display the three-dimensional projection image with a wire frame, or may map the captured image to the three-dimensional model. In addition, the three-dimensional model display unit 315 displays the position of the camera 12 and the direction of the viewpoint axis superimposed on the three-dimensional model for each captured image that created the three-dimensional model. In the present embodiment, for ease of explanation, the position of the camera 12 and the direction of the viewpoint axis are displayed. However, the setting of whether or not to display and the switching method thereof are appropriately selected according to the application and needs. Can be changed.

== Processing ==
Hereinafter, processing in the inspection system of the present embodiment will be described.

== Shooting process ==
FIG. 10 is a diagram for explaining the flow of processing for photographing the inspection object 1. The imaging process shown in FIG. 10 is periodically executed while the flying device 10 is flying.

  In the flying device 10, the imaging processing unit 114 controls the camera 12 to acquire a captured image captured by the camera 12 (S201), and the position / orientation information acquisition unit 113 includes a GPS sensor 104, an atmospheric pressure sensor 105, and a temperature sensor. Based on the signals from 106 and the acceleration sensor 107, the photographing position, the photographing altitude and the posture are obtained (S202). The shooting information transmission unit 115 creates shooting information in which the current date and time (shooting date and time), shooting position, shooting altitude, and posture are added to the shot image acquired from the camera 12, and transmits the shooting information to the inspection server 30 (S203). .

  In the inspection server 30, the imaging information reception unit 312 receives imaging information transmitted from the flying device 10 (S204), and registers the received imaging information in the imaging information storage unit 351 (S205).

  As described above, the images photographed by the flying device 10 are sequentially registered in the photographing information storage unit 351 together with the photographing date and time, the photographing position, the photographing altitude, and the attitude of the flying device 10 (the inclination of the optical axis 121). It will be.

== Inspection processing ==
FIG. 11 is a diagram showing a flow of inspection processing executed by the inspection server 30. In the present embodiment, it is assumed that the processing shown in FIG. 11 is performed after all shooting by the flying device 10 is completed, but may be performed while the flying device 10 continues shooting.

  The three-dimensional model creation unit 313 creates a three-dimensional model based on the shooting information registered in the shooting information storage unit 351 (S211). As for the creation process of the 3D model, it is assumed that the 3D model is created by a general photogrammetry (photogrammetry) technique. For example, the three-dimensional model creation unit 313 searches for corresponding points for two pairs of shooting information that are consecutive in the order of shooting times, and for the corresponding points that are found, the world of the corresponding points according to the flight position, flight altitude, and posture. The position in the coordinate system can be specified. Note that the creation of the three-dimensional model is an example, and in addition to the above, it can be created by various methods.

  The 3D model creation unit 313 registers the created 3D model in the 3D model storage unit 352 (S212). In the present embodiment, it is assumed that the three-dimensional model is composed of a three-dimensional point group. Therefore, the coordinates of the corresponding point in the world coordinate system can be registered in the three-dimensional model storage unit 352.

  Next, the abnormality detection unit 314 performs the following processing for each piece of shooting information registered in the shooting information storage unit 351. That is, the abnormality detection unit 314 analyzes the image data included in the imaging information and inspects whether there is an abnormality in the inspection object 1 (S213). When an abnormality of the inspection object 1 is detected from the image data (S214: YES), the abnormality detection unit 314 indicates the light indicated by the inclination of the shooting information from the coordinates of the shooting position and the shooting altitude of the shooting information in the world coordinate system. The coordinates where the straight line extending in the direction of the axis 121 (viewpoint axis) and the three-dimensional model intersect are specified as the position where the abnormality has occurred (abnormal position) (S215). It should be noted that the abnormality detection unit 314 passes through the nearest point when the optical axis 121 from the camera 12 does not intersect any of the points included in the three-dimensional point group, and is the closest point parallel to the optical axis 121. May be determined as an abnormal position. The abnormality detection unit 314 associates information for specifying shooting information (for example, a file name) and a location where an abnormality is found on the image data (image abnormal position) in association with the abnormal position, and stores abnormality information. Registered in the unit 353 (S216).

  After performing the above processing for each piece of shooting information, the 3D model display unit 315 displays an image obtained by projecting the 3D model (S217). FIG. 12 is a diagram for explaining a three-dimensional projection image displayed by the three-dimensional model display unit 315. In the example of FIG. 12, a three-dimensional projection image 41 obtained by mapping a photographed image with respect to a three-dimensional model created by the three-dimensional model creation unit 313 based on a plurality of photographed images is displayed on the screen 40. The three-dimensional model display unit 315 can create a three-dimensional projection image 41 by projecting a point group registered in the three-dimensional model storage unit 352, for example.

  Next, the three-dimensional model display unit 315 superimposes and displays a figure indicating the camera 12 that has captured the captured image on the three-dimensional projection image 41. Specifically, the three-dimensional model display unit 315 has the camera 12 tilted in the direction indicated by the tilt from the shooting position and the shooting altitude included in the shooting information for each shooting information registered in the shooting information storage unit 351. A graphic representing the image is displayed so as to be superimposed on the three-dimensional model. In the example of FIG. 12, a cone 42 indicating the position of the camera 12 and the direction of the viewpoint axis is displayed superimposed on the three-dimensional projection image 41. As described above, in the present embodiment, the position of the camera 12 and the direction of the viewpoint axis are displayed for ease of explanation. It can be appropriately changed according to the above.

  The three-dimensional model display unit 315 superimposes and displays a graphic indicating the abnormal part on the three-dimensional projection image 41 (S219). Specifically, the three-dimensional model display unit 315 displays a predetermined graphic at the coordinates indicated by the abnormal position for each abnormality information registered in the abnormality information storage unit 353. In the example of FIG. 12, the mark 43 is displayed superimposed on the three-dimensional projection image 41. In the example of FIG. 12, only one mark 43 is displayed. However, when a plurality of abnormal locations are registered in the abnormality information storage unit 353, a plurality of marks 43 are displayed. .

  As described above, a three-dimensional model can be created based on a plurality of images captured by the flying device 10, and an abnormal location detected from the captured image can be mapped and displayed on the three-dimensional model.

== Confirmation processing ==
FIG. 13 is a diagram illustrating a flow of processing for displaying a captured image in which an abnormal portion at a specified position is captured. In the screen 40, when the user specifies the position of the abnormal part whose details are to be confirmed by clicking the mark 43, for example, the captured image display unit 316 receives the specification of the position (S231), and the received position. The abnormality information corresponding to is retrieved from the abnormality information storage unit 353 (S232). The captured image display unit 316 reads out the shooting information indicated by the image included in the searched abnormality information from the shooting information storage unit 351 (S233), and outputs the image data included in the read out shooting information to the screen 40 (S234). ).

  As described above, the captured image display unit 316 can display the captured image obtained by capturing the position instructed by the user on the screen 40.

== Effect ==
As described above, according to the inspection system of the present embodiment, the position on the photographed image can be associated with the position on the three-dimensional model based on the photographing position, the photographing altitude, and the posture. Therefore, the abnormal part detected from the image can be captured not only as an image but also as an abnormal part on the three-dimensional model. In addition, when a position on the three-dimensional model is specified, it is possible to easily specify an image obtained by photographing the abnormal portion by specifying the abnormal information corresponding to the specified position. And a detailed two-dimensional image can be provided quickly and accurately.

  As mentioned above, although one Embodiment of this invention was described, the said embodiment is for making an understanding of this invention easy, and is not for limiting and interpreting this invention. The present invention can be changed and improved without departing from the gist thereof, and the present invention includes equivalents thereof.

  For example, in the present embodiment, the camera 12 is fixed to the lower part of the airframe. However, the present invention is not limited to this, and the camera 12 may be movably mounted via a gimbal. In this case, the shooting information includes the shooting direction of the camera 12 in addition to the tilt of the flying device 10.

  In the present embodiment, the photographing altitude is obtained using the atmospheric pressure sensor 105 and the temperature sensor 106. However, the present invention is not limited to this, and the photographing altitude may be obtained using a known method.

  In this embodiment, the shooting information from the flying device 10 is transmitted to the inspection server 30 every time shooting is performed by the camera 12. However, the flying device 10 stores the shooting information in the shooting information storage unit 152. In addition, the shooting information stored in the shooting information storage unit 152 may be transmitted to the inspection server 30 periodically during the flight or once after the flight.

  Further, in the present embodiment, the user selects the position of the abnormal location on the screen 40, the abnormal location corresponding to the selected position is searched, and the captured image corresponding to the searched abnormal location is displayed. The abnormality information may be associated with the mark 43 and the abnormality information may be specified from the selected mark 43.

  In the present embodiment, the user can select only the position of the abnormal location on the screen 40, but may select any position (coordinates) other than the abnormal location. In this case, as shown in FIG. 14, images (44a, 44b, 44c) corresponding to the selected locations (for example, 43a, 43b, 43c) are displayed. In this case, the captured image display unit 316 first sets a straight line from the flight position and flight altitude coordinates to the designated coordinates for each of the captured information registered in the captured information storage unit 351. Whether or not the designated coordinates can be photographed in the image data related to the photographing information is determined based on whether or not the angle is within the angle of view of the camera 12 when the camera 12 is directed from the coordinates to the viewpoint axis. . Next, the photographic image display unit 316 has the above straight line passing through another point of the three-dimensional model (or less than a predetermined distance from the other point) for the photographic information determined that the designated coordinates can be photographed. Whether or not shooting of the designated coordinates is hindered is determined by whether or not it passes through the polygon of the three-dimensional model. Finally, the captured image display unit 316 can display image data for the captured information that is determined not to prevent the capturing of the designated coordinates.

DESCRIPTION OF SYMBOLS 10 Flight apparatus 11 Flight controller 12 Camera 14 Transmission / reception part 16 ESC
DESCRIPTION OF SYMBOLS 17 Motor 18 Propeller 30 Inspection server 50 Communication network 101 Processor 102 Memory 103 Sensors 104 GPS sensor 105 Atmospheric pressure sensor 106 Temperature sensor 107 Acceleration sensor 111 Instruction receiving part 112 Flight control part 113 Position and orientation information acquisition part 114 Shooting processing part 115 Shooting information Transmission unit 151 Position and orientation information storage unit 152 Imaging information storage unit 301 CPU
302 Memory 303 Storage Device 304 Communication Device 305 Input Device 306 Output Device 311 Flight Control Unit 312 Imaging Information Reception Unit 313 3D Model Creation Unit 314 Abnormality Detection Unit 315 3D Model Display Unit 316 Captured Image Display Unit 351 Imaging Information Storage Unit 352 3D model storage unit 353 Abnormal information storage unit

Claims (5)

An inspection system for inspecting an inspection object,
A three-dimensional model generation unit that generates a three-dimensional model of the inspection object based on a plurality of images obtained by photographing a flying object with a camera;
For each of the plurality of images, a photographing information acquisition unit that acquires a photographing position where the image is photographed in a three-dimensional coordinate system and a viewpoint axis direction of the camera;
For each of the plurality of images, an abnormality detection unit that detects an abnormality of the inspection object based on the images;
An abnormal position specifying unit that specifies an abnormal position that is a position in the three-dimensional coordinate system according to the photographing position and the viewpoint axis direction with respect to the detected abnormality,
A three-dimensional model display unit that displays the three-dimensional model in which the abnormal position is mapped;
An inspection system comprising: The inspection system according to claim 1,
The abnormal position specifying unit specifies, as the abnormal position, coordinates at which a straight line from the shooting position toward the abnormal point intersects the three-dimensional model based on the shooting position and the viewpoint axis direction;
Inspection system characterized by The inspection system according to claim 1,
An abnormal position storage unit that stores the image in association with the abnormal position;
An input unit for receiving designation of a position on the three-dimensional model;
An image specifying unit for specifying the image corresponding to the specified position;
An inspection system comprising: An inspection system for inspecting an inspection object,
A three-dimensional model generation unit that generates a three-dimensional model of the inspection object based on a plurality of images obtained by photographing a flying object with a camera;
For each of the plurality of images, a photographing information acquisition unit that acquires a photographing position where the image is photographed in a three-dimensional coordinate system and a viewpoint axis direction of the camera;
For each of the plurality of images, an intersection position calculation unit that calculates an intersection position that intersects the three-dimensional model in the viewpoint axis direction from the photographing position;
An image specifying unit that receives a specified position on the three-dimensional model and specifies the image in which the specified position is captured according to a distance between the specified position and the intersection position;
An inspection system comprising: The inspection system according to claim 4,
For each of the plurality of images, an abnormality detection unit that detects an abnormality of the inspection object based on the images;
An abnormal position specifying unit that specifies an abnormal position that is a coordinate at which a straight line from the shooting position toward the abnormal point intersects the three-dimensional model based on the shooting position and the viewpoint axis direction;
An inspection system comprising: