JP2020005186A - Image display system and method - Google Patents

Abstract

To provide a technology capable of easily selecting an appropriate image and reducing time and effort of user's work when selecting the image for a specific application from an image group photographed by a camera.SOLUTION: In a computer system 1 in this image display system, a photographed image group 201 is inputted, a list of the image group 201 is displayed on a list screen 401, and a first image selected from the image group 201 is displayed on an individual screen 402 based on operation of a user. In the computer system 1, a spatial positional relationship in a set of the first image and candidate images therearound is determined, an image adjacent to the first image is determined on the basis of determination of an overlapping state regarding a photographing range, and based on the operation of the user, the image adjacent to the first image is selected and displayed on the individual screen 402.SELECTED DRAWING: Figure 3

Description

  The present invention relates to a technique for processing and displaying an image group. Further, the present invention relates to a technique which is useful when applied to, for example, support for structural deterioration inspection.

  In various applications, a user captures an image group (moving image or a plurality of still images) using a camera and inputs the captured image group to a computer such as a PC. The user displays an image group processed by the computer and individual images selected from the image group on a display screen. The user confirms a group of images and individual images on the display screen, and performs an operation according to the application. The work includes, for example, a general photographic image editing work.

  As an example of the use of the image group described above, a system or function for inspecting or diagnosing a state of deterioration of the surface of a structure from an image obtained by photographing the surface of the structure (in some cases, described as a structure deterioration inspection or the like). Is). A user who is a worker who performs maintenance and inspection work captures the surface of a structure to be maintained and inspected with a camera manually or using a drone or a robot. The structures include various buildings and infrastructure facilities. The user inputs a group of images captured by the camera to a computer, and displays a group of images processed by the computer on a display screen. The user selects an individual image to be inspected from the image group on the display screen and displays the selected image on the display screen. The user performs, for example, visual inspection work. In this case, the user looks at individual images on the display screen and visually confirms and detects the presence or absence and location of a state such as cracking, corrosion, or peeling (which may be collectively referred to as deterioration).

  As another example of the use of the image group, a three-dimensional model of the structure is generated and acquired from an image obtained by photographing the surface of the structure based on a known process such as SFM (Structure For Motion). A system and a function to be displayed on the display screen (sometimes referred to as a structure three-dimensional model generation) are included.

  Japanese Patent Application Laid-Open No. 2000-90232 (Patent Document 1) is an example of a prior art related to the image display. Patent Document 1 discloses a panoramic image synthesizing apparatus or the like, in which a digital camera divides a subject image into a plurality of images so that a part of the images overlaps, captures the images, and combines the captured images to generate a panoramic image. It describes a step of generating an image, a step of simultaneously displaying an entire preview and a partial preview of a panoramic image in the same window, and the like.

  Japanese Patent Application Laid-Open No. 11-132961 (Patent Document 2) is an example of a prior art related to the above-described structural deterioration inspection. Patent Document 2 discloses an inspection apparatus for a structure in which an inspection object is divided into a plurality of parts, an image is taken, and image data obtained by adding corresponding position information and imaging information to a plurality of imaged image data is stored. It is described that, by designating the inspection position of the inspection object, image data is selected and the inspection position and a plurality of image data around the inspection position are displayed as a connected image.

JP 2000-90232 A JP-A-11-132961

  In a conventional image display system, when a user selects an individual image for a predetermined application from a group of images on a display screen, there are cases where it is difficult to select a suitable image and cases where work is laborious. For example, in the case of use for structural deterioration inspection support, a user needs to select an inspection target image from a large number of images on a display screen. At this time, it may be difficult to select a suitable image having few overlapping portions between the images, and the deterioration inspection work is troublesome.

  An object of the present invention is to provide an image display technology, in which when selecting an image for a predetermined application from a group of images taken by a camera, a suitable image can be easily selected, and the user's work is reduced. To provide technologies that can be used.

  A typical embodiment of the present invention is an image display system, having the following configuration. An image display system according to an embodiment is an image display system configured by a computer system, wherein the computer system inputs an image group including a plurality of images having different shooting dates and times, positions, and directions, and Displaying a list of image groups on a list screen, displaying a first image selected from the image group on an individual screen based on a user operation, and spatially surrounding the first image and the first image; Determining an adjacent image with respect to the first image based on a determination of a spatial positional relationship in a set with a candidate image and a determination of an overlapping state with respect to a shooting range, based on an operation of the user on the individual screen. Then, the adjacent image to the first image is selected as a second image, and the second image is displayed as a new first image.

  According to the representative embodiment of the present invention, it is possible to easily select a suitable image when selecting an image for a predetermined use from a group of images taken by a camera with respect to an image display technique. In addition, the work of the user can be reduced.

FIG. 1 is a diagram showing a configuration of a structure deterioration inspection support system including an image display system according to an embodiment of the present invention. FIG. 3 is a diagram illustrating a configuration of cooperation between a drone and a computer system in the embodiment. FIG. 1 is a diagram illustrating a configuration of an image display system according to an embodiment. FIG. 3 is a diagram illustrating a processing flow of the image display system according to the embodiment. FIG. 3 is a diagram illustrating a first example of an aerial photographing method of a structure in the embodiment. FIG. 4 is a diagram illustrating a second example of an aerial photographing method of a structure in the embodiment. FIG. 3 is a diagram illustrating a shooting range in the embodiment. FIG. 4 is a diagram illustrating overlap between images in the aerial photographing traveling direction in the embodiment. FIG. 7 is a diagram illustrating an overlap between images in an image planar view in the embodiment. FIG. 4 is a diagram illustrating an example of an adjacent image and an overlap in the embodiment. FIG. 7 is a diagram illustrating an overlapping state in a shooting range in the embodiment. FIG. 8 is a diagram illustrating an overlapping state in adjacent images in the embodiment. FIG. 3 is a diagram illustrating SFM in the embodiment. FIG. 7 is a diagram illustrating a first example of a list screen in the embodiment. FIG. 5 is a diagram illustrating a first example of an individual screen in the embodiment. FIG. 9 is a diagram illustrating a second example of an individual screen in the embodiment. FIG. 13 is a diagram illustrating a third example of an individual screen in the embodiment. FIG. 14 is a diagram illustrating a fourth example of an individual screen in the embodiment. FIG. 14 is a diagram illustrating a fifth example of an individual screen in the embodiment. FIG. 14 is a diagram illustrating a sixth example of an individual screen in the embodiment. FIG. 5 is a diagram illustrating an example of an individual screen (deterioration inspection screen) in the embodiment. FIG. 5 is a diagram illustrating an example of a structure three-dimensional model screen in the embodiment. FIG. 13 is a diagram illustrating a second example of a list screen in the embodiment. FIG. 14 is a diagram illustrating a third example of a list screen in the embodiment. FIG. 4 is a diagram illustrating an example of an image relationship between side surfaces of a structure in the embodiment. FIG. 5 is a diagram illustrating image selection between side surfaces of a structure in the embodiment. FIG. 9 is a diagram illustrating a third example of an aerial photographing method of a structure in the embodiment.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In all the drawings for describing the embodiments, the same portions are denoted by the same reference numerals in principle, and the repeated description thereof will be omitted.

[Issues, etc.]
Supplementary explanation of the prerequisite technologies and issues will be given. 2. Description of the Related Art Conventionally, for example, in an application for supporting structural deterioration inspection and a system thereof, the following operations and processes are performed. A user who is an operator goes to a site where a structure to be inspected is located, and photographs the surface of the structure using a camera. At that time, the whole area of the surface of the structure may be photographed, or the image may be narrowed down to a part of the area. In particular, there is a case where a person (expert technician) who has knowledge about the structure in advance narrows down a portion where deterioration is likely to occur from a design drawing of the structure or the like and sets the portion as a shooting target portion. In this case, the shooting location and the number of images can be limited to reduce the labor of the operation.

  The user takes the data of the captured image group back to an office or the like and inputs the data to a computer such as a PC. The user selects an individual image to be an inspection target image from a list of image groups on the display screen, and displays the image on the display screen. When performing a visual inspection, the user visually searches the image for the presence or absence and location of a deteriorated state such as a crack for each image.

  The manual work of photographing a structure and the work of inspecting deterioration of the structure as described above require a lot of trouble. In addition, there are general problems such as a shortage of specialist engineers and high costs. Therefore, as a countermeasure, there is a technique of photographing a structure using a flying object such as a drone or a UAV, a robot, or the like. In the example of this technique, while the drone is autonomously sailing, the surface area of the structure is, for example, comprehensively photographed by a camera mounted on the drone, and an image group is automatically obtained. Thereby, it is possible to reduce the work of narrowing down the inspection target portion by the user, the manual photographing work at the site, and the like. In addition, the efficiency of maintenance inspection and deterioration inspection of the structure can be improved.

  Also, in the case of generating a three-dimensional model of a structure from a group of images, similarly, it takes time and effort to manually perform a photographing operation. Also in this case, the use of a drone or the like can reduce photographing work and the like. The computer generates a three-dimensional model of the structure from the input image group based on the SFM processing. Thus, for example, even in the case of a structure having only a two-dimensional design drawing, a three-dimensional model can be obtained and displayed on a display screen, and can be used for structure management and the like.

  The image group obtained by the manual operation or the drone or the like is an image group in which the surface area of the target object is comprehensively photographed, for example, and often has an overlapping area between the images. In other words, in one image, and in another image surrounding the image in a spatial positional relationship, a partial area of the same portion in the image content overlaps. When inspecting the surface area of a structure without omission for the purpose of deterioration inspection, when capturing an image group, images are taken so as to have a margin so that some areas overlap so that there is no shooting omission area. . In particular, in the application of generating a three-dimensional model, an image group having a sufficient overlap between images is required as a condition necessary for applying the known SFM processing. For example, 80% or more overlap is required in the horizontal direction of an image corresponding to the direction of camera movement, and 60% or more overlap is required in the vertical direction.

  Further, in the surface area of the external appearance of the structure, there are many similar shapes due to the architectural structure and the wall design. That is, similar parts increase in the image content of the image group. Therefore, when a user views a certain image, it may be difficult for the user to recognize which part of the image has been shot. In some cases, it is difficult to grasp the spatial position of a certain image in the vicinity of the image.

  Therefore, when the user selects an individual image to be worked from the list of images on the display screen, it may be difficult to select a suitable image. For example, in the application of the deterioration inspection, the user wants to visually inspect an area in one image and then visually inspect an area adjacent to the area of the image. However, it is difficult for the user to select an image in an adjacent area because the image contents between images are so similar that differences are difficult to understand and the positional relationship between images is difficult to understand. Further, when the user simply and sequentially selects the next image of a certain image from the list of image groups, the selected image is not necessarily an image of an adjacent region, and may be an image of a distant position. It may be.

  Further, it is assumed that the user selects the next image (second image) from the certain image (first image) on the display screen by himself / herself determining the image content and the spatial positional relationship. For example, it is determined that the same object appears in two images. Even in such a case, the next image often includes an area already inspected in the first image as an overlapping area. Overlapping areas occur between the images in the contents of the image group depending on the use or case, such as when the surface area of the structure is comprehensively photographed finely or when a three-dimensional model is generated. Therefore, the overlapping area in the next image is included again as a target of the visual inspection, and the burden on the user increases. Since there are overlapping areas in each of a large number of images, work such as deterioration inspection becomes inefficient.

  As described above, in the conventional image display system, for example, in a case of a use for supporting a deterioration inspection of a structure or generating a three-dimensional model, it is difficult and troublesome to select an individual image from an image group. That is, the conventional system has a problem in terms of the support of the user's work and the efficiency.

  In view of the above problems and the like, the image display system according to the embodiment can easily select a suitable image when the user selects an individual image from a group of images on the display screen according to the application, and can save time and effort. To provide a mechanism to support the user so that the number can be reduced. The mechanism includes a graphical user interface (GUI). In the present image display system, when selecting the next target image from a certain image in the image group, there is a mechanism for navigating and linking to the next image in consideration of the overlapping state between the images. The present image display system automatically determines, for example, an image (adjacent image) with the minimum degree of overlap, and presents a link image that allows the image to be easily selected.

  The image display system according to the embodiment receives, for example, data of a group of images captured so as to partially overlap the surface region of the structure between the images. This image group may include a plurality of images having different shooting dates and times, positions, directions, and the like. The image display system displays a list of image groups on a GUI screen. When an individual image (first image) is selected from a screen by a user operation, the image display system displays the selected first image on the screen. The image display system determines a spatial positional relationship between images (a set of the first image and the candidate image) for each image (candidate image) spatially surrounding the first image, The overlapping state of the photographing range between the images is determined. In particular, the image display system calculates the overlapping state as an overlapping rate. In addition, the image display system determines a spatial positional relationship between images by using positional information of the images. The image display system determines an image such as an adjacent image to the first image based on the overlapping ratio, the positional relationship, and the like. For example, the present image display system selects, as a neighboring image, an image in which the overlap ratio between images is a minimum below a predetermined overlap ratio threshold among candidate images.

  The image display system presents an image such as the adjacent image as an image (next image) which is a candidate to be selected next as a target by the user. At this time, the present image display system displays, on the screen, a link image indicating the presence or positional relationship of the next image with respect to the first image. When a link image is selected and operated, the present image display system displays the next image (second image) associated with the link image as an individual image (new first image).

  With the image display system described above, the user can easily select a suitable image with few overlaps from a certain image, and can efficiently perform operations such as deterioration inspection. The first image and the candidate images (adjacent images and the like) may have different spatial positions and different planes or directions on which the imaging ranges are arranged. Even in this case, the user can select and display the second image in a different direction from the first image by operating the link image on the screen.

(Embodiment)
An image display system according to an embodiment of the present invention will be described with reference to FIGS.

[Structural deterioration inspection support system]
FIG. 1 shows an overall configuration of a structure deterioration inspection support system including an image display system according to an embodiment. The image display system according to the embodiment is configured by a computer system 1. The deterioration inspection support system in FIG. 1 includes a computer system 1 as an image display system, a drone 10 as a flying object, and a structure 5 as an object for deterioration inspection or the like. The computer system 1 and the drone 10 are connected by wireless communication. The computer system 1 is, for example, configured as a client-server system having a PC 2 and a server 3. The PC 2 and the server 3 are connected via a communication network 6. A user (operator) operates the PC 2 to use the stem. The user can input an instruction to the system, set a user, and the like on the GUI screen 22 of the PC 2, and can confirm a work state, a result, and the like.

  The structure deterioration inspection support system of FIG. 1 has a deterioration inspection support function, and the deterioration inspection support function includes a three-dimensional model generation function. The deterioration inspection support function is realized by the deterioration inspection support software 100 or the like. The deterioration inspection support function is a function of providing a GUI screen for assisting a user (operator) in performing a deterioration inspection operation of the structure 5. This deterioration inspection includes a visual inspection of the deterioration state of the image captured by the camera 4 by the user. This deterioration inspection may further include automatic diagnosis by a computer. The three-dimensional model generation function is a function of generating a three-dimensional model of the structure 5 from the image group of the camera 4 based on the SFM processing and providing a screen for displaying the three-dimensional model.

  The PC 2 is a computer having the drone control function 21 and the like, and is a client terminal device used by each user (operator). The server 3 is, for example, a server device in a business cloud computing system, a data center, or the like, and performs calculation processing in cooperation with the PC 2. A plurality of PCs 2 of a plurality of users may be similarly connected to the server 3. The PC 2 performs navigation control of the drone 10 and imaging control of the camera 4 by wireless communication. The PC 2 transmits navigation control information, photographing setting information, and the like to the drone 10.

  The structure 5 is an object to be inspected for deterioration and to generate a three-dimensional model, and is an object photographed by the camera 4 of the drone 10. The structure 5 is, for example, a building or an infrastructure facility. The buildings include general buildings, houses, public buildings, and the like. Infrastructure facilities include, for example, power facilities, road traffic facilities, communication facilities, bridges, and the like. A predetermined area on the surface of the structure 5 is a target area.

  The drone 10 is an air vehicle that performs autonomous navigation based on remote control by wireless communication from the PC 2. The drone 10 autonomously sails on the set route in the space around the structure 5. In advance, in the computer system 1, control information on autonomous navigation and aerial photography of the drone 10 is created and set as aerial photography setting information based on the imaging plan data d2 so that the target area of the structure 5 can be aerial photography. You. The aerial photography setting information includes the navigation route of the drone 10, the imaging timing (imaging date and time), the imaging setting information of the camera 4, and the like.

  As a modified example, a mode in which the user controls the navigation of the drone 10 from the PC 2 or the like may be adopted.

  The drone 10 has a camera 4 and various sensors mounted thereon. The drone 10 takes an aerial image of the structure 5 with the camera 4 when traveling on the route. The drone 10 transmits image data and sensor data at the time of shooting to the PC 2 by wireless communication. The known sensor group of the drone 10 can detect the position, direction, speed, acceleration, and the like of the drone 10 as sensor data. The position includes three-dimensional coordinate information and can be obtained in the form of latitude, longitude, altitude (elevation from the ground) or the like based on, for example, a GPS, an altitude sensor, or another positioning system. When GPS is used, sufficient positioning accuracy is assumed.

  The camera 4 includes, as shooting setting information, a camera direction (shooting direction), shooting timing, shooting conditions, and the like. The camera direction is a direction facing a location on the surface of the object 5 with reference to the camera position. The shooting timing is a timing at which a plurality of continuous images (still images) are shot, and is defined by a shooting interval or the like. The shooting conditions are defined by known parameters such as exposure.

  The PC 2 has a drone control function 21, a client program 20, a GUI screen 22, and the like. The drone control function 21 is a function for controlling navigation of the drone 10 and shooting of the camera 4. The client program 20 is a client program of the deterioration inspection support software 100. The client program 20 performs processing in cooperation with the server program 30 through client-server communication. The client program 20 controls a drone control function 21, an image display function, a GUI screen 22, and the like. Various data and information used for processing by the client program 20 and the like are stored in the memory of the PC 2. The memory stores control information of the drone 10, image data and sensor data acquired from the drone 10, various data acquired from the server 3, and the like.

  The server 3 has a server program 30, a DB 31, and the like. The server program 30 is a server program of the deterioration inspection support software 100. The server program 30 is in charge of processing with a high calculation processing load, such as the deterioration inspection support processing and the three-dimensional model generation processing. The server program 30 executes a predetermined process in response to a request from the client program 20, and responds with process result information.

  The DB 31 stores various data and information used by the server program 30 and the client program 20 for processing. The DB 31 may be realized by a DB server or the like. The DB 31 stores, for example, structure data d1, imaging plan data d2, image data d3, inspection data d4, and the like.

  The structure data d1 includes basic information and management information on the structure 5, design drawing data, three-dimensional data, and the like. In the present image display system, the three-dimensional data includes three-dimensional model data generated by the three-dimensional model generation function. The three-dimensional data is not limited to this, and if there is data created by an existing three-dimensional CAD system or the like, it can be used.

  The photographing plan data d2 is data of a plan relating to deterioration inspection and aerial photography. Based on the shooting plan data, aerial shooting setting information by the actual drone 10 is created and set. The image data d3 is data of an image group captured by the camera 4. The inspection data d4 is data representing the state and the result of the deterioration inspection.

  The mounting configuration of the computer system 1 is not limited to the above example, but is possible. For example, the PC 2 and the server 3 may be integrated as one computer, or may be separated into a plurality of devices for each function. A form having a drone control device separately from the PC 2 may be used. Not only the drone 10 but also other flying objects, robots, and the like are applicable. As the drone 10, a general-purpose drone can be applied, but a dedicated drone having a dedicated function for the present image display system may be applied.

[Drone, calculator]
FIG. 2 shows a schematic configuration of the drone 10 and the drone control function 21 in the computer system 1. The drone 10 includes a propeller driving unit 11, a navigation control unit 12, a sensor 13, a gimbal 14, a camera 4, an image storage unit 15, a wireless communication unit 16, a battery 17, and the like. The propeller driving unit 11 drives a plurality of propellers. The navigation control unit 12 controls navigation of the drone 10 according to navigation control information from the navigation control unit 102 of the PC 2. For that purpose, the navigation control unit 12 drives and controls the propeller driving unit 11 while using the information of the sensor 13. The sensor 13 is a group of sensors such as a known GPS receiver, an electronic compass, a gyro sensor, and an acceleration sensor, and outputs predetermined sensor data. The gimbal 14 is a known mechanism that holds the camera 4, and is a mechanism that automatically maintains the camera 4 in a certain state without blurring during navigation. Each direction and each position of the drone 10, the gimbal 14, and the camera 4 are basically independent parameters.

  The camera 4 aerially shoots the object 5 according to the shooting control information and the shooting setting information from the shooting control unit 104 of the PC 2 and outputs image data. The image storage unit 15 stores image data and the like. The wireless communication unit 16 includes a wireless communication interface device, and performs wireless communication with the computer system 1 using a predetermined wireless communication interface. The battery 17 supplies power to each unit.

  The computer system 1 includes a processor 111, a memory 112, a communication device 113, an input device 114, a display device 115, and the like, which are interconnected by a bus or the like. The processor 11 executes a process according to the program read into the memory 112. Thereby, each function of the deterioration inspection support software 100 and each unit such as the GUI unit 101 are realized.

  The computer system 1 includes a GUI unit 101, an aerial photography setting unit 107, a navigation control unit 102, an imaging control unit 104, a sensor data storage unit 103, an image data storage unit 105, a wireless communication unit 106, a deterioration inspection support unit 121, and a three-dimensional It has a model generation unit 122, an image display control unit 123, and the like.

  The GUI unit 101 configures a GUI screen 22 for a user based on an application such as an OS, middleware, and a Web browser, and displays the GUI screen 22 on the display device 114 (for example, a touch panel). The user can perform user settings and instruction input operations on the GUI screen 23, and can check the state and result of each function. As the user setting, it is possible to set whether or not to use each function of the image display system, and to set a control threshold for each function. The GUI unit 101 generates screen data corresponding to a screen such as a list screen and an individual screen described later.

  The aerial photography setting unit 107 creates aerial photography setting information based on the operation of the user and the imaging plan data d2, and sets the information in the computer system 1 and the drone 10. The navigation control unit 102 controls navigation of the drone 10 based on the aerial photography setting information. The navigation control unit 102 transmits navigation control information d101 to the drone 10 through wireless communication, and receives sensor data d102 and the like indicating the navigation state from the drone 10. The shooting control unit 104 controls shooting by the camera 4 based on the aerial shooting setting information. The imaging control unit 104 transmits the imaging control information d103 to the drone 10 through wireless communication, and receives image data d104 and the like from the drone 10.

  The image data storage unit 105 stores image data obtained from the drone 10. The image data is managed in association with the shooting date and time, sensor data, shooting setting information, and the like as appropriate. The wireless communication unit 106 includes a wireless communication interface device, and performs wireless communication with the drone 10 using a predetermined wireless communication interface.

  The deterioration inspection support unit 121 performs a process corresponding to the deterioration inspection support function. The deterioration inspection support unit 121 performs, for example, visual inspection support processing or automatic diagnosis processing. The three-dimensional model generation unit 122 performs a process corresponding to the three-dimensional model generation function. The image display control unit 123 determines the spatial positional relationship of the image group and performs control related to image selection.

[Image display system]
FIG. 3 shows a configuration of the image display system (computer system 1) according to the embodiment. The image display system has, as inputs (including storage and the like), an image group 201 and image information 202 based on the image data d3, structure data d1, photographing plan data d2, and the like. In the embodiment, the input image data d3 (image group 201) is mainly obtained by the camera 4 of the drone 10, but is not limited thereto, and may be an image group photographed manually by an operator. The combination of the image groups obtained by the above means may be used. The image group 201 is data of the main body of each image. The image information 202 is metadata or property information associated with each image in a format such as, for example, a known Exif. The image information 202 includes, for example, information such as ID (identification information), shooting date and time, camera position, camera direction, camera model, focal length, angle of view, image size (number of pixels), and the like. Note that the camera position and the camera direction generated by the SFM processing unit 122A are used.

  The image display system includes a list display unit 301, an individual display unit 302, an image display control unit 123, a deterioration inspection support unit 121, a three-dimensional model generation unit 122, and the like as processing (corresponding processing units). The image display control unit 123 includes an adjacent image determination unit 303 and an overlap ratio calculation unit 304. The image display system outputs screens such as a list screen 401 and an individual screen 402 as outputs (including display, storage, and the like).

  The list display unit 301 displays a list screen 401 based on the image group 201 and the image information 202 (for example, ID, shooting date and time). On the list screen 401, a plurality of images of the image group 201 are arranged and displayed in parallel in an order using IDs and the like.

  The individual display unit 302 displays the individual screen 402 based on the image group 201 and the first image selection information 321 on the list screen 401. The individual display unit 302 displays the first image 322 selected by the user on the individual screen 402. Further, the individual display unit 302 displays the second image 324 associated with the link image based on the link image selection information 323 in the individual screen 402.

  The image display control unit 123 uses the adjacent image determination unit 303 and the like to grasp the spatial positional relationship and the degree of overlap between the images, and controls the link between the images and the selection of the next image.

  Based on the image information 202 and the first image selection information 321 or the link image selection information 323 (that is, the second image selection information), the adjacent image determination unit 303 determines a candidate image, an adjacent image, and the like around the first image. Judge. Adjacent images are images that are adjacent with a minimum degree of overlap. The adjacent image determination unit 303 determines a positional relationship between images using, in particular, position information (camera position and the like) and direction information (camera direction and the like) of the image information 202. In addition, the adjacent image determination unit 303 determines an adjacent image using the information on the overlap ratio calculated by the overlap ratio calculation unit 304. The adjacent image determination unit 303 determines the distance between the position of a certain image (first image) and another image in the vicinity thereof, and determines the direction of each image (corresponding camera direction or image capturing range). The direction of the arranged surface) is determined.

  The overlap ratio calculation unit 304 calculates the overlap ratio between images based on the first image selection information 321 and the like and the candidate image information determined by the adjacent image determination unit 303.

  The image display control unit 123 displays the link image 325 corresponding to the image such as the adjacent image determined by the adjacent image determination unit 303 on the individual screen 402. The link image 325 is a link image (GUI component) for navigating an image (next image) that is a candidate to be selected next to the first image.

  The deterioration inspection support unit 121 performs a visual inspection support process or the like based on a user operation using the individual screen 402 as a deterioration inspection screen, displays a processing state or result on the deterioration inspection screen, and displays the processing state or result. The inspection data d4 to be represented is created and stored.

  The three-dimensional model generation unit 122 includes an SFM processing unit 122A. The SFM processing unit 122A generates a three-dimensional model of the structure 5 and information on the camera direction and the camera position by performing the SFM processing with the image group 201 as an input, and creates the corresponding three-dimensional model data d5. The three-dimensional model generation unit 122 displays a structure three-dimensional model screen based on the three-dimensional model data d5. The three-dimensional model generation unit 122 describes the obtained camera direction and camera position information in the image information 202 or manages the information in association with the image data d3.

  The user performs an operation (first image selection operation) OP1 on the list screen 401 to select a desired image from the image group as the first image. In addition, the user performs an operation (second image selection operation = link image selection operation) OP2 on the individual screen 402 to select a link image. Further, the user performs the deterioration inspection operation OP3 on the image of the individual screen 402 (deterioration inspection screen).

[Processing flow]
FIG. 4 shows a processing flow of the structure deterioration inspection support system including the image display system according to the embodiment. The flow in FIG. 4 includes steps S1 to S13. Hereinafter, the steps will be described in order.

  (S1) The computer system 1 creates and sets aerial photography setting information of the drone 10 based on a user operation, the photography plan data d2 of the structure 5, and the like. Aerial photography by the drone 10 is performed based on the aerial photography setting information. By the aerial photographing, image data or the like obtained by photographing with the camera 4 is transmitted to the PC 2. The PC 2 transmits image data and the like to the server 3. The server 3 stores the image data d3 and the like in the DB 31.

  (S2) The computer system 1 reads out image data d3 (image data 201, image information 202) and the like from the DB 31 and the like, and stores them in the processing memory.

  (S3) The computer system 1 (especially the list display unit 301) displays a list of image groups on the list screen 401 (FIG. 14) based on the image data d3 (image group 201, image information 202).

  (S4) The computer system 1 receives the user operation on the list screen 401 and, when receiving a predetermined operation (first image selection operation OP1), obtains the first image selection information 321 and selects the first image. . The first image selection operation OP1 is, for example, a click or a tap of one image.

  (S5) The computer system 1 (especially the individual display unit 302) displays the selected first image 322 on the individual screen 402 (FIG. 15) based on the operation in S4 and the first image selection information 321. The user can perform the deterioration inspection work using the individual screen 402 as the deterioration inspection screen (FIG. 21).

  (S6) Based on the image information 202 and the first image selection information 321 (or the link image selection information 323), the computer system 1 (especially the adjacent image determination unit 303) executes each image (candidate image) in the vicinity of the first image. Explore). At this time, the adjacent image determination unit 303 determines the positional relationship between the images using information such as the camera position and the camera direction of the images. In particular, the adjacent image determination unit 303 calculates a distance between the position of the first image and the candidate image, and picks up a candidate image for the first image based on the distance. At this time, the adjacent image determination unit 303 picks up candidate images in ascending order of the distance. At that time, the adjacent image determination unit 303 narrows down the candidate images within a range in which the distance is within the distance threshold, using a preset distance threshold. At this point, the adjacent image has not been determined yet.

  Further, in particular, in this processing example, when picking up a candidate image, the adjacent image determination unit 303 first extracts an image whose camera direction is substantially the same, for example, an image arranged in the same plane. Is picked up as a candidate image.

  (S7) The computer system 1 (especially the overlap ratio calculation unit 304) sequentially calculates the overlap ratio (OL) between the pairs of the first image and the candidate image for each candidate image obtained in S6. I do. At that time, the duplication ratio calculation unit 304 calculates a photographing range for each image (FIG. 7), and calculates the duplication ratio OL using the photographing range and the like (FIG. 12 and the like), as described later.

  (S8) The computer system 1 (especially the adjacent image judging unit 303) judges the duplication state by comparing the duplication rate OL obtained in S7 with a predetermined duplication rate threshold value (TOL) set in advance. I do. In this processing example, the adjacent image determination unit 303 determines whether the OL value of the candidate image is equal to or less than the TOL value, is greater than 0, and is minimum. The computer system 1 selects a candidate image that satisfies the condition as an adjacent image (an image with the minimum overlap and an adjacent image). The computer system 1 similarly determines an adjacent image for each direction such as up, down, left, and right in the plane.

  Note that the overlap ratio threshold TOL has already been set in the program of the deterioration inspection support software 100. The present invention is not limited to this, and the TOL value may be set by the user.

  (S9) Based on the result of S8, the computer system 1 confirms whether or not an adjacent image satisfying the condition of the overlapping state has been found. If it has been found (Y), the process proceeds to S11. If it has not been found (N), Proceed to S10. Here, the case where no image is found corresponds to the case where no image is found from an image whose camera direction is substantially the same, for example, an image arranged in the same plane.

  (S10) In S10, the computer system 1 selects a candidate image other than the searched image and having the smallest distance between image positions as the next image. As will be described later, for example, this next image is an image arranged on another surface (side surface or the like) having a different camera direction from the first image. Alternatively, this next image is an image located at a distance without any overlap even in the same plane as the first image.

  (S11) The computer system 1 (image display control unit 123) displays the link image 325 (the link image 403 in FIG. 15 and the link in FIG. 16) of the image (the next image) selected in S8 or S10 in the individual screen 402. Image 404). In particular, the image display control unit 123 displays the link image 325 according to the type of the next image in each direction with respect to the first image.

  (S12) The computer system 1 receives the user's predetermined operation (second image selection operation OP2) on the link image 325 on the individual screen 402. The second image selection operation OP2 is a click, a tap, or the like on the link image. Further, the computer system 1 receives another operation such as an operation for returning to the list screen 401 and an end operation on the individual screen 402. The computer system 1 proceeds to S13 when receiving a selection operation (second image selection operation OP2) for the link image 325 (Y), and returns to S3 when receiving an operation for returning to the list screen 401.

  (S13) In S13, the computer system 1 (in particular, the individual display unit 302) is displayed on the individual screen 402 by the selected link image 325 based on the link image selection information (second image selection information) 324 in S12. The next image (second image) is displayed as a new first image. After S13, the process returns to S6, and the same repetitive processing is performed. The user can perform the deterioration inspection work by viewing the new first image on the individual screen 402 (deterioration inspection screen). Thereafter, similarly, selection and display transition from one image to another image in the image group are possible.

[Aerial photography method (1)]
FIG. 5 shows a first example of an aerial photographing method of the structure 5 using the drone 10. The method of aerial photography is not limited to this. As the aerial photography method, a suitable method is selected in consideration of the processing and characteristics of the image display system. Using the drone control function 21 of the computer system 1, the route of the aerial photographing by the drone 10, the photographing setting of the camera 4, and the like are set as the aerial photographing setting information based on the photographing plan data d2 and the like. The aerial photography setting information is held in the DB 31 or the like.

  FIG. 5A shows a bird's-eye view configuration in the Z direction when the structure 5 is roughly a rectangular parallelepiped. This structure 5 has four side surfaces A1 to A4 standing vertically. The image display system comprehensively captures, for example, the entire surface area of these side surfaces A1 to A4 and sets the image as a degradation inspection target. In addition, the structure 5 has corners, such as corners 501, in which side surfaces in different directions are connected to each other.

  In the present example, aerial photography is performed separately for each side surface of the object 5 as a group (side surface group). In the present example, four routes R1 to R4 are set corresponding to four groups corresponding to the four side surfaces A1 to A4, and aerial photography is performed four times. In this example, the route of each group is basically a straight route (indicated by a dashed arrow), and is controlled so that the camera direction (indicated by an alternate long and short dash line) and the target distance D0 are substantially constant.

  For example, first, in the group on the side surface A1, aerial photography is performed on the route R1. In the route R1, the camera direction is the Y direction (the direction Y1 from the front to the back). Next, in the group of the side surface A2, aerial photography is performed along the route R2. On route R2, the camera direction is the X direction (right-to-left direction X1). The same applies to the route R3 of the side surface A3 and the route R4 of the side surface A4.

  FIG. 5B corresponds to FIG. 5A and shows the structure 5 and the route of aerial photography in a perspective view. Particularly, a detailed setting example of a route R1 for aerial photographing the group of the side surface A1 is shown. In this example, a method similar to a so-called line-sequential scanning method is used to comprehensively photograph the entire area of the side surface A1 (XZ plane). In this method, first, the drone 10 is linearly moved from one end of the side surface A1 to the other end in the X direction, which is a horizontal direction, as in main scanning. During the horizontal movement, the direction of the camera 4 and the target distance D0 are controlled so as to be substantially constant. After the drone 10 reaches the other end of the side surface A1, the drone 10 is then moved linearly in the vertical direction (Z direction) like a sub-scan. Next, similarly, the side surface A1 is linearly moved from the other end to one end of the side surface A1 like a main scan so as to be folded in the opposite direction in the X direction. Thereafter, the main scanning and the sub-scanning are similarly repeated.

  Note that, in the example of the embodiment, the description is mainly directed to the side surface that stands vertically, but the horizontal surface such as the roof of the structure 5 can be similarly processed as the target. In this case, the direction of the camera 4 during aerial photography is, for example, downward in the Z direction. Further, in the example of the embodiment, the case where the direction of the camera 4 at the time of aerial photography is the X direction or the Y direction which is a horizontal direction in which the side surface is viewed in a plan view is described, but is not limited thereto. The same processing can be performed even in an obliquely downward direction.

  When setting the aerial photography, the route of the drone 10, the photographing timing (interval) of the camera 4, and the like are set in advance in consideration of the SFM processing of the three-dimensional model generation function so that the overlapping state between the images is suitable. Is set. For example, the overlapping state in the traveling direction of the drone 10 and the camera 4 is set to be about 80%. By performing aerial photography based on this setting, a group of images in a suitable overlapping state is obtained. The SFM processing can be reliably performed by using this image group as an input, and a highly accurate three-dimensional model can be obtained.

  However, when the image group corresponding to the use of the three-dimensional model generation is also used for the deterioration inspection, there are many overlapping areas between the images. Therefore, at the time of the deterioration inspection of each image, the inspected area (that is, the overlapping area) is reduced. In many cases, the area to be inspected again increases, which is inefficient. Therefore, in order to perform an efficient deterioration inspection, it is preferable to select an image group in which the overlap between the images is as small as possible from those image groups as an image group to be subjected to the deterioration inspection. However, it is difficult and time-consuming to manually select the image. Therefore, the present image display system provides a mechanism such as a GUI that can support and increase the efficiency of the image selection.

[Aerial photography method (2)]
FIG. 6 similarly shows a second example of a method of aerial photographing of the structure 5 using the drone 10. Such an aerial photographing method is appropriately selected according to the shape of the structure 5 and the like. In the present example, the structure 5 in FIG. 6A is different from the side surface A1 of the rectangular parallelepiped structure 5 in FIG. 5 on a part of the right side (direction X2) on the near side (direction Y2). Has a shape such that a rectangular parallelepiped portion of the is connected additionally. This structure 5 has six side surfaces a1 to a6, and it is assumed that the entire surface area thereof is to be inspected. The structure 5 has corners such as a corner 601, a corner 602, and a corner 603. The corner portion 601 is a portion where the side surface a1 extending in the X direction and the side surface a2 extending in the Y direction are connected, and is a corner portion that is concave when viewed from the drone 10. The corner portion 602 is a portion where the side surface a2 extending in the Y direction and the side surface a3 extending in the X direction are connected, and is a convex portion viewed from the drone 10. The corner portion 603 is a portion where the side surface a3 extending in the X direction and the side surface a4 extending in the Y direction are connected, and is a convex corner portion.

  In this example, similarly, aerial photography is performed by dividing the object 5 into groups for each side surface. In this example, the six side surfaces a1 to a6 are divided into six groups, six routes r1 to r6 are set correspondingly, and aerial photography is performed six times. The route of each group is a straight route, and is controlled so that the camera direction and the target distance D0 are substantially constant. As a whole, camera directions are divided into four types: a direction Y1, a direction Y2, a direction X1, and a direction X2. For example, in the route r1 of the group of the side surface a1, the camera direction is the Y direction (direction Y1). On the route r2 of the group of the side surface a2, the camera direction is the X direction (direction X2). On the route r3 of the side surface a3, the camera direction is the Y direction (direction Y1). On route r4 of side surface a4, the camera direction is the X direction (direction X1). The same applies to the routes of other aspects.

  FIG. 6B shows the structure 5 and the route of the aerial photography in perspective, corresponding to FIG. In particular, a detailed setting example of the route r1 of the side surface a1 and the route r3 of the side surface a3 will be described. As in FIG. 5B, the same method as the line-sequential scanning method is used for each side surface.

[Shooting range]
FIG. 7 is a schematic diagram illustrating an image capturing range and the like. The image display control unit 123 of the image display system calculates the image capturing range as described below. This imaging range is used for calculating the overlap rate. The shooting range is a size in the real world defined by the length and width of the image and the like. For calculating the photographing range, a focal length (F), a sensor size (SS), a target distance (D), and the like are used. In FIG. 7, the bird's-eye view in the Z direction shows the wall surface 701 (for example, the side surface A1) of the structure 5, the traveling direction K1 (for example, the X direction) in a part of the aerial photography route of the drone 10, and the position of the camera 4 (the camera position C). ), The direction of the camera 4 (referred to as a camera direction V), the sensor 702 of the camera 4, an image capturing range (referred to as an SA, particularly, a width in the X direction), and the like.

  The sensor 702 is an image sensor of the camera 4, and FIG. 7 shows a portion having a width in the X direction. The sensor 702 has a predetermined sensor size SS (particularly, a width in the X direction). The sensor size SS depends on the camera model, and is defined by a width, a height, a diagonal distance, and the like. The sensor size SS is included in the image information 202 or can be grasped from the camera model and other information even if it is not included. The center position of the sensor 702 is shown as camera position C. From the sensor 702 and the camera position C, for example, there is a camera direction V (indicated by an alternate long and short dash line arrow) in the Y direction. The position where the camera direction V intersects the wall surface 701 is indicated by a position Q. The position Q corresponds to the center position of the image and the shooting range SA. In the distance from the camera position C to the position Q of the wall surface 701 (side surface A1) in the camera direction V, there are a focal length F and a target distance D. The focal length F is obtained from the image information 202. The target distance D is a distance to the target. The target distance D is obtained based on the shooting plan data d2. For example, the target distance D is obtained from the target distance D0 in FIG.

  The computer system 1 (especially the overlap ratio calculation unit 304) calculates the shooting range SA using the sensor size SS, the focal length F, the target distance D, and the like. The basic calculation formula is [photographing range SA] = [sensor size SS] ÷ [focal length F] × [target distance D]. A specific example is as follows. It is assumed that the sensor size SS is 23.5 mm in width (width) and 15.6 mm in height (height). Assume that the focal length F is 35 mm. It is assumed that the target distance D is 5 m (= 5000 mm). In the photographing range SA, when the width in the horizontal (X direction) is SAX and the height in the vertical (Z direction) is SAZ, SAX = (23.5 / 35) × 5000 ≒ 3357 mm, SAZ = (15.6) / 35) × 5000 ≒ 2229 mm.

  The computer system 1 similarly calculates the shooting range SA of each image in the image group. The computer system 1 (overlap ratio calculation unit 304) calculates the overlap ratio between images using the shooting range SA of each image. The computer system 1 (adjacent image determining unit 303) determines the positional relationship between images using information such as the camera position C, the camera direction V, and the shooting range SA.

[Camera position and camera direction]
The computer system 1 uses information on the camera position C and the camera direction V for processing. Information on the camera position C and the camera direction V can be obtained by, for example, the following methods.

  (1) When the computer system 1 includes information on the camera position C and the camera direction V in the image information 202 or the sensor data at the time of aerial photography, the computer system 1 transmits the information on the camera position C and the camera direction V. refer. For example, the camera 4 itself may include a sensor that can detect the camera position C and the camera direction V. In this case, the information can be referred to by associating the information on the detected camera position C and the camera direction V with the image data of the image captured by the camera 4.

  (2) When the image information 202 does not include the information on the camera position C and the camera direction V, the computer system 1 calculates the camera position C and the camera direction V from other information. For example, the computer system 1 obtains information on the position and direction of the drone 10 and the position and direction of the gimbal 14 based on the sensor data at the time of aerial photography. The computer system 1 obtains the camera position C and the camera direction V by calculation using the information. The camera position C and the like can be calculated by matching the relative positional relationship of the gimbal 14 installed on the drone 10 with the relative positional relationship of the camera 4 installed on the gimbal 14.

  (3) The computer system 1 obtains a camera position C and a camera direction V from an image group (for example, three continuous images) by SFM processing. This method is used in the image display system of the embodiment. The SFM processing unit 122A of the three-dimensional model generation unit 122 obtains the camera position C and the camera direction V by performing the SFM processing on the input image group.

[Overlap between images (1)]
FIG. 8 shows the overlap between images and the photographing range in the aerial photographing traveling direction. In FIG. 8, a part of the route to the wall surface 701 is shown as a bird's-eye view in the Z direction. The traveling direction K1 of the drone 10 during the aerial photography is, for example, the main scanning direction X2 of the route R1 in FIG. 5 and the main scanning direction X2 of the route r1 in FIG. The camera direction C is the direction Y1. On the traveling direction K1, as an example of the position P of the drone 10, positions P1 to P7 are indicated by dots. Correspondingly, positions C1 to C7 are indicated by dots as examples of the camera position C. The target distance D0 at each position is constant. The positions P1 to P7 and the positions C1 to C7 have a chronological order. In addition, corresponding to the positions C1 to C7 and the like, the center position on the image and photographing range side is indicated by points Q1 to Q7. For example, it has images g1 to g7 and shooting ranges SA1 to SA7. The imaging range here indicates the width in the X direction. For example, the shooting range SA of the image g1 shot at the position P1 and the position C1 is the shooting range SA1, and the center position is the position Q1. The example of FIG. 8 illustrates a case where the photographing ranges SA1 to SA7 of the images g1 to g7 overlap by about 80% between the images in the traveling direction K1.

[Overlap between images (2)]
FIG. 9 shows the overlap between images in plan view (XZ plane, for example, side A1), corresponding to the example of FIG. FIG. 9 illustrates a case where the images g1 to g7 are slightly shifted in the Z direction (the traveling side direction K2) so that the state of the overlap can be easily understood. The traveling direction K1 indicates the case of the direction X2 as in FIG. 8, and the traveling side direction K2 is a vertical direction perpendicular to the traveling direction K1. The traveling direction K3 is a direction when a displacement (movement component) of the traveling side direction K2 is added to the traveling direction K1. In the present example, an overlapping area 901 indicated by a diagonal line pattern indicates an overlapping area in a set of an image g1 (imaging area SA1) and an image g2 (imaging area SA2). The overlapping area 902 indicates an overlapping area in the set of the image g1 and the image g6 (the imaging range SA6).

  In this example, the images g1 to g7 are a group of images photographed so as to secure an overlap rate (for example, about 80%) necessary for generating a three-dimensional model. For example, in a set of the image g1 and the next image g2, the overlap ratio OL is 80% or more (for example, 85%). Similarly, the overlap ratio OL for the image g1 is 65% for the image g3, 45% for the image g4, 25% for the image g5, and 5% for the image g6. In the set of the image g1 and the image g7, there is no overlap and there is a gap between the images.

  It is assumed that the overlap ratio threshold TOL set for the deterioration inspection is, for example, 20%. When considering each of the candidate images (images g2 to g7, etc.) on the right side (direction X1) with respect to the image g1, the image g6 is selected as the image that satisfies the above-described (FIG. 4, S8) overlapping rate condition. The overlap rate OL (= 5%) of the image g6 with respect to the image g1 is the minimum within a range that is equal to or less than the overlap rate threshold TOL (= 20%) and greater than 0.

  In addition, not only the overlapping rate in the traveling direction K1 (X direction, the horizontal direction of the image) can be considered, but also the overlapping rate in the traveling side direction K2 (the vertical direction of the image). The present image display system performs the same processing for the traveling side direction K2 using a predetermined overlap ratio threshold.

[Overlap between images (3)]
FIG. 10 shows an example of an adjacent image in a plan view (XZ plane, for example, side A1), corresponding to the example of FIG. It is assumed that the first image selected by the user is, for example, the image g1. The present image display system examines three-dimensional spatially peripheral images of the image g1 as candidate images. In the example of FIG. 10, there are images g2 to g7 and the like in the right direction (direction X2) with respect to the image g1, as in FIG. In the right direction of the image g1, the image g6 is selected as an image (adjacent image) with the minimum overlap as shown in FIG. The image g6, which is an adjacent image in the right direction, is also shown as an image ga. The position Qa of the image ga is the position Q6 of the image g6. The image display system displays a link image representing the image ga on the individual screen 402.

  In the left direction (direction X1) with respect to the image g1, there is, for example, an image gb (position Qb) as an adjacent image. Each image between the image g1 and the image gb is omitted. Similarly, in the upward direction (direction Z1) with respect to the image g1, there is, for example, an image gc (position Qc) as an adjacent image. In the downward direction (direction Z2) with respect to the image g1, there is an image gd (position Qd) as an adjacent image, for example. Similarly, there is, for example, an image ge (position Qe) as an adjacent image in a diagonally lower right direction with respect to the image g1. Although not shown, there are adjacent images in the diagonally upper right direction, the diagonally upper left direction, and the diagonally lower left direction of the image g1. The user can select a desired next image (second image) from the link images (corresponding adjacent images) displayed in each direction with respect to the first image on the individual screen 402.

  Further, in the present image display system, when determining an image in an oblique direction with respect to a certain first image (for example, image g1), for example, an image ge at a position Qe that is obliquely lower right, the following method may be used. Good. As a first method, an image in the oblique direction is directly searched for the image g1 at the position Q1, an overlap ratio between the image g1 and the image in the oblique direction is calculated, and a predetermined overlap ratio threshold (in the oblique direction) is calculated. Is set to determine the adjacent image. As another second method, first, left and right or top and bottom adjacent images are determined for the image g1 at the position Q1, and the adjacent images (eg, the image ga and the image gd) obtained by the determination are further processed. Alternatively, it is also possible to determine the adjacent images that are above, below, right and left. That is, oblique adjacent images may be obtained based on the result of the same determination in two directions in two stages from the first image.

[Calculation of duplication rate (1)]
The calculation of the overlap rate will be described with reference to FIGS. The basic definition of the overlap ratio is as follows. Ideally, a state of overlap in the surface area of the structure 5 (corresponding imaging range SA) is considered. The computer system 1 considers a region where the images overlap in the surface region in the image centered on the position Q in FIG. The computer system 1 calculates the size of the image and the overlapping area. The rate indicating the degree of overlap of the overlapping area between the images is defined as the overlap rate. The computer system 1 calculates an overlap rate based on the size of the image and the overlap area. When the distance between the camera 4 and the surface of the structure 5 at the time of photographing (target distance D0) is constant, the size of the region shown in each image almost corresponds to the size of the object in the real world. Therefore, in that case, a sufficient effect can be obtained by calculating the overlap ratio using the size of the image area.

  FIG. 11 shows the overlapping state of the two images in the shooting range SA. In FIG. 11, it is shown as a bird's-eye view in the Z direction. An image G1 taken from the camera position C1 and an image G2 taken from the camera position C2 are shown for the side surface A1 and the like. The position Q1 of the image G1 and the position Q2 of the image G2 are shown. Each of the position C1, the position Q1, and the like has three-dimensional position coordinates. The width W1 in the X direction of the photographing range SA of the image G1 and the width W2 in the X direction of the photographing range SA of the image G2 are shown. The width of the overlapping area in the set of the image G1 and the image G2 is indicated by a width W12. For example, the overlapping rate in the X direction between the image G1 and the image G2 can be calculated using the width W1 of the image G1 and the width W12 of the overlapping area.

  Since the drone 10 travels along the route and is shaken by the wind, the position of the drone 10, the position of the camera 4, the direction of the camera, and the like may be shaken or shifted. That is, the camera direction C of each image may be different. Even in that case, the photographing range SA of the image on the surface of the structure 5 and the overlap ratio can be calculated in the same manner.

[Calculation of duplication rate (2)]
FIG. 12 corresponds to FIG. 11 and shows an overlapping state in an image in plan view (XZ plane). Further, in this example, a case where the image size is different between the image G1 at the position Q1 and the image G2 at the position Q2 is shown. An overlapping area 1201 (a hatched pattern) in a set of an image G1 at a position Q1 and an image G2 at a position Q2 is shown. The image G1 has a width W1 as a width W in the X direction and a height H1 as a height H in the Z direction. Image G2 has width W2 and height H2. A pair of the image G1 and the image G2 indicates a width W12 of the overlapping area 1201 and a height H12 of the overlapping area 1201. The width NW1 and height NH1 of the non-overlapping part of the image G1 and the width NW2 and height NH2 of the non-overlapping part of the image G1 are also shown.

  Here, as the overlapping rate OL, an overlapping rate in the horizontal direction (X direction) of the image (OLW) and an overlapping rate in the vertical direction (Z direction) of the image (OLH) are considered. The overlap rate OL can be defined by a ratio of the length of the overlap portion to the original length of the first image. In this example, the horizontal (X direction) overlap rate OLW can be calculated by OLW = W12 / W1, and the vertical (Z direction) overlap rate OLH can be calculated by OLH = H12 / H1. Note that the three-dimensional position coordinates (X, Y, Z) are appropriately converted into two-dimensional position coordinates (x, y) in the processing of the two-dimensional image and processed.

  In the example of FIG. 12, the coordinates of the position Q1 of the image G1 are (X1, Y1, Z1), and the coordinates of the position Q2 of the image G2 are (X2, Y2, Z2). In the shooting range SA of the image G1, the left end position in the X direction is X3, and the right end position is X4. In the shooting range SA of the image G2, the left end position in the X direction is X5, and the right end position is X6. In the shooting range SA of the image G1, the position of the upper end in the Z direction is Z3, and the position of the lower end is Z4. In the shooting range SA of the image G2, the position of the upper end in the Z direction is Z5, and the position of the lower end is Z6. In this example, the positions in the Y direction are the same.

  The positions of the upper, lower, left, and right ends (sides) of the image G1 are obtained as follows using the center position Q1, the width W1, and the height H1. The position X3 at the left end of the image G1 is obtained by X3 = X1- (W1 / 2). The position X4 at the right end of the image G1 is obtained by X4 = X1 + (W1 / 2). The position Z3 of the upper end of the image G1 is obtained by Z3 = Z1- (H1 / 2). The position Z4 of the lower end of the image G1 is obtained by Z4 = Z1 + (W1 / 2). Similarly, the position (X5, X6, Z5, Z6) of each end of the image G2 can be obtained. Using the position of each end of each image, the width W12 and the height H12 of the overlapping area 1201 can be calculated. For example, W12 = X4-X5 and H12 = Z5-Z4.

  The following is a specific example of the calculation. It is assumed that the size (expressed by the number of pixels) of the image G1 is W1 = 339 and H1 = 252, and the size of the image G2 is W2 = 373 and H2 = 244. The coordinates (X1, Z1) of the position Q1 are (1420, 2395), and the coordinates (X2, Z2) of the position Q2 are (1510, 2376). As the coordinates of each image end, it is assumed that X3 = 1251, X4 = 1589, X5 = 1324, X6 = 1696, Z3 = 2520, Z4 = 2269, Z5 = 2497, and Z6 = 2254. Using these pieces of information, the width W12 and the height H12 of the overlapping area 1201 are obtained as W12 = 265 and H12 = 228. The horizontal overlap ratio OLW is OLW = 265/339 ≒ 0.78, that is, 78%. The vertical overlap ratio OLH is OLH = 228/252 ≒ 0.90, that is, 90%.

  The duplication ratio is not limited to the above method, and may be processed by the following method, for example. Consider the state of an object in a surface area represented by a pixel color, a feature point, or the like in the image content shown in the image. It is possible to extract feature points, color regions, and the like from an image by known image processing. The computer system 1 determines the position of the same thing (represented by a feature point or the like) in each image, and determines the overlapping area between the images based on the position of the same thing. The computer system 1 calculates an overlap rate from the size of the overlap area (represented by the number of pixels).

[SFM processing]
FIG. 13 is an explanatory diagram of a known SFM. The SFM uses a plurality of images obtained by photographing the same object (represented by a plurality of feature points) from a plurality of viewpoints (camera positions), and information on the three-dimensional coordinates of the object, the camera direction and the camera position. (Camera parameters). For example, the information can be obtained using three images from three camera positions. In SFM, three-dimensional coordinates are estimated and calculated based on how much the same feature point of the target object has been displaced using parallax based on the movement of the camera. The calculated three-dimensional coordinates represent the structure and shape of the object. In the example of FIG. 13, three images PIC <b> 1 to PIC <b> 3 continuously photographed with respect to the object OB (for example, a cube) are shown. The characteristic points A and B are shown as examples of the characteristic points of the object OB. In the SFM processing in which the images PIC1 to PIC3 are input, a feature point group (for example, feature points A and B) is extracted from each image, and three-dimensional coordinates of the feature point group are obtained by a factor decomposition method or the like. For example, the coordinates (Xa, Ya, Za) of the feature point A and the coordinates (Xb, Yb, Zb) of the feature point B are obtained. At the same time, the camera direction (for example, directions J1 to J3) and the camera position (for example, positions E1 to E3) are estimated. As for the structure three-dimensional model generation function, various applied methods based on the SFM method and other methods can be similarly applied.

[List screen (1)]
FIG. 14 shows a first example of the list screen 401 in the embodiment. On the list screen 401, a list of image groups is displayed. In this example, a plurality of images (indicated by, for example, # 1 to # 24) of the image group 201 are arranged in a thumbnail format in a vertical and horizontal manner in an order using the ID and the shooting date and time in the image information 202. Is displayed. These image groups are, for example, image groups taken in consideration of a predetermined overlapping rate for generating a three-dimensional model. As another display example, not only the thumbnail but also the information of the image information 201 may be displayed for each image.

[Individual screen (1)]
FIG. 15 shows a first example of the individual screen 402. On the individual screen 402, the first image selected by the user from the list screen 401 is displayed in a sufficiently large main area frame 1501. Also, when the second image is selected by selecting the link image on the individual screen 402, the second image is displayed in the frame 1501. Note that the user can return to the list screen 401 by performing a predetermined operation such as pressing a “return to list” button 1502 in the individual screen 402.

  In the individual screen 402, a link image 403 is displayed around the first image. The link image 403 is, in other words, a navigation image. In the example of FIG. 15, the link images 403 are displayed outside the first image (frame 1501) at the top, bottom, left, right, and diagonally (upper right, lower right, upper left, lower left) positions. The link image 403 is shown as having an arrow shape, but is not limited thereto, and may be a GUI component, an icon image, or the like having a predetermined shape. Another example of the link image 403 may be a thumbnail image.

  These link images 403 are images and parts for selecting an adjacent image in the direction indicated by the arrow as the next image. These link images 403 are images and components for selecting an adjacent image in a certain plane. The user can perform an operation (such as tapping) for selecting the link image 403.

  In the GUI example of FIG. 15, all of these link images 403 are displayed in each direction regardless of the presence or absence of an actual adjacent image in each direction. The user selects and operates a desired link image 403 (for example, a right arrow). In that case, if there is an adjacent image in the direction represented by the selected link image 403, the adjacent image is selected as the second image. That is, the screen display state is changed so that the selected second image is displayed instead of the first image in the frame 1501 of the individual screen 402. The second image is treated as a new first image, and the link image 403 is displayed similarly. If there is no adjacent image in the direction represented by the link image 403 selected and operated by the user, no transition is made.

  Also, in this GUI example, when the user selects and operates a desired link image 403 (for example, a right arrow), the direction represented by the link image 403 in the same plane (for example, side A1) where the first image is located. If there is an adjacent image, the adjacent image is selected. Also, if there is no adjacent image in the same plane in the direction represented by the link image 403, and there is an image in another plane in the back or front direction with the aforementioned corner interposed, The image is selected.

  As another example of the GUI, the link image 403 may be displayed so as to be superimposed on the image in the frame 1501.

  As another GUI example, the link image 403 may not be displayed, and the adjacent image (second image) may be selected according to a predetermined operation specified in advance by user settings or the like. For example, a function as a link in each direction may be assigned to each key of the keyboard.

  As another GUI example, instead of the link image 403, an adjacent image in a direction corresponding to the position of the line of the frame 1501 may be selected by a predetermined operation on the position of the line. For example, when the selection operation near the right side of the frame 1501 is performed, the same function as the selection operation of the right arrow is performed.

  As another GUI example, a well-known operation such as dragging and swiping may be used instead of the link image 403. For example, when a swipe operation from right to left is performed within the frame 1501, the same function as the selection operation of the right arrow is performed.

[Individual screen (2)]
FIG. 16 shows a second example of the individual screen. In the example of the GUI in FIG. 16, when there is an image of another surface with a corner interposed (a closest image corresponding to an adjacent image) around the first image, the image and the direction of the type are changed. A specific link image 404 to be displayed is displayed. The link image 404 has a different shape from the link image 403. For example, if there is no adjacent image on the right side in the same side of the first image and there is an image on the other side near or behind the corner, a link image 404 indicating that is displayed. Is done. In this example, when there is an adjacent image in the right back direction with respect to the first image (for example, image # 21 of side surface a3 with respect to image # 3 of side surface a2 near corner 602 in FIG. 25), the right back direction is changed. The displayed link image 404 is displayed. When there is an adjacent image in the right front direction with respect to the first image (for example, an image of the side surface a2 with respect to the image of the side surface a1 near the corner 601 in FIG. 25), a link image 404 indicating the right front direction is displayed. You.

  A link image 405 is shown as an example of another link image. If there is an adjacent image above the first image in the same plane in the Z direction (direction Z1), a link image 403 of an arrow indicating the upward direction is displayed. If there is no adjacent image above the first image in the same plane in the Z direction (direction Z1) and an image corresponding to a horizontal plane such as the roof of the structure 5 exists, a link image 405 indicating that. Is displayed. The link image 405 has a shape (for example, a shape that is obliquely bent) representing the upper back direction. By the selection operation of the link image 405, the image located in the upper depth direction can be selected as the next image.

  A link image 406 is shown as an example of another link image. The link image 406 is a link image indicating that, when there is no adjacent image that minimizes overlap in a certain direction in the same plane with respect to the first image, but there is an image at a distance without overlapping, the link image Will be displayed as The link image 406 has a different shape from the link image 403. For example, when there is no adjacent image below the first image (direction Z2) but there is an image at a distant position, a link image 406 representing a position distant in the downward direction is displayed. By the operation of selecting the link image 406, an image at a position apart from the image can be selected.

[Individual screen (3)]
FIG. 17 shows a third example of the individual screen. In the GUI example of the individual screen 402, the link image 403 is limitedly displayed according to the presence or absence of the adjacent image. In this example, it is assumed that adjacent images exist at the left, lower, and lower left positions as adjacent images in the same plane with respect to the first image, and do not exist at other positions. In this case, as shown in the figure, a link image 403 of three arrows indicating left, lower, and lower left is displayed. Further, in this example, when an image on the other side exists on the right side of the first image, particularly on the position on the right back side, a link image 404 of an arrow showing the right back side is displayed as shown in the figure. I have. In this GUI example, the amount of display information is limited, so that the user can work more easily.

[Individual screen (4)]
FIG. 18 shows a fourth example of the individual screen. In the individual screen 402, when displaying the first image in the frame 1501, the image content of the overlapping area with the adjacent image is not displayed as it is, but the overlapping area is deleted (in other words, masked). I do. More specifically, the overlapping area is displayed by, for example, blackening or hatching. In this example, the first image (or the second image) has an overlapping area 408. The overlapping area 408 is located on the left side of the image selected as the next image and displayed in the frame 1501 in the vicinity of the right side of the previous image (the area is indicated by a broken line for the sake of explanation). The vicinity is an overlapping area.

  By deleting and displaying the overlapping area 408 in this manner, the user does not need to look at the image content of the overlapping area 408 in the selected image, and the visual information amount is reduced, so that the work can be performed. It's easy to do. In this example, the user knows that the overlapping area 408 of this image has been subjected to the deterioration inspection in the previous image, so that it is not necessary to perform the deterioration inspection of the overlapping area 408 again at the time of the deterioration inspection of this image. Therefore, the efficiency of the deterioration inspection work can be improved.

  As another example of the GUI, the overlapping area 408 may be further displayed with information indicating that the deterioration has been inspected. As another example of the GUI, the overlapping area 408 may be displayed in a frame line expression.

[Individual screen (5)]
FIG. 19 shows a fifth example of the individual screen. On this individual screen 402, the unphotographed area (missing area) is displayed in a predetermined expression so that the user can understand it. An unphotographed area (missing area) is an area where a corresponding image does not exist. FIG. 19A shows an example of the relationship between two images. The case where the image 191 and the image 192 overlap and the overlapping rate is large is shown. The unphotographed area 193 is an area outside these two images, and is an example of an area that has not been photographed in an image group including these two images. The unphotographed area 193 among the areas such as the side surfaces of the structure 5 does not exist as an image. Therefore, the unphotographed area 193 cannot be a target of a deterioration inspection or the like. On the individual screen 402, such a portion of the unphotographed area 193 is displayed so as to be understood by the user. Accordingly, the user can recognize the location of the unphotographed area 193, recognize the location where the deterioration inspection has been omitted, and perform the degradation inspection by separately capturing the location.

  Also, a non-photographed area 193b of another example is shown. The unphotographed area 193b is an area where there is a gap between the image 192 and the image 194. Basically, the images are photographed so as to be overlapped between the images. However, for some reason (for example, an error during aerial photographing), it is conceivable that such a gap is generated and the unphotographed area 193b is generated. In this case as well, the image is displayed so that the existence of the unphotographed area 193b can be recognized.

  FIGS. 19B and 19C show display examples of the unphotographed area 193 on the individual screen 402 corresponding to FIG. First, the example of (B) shows an example in which the selected image 191 is displayed in the frame 1501 at the image size of the image 191 as it is. For the purpose of explanation, the overlapping image 192 is also indicated by a frame line. At this time, an unphotographed area 193 exists outside the frame 1501. As described above, when the unphotographed area 193 exists outside the frame 1501, an image representing the unphotographed area 193 may be displayed in a predetermined expression (for example, a dot pattern).

  The example of (C) shows a case where the image 191 is shifted (translated in parallel) and displayed according to a predetermined operation of the user from the state of (B). The image display system also has a function of shifting an image displayed in the frame 1501 on the individual screen 402, a function of enlarging / reducing the image, and the like. For example, this shift display is enabled by an operation such as dragging or swiping on the first image, and enlargement / reduction display is enabled by a pinch operation or the like on the first image.

  In this example, the center position (position Q1a) of the image 191 is moved to, for example, a slightly upper left position Q1b by the shift operation from the state of (B). Due to this shift, a part of the image 191, a part 193 c of the unphotographed area 193, and a part 192 c of the image 192 are put in the frame 1501. On the individual screen 402, an image portion included in the frame 1501 is displayed, and a part 193c of the unphotographed area 193 is also displayed in a predetermined expression.

[Individual screen (6)]
FIG. 20 shows a sixth example of the individual screen. In the individual screen 402, when there is an adjacent image that overlaps with a certain first image, the image is displayed by a frame line or the like so that the user can recognize the existence of the adjacent image. In this example, when there is an adjacent image on the right side of the first image in the frame 1501, not only the link image 403 but also an outline image 409 (for example, a broken line) representing the outline (side) of the adjacent image is displayed. indicate. This allows the user to know that, for example, there is an adjacent image on the right side of the first image, and also knows the degree of overlap. When the user wants to select the adjacent image, the user operates the link image 403 or the outline image 409 thereof. As a result, the adjacent image is selected and displayed in the frame 1501.

  In the example of FIG. 20, a GUI component (duplication ratio threshold setting column) 1503 for user setting of the duplication ratio threshold TOL is displayed in the individual screen 402. The GUI component 1503 is provided with a slide bar or the like so as to display the currently set overlapping rate threshold TOL and allow the user to change the TOL value. In this example, the user can check the state of the TOL value on the individual screen 402. In addition, the user can change the setting of the TOL value by operating the GUI component 1503 according to the application, the working state, and the like. Also, as in the example of FIG. 20, the actual overlap ratio between the first image and the adjacent image may be displayed.

  As for the setting of the overlap ratio threshold TOL, the range may be set not only by one value but also by two values. For example, the range of the overlap rate is set to be 40% or more and 60% or less, and in that case, an image corresponding to the range can be selected.

[Deterioration inspection screen]
FIG. 21 shows an example of the deterioration inspection screen. In the image display system according to the embodiment, the individual screen 402 is a deterioration inspection screen as it is. With the individual image selected on the individual screen 402, the user can perform the deterioration inspection work on the image as it is. On the deterioration inspection screen, the selected image is displayed as a deterioration inspection target image in a frame 1501.

  The deterioration inspection includes, for example, a visual inspection by a person and an automatic diagnosis by a computer. The present image display system supports both of these functions. In this example, the user performs a visual inspection on the image in the deterioration inspection screen. The user visually checks this image and determines whether or not there is any deterioration and the location. For example, a case where there is a deteriorated portion 2101 such as a crack in an image is shown. On the deterioration inspection screen, as an example of the deterioration inspection support processing, a deterioration spot found by a user through a visual inspection can be marked and displayed. The user performs marking on the deteriorated portion 2101 by a predetermined operation, for example. An example of a marking operation is an operation of enclosing a frame by dragging or the like. A red frame 2102 is displayed as an example of the marking image.

  Further, image information, structure information, and the like may be displayed outside the frame 1501. Outside the frame 1501, a check item for deterioration inspection (an inspected / uninspected state can be input), a deterioration presence / absence item, a comment input column, and the like may be provided.

  When the automatic diagnosis function is used, the computer system 1 estimates a deteriorated portion using a target image as an input. The image display system displays automatic diagnosis result information by the computer system 1 on a deterioration inspection screen. For example, the estimated deteriorated portion is displayed in a predetermined expression. The user confirms or makes a final decision by looking at the result information.

  In the image display system according to the modified example, a deterioration inspection screen may be provided separately from the individual screen 402, and transition may be made between these screens based on a predetermined operation (for example, double tap).

[Structure 3D model screen]
FIG. 22 shows an example of a structure three-dimensional model screen. This image display system displays a structure three-dimensional model screen when a predetermined operation is received on the list screen 401 or the like. The present image display system displays a structure three-dimensional model in an area within this screen based on the three-dimensional model data d5, and also displays structure information and the like. In this area, for example, a three-dimensional structure is superimposed on a map (eg, a site) and displayed in a perspective view. According to a predetermined operation on this screen, the direction and position of viewing the three-dimensional structure model can be changed.

  Further, a region such as a side surface may be selected and designated by a user's predetermined operation with respect to the three-dimensional structure model. When, for example, a certain aspect is selected and designated, the image display system displays an image group associated with the aspect on the list screen 401. Alternatively, when a certain position in the three-dimensional structure model is selected and designated, an image associated with the position may be displayed on the individual screen 402.

[List screen (2)]
FIG. 23 shows a second example of the list screen. On the list screen 401, among the original image groups, a group of images to be subjected to deterioration inspection is displayed in a list. First, the user selects a group of images to be subjected to deterioration inspection through the individual screen 402 described above. After that, the image display system displays a list of the images (selected images) to be subjected to the deterioration inspection on the list screen 401. The selected image group may be displayed by a predetermined operation (for example, pressing a selected image group display button) in the list screen 401. Further, on this screen, the deterioration inspection information (for example, inspection completed, presence or absence of deterioration, etc.) may be displayed together for each image with respect to the image group to be subjected to deterioration inspection (selected image group).

[List screen (3)]
FIG. 24 shows a third example of the list screen. This list screen 401 shows, as a modification example of the list screen 401 of FIG. 23, another method for displaying a list of the image group to be subjected to deterioration inspection. In the list screen 401, the image group to be subjected to the deterioration inspection is arranged and displayed in a positional relationship along the shape of the surface area of the structure 5 (or the generated structure three-dimensional model). For example, in an area 2401 in the list screen 401, an image group (adjacent image group) to be subjected to deterioration inspection is displayed for each selected side group. For example, regarding the side surface a2 in FIG. 25, there are images # 1 to # 12 as an adjacent image group. In this area, first, an area 2401 corresponding to the side surface of the selected group (that is, a partially planar area) is displayed. In the area 2401, a plurality of adjacent images (images # 1 to # 12) are arranged and displayed according to the actual positional relationship. In addition, for each image (thumbnail), it may be displayed in a predetermined expression so that the state such as the inspection completed or the presence or absence of deterioration can be understood. When one image in the area 2401 is selected, the image is displayed on the individual screen 402.

  Further, a link image 2402 for selecting another side group may be displayed in the area 2401. For example, when the right arrow link image 2402 is selected, the group of the side surface a3 is selected and displayed in the area 2401 in the same manner. Further, in the example of displaying the link image 2402 in units of side groups in FIG. 24, the right and left arrows are used, but the present invention is not limited to this, and various expressions can be used. For example, similarly to the example of the link image 404 in FIG. 16, the link image 2402 may have an arrow shape indicating that the link image changes in a three-dimensional space in a depth direction or a front direction. For example, a link image 2402 for a link from the side surface a2 to the side surface a3 is displayed in a curved arrow shape representing the right back.

[Judgment of adjacent images]
The determination of the adjacent image (next image) based on the spatial positional relationship, the GUI, and the like will be described with reference to FIGS. In particular, as in the example of FIG. 6, a case will be described in which the positional relationship between images on the side surfaces interposing the corners of the structure 5 and the selection of the next image are determined. A plurality of images (and shooting ranges) are arranged in a three-dimensional space. When handling an image group in which a corner, a different side surface, a curved surface, or the like of the three-dimensional structure 5 is photographed, the camera direction C of each image may be different. In such images having different camera directions C, it may be difficult to determine the overlap ratio and the adjacent image. Therefore, in the present image display system (particularly, the adjacent image determination unit 303), even when the camera direction C is different, the positional relationship between the images is determined using the information on the position and direction of the image (for example, the camera direction C and the camera direction V). Has the function of determining the next image for a certain image. In the present image display system, when calculating the overlap ratio of a set of images in space, the set of images is determined using the positional information of the images in consideration of efficiency.

  FIG. 25 shows an example of an image relationship between the side surfaces of the structure 5 corresponding to the example of FIG. In this example, examples of adjacent images on the side surface a2 and the side surface a3 of the structure 5 are shown. Here, the adjacent image indicates an image group selected as an image suitable for deterioration inspection. Although each image is indicated by a broken-line frame, in reality, the ends of each image partially overlap.

  In the side surface a2, there is a corner 601 on the left side in the plane, on the far side in the Y direction (direction Y1), and on the right side in the plane, on the near side in the Y direction (direction Y2). In the side surface a3, the corner 602 is on the left side (direction X1) in the plane, and the corner 603 is on the right side (direction X2) in the plane. On the route r2 of the group on the side surface a2, an image group is captured by aerial photography like the above-described scanning. Images # 1 to # 12 show examples of adjacent image groups selected to minimize overlap among the image groups. Similarly, on the route r3 of the group on the side surface a3, an image group is captured by aerial photography such as the above-described scanning. Examples of adjacent image groups selected to minimize the overlap among the image groups are shown as images # 21 to # 32. These image numbers also correspond to the order of the shooting date and time. Images that are not selected due to a high overlap rate are not shown.

  The image display system displays one image selected by the user on the individual screen 402. For example, assume that image # 3 is selected as the first image. In this case, considering only the YZ plane of the side surface a2, there are images # 2, # 4, and # 5 as adjacent images. Here, in a three-dimensional space, as shown, at the corner 602, the left end of the side surface a3 is connected to the right end of the side surface a2. When the user views the image # 3 on the individual screen 402, there is no adjacent image on the right side (direction Y2) of the image # 3 because there is no structure 5. However, when considering a three-dimensional positional relationship along the surface shape of the structure 5 including the direction of bending from the side surface a2 to the side surface a3, in the direction going to the far right with respect to the image # 3, An image # 21 of a3 exists. Similarly, when viewed from the image # 21 of the side surface a3, there is an image # 3 of the side surface a2 in the direction going to the far left.

  For applications such as deterioration inspection, it is useful to be able to select and display between these images even in the case of the positional relationship between the images # 3 and # 21. Thus, in the present image display system, even when the image of one side and the image of another side are continuous (when there is an image whose position is different in the camera direction is close), selection and display between the images can be performed. It has a function that enables For this reason, in the present image display system, the three-dimensional positional relationship between the images is determined using the information on the position and direction of the image (FIG. 26), the adjacent images between the side surfaces are also determined, and a link image is determined. Present. For example, when the image # 3 is displayed on the individual screen 402, as the link image, not only the link image indicating the left or the bottom, but also the link image indicating the right back (the link image 404 in FIG. 17) is displayed.

  In the case of the adjacent images between the side surfaces as described above, there is a case where there is no overlapping region between the images. Therefore, the present image display system mainly determines the three-dimensional positional relationship and the distance, not the overlap ratio in the same plane, when determining the adjacent image between the side surfaces (see FIG. 4). S10 etc.).

  Similarly, in the relationship between the side surface a2 and the side surface a3, a pair of the image # 4 and the image # 26, a pair of the image # 9 and the image # 27, a pair of the image # 10 and the image # 32, Is obtained. Similarly, adjacent images can also be obtained on the side surfaces interposing other corners.

  FIG. 26 shows an example of image selection determination between the side surfaces of the structure 5 corresponding to FIG. In FIG. 26, the side surface a1, the side surface a2, the side surface a3, the corner 601 and the corner 602 are shown in a bird's-eye view. Positions c1, c2, and c3 are shown as examples of the camera position C at the time of shooting along the route r1 of the side surface a1. These positions are positions near the corner 601 of the side surface a1. At any position, the camera direction is the direction Y1, and the target distance D0 is constant. For example, it shows an image g11 (especially the width of the photographing range) photographed at the position c3. Further, positions c4, c5, c6, and the like are shown as examples of the camera position C at the time of shooting along the route r2 of the side surface a2. These positions are positions near the corner 601 of the side surface a2. At any position, the camera direction is the direction X2, and the target distance D0 is constant. For example, it shows an image g12 (particularly the width of the photographing range) photographed at a position c4 (it is assumed to be almost the same position as the position c3).

  The image display system determines not only the adjacent image in the same side surface but also the image (image g12) of a different side surface as the adjacent image to the image g11 of the side surface a1. At this time, the present image display system calculates and determines the distance between the camera positions C using the position information of the camera position C of each image. The image display system determines, for example, an image in which the distance between the camera positions C is the smallest with respect to the position c3 of the image g11, except for images in the same side surface (images of the positions c1, c2, etc.). In this example, the position c4 has the smallest distance with respect to the position c3. Therefore, the image g12 at the position c4 is picked up as an adjacent image between the side surfaces. The image display system associates the image g11 with the image g11 as an adjacent image between the side surfaces (in the case of the side surface, it is called an adjacent image even if there is no overlap), and displays a corresponding link image.

  As another processing example, the present image display system may determine the above-mentioned distance approximately using the information of the drone position P instead of using the camera position C. Further, as another processing example, the present image display system may determine the distance using the information of the corresponding image and the position Q of the center of the shooting range instead of the camera position C. For example, it is assumed that the center position of the image g11 is the position q3 and the center position of the image g12 is the position q4. The image display system determines an image in which the distance between the center positions of the images is the smallest other than the images in the same side surface. In this example, the position q4 has the smallest distance with respect to the position q3. Therefore, the image g12 at the position q4 is picked up as an adjacent image between the side surfaces.

  Similarly to the above, it is possible to determine the adjacent image also in the relationship between the side surface a2 and the side surface a3 that interpose the corner 602. For example, when the image g13 at the position c13 (or the position q13) on the side surface a2 is the first image, the image g14 at the position c14 (or the position q14) on the side surface a3 can be picked up as an adjacent image.

  When the image g11 is displayed on the individual screen 402, a link image representing the front right (direction Y2) is displayed as the link image. It is possible to transit to the image g12 by selecting this link image. When the image g12 is displayed on the individual screen 402, a link image representing the front left (direction X1) is displayed as the link image. It is possible to transit to the image g11 by selecting this link image. When the image g13 is displayed on the individual screen 402, a link image representing the right back (direction X2) is displayed as the link image. It is possible to transit to the image g14 by selecting this link image. When the image g14 is displayed on the individual screen 402, a link image representing the back left (direction Y1) is displayed as the link image. It is possible to transit to the image g13 by selecting this link image.

  The transition of the display image via such a link image corresponds to a change in the user's line of sight corresponding to the change in the camera direction C. The GUI of the present image display system enables image selection and display image transition in a direction along the surface shape of such a three-dimensional structure 5. Thereby, the user can more efficiently perform the work according to the use such as the deterioration inspection.

  It is to be noted that the present invention is not limited to the above-described embodiment in which the positional relationship between images is determined using information on the position and direction of the image. As a modification, the positional relationship of the images may be determined based on the determination of the feature points and the like extracted from within the image by the image processing.

[Aerial photography method (3)]
FIG. 27 shows a third example of an aerial photographing method of another structure 5 as an application example. The structure 5 of this example shows a case having a columnar shape. When the area of the side surface A30 having the curved surface of the structure 5 is to be subjected to deterioration inspection, the following aerial photography method is used. 27A shows a perspective view, and FIG. 27B shows an overhead view in the Z direction. In the aerial photography setting example of this method, as shown in the drawing, the drone 10 and the camera 4 are moved, for example, circumferentially in the horizontal direction with respect to the curved side surface A30 so that the target distance D0 is substantially constant. Route R30. In the route R30, the camera is controlled so as to face, for example, perpendicularly to the side surface A30 of the curved surface. For example, an image g31 at the camera position c31 on the route R30, an image g32 at the camera position c32, an image g33 at the camera position c33, and the like are shown.

  Also in the case of an image group obtained by such an aerial photography method, the processing of the image display system can be similarly applied, and substantially the same effect can be obtained. The image display system similarly determines the overlap ratio, adjacent images, and the like for a plurality of images (imaging ranges) formed along the side surface A30. In this case, on the individual screen 402, each image can be selected in a direction along the side surface A30. In the case of this aerial photography method, in particular, by setting the target distance D0 to be constant, it is possible to easily perform the deterioration inspection and to ensure the accuracy of the SFM processing. The method is not limited to the aerial photographing method, but may be a method of dividing into a plurality of linear routes as described above.

[Processing timing]
In the image display system according to the embodiment, the computer system 1 executes the processing of the overlap ratio and the determination of the adjacent image at each timing when a certain first image is selected by a user operation, as shown in the flow (S6) of FIG. are doing. The processing timing is not limited to this, and is possible. In a modified example, the computer system 1 (for example, the server 3) calculates the overlap ratio and the adjacent image judgment for the image group at the timing when the data of the image group is input or when the execution of the batch processing is designated by the user in advance. Execute the process as a batch process. As a result of this processing, the target image group can be collectively selected with respect to the original image group according to the use such as the deterioration inspection. For example, as shown in the example of FIG. 25, a group of adjacent images having a positional relationship in a three-dimensional space can be configured. The computer system 1 manages the information of the adjacent image group formed by this processing together with the three-dimensional model data of the structure 5 and the like. Then, the computer system 1 can display information of the adjacent image group on a GUI screen (for example, FIGS. 23 and 24) in accordance with a user operation.

  Also, the input image group may be changed. For example, some images may be added later. For example, when there is an unphotographed area in the first photography, the second photography is performed for the unphotographed area as a target. In such a case, the computer system 1 performs the same process on the added image. That is, the computer system 1 re-executes the similar duplication ratio calculation and the adjacent image determination process using the additional image as the first image and the processed image group as the candidate image. Thereby, an adjacent image group is formed for the additional image, and the relationship between the image groups is reconstructed and updated.

[Effects]
As described above, according to the image display system of the embodiment, when selecting an image for use such as deterioration inspection from a group of images taken by the camera 4 of the drone 10, a suitable image is easily selected. This can reduce the work of the user. Depending on the application, the user checks and works by seeing a certain image (first image), then selects from images surrounding the image, and checks and works by viewing the image (second image). In many cases. In such a case, the present image display system can efficiently realize selection and operation of an image group. In the embodiment, the user can efficiently perform the work of the visual inspection for the purpose of the deterioration inspection. Even when automatic diagnosis is performed by the computer, the computer can efficiently perform processing on the selected image group.

  As described above, the image display system according to the embodiment has a mechanism for judging an overlapping state of an object in an image with respect to a plurality of two-dimensional images having a positional relationship in a three-dimensional space, It has a mechanism such as a GUI for navigating selection and transition between images according to the overlapping state. In particular, in the present image display system, it is possible to select and transition an image in a positional relationship and a direction along the three-dimensional shape of the surface of the structure 5.

  In the present image display system, when the user selects the first image and displays it on the individual screen 402, it searches for an image around the first image, determines an adjacent image based on the overlap rate, and The link image to be presented is presented. As described above, in the present image display system, since the processing is performed efficiently, the waiting time of the user can be reduced.

  The present image display system is applicable not only to the use for the deterioration inspection and the three-dimensional model generation but also to other uses. Even when applied to other uses, a suitable image can be selected according to the overlap rate according to the use between the image group and the image, and work and processing according to the use can be efficiently performed.

  As a modified image display system, a form without a three-dimensional model generation function is also possible.

  In the present image display system, the input image group is obtained by hand or by using the drone 10. However, the present invention is not limited to this, and the input image group may be obtained by any means. In addition, the present image display system has described a case in which an image group used for both the purpose of the deterioration inspection and the generation of the three-dimensional model is handled. In particular, a case has been described in which a partial image group suitable for deterioration inspection is selected from an image group photographed so that a three-dimensional model can be generated. The present invention is not limited to this, and the present image display system is effective when a second image group for another second use is selected from a first image group taken for a certain first use.

  As described above, the present invention has been specifically described based on the embodiments. However, the present invention is not limited to the above-described embodiments, and can be variously modified without departing from the gist thereof.

  DESCRIPTION OF SYMBOLS 1 ... Computer system, 201 ... Image group, 202 ... Image information, d1: Structure data, d2 ... Photographing plan data, d4 ... Inspection data, d5 ... 3D model data, 121 ... Deterioration inspection support part, 122 ... 3D Model generation unit, 122A: SFM processing unit, 123: Image display control unit, 301: List display unit, 302: Individual display unit, 303: Adjacent image determination unit, 304: Overlap ratio calculation unit, 401: List screen, 402 ... Individual screen.

Claims (30)

An image display system configured by a computer system,
The computer system includes:
Enter a group of images containing multiple images with different shooting date and time, position, and direction,
Displaying a list of the image group on a list screen,
Displaying, on an individual screen, a first image selected from the image group based on a user operation;
Neighboring to the first image is determined based on a determination of a spatial positional relationship in a set of the first image and a candidate image spatially peripheral to the first image and a determination of an overlapping state regarding a shooting range. Judge the image,
On the individual screen, based on an operation of the user, the adjacent image to the first image is selected as a second image, and the second image is displayed as a new first image.
Image display system. The image display system according to claim 1,
The computer system displays, on the individual screen, a link image representing the existence and direction of the adjacent image with respect to the first image, and displays the adjacent image on the basis of the user's operation on the link image. Selecting the two images and displaying the second image as the new first image;
Image display system. The image display system according to claim 1,
The computer system calculates an overlapping rate of the photographing range as the overlapping state with a set of the first image and the candidate image, and calculates the candidate image in which the overlapping rate is minimum within a range of an overlapping rate threshold. Is selected as the adjacent image,
Image display system. The image display system according to claim 1,
The computer system uses the image information including the shooting position of each image in the image group, and sets the position of the first image as the spatial positional relationship in a set of the first image and the candidate image. And calculating the distance between the position of the candidate image and selecting the candidate image having the smallest distance as the adjacent image,
Image display system. The image display system according to claim 1,
The computer system captures the first image as a spatial positional relationship in a set of the first image and the candidate image using image information including a capturing position of each image in the image group. The direction and the direction of shooting of the candidate image are substantially the same, and divided into different cases, and in each case, the adjacent image is determined.
Image display system. The image display system according to claim 2,
The computer system, on the individual screen, displays, as the link image, a link image having a shape indicating a direction in which the adjacent image to the first image exists,
Image display system. The image display system according to claim 2,
As the spatial positional relationship, with respect to a first direction in the first plane where the shooting range of the first image is arranged, a first direction in the second plane where the shooting range of the adjacent image is arranged. When there are two directions,
The computer system displays, on the individual screen, a link image having a shape representing a change from the first direction to the second direction, as the link image.
Image display system. The image display system according to claim 1,
When there is an overlapping area between the first image and the adjacent image,
The computer system, on the individual screen, displays the overlapping area of the first image in a predetermined expression with image content deleted,
Image display system. The image display system according to claim 1,
When there is an unphotographed area around the first image,
The computer system displays, on the individual screen, the unphotographed area around the first image in a predetermined expression indicating that the area is the unphotographed area.
Image display system. The image display system according to claim 1,
When there is an overlapping area between the first image and the adjacent image,
The computer system, on the individual screen, displays the overlap area of the first image in a predetermined expression representing the outline of the overlap area or the overlap area,
Image display system. The image display system according to claim 1,
The computer system displays, on the list screen, a list of image groups selected through the operation of the user on the individual screen among the image groups.
Image display system. The image display system according to claim 1,
The computer system, of the image group, an image group selected through the user's operation on the individual screen is arranged according to the spatial positional relationship, and displayed on the list screen.
Image display system. The image display system according to claim 1,
The group of images is a group of images taken by a camera for use in deterioration inspection of the surface area of the structure,
The individual screen is a deterioration inspection screen that receives an operation for the deterioration inspection work of the user,
Image display system. The image display system according to claim 1,
The group of images is a group of images captured by a camera for use in generating a three-dimensional model of a structure.
Image display system. The image display system according to claim 1,
The image group is a group of images taken aerial by a flying object equipped with a camera,
Image display system. An image display method in an image display system configured by a computer system,
As steps performed in the computer system,
Inputting an image group including a plurality of images having different shooting dates and times, positions, and directions;
Displaying a list of the image group on a list screen;
Displaying, on an individual screen, a first image selected from the image group based on a user operation;
Neighboring to the first image is determined based on a determination of a spatial positional relationship in a set of the first image and a candidate image spatially peripheral to the first image and a determination of an overlapping state regarding a shooting range. Determining an image;
On the individual screen, based on the operation of the user, selecting the adjacent image to the first image as a second image, and displaying the second image as a new first image;
An image display method comprising: The image display method according to claim 16,
The computer system displays, on the individual screen, a link image representing the existence and direction of the adjacent image with respect to the first image, and displays the adjacent image on the basis of an operation performed by the user on the link image. Selecting the two images and displaying the second image as the new first image.
Image display method. The image display method according to claim 16,
The computer system calculates an overlapping ratio of the photographing range as the overlapping state in a set of the first image and the candidate image, and the candidate image in which the overlapping ratio is the minimum within a range of an overlapping ratio threshold. Is selected as the adjacent image,
Image display method. The image display method according to claim 16,
The computer system uses the image information including the shooting position of each image in the image group, and sets the position of the first image as the spatial positional relationship in a set of the first image and the candidate image. And calculating the distance between the position of the candidate image and selecting the candidate image having the smallest distance as the adjacent image,
Image display method. The image display method according to claim 16,
The computer system captures the first image as a spatial positional relationship between the first image and the candidate image using image information including a capturing position of each image in the image group. And the case where the direction of shooting of the candidate image is substantially the same as the direction of shooting the candidate image, and the case where the direction is different, and a step of determining the adjacent image in each case,
Image display method. The image display method according to claim 17,
The computer system further comprises a step of displaying, on the individual screen, a link image having a shape indicating a direction in which the adjacent image to the first image exists as the link image,
Image display method. The image display method according to claim 17,
As the spatial positional relationship, with respect to a first direction in the first plane where the shooting range of the first image is arranged, a first direction in the second plane where the shooting range of the adjacent image is arranged. When there are two directions,
The computer system has a step of displaying, on the individual screen, a link image having a shape representing a change from the first direction to the second direction, as the link image.
Image display method. The image display method according to claim 16,
When there is an overlapping area between the first image and the adjacent image,
The computer system includes a step of displaying the overlapping area of the first image in a predetermined expression with image content deleted on the individual screen,
Image display method. The image display method according to claim 16,
When there is an unphotographed area around the first image,
The computer system has a step of displaying, on the individual screen, the unphotographed area around the first image in a predetermined expression representing the unphotographed area,
Image display method. The image display method according to claim 16,
When there is an overlapping area between the first image and the adjacent image,
The computer system further comprising a step of, on the individual screen, displaying the overlap area of the first image in a predetermined expression representing the outline of the overlap area or the overlap area,
Image display method. The image display method according to claim 16,
The computer system has a step of displaying, on the list screen, a list of image groups selected through the operation of the user on the individual screen among the image groups,
Image display method. The image display method according to claim 16,
The computer system has a step of, among the image groups, arranging an image group selected through the operation of the user on the individual screen according to the spatial positional relationship, and displaying the image group on the list screen.
Image display method. The image display method according to claim 16,
The group of images is a group of images taken by a camera for use in deterioration inspection of the surface area of the structure,
The individual screen is a deterioration inspection screen that receives an operation for the deterioration inspection work of the user,
Image display method. The image display method according to claim 16,
The group of images is a group of images captured by a camera for use in generating a three-dimensional model of a structure.
Image display method. The image display method according to claim 16,
The image group is a group of images taken aerial by a flying object equipped with a camera,
Image display method.

Images