JP6890156B2 - Image generator, image generator and image generator - Google Patents

Description

The present invention relates to an image generator, an image generation method and an image generation program.

Conventionally, regular inspection work has been carried out to confirm whether or not defects such as deterioration have occurred in high-altitude equipment such as steel towers of wireless base stations. For example, deterioration of equipment in high places includes symptoms such as rust on the metal surface, loosening of bolts, and distortion of the housing, and at present, it is usual to visually inspect the presence or absence of these.

Manual inspection of equipment in high places is not efficient in terms of worker safety, labor costs, time and effort required for inspection work, etc. Therefore, in recent years, drones equipped with cameras have been used. Labor saving is achieved by taking an image of equipment at a high place (see, for example, Patent Document 1). Patent Document 1 discloses a method of confirming the state and location of a defect by photographing a portion of the structure to be inspected where a defect has occurred with a camera.

In addition, when an object is imaged using a drone equipped with a camera and the inspection result of the object is reported, a plurality of partially wrapped photographs are taken and imaged for one object. In many cases, all the photos are output to the report and reported.

JP-A-2017-166241

However, if any one rust appears in multiple photos, it will be output as separate rust when the report is output, and whether or not one rust for the report viewer is the same rust as the other photos. Is difficult to grasp and may cause confusion.

Therefore, the present invention is an image generation device and an image generation method capable of generating a three-dimensional image capable of efficiently and clearly visually recognizing the position and state in which deterioration symptoms have occurred from an image obtained by photographing the appearance of high-altitude equipment such as a steel tower. And to provide an image generation program.

In order to solve the above problems, the image generator according to the present invention is an image generator that generates a three-dimensional image from a two-dimensional image captured by a camera mounted on a flying object, and indicates an imaging position of a subject. An acquisition unit that includes position information and acquires a plurality of images that partially overlap each other for each predetermined shooting position, and a color definition unit that defines the colors included in the images that are recognized as deterioration symptoms of the subject. A selection unit that selects one image for each predetermined shooting position from a plurality of images acquired by the acquisition unit, and a color defined by the color definition unit among the images acquired by the acquisition unit are included. An extraction unit that extracts a part to be used, a generation unit that generates a three-dimensional image of a subject using an image selected by the selection unit, and a generation unit that includes the color extracted by the extraction unit. It is characterized by including a display unit for identifiable display in the generated three-dimensional image.

According to the present invention, after defining a color to be recognized as a deterioration symptom, a plurality of partially overlapping images are associated with a three-dimensional image based on a shooting position, and a portion corresponding to the defined color is three-dimensional. Since it is displayed as an image, it is possible to visually check the position where the deterioration symptom occurs efficiently and clearly at a glance.

It is a system diagram which shows the structure of the image generation system which concerns on embodiment of this invention. It is a block diagram which shows the functional structure example of the image generation apparatus in embodiment of this invention. 3 (a) to 3 (c) are diagrams showing images in which a part of the image overlaps at a predetermined shooting position. It is a figure which shows an example of the image which was divided into predetermined regions by the color definition part in embodiment of this invention. It is a figure which showed all the components constituting the subject in embodiment of this invention in one image. It is a figure which shows an example of the 3D image of the steel tower which is the subject generated based on the 2D image by the generation part in embodiment of this invention. It is a flowchart which shows the flow of the 3D image generation processing of a subject by the image generation apparatus in embodiment of this invention. It is a hardware block diagram which shows an example of the computer which can realize the image generation apparatus in this embodiment.

FIG. 1 is a system diagram showing an overall configuration of an image generation system according to an embodiment of the present invention. As shown in FIG. 1, the image generation system 1 includes an image generation device 100 and a projectile 200, which can communicate with each other via a network 10. The network 10 plays a role of connecting each of the image generator 100, the flying object 200a, the flying object 200b, and the flying object 200c to each other. For example, the network 10 may be a wired network or a wireless network. In FIG. 1, two units, a flying object 200a and a flying object 200b, are shown, but there may be a plurality of units (hereinafter, unless otherwise specified, they are collectively referred to as the flying object 200).

The image generation device 100 acquires an image of a steel tower 300a or a steel tower 300b (hereinafter, the steel tower 300a and the steel tower 300b are collectively referred to as a steel tower 300 unless otherwise specified), which is an image-imaging object imaged by the flying object 200. It is a device that analyzes the acquired two-dimensional image of the steel tower 300 and generates a three-dimensional image including the deterioration symptom of the steel tower 300. The steel tower 300 is an example of an object to be imaged, and may be any structure that requires inspection to confirm deterioration symptoms. Other examples include, for example, iron bridges, cultural buildings, statues, and bridges.

The projectile 200 is a device having at least an autonomous flight function and an imaging function, and is, for example, a drone. Specifically, the flying object 200 is provided with a flight control unit that controls flight according to a set flight route, a camera as an imaging function, a GPS function for acquiring the current position, and the like, and is driven by a battery charged in the battery. To do.

FIG. 2 is a block diagram showing a functional configuration example of the image generation device 100 in the present invention. As shown in FIG. 2, the image generation device 100 includes a control unit 110, a communication control unit 120, an input / output unit 130, and a storage unit 140.

The communication control unit 120 is an interface connected to a communication device (not shown) such as an antenna or a router connected to a communication line, a telephone line, or the like. That is, the communication control unit 120 has a function of communicating data with an external device such as a flying object 200 or an information processing device (not shown) as an output destination via a communication line.

The input / output unit 130 is a screen such as a liquid crystal monitor for inputting or outputting information. In FIG. 2, the input / output unit 130 exemplifies a screen in which an input function and an output function are integrated, but the present invention is not limited to this, and an input function can be provided by separately providing an input unit such as a keyboard. And the output function may be separate functional parts.

The storage unit 140 stores the image information database 141. The image information database 141 stores image information and color sample information. Here, the image information is the image data of the subject. Further, the color sample information is sample data showing the colors of rust and cracks, which are examples of deterioration symptoms of the subject, and as sample data, a plurality of sample data are used according to the RGB and brightness of the entire captured image. Data is saved.

The control unit 110 includes an acquisition unit 111, a color definition unit 112, an image classification unit 113, an extraction unit 114, a generation unit 115, a related unit 116, a display control unit 117, and a transmission / reception unit 118.

The transmission / reception unit 118 transmits / receives various information to / from an external terminal such as the flying object 200 or an external device. For example, the transmission / reception unit 118 receives an image of a subject from the projectile 200 and receives map information from an external server. The image of the subject includes the position information of the subject and the camera coordinates. Here, the position information of the subject is information indicating the shooting position of the subject, for example, latitude / longitude information indicating the position of the position where the subject is imaged.

The acquisition unit 111 acquires an image of the subject from the projectile 200 via the transmission / reception unit 118. Specifically, the acquisition unit 111 acquires a plurality of images (for example, eight images at one shooting position) in which some of the images overlap each other at a predetermined shooting position. Further, the predetermined shooting position is a shooting position determined for each part of the subject as a shooting position of the subject, and may be, for example, a shooting position for each member constituting the steel tower 300 which is the subject. , Upper surface, side surface, back surface, lower surface, inner side surface, etc., may be a shooting position uniformly defined as each surface constituting the developed view of the three-dimensional image.

For example, FIGS. 3 (a) to 3 (b) are diagrams showing images in which a part of the image overlaps at a predetermined shooting position. 3 (a) to 3 (c) show three images taken in an overlapping manner when the shooting position of the member 301 included in the steel tower 300 is defined as a predetermined shooting position. is there. In FIG. 3A, the lower part of the member 301 is cut off from the image frame, and the parts other than the lower part overlap with FIG. 3B. Further, in FIG. 3B, the entire view of the member 301 is included in the image frame, which overlaps with a part of FIGS. 3A and 3C. Further, in FIG. 3C, the upper part of the member 301 is cut off from the image frame, and the parts other than the upper part overlap with FIG. 3B.

The color definition unit 112 defines the colors to be recognized as deterioration symptoms of the subject among the colors included in the image acquired by the acquisition unit 111. Specifically, the color definition unit 112 divides the image into regions having a predetermined number of pixels, and defines for each region the colors to be recognized as deterioration symptoms of the subject among the colors included in the divided regions. FIG. 4 is a diagram showing an example of an image divided into predetermined regions by the color definition unit 112. In the image shown in FIG. 4, each of the vertical direction and the horizontal direction is divided into four, and the image is divided into 16 regions in total. In addition, coordinates in the image are given to each area. For example, coordinates (x1, y1) are assigned to the area located in the leftmost column at the bottom of the image, and the x-coordinate of a numerical value that is one coordinate larger than the area located in the right column with respect to the area is assigned. It is assigned, and a numerical y-coordinate that is larger by one coordinate in order with respect to the region located in the upper row with respect to the region is assigned.

Further, the color definition unit 112 defines a color that recognizes rust or crack as a deterioration symptomatism of the subject among the colors included in the divided region. Specifically, the color definition unit 112 refers to the color sample information from the image information database 141, and recognizes each divided region (hereinafter, referred to as a divided region) as a deterioration symptom of the colors included in the region. To define. Further, the color definition unit 112 may define the colors to be recognized as deterioration symptoms by dividing them into a plurality of levels. For example, the color definition unit 112 defines the colors to be recognized as deterioration symptoms in five ranks from a level showing mild deterioration symptoms to a level showing severe deterioration symptoms.

The image classification unit 113 groups a plurality of images acquired by the acquisition unit 111 in association with each predetermined shooting position. For example, the image classification unit 113 groups the three images shown in FIGS. 3A to 3C and associates them with the imaging position of the member 301. The number of images to be grouped may be a plurality for one shooting position.

The related unit 116 associates the image acquired by the acquisition unit 111 with the three-dimensional image generated by the generation unit 115, which will be described in detail later, by position information. For example, the related unit 116 may associate the grouped image group with the three-dimensional image generated by the generation unit 115 with the position information indicated by the predetermined imaging position. Further, the related unit 116 assigns an image ID to the image extracted by the extraction unit 114, and associates the image with the image ID with the position information of the developed image of the three-dimensional image generated by the generation unit 115.

The generation unit 115 generates a three-dimensional image of the subject by using the position information included in the image. The generation unit 115 selects one image from a plurality of images that partially overlap each shooting position, cuts out each component included in the selected image from the image, and generates a developed view of a three-dimensional image. A three-dimensional image is generated based on the position information included in the generated development view. FIG. 5 is a diagram showing all the components constituting the subject in one image. In FIG. 5, the image of each component is developed on one image for convenience, and the image portion showing each component includes position information. Further, FIG. 6 is a diagram showing an example of a three-dimensional image of the steel tower 300, which is a subject generated based on the two-dimensional image by the generation unit 115. Of the three-dimensional images of the steel tower 300 shown in FIG. 6, the members 302, 303, and 304, which are the members with the same reference numerals, are the same as the members 302, 303, and 304 in FIG. 5, respectively. It is the same. More specifically, for example, the generation unit 115 makes the image acquired by the acquisition unit 111 three-dimensional by the processing of aerial triangulation, and generates a frame of the three-dimensional model. Next, the generation unit 115 acquires the relative camera coordinates ((position: xyz coordinates), (rotation: opk coordinates)) included in each image, and specifies the lap ratio and the lap area of each image. Further, the generation unit 115 generates a texture as a color of each place of the component included in each three-dimensional image from the specified lap ratio and the color of the image of the place corresponding to the wrap area. Then, the generation unit 115 attaches the generated texture to the frame of the three-dimensional model to generate a three-dimensional image.

The extraction unit 114 extracts an image including the color defined by the color definition unit 112 from among the plurality of images for each divided region. Specifically, the extraction unit 114 extracts the color portion defined by the color definition unit 112 from the developed view shown in FIG. In FIG. 5, the black-painted portions of the members 302, 303, and 304 of the steel tower 300 indicate the colored portions extracted by the extraction unit 114. Similarly, in the three-dimensional image of the steel tower 300 in FIG. 6, the colored portion is displayed in black on the member 302, the member 303, and the member 304 corresponding to the black-painted portion in FIG. .. Further, the extraction unit 114 extracts the image having the largest amount of color features defined by the color definition unit 112 from the images grouped by the image classification unit 113. Further, the extraction unit 114 may extract an image in which the color feature amount included in the image is included at a position closer to the center of the image from the images grouped by the image classification unit 113. In addition, for example, as equipment for which a deteriorated portion is extracted by the extraction unit 1134, there are a steel tower, a steel pipe column, a concrete column, a building, a bridge, and the like.

The display control unit 117 identifiablely displays the colors included in the image extracted by the extraction unit 114 among the images associated with the related unit 116 in the three-dimensional image. For example, the display control unit 117 highlights and displays the colors included in the image extracted by the extraction unit 114 among the three-dimensional images.

Further, the display control unit 117 outputs the display information capable of accepting the designation of any part of the three-dimensional image generated by the generation unit 115 to the input / output unit 130, and at the place where the designation is accepted. Output the corresponding 2D image. For example, when the display control unit 117 receives the designation of any of the components (members) included in the three-dimensional image or the part, the display control unit 117 is grouped in association with the position information of the place where the designation is received. The image group is displayed on the input / output unit 130. As for the display method, the images of the grouped image group may be displayed one by one, or all the images of the grouped image group or the preview images of a plurality of images are displayed at the same time. All images may be displayed by selection by the user.

Next, the process of generating a three-dimensional image of the subject by the image generation device 100 will be described. FIG. 7 is a flowchart showing a flow of a three-dimensional image generation process of a subject by the image generation device 100.

The acquisition unit 111 acquires an image from the projectile 200 (step S1). The color definition unit 112 divides the image acquired in step S1 into a predetermined area (step S2). The color definition unit 112 defines the color (step S3).

The image classification unit 113 groups images for each shooting position (step S4). The generation unit 115 selects one image for each shooting position (step S5). The generation unit 115 generates a developed view of a three-dimensional image from the image selected in step S5 (step S6).

The extraction unit 114 extracts an image containing the colors defined in step S3 (step S7). For example, the extraction unit 114 extracts an image containing the most defined colors from a plurality of images. The related unit 116 assigns an image ID to the image extracted in step S7 (step S8). The related unit 116 associates the image to which the image ID is given in step S8 with the developed view (step S9).

The display control unit 117 displays the defined color portion of the image extracted in step S7 so as to be distinguishable from other portions (step S10). For example, the display control unit 117 highlights a color portion defined in the three-dimensional image. The generation unit 115 generates a three-dimensional image of the subject using the developed view of the three-dimensional image generated in step S6 and the image highlighted in step S10 (step S11).

As described above, according to the image generation device 100 in the present embodiment, a three-dimensional image is generated using a plurality of images acquired from the flying object 200, and there are a plurality of images showing deterioration symptoms at the same location. In addition, since the plurality of images are associated with the corresponding parts in the generated three-dimensional image, it is possible to generate a three-dimensional image capable of efficiently and clearly identifying the position where the deterioration symptom occurs.

FIG. 8 is a hardware configuration diagram showing an example of a computer 20 capable of realizing the image generator 100 according to the present embodiment. As shown in FIG. 8, the computer 20 includes a CPU (Central Processing Unit) 21, a RAM (Random Access Memory) 22, a ROM (Read Only Memory) 23, an HDD (Hard Disk Drive) 24, and a communication interface (I / F). 25, an input / output interface (I / F) 26, and a media interface (I / F) 27 are provided.

The CPU 21 operates by a program stored in the ROM 23 or the HDD 24, and controls each part. The ROM 23 stores a boot program executed by the CPU 21 when the computer 20 is started, a program that depends on the hardware of the computer 20, and the like.

The HDD 24 stores a program executed by the CPU 21 and data used by the program. The communication interface 25 sends the data received from the external device via the communication line to the CPU 21, and transmits the data generated by the CPU 21 to the external device via the communication line.

The CPU 21 controls an output device such as a display or a printer and an input device such as a keyboard or a mouse via the input / output interface 26. The CPU 21 acquires data from the input device via the input / output interface 26. Further, the CPU 21 outputs the generated data to the output device via the input / output interface 26.

The media interface 27 reads the program or data stored in the storage medium 28 and provides it to the CPU 21 via the RAM 22. The CPU 21 loads the program from the storage medium 28 onto the RAM 22 via the media interface 27, and executes the loaded program. The storage medium 28 is, for example, an optical storage medium such as a DVD (Digital Versatile Disc), a magnetic storage medium, a semiconductor memory, or the like.

When the computer 20 functions as the image generation device 100 in the present embodiment, the CPU 21 of the computer 20 executes the program loaded on the RAM 22 to execute the acquisition unit 111, the color definition unit 112, the image classification unit 113, and the extraction unit. Each function of the unit 114, the generation unit 115, the related unit 116, the display control unit 117, and the transmission / reception unit 118 is realized. Further, the data in the image information database 141 is stored in the HDD 24.

The image generation system is a computer-readable storage medium such as a CD-ROM, CD-R, memory card, DVD (Digital Versatile Disk), or flexible disk (FD) in an installable or executable format. Memorized and provided. The CPU 21 of the computer 20 reads and executes these programs from the above storage medium via the media interface 27, but as another example, even if these programs are acquired from an external device via a communication line. Good.

The image generation system uses, for example, script languages such as ActionScript, JavaScript (registered trademark), Python, and Ruby, and compiler languages such as C language, C ++, C #, Objective-C, Swift, and Java (registered trademark). Can be implemented using.

1 Image generation system 100 Image generation device 110 Control unit 111 Acquisition unit 112 Color definition unit 113 Image classification unit 114 Extraction unit 115 Generation unit 116 Related unit 117 Display control unit 118 Transmission / reception unit 120 Communication control unit 130 Input / output unit 140 Storage unit 141 Image information database 200 Flying object 300 Iron tower

Claims (13)

An image generator that generates a three-dimensional image from a two-dimensional image captured by a camera mounted on a flying object.
An acquisition unit that includes position information indicating the imaging position of the subject and acquires a plurality of images that partially overlap each other at a predetermined shooting position.
A color definition unit that defines the colors that are recognized as deterioration symptoms of the subject among the colors included in the image, and
An extraction unit that extracts an image containing the color defined by the color definition unit from a plurality of images acquired by the acquisition unit, and an extraction unit.
A generator that generates a three-dimensional image of the subject using the plurality of images,
A related part that associates the image acquired by the acquisition unit with the three-dimensional image generated by the generation unit by the position information, and
In the three-dimensional image, a display control unit that identifiablely displays the colors included in the image extracted by the extraction unit among the images associated with the related unit.
An image classification unit that groups a plurality of images acquired by the acquisition unit in association with each predetermined shooting position, and an image classification unit.
Equipped with a,
The extraction unit extracts the image having the largest amount of color features from the images grouped by the image classification unit.
An image generator characterized by. The color definition unit divides the image into regions having a predetermined number of pixels, and defines the colors included in the divided regions that are recognized as deterioration symptoms of the subject for each region.
The extraction unit extracts an image including a region containing the color for each region.
The image generator according to claim 1. The extraction unit extracts an image in which the feature amount of the color contained in the image is contained at a position closer to the center of the image.
The image generator according to claim 1 or 2. The acquisition unit acquires a plurality of images that include the position information and camera coordinates and partially overlap each other at each predetermined shooting position.
The generation unit derives the overlap rate and the overlap area of a plurality of images from the camera coordinates included in the image acquired by the acquisition unit, and identifies and identifies the overlap portion based on the derived overlap rate and the overlap area. Generating a three-dimensional image of the subject based on the duplication rate and the color of the image of the part corresponding to the derived overlapping part,
The image generator according to any one of claims 1 to 3. The color definition unit defines a plurality of ranks according to the degree of the deterioration symptomatology.
The extraction unit extracts an image containing a color corresponding to any of the ranks from the three-dimensional images generated by the generation unit.
The display control unit displays the portion including the color so as to be identifiable by the rank.
The image generator according to any one of claims 1 to 4. The color definition unit defines a plurality of ranks according to the degree of deterioration symptomatology.
The extraction unit extracts an image containing the color having the highest rank among the images acquired by the acquisition unit.
The image generator according to any one of claims 1 to 5. The display control unit highlights the colors included in the image associated with the related unit in the three-dimensional image generated by the generation unit.
The image generator according to any one of claims 1 to 6. The display control unit accepts the designation of any part of the three-dimensional images generated by the generation unit, and outputs the two-dimensional image corresponding to the designated portion.
The image generator according to any one of claims 1 to 7. An image classification unit that groups a plurality of images acquired by the acquisition unit in association with each predetermined shooting position.
With more
The color definition unit divides the image into a region having a predetermined number of pixels.
The display control unit accepts the designation of any part of the three-dimensional images generated by the generation unit, and one image of the grouped image group corresponding to the area corresponding to the designated portion. To output the above,
The image generator according to any one of claims 1 to 8. The color definition unit defines the colors included in the image that are recognized as deterioration symptoms indicating rust or cracks in the subject.
The image generator according to any one of claims 1 to 9. It is an image generation method that generates a three-dimensional image from a two-dimensional image captured by a camera mounted on a flying object.
An acquisition step of acquiring a plurality of images that partially overlap each other for each predetermined shooting position, including position information indicating the imaging position of the subject.
A color definition step that defines a color that is recognized as a deterioration symptomatology of the subject among the colors included in the image, and
An extraction step of extracting an image containing the color defined by the color definition step from a plurality of images acquired by the acquisition step, and an extraction step.
A generation step of generating a three-dimensional image of a subject using the plurality of images, and
A related step of associating the image acquired by the acquisition step with the three-dimensional image generated by the generation step by the position information.
In the three-dimensional image, a display control step for identifiablely displaying the colors included in the image extracted by the extraction step among the images associated with the related step, and
An image classification step of associating and grouping a plurality of images acquired by the acquisition step for each predetermined shooting position, and an image classification step.
With
The extraction step is to extract the image having the largest amount of color features from the images grouped by the image classification step.
An image generation method characterized by. Computer,
An acquisition function that includes position information indicating the imaging position of the subject and acquires multiple images that partially overlap each other at each predetermined shooting position.
A color definition function that defines the colors that are recognized as deterioration symptoms of the subject among the colors included in the image, and
An extraction function that extracts an image containing the color defined by the color definition function from a plurality of images acquired by the acquisition function, and an extraction function.
A generation function that generates a three-dimensional image of the subject using the plurality of images, and
A related function that associates the image acquired by the acquisition function with the three-dimensional image generated by the generation function by the position information, and
In the three-dimensional image, and a display control function for identifiably displaying the color included in the extracted image by the extraction function of the image associated with the related functions,
It is realized as an image classification function that groups a plurality of images acquired by the acquisition function in association with each predetermined shooting position.
The extraction function extracts an image having the largest amount of color features from the images grouped by the image classification function.
An image generation program characterized by. An image generator that generates a three-dimensional image based on an image captured by a camera mounted on a flying object.
A color definition unit that defines the colors that indicate a defect in the subject among the colors included in the image,
Based on an image in which a plurality of images are grouped, an extraction unit that extracts an image containing the largest amount of colors indicating a defect of the subject, and an extraction unit.
In the three-dimensional image of the subject generated by using the plurality of images and in which the position information regarding the imaging position of the subject is associated with the three-dimensional image, the color indicating the defect of the subject is identifiablely displayed. Display control unit and
An image generator characterized by comprising.

Images