JP2020155149A - Control method and control device - Google Patents

Abstract

To suggest a control method and a control device capable of intuitively instructing movement to a mobile body by use of an image photographed by the mobile body.SOLUTION: A control method includes: generating an operation image from an image photographed by a mobile body included in a photographing device; and generating movement information for moving the mobile body in accordance with operation performed on the generated operation image.SELECTED DRAWING: Figure 2

Description

The present disclosure relates to control methods and control devices.

A technique relating to a method of capturing a photograph in which a camera is attached to a flying object that can be operated wirelessly and the camera is used to capture a photograph is disclosed (see, for example, Patent Document 1 and the like). By attaching a camera to the flying object, it is possible to take pictures from the sky or from a place where a tripod cannot be set up. Also, by attaching a camera to the flying object and taking an image, the cost can be reduced compared to using a real airplane or helicopter, it is possible to take an image safely, it is possible to take an image even at low altitude or in a narrow place, and it approaches the target. It brings various advantages such as being able to image.

JP-A-2006-27448

The operation of a moving object such as a flying object or a robot equipped with such a camera usually requires a dedicated controller. Here, if the user can easily specify a movement instruction for the moving body using the image captured by the moving body, even a user who is not accustomed to operating the moving body can move the moving body to a target position. Is expected to be easier.

Therefore, the present disclosure proposes a new and improved control method and control device capable of intuitively instructing a moving body to move using an image captured by the moving body.

According to the present disclosure, operation information is generated from an image captured by a moving body equipped with an imaging device, and movement information for moving the moving body in response to an operation on the generated operation image. Control methods are provided, including the generation of.

Further, according to the present disclosure, an image processing unit that generates an operation image from an image captured by a moving body equipped with an imaging device, and the moving body according to an operation on the operation image generated by the image processing unit. A control device is provided that includes a movement information generation unit that generates movement information for moving the image.

As described above, according to the present disclosure, there is provided a new and improved control method and control device capable of intuitively instructing a moving body to move using an image captured by the moving body. To.

It should be noted that the above effects are not necessarily limited, and together with or in place of the above effects, any of the effects shown herein, or any other effect that can be grasped from this specification. May be played.

It is explanatory drawing explaining the outline of one Embodiment of this disclosure. It is explanatory drawing which shows the system configuration example of the inspection system 10 which concerns on one Embodiment of this disclosure. It is explanatory drawing which shows the functional structure example of the hovering camera 100 which concerns on one Embodiment of this disclosure. It is explanatory drawing which shows the functional structure example of the control terminal 200 which concerns on one Embodiment of this disclosure. It is a flow chart which shows the operation example of the inspection system 10 which concerns on one Embodiment of this disclosure. It is explanatory drawing which shows an example of the screen which is displayed on the display part 210 of the control terminal 200. It is explanatory drawing which shows an example of the screen which is displayed on the display part 210 of the control terminal 200. It is explanatory drawing which shows an example of the screen which is displayed on the display part 210 of the control terminal 200. It is explanatory drawing which conceptually shows the state of the image | image of the bottom surface of the bridge 1 by a hovering camera 100. It is explanatory drawing which conceptually shows the operation of the hovering camera 100 in the inspection system 10 which concerns on one Embodiment of this disclosure. It is explanatory drawing which conceptually shows the operation of the hovering camera 100 in the inspection system 10 which concerns on one Embodiment of this disclosure. It is explanatory drawing which shows an example of the screen which is displayed on the display part 210 of the control terminal 200. It is explanatory drawing which shows the outline in the case of inspecting the bottom surface of a bridge girder 3. It is explanatory drawing which shows the example of the image 20 obtained by stitching the still image captured by the hovering camera 100. It is explanatory drawing which shows the functional structure example of the information processing apparatus 300 which concerns on one Embodiment of this disclosure. It is a flow chart which shows the operation example of the information processing apparatus 300 which concerns on one Embodiment of this disclosure. It is a flow chart which shows the operation example of the control terminal 200 which concerns on one Embodiment of this disclosure. It is explanatory drawing which shows the state that the hovering camera 100 is made to image the direction of the ground. It is explanatory drawing which shows a mode that a user is made to specify a flight path of a hovering camera 100 using a composite image. It is explanatory drawing which shows the state which the hovering camera 100 is made to image the upward direction (the back surface of a bridge). It is explanatory drawing which shows a mode that a user is made to specify a flight path of a hovering camera 100 using a composite image. It is explanatory drawing which shows the state which the control terminal 200 generates and displays a composite image. It is explanatory drawing explaining the generation process of the flight path of the control terminal 200 based on the input of the user with respect to the composite image. It is explanatory drawing which shows the example of the composite image. It is explanatory drawing which shows the example of the composite image. It is a flow chart which shows the operation example of the control terminal 200 which concerns on one Embodiment of this disclosure.

Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In the present specification and the drawings, components having substantially the same functional configuration are designated by the same reference numerals, so that duplicate description will be omitted.

The explanations will be given in the following order.
1. 1. One Embodiment of the present disclosure 1.1. Overview 1.2. System configuration example 1.3. Functional configuration example 1.4. Operation example 1.5. Damage data generation example 15.1 Functional configuration example 1.5.2 Operation example 1.6. Example of flight instruction using composite image 2. Summary

<1. Embodiment of the present disclosure>
[1.1. Overview]
In explaining one embodiment of the present disclosure in detail, first, an outline of one embodiment of the present disclosure will be described.

Humans are indispensable for confirming the condition of structures such as roads, bridges, tunnels, and buildings. Normally, visual confirmation of such a structure is performed by a worker who approaches the structure and visually confirms whether the structure is damaged such as corrosion or cracks, or loosening of connecting members such as bolts. Or, it is common to perform a tapping sound inspection to confirm the presence or absence of these abnormalities.

For the maintenance of bridges, especially concrete bridges, for workers who perform visual inspections and tapping sound inspections of bridge piers and piers, for example, it is necessary to build a foothold on the back of the piers and bridge girders, or for workers. In order to ensure safety and to arrange work vehicles, it was necessary to close some or all lanes. Due to these factors, not only the cost required for inspection, but also the cost required for arranging road guides due to road closure, and the traffic congestion of the detour that may occur due to road closure may become a problem.

In addition, there are bridges that are not easy to build scaffolding because they are built on rivers or the sea, or that scaffolding cannot be built in the first place. Therefore, in view of these circumstances, a technology that is safe, does not affect traffic, and can realize low-cost structure inspection work is desired.

Therefore, in view of the above circumstances, the Disclosers examined a technology that is safe, does not affect traffic, and can realize low-cost structure inspection work. Then, as described below, the Disclosers use an air vehicle equipped with an imaging device (in the following description, the air vehicle equipped with an imaging device is also referred to as a "hovering camera") to be safe. , We have come up with a technology that does not affect traffic and enables inspection work that can be done at low cost.

FIG. 1 is an explanatory diagram illustrating an outline of an embodiment of the present disclosure. In FIG. 1, for example, a bridge 1 constructed of concrete is schematically shown. When inspecting a bridge 1 constructed of concrete, as described above, conventionally, in order for workers to visually inspect whether there is any damage such as cracks or corrosion, the back surface of the pier 2 and the bridge girder 3 is inspected. It was necessary to build a foothold in the concrete, to ensure the safety of workers, and to close some or all lanes in order to place work vehicles.

In one embodiment of the present disclosure, the hovering camera 100 is used when inspecting the bridge 1. The hovering camera 100 is a flying object including an imaging device configured to automatically fly according to preset flight information (in the present embodiment, including information on a flight path and an imaging position of a still image). .. The information on the imaging position of the still image may include, for example, a position for executing the imaging process, a direction for imaging, a moving time to a position for executing the next imaging process, and the like.

For example, when inspecting the back side (bottom surface) of the bridge girder 3, the hovering camera 100 is automatically made to fly, and the hovering camera 100 is made to image the back side of the bridge girder 3. By having the hovering camera 100 image the back side of the bridge girder 3, it is not necessary to build a scaffolding on the back side of the pier 2 or the bridge girder 3 for inspection of the bridge girder 3, the frequency of closing lanes is reduced, or the road is closed. Is no longer needed. Further, for example, when inspecting the lateral side (side surface) of the bridge girder 3, the hovering camera 100 is automatically made to fly, and the hovering camera 100 is made to image the lateral side of the bridge girder 3. Therefore, by automatically flying the hovering camera 100 and causing the hovering camera 100 to image the back side and the side side of the bridge girder 3, the safety of the workers is ensured, the traffic is not affected, and the bridge 1 is operated at low cost. Inspection is possible.

In order to automatically fly the hovering camera 100 and image the back side of the bridge girder 3, it is necessary to set the flight path of the hovering camera 100 and the information of the image pickup position of the still image at the position on the back side of the bridge girder 3. In one embodiment of the present disclosure, it is an object of the present invention to enable efficient inspection of the bridge 1 by efficiently creating flight information set in the hovering camera 100 using information on the general condition of the bridge 1. To do.

The outline of one embodiment of the present disclosure has been described above. Subsequently, a configuration example of the inspection system according to the embodiment of the present disclosure will be described.

[1.2. System configuration example]
FIG. 2 is an explanatory diagram showing a system configuration example of the inspection system 10 according to the embodiment of the present disclosure. The inspection system 10 according to the embodiment of the present disclosure shown in FIG. 2 is a system for the purpose of efficiently inspecting a structure, for example, a bridge 1. Hereinafter, a system configuration example of the inspection system 10 according to the embodiment of the present disclosure will be described with reference to FIG.

As shown in FIG. 2, the inspection system 10 according to the embodiment of the present disclosure includes a hovering camera 100, a control terminal 200, an information processing device 300, a wireless relay node 400, a position estimation node 500, and a base. It includes a station 600, a charging station 700, and a server device 800.

The hovering camera 100 is an example of the image pickup device of the present disclosure, and is an air vehicle including the image pickup device as described above. The hovering camera 100 is a flying object configured to automatically fly based on a designated flight path and to capture a still image with an imaging device at a designated imaging position. The hovering camera 100 can fly by, for example, four rotors, and by controlling the rotation of each rotor, it can fly while moving up, down, and horizontally. Of course, the number of rotors is not limited to this example.

The flight path and imaging position from the flight start position to the flight end position set in the hovering camera 100 are set, for example, as position information of GPS (Global Positioning System). Therefore, the hovering camera 100 may incorporate a GPS receiver that receives radio waves from GPS satellites and calculates the current position. In the flight path set in the hovering camera 100, latitude, longitude, and altitude may all be set as GPS position information, and only latitude and longitude are set as GPS position information. The altitude will be described later, for example. It may be set as a relative height from the base station 600.

The control terminal 200 is an example of the control device of the present disclosure, and is a terminal that executes control regarding flight of the hovering camera 100. The control terminal 200 controls the flight of the hovering camera 100, for example, generation of flight information to be sent to the hovering camera 100, a takeoff instruction to the hovering camera 100, a return instruction to the base station 600 described later, and the hovering camera 100 for some reason. There are operations such as maneuvering the hovering camera 100 when it becomes impossible to fly automatically. The process of generating flight information of the hovering camera 100 by the control terminal 200 will be described in detail later, but an example will be briefly described here.

When generating the flight information of the hovering camera 100, the control terminal 200 reads the information on the general condition of the bridge 1 to be inspected, for example, the general condition map of the bridge 1 to be inspected, and displays it on the screen. The points on the map of the bridge 1 are associated with the points on the map data containing detailed GPS information in advance. It is desirable that this association is performed at a minimum of two sets of points. The flight path of the hovering camera 100 is defined as a GPS value by associating a point on the map data containing detailed GPS information in advance with the general condition map of the bridge 1. Then, the control terminal 200 generates a flight path of the hovering camera 100 based on the general condition map of the bridge 1. The flight path of the hovering camera 100 is superimposed on the general condition map so that the user (construction inspection worker) can easily understand it.

When generating the flight information of the hovering camera 100, the control terminal 200 may consider the structure and dimensions of the bridge 1 and the portion of the bridge 1 to be imaged by the hovering camera 100. When the control terminal 200 generates the flight information of the hovering camera 100, the control terminal 200 may generate the flight information so that the hovering camera 100 can take a detailed image of a portion that is considered to be damaged.

As described above, in the flight path set in the hovering camera 100, latitude, longitude, and altitude may all be set as GPS position information, but there are cases where altitude data does not exist in the general condition map of the bridge 1. Conceivable. If there is no altitude data in the general condition map of the bridge 1, only the latitude and longitude of the flight path set in the hovering camera 100 are set as GPS position information, and the altitude is relative to, for example, the base station 600. It may be set as a height.

When the control terminal 200 sets the flight information to be set in the hovering camera 100, the control terminal 200 may generate the flight information so that the distance to the image target surface becomes constant when the hovering camera 100 images the bridge 1. desirable. When the hovering camera 100 images the bridge 1, the control terminal 200 can cause the hovering camera 100 to generate an image of the same scale by generating flight information such that the distance to the image target surface is constant. You can.

The control terminal 200 is a portable device such as a notebook computer or a tablet terminal, and wirelessly transmits and receives information to and from the hovering camera 100. The control terminal 200 may directly perform wireless communication with the hovering camera 100 with the hovering camera 100, but when inspecting a structure, particularly a bridge 1, hovering exceeds the communicable range of the control terminal 200. Since the camera 100 may fly, it may be performed via the wireless relay node 400 installed at the time of inspection.

The control terminal 200 acquires an image captured by the image pickup device during flight by the hovering camera 100 and displays it as necessary. The control terminal 200 may acquire and display a moving image captured by the image pickup device during flight of the hovering camera 100 by streaming. By acquiring and displaying the moving image captured by the imaging device during flight by the hovering camera 100 by streaming, the control terminal 200 presents to the user what position the hovering camera 100 is flying. Can be done.

The information processing device 300 is a device that processes various information, and may be a device having a function of processing information such as a personal computer (PC) or a game machine. In the present embodiment, the information processing device 300 is a device having a function of displaying an image captured by the hovering camera 100 and allowing the user to confirm the state of the bridge 1. Further, the information processing device 300 has a function of calculating the absolute damage position of the bridge girder 3 from the image captured by the hovering camera 100 and generating the damage data described later. The information processing device 300 may have a function of transmitting the generated damage data to the server device 800. The control terminal 200 may have a function of calculating the absolute damage position of the bridge girder 3 from the image captured by the hovering camera 100 and generating the damage data described later.

The information processing device 300 acquires an image captured by the hovering camera 100 from, for example, the control terminal 200. The information processing device 300 acquires the image captured by the hovering camera 100 at a specific timing. For example, the information processing device 300 is captured by the hovering camera 100 at the timing when one flight of the hovering camera 100 is completed. The image may be acquired from the control terminal 200.

The wireless relay node 400 is a device that relays wireless communication between the hovering camera 100 and the control terminal 200. As described above, in the inspection of the structure, particularly the bridge 1, the hovering camera 100 may fly beyond the communicable range of the control terminal 200. Therefore, wireless communication between the hovering camera 100 and the control terminal 200 can be performed via the wireless relay node 400 installed at the time of inspection of the structure. The number of wireless relay nodes 400 is not limited to one, and a plurality of wireless relay nodes 400 may be installed depending on the inspection range of the bridge 1. Therefore, wireless communication between the hovering camera 100 and the control terminal 200 can be performed via a plurality of wireless relay nodes 400. The hovering camera 100 can switch the communication destination between the control terminal 200 and the wireless relay node 400 according to the radio wave condition.

The radio relay node 400 may be installed at an appropriate location on the bridge surface (preferably on the sidewalk) when inspecting the bridge 1. Further, the wireless relay node 400 may be installed so as to be suspended from the parapet portion of the bridge girder 3. Before the inspection of the bridge 1, it is desirable to confirm that the wireless relay node 400 operates normally by a predetermined method, for example, by using a control terminal 200.

The position estimation node 500 is a device for causing the hovering camera 100 to estimate the current position. As described above, the flight path of the hovering camera 100 is set as, for example, GPS position information. At this time, if there is nothing to block the radio waves from the GPS satellites, the hovering camera 100 can know the current position with extremely high accuracy. However, it is inevitable that the hovering camera 100 sneaks under the bridge girder 3, and the bridge girder 3 blocks the radio waves from the GPS satellites, and the bridge 1 causes multipath due to the reflection of the radio waves. If this happens, the hovering camera 100 may not be able to know the current position with high accuracy.

Therefore, in the present embodiment, the position estimation node 500 is provided under the bridge girder 3 in order for the hovering camera 100 to accurately acquire the current position. As the position estimation node 500, for example, an AR (Augmented Reality) marker may be used, or a GPS signal transmitter may be used.

When the AR marker is used as the position estimation node 500, in order for the hovering camera 100 to recognize the current position, for example, the position estimation node 500 is hung from both ends of the bridge 1 and the hovering camera 100 is made to image the position estimation node 500. .. Then, the hovering camera 100 that has imaged the position estimation node 500 is made to fly between the designated position estimation nodes 500. The hovering camera 100 determines the position between the position estimation nodes 500, for example, the integrated value of a sensor (for example, an IMU (Inertial Measurement Unit) sensor) provided in the hovering camera 100, and the position estimation node 500 of the destination calculated from the captured image. It becomes possible to grasp by the distance to. Therefore, the hovering camera 100 can accurately acquire the current position even under the bridge girder 3 by imaging the position estimation node 500.

When a GPS signal transmitter is used as the position estimation node 500, in order for the hovering camera 100 to recognize the current position, for example, the position estimation nodes 500 are installed diagonally or at the four corners of the bridge 1. By receiving the GPS signal emitted from the position estimation node 500, the hovering camera 100 can accurately acquire the current position even under the bridge girder 3.

The base station 600 is a device provided for takeoff and landing of the hovering camera 100. The base station 600 includes a GPS receiver and calculates the current position by receiving radio waves from GPS satellites. The current position calculated by the base station 600 is sent to the control terminal 200. By sending the current position calculated by the base station 600 to the control terminal 200, the control terminal 200 can display the position of the base station 600 on the general condition map of the bridge 1.

The base station 600 may have a function of checking the operation of the hovering camera 100. The operation confirmation of the hovering camera 100 executed by the base station 600 may include, for example, confirmation of a communication function, confirmation of an imaging function, confirmation of an airplane function, calibration of various sensors, and the like. Needless to say, the method for calibrating the sensor of the hovering camera 100 is not limited to the method using the base station 600, and the method for calibrating the sensor of the hovering camera 100 is, for example, a hovering camera on a jig dedicated to calibration. There may be a method of fixing the 100 and rotating the hovering camera 100 in the pitch direction or the roll direction to calibrate the sensor.

The charging station 700 charges the secondary battery provided in the hovering camera 100. The hovering camera 100 uses a battery as a power source, and consumes the electric power stored in the battery during flight and imaging. If the battery provided in the hovering camera 100 is a secondary battery, the charging station 700 can recover the power consumed by the hovering camera 100 by charging the battery. The charging station 700 may charge the hovering camera 100 by connecting a cable or the like to the hovering camera 100 and supplying power to the hovering camera 100. The charging station 700 may charge the hovering camera 100 by supplying power to the hovering camera 100 by a non-contact power transmission method. May be charged.

The server device 800 is a device that stores various data. In the present embodiment, the server device 800 may store the damage data generated by the information processing device 300.

The inspection system 10 according to the embodiment of the present disclosure has a configuration as shown in FIG. 2, so that the hovering camera 100 can image the bridge 1 and acquire an image of the bridge 1. By causing the hovering camera 100 to image the bridge 1, the inspection system 10 according to the embodiment of the present disclosure eliminates the need for scaffolding on the piers and bridge girders, and partially or completely in order to ensure the safety of workers. By reducing the frequency of closing the lanes or eliminating the need for closing the lanes, it is possible to inspect the bridge 1 at low cost and efficiently.

An example of a system configuration of the inspection system 10 according to the embodiment of the present disclosure has been described. Subsequently, a functional configuration example of the hovering camera 100 and the control terminal 200 constituting the inspection system 10 according to the embodiment of the present disclosure will be described.

[1.3. Function configuration example]
First, a functional configuration example of the hovering camera 100 according to the embodiment of the present disclosure will be described. FIG. 3 is an explanatory diagram showing a functional configuration example of the hovering camera 100 according to the embodiment of the present disclosure. Hereinafter, an example of the functional configuration of the hovering camera 100 according to the embodiment of the present disclosure will be described with reference to FIG.

As shown in FIG. 3, the hovering camera 100 according to the embodiment of the present disclosure includes an image pickup device 101, rotors 104a to 104d, motors 108a to 108d, a control unit 110, a communication unit 120, and a sensor unit. It includes 130, a position information acquisition unit 132, a storage unit 140, and a battery 150.

The control unit 110 controls the operation of the hovering camera 100. For example, the control unit 110 adjusts the rotation speeds of the rotors 104a to 104d by adjusting the rotation speeds of the motors 108a to 108d, performs an imaging process by the image pickup device 101, and communicates with another device (for example, the control terminal 200) via the communication unit 120. It is possible to control the transmission / reception processing of information between, and the storage / reading of information to the storage unit 140.

In the present embodiment, the control unit 110 controls the flight by adjusting the rotation speeds of the motors 108a to 108d based on the flight information transmitted from the control terminal 200, and the execution of the still image imaging process on the imaging device 101. .. The control unit 110 can provide the control terminal 200 with an image based on the request of the control terminal 200 by controlling the motors 108a to 108d and the image pickup device 101 based on the flight information transmitted from the control terminal 200. become.

The image pickup device 101 is composed of a lens, a CCD image sensor, an image pickup element such as a CMOS image sensor, a flash, and the like. The imaging device 101 provided in the hovering camera 100 executes imaging of a still image or a moving image under the control of the control terminal 200. The image captured by the image pickup device 101 is transmitted from the communication unit 120 to the control terminal 200. Further, in the present embodiment, the imaging device 101 executes the imaging process based on the information on the imaging position of the still image included in the flight information transmitted from the control terminal 200. The image obtained by the imaging process of the imaging device 101 may be stored in the storage unit 140 or transmitted from the communication unit 120 to the control terminal 200. When the hovering camera 100 images the bottom surface side of the bridge 1, it is possible that the sunlight is blocked by the bridge 1 and the brightness is insufficient. Therefore, the hovering camera 100 uses a flash when imaging the bottom surface side of the bridge 1. It is good to make it emit light.

The imaging device 101 can change the imaging direction to any direction by, for example, control from the control unit 110. For example, when the horizontal direction of the hovering camera is set to 0 degrees, it is possible to capture an imaging direction represented in a range of ± 90 degrees in the vertical direction. Since the imaging device 101 can change the imaging direction, the hovering camera 100 can capture an image in a predetermined direction and provide the captured image to the control terminal 200. Then, the control unit 110 may include the position information of the hovering camera 100 at the time when the image pickup device 101 captures a still image (position information by GPS positioning or positioning using the position estimation node 500. The position estimation node 500 may be included. The positioning used will be described later), the aircraft information at the time of imaging (for example, yaw angle, pitch angle, acceleration, angular velocity), and information on the imaging direction are associated with the metadata of the still image. As a method of storing the associated metadata, the metadata may be added to the additional information area of the still image data (for example, a specific area of the Exif format), or the metadata may be recorded in a separate file from the image file. It may be body data.

The rotors 104a to 104d fly the hovering camera 100 by generating lift by rotation. The rotation of the rotors 104a to 104d is performed by the rotation of the motors 108a to 108d. The motors 108a to 108d rotate the rotors 104a to 104d. The rotation of the motors 108a to 108d can be controlled by the control unit 110.

The communication unit 120 performs information transmission / reception processing by wireless communication with the control terminal 200. The hovering camera 100 transmits the image captured by the image pickup device 101 from the communication unit 120 to the control terminal 200. Further, the hovering camera 100 receives an instruction regarding flight from the control terminal 200 by the communication unit 120.

The sensor unit 130 is a group of devices for acquiring the state of the hovering camera 100, and may include, for example, an acceleration sensor, a gyro sensor, an ultrasonic sensor, a pressure sensor, an optical flow sensor, a laser range finder, and the like. The sensor unit 130 can convert the acquired state of the hovering camera 100 into a predetermined signal and provide it to the control unit 110 as needed. The position information acquisition unit 132 acquires information on the current position of the hovering camera 100 by using, for example, GPS or a vision sensor. The position information acquisition unit 132 may provide the acquired information on the current position of the hovering camera 100 to the control unit 110 as needed. The control unit 110 executes flight control of the hovering camera 100 based on the flight information received from the control terminal 200 by using the information on the current position of the hovering camera 100 acquired by the position information acquisition unit 132.

In addition, the sensor unit 130 detects obstacles that may hinder the flight during flight. When the sensor unit 130 detects an obstacle, the hovering camera 100 can provide the control terminal 200 with information about the detected obstacle.

The storage unit 140 stores various information. The information stored in the storage unit 140 may include, for example, flight information of the hovering camera 100 transmitted from the control terminal 200, an image captured by the image pickup device 101, and the like.

The battery 150 stores electric power for operating the hovering camera 100. The battery 150 may be a primary battery that can only be discharged or a secondary battery that can be charged. However, when the battery 150 is a secondary battery, the battery 150 may be, for example, shown in FIG. Power can be supplied from the charging station 700 shown.

The hovering camera 100 according to the embodiment of the present disclosure has a configuration as shown in FIG. 3, and automatically flies based on the flight path included in the flight information transmitted from the control terminal 200 to control the control terminal. The imaging process can be executed based on the information on the imaging position of the still image included in the flight information transmitted from the 200.

As described above, the functional configuration example of the hovering camera 100 according to the embodiment of the present disclosure has been described with reference to FIG. Subsequently, an example of the functional configuration of the control terminal 200 according to the embodiment of the present disclosure will be described.

FIG. 4 is an explanatory diagram showing a functional configuration example of the control terminal 200 according to the embodiment of the present disclosure. Hereinafter, an example of the functional configuration of the control terminal 200 according to the embodiment of the present disclosure will be described with reference to FIG.

As shown in FIG. 4, the control terminal 200 according to the embodiment of the present disclosure includes a display unit 210, a communication unit 220, a control unit 230, and a storage unit 240.

The display unit 210 includes, for example, a flat plate display device such as a liquid crystal display device or an organic EL display device. The display unit 210 may display, for example, an image captured by the image pickup apparatus 101 and information for controlling the operation of the hovering camera 100. The display unit 210 is provided with a touch panel, and the user can directly operate the information displayed on the display unit 210 by touching the display unit 210 with a finger or the like.

The communication unit 220 transmits / receives information to / from the hovering camera 100 by wireless communication. The control terminal 200 receives the image captured by the image pickup device 101 from the hovering camera 100 by the communication unit 220. Further, the control terminal 200 transmits an instruction regarding the flight of the hovering camera 100 from the communication unit 220 to the hovering camera 100. The flight command of the hovering camera 100 may be generated by the control unit 230.

The control unit 230 controls the operation of the control terminal 200. For example, the control unit 230 can control the display processing of characters, figures, images and other information on the display unit 210, and the transmission / reception processing of information with another device (for example, the hovering camera 100) via the communication unit 220. .. Further, the control unit 230 includes a flight information generation unit 232 and a display control unit 234.

The flight information generation unit 232 generates flight information to be transmitted to the hovering camera 100. When generating flight information, the flight information generation unit 232 uses, for example, information about a structure to be inspected, which is stored in a storage unit 240 described later. When the flight information generation unit 232 generates the flight information, the flight information generation unit 232 causes the communication unit 220 to transmit the generated flight information before the hovering camera 100 takes off.

The flight information generation process by the flight information generation unit 232 will be described in detail later, but an example of the flight information generation process by the flight information generation unit 232 will be briefly described. The flight information generation unit 232 reads a general condition map of the bridge 1 to be inspected when generating flight information of the hovering camera 100. The read general condition map of the bridge 1 is displayed on the display unit 210 by the display control unit 234. As described above, the points on the general condition map of the bridge 1 are associated with the points on the map data containing detailed GPS information in advance. It is desirable that this association is performed at a minimum of two sets of points. The flight path of the hovering camera 100 is defined as a GPS value (latitude and longitude set) by associating points on map data containing detailed GPS information in advance with the general condition map of the bridge 1. To.

Then, the flight information generation unit 232 generates the flight path of the hovering camera 100 based on the general condition map of the bridge 1. When the flight information generation unit 232 generates the flight path of the hovering camera 100, the flight information generation unit 232 provides information on the structure such as the construction method of the bridge 1, the width and the span length, the flight time of the hovering camera 100, the inspection method of the bridge 1, and the like. Use the information in. Concrete bridges are divided into reinforced concrete (RC) and PC (prestressed concrete) according to the reinforcement method, and depending on the shape of the girder, for example, RCT girder bridge, PCT girder bridge, PC hollow slab bridge, RC box girder bridge, PC box girder bridge. And so on. Therefore, if the construction method of the bridge 1 to be inspected is known, the flight information generation unit 232 can generate a flight path suitable for the construction method of the bridge 1. Then, the flight information generation unit 232 displays the flight path of the hovering camera 100 on the general condition map of the bridge 1.

The flight information generation unit 232 defines the flight path of the hovering camera 100 as a GPS value (a set of latitude and longitude) as described above. By defining the flight path of the hovering camera 100 as a GPS value by the flight information generation unit 232, the hovering camera 100 can determine at which position the imaging process should be executed during flight based on the GPS value.

The display control unit 234 controls the display of characters, figures, images, and other information on the display unit 210. Characters, figures, symbols, images and other information displayed on the display unit 210 in the drawings referred to in the description below are displayed by the display control unit 234. For example, when the flight information generation unit 232 generates the flight information to be transmitted to the hovering camera 100, the display control unit 234 transmits the general condition map of the structure to be inspected (bridge 1) and the generated flight information to the display unit 210. Perform display control.

The storage unit 240 stores various information. As the information stored in the storage unit 240, for example, there is information about the structure (bridge 1) to be inspected. Information about the structure to be inspected may include, for example, a general view of the structure to be inspected (bridge 1) and a method of constructing the structure to be inspected. Further, if the location of the structure to be inspected that is likely to be damaged is known in advance, the information on the structure to be inspected may include information on a portion that is likely to be damaged.

Note that the control terminal 200 may receive information about the structure to be inspected (bridge 1) from, for example, the information processing device 300 at the time of inspection of the structure, even if the information is not stored in the storage unit 240 in advance.

The control terminal 200 according to the embodiment of the present disclosure has a configuration as shown in FIG. 4, and generates flight information to be transmitted to the hovering camera 100 based on the information regarding the structure (bridge 1) to be inspected. Then, the hovering camera 100 that flies based on the flight information can acquire the image captured based on the flight information.

As described above, the functional configuration example of the control terminal 200 according to the embodiment of the present disclosure has been described with reference to FIG. Subsequently, an operation example of the inspection system 10 according to the embodiment of the present disclosure will be described.

[1.4. Operation example]
FIG. 5 is a flow chart showing an operation example of the inspection system 10 according to the embodiment of the present disclosure. FIG. 5 shows an operation example of the inspection system 10 according to the embodiment of the present disclosure when the hovering camera 100 is flown and the hovering camera 100 is made to image the bridge 1 to inspect the bridge 1. is there. When inspecting the bridge 1 using the hovering camera 100, it is assumed that the wireless relay node 400 and the position estimation node 500 are installed in advance at appropriate positions of the bridge 1. Hereinafter, an operation example of the inspection system 10 according to the embodiment of the present disclosure will be described with reference to FIG.

The control terminal 200 that generates flight information of the hovering camera 100 reads information about the bridge 1 including a general condition map of the bridge 1 to be inspected, and displays the general condition map of the bridge 1 on the display unit 210 (step S101). For example, the flight information generation unit 232 executes reading of information about the bridge 1, and the display control unit 234 executes the display of the general condition map of the bridge 1 on the display unit 210, for example. The control terminal 200, which displays the general condition map of the bridge 1 on the display unit 210, gives the user the area of the bridge 1 to be inspected by using the general condition map of the bridge 1 displayed on the display unit 210. It is specified (step S102). For example, the flight information generation unit 232 executes the process of causing the user to specify in step S102.

For example, when a part of the bridge 1 is to be inspected, the control terminal 200 causes the user to specify the area to be inspected in the general condition map of the bridge 1 displayed on the display unit 210. Further, for example, when the entire bridge 1 is to be inspected, the control terminal 200 causes the user to specify the area of all the parts of the bridge 1 in the general condition map of the bridge 1 displayed on the display unit 210.

FIG. 6 is an explanatory diagram showing an example of a screen displayed on the display unit 210 of the control terminal 200. FIG. 6 shows an example of a screen displayed on the display unit 210 when the user specifies the area of the bridge 1 to be inspected in step S102. In FIG. 6, it is assumed that the screen displayed on the display unit 210 is displayed when the bridge girder is designated as the area of the bridge 1 to be inspected. The control terminal 200 has, for example, a touch panel as an input unit (not shown), and allows the user to specify the area of the bridge 1 by, for example, tracing the screen or selecting the span to be inspected. Can be done. Of course, the method of having the user specify the area of the bridge 1 to be inspected is not limited to the above example. Further, the display of the area designated by the user is not limited to the example shown in FIG.

FIG. 6 also shows how the mark B1 indicating the position of the base station 600 is superimposed and displayed on the general condition map of the bridge 1. As described above, the base station 600 is provided with a GPS receiver, and the current position can be calculated by receiving radio waves from GPS satellites. Therefore, the control terminal 200 can superimpose the mark B1 indicating the position of the base station 600 on the general condition map of the bridge 1 and display it based on the information of the current position calculated by the base station 600.

After having the user specify the area of the bridge 1 to be inspected, the control terminal 200 subsequently generates flight information of the hovering camera 100 in the area to be inspected specified by the user based on the information about the bridge 1 (step S103). For example, the flight information generation unit 232 executes the flight information generation process in step S103.

When the control terminal 200 generates the flight information of the hovering camera 100 in step S103, the control terminal 200 provides information on the structure such as the construction method of the bridge 1, the width, and the span length, the flight time of the hovering camera 100, and the inspection of the bridge 1. Use information such as the method. For example, when the T girder is used as the construction method of the bridge 1, the control terminal 200 generates flight information as flight information such that the hovering camera 100 repeatedly ascends and descends on the bottom surface side of the bridge 1. Further, the control terminal 200 may use the information of the image pickup target surface of the bridge 1 when generating the flight information of the hovering camera 100 in the step S103. For example, when the user selects to image the side surface of the bridge 1, the control terminal 200 generates a flight path along the side surface of the bridge 1 as flight information, and the user selects to image the bottom surface of the bridge 1. If so, the control terminal 200 generates flight information as flight information that reciprocates on the bottom surface side of the bridge 1.

An example of flight information generated by the control terminal 200 will be described. For the flight information, for example, a list for each position where the imaging process is executed may be specified in the following format.
ID: (Relative coordinates of imaging point, imaging direction, speed at the time of imaging, travel time to the next imaging point, etc.)
The relative coordinates of the imaging points are specified by three points, the X-axis, the Y-axis, and the Z-axis. The X-axis is the latitude direction, the Y-axis is the longitude direction, and the Z-axis is the height direction. In addition, other information may include, for example, information for controlling special imaging. Information for controlling special imaging includes, for example, information for imaging at the same position in a plurality of imaging directions, bracket imaging (imaging at the same position and imaging direction with different exposures, shutter speeds, ISO sensitivities, etc.). Information on parameters for), information on the wavelength of infrared rays at the time of imaging may be included. According to this format, the flight information generated by the control terminal 200 can be composed of a list of values as follows.
0: (0,0,0,0,0,2,1.0)
1: (5,0,0,0,0,2,1.0)
2: (7,0,0,0,0,2,1.0)
3: (9,0,0,0,0,2,1.0)
The imaging point included in the flight information generated by the control terminal 200 is designated by the absolute coordinates of the base station 600, the absolute coordinates of an arbitrary position such as the first imaging position, and the relative coordinates from the reference point. May be done. The hovering camera 100 may convert the relative coordinates from the absolute coordinates of the reference point into absolute coordinates and refer to the converted coordinates during flight. Further, the imaging point included in the flight information generated by the control terminal 200 may be specified in absolute coordinates instead of relative coordinates. Further, a predetermined value may be stored in the information for controlling the special imaging included in the flight information generated by the control terminal 200. For example, the information for controlling special imaging includes values such as 1: imaging in multiple imaging directions, 2: bracket imaging (change in shutter speed), 3: bracket imaging (change in ISO sensitivity), and the like. Can be stored. The control terminal 200 may include information for controlling special imaging in the flight information, for example, for a place stored in the storage unit 240, which is considered to be easily damaged in the bridge girder 3.

The control terminal 200 may generate flight information such that the hovering camera 100 images the back surface of the bridge girder 3 of the bridge 1 at equal intervals, for example, during the flight information generation process of step S103. Therefore, the control terminal 200 may generate flight information so that the imaging positions of the still images are evenly spaced during the flight information generation process in step S103.

When the control terminal 200 generates the flight information of the hovering camera 100 in step S103, if the information of the portion that is considered to be highly damaged is stored in the storage unit 140 in advance, the information is stored. The flight information may be read out and the flight information may be generated so that the hovering camera 100 captures the portion in detail. When the hovering camera 100 images a portion that is considered to be damaged, the control terminal 200 may include information for controlling the above-mentioned special imaging in the flight information. Of course, the information of the part that is highly likely to be damaged does not have to be stored in the storage unit 140 in advance, and in that case, the user can obtain the information of the part that is highly likely to be damaged. You may enter it at the time of inspection.

When the area of the bridge 1 to be inspected is made to fly to the hovering camera 100, it is considered that the area cannot be made to fly to the hovering camera 100 at once depending on the flight time of the hovering camera 100. The flightable time of the hovering camera 100 is determined in advance from the capacity of the battery 150, the power consumption of the motors 108a to 108d for moving the rotors 104a to 104d, the power consumption of the image pickup device 101, the control unit 110, the communication unit 120, and the like. Is possible. Then, when generating flight information, the scheduled travel time from the start position (for example, base station 600) to the first imaging point, the scheduled travel time between the imaging points, and the scheduled travel time from the last imaging point to returning to the start position. From the above, it is also possible to estimate the time required for one inspection flight of the hovering camera 100. Therefore, if the hovering camera 100 cannot fly the flight path to the area of the bridge 1 to be inspected in one inspection flight, the control terminal 200 may divide the generated flight path into several parts.

Further, the control terminal 200 may generate a plurality of flight paths when generating flight information of the hovering camera 100 in step S103, and display the flight paths on the display unit 210. FIG. 7 is an explanatory diagram showing an example of a screen displayed on the display unit 210 of the control terminal 200. FIG. 7 shows an example of a state in which a plurality of flight paths are generated when the flight information of the hovering camera 100 is generated in S103, and the flight paths R1 and R2 are displayed on the display unit 210. Is. The control terminal 200 displays a plurality of flight paths on the display unit 210 and allows the user to select one flight path. The control terminal 200 generates flight information based on the selection of the flight path of the user.

When the flight information of the hovering camera 100 is generated in step S103, the control terminal 200 subsequently transmits the generated flight information to the hovering camera 100 and the takeoff instruction to the hovering camera 100 (step S104). For example, the flight information generation unit 232 transmits the generated flight information and the takeoff instruction through the communication unit 220.

FIG. 8 is an explanatory diagram showing an example of a screen displayed on the display unit 210 of the control terminal 200. FIG. 8 shows an example of a screen displayed on the display unit 210 of the control terminal 200 when a takeoff instruction is transmitted to the hovering camera 100. By touching the takeoff instruction button 211 displayed on the display unit 210, the user can cause the control terminal 200 to transmit the takeoff instruction to the hovering camera 100. When the control terminal 200 transmits the takeoff instruction to the hovering camera 100, the flight information generated in step S103 may be transmitted from the control terminal 200 to the hovering camera 100 before the takeoff instruction. After transmitting the takeoff instruction from the 200 to the hovering camera 100, the flight information generated in step S103 may be transmitted from the control terminal 200 to the hovering camera 100.

The hovering camera 100 that receives the flight information and the takeoff instruction from the control terminal 200 and takes off from the base station 600 flies based on the flight information sent from the control terminal 200 and executes an imaging process to obtain a still image. (Step S105). The hovering camera 100 acquires position information when an imaging process for obtaining a still image is executed and information on the aircraft during the imaging process, and associates the hovering camera 100 with the still image. Information on the airframe during the imaging process may include, for example, yaw angle, pitch angle, acceleration, and angular velocity information. Further, the hovering camera 100 may stream the moving image captured by the image pickup device 101 during flight to the control terminal 200. The control terminal 200 acquires and displays a moving image captured by the image pickup device during flight of the hovering camera 100 by streaming, so that the control terminal 200 is flying at what position the hovering camera 100 is flying. Can be presented to the user.

When the hovering camera 100 executes the imaging process, it is desirable that the distance between the hovering camera 100 and the imaging target surface (for example, the side surface or the bottom surface of the bridge girder 3) is kept constant at all imaging points. By keeping the distance to the imaging target surface constant at all imaging points, the hovering camera 100 can obtain still images captured with the same size.

If a part that is considered to be damaged is included in the flight path of the hovering camera 100, the hovering camera 100 may change the imaging direction of the image pickup apparatus or infrared rays having different wavelengths for that part. A plurality of still images may be captured by hitting the camera or changing the shutter speed. Further, when a part that is considered to be damaged is included in the flight path of the hovering camera 100, the hovering camera 100 narrows the interval between the positions where the imaging process is performed for that part to be narrower than the other parts. You may.

FIG. 9 is an explanatory diagram conceptually showing the operation of the hovering camera 100 in the inspection system 10 according to the embodiment of the present disclosure. When the hovering camera 100 flies on the bottom surface side of the bridge 1 based on the flight information, the hovering camera 100 stops at, for example, at time t1 to image the bottom surface of the bridge 1, and after imaging, reaches a position where the image is taken at time t2. It flies, stops at time t2, images the bottom surface of the bridge 1 at another position, and then repeats flying, stopping, and imaging until the time tn to image the bottom surface side of the bridge 1. An image of the bottom surface of the bridge 1 can be obtained by repeating flight, stop, and imaging of the hovering camera 100.

When flying based on flight information, the hovering camera 100 can accurately grasp the current position if it can receive radio waves from GPS satellites without any obstacles. However, at a position where it is difficult to receive radio waves from GPS satellites, such as under a bridge 1, it becomes difficult for the hovering camera 100 to accurately grasp the current position. Therefore, in the present embodiment, by using the position estimation node 500, the hovering camera 100 is made to grasp the accurate current position even at a position where it is difficult to receive radio waves from GPS satellites.

FIG. 10 is an explanatory diagram conceptually showing the operation of the hovering camera 100 in the inspection system 10 according to the embodiment of the present disclosure. For example, when a route is set such that the hovering camera 100 flies from Start to Goal in FIG. 10, the hovering camera 100 has a GPS positioning area 40 that receives and positions radio waves from GPS satellites 30 without any obstacles, for example. It goes back and forth between the sensor positioning area 50, which estimates the current position using the vision sensor.

The hovering camera 100 grasps the current position in the GPS positioning area 40 by using radio waves from the GPS satellites 30. The hovering camera 100 determines the position between the position estimation nodes 500 in the sensor positioning area 50, and if the position estimation node 500 is an AR marker, the integrated value of the sensor (for example, IMU sensor) provided in the hovering camera 100 and the imaging device 101. The current position is grasped by the distance to the position estimation node 500 of the movement destination calculated from the image captured in. If the position estimation node 500 is a GPS signal transmitter, the hovering camera 100 grasps each of them by using the signal transmitted from the position estimation node 500.

By using the position estimation node 500 in this way, the hovering camera 100 can grasp the accurate current position even if it moves to a position where it is difficult to receive radio waves from GPS satellites.

When the hovering camera 100 completes the imaging process at the final imaging point, it automatically flies to the base station 600 and returns to the base station 600 (step S106). Then, the control terminal 200 acquires an image captured by the hovering camera 100 that has returned to the base station 600 from the hovering camera 100 (step S107). The image captured by the hovering camera 100 may be acquired after the hovering camera 100 returns to the base station 600 in this way, but the control terminal 200 is stationary after the hovering camera 100 executes the imaging process. Each time an image is obtained, the still image may be acquired sequentially.

In the inspection system 10 according to the embodiment of the present disclosure, the hovering camera 100 and the control terminal 200 execute the operations as shown in FIG. 5, and the flight information transmitted to the hovering camera 100 is transmitted to the structure to be inspected ( The control terminal 200 generates an image based on the information about the bridge 1), the hovering camera 100 that flies based on the flight information captures an image based on the flight information, and the control terminal 200 acquires the image captured by the hovering camera 100. Can be done.

It is assumed that the user finds a portion to be captured in detail by looking at the moving image captured by the hovering camera 100 while the hovering camera 100 is flying. In that case, for example, the user may operate the control terminal 200 to send an instruction to the hovering camera 100 to stop the automatic flight and switch to the manual operation from the control terminal 200.

In the above example, the flight information is generated by the control terminal 200, the hovering camera 100 automatically flies based on the generated flight information, and the imaging process is executed. However, it is possible that there are obstacles on the flight path that cannot be seen in the overview map of the bridge 1.

FIG. 11 is an explanatory diagram conceptually showing the operation of the hovering camera 100 in the inspection system 10 according to the embodiment of the present disclosure. FIG. 11 shows a state in which a tree 4 grows under the bridge girder 3. This tree 4 is an obstacle that does not appear in the general condition map of the bridge 1, and its existence may be revealed only when the hovering camera 100 flies.

Therefore, in the present embodiment, the hovering camera 100 may be made to perform a test flight once based on the flight information generated by the control terminal 200, and it may be confirmed whether or not there is an obstacle on the flight path included in the flight information. ..

When the hovering camera 100 is once test-flighted based on the flight information generated by the control terminal 200, the moving image captured by the hovering camera 100 is streamed and received by the control terminal 200, and the flight path included in the flight information. The user may check whether there is an obstacle on the top while looking at the moving image, or the sensor unit 130 of the hovering camera 100 may detect the obstacle. For the detailed position of the obstacle, a stereo camera is provided as the image pickup device 101 of the hovering camera 100, the distance to the obstacle can be grasped by imaging with the stereo camera, and the direction of the obstacle can be specified by the direction of the hovering camera 100. It is possible to grasp by doing. If an obstacle exists on the flight path during the test flight of the hovering camera 100, the hovering camera 100 stops the automatic flight, shifts to the hovering state, and waits for the operation from the user. Alternatively, it may be automatically returned to the base station 600.

If it is found that an obstacle exists on the flight path included in the flight information, the control terminal 200 may register the location of the obstacle in the general condition map of the bridge 1. The location of the obstacle may be manually input by the user, and when the hovering camera 100 detects an obstacle by the sensor unit 130, the location of the detected obstacle is acquired from the hovering camera 100 and the location of the obstacle is obtained. May be registered in the overview map of the bridge 1.

FIG. 12 is an explanatory diagram showing an example of a screen displayed on the display unit 210 of the control terminal 200. FIG. 12 shows an example of a screen displayed on the display unit 210 when an obstacle is found on the flight path by a test flight of the hovering camera 100. If the test flight of the hovering camera 100 reveals that an obstacle exists on the flight path, the control terminal 200 superimposes the mark O1 indicating the location of the obstacle on the general condition map of the bridge 1.

If the location of the obstacle is known, the control terminal 200 regenerates the flight information so as to avoid the location of the obstacle, and transmits the regenerated flight information to the hovering camera 100. By flying based on the flight information regenerated by the control terminal 200, the hovering camera 100 can perform flight and imaging processing while avoiding obstacles (trees 4).

The method of flying the hovering camera 100 to grasp the location of an obstacle is not limited to the above example. For example, the outer circumference of the flight path generated by the control terminal 200 is made to fly on a simple path while the hovering camera 100 captures a moving image with the image pickup device 101, and it is confirmed whether or not there is an obstacle under the bridge girder 3. You may do so.

[1.5. Damage data generation example]
By flying the hovering camera 100 and causing the hovering camera 100 to image the bridge 1, it is possible to grasp the state of a place where a worker cannot easily approach, for example, the bottom surface of the bridge girder 3. The still image captured by the hovering camera 100 may include, for example, the position information of the hovering camera 100 that captured the still image (position information by GPS positioning or positioning using the position estimation node 500), and at the time of imaging. Aircraft information (for example, yaw angle, pitch angle, acceleration, angular velocity), and information on the imaging direction are linked. Further, when the hovering camera 100 takes an image while keeping the distance between the image pickup target surface and the image pickup target surface constant at all the imaging points, the relative position of the damaged place in the image can be known. Therefore, if the still image captured by the hovering camera 100 includes a damaged portion of the bridge girder 3, it is possible to grasp the absolute location of the damaged portion. For example, by calculating the relative value of the damaged part with the center of the still image as the origin and calculating the relative value in the position information of the hovering camera 100 when the image is captured, the position information of the damaged part can be obtained. Desired. In addition, for example, the following data can be recorded as the position information of the damaged portion.
(1) Information on the imaging position of the still image is recorded as the position of the damaged portion (relative value (offset) is not recorded).
(2) The information on the imaging position of the still image and the relative value (offset) corresponding to the damaged part are recorded as the position of the damaged part.
(3) The absolute value (for example, as will be described later, the imaging position of the still image at the four corners and the coordinates of the position estimation node 500) and the relative value (offset) of the damaged part, which are considered to have high accuracy of the position information, are used as the reference. Record as position.
(4) Record the calculated absolute value (for example, latitude, longitude, altitude) as the position of the damaged part.

There is known a technique for obtaining the physical size of an imaging range by using numerical values such as the focal length of a lens, the size of an image sensor, and the distance to an imaging target. Therefore, when grasping the damaged part, the physical size of the imaging range of the hovering camera 100 is used by using the distance information from the hovering camera 100 to the imaging target (for example, the back surface or the side surface of the bridge girder 3) and the angle of view information of the imaging device 101. May be estimated. With the center position of the captured image (the position where the hovering camera 100 imaged) as the origin, the physical relative position from the origin to the damaged part is estimated, and the position coordinates of the origin of the captured image are added to this relative position. By being combined, the physical position information of the damaged part is determined. It should be noted that these distance information and angle of view information may be acquired through the sensor of the hovering camera 100 at the time of imaging and recorded in association with the image, or the hovering camera 100 may be used in advance. Alternatively, the value set in the image pickup apparatus 101 may be used. In addition to the imaging position information, the distance information, and the angle of view information, the position information of the damaged portion may be calculated by using the aircraft information (for example, yaw angle, pitch angle, acceleration, angular velocity) at the time of imaging and the information of the imaging direction. ..

The detection of the damaged portion from the still image captured by the hovering camera 100 may be visually performed by the user, or may be automatically performed by, for example, the information processing apparatus 300 by image processing. When automating the detection of a damaged portion, an image processing technique such as a pattern matching process can be used.

The data structure of the damage data is defined, for example, in the following format.
(Image ID, damage ID, position information of damaged part, coordinates of damaged part on image, damage type ID, degree of damage)
The damage type ID refers to an ID given to each type of damage such as cracks, peeling, water leakage, and free lime. Further, in the field of the degree of damage, the maximum width of the damage, the length of the damaged portion in the image, and the like can be recorded. From the still image captured by the hovering camera 100, the inspection system 10 according to the present embodiment can generate damage data according to the above-mentioned format by manual input by the user or automatic processing by the information processing device 300. Then, the damage data generated by the inspection system 10 according to the present embodiment can be used for order processing to the contractor for repairing the damage caused in the bridge 1.

However, the hovering camera 100 captures a large number of still images in one inspection flight. Therefore, it is a heavy burden on the user to check the still images captured by the hovering camera 100 during the inspection flight one by one.

Therefore, the still images captured by the hovering camera 100 are joined (stitched) to obtain one image. By stitching the still image captured by the hovering camera 100, for example, the state of the bottom surface of the bridge girder 3 for one span can be obtained as one image. Then, by checking the image of the bottom surface of the bridge girder 3 for one span obtained by stitching the still image captured by the hovering camera 100, the user confirms whether or not the bottom surface of the bridge girder 3 is damaged. Can be done. The stitching process of the still image may be performed by the control terminal 200 or the information processing device 300.

FIG. 13 is an explanatory diagram showing an outline of a case where the bottom surface of the bridge girder 3 is inspected based on a still image captured by the hovering camera 100. A part of the bottom surface of the bridge girder 3 (for example, one span length of the bridge girder 3) is imaged by the hovering camera 100, and the still image captured by the hovering camera 100 is stitched to capture the bottom surface of the bridge girder 3. Image 20 is obtained. Reference numeral 21 indicates an image captured by the hovering camera 100 in a single imaging process.

When determining the absolute location of the damaged part from the image obtained by stitching the still image captured by the hovering camera 100, the position of the stitched image in which the accuracy of the position information at the time of imaging is relatively high is relatively high. Information can be selected as a reference point. You may use the position information of the hovering camera 100 at the time of capturing the still images of the four corners which are the basis of the stitched image as the reference point. The still images at the four corners, which are the basis of the stitched image, have the least distortion, and the GPS positioning area has less error in position information, which is the GPS positioning area, and when imaging the four corners near the GPS positioning area. Since it is considered preferable to use the position information of the above as a reference point, the position of the damaged part can be accurately determined by obtaining the absolute location of the damaged part from the position information of the hovering camera 100 corresponding to the still images at the four corners. It becomes possible to ask well. As the accuracy of each position information, for example, positioning status information in GPS positioning data (information indicating a state during 2D positioning, 3D positioning, or inability to position, data such as the number of received satellites) may be used.

FIG. 14 is an explanatory diagram showing an example of an image 20 obtained by stitching a still image captured by the hovering camera 100. The centers G1 to G4 of the four corners of the still images C1 to C4, which are the basis of the image 20, correspond to the positions of the hovering camera 100 when the respective still images are captured. In the present embodiment, the absolute position of each damaged portion in the image 20 is calculated by using the position information of the hovering camera 100 corresponding to the still images C1 to C4 at the four corners.

When generating damage data from stitched images, the data structure of the damage data is defined, for example, in the following format. That is, the image ID is deleted from the damage data described above.
(Damage ID, location information of damaged part, coordinates of damaged part on image, damage type ID, degree of damage)
The image ID of the stitched image may be generated, and the image ID of the stitched image may be included in the damage data. The inspection system 10 according to the present embodiment can generate damage data according to the above-mentioned format from the stitched image by manual input by the user or automatic processing by the information processing apparatus 300.

[1.5.1. Function configuration example]
FIG. 15 is an explanatory diagram showing a functional configuration example of the information processing apparatus 300 according to the embodiment of the present disclosure. FIG. 15 shows the information processing apparatus 300 according to the embodiment of the present disclosure, which has a function of obtaining the absolute damage position of the bridge girder 3 from the still image captured by the hovering camera 100 and generating damage data. This is a functional configuration example. Hereinafter, an example of the functional configuration of the information processing apparatus 300 according to the embodiment of the present disclosure will be described with reference to FIG.

As shown in FIG. 15, the information processing device 300 according to the embodiment of the present disclosure includes a display unit 310, a communication unit 320, a control unit 330, and a storage unit 340.

The display unit 310 includes, for example, a flat plate display device such as a liquid crystal display device or an organic EL display device. The display unit 310 can display, for example, an image captured by the image pickup device 101 of the hovering camera 100, information on damage to the bridge 1 obtained from the image captured by the image pickup device 101, and the like.

The communication unit 320 transmits / receives information by wireless communication to, for example, the control terminal 200. The information processing device 300 receives the image captured by the hovering camera 100 from the control terminal 200 by the communication unit 320 together with the information on the absolute imaging position of the image.

The control unit 330 controls the operation of the information processing device 300. For example, the control unit 330 can control the display processing of characters, figures, images and other information on the display unit 310, and the transmission / reception processing of information with another device (for example, the control terminal 200) via the communication unit 320. .. Further, the control unit 330 includes an imaging position information acquisition unit 332, a damage position calculation unit 334, an image composition unit 336, and a damage data generation unit 338.

The imaging position information acquisition unit 332 acquires information on the imaging position at the time of imaging, which was acquired by the hovering camera 100 when the hovering camera 100 imaged the bridge 1. The damage position calculation unit 334 detects the damaged part of the bridge 1 from the image captured by the hovering camera 100 by using an image processing technique such as pattern matching processing, and obtains the absolute position of the damaged part as the imaged position information. It is calculated using the information of the imaging position acquired by the acquisition unit 332.

The image compositing unit 336 executes stitching image processing in which still images captured by the hovering camera 100 are stitched together to generate one image. When stitching the still images captured by the hovering camera 100, the image synthesizing unit 336 may use the information of the imaging position at the time of capturing each still image.

When calculating the damage position, the damage position calculation unit 334 may use the information of the image pickup position of the corner image (for example, each of the four corners) in the image image to be the basis of the image stitched by the image synthesis unit 336. Good. As described above, since the still images at the four corners in the captured image that is the basis of the stitched image are considered to have the least distortion, the damage position calculation unit 334 is the corner in the captured image that is the basis of the stitched image. By using the information on the imaging position of the captured image of, a more accurate damage position can be obtained.

The damage data generation unit 338 generates the damage data as described above by using the absolute position of the damaged part of the bridge 1 calculated by the damage position calculation unit 334. The damage data generation unit 338 may generate damage data in units of one still image, or the image composition unit 336 may generate one damage data for the stitched image.

The storage unit 340 stores various information. The information stored in the storage unit 340 includes, for example, a still image captured by the imaging device 101 of the hovering camera 100, information on the absolute imaging position of the hovering camera 100 when the still image is captured, and a damage data generation unit. Information on damage data generated by 338 may be included.

The information processing device 300 according to the embodiment of the present disclosure has a configuration as shown in FIG. 15, so that damage data can be generated from a still image captured by the hovering camera 100. Therefore, the present disclosure The information processing device 300 according to the embodiment can efficiently generate the inspection result of the bridge 1 which is the structure to be inspected. As described above, the damage data may be generated by the control terminal 200 instead of the information processing device 300. Therefore, in the operation shown in FIG. 15, the control terminal 200 may have the configuration of the control unit 330 of the information processing device 300. In addition, the inspection results of the bridge 1, which is the structure to be inspected, can be accumulated and utilized in a public and private database. As described above, the damage data may be generated by the control terminal 200 instead of the information processing device 300. Therefore, in the operation shown in FIG. 15, the control terminal 200 may have the configuration of the control unit 330 of the information processing device 300.

As described above, the functional configuration example of the information processing apparatus 300 according to the embodiment of the present disclosure has been described with reference to FIG. Subsequently, an operation example of the information processing apparatus 300 according to the embodiment of the present disclosure will be described.

[1.5.2. Operation example]
FIG. 16 is a flow chart showing an operation example of the information processing apparatus 300 according to the embodiment of the present disclosure. FIG. 16 shows the operation of the information processing device 300 according to the embodiment of the present disclosure when the damage data is generated by obtaining the absolute damage position of the bridge girder 3 from the still image captured by the hovering camera 100. This is an example. Hereinafter, an operation example of the information processing apparatus 300 according to the embodiment of the present disclosure will be described with reference to FIG.

The information processing device 300 first acquires an image associated with the information of the imaging position, which is captured by the hovering camera 100 while flying around the periphery of the bridge 1 (step S301). When an image associated with the imaging position information is acquired in step S301, the information processing apparatus 300 subsequently detects the damaged portion from the image by using an image processing technique such as pattern matching processing (step S302). For example, the damage position calculation unit 334 may execute the detection process of the damaged portion in step S302.

When the damaged portion is detected from the image in step S302, the information processing apparatus 300 subsequently calculates the absolute position of the damaged portion (step S303). The calculation process of step S303 can be executed by, for example, the damage position calculation unit 334. The information processing device 300 calculates the absolute position of the damaged portion in step S303 based on the information on the imaging position of the still image captured by the hovering camera 100. When calculating the absolute position of the damaged portion, the information processing device 300 is hovering based on the distance information from the hovering camera 100 to the image pickup target (for example, the back surface or the side surface of the bridge girder 3) and the angle of view information of the image pickup device 101. The physical size of the imaging range of the camera 100 may be estimated. By estimating the physical relative position from the center of the captured image to the damaged portion and adding the position coordinates as the origin of the captured image to this relative position, the information processing apparatus 300 can perform the physical position of the damaged portion. Information can be decided. Then, the information processing device 300 generates damage data including the absolute position of the damaged portion (step S304). The damage data generation process in step S304 can be executed, for example, by the damage data generation unit 338.

The information processing apparatus 300 according to the embodiment of the present disclosure can generate damage data from a still image captured by the hovering camera 100 by executing the operation as shown in FIG. The information processing device 300 according to the embodiment can efficiently generate the inspection result of the bridge 1 which is the structure to be inspected. As described above, the damage data may be generated by the control terminal 200 instead of the information processing device 300. Therefore, the operation shown in FIG. 16 may be executed by the control terminal 200.

FIG. 17 is a flow chart showing an operation example of the control terminal 200 according to the embodiment of the present disclosure. FIG. 17 shows an example of flight information generation processing by the control terminal 200 using the damage data generated by the information processing device 300. Hereinafter, an operation example of the control terminal 200 according to the embodiment of the present disclosure will be described with reference to FIG.

When the control terminal 200 displays a still image captured by the hovering camera 100 and the location of damage is specified by the user on the still image (step S311), the control terminal 200 determines the damaged position at the designated location. , Acquired from the damage data generated by the information processing apparatus 300 (step S312). The method of designating the damaged location is not limited, but for example, the user may be made to specify the damaged location by displaying a still image and touching the touch panel of the control terminal 200 with a finger or the like.

When the damage position of the location specified by the user is acquired from the damage data, the control terminal 200 subsequently flies the hovering camera 100 to the damage position acquired from the damage data to generate flight information for imaging the damage position ( Step S313). The process of step S313 is executed by, for example, the flight information generation unit 232. Since the flight information generated by the control terminal 200 in step S313 is for confirming the damage position in detail, the interval between the imaging positions is compared with the flight information generated by the control terminal 200 described in the above description of FIG. The hovering camera 100 may be instructed to narrow the distance and perform a special imaging at each imaging position.

When flight information is generated in step S313, the control terminal 200 transmits the generated flight information to the hovering camera 100, and the hovering camera 100 flies and operates based on the flight information, as shown in steps S104 and S105 of FIG. Execute the imaging process. Then, as shown in steps S106 and S107 of FIG. 5, when the hovering camera 100 completes the imaging process at the final imaging point, it automatically flies to the base station 600, returns to the base station 600, and controls. The terminal 200 acquires an image captured by the hovering camera 100 from the hovering camera 100.

The control terminal 200 according to the embodiment of the present disclosure uses the damage data generated by the information processing apparatus 300 by executing the operation as shown in FIG. 17 to transfer the damaged portion of the bridge girder 3 to the hovering camera 100. Flight information for detailed imaging can be generated.

[1.6. Example of flight instructions using composite images]
When the hovering camera 100 captures a moving image or a still image at a predetermined interval and transmits the moving image to the control terminal 200, the control terminal 200 captures the moving image captured by the hovering camera 100 at predetermined frames or at predetermined intervals. It is possible to perform a process of synthesizing captured still images in real time (real-time stitching process). The control terminal 200 can allow the user to specify the target position of the hovering camera 100 by using the composite image. Then, the control terminal 200 transmits flight information for flying to the target position specified by the user to the hovering camera 100.

The control terminal 200 synthesizes the images in this way, displays the composite image on the display unit 210, and generates flight information for flying to the target position designated for the composite image, thereby more intuitively for the user. The flight path of the hovering camera 100 can be generated. Further, when it is difficult to obtain an aerial photograph by generating a composite image from a moving image or a still image captured by the hovering camera 100 and displaying the composite image on the display unit 210, the control terminal 200, for example, on the back side of a bridge. Even for such places, it is possible to specify the flight route using detailed image information and the current situation that cannot be obtained from maps and aerial photographs, and it is necessary to access a database such as external map information. do not do. Further, the control terminal 200 can also specify the flight path of the hovering camera 100 indoors by generating a composite image from the moving image or the still image captured by the hovering camera 100.

FIG. 18 is an explanatory diagram showing a state in which the hovering camera 100 is made to image the direction of the ground while the hovering camera 100 is flying. FIG. 19 is an explanatory diagram showing how the images captured by the hovering camera 100 are combined and the user is made to specify the flight path of the hovering camera 100 using the combined images.

FIG. 19 shows a current position mark 252 indicating the current position of the hovering camera 100, which is superimposed on the composite image 251 obtained as a result of imaging from the sky. When the control terminal 200 generates the composite image 251, for example, feature point matching or the like is used as described later.

When the user specifies the target position 253 of the hovering camera 100 on the composite image 251 as shown in FIG. 19, for example, the control terminal 200 flies the hovering camera 100 from the current position mark 252 to the target position 253 on the flight path 254. The flight information for the purpose is generated and transmitted to the hovering camera 100. The hovering camera 100 flies based on the flight information transmitted from the control terminal 200.

FIG. 20 is an explanatory view showing a state in which the hovering camera 100 is imaged in the upward direction (the back surface of the bridge) while the hovering camera 100 is flying. FIG. 21 is an explanatory diagram showing how the images captured by the hovering camera 100 are combined and the user is made to specify the flight path of the hovering camera 100 using the combined images.

Similar to FIG. 19, FIG. 21 shows a current position mark 262 indicating the current position of the hovering camera 100, which is superimposed on the composite image 261 obtained as a result of imaging the back surface of the bridge. When the control terminal 200 generates the composite image 261, for example, feature point matching or the like is used as described later.

When the user specifies the target position 263 of the hovering camera 100 on the composite image 261 as shown in FIG. 21, for example, the control terminal 200 makes the hovering camera 100 fly from the current position mark 262 to the target position 263 on the flight path 264. The flight information for the purpose is generated and transmitted to the hovering camera 100. The hovering camera 100 flies based on the flight information transmitted from the control terminal 200.

First, an example of generating a composite image by the control terminal 200 is shown. FIG. 22 is an explanatory diagram showing a state in which the control terminal 200 generates a composite image and displays it on the display unit 210. The process described below may be executed by, for example, the display control unit 234 of the control terminal 200. Therefore, the display control unit 234 can function as an example of the image processing unit of the present disclosure.

In FIG. 22, for example, using the image group 271 captured at time t-5 to t-1 and the image 272 captured at time t, the control terminal 200 generates and displays a composite image. It is shown that it is. When the image 272 captured at time t was transmitted to the control terminal 200, the control terminal 200 was captured at time t-1 from the feature points of the image 272 captured at time t and the image group 271. Collate with the feature points of the image. The control terminal 200 may collate the feature points of the image 272 captured at time t with the feature points of the composite image already synthesized using the image group 271.

Subsequently, the control terminal 200 obtains a rotation / translation parameter (or an affine transformation parameter) that minimizes the position error of the feature point. Subsequently, the control terminal 200 synthesizes (α-blending) the composite image generated by the stitching process so far with reference to the new image 272 by conversion using rotation / translation parameters (or affine transformation parameters). By this composition, the control terminal 200 can generate a new composite image in which the image captured at time t comes to the center of the composite image. And. The control terminal 200 displays the composite image so that the center of the composite image, that is, the center 273 of the image captured at time t is the position of the hovering camera 100.

Subsequently, the flight path generation process of the control terminal 200 based on the user input to the composite image generated in this way will be described.

FIG. 23 is an explanatory diagram illustrating a flight path generation process of the control terminal 200 based on the user's input to the composite image. When the user specifies the target position 276 with respect to the composite image 275 generated by the control terminal 200, the direction from the current position of the hovering camera 100 to the target position can be known. Since the composite image is generated based on the image captured by the hovering camera 100, the direction with respect to the target position represents the direction from the current hovering camera 100.

The target position 276 may be designated by the user as a location that has not yet been imaged by the hovering camera 100.

For example, it is assumed that the hovering camera 100 is provided with an image pickup device downward so that the upper part of the captured image is the front of the hovering camera 100 and the left direction is the left direction of the hovering camera 100. Therefore, when the user specifies the target position 276 with respect to the composite image 275 shown in FIG. 23, the target position 276 is in the left rear direction when viewed from the current position (that is, the center 273) of the hovering camera 100.

Therefore, the control terminal 200 determines a flight path 277 such that the hovering camera 100 flies in the direction to the left rearward when viewed from the current position of the hovering camera 100, and generates flight information such that the hovering camera 100 flies horizontally on the flight path 277. Then, it is transmitted to the hovering camera 100. For example, the flight information generation unit 232 may execute the determination of the flight path 277 and the generation of flight information. The hovering camera 100 makes a horizontal flight to the left rear based on the flight information transmitted from the control terminal 200.

Then, with the start of horizontal flight of the hovering camera 100, the control terminal 200 can obtain a new image captured by the hovering camera 100. The control terminal 200 updates the composite image using a new captured image obtained by moving the hovering camera 100. When updating the composite image, the control terminal 200 not only updates the orientation and position of the image, but also updates the target position. By updating the orientation, position, and target position of the image, the target position moves together with the position specified by the user. FIG. 24 is an explanatory diagram showing an example of an updated composite image using a new captured image obtained by moving the hovering camera 100 to the target position 276. In FIG. 24, as compared with FIG. 23, the position of the center 273 is always the same, and the target position 276 is closer to the center 273. That is, in FIG. 24, it can be seen from what is shown in FIG. 23 that the target position 276 is moving.

The control terminal 200 generates flight information by applying feedback control so that the target position 276 approaches the center 273 to a predetermined distance or less by newly capturing the image and updating the composite image by the hovering camera 100, and generates the generated flight information. It is transmitted to the hovering camera 100.

In the above-mentioned example, a method of moving the hovering camera 100 to a target position without rotation is shown, but the present disclosure is not limited to such an example. When the target position is specified, the control terminal 200 first generates flight information for rotating the hovering camera 100 toward the target position, and then generates flight information for rotating the hovering camera 100 and then moving the hovering camera 100 toward the target position. You may. Even when the hovering camera 100 is rotated, the control terminal 200 performs feature point matching as described above, so that the composite image can be rotated as the hovering camera 100 is rotated.

The moving direction in the image differs depending on how the image pickup device is attached to the hovering camera 100. Therefore, for example, if the image pickup device 101 is attached upward to the hovering camera 100, the moving direction is reversed from that when the image pickup device 101 is attached downward.

Of course, it is also possible to install the image pickup device 101 laterally on the hovering camera 100. When the image pickup device 101 is installed in the hovering camera 100 in the horizontal direction, the moving direction of the hovering camera 100 is a combination of up and down and rotation, or up and down and left and right. FIG. 25 is an explanatory diagram showing an example of a composite image when the image pickup device 101 is installed in the hovering camera 100 in the lateral direction. FIG. 25 shows a composite image 281 generated from an image captured by the hovering camera 100. Further, what is surrounded by reference numeral 282 is an image captured by the hovering camera 100 from the current position.

Based on the above description, an operation example of the control terminal 200 according to the embodiment of the present disclosure will be described.

FIG. 26 is a flow chart showing an operation example of the control terminal 200 according to the embodiment of the present disclosure. FIG. 26 shows an operation example of the control terminal 200 when a composite image is generated from the image captured by the hovering camera 100, flight information is generated based on the composite image, and the flight information is sent to the hovering camera 100. is there.

The control terminal 200 first performs the initialization process in the display control unit 234 (step S401). This initialization process is a process in which the captured image first sent from the hovering camera 100 is used as an input image, and the input image is used as the first composite image.

The control terminal 200 subsequently uses the image captured by the hovering camera 100 as an input image, and the display control unit 234 extracts the feature points of the input image and the composite image (step S402). After extracting the feature points, the control terminal 200 subsequently matches the extracted feature points, and the display control unit 234 calculates the movement amount and the rotation amount between the feature points (step S403).

Subsequently, the control terminal 200 converts the composite image in the display control unit 234 according to the amount of movement and the amount of rotation between the feature points obtained in step S403 (step S404). At the time of this conversion, if the target position has already been specified, the control terminal 200 also converts the target position by the display control unit 234.

Subsequently, the control terminal 200 synthesizes the input image with the composite image so far to obtain a new composite image, and the display control unit 234 performs display processing of the composite image (step S405).

The control terminal 200 subsequently determines in the display control unit 234 whether or not there is a user input to the display unit 210 on which the composite image is displayed (step S406). The user input to the display unit 210 on which the composite image is displayed is, for example, a user input to the touch panel provided on the display unit 210. Then, when there is a user input to the touch panel provided on the display unit 210, the control terminal 200 detects the coordinate position on the image input by the user, for example, by the display control unit 234. When there is a user input for the composite image (step S406, Yes), the control terminal 200 performs a process of registering the user's touch position as a target position in the display control unit 234 (step S407). If there is no user input for the composite image (step S406, No), the control terminal 200 skips the process of step S407.

Subsequently, the control terminal 200 determines whether or not the target position is registered by the display control unit 234 (step S408). If the target position is registered (step S408, Yes), the control terminal 200 subsequently obtains the direction of the target position from the current position by the display control unit 234 (step S409).

When the direction of the target position from the current position is obtained, the control terminal 200 subsequently converts the direction on the image into the moving direction of the hovering camera 100 in accordance with the attachment of the imaging device 101 (step S410). .. Then, the control terminal 200 generates a command for moving in the moving direction after the conversion as flight information by the flight information generation unit 232 and sends it to the hovering camera 100 (step S411).

When the control terminal 200 sends flight information to the hovering camera 100, the control terminal 200 returns to the extraction process of the feature points in step S402.

On the other hand, if the target position is not registered as a result of the determination in step S408 (step S408, No), the control terminal 200 skips the processes in steps S409 to S411 and extracts the feature points in step S402. Return to processing.

By executing the above-described processing, the control terminal 200 according to the embodiment of the present disclosure synthesizes the images captured by the hovering camera 100 at any time to generate a composite image, and is designated for the composite image. It is possible to generate flight information such as flying to a position and send it to the hovering camera 100.

In the above-mentioned example, the control terminal 200 generates flight information for controlling the flight of the hovering camera 100 in response to the touch processing on the composite image, but the present disclosure is not limited to such an example.

For example, the control terminal 200 may generate flight information for controlling the flight of the hovering camera 100 in response to a reduction process by a pinch-in operation on the composite image, an enlargement process by a pinch-out operation, and a rotation operation on the composite image.

That is, the control terminal 200 may generate flight information that moves the hovering camera 100 away from the imaging target when a pinch-in operation is performed on the composite image. Further, the control terminal 200 may generate flight information that brings the hovering camera 100 closer to the imaging target when the pinch-in operation is performed on the composite image. Further, the control terminal 200 may generate flight information such that when the rotation operation for the composite image is performed, the image pickup device 101 of the hovering camera 100 rotates while facing the image pickup target.

Further, in the above-described example, the control terminal 200 performs a compositing process on the image captured by the hovering camera 100 and sends a flight instruction from the control terminal 200 to the hovering camera 100. For example, a robot equipped with an image pickup device. Needless to say, the technique of the present embodiment can be applied to the control of the moving body in general.

In the above example, the control terminal 200 performs a compositing process on the image captured by the hovering camera 100, and the control terminal 200 sends a flight instruction to the hovering camera 100. In this case, the image captured by the hovering camera 100 is transmitted from the hovering camera 100 to the control terminal 200 each time the image is taken, and the flight information generated by the control terminal 200, that is, hovering, is transmitted from the control terminal 200 to the hovering camera 100. Information that can be directly interpreted by the camera 100 in the direction of movement is transmitted each time the flight information is generated and updated.

However, the present disclosure is not limited to such examples. For example, the hovering camera 100 executes parameter extraction by image processing on the image captured by the hovering camera 100 and generates a command for controlling the flight of the hovering camera 100, and the control terminal 200 synthesizes the image captured by the hovering camera 100. Only the processing and the input of the target position from the user may be accepted. In this case, the image captured by the hovering camera 100 is transmitted from the hovering camera 100 to the control terminal 200 each time the image is captured, and the information of the target position specified by the user is transmitted from the control terminal 200 to the hovering camera 100 by the user. It is sent every specified time.

When the hovering camera 100 executes the extraction of parameters by image processing for the image captured by the hovering camera 100 and the generation of commands for controlling the flight of the hovering camera 100, the feedback control is completed only in the hovering camera 100. Therefore, the amount of communication required for exchanging information between the control terminal 200 and the hovering camera 100 can be reduced, and the hovering camera 100 can be safely flown even when the communication between the control terminal 200 and the hovering camera 100 is cut off. Becomes possible.

Further, in the above-described example, the composite image is generated so that the image most recently captured by the hovering camera 100 comes to the central portion of the screen, but the present disclosure is not limited to such an example. That is, the display position of the composite image may be fixed, and the image captured by the hovering camera 100 may be composited with the composite image. When the display position of the composite image is fixed, the target position is fixed and the current position of the hovering camera 100 moves with the update of the composite image. Also, when the display position of the composite image is fixed and the composite image reaches the edge of the screen due to the update of the composite image, the entire composite image is scrolled so that the most recently captured image fits on the screen. , The current position and the target position may be updated.

The image pickup apparatus 101 may be attached to the hovering camera 100 so that the image pickup direction is fixed, or may be attached to the hovering camera 100 so that the image pickup direction is changed by, for example, a motor or the like. When the image pickup device 101 is attached to the hovering camera 100 so that the image pickup direction changes, the image pickup direction of the image pickup device 101 is detected by, for example, the control unit 110, and the image pickup direction is detected by the control unit 110. , A process of generating a composite image and a process of designating the moving direction of the hovering camera 100 may be performed.

<2. Summary>
As described above, according to one embodiment of the present disclosure, it is based on a hovering camera 100 that automatically flies based on set flight information and images a structure to be inspected, and a still image captured by the hovering camera 100. An inspection system 10 capable of confirming the damaged state of the structure is provided.

The inspection system 10 according to the embodiment of the present disclosure uses information about the structure to be inspected when the control terminal 200 generates flight information to be transmitted to the hovering camera 100. By using the information regarding the structure to be inspected, the control terminal 200 can generate flight information for efficiently inspecting the structure to be inspected by flying the hovering camera 100.

Further, according to one embodiment of the present disclosure, a control capable of generating a composite image from an image captured by the hovering camera 100 and generating flight information for moving the hovering camera 100 by inputting to the composite image. Terminal 200 is provided. The control terminal 200 according to the embodiment of the present disclosure generates a composite image from an image captured by the hovering camera 100, and generates flight information for moving the hovering camera 100 by inputting to the composite image, thereby generating a user. It becomes possible to intuitively operate the hovering camera 100. Therefore, the control terminal 200 according to the embodiment of the present disclosure can easily operate the hovering camera 100 by the user without forcing the user to perform a complicated operation.

In the above embodiment, the image captured by the hovering camera 100 is a still image, and an example of the inspection system 10 in which the damaged state of the bridge 1 is inspected using the still image is shown, but the present disclosure relates to this. It is not limited to the example. The hovering camera 100 may capture the bridge 1 as a moving image while flying, and the information processing device 300 may generate damage data using the moving image captured by the hovering camera 100. The hovering camera 100 periodically acquires position information when the moving image imaging process is executed, and by associating the imaging time of the moving image with the acquisition time of the position information, the information processing device 300 obtains the moving image. It becomes possible to generate the damage data used.

Each step in the process performed by each device of the present specification does not necessarily have to be processed in chronological order in the order described as a sequence diagram or a flowchart. For example, each step in the process executed by each device may be processed in an order different from the order described in the flowchart, or may be processed in parallel.

It is also possible to create a computer program for making the hardware such as the CPU, ROM, and RAM built in each device exhibit the same functions as the configuration of each device described above. It is also possible to provide a storage medium in which the computer program is stored. Further, by configuring each functional block shown in the functional block diagram with hardware or a hardware circuit, a series of processes can be realized by the hardware or the hardware circuit. Further, a part or all of each functional block shown in the functional block diagram used in the above description may be realized by a server device connected via a network such as the Internet. Further, the configuration of each functional block shown in the functional block diagram used in the above description may be realized by a single device or a system in which a plurality of devices are linked. A system in which a plurality of devices are linked may include, for example, a combination of a plurality of server devices, a combination of a server device and a terminal device, and the like.

Although the preferred embodiments of the present disclosure have been described in detail with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that a person having ordinary knowledge in the technical field of the present disclosure can come up with various modifications or modifications within the scope of the technical ideas described in the claims. Of course, it is understood that the above also belongs to the technical scope of the present disclosure.

Moreover, the effects described in the present specification are merely explanatory or exemplary and are not limited. That is, the technique according to the present disclosure may exhibit other effects apparent to those skilled in the art from the description herein, in addition to or in place of the above effects.

The following configurations also belong to the technical scope of the present disclosure.
(1)
Generating an operation image from an image captured by a moving body equipped with an image pickup device, and
To generate movement information for moving the moving body in response to an operation on the generated operation image, and
Control methods, including.
(2)
Generating the movement information generates movement information for moving the moving body to a place based on a designated position designated for the operation image.
The control method according to (1) above.
(3)
Generating the movement information determines the moving direction of the moving body based on the installation state of the imaging device in the moving body at the time of generating the moving information.
The control method according to (2) above.
(4)
To generate the movement information, the movement information for moving the moving body to the place after changing the direction of the moving body so as to face the front to the place based on the designated position is generated.
The control method according to (2) or (3) above.
(5)
Generating the movement information generates movement information for moving the moving body so as to approach the image pickup target of the imaging device based on the enlargement processing for the operation image.
The control method according to any one of (1) to (4) above.
(6)
Generating the movement information generates movement information for moving the moving body so as to move away from the image pickup target of the imaging device based on the reduction process for the operation image.
The control method according to any one of (1) to (5) above.
(7)
Generating the movement information generates movement information for moving the moving body so that the imaging device rotates while facing the image pickup target, based on the rotation processing for the operation image.
The control method according to any one of (1) to (6) above.
(8)
Generating the operational image generates the operational image so that the center of the image most recently captured by the moving body is located at the center of the screen.
The control method according to any one of (1) to (7) above.
(9)
The control method according to any one of (1) to (8) above, wherein the moving body is a flight device.
(10)
An image processing unit that generates an operation image from an image captured by a moving body equipped with an image pickup device,
A movement information generation unit that generates movement information for moving the moving body in response to an operation on the operation image generated by the image processing unit.
A control device.

10 Inspection system 100 Hovering camera 101 Imaging device 104a to 104d Rotor 108a to 108d Motor 110 Control unit 120 Communication unit 130 Sensor unit 132 Position information acquisition unit 140 Storage unit 150 Battery 200 Control terminal 300 Information processing device 400 Wireless relay node 500 Position estimation Node 600 Base Station 700 Charging Station

Claims (10)

Generating an operation image from an image captured by a moving body equipped with an image pickup device, and
To generate movement information for moving the moving body in response to an operation on the generated operation image, and
Control methods, including. Generating the movement information generates movement information for moving the moving body to a place based on a designated position designated for the operation image.
The control method according to claim 1. Generating the movement information determines the moving direction of the moving body based on the installation state of the imaging device in the moving body at the time of generating the moving information.
The control method according to claim 2. To generate the movement information, the movement information for moving the moving body to the place after changing the direction of the moving body so as to face the front to the place based on the designated position is generated.
The control method according to claim 2. Generating the movement information generates movement information for moving the moving body so as to approach the image pickup target of the imaging device based on the enlargement processing for the operation image.
The control method according to claim 1. Generating the movement information generates movement information for moving the moving body so as to move away from the image pickup target of the imaging device, based on the reduction process for the operation image.
The control method according to claim 1. Generating the movement information generates movement information for moving the moving body so that the imaging device rotates while facing the image pickup target, based on the rotation processing for the operation image.
The control method according to claim 1. Generating the operational image generates the operational image so that the center of the image most recently captured by the moving body is located at the center of the screen.
The control method according to claim 1. The control method according to claim 1, wherein the moving body is a flight device. An image processing unit that generates an operation image from an image captured by a moving body equipped with an image pickup device,
A movement information generation unit that generates movement information for moving the moving body in response to an operation on the operation image generated by the image processing unit.
A control device.