JP7440272B2 - Aircraft, 3D position/attitude measurement device, and 3D position/attitude measurement method - Google Patents

Description

The present invention relates to a flying object, a three-dimensional position/attitude measurement device, and a three-dimensional position/attitude measurement method.

BACKGROUND ART Conventionally, an imaging device is attached to a flying object, the exterior walls of a building, etc. are photographed by the imaging device, and the generated images are combined to obtain an overall image of the building. In image synthesis processing, information on the three-dimensional position and attitude of the flying object at the time of image capture is required. To this end, techniques have been devised to measure the three-dimensional position and attitude of an aircraft.

For example, a technology has been proposed that measures the three-dimensional position and attitude of an aircraft by mounting a GPS signal receiver, a distance measuring sensor, or the like on the aircraft. For example, a plurality of light sources may be attached to a flying object, and a fixed two-dimensional image sensor may be used to photograph the flying object. Based on the position of the light source included in the generated image, the three-dimensional image of the flying object may be displayed. A method of measuring position and orientation has been proposed (for example, Patent Document 1).

Japanese Patent Application Publication No. 2003-254716

However, for example, in a space such as the inside of a boiler furnace, it may be difficult to receive a GPS signal, or it may be difficult for an aircraft that has become larger due to mounting a sensor to enter the space. Furthermore, when measuring the three-dimensional position and attitude of an aircraft based only on the position of a light source attached to the aircraft, a large number of light sources are required.

Therefore, the present invention provides an aircraft that can grasp the three-dimensional position and attitude using a small number of light sources, even in environments where it is difficult to receive GPS signals or in narrow environments where it is difficult for large devices to enter. Another object of the present invention is to provide a three-dimensional position/attitude measurement device and a three-dimensional position/attitude measurement method for measuring the three-dimensional position/attitude of the flying object.

A three-dimensional position/attitude measuring device according to one aspect of the present invention is a three-dimensional position/attitude measuring device that calculates a spatial position and attitude of an aircraft having at least one light source, and is a three-dimensional position/attitude measuring device that calculates a spatial position and attitude of an aircraft that includes a scene including the aircraft. an imaging unit that generates an image by capturing an image; an extraction unit that extracts a light emission mode of at least one light source from the image generated by the imaging unit; two-dimensional coordinates of the at least one light source in the image; and a calculation unit that calculates the three-dimensional position and attitude of the flying object based on the light emission mode extracted by.

According to this aspect, it is possible to grasp the three-dimensional position and attitude of the aircraft using a small number of light sources, even in environments where it is difficult to receive GPS signals or in narrow environments where it is difficult for large devices to enter. becomes. By making it possible to grasp the three-dimensional position and attitude of the aircraft, images generated by the imaging device installed on the aircraft can be used to determine which part of the large equipment being observed and in which direction. It becomes possible to ascertain whether the image is a captured image, and it becomes possible to observe large equipment, etc.

According to the present invention, there is provided a flying object whose three-dimensional position and attitude can be determined using a small number of light sources, a three-dimensional position and attitude measuring device which measures the three-dimensional position and attitude of the flying object, and a three-dimensional A position/orientation measurement method can be provided. By making it possible to grasp the three-dimensional position and attitude of the flying object, it becomes possible to control the flight of the flying object and perform image diagnosis of large equipment using an imaging device installed on the flying object.

1 is a schematic diagram showing an example of the configuration of a three-dimensional position/orientation measurement system 1 according to a first embodiment. It is a block diagram showing an example of composition of flying object 10a concerning a 1st embodiment. FIG. 1 is a block diagram showing an example of the configuration of a three-dimensional position/orientation measuring device 20 according to the first embodiment. FIG. 3 is a schematic diagram for explaining an example of reference information. FIG. 3 is a diagram illustrating an example of an operation flow of a three-dimensional position/attitude measurement method of a flying object 10a by the three-dimensional position/attitude measurement system 1 according to the first embodiment. 3 is a diagram schematically showing an example of the positional relationship between a camera 21a and a light source 13. FIG. FIG. 2 is a schematic diagram showing an example of the configuration of a three-dimensional position/orientation measurement system 2 according to a second embodiment. It is a block diagram showing an example of composition of flying object 10a concerning a 2nd embodiment. FIG. 2 is a block diagram showing an example of the configuration of a three-dimensional position/orientation measuring device 20 according to a second embodiment. FIG. 7 is a diagram illustrating an example of an operation flow of a method for measuring the three-dimensional position and attitude of the flying object 10a using the three-dimensional position and attitude measurement system 2 according to the second embodiment. It is a figure which shows typically an example of the positional relationship of the camera 21a and light sources 13A and 13B. FIG. 3 is a schematic diagram showing an example of the configuration of a three-dimensional position/orientation measurement system 3 according to a third embodiment. It is a block diagram showing an example of composition of flying object 10a concerning a 3rd embodiment. FIG. 3 is a block diagram showing an example of the configuration of a three-dimensional position/orientation measuring device 20 according to a third embodiment. FIG. 7 is a diagram showing an example of an operation flow of a three-dimensional position/attitude measurement method of a flying object 10a by a three-dimensional position/attitude measurement system 2 according to a third embodiment. It is a figure which shows typically an example of the positional relationship of camera 21a and light sources 13A, 13B, and 13C.

Preferred embodiments of the present invention will be described with reference to the accompanying drawings. (In each figure, items with the same reference numerals have the same or similar configurations.)

[First embodiment]
Below, a three-dimensional position/orientation measurement system 1 according to the first embodiment will be described. In the three-dimensional position/attitude measurement system 1 according to the first embodiment, the flying object 10a includes one light source that emits light in a light emission mode depending on the attitude of the flying object 10a. In the three-dimensional position/attitude measurement system 1 according to the first embodiment, the three-dimensional position of the flying object 10a is determined based on the two-dimensional coordinates and brightness of the light source 13 in the image obtained by imaging the flying object 10a with the camera 21a. is calculated. Further, the light emission mode of the light source 13 is extracted based on a plurality of images that are sequentially preceding and following the image, and then the three-dimensional attitude of the flying object 10a is calculated based on the light emission mode.

(1) Configuration (1-1) Three-dimensional position/orientation measurement system 1
FIG. 1 is a schematic diagram showing an example of the configuration of a three-dimensional position/orientation measurement system 1 according to the first embodiment. The three-dimensional position/attitude measurement system 1 includes, for example, a flying object 10a, a control device 10b, and a three-dimensional position/attitude measurement device 20.

One light source 13 is provided in the aircraft 10a. The flying object 10a performs operations according to the control signal and imaging signal received from the control device 10b. The control device 10b is an information processing device for remotely controlling the flying object 10a, and includes, for example, a memory, a communication circuit, a processor, and the like. When the control device 10b receives an operation by the operator, it transmits a control signal and an imaging signal corresponding to the operation to the flying object 10a, for example, by wireless communication.

The three-dimensional position/attitude measurement device 20 is a device that measures the three-dimensional position and attitude of the flying object 10a, and includes, for example, a camera 21a and a control unit 21b. The camera 21a generates an image by capturing a scene including the flying object 10a, and supplies the image to the control unit 21b. The control unit 21b analyzes the image supplied from the camera 21a, and calculates the three-dimensional position and attitude of the flying object 10a based on the two-dimensional coordinates and light emission mode of the light source 13 in the image.

(1-2) Aircraft 10a
FIG. 2 is a block diagram showing an example of the configuration of the flying object 10a according to the first embodiment.

As shown in FIG. 2, the flying object 10a includes four motors 11, a camera 12, a light source 13, a control unit 14, and the like. The control unit 14 includes a control section 140, a memory 141, a motor controller 142, a communication section 143, an attitude sensor 144, a battery 145, and the like.

The four motors 11 are motors that individually rotate rotary blades attached to respective output shafts. Each motor 11 is connected to a propeller, and as the motor 11 rotates, the propeller generates buoyancy, which allows the flying object 10a to fly.

The camera 12 includes an image sensor such as a CCD (Charge-Coupled Device) or a CMOS (Complementary MOS), and uses the image sensor to detect an object (for example, the outer wall of a large facility) based on control data from the camera control unit 140b. The image is captured and the image obtained by the capture is output to the control unit 140.

The light source 13 has an LED (Light Emitting Diode) light emitting section that is driven by a PWM (Pulse Width Modulation) signal. Note that the light source 13 may use other light emitting parts such as a xenon lamp in addition to the LED. The light source 13 has a setting register for setting the brightness of the illumination, and drives the LED light emitting section by adjusting the pulse width of the PWM signal to the brightness set in the setting register. The control unit 140 sets the brightness in the setting register, and sets control data for adjusting the pulse width. In addition to the PWM signal, the method for varying the brightness of the light emitting section may be modified as appropriate, such as by varying the value of the current flowing through the light emitting section.

The control unit 140 includes a microcomputer including a CPU (Central Processing Unit) and the like. The control unit 140 outputs control data to each part of the aircraft 10a by executing a control program stored in the memory 141, thereby controlling the entire aircraft 10a in an integrated manner. The control unit 140 includes, for example, a motor control unit 140a, a camera control unit 140b, and a light source control unit 140c.

The motor control unit 140a controls each motor 11 by outputting control data corresponding to the control signal from the control device 10b received by the communication unit 143 to the motor controller 142. Thereby, the motor control unit 140a performs control regarding the flight of the flying object 10a.

The camera control unit 140b outputs control data corresponding to the imaging signal from the control device 10b received by the communication unit 143 to the camera 12, and performs control regarding imaging by the camera 12.
The camera control unit 140b converts the image output from the camera 12 into a predetermined format and stores it in the memory 141.

The light source control unit 140c monitors the output of the attitude sensor 144 indicating the three-dimensional attitude of the flying object 10a, and outputs control data to the light source 13 so that the light source 13 emits light in a light emission mode corresponding to the output value. Controls light emission.

Specifically, the light source control unit 140c refers to the reference information stored in the memory 141, and based on the output of the attitude sensor 144, determines the rotation angle α, Y around the X axis, which is the three-dimensional attitude of the flying object 10a. The light source 13 is caused to emit light in a light emitting manner such as color information and blinking frequency corresponding to the rotation angle β around the axis and the rotation angle γ around the Z axis. For example, the light source control unit 140c causes the light source 13 to sequentially emit light for a predetermined period using color information C1 corresponding to the rotation angle α, color information C2 corresponding to the rotation angle β, and color information C3 corresponding to the rotation angle γ. It's okay. Further, the light source control unit 140c causes the light source 13 to sequentially emit light for a predetermined period at a blinking frequency f1 corresponding to the rotation angle α, a blinking frequency f2 corresponding to the rotation angle β, and a blinking frequency f3 corresponding to the rotation angle γ. It's okay.

The memory 141 is composed of ROM, RAM, etc., and stores various programs such as driver programs, operating system programs, and application programs, and various data. The memory 141 stores, for example, programs for the control unit 140 to implement the motor control unit 140a, the light source control unit 140c, the camera control unit 140b, and the like. Further, the memory 141 stores, for example, images output from the camera control unit 140b. Further, the memory 141 stores, for example, reference information that the light source control unit 140c refers to when controlling light emission from the light source 13. Reference information will be described later. The various programs stored in the memory 141 may be installed in the memory 141 from a computer-readable portable recording medium such as a CD-ROM or a DVD-ROM using a known setup program or the like.

The motor controller 142 individually controls the rotation speeds of the four motors 11 based on control data from the motor control section 140a of the control section 140.

The communication unit 143 is a wireless communication device that performs wireless communication using a communication method such as WiFi or Bluetooth (registered trademark). For example, the communication unit 143 receives radio waves, demodulates received data from the received radio waves, and outputs data related to the control signal and imaging signal in the received data to the control unit 140.

The attitude sensor 144 includes, for example, an angular velocity sensor such as a three-axis gyro sensor, detects the three-dimensional attitude of the flying object 10a, and outputs the detection result to the control unit 140. Here, the three-dimensional attitude of the flying object 10a detected by the attitude sensor 144 is, for example, rotated around the X axis by a rotation angle α, then rotated around the Y axis by a rotation angle β, and then rotated around the Z axis. It is defined as a three-dimensional posture when rotated by a rotation angle γ.

The battery 146 is a lithium ion battery or the like, and supplies driving power to each part of the aircraft 10a.

(1-3) Three-dimensional position/orientation measuring device 20
FIG. 3 is a block diagram showing an example of the configuration of the three-dimensional position/orientation measuring device 20 according to the first embodiment.

The three-dimensional position/orientation measuring device 20 includes, for example, a camera 21a, a communication section 22 provided in a control unit 21b, a memory 23, and a control section 24.

The camera 21a includes an image sensor such as a CCD (Charge-Coupled Device) or a CMOS (Complementary MOS). The camera 21a generates an image by capturing an image of the flying object 10a using the image sensor at a predetermined frame rate based on the control data supplied from the camera control unit 25, and outputs the image to the memory 23. The frame rate of image generation by the camera 21a may be sufficiently large so that the blinking frequency of the light source 13 can be determined.

The communication unit 22 is a wireless communication device that performs wireless communication using a communication method such as WiFi or Bluetooth (registered trademark). The communication unit 22 transmits and receives information to and from other devices including the flying object 10a.

The memory 23 is composed of a ROM, a RAM, etc., and stores various programs such as driver programs, operating system programs, and application programs, and various data. Furthermore, the memory 23 stores, for example, images generated by the camera 21a in chronological order. Further, for example, the memory 23 stores various types of reference information for calculating the three-dimensional position and attitude of the flying object 10a based on the image. The reference information is referred to, for example, when the three-dimensional attitude calculation unit 273 calculates the attitude of the flying object 10a based on the light emission mode of the light source 13 extracted from the moving image to be analyzed. Reference information will be described later.

The control unit 24 includes a microcomputer including a CPU (Central Processing Unit) and the like. The control unit 24 outputs control data to each part of the three-dimensional position/orientation measuring device 20 by executing the control program stored in the memory 23, thereby controlling the entire three-dimensional position/orientation measuring device 20. to control.

The camera control unit 25 outputs control data to the camera 21a to control imaging by the camera 21a. The camera control unit 25 adds a time stamp to the image output from the camera 21a, converts it into a predetermined format, and stores it in the memory 23.

The image processing unit 26 extracts a predetermined image from the images stored in the memory 23 and processes the image to calculate and extract various parameters for three-dimensional position/orientation measurement. The image processing unit 26 includes, for example, a light source identification unit 261 and a light emission mode extraction unit 262.

The light source identifying unit 261 identifies the light source 13 from the image to be analyzed. Although the method by which the light source specifying unit 261 specifies the light source 13 is not particularly limited, the light source 13 may be specified as follows, for example. That is, the light source specifying unit 261 refers to the brightness and/or color information (color difference, etc.) of each pixel included in the image, and identifies pixels whose brightness and/or color information (color difference, etc.) is equal to or higher than a predetermined threshold. A pixel corresponding to the light source 13 may be specified. In addition, the light source specifying unit 261 determines the luminance and/or Alternatively, the area may be identified as a pixel (area) corresponding to the light source 13 when the color information (color difference, etc.) is greater than or equal to a predetermined threshold. Then, two-dimensional coordinates of the light source 13 are determined according to the position of the specified pixel (area).

The light emission mode extraction unit 262 extracts a plurality of images captured within a predetermined period (which may be a sufficiently short period to the extent that it can be considered that the flying object 10a is not moving) including the image to be analyzed from the memory 23. Then, the light emission mode of the light source 13 is extracted from the plurality of images. Here, the light emission mode is defined by color information of the light source, blinking frequency, and the like. For example, the light emission mode may be such that the color information of the light source 13 changes sequentially to color information C1, color information C2, and color information C3. Further, for example, the light emission mode may be such that the frequency of blinking by the light source 13 changes sequentially from frequency f1 to frequency f2 to frequency f3 for each predetermined period.

The three-dimensional position/attitude calculation unit 27 uses various parameters calculated and extracted by the image processing unit 26 to calculate the three-dimensional position and three-dimensional attitude of the flying object 10a. The three-dimensional position/orientation calculation unit 27 includes, for example, a three-dimensional direction calculation unit 271, a three-dimensional position calculation unit 272, and a three-dimensional orientation calculation unit 273.

The three-dimensional direction calculation unit 271 calculates the three-dimensional direction of the light source 13 based on the two-dimensional coordinates of the light source 13 in the image to be analyzed. Here, the "three-dimensional direction" is a direction with respect to the optical origin O. The calculation method will be described later.

The three-dimensional position calculation unit 272 calculates the three-dimensional position of the light source 13 based on the three-dimensional direction of the light source 13 calculated by the three-dimensional direction calculation unit 271 and the brightness of the light source 13 included in the image to be analyzed. do. The calculation method will be described later.

The three-dimensional attitude calculation section 273 calculates the attitude of the flying object 10a based on the light emission mode of the light source 13 extracted by the light emission mode extraction section 273. The calculation method will be described later.

Here, reference information will be explained with reference to FIG. 4. FIG. 4 is a schematic diagram for explaining an example of reference information. As described above, the reference information is stored, for example, in the memory 141 of the aircraft 10a, and is referenced when the light source control unit 140c of the aircraft 10a controls the light emission of the light source 13. Further, as described above, the reference information is, for example, the 3D attitude calculation unit 273 of the 3D position/attitude measurement device 20 based on the light emitting mode of the light source 13 extracted from the moving image to be analyzed. It is referred to when calculating the three-dimensional attitude of 10a.

FIG. 4 shows a graph in which a relational expression between the color information C or blinking frequency f of the light source 13 and the rotation angle θ around the axis is defined. The reference information is not particularly limited as long as it is a relational expression in which the rotation angle θ is uniquely determined by the color information C or the blinking frequency f, and in addition to the proportional relationship shown in FIG. Alternatively, any relational expression that is monotonically decreasing (one-to-one mapping relationship) may be used. As an example, FIG. 4 shows a graph in which the rotation angle θ monotonically increases with respect to the color information C or the blinking frequency f. By using the graph, it becomes possible to express the rotation angle θ included in the three-dimensional attitude of the flying object 10a using the color information C and the blinking frequency f included in the light emission mode of the light source 13. Then, by sequentially causing the light sources 13 to emit light using the reference information, the flying object 10a can express the three-dimensional attitude of the flying object 10a detected by the attitude sensor 144 by the light emitting mode of the light sources 13. Furthermore, the three-dimensional position/attitude measuring device 20 is capable of calculating the three-dimensional attitude of the flying object 10a based on the light emission mode of the light source 13 extracted from the image obtained by imaging the flying object 10a. Become.

(2) Three-dimensional position/attitude measurement method Referring to FIGS. 5 and 6, the operation flow of the three-dimensional position/attitude measurement method of the flying object 10a by the three-dimensional position/attitude measurement system 1 according to the first embodiment explain.

In the following, as shown in FIG. 6, the optical origin of the camera 21a is O, and the axes of three-dimensional coordinates are the X-axis, Y-axis, and Z-axis that are orthogonal to each other, and the analysis target generated by the camera 21a is Let the image be P and the focal length of the camera 21a be F. In addition, the two-dimensional coordinate origin in the image P is O _P , the two-dimensional coordinate axes in the image P are the x axis parallel to the X axis, and the y axis parallel to the Y axis, and the two dimensional coordinate origin O _P is Let the three-dimensional position coordinates be (0, 0, F). Also, let the three-dimensional position of the light source 13 provided on the aircraft 10a be a point A, and let the three-dimensional coordinates of the point A be (X _A , Y _A , Z _A ). Let the position of point A in image P be point a, the two-dimensional coordinates of point a in image P be (x _a , y _a ), and the brightness of point a be I _a .

(S11: Identify the light source)
First, the light source identifying unit 261 identifies the light source 13 from the image P to be analyzed acquired from the memory 23 . For example, the light source identifying unit 261 refers to the brightness of each pixel included in the image and identifies a pixel whose brightness is equal to or higher than a predetermined threshold as a pixel corresponding to a light source. Then, the two-dimensional coordinates of the light source 13 are specified based on the specified pixels. Thereby, the two-dimensional coordinates and brightness of the light source 13 are specified.

(S12: Extract light emission mode)
Next, the light emission mode extraction unit 262 extracts from the memory 23 a plurality of images captured within a predetermined period, including the image P to be analyzed, and extracts the light emission mode of the light source 13 from the plurality of images. The light emission mode is defined by color information, blinking frequency, etc. of the light source 13. For example, the light emission mode may be such that the color information of the light source 13 changes sequentially to color information C1, color information C2, and color information C3. Note that in this case, the light emission mode extraction unit 262 may calculate color information based on chromaticity information of pixels included in the image, for example. Further, for example, the light emission mode may be such that the frequency of blinking by the light source 13 changes sequentially to frequency f1, frequency f2, and frequency f3.

(S13: Calculate the three-dimensional direction of the light source)
Next, the three-dimensional direction calculation unit 271 calculates the three-dimensional direction of the point A based on the image P to be analyzed. That is, equation (1) holds for the three-dimensional coordinates X _A and Z _A of point A in the X-axis direction and Z-axis direction, the two-dimensional coordinate x _a of point a in the x-axis direction, and the focal length F of the camera 12.

When formula (1) is solved for X _A , the following formula (2) is obtained.

Similarly, based on the three-dimensional coordinates Y _A and Z _A of point A in the Y-axis direction and Z-axis direction, the two-dimensional coordinate _ya of point a in the Y-axis direction, and the focal length F of the camera 12, the following We obtain equation (3).

From equations (2) and (3), the three-dimensional coordinates (X _A , Y _A , Z _A ) of point A are expressed as shown in equation (4) below.

That is, the three-dimensional direction of point A matches the direction of vector l shown in the following equation (5).

(S14: Calculate the three-dimensional position of the light source)
Next, the three-dimensional position calculation unit 272 calculates the three-dimensional position of the light source 13. First, the brightness I _a of the light source 13 in the image P to be analyzed and the distance R from the optical origin O of the light source 13 satisfy the following equation (6), where C is a constant.

When formula (4) is substituted into formula (6), the following formula (7) holds true.

Equation (7) is solved for Z _A to obtain the following equation (8).

From the above, the three-dimensional position of the light source 13 is expressed as in the following equation (9).

(S15: Calculate the three-dimensional posture of the light source)
Next, the three-dimensional attitude calculation unit 273 calculates the three-dimensional attitude of the flying object 10a based on the light emission mode of the light source 13 extracted in S12.

For example, when the light emission mode is defined by the color information C1, C2, and C3 of the light source 13, in the reference information stored in the memory 23, the color information C1 is set at the angle θ1, the color information C2 is set at the angle θ2, and the color information C1 is set at the angle θ2. It is assumed that the information C3 corresponds to the angle θ3. In this case, the three-dimensional attitude calculation unit 263 calculates that the attitude of the flying object 10a is an attitude in which the aircraft 10a is rotated by θ1 around the X-axis, rotated by θ2 around the Y-axis, and rotated by θ3 around the Z-axis. .

For example, when the light emission mode is defined by the blinking frequencies f1, f2, and f3 of the light source 13, in the reference information stored in the memory 23, the blinking frequency f1 is set to the angle θ1, and the blinking frequency f2 is set to the angle θ2. , blinking frequency f3 corresponds to an angle θ3 degrees, respectively. In this case, the three-dimensional attitude calculation unit 263 calculates that the attitude of the flying object 10a is an attitude in which the aircraft 10a is rotated by θ1 around the X-axis, rotated by θ2 around the Y-axis, and rotated by θ3 around the Z-axis. . With the above steps, the three-dimensional position and attitude of the flying object 10a are calculated.

(3) Applications of the three-dimensional position/orientation measurement system 1 The three-dimensional position/orientation measurement system 1 according to the first embodiment can be used, for example, as follows. First, the flying object 10a flies while imaging an object such as an outer wall of a large facility with the camera 12 based on a control signal and an imaging signal supplied from the control device 10b. At this time, the camera 21a of the three-dimensional position/attitude measurement device 20 continuously images the flying object 10a at a predetermined frame rate.

Then, after the imaging of the object by the camera 12 of the aircraft 10a and the imaging of the aircraft 10a by the camera 21a of the three-dimensional position/attitude measurement device 20 are completed, the three-dimensional position/attitude measurement 20 is performed as described above. Based on the image generated by the camera 21a, the three-dimensional position and attitude of the flying object 10a at the time the image was captured are calculated. Further, the information on the three-dimensional position and attitude calculated in this way is used when synthesizing images of the object (such as the outer wall of a large facility) generated by the camera 12 of the flying object 10a.

For example, if the three-dimensional attitude of the flying object 10a is rotated around the X-axis by a rotation angle α, rotated around the Y-axis by a rotation angle β, and rotated around the Z-axis by a rotation angle γ, the three-dimensional attitude The rotation matrix corresponding to the orientation is expressed as the following equation (R).

Therefore, by multiplying the image generated by the camera 12 of the aircraft 10a by the inverse matrix of the rotation matrix corresponding to the three-dimensional attitude, a corrected image in which the three-dimensional attitude of the aircraft 10a is corrected is obtained. be able to. Then, by generating a corrected image in this manner for each three-dimensional position of the aircraft 10a, and then composing the generated corrected images, an image of a target object such as an outer wall of a large facility is generated. .

[Second embodiment]
Below, a three-dimensional position/orientation measurement system 2 according to a second embodiment will be described.

(1) Configuration FIG. 7 is a schematic diagram showing an example of the configuration of a three-dimensional position/orientation measurement system 2 according to the second embodiment. FIG. 8 is a block diagram showing an example of the configuration of a flying object 10a according to the second embodiment. FIG. 9 is a block diagram showing an example of the configuration of a three-dimensional position/orientation measuring device 20 according to the second embodiment.

As shown in FIGS. 7 and 8, in the three-dimensional position/attitude measurement system 2 according to the second embodiment, the flying object 10a has two light sources 13A and 13B that emit light in a light emission mode according to the attitude of the flying object 10a. Equipped with Once the three-dimensional positions of the light sources 13A and 13B are calculated, a vector passing through the two light sources 13A and 13B can be calculated based on the two three-dimensional positions. Once the vector is calculated, only one parameter, the rotation angle around the vector, remains as an unknown parameter regarding the three-dimensional attitude of the flying object 10a. Therefore, the flying object 10a in the three-dimensional position/attitude measurement system 2 according to the second embodiment has a light emission mode (color information and blinking frequency) based on the rotation angle of the flying object 10a around the vector passing through the two light sources 13A and 13B. etc.) to emit light. As a result, the three-dimensional position/orientation measurement device 20 only needs to specify the one parameter using the light emission mode of the light sources 13A and/or 13B, so that the control of the light emission of the light sources 13A and/or 13B is simplified. Become.

Therefore, as shown in FIG. 8, in the flying object 10a according to the second embodiment, the light source control section 140c included in the control section 140 includes, for example, a vector calculation section 140c1 and a rotation angle calculation section 140c2. Here, the vector calculation unit 140c1 calculates the direction of a vector passing through the two light sources 13A and 13B based on the output of the attitude sensor 144 indicating the three-dimensional attitude of the flying object 10a. Further, the rotation angle calculation unit 140c2 calculates the following based on the output of the attitude sensor 144 indicating the three-dimensional attitude of the flying object 10a and the direction of the vector (vector passing through the two light sources 13A and 13B) calculated by the vector calculation unit 140c1. The rotation angle of the flying object 10a around the vector is calculated. Then, the light source control unit 140c causes the light source 13A and/or the light source 13B to emit light in a light emission mode (color information and blinking frequency) according to the rotation angle calculated by the rotation angle calculation unit 140c2.

As shown in FIG. 9, the three-dimensional orientation calculation unit 273 of the three-dimensional position/orientation measuring device 20 according to the second embodiment includes, for example, a vector calculation unit 273a and a rotation angle calculation unit 273b. The vector calculation unit 273a calculates the direction of a vector passing through the two light sources 13A and 13B based on the three-dimensional positions of the two light sources 13A and 13B. The rotation angle calculating section 273b calculates the rotation angle of the flying object 10a around the vector passing through the two light sources 13A and 13B, based on the light emission mode extracted by the light emission mode extraction section 262.

(2) Three-dimensional position/attitude measurement method Referring to FIGS. 10 and 11, the operation flow of the three-dimensional position/attitude measurement method of the flying object 10a by the three-dimensional position/attitude measurement system 2 according to the second embodiment explain.

In the following, as shown in FIG. 11, the three-dimensional position of the light source 13A provided on the aircraft 10a is taken as point A, the three-dimensional coordinates of point A are taken as (X _A , Y _A , Z _A ), and the image of point A is Let the position in P be point a, the two-dimensional coordinates of point a in image P be (x _a , y _a ), and the brightness of point a be I _a . Further, the three-dimensional position of the light source 13B provided on the aircraft 10a is defined as point B, the three-dimensional coordinates of point B are (X _B , Y _B , Z _B ), the position of point B in image P is defined as point B, and the point Let the two-dimensional coordinates of point B in image P be (x _B , y _B ), and the brightness of point B be I _B .

(S21: Identify the light source)
First, the light source identifying unit 261 identifies each of the light sources 13A and 13B from the image P to be analyzed acquired from the memory 23. For example, the light source identifying unit 261 refers to the brightness of each pixel included in the image and identifies a pixel whose brightness is equal to or higher than a predetermined threshold as a pixel corresponding to a light source. Then, based on the identified pixels, two-dimensional coordinates of each of the light sources 13A and 13B are identified. Thereby, the two-dimensional coordinates and brightness of each of the light sources 13A and 13B are specified.

(S22: Extract light emission mode)
Next, the light emission mode extraction unit 262 extracts the light emission mode of the light source 13A and/or the light emission mode of the light source 13B. For example, the light emission mode extraction unit 262 may extract color information of the light source 13A and/or color information of the light source 13B from the image P to be analyzed. Alternatively, the light emission mode extraction unit 262 extracts from the memory 23 a plurality of images captured within a predetermined period including the image P to be analyzed, and extracts the blinking frequency of the light source 13A and/or the blinking frequency of the light source 13B from the plurality of images. Frequencies may also be extracted.

(S23: Calculate the three-dimensional direction of the light source)
Next, the three-dimensional direction calculation unit 271 calculates the three-dimensional directions of each of the points A and B based on the image P to be analyzed. That is, based on the three-dimensional coordinates X _A and Z _A of point A in the X-axis direction and Z-axis direction, the two-dimensional coordinate x _a of point a in the x-axis direction, and the focal length F of the camera 12, The dimensional direction coincides with the direction of vector l1 shown in the following equation (10-1).

Also, based on the three-dimensional coordinates X _B and Z _B of point B in the X-axis direction and Z-axis direction, the two-dimensional coordinate x b of point _b in the x-axis direction, and the focal length F of the camera 12, The dimensional direction coincides with the direction of vector l2 shown in the following equation (10-2).

(S24: Calculate the three-dimensional position of the light source)
Next, the three-dimensional position calculation unit 272 calculates the three-dimensional positions of each of the light sources 13A and 13B. First, the brightness I _a and I _b of the image P to be analyzed for the light sources 13 and 13B, and the distances R _A and R _B of the light sources 13 and 13B from the optical origin O, respectively, are as follows, where C is a constant. Formula (11) is satisfied.

Based on formulas (10-1), (10-2), and (11), the three-dimensional positions of the light sources 13A and 13B are expressed as in the following formulas (12) and (13). Ru.

(S25: Calculate the direction of the vector passing through the two light sources)
Next, the vector calculation unit 273a of the three-dimensional attitude calculation unit 273 calculates a vector passing through the light sources 13A and 13B of the flying object 10a as shown in the following equation (14).

(S26: Calculate the rotation angle around the vector)
Next, the rotation angle calculation section 273b of the three-dimensional posture calculation section 273 refers to the reference information stored in the memory 23, and based on the light emission mode of the light source 13A and/or the light emission mode of the light source 13B extracted in S22. Then, the rotation angle of the flying object 10a around the vector AB is calculated. For example, if the color information of the light source 13A and/or the color information of the light source 13B is C4, and the color information C4 corresponds to the rotation angle θ4 in the reference information, the three-dimensional attitude calculation unit 273 calculates the attitude of the flying object 10a by It is calculated that the posture is rotated by θ4 around the vector AB. Alternatively, if the blinking frequency of the light source 13A and/or the blinking frequency of the light source 13B is f4, and the blinking frequency f4 corresponds to the rotation angle θ4 in the reference information, the three-dimensional attitude calculation unit 273 calculates the attitude of the flying object 10a by It is calculated that the posture is rotated by θ4 around the vector AB. With the above steps, the three-dimensional position and attitude of the flying object 10a are calculated.

[Third embodiment]
Below, a three-dimensional position/orientation measurement system 3 according to a third embodiment will be described.

(1) Configuration FIG. 12 is a schematic diagram showing an example of the configuration of a three-dimensional position/orientation measurement system 3 according to the third embodiment. FIG. 13 is a block diagram showing an example of the configuration of a flying object 10a according to the third embodiment. FIG. 14 is a block diagram showing an example of the configuration of a three-dimensional position/orientation measuring device 20 according to the third embodiment.

As shown in FIGS. 12 and 13, in the three-dimensional position/attitude measurement system 3 according to the third embodiment, the flying object 10a includes three light sources 13A, 13B, and 13C. The light source control unit 140c of the aircraft 10a does not control the light sources 13A, 13B, and 13C to emit light in a light emission mode based on the three-dimensional attitude of the aircraft 10a. The light source control unit 140c controls the light sources 13A, 13B, and 13C to emit light in a manner that is based on the three-dimensional attitude of the flying object 10a, but is not particularly limited in its content. Each of the light sources 13A, 13B, and 13C may be caused to emit light based on color information, intensity, or the like.

In the three-dimensional position/attitude measurement system 3 according to the third embodiment, the flying object 10a does not need to include the attitude sensor 144. Furthermore, in the three-dimensional position/attitude measurement system 3 according to the third embodiment, the memory 141 of the flying object 10a controls the light sources 13A, 13B, and 13C to emit light in a light emission mode according to the three-dimensional attitude of the flying object 10a. There is no need to store the reference information to be referred to when doing so.

As shown in FIG. 14, in the three-dimensional position/orientation measurement system 3 according to the third embodiment, the image processing section 26 of the three-dimensional position/orientation measurement device 20 does not need to include the light emission mode extraction section 262. Further, in the three-dimensional position/orientation measurement system 3 according to the third embodiment, the three-dimensional direction calculation unit 271 calculates the light source based on the two-dimensional coordinates of each of the light sources 13A, 13B, and 13C in the image to be analyzed. The three-dimensional directions of each of 13A, 13B, and 13C are calculated. In the three-dimensional position/orientation measurement system 3 according to the third embodiment, the three-dimensional position calculation unit 272 calculates the three-dimensional directions of the light sources 13A, 13B, and 13C, and the light source 13A in the image to be analyzed. The three-dimensional position of each light source 13 is calculated based on the distance to each light source 13 calculated from the brightness of each of 13B and 13C. Further, in the three-dimensional position/attitude measurement system 3 according to the third embodiment, the three-dimensional attitude calculation unit 273 calculates the three-dimensional attitude of the flying object 10a based on the three-dimensional positions of the light sources 13A, 13B, and 13C. Calculate.

(2) Three-dimensional position/attitude measurement method Referring to FIGS. 15 and 16, the operation flow of the three-dimensional position/attitude measurement method of the flying object 10a by the three-dimensional position/attitude measurement system 3 according to the third embodiment explain.

In the following, as shown in FIG. 16, the three-dimensional position of the light source 13A provided on the aircraft 10a is referred to as point A, the three-dimensional coordinates of point A are (X _A , Y _A , Z _A ), and the image of point A is Let the position in P be point a, the two-dimensional coordinates of point a in image P be (x _a , y _a ), and the brightness of point a be I _a . Further, the three-dimensional position of the light source 13B provided on the aircraft 10a is defined as point B, the three-dimensional coordinates of point B are (X _B , Y _B , Z _B ), the position of point B in image P is defined as point B, and the point Let the two-dimensional coordinates of point B in image P be (x _B , y _B ), and the brightness of point B be I _B . Further, the three-dimensional position of the light source 13C provided on the aircraft 10a is a point C, the three-dimensional coordinates of the point C are (X _C , Y _C , Z _C ), the position of the point C in the image P is a point C, and the point Let the two-dimensional coordinates of point C in image P be (x _C , y _C ), and the brightness of point C be I _C .

(S31: Identify the light source)
First, the light source identifying unit 261 identifies each of the light sources 13A, 13B, and 13C from the image P to be analyzed acquired from the memory 23. For example, the light source identifying unit 261 refers to the brightness of each pixel included in the image and identifies a pixel whose brightness is equal to or higher than a predetermined threshold as a pixel corresponding to a light source. Then, based on the identified pixels, the two-dimensional coordinates of each of the light sources 13A, 13B, and 13C are identified. Thereby, the two-dimensional coordinates and brightness of each of the light sources 13A, 13B, and 13C are specified.

(S32: Calculate the three-dimensional direction of the light source)
Next, the three-dimensional direction calculating unit 271 calculates the three-dimensional directions of each of the points A, B, and C based on the image P to be analyzed. That is, based on the three-dimensional coordinates X _A and Z _A of point A in the X-axis direction and Z-axis direction, the two-dimensional coordinate x _a of point a in the x-axis direction, and the focal length F of the camera 12, The dimensional direction coincides with the direction of vector l1 shown in the following equation (15-1).

Also, based on the three-dimensional coordinates X _B and Z _B of point B in the X-axis direction and Z-axis direction, the two-dimensional coordinate x b of point _b in the x-axis direction, and the focal length F of the camera 12, The dimensional direction coincides with the direction of vector l2 shown in the following equation (15-2).

Also, based on the three-dimensional coordinates X _C and Z _C of point C in the X-axis direction and Z-axis direction, the two-dimensional coordinate x _c of point c in the x-axis direction, and the focal length F of the camera 12, the three-dimensional coordinates of point C The dimensional direction coincides with the direction of vector l3 shown in the following equation (15-3).

(S33: Calculate the three-dimensional position of the light source)
Next, the three-dimensional position calculation unit 272 calculates the three-dimensional positions of each of the light sources 13A, 13B, and 13C. First, the luminances I _a , I _b , and I _c of each of the light sources 13A, 13B, and 13C in the image P to be analyzed, and the distances R _A , RB , of each of the light sources 13A, 13B, and _13C from the optical origin O, R _C satisfies the following formula (16), where C is a constant.

Based on Equation (15-1), Equation (15-2), Equation (15-3), and Equation (16), the three-dimensional positions of the light sources 13A, 13B, and 13C are calculated using the following equation (17). , Expression (18), and Expression (19).

(S34: Calculate the three-dimensional posture of the light source)
Next, the three-dimensional attitude calculation unit 273 calculates the three-dimensional position and attitude of the flying object 10a based on the three-dimensional positions of the light sources 13A, 13B, and 13C. The three-dimensional position of the flying object 10a is not particularly limited as long as it is defined based on the three-dimensional positions of the light sources 13A, 13B, and 13C, but for example, the center of gravity of the light sources 13A, 13B, and 13C, etc. It may be. Further, the three-dimensional attitude of the flying object 10a is not particularly limited as long as it is defined based on the three-dimensional positions of the light sources 13A, 13B, and 13C, but for example, the following equation (20) is used. It may be expressed as a vector directed from the midpoint between the light source 13B and the light source 13C toward the light source 13A, as shown in FIG.

Various embodiments have been described above. According to the flying object, the three-dimensional position/attitude measurement device, and the three-dimensional position/attitude measurement method according to each of the embodiments described above, the flying object, the three-dimensional position/attitude measurement device, and the three-dimensional position/attitude measurement method can be used in environments where it is difficult to receive GPS signals or where it is difficult for large devices to enter. Even in a confined environment, it is possible to grasp the three-dimensional position and attitude of an aircraft using a small number of light sources.

The embodiments described above are intended to facilitate understanding of the present invention, and are not intended to be interpreted as limiting the present invention. Each element included in the embodiment, as well as its arrangement, material, conditions, shape, size, etc., are not limited to those illustrated, and can be changed as appropriate. Further, it is possible to partially replace or combine the structures shown in different embodiments.

1, 2, 3... Three-dimensional position/attitude measurement system, 10a... Flying object, 10b... Control device, 11... Motor, 12... Camera, 13, 13A, 13B, 13C... Light source, 140... Control unit, 140a... Motor Control unit, 140b...Camera control unit, 140c...Light source control unit, 140c1...Vector calculation unit, 140c2...Rotation angle calculation unit, 141...Memory, 142...Motor controller, 143...Communication unit, 144...Posture sensor, 145...Battery , 20...Three-dimensional position/orientation measuring device, 21a...Camera, 21b...Control unit, 22...Communication unit, 23...Memory, 24...Control unit, 25...Camera control unit, 26...Image processing unit, 261...Light source identification Part, 27...3D position/attitude calculation unit, 271...3D direction calculation unit, 272...3D position calculation unit, 273...3D attitude calculation unit, 273a...Vector calculation unit, 273b...Rotation angle calculation unit

Claims (9)

A flying object,
an attitude detection unit that detects the attitude of the flying object;
at least one light source;
a light source control unit that causes the at least one light source to emit light in a light emission mode based on the orientation detected by the orientation detection unit;
The attitude detection unit detects three rotation angles with each of three orthogonal coordinate axes as rotation axes as the attitude of the flying object ,
The flying object, wherein the light source control unit causes the at least one light source to emit light with color information and/or blinking frequency according to each of the three rotation angles detected by the attitude detection unit. The flying object according to claim 1 , wherein the light source control unit causes the at least one light source to sequentially emit light with color information corresponding to each of the three rotation angles. The flying object according to claim 1 , wherein the light source control unit causes the at least one light source to sequentially emit light at a blinking frequency corresponding to each of the three rotation angles. The flying object according to any one of claims 1 to 3 , wherein the attitude detection section is an angular velocity sensor. The aircraft according to any one of claims 1 to 4 , wherein the at least one light source is an LED. The flying object according to any one of claims 1 to 5 , wherein the number of the at least one light source is one or two. A three-dimensional position/attitude measuring device for calculating the spatial position and attitude of a flying object according to any one of claims 1 to 6 ,
an imaging unit that generates an image by imaging a scene including the flying object;
an extraction unit that extracts a light emission mode of the at least one light source from the image generated by the imaging unit;
a three-dimensional position, comprising: a calculation unit that calculates a three-dimensional position and attitude of the flying object based on two-dimensional coordinates of the at least one light source in the image and the light emission mode extracted by the extraction unit;・Posture measurement device. Generating an image by capturing a scene including the flying object according to any one of claims 1 to 7;
extracting a light emission mode of the at least one light source from the generated image;
A three-dimensional position/attitude measurement method comprising: calculating a three-dimensional position and attitude of the flying object based on two-dimensional coordinates of the at least one light source in the image and the extracted light emission mode. A three-dimensional position/attitude measurement device that calculates the spatial position and attitude of a flying object comprising at least three light sources,
an imaging unit that generates an image by imaging a scene including the flying object;
a calculation unit that calculates the position and attitude of the flying object by calculating the three-dimensional position of each of the at least three light sources based on the two-dimensional coordinates and brightness in the image of each of the at least three light sources; , comprising;
A three-dimensional position/attitude measurement device, wherein the attitude of the flying object is defined based on the three-dimensional positions of each of the at least three light sources.