JP6093226B2 - Plant position measuring apparatus and method - Google Patents

Description

  The present invention relates to a plant position measuring apparatus and method for measuring the position of a plant leaf existing on the ground.

It is desired to acquire the distribution of the leaf position by measuring the position of the leaf of the plant and use this distribution as follows.
(1) A vehicle or robot that autonomously moves on an unpaved ground determines whether or not the position can pass based on the distribution of the positions where the leaves of the plants exist. Thereby, it can avoid determining the leaf of a plant as an obstruction which cannot pass immediately.
(2) A map showing the distribution of leaf positions can be used for forest resources and disaster prevention. For example, a mountain area where a plant leaf is located can be recorded as an area where forest resources exist, and a mountain area where a plant leaf does not exist can be recorded as an area where landslides are likely to occur.
(3) Create a map showing the distribution of leaf positions in the field. Based on this map, the position where the leaf of the plant exists outside the range where the crop is planted can be recognized as the position of the weed. Based on the position of the weeds, the agricultural machine that moves autonomously can remove the weeds.

  The technique regarding the detection of the leaf of a plant is described in the following patent documents 1 and 2, for example.

  In Patent Document 1, sunlight reflected from the leaves of a plant is dispersed and received, and the degree of plant growth is determined based on the received light intensity of each of the dispersed light.

  In Patent Document 2, an image captured through an optical filter that transmits only light in the first wavelength range that is not absorbed by the leaves of the plant and an optical filter that transmits only light in the second wavelength range that is absorbed by the leaves of the plant Whether or not the observation object is a plant is determined based on the image picked up via the.

JP 2006-314215 A JP 2010-44623 A

  However, based on the received light intensity detected as in Patent Document 1 or the image data acquired as in Patent Document 2, the position of the plant cannot be obtained with high accuracy. In Patent Literature 1, the degree of plant growth is detected, but the position of the leaf of the plant is not strictly detected. As in Patent Document 2, in the detection of the leaf position based on the image data, the accuracy of the position detection is limited.

  Therefore, an object of the present invention is to provide a plant position measuring apparatus and method capable of specifying the position of a leaf existing on the ground with high accuracy.

In order to achieve the above-mentioned object, according to the present invention, a plant position measuring device for measuring a position where a leaf of a plant exists on the ground,
A distance sensor that measures the distance to each measurement point on a plant or other object that exists in each measurement direction by emitting laser light in each measurement direction from a base point set on the moving body;
An imaging device that images an area including each of the measurement points;
The distance sensor and the imaging device are provided on a moving body that moves on the ground.
While moving the moving body, the distance sensor measures the distance at multiple measurement times,
Furthermore, an operation amount measuring device for determining the amount of change in the direction and position of the moving body between the imaging time and each measurement time;
A data processing device for determining a leaf position based on a distance and a measurement direction measured by a distance sensor, an image obtained by an imaging device, and the amount of change obtained by an operation amount measuring device;
The data processing device
(A) Based on the distance to the measurement point measured by the distance sensor at the measurement time and the measurement direction, the position of the measurement point is represented by a coordinate value of a sensor coordinate system fixed to the distance sensor,
(B) Based on the amount of change between the imaging time and the measurement time, the coordinate value of the sensor coordinate system is converted into a coordinate value of a camera coordinate system fixed to the imaging device at the imaging time,
(C) determining whether the converted coordinate value corresponds to a pixel indicating a leaf in an image obtained by the imaging device;
Converting the coordinate value determined to correspond to a pixel indicating a leaf in (C) to a coordinate value of a stationary coordinate system fixed on the ground, and recording the converted coordinate value as a leaf position; A characteristic plant position measuring apparatus is provided.

According to a preferred embodiment of the present invention, the posture of the distance sensor with respect to the main body of the moving body is changed so that the posture of the sensor coordinate system with respect to the main body of the moving body is different between a plurality of measurement times.
In (B) above, the data processing device
When the sensor coordinate system when the distance sensor is in the reference posture is a reference sensor coordinate system, and the sensor coordinate system at the measurement time does not match the reference sensor coordinate system, the coordinate value of the sensor coordinate system is Convert to the coordinate value of the main body coordinate system fixed to the main body of the moving body at the measurement time of the distance sensor,
The coordinate value of the main body coordinate system is converted into the coordinate value of the main body coordinate system fixed to the main body of the moving body at the imaging time of the imaging device,
The coordinate value of the main body coordinate system is converted into the coordinate value of the camera coordinate system fixed to the imaging device at the imaging time.

In addition, the data processing device, in (B), for each measurement time,
The coordinate value of the sensor coordinate system is converted into the coordinate value of the main body coordinate system fixed to the main body of the moving body at the measurement time,
Convert the coordinate value of the main body coordinate system to the coordinate value of the main body coordinate system fixed to the main body of the moving body at the imaging time,
The coordinate value of the main body coordinate system may be converted into the coordinate value of the camera coordinate system fixed to the imaging device at the imaging time.

Moreover, according to the present invention, there is a plant position measuring method for measuring a position where a plant leaf exists on the ground,
A distance sensor that measures the distance to each measurement point on a plant or other object that exists in each measurement direction by emitting laser light in each measurement direction from a base point set on the moving body;
An imaging device that images the area including each of the measurement points, and a moving body that moves on the ground,
While moving the moving body, the distance sensor performs distance measurement at a plurality of measurement times, and the imaging device performs imaging.
Find the amount of change in the direction and position of the moving body between the imaging time and each measurement time,
For each measurement time,
(A) Based on the distance to the measurement point measured by the distance sensor at the measurement time and the measurement direction, the position of the measurement point is represented by a coordinate value of a sensor coordinate system fixed to the distance sensor,
(B) Based on the amount of change between the imaging time and the measurement time, the coordinate value of the sensor coordinate system is converted into a coordinate value of a camera coordinate system fixed to the imaging device at the imaging time,
(C) determining whether the converted coordinate value corresponds to a pixel indicating a leaf in an image obtained by the imaging device;
Converting the coordinate value determined to correspond to a pixel indicating a leaf in (C) to a coordinate value of a stationary coordinate system fixed on the ground, and recording the converted coordinate value as a leaf position; A featured plant location measurement method is provided.

According to the above-described present invention, since both the distance sensor and the imaging device are used, the position of the leaf can be specified with high accuracy. That is, based on the distance to each measured point measured by the distance sensor, the position of each measured point is represented by the coordinate value of the sensor coordinate system fixed to the distance sensor, and the measured point represented by the sensor coordinate system Each coordinate value is converted into each coordinate value of the camera coordinate system, and among the converted coordinate values, the coordinate value corresponding to the pixel indicating the leaf in the image obtained by the imaging device is specified, thereby existing on the ground It is possible to specify the position of the leaf in high accuracy.
Further, for each measurement time, the coordinate value of the sensor coordinate system is fixed to the imaging device at the imaging time based on the amount of change in the direction and position of the moving body between the measurement time and the imaging time. Therefore, the position of the leaf existing on the ground can be specified with higher accuracy.

1 shows a plant position measuring apparatus according to an embodiment of the present invention. 2 shows an example of the configuration of an imaging apparatus. It is a flowchart which shows the plant position measuring method by embodiment of this invention. It is explanatory drawing of coordinate transformation. It is explanatory drawing of the effect by embodiment of this invention.

  A preferred embodiment of the present invention will be described with reference to the drawings. In addition, the same code | symbol is attached | subjected to the common part in each figure, and the overlapping description is abbreviate | omitted.

  FIG. 1 shows a plant position measuring apparatus 10 according to an embodiment of the present invention.

  The plant position measuring device 10 includes a distance sensor 3, an imaging device 5, an operation amount measuring device 7, and a data processing device 9.

  The distance sensor 3, the imaging device 5, and the operation amount measuring device 7 are installed on the moving body 11. The data processing device 9 is preferably installed on the moving body 11. When the data processing device 9 is not installed on the moving body 11, for example, the data processing device 9 receives data from the distance sensor 3, the imaging device 5, and the operation amount measuring device 7 by radio. In the example of FIG. 1, the moving body 11 is a vehicle that moves on the ground by wheels that are rotated and driven in contact with the ground. However, a robot having legs that walk, a traveling body that travels on the ground by a crawler, and the like. Other moving bodies may be used.

  The distance sensor 3 uses a plane including the base point Pa as a scanning plane, and emits laser light from the base point Pa in each measurement direction included in the scan plane, thereby causing a plant or other object existing in each measurement direction to Measure the distance to each measurement point.

The distance sensor 3 performs distance measurement at a plurality of measurement times. Between a plurality of measurement times, the posture of the distance sensor 3 with respect to the main body 11a of the moving body 11 (that is, the direction of the reference line fixed to the distance sensor 3 with respect to the reference line fixed to the main body 11a) is changed from the reference posture. By doing so, the direction of the scanning plane may be changed. For example, as shown in FIG. 1, the posture of the distance sensor 3 with respect to the main body 11a is made by swinging the distance sensor 3 with respect to the main body 11a around the rotation axis that is perpendicular to the paper surface of FIG. To change. In this way, the scanning plane on which the laser light travels is different between the measurement times. For example, these scanning planes are represented by broken lines X1, X2, X3, and X4 in FIG. 1, respectively, and these scanning planes X1, X2, X3, and X4 are orthogonal to the paper plane of FIG.
The measurement direction by the distance sensor 3 is preferably directed to the traveling direction side of the moving body 11. In this case, the measurement range (viewing angle) by the distance sensor 3 is set to be wider than the imaging range (viewing angle) by the imaging device 5 in the left-right direction with respect to the traveling direction of the moving body 11.

  The imaging device 5 captures an area including each measurement point measured by the distance sensor 3. The imaging direction by the imaging device 5 is a direction in which the moving body 11 faces the traveling direction side and is inclined downward from the horizontal direction. The imaging range (viewing angle) by the imaging device 5 is, for example, 60 degrees to the left and right. Note that the imaging range of the imaging device 5 includes at least some of the plurality of measurement points whose distances are measured by the distance sensor 3.

As illustrated in FIG. 2, the imaging device 5 includes a first imaging unit 5 a that generates a first image by imaging the region via the first filter 13, and the region via the second filter 15. And a second imaging unit 5b that generates a second image by imaging.
The first filter 13 transmits light in a wavelength range included in the first wavelength range. Preferably, the first filter 13 transmits only light in the wavelength range included in the first wavelength range. Here, the first wavelength range is a wavelength range of light reflected by the leaves of the plant (that is, chlorophyll). Specifically, the first wavelength range is a wavelength range of 800 nm to 1200 nm, and includes 800 nm and 1200 nm. The wavelength range included in the first wavelength region preferably has a width of 100 nm or more.
The second filter 15 transmits light in a wavelength range included in the second wavelength range. Preferably, the second filter 15 transmits only light in a wavelength range included in the second wavelength range. Here, the second wavelength range is a wavelength range of light absorbed by the leaves of the plant (that is, chlorophyll). Specifically, the second wavelength range is a wavelength range of 600 nm to 680 nm, and includes 600 nm and 680 nm. In this case, the wavelength range included in the second wavelength region has a maximum width of 80 nm. Preferably, the first filter 13 may transmit light in the wavelength range having a width of 30 nm or more and 100 nm or less.

  The first imaging unit 5 a has an imaging surface 17 that receives light from the region via the first filter 13. A large number of CCD light receiving elements are two-dimensionally arranged on the imaging surface 17. The first imaging unit 5a generates a first image (in this example, a two-dimensional image) corresponding to the received light. The second imaging unit 5 b has an imaging surface 19 that receives light from the region via the second filter 15. A large number of CCD light receiving elements are two-dimensionally arranged on the imaging surface 19. The second imaging unit 5b generates a second image (in this example, a two-dimensional image) corresponding to the received light.

The motion amount measurement device 7 obtains the amount of change in the direction and position of the moving body 11 between each measurement time of the distance sensor 3 and the imaging time of the imaging device 5.
Specifically, the motion amount measuring device 7 is configured such that the relative direction of the moving body 11 at the imaging time by the imaging device 5 with respect to the orientation of the moving body 11 at the time measured by the distance sensor 3 and the position of the moving body 11 at the time of distance measurement. The relative position of the moving body 11 at the imaging time with respect to is determined as the amount of change. The direction of the moving body 11 is, for example, the direction of the reference point set at the front portion of the moving body 11 as viewed from the center point of the moving body 11.

  The operation amount measuring device 7 includes a measurement unit 7a and a calculation unit 7b.

  The measurement unit 7a includes each time included in a period between the distance measurement time and the imaging time (for example, each time including an intermediate time between the distance measurement time and the imaging time), and the distance measurement time and the imaging time. The direction of the moving body 11 and the moving speed of the moving body 11 are measured. For example, when the moving body 11 has a wheel that moves by rolling on the ground, the measuring unit 7a measures the rotational speed of the wheel, and the moving body 11 determines the moving body from the rotational speed. 11 movement speed is calculated. The measurement unit 7 a detects the direction (traveling direction) of the moving body 11 using a gyro sensor installed on the moving body 11.

  Based on the direction and moving speed of the moving body 11 at each time measured by the measuring unit 7a, the calculation unit 7b determines the relative direction of the moving body 11 at the imaging time with respect to the direction of the moving body 11 at the time of distance measurement. Then, the relative position of the moving body 11 at the imaging time with respect to the position of the moving body 11 at the time of distance measurement is obtained.

  The data processing device 9 is preferably installed on the moving body 11. The data processing device 9 obtains the leaf position based on the distance measured by the distance sensor 3 and the image captured by the imaging device 5.

The data processing device 9 performs the following processes (A) to (C) for each measurement time.
(A) Based on the measurement direction measured by the distance sensor 3 at the measurement time and the distance to the measurement point existing in the measurement direction, the position of the measurement point is a sensor coordinate fixed to the distance sensor 3. Expressed in the coordinate value of the system.
(B) Based on the change amount measured by the motion amount measuring device 7 between the imaging time and the measurement time, the coordinate values of the sensor coordinate system are fixed to the imaging device 5 at the imaging time. Convert to coordinate value of system.
(C) It is determined whether the converted coordinate value corresponds to a pixel indicating a leaf in the image data obtained by the imaging device 5 by imaging.

  The data processing device 9 further converts the coordinate value determined to correspond to the pixel indicating the leaf in (C) for each measurement time into a coordinate value of a stationary coordinate system fixed on the ground, and the converted coordinate Record the value as the leaf position.

  FIG. 3 is a flowchart showing a plant position measuring method according to an embodiment of the present invention.

  This method is performed by the plant position measuring apparatus 10. The method also includes steps S1 to S8. During steps S1 to S8, the moving body 11 may continue to move on the ground.

In step S1, the distance sensor 3 emits a laser beam from the base point Pa in each measurement direction included in the scanning plane, so that the distance from each measurement point on the plant or other object that exists in each measurement direction. Measure.
The distance measured in each measurement direction and the measurement direction are stored in the storage unit of the data processing device 9 as measurement data in a state of being associated with each other.

  In step S2, the position and orientation of the moving body 11 when step S1 is performed are measured by the measuring unit 7a of the motion amount measuring apparatus 7. In this regard, the position and orientation of the moving body 11 to be measured are expressed in a stationary coordinate system fixed on the ground.

  On the other hand, in step S3, an area including each measurement point measured in step S1 performed a plurality of times as will be described later (in a modified example 1 described later, measured in a plurality of steps S1 performed in the future). Take an image.

  Next, in step S4, the movement amount measuring device 7 measures the position and orientation of the moving body 11 when step S3 is performed. In this regard, the position and orientation of the moving body 11 to be measured are expressed in a stationary coordinate system.

  The imaging in step S3 is performed at time intervals. Step S1 is performed a plurality of times from step 3 to the next step S3. In each step S3, the first image is acquired by the first imaging unit 5a, and the second image is acquired by the second imaging unit 5b. That is, each step S3 obtains image data composed of the first image and the second image.

  Following step S2 described above, step S5 is performed. In step S5, it is determined whether the imaging in step S3 has been performed. If the determination in step S5 is “No”, the process returns to step S1.

  When returning to step S1, this step S1 is performed in a state where the direction of the scanning surface with respect to the main body 11a of the movable body 11 is different from that in the previous step S1. Thereby, a wider range can be measured by the distance sensor 3.

If the determination in step S5 is “yes”, the distance to the point to be measured in each measurement direction measured by repeating step S1 and the measurement data including the measurement direction, and imaging in the immediately preceding step S3. The processing of steps S6 to S8 is performed on the image data obtained by the apparatus 5.
On the other hand, if the determination in step S5 is “Yes”, the process returns to step S1 again, and steps S1, S2, and S5 are repeated as described above, and the next step S3 is performed as described above. This is performed at a time interval from the previous step S3. Thus, when the determination in step S5 is “Yes” again, new measurement data and image data are obtained. The process of steps S6 to S8 is performed again on the new measurement data and image data.
Thereafter, similarly, the new measurement data and the image data are acquired, and the processes of steps S6 to S8 are performed again on the new measurement data and the image data.

  In step S <b> 6, the data processing device 9 specifies a pixel indicating a leaf in the image data. Step S6 includes the following steps S61 and S62 (for details of step S6, see, for example, Patent Document 2).

  In step S61, for each pixel of the first image, the luminance of the pixel is set to A, and the luminance of the pixel of the second image corresponding to the same position as the pixel in the region is set to B. The ratio B / A of the luminance B with respect to is obtained.

  In step S62, the position of the leaf of the plant in the first image or the second image is specified by the data processing device 9 based on the ratio B / A in the first image or the second image. Here, in the first image or the second image, the position of the pixel where the ratio B / A is equal to or greater than the threshold is specified as the position indicating the leaf of the plant in the first image or the second image.

Of the measurement data, step S7 is performed for each data obtained at each measurement time.
That is, when the measurement data is data obtained at n measurement times, step S7 is performed on the data about the distance to the measurement point in the measurement direction obtained at the first measurement time and the measurement direction. Step S7 is performed on the data of the distance to the measurement point in the measurement direction obtained at the second measurement time and the data in the measurement direction, and in this way, finally, the nth measurement is performed. Step S7 is performed on the data of the distance to the measurement point in the measurement direction obtained at the time and the measurement direction.

  Step S7 includes steps S71 to S74.

In step S <b> 71, the amount of change in the position and orientation of the moving body 11 between the target measurement time and the imaging time is calculated by the calculation unit 7 b of the motion amount measurement device 7.
This calculation is performed based on the position and orientation of the moving body 11 at the target measurement time measured in step S2 and the position and orientation of the moving body 11 at the target imaging time measured in step S4.

In step S72, based on the measurement direction measured by the distance sensor 3 at the target measurement time and the distance to the measurement point in the measurement direction by the data processing device 9, the position of the measurement point is measured. This is represented by the coordinate value of the sensor coordinate system fixed to the distance sensor 3 at the time.
This sensor coordinate system moves with respect to the stationary coordinate system (that is, the ground) by the movement of the moving body 11. The sensor coordinate system used in step S72 is a coordinate system at the target measurement time, and the origin is located at a certain point in the stationary coordinate system at this measurement time.

In step S73, based on the change amount obtained in step S71 by the data processing device 9, the coordinate value of the sensor coordinate system obtained in step S72 is obtained from the camera coordinate system fixed to the imaging device 5 at the imaging time. Convert to coordinate values.
The camera coordinate system moves with respect to the stationary coordinate system (that is, the ground) by the movement of the moving body 11. The camera coordinate system used in step S73 is a coordinate system at the imaging time of the target, and the origin is located at a certain point in the stationary coordinate system at this imaging time.

  In step S74, the data processing device 9 determines whether the coordinate value converted in step S73 corresponds to the pixel of the image data (first image or second image) specified as the leaf position in step S6. To do.

  Step S7 is performed on the data obtained at all measurement times included in the measurement data. Accordingly, in step S74, the data processing device 9 corresponds to the pixel of the image data (first image or second image) identified as the leaf position in step S6 among the coordinate values converted in step S73. The coordinate value to be specified is specified. Thereafter, the process proceeds to step S8.

  Step S73 will be described with reference to FIG.

In FIG. 4, a symbol t indicates an imaging time, and a symbol (t + Δti) indicates a target measurement time. Here, the suffix i of Δti is for identifying a plurality of measurement times different from each other with respect to one imaging time t (that is, the time when the immediately preceding step S3 was performed), and one imaging time t If there are n measurement times, i is one of 1 to n. In the present embodiment, Δti is a negative value, whereas in the following modification example, it is a positive value or includes both positive and negative values. Each coordinate system S _L0 , S _L1 , S _C , S _B moves with respect to the stationary coordinate system S _S by the movement of the moving body 11 with respect to the stationary coordinate system S _S fixed on the ground. Therefore, the coordinate system shape t is attached served, the coordinate system of the imaging time t, in the imaging time t, the origin point in the stationary coordinate system S _S is positioned. Similarly, subscript (t +? Ti) is a coordinate system attached is a coordinate system in the target measurement time (t +? Ti), at the measurement time (t +? Ti), the origin point in the stationary coordinate system S _S is located doing. Further, the sensor coordinate system when the distance sensor 3 is in the reference attitude and reference sensor coordinate system S _L0, representing the sensor coordinate system of the measuring time (t +? Ti) in S _L1.

  In step S73, the data processor 9 performs coordinate transformations (1) to (6) as follows.

(1) If the sensor coordinate system S _{L1, t + Δti at} the target measurement time (t + Δti) does not match the reference sensor coordinate system S _{L0, t + Δti} , the sensor coordinate system S _{L1 at the} target measurement time (t + Δti) _{, T + Δti} are converted into coordinate values of the reference sensor coordinate system S _{L0, t + Δti} , respectively. This coordinate conversion is performed by a matrix T _{LRF−LRF (θ)} described later.
Here, the sensor coordinate system S _{L1, t + Δti} is a coordinate system fixed to the distance sensor 3 and is a coordinate system viewed from the main body 11a. Therefore, the posture of the sensor coordinate system _{SL1, t + Δti} with respect to the main body 11a is the posture of the distance sensor 3 with respect to the main body 11a (that is, the direction of the reference line fixed to the distance sensor 3 with respect to the reference line fixed to the main body 11a). means.

(2) The coordinate values of the reference sensor coordinate system S _{L0, t + Δti} are converted into coordinate values of the camera coordinate system _{SC, t + Δti at} the target measurement time (t + Δti), respectively. This coordinate conversion is performed by a matrix T _CAM-LRF described later.

(3) The coordinate values of the camera coordinate system _{SC, t + Δti} are respectively coordinated to the coordinate values of the main body coordinate system _{SB, t + Δti} fixed to the main body 11a of the moving body 11 at the target measurement time (t + Δti). Convert. This coordinate conversion is performed by a matrix T _BODY-CAM described later.

(4) The body coordinate system S _B, the coordinate values of the _{t +? Ti,} respectively, to coordinate conversion into coordinate values of the stationary coordinate system S _S fixed on the ground. This coordinate conversion is performed by a matrix T _{EARTH-BODY (t + Δti)} described later.

(5) the coordinates of the stationary coordinate system S _S, the body coordinate system S _B which is fixed to the body 11a of the movable body 11 in the imaging time _t, a coordinate conversion into coordinate values of _t. This coordinate conversion is performed by a matrix T _BODY-EARTH described later.

(6) The coordinate values of the main body coordinate system SB _{, t} are converted into the coordinate values of the camera coordinate system SC _{, t} fixed to the imaging device 5 at the imaging time t, respectively. This coordinate conversion is performed by a matrix T _CAM-BODY described later.

That is, in step S73, the calculation according to the following equation is performed.

V _{CAM, t} = T _CAM-BODY T _BODY-EARTH T _{EARTH-BODY (t + Δti)} T _BODY-CAM T _CAM-LRF T _{LRF-LRF (θ)} V _{LRF (θ, t + Δti)}

Each symbol in this formula is defined as follows.
V _{LRF (θ, t + Δti)} : The coordinate value of the sensor coordinate system S _{L0, t + Δti at} the target measurement time (t + Δti).
T _{LRF−LRF (θ)} : a matrix representing the coordinate transformation of (1).
_TCAM-LRF : a matrix representing the coordinate transformation of (2).
T _BODY-CAM : a matrix representing the coordinate transformation of (3) above.
T _{EARTH-BODY (t + Δti)} : a matrix representing the coordinate transformation of (4).
T _BODY-EARTH : a matrix representing the coordinate transformation of (5).
_TCAM-BODY : a matrix representing the coordinate transformation of (6) above.
V _{CAM, t:} which is the coordinate value of the camera coordinate system _{S C, t} at imaging time t of interest.

Each matrix is obtained by an appropriate means. For example, each matrix is obtained as follows. T _{LRF−LRF (θ)} is obtained by measuring the direction of the distance sensor 3 with respect to the main body 11a at the measurement time (t + Δti) by an appropriate means. T _CAM-LRF is obtained based on the position and orientation relationship between the distance sensor 3 and the imaging device 5 in the reference posture. T _BODY-CAM and T _CAM-BODY are obtained based on the relationship between the position and orientation of the imaging device 5 and the main body 11a. When the orientation of the imaging device 5 with respect to the main body 11a changes, T _BODY-CAM and T _CAM-BODY are further measured between the imaging device 5 and the main body 11a at the measurement time (t + Δti) and the imaging time t, respectively. It is calculated | required by measuring the relationship of direction by an appropriate means. _{T EARTH-BODY (t + Δti} ) is in the stationary coordinate system _{S S,} it is determined based on the position and orientation of the body 11a at measurement time (t + Δti). T _BODY-EARTH is in the still coordinate system _{S S,} is determined based on the position and orientation of the body 11a in the imaging time t.

Preferably, the coordinate transformations (T _{EARTH-BODY (t + Δti)} and T _BODY-EARTH ) of (4) and (5) are performed by reflecting the orientation of the main body 11a with respect to the ground (ground or horizontal plane). In this case, an appropriate sensor for measuring the orientation of the main body 11a with respect to the ground (for example, a horizontal plane) may be provided in the main body 11a, and the measured value of the orientation of the main body 11a with respect to the ground is calculated by T _{EARTH-BODY (t + Δti)} and Reflected in T _BODY-EARTH .
Thereby, the coordinate conversion of (4) and (5) can be performed in consideration of the position and orientation of the main body 11a with respect to the ground.

  Step S74 will be described in more detail.

In step S74, the image data (first image or second image) is composed of a large number of pixels arranged in a two-dimensional coordinate system. In this example, the coordinate axes of this two-dimensional coordinate system are the u-axis and the v-axis that are orthogonal to each other, and the position of each pixel k (k indicates the number of the pixel) in this two-dimensional coordinate system is represented by the u-coordinate value. It is represented by u (k) and v coordinate value v (k). In this example, the camera coordinate system SC _{, t} is a three-dimensional coordinate system having an x-axis, a y-axis, and a z-axis that are orthogonal to each other. In this three-dimensional coordinate system, the measurement position corresponding to the pixel k is represented by an x coordinate value x (k), a y coordinate value y (k), and a z coordinate value z (k).

In this case, the coordinate values u (k), v (k) indicating the positions of the pixels arranged in the two-dimensional coordinate system, and the coordinate values x (k), y (k), z (k) of the camera coordinate system. ) Is associated with the following equation.

u (k) = − f _u y (k) / x (k) + C _u

v (k) = − f _v z (k) / x (k) + C _v

Here, f _u and f _v are parameters related to the focal length of the imaging device 5, and C _u and C _v are parameters related to the optical axis of the imaging device 5.

  In step S8, the data processing device 9 converts the coordinate value specified as the leaf position in step S74 for each measurement time into a coordinate value of a stationary coordinate system fixed on the ground, and the converted coordinate value is converted into the coordinate value. Record as leaf position. Here, the stationary coordinate system fixed on the ground is an orthogonal coordinate system having an x-axis and a y-axis that are orthogonal to each other in the horizontal direction, and a z-axis that is in the vertical direction. When converting to a two-dimensional coordinate system based on a bird's eye view, the position of the leaf in the two-dimensional coordinate system is indicated by the x coordinate value and the y coordinate value of the position specified as the leaf position in the orthogonal coordinate system.

  Hereinafter, the effect by embodiment of this invention is demonstrated.

FIG. 5 is an explanatory diagram of effects according to the embodiment of the present invention.
As shown in FIG. 5A, it is assumed that the camera is placed above the ground surface and the leaves of the plant are imaged obliquely downward. In this case, when a bird's-eye view showing the distribution of leaf positions is created based on an image obtained by imaging, a plant distribution indicated by diagonal lines in FIG. 5B is obtained. The horizontal length of this distribution is La, which is larger than the actual length Lb.
On the other hand, in the embodiment of the present invention, the measurement direction and the imaging direction are the same as those of the camera of FIG. 5A, but the length of the leaf position distribution in the horizontal direction can be obtained with high accuracy. . In the present embodiment, measurement by the distance sensor 3 and imaging by the imaging device 5 are performed in a direction facing obliquely downward as in the camera of FIG. However, in this embodiment, since the three-dimensional position can be measured by the distance sensor 3, the plant distribution indicated by the oblique lines in FIG. 5C is obtained. The horizontal length of this distribution matches the actual length Lb with high accuracy.
For example, when the imaging device 5 (camera) is located at a height of 2.2 m from the ground, the height of the plant is 0.2 m from the ground, and the horizontal distance from the imaging device 5 to the plant is 10 m. It becomes as follows. In FIG. 5B, the error La-Lb is 1 m. On the other hand, in FIG. 5C of the embodiment of the present invention, the error La−Lb does not occur, and even if an error occurs, the error is a small distance measurement error (for example, several cm) by the distance sensor 3. Degree).

Moreover, for each measurement time, the coordinate values of the sensor coordinate system S _{L0, t + Δti} are fixed to the imaging device 5 at the imaging time based on the amount of change in the direction and position of the moving body between the measurement time and the imaging time. Since the camera coordinate system SC _{, t} is converted into the coordinate value, the position of the leaf existing on the ground can be specified with high accuracy. Therefore, the distribution of leaves existing on the ground can be obtained with high accuracy.

Further, it is possible to obtain the leaf distribution with high accuracy by avoiding errors due to changes in the position and orientation of the moving body 11 with respect to the ground (the ground surface or the horizontal surface) as follows.
For example, the distance sensor 3 measures at 100 Hz (that is, measures the distance to 100 measurement points in 1 second), and the imaging device 5 captures images at 10 Hz (that is, acquires 10 images in 1 second). ), The time difference between the measurement time by the distance sensor 3 and the imaging time by the imaging device 5 can be 90 milliseconds.
In this case, when the moving body 11 is moving at a speed of 50 km / sec during the above-described time difference, the moving distance 1.25 m of the moving body 11 becomes an error. On the other hand, in the above-described embodiment, this error can be avoided by the coordinate transformations (1) to (6).
In addition, when the orientation of the main body 11a changes with respect to the ground or the horizontal plane at a speed of 10 degrees / second during the above-described time difference, the amount of change in orientation between the distance sensor 3 and the imaging device 5 is 0. Nine degrees is an error. On the other hand, in the above-described embodiment, this error can be avoided by the coordinate transformations (1) to (6).

  The present invention is not limited to the above-described embodiment, and various changes can be made without departing from the scope of the present invention. For example, any one of the following modification examples 1 to 4 may be employed alone, or modification examples 1 to 4 may be combined in an appropriate manner. In this case, the points not described below are the same as described above.

(Modification 1)
At the start of the method of FIG. 3, step S1 may be started after step S3 is started. In this case, if the determination in step S5 is “Yes” by performing step S3, a plurality of image data obtained in step S3 and the next step S3 after step S3 are performed. Steps S6 to S8 may be performed on the measurement data obtained in step S1 that has been performed once.
Instead, when starting step S3 after starting step S3 at the start of the method of FIG. 3, if the determination in step S5 is “yes” by performing step S3, the result obtained in step S3 is obtained. Steps S6 to S8 may be performed on the image data and the measurement data. This measurement data is obtained from an intermediate time (intermediate time point) between step S3 where the image data is obtained and step S3 performed immediately before step S3, and step S3 next to step S3 where the image data is obtained. The data obtained in step S1 performed a plurality of times until the intermediate time (intermediate time).

(Modification 2)
In the above description, the distance sensor 3 uses a plane including the base point Pa as a scanning plane, and emits laser light from the base point Pa in each measurement direction included in the scan plane, thereby causing a plant or other object present in each measurement direction. Although the distance to each measurement point on the object was measured, the present invention is not limited to this. In other words, according to the present invention, the distance measuring method by the distance sensor 3 is not limited to the above-described method (line scanning in which laser light is emitted in a plurality of directions within the scanning plane), and the distance sensor 3 is used in each measuring direction. It is only necessary to measure the distance to each measurement point on the existing plant or other object. In step S1, the laser beam may be emitted in one measurement direction instead of a plurality of measurement directions, and the process may proceed to step S2.

(Modification 3)
In the above description, the distance sensor is configured so that the direction of the scanning surface with respect to the main body 11a of the moving body 11 is different between a plurality of measurement times (for example, between a plurality of steps S1) while the moving body 11 is moving. However, the present invention is not limited to this. That is, according to the present invention, the direction of the scanning surface with respect to the main body 11a of the moving body 11 may be the same between a plurality of measurement times (for example, between a plurality of steps S1). In this case, the sensor coordinate system is maintained at the reference sensor coordinate system S _{L0, t + Δti,} and the coordinate conversion of (1) is omitted.

(Modification 4)
In the above description, the coordinate transformations (1) to (6) are performed in the coordinate transformation of FIG. 4, but the coordinate transformations (2) and (3) may be omitted. In this case, the coordinate value of the reference sensor coordinate system S _{L0, t + Δti} obtained by the coordinate transformation of (1) is fixed to the main body coordinate system 11a of the moving body 11 at the target measurement time (t + Δti). The coordinates are directly converted to the coordinate values of _{SB, t + Δti} , and the coordinates conversion of (4) to (6) is performed.
In the above description, the coordinate transformations (1) to (6) are performed in the coordinate transformation of FIG. 4, but the coordinate transformations (2) to (5) may be omitted. In this case, the coordinate value of the reference sensor coordinate system S _{L0, t + Δti} obtained by the coordinate transformation of (1) is fixed to the main body coordinate system 11a of the moving body 11 at the target measurement time (t + Δti). S _B, to the coordinate values of the _{t +? ti,} direct and coordinate transformation, the body coordinate system _{S B,} the coordinate values of _{t +? ti,} the body coordinate system _{S B} which is fixed to the body 11a of the movable body 11 in the imaging time _t, the _t The coordinate conversion is directly performed on the coordinate value, and the coordinate conversion of (6) is performed.
In the above description, the coordinate transformations (1) to (6) are performed in the coordinate transformation of FIG. 4, but the coordinate transformations (1) to (3) may be omitted. In this case, when the sensor coordinate system S _{L1, t + Δti} at the target measurement time (t + Δti) does not match the reference sensor coordinate system S _{L0, t + Δti} , the sensor coordinate system S _L1, at the target measurement time (t + Δti) _{. The} coordinate value of _{t + Δti} is directly transformed into the coordinate value of the main body coordinate system _{SB, t + Δti} fixed to the main body 11a of the moving body 11 at the target measurement time (t + Δti), and the above (4) to (4) 6) Coordinate conversion is performed. On the other hand, if the sensor coordinate system _{S L1, t +? Ti} of the measurement time of interest (t +? Ti) coincides with the reference sensor coordinate system _{S L0, t +? Ti,} the reference sensor coordinate system _{S L0} in the subject of measurement time (t +? Ti) _{, T + Δti} , and the coordinate value is directly converted into the coordinate values of the main body coordinate system _{SB, t + Δti} fixed to the main body 11a of the moving body 11 at the target measurement time (t + Δti), The coordinate conversion of (6) is performed.
In the above description, the coordinate transformations (1) to (6) are performed in the coordinate transformation of FIG. 4, but the coordinate transformations (1) to (5) may be omitted. In this case, when the sensor coordinate system S _{L1, t + Δti} at the target measurement time (t + Δti) does not match the reference sensor coordinate system S _{L0, t + Δti} , the sensor coordinate system S _L1, at the target measurement time (t + Δti) _. the coordinate values of the _{t +? ti,} measurement time of interest (t +? ti) body coordinate system fixed to the body 11a of the movable body 11 in the _{S B,} to the coordinate values of the _{t +? ti,} direct and coordinate transformation, the body coordinate system _{S B ,} the coordinate values of _{t +? ti,} the body coordinate system S _B which is fixed to the body 11a of the movable body 11 in the imaging time _t, the coordinate values of _t, direct and coordinate transformation, performs the coordinate transformation of the (6). On the other hand, if the sensor coordinate system _{S L1, t +? Ti} of the measurement time of interest (t +? Ti) coincides with the reference sensor coordinate system _{S L0, t +? Ti,} the reference sensor coordinate system _{S L0} in the subject of measurement time (t +? Ti) _{, T + Δti} , the coordinate value is directly converted into the coordinate value of the main body coordinate system _{SB, t + Δti} fixed to the main body 11a of the moving body 11 at the target measurement time (t + Δti), and the main body coordinate system S The coordinate value of _{B, t + Δti} is directly converted into the coordinate value of the main body coordinate system S _{B, t} fixed to the main body 11a of the moving body 11 at the imaging time t, and the coordinate conversion of (6) is performed.

3 distance sensor, 5 imaging device, 5a first imaging unit, 5b second imaging unit, 7 motion amount measuring device, 7a measuring unit, 7b computing unit, 9 data processing device, 10 plant position measuring device, 11 moving object, 11a Main body of moving body, 13 1st filter, 15 2nd filter, 17, 19

Claims (4)

A plant position measuring device that measures the position of a plant leaf on the ground,
A distance sensor that measures the distance to each measurement point on a plant or other object that exists in each measurement direction by emitting laser light in each measurement direction from a base point set on the moving body;
An imaging device that images an area including each of the measurement points;
The distance sensor and the imaging device are provided on a moving body that moves on the ground.
While moving the moving body, the distance sensor measures the distance at multiple measurement times,
Furthermore, an operation amount measuring device for determining the amount of change in the direction and position of the moving body between the imaging time and each measurement time;
A data processing device for determining a leaf position based on a distance and a measurement direction measured by a distance sensor, an image obtained by an imaging device, and the amount of change obtained by an operation amount measuring device;
The data processing device
(A) Based on the distance to the measurement point measured by the distance sensor at the measurement time and the measurement direction, the position of the measurement point is represented by a coordinate value of a sensor coordinate system fixed to the distance sensor,
(B) Based on the amount of change between the imaging time and the measurement time, the coordinate value of the sensor coordinate system is converted into a coordinate value of a camera coordinate system fixed to the imaging device at the imaging time,
(C) determining whether the converted coordinate value corresponds to a pixel indicating a leaf in an image obtained by the imaging device;
Converting the coordinate value determined to correspond to a pixel indicating a leaf in (C) to a coordinate value of a stationary coordinate system fixed on the ground, and recording the converted coordinate value as a leaf position; A plant position measuring device. The posture of the distance sensor with respect to the main body of the moving body is changed so that the posture of the sensor coordinate system with respect to the main body of the moving body is different between a plurality of measurement times.
In (B) above, the data processing device
If the sensor coordinate system when the distance sensor is in the reference posture is the reference sensor coordinate system, and the sensor coordinate system at the measurement time does not match the reference sensor coordinate system, the coordinate value of the sensor coordinate system is calculated as the distance. Convert to the coordinate value of the body coordinate system fixed to the body of the moving body at the measurement time of the sensor,
The coordinate value of the main body coordinate system is converted into the coordinate value of the main body coordinate system fixed to the main body of the moving body at the imaging time of the imaging device,
The plant position measuring device according to claim 1, wherein the coordinate value of the main body coordinate system is converted into a coordinate value of a camera coordinate system fixed to the imaging device at the imaging time. In (B) above, the data processing device
The coordinate value of the sensor coordinate system is converted into the coordinate value of the main body coordinate system fixed to the main body of the moving body at the measurement time,
Convert the coordinate value of the main body coordinate system to the coordinate value of the main body coordinate system fixed to the main body of the moving body at the imaging time,
The plant position measuring device according to claim 1, wherein the coordinate value of the main body coordinate system is converted into a coordinate value of a camera coordinate system fixed to the imaging device at the imaging time. A plant position measurement method for measuring the position of a plant leaf on the ground,
A distance sensor that measures the distance to each measurement point on a plant or other object that exists in each measurement direction by emitting laser light in each measurement direction from a base point set on the moving body;
An imaging device that images the area including each of the measurement points, and a moving body that moves on the ground,
While moving the moving body, the distance sensor performs distance measurement at a plurality of measurement times, and the imaging device performs imaging.
Find the amount of change in the direction and position of the moving body between the imaging time and each measurement time,
For each measurement time,
(A) Based on the distance to the measurement point measured by the distance sensor at the measurement time and the measurement direction, the position of the measurement point is represented by a coordinate value of a sensor coordinate system fixed to the distance sensor,
(B) Based on the amount of change between the imaging time and the measurement time, the coordinate value of the sensor coordinate system is converted into a coordinate value of a camera coordinate system fixed to the imaging device at the imaging time,
(C) determining whether the converted coordinate value corresponds to a pixel indicating a leaf in an image obtained by the imaging device;
Converting the coordinate value determined to correspond to a pixel indicating a leaf in (C) to a coordinate value of a stationary coordinate system fixed on the ground, and recording the converted coordinate value as a leaf position; A plant position measuring method as a feature.

Images