AU2007201452A1 - Methods and apparatus for measuring geometrical parameters of foliage - Google Patents

Description

k e, Cc, P/00/011 Regulation 3.2

AUSTRALIA

Patents Act 1990

ORIGINAL

COMPLETE SPECIFICATION STANDARD PATENT Invention Title: "METHODS AND APPARATUS FOR MEASURING GEOMETRICAL PARAMETERS OF FOLIAGE" The following statement is a full description of this invention, including the best method of performing it known to me/us:

TITLE

k METHODS AND APPARATUS FOR MEASURING GEOMETRICAL PARAMETERS OF FOLIAGE t'q C 5 FIELD OF THE INVENTION The present invention relates to methods and apparatus for measuring geometrical parameters of foliage.

BACKGROUND TO THE INVENTION In an age of water shortages and increasing populations, the need to maximise plant yields and optimise water usage is greater than ever. Irrigation is one means of controlling the amount, location and frequency of water application. Monitoring plant attributes is one means of diagnosing plant conditions, which can be treated to maintain the health of the plant.

Large mobile irrigation machines (LMIMs) apply water directly to crop plants at high temporal frequency and variable-rate LMIMs adjust the water application in response to real-time irrigation requirements. However, water application needs to be carefully controlled because plants can suffer from water stress caused by insufficient soil moisture. Water stress can lead to poor plant health and growth, increased susceptibility to disease and reduced tolerance to insects feeding on the plant. Excessive watering is wasteful and can lead to other problems with plant health.

A number of plant attributes may be used to infer water stress in plants, such as cotton. For example, for cotton, nodes above white flowers and distances between fourth and fifth mainstem nodes can be measured because 2 these attributes indicate a balance between vegetative and reproductive growth that is required during crop flowering to optimise cotton yield.

The desire for on-the-go measurement of physical plant characteristics Sengages machine vision as a potential sensing technology for determining cotton plant water stress and other conditions in the same and other plants. Physical plant attributes which have been successfully measured on-the-go using (machine vision include plant height and biomass. These attributes require discrimination of plant and non-plant elements, but not specifically plant structures such as stems, petioles and leaves.

Hence, it is desirable to determine a method and/or an apparatus for measuring geometrical parameters of foliage because such parameters can be used to maintain plant health, maximize plant yield and optimize water usage.

Preferably, such a method and/or apparatus should be capable of being utilised on-the-go.

In this specification, the terms "comprises", "comprising" or similar terms are intended to mean a non-exclusive inclusion, such that a method, system or apparatus that comprises a list of elements does not include those elements solely, but may well include other elements not listed.

SUMMARY OF THE INVENTION In one form, although it need not be the only or indeed the broadest form, the invention resides in an apparatus for measuring geometrical parameters of foliage comprising: an image capturing device; and a transparent surface for contacting the foliage, said transparent surface O mounted to, and a fixed distance from, the image capturing device; k wherein foliage contacting the transparent surface is in a field of view of the image capturing device.

0 Suitably, the image capturing device is a still camera or a video camera.

N 5 Suitably, the image capturing device is mounted within an enclosure comprising the transparent surface.

(Suitably, the enclosure comprises a curved surface to guide foliage past 0the enclosure.

(N

In another form, although again not necessarily the broadest form, the invention resides in a method of measuring geometrical parameters of foliage including: extracting strong linear components from an image of the foliage captured by an image capturing device; defining pairs of the extracted lines that are substantially parallel as branch segments of the foliage; determining a mainstem of the foliage from a sequence of the branch segments; constructing branches of the foliage from linked branch segments; projecting the branches to the mainstem; estimating branch intersections where the branches intersect the mainstem; and filtering the branch intersections to identify one or more nodes of the foliage.

In another form, although again not necessarily the broadest form, the invention resides in a method of measuring geometrical parameters of foliage including: detecting significant line segments in an image captured by an image capturing device; identifying a set of line segments as a main stem if the line segments are approximately collinear and if collectively the line segments have an incline within a threshold angle of vertical; and identifying branch junctions or nodes on the main stem.

Further features of the present invention will become apparent from the following detailed description.

BRIEF DESCRIPTION OF THE DRAWINGS By way of example only, preferred embodiments of the invention will be described more fully hereinafter with reference to the accompanying drawings, wherein: FIG 1 is a schematic side view of an apparatus for measuring geometrical parameters of foliage according to an embodiment of the present invention; FIGS 2(a) and 2(b) are plan views of the apparatus of FIG 1 measuring geometrical parameters of foliage; FIG 3 is an image of foliage captured with the apparatus of FIG 1; FIG 4 is a flow diagram illustrating a method of for measuring geometrical parameters of foliage; and FIG 5 is a schematic plan view of an apparatus for measuring geometrical parameters of foliage according to an alternative embodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION A machine vision solution for measurement of internode length, and in the case of cotton, nodes above white flower, requires image processing algorithms to identify cotton plant features (such as the mainstem and node positions), and a suitable camera platform to collect imagery of cotton plants in the field.

However, it should be appreciated that the present invention can be utilised for plants other than cotton and for diagnosing plant conditions other than water stress.

Referring to FIG 1, there is provided an apparatus 100 for measuring geometrical parameters of foliage 110 in accordance with embodiments of the present invention. The apparatus 100 comprises an image capturing device 120, such as a still camera or a video camera and a transparent surface 130 for contacting the foliage 110. The transparent surface 130 is mounted to, and is a fixed distance from, the image capturing device 120 such that foliage 110 contacting the transparent surface 130 is in a field of view of the image capturing device 120.

According to the embodiment shown in FIG 1, the image capturing device 120 is mounted within an enclosure 140 comprising the transparent surface 130 and the enclosure comprises a curved surface 150 to guide foliage 110 past the enclosure 140. According to one embodiment, the enclosure 140 is made from fibreglass and the transparent surface 130 is in the form of a glass panel. The transparent surface 130 can include indicia to aid with measurements.

In an alternative embodiment, an elongate member, such as a formed bar or rod 160 can be used with the transparent surface 130 mounted at, or toward, one end and the image capturing device 120 mounted at, or toward, an opposite end. The elongate member 160 can also be formed to incorporate the curved surface 150 to guide foliage 110 past the apparatus.

According to one embodiment, the image capturing device 120 is a Prosilica EC750C machine vision camera fitted with an optical component in the form of a varifocal, manual iris Tamron lens, which allows for adjustment of optical parameters such as depth of field. The optical components can be selected to obtain a small depth of field for the image capturing device 120, hence simplifying discrimination of foliage hitting the transparent surface 130 and foliage a distance from the surface 130. Other optical components that can be used include optical filters coupled to the image capturing device 120 to improve discrimination of plant features from other objects in the field of view. Natural lighting and/or lighting supplied from within the enclosure 140 can be used.

Some embodiments of the apparatus 100 include a canopy extending above the image capturing device 120 and the transparent surface 130 to reduce the level of natural lighting incident on the transparent surface 130.

In another embodiment, the image capturing device 120 is a Sony DCR- TRV19E camcorder. However, it will be appreciated that other models and versions of image capturing device 120 can be utilised.

Both embodiments of the image capturing device 120 were trialed on four varieties of cotton (Sicot 80B, Sicot 71B, Sicot 289B and DP 408B) during crop flowering in the cotton growing season. Both embodiments of the image capturing device 120 were used for relative performance comparison and were used to capture video footage of cotton plants with up to 25 frames per second being captured by each device.

b The apparatus 100 is moved through the plants, such as a crop as k indicated by the arrow in FIG 1. When the foliage 110 hits the transparent surface 130, the foliage 110 is at a fixed known distance from the image 0 capturing device 120 and hence reliable geometrical data about the foliage 110 can be obtained. Foliage parameters that can be measured include, but are not limited to, distance between nodes, stem diameter, leaf area, plant height and plant spacing.

According to some embodiments, the apparatus 100 further comprises a powered module to propel the apparatus through the crop. The powered module can comprise a power source, such as motor, coupled to wheels or tracks. In some embodiments, the apparatus 100 can be mounted to a powered module, such as a tractor or to a mobile irrigation machine, such as a LMIM.

With reference to FIGS 2(a) and three replicates of multiple orientations and approach angles of the apparatus 100 were conducted to test the repeatability of the method and apparatus of the present invention. As shown in FIG the apparatus 100 approached the shaded area of interest with the transparent surface 130 substantially perpendicular to the directions of motion shown by the arrows at approach angles of 00, 1350, 1800 and 3150. As shown in FIG the apparatus 100 approached the shaded area of interest with the transparent surface 130 approximately 450 to the directions of motion shown by the arrows at approach angles of 0' and 1800, i.e. a yaw angle of 450 However, other approach angles and angles of the transparent surface 130 to the direction of motion can be employed.

A yaw angle of 00 produces a bending action in the plant's main stem and therefore the plant is more likely to flatten against the transparent surface 130. A yaw angle of 45°causes a shearing motion between the plant and transparent surface 130 that is not conducive to flattening the plant against the transparent surface 130. Hence, a yaw angle of 00 is preferred.

Different approach speeds and illumination configurations were also tested. Approach speeds of 0.1, 0.2, 0.3 and 0.4 m/s were used and all yielded good results. According to some embodiments, suitable illumination configurations comprise one or more light sources in the form of a linear array of near infrared or white LEDs affixed by any suitable means to each vertical and horizontal edge of the transparent surface 130 having a power consumption of about 30 Watts and three 10 Watt halogen globes spaced evenly on each vertical and horizontal edge of the transparent surface 130 with a diffuser board.

Preferences for artificial lighting include that the light sources be mounted between the image capturing device 120 and the transparent surface 130 for localised, uniform illumination of the region immediately behind the transparent surface 130. In this configuration, there is uniform illumination of the plant contacting the transparent surface 130 and preferably not of plants a further distance from the transparent surface 130. Restricted natural lighting is desirable, e.g. at night time or by use of the canopy above the transparent surface 130.

For each of the plants targeted by the apparatus 100, manual measurements were made of the top five internode lengths, plant height, stem diameters of the top five internodes, nodes above white flower, retention of firstposition fruit on the top five nodes, number of fruiting branches and plant spacing. Other measurements can be determined for other plants or for identifying other plant conditions.

A stylised sample image from the Sony camcorder is shown in FIG 3. By visual inspection of the acquired image, the apparatus 100 effectively isolates the target plant attributes and enables distance measurements of plant features.

Image processing algorithms as described hereinafter based on edge detection and plant feature geometry, are applied to the images to determine plant attributes for comparison with the manual measurements taken. As expected, the orientation of leaves obscured essential points of geometry within some images such that the required measurements could not be deduced. However, this problem is overcome because of the multiple images available (up to 25 per second).

Natural flexure in the foliage adjoining the plant's main stem may sometimes prevent the plant's main stem from flattening against the transparent surface 130. In cases where the plant does not flatten against the transparent surface 130, a simple geometric correction factor for foliage measurements can be calculated with the alternative embodiment of the apparatus 100 shown in FIG 5. According to alternative embodiments, the apparatus 100 of the present invention further comprises one or more secondary image capturing devices 500 in addition to the primary image capturing device 120. The secondary image capturing devices 500 are low-resolution cameras mounted by any suitable means to the front of the transparent surface 130. The secondary image capturing devices 500 are mounted with an optical axis parallel to the transparent surface 130 to measure the distance d between the plant's main stem and the transparent surface 130.

Methods 400 of measuring geometrical parameters of foliage in accordance with embodiments of the present invention will now be described with reference to the flow diagram in FIG 4.

According to one embodiment, the method 400 includes at 410 generating an edge map for an image of the plant captured by the image capturing device 120. The edge map can be generated by a mask operator or edge following routine. Alternatively, the edge map can be generated by an automatic thresholding technique which separates plant pixels from unnecessary background pixels, from which a plant outline may be derived.

The method includes at 420 extracting strong linear components or lines from the image of the foliage. This can be achieved by accumulating groups of pixels based on similar slopes and position or by using a Hough transform.

At 430, the method includes defining pairs of the extracted lines that are substantially parallel as branch segments of the foliage. This can be achieved by determining parallelism within a specified threshold, identifying start points and endpoints coinciding within a threshold pixel radius or by identifying a transverse separation of no more than a threshold number of pixels.

The method includes at 440 determining a mainstem of the foliage from a sequence of the branch segments. The mainstem can be constructed by finding a path of branch segments from the bottom of the image to the top, where for each y-position the mainstem features the branch segment which has one or more characteristics including, but not limited to, the maximum width of all the branches at the same y-position, a specified incline angle, a slope within a specified threshold of the previous mainstem element slope, and/or a starting position within a threshold pixel radius from the endpoint of the previous mainstem element. Where the mainstem is a series of unconnected elements, the method includes joining the mainstem elements with straight lines.

O With further reference to FIG 4, the method includes at 450 constructing branches of the foliage from linked branch segments. This can be achieved by finding a path of branch segments from the mainstem to the extremities of the image. A branch segment is added to a branch if the branch segment has one N 5 or more characteristics including, but not limited to, a slope within a specified threshold of the previous branch element slope and/or a starting position within a N threshold pixel radius from the endpoint of the previous branch element.

OAt 460, the method includes projecting the branches to the mainstem using a straight line or a curve. The method includes at 470 estimating branch intersections where the branches intersect the mainstem. At 480, the method includes filtering the branch intersections to identify one or more nodes of the foliage. Nodes are identified as those branch intersections which have one or more characteristics including, but not limited to, a position within a specified pixel radius from neighbouring nodes and/or coincidence with a localised increase in width of the mainstem.

If the plant structure is incomplete, the aforementioned method can be attempted on another frame. Data from more than one frame can be combined to obtain the required geographical parameters. For example, the method can include repeating the process on sequential frames. Node trajectories are formed by tracking nodes across sequential frames. Nodes that cannot be tracked across sequential frames are discarded as false positives. The maximum distance between node trajectories corresponds to the true distance between mainstem nodes.

According to other embodiments, the method of measuring geometrical parameters of foliage includes detecting significant line segments in the image captured by the image capturing device 120 using one or more of the following techniques: edge detection, line fitting or line linking. The method includes identifying a set of line segments as the main stem if the line segments are approximately collinear and if collectively the line segments have an incline within a threshold angle of vertical. The method further includes identifying branch junctions or nodes on the main stem. This can be achieved by using shape, edge and/or colour properties of the image in the vicinity of the main stem to identify localised increases in width/line intersections along the main stem.

Alternatively, this can be achieved by identifying a set of line segments as a subtending branch if the line segments are approximately collinear and within a threshold distance of the main stem and then projecting the subtending branch onto the main stem to estimate node position.

Hence, the method and apparatus of the present invention thus provides a solution to measuring geometrical parameters of foliage. The apparatus comprising an image capturing device 120 and a transparent surface 130 at a fixed distance from the image capturing device 120 that moves within the crop canopy can be used to automatically and accurately measure plant attributes, particularly the top five internode lengths during the crop flowering stage. The apparatus can be mounted to and integrated with a mobile irrigation device, such as an overhead irrigation machine, that enables the modification of irrigation volumes in real-time in response to measured parameters and diagnosed water requirements. Hence, water is used only as and when needed to maintain healthy crops and maximize yields.

Throughout the specification the aim has been to describe the invention without limiting the invention to any one embodiment or specific collection of 13 features. Persons skilled in the relevant art may realize variations from the specific embodiments that will nonetheless fall within the scope of the invention.

Claims (27)

1. An apparatus for measuring geometrical parameters of foliage comprising: an image capturing device; and a transparent surface for contacting the foliage, said transparent surface mounted to, and a fixed distance from, the image capturing device; wherein foliage contacting the transparent surface is in a field of view of the image capturing device.

2. The apparatus of claim 1, wherein the image capturing device is a still camera or a video camera.

3. The apparatus of claim 1 or 2, wherein the image capturing device is mounted within an enclosure comprising the transparent surface.

4. The apparatus of claim 3, wherein the enclosure comprises a curved surface to guide foliage past the enclosure.

The apparatus of claim 1 or 2, wherein the image capturing device is mounted at or toward one end of an elongate member and the transparent surface is mounted at or toward an opposite end of the elongate member.

6. The apparatus of any preceding claim, further comprising a canopy coupled to the image capturing device and/or the transparent surface to limit a level of natural light incident on the transparent surface. O

7. The apparatus of any preceding claim, further comprising a lens and/or an C optical filter coupled to the image capturing device to improve discrimination of features of the foliage from other objects in the field of view.

8. The apparatus of any preceding claim, further comprising one or more light _sources mounted to one or more edges of the transparent surface.

9. The apparatus of any preceding claim, further comprising one or more secondary image capturing devices mounted to, and in front of, the transparent surface.

The apparatus of any preceding claim, further comprising a powered module coupled to the image capturing device and/or the transparent surface to propel the apparatus through the foliage.

11. The apparatus of claim 10, wherein the powered module is a mobile irrigation machine.

12. A method of measuring geometrical parameters of foliage including: extracting strong linear components from an image of the foliage captured by an image capturing device; defining pairs of the extracted lines that are substantially parallel as branch segments of the foliage; determining a mainstem of the foliage from a sequence of the branch segments; O constructing branches of the foliage from linked branch segments; k projecting the branches to the mainstem; estimating branch intersections where the branches intersect the Smainstem; and filtering the branch intersections to identify one or more nodes of the _foliage.

13. The method of claim 12, further including initially generating an edge map for the image of the foliage using one of the following: a mask operator; an edge following routine; an automatic thresholding technique to separate foliage pixels from background pixels to derive a foliage outline.

14. The method of claim 12 or 13, wherein defining pairs of extracted lines as branch segments includes accumulating groups of pixels based on similar slopes and position or using a Hough transform.

The method of any of claims 12-14, wherein determining a mainstem includes finding a path of branch segments, each branch segment in the path having one or more of the following characteristics: the maximum width of all the branches at the same y-position; a specified incline angle; a slope within a specified threshold of the previous mainstem element slope; a starting position within a threshold pixel radius from the endpoint of the previous mainstem element.

16. The method of any of claims 12-15, wherein constructing branches includes finding a path of branch segments from the mainstem to the extremities of the image.

17. The method of claim 16, further including adding a branch segment to a branch if the branch segment has one or more of the following characteristics: a slope within a specified threshold of a slope of the previous branch segment; a starting position within a threshold pixel radius from the endpoint of the previous branch segment.

18. The method of any of claims 12-17, wherein filtering the branch intersections includes identifying nodes as those branch intersections which have one or more of the following characteristics: a position within a specified pixel radius from neighbouring nodes; coincidence with a localised increase in width of the mainstem.

19. The method of any of claims 12-18, further including repeating the method on a different image if the required geometrical parameters of the foliage are not obtainable.

20. The method of any of claims 12-18, further including repeating the method on images from sequential frames.

21. The method of claim 20, further including tracking nodes across sequential frames. O

22. The method of claim 21, further including discarding nodes as false k positives that cannot be tracked across sequential frames. C,

23. A method of measuring geometrical parameters of foliage including: detecting significant line segments in an image captured by an image capturing device; Sidentifying a set of line segments as a main stem if the line segments are approximately collinear and if collectively the line segments have an incline within a threshold angle of vertical; and identifying branch junctions or nodes on the main stem.

24. The method of claim 23, wherein detecting significant line segments includes one or more of the following techniques: edge detection, line fitting, line linking.

The method of claim 23, wherein identifying branch junctions or nodes includes identifying localised increases in width/line intersections along the main stem using shape, edge and/or colour properties of the image in the vicinity of the main stem.

26. The method of claim 23, wherein identifying branch junctions or nodes includes identifying a set of line segments as a subtending branch if the line segments are approximately collinear and within a threshold distance of the main stem. 19 O

27. The method of claim 26, further including projecting the subtending branch Sonto the main stem to estimate node position. Dated this Third day of April 2007 In IF TECHNOLOGIES PTY LTD By their Patent Attorneys FISHER ADAMS KELLY