JP2006101816A - Method and apparatus for controlling steering - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a method for controlling steering, with which only one straight line is determined for the whole one line of seedlings in a visual field transplanted before one traveling and a transplanter is traveled in parallel with the straight line. <P>SOLUTION: The method for controlling steering has a step for imaging a planted seedling line by an imaging means attached to the transplanter and acquiring image information on the seedling line, a step for acquiring a binary image obtained by extracting the seedling line region from the image information, a step for subjecting the binary image to inverse perspective transformation to acquire the virtual horizontal surface image of the binarized seedling line region, a step for approximating the seedling line region by a straight line in the virtual horizontal surface image and a step for controlling steering on the basis of position information of the transplanter relative to the straight line. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

The present invention relates to a steering control method and apparatus for a traveling vehicle, and in a preferred example relates to a steering control method and apparatus for a transplanter (particularly, a rice transplanter).

Rice transplanters were developed in the 1960s and are currently being transplanted by more than 90% of paddy fields. In the early days, the walking type was the main, but now the riding type is the mainstream. In order to increase work efficiency, various improvements have been made, such as increasing the number of strips (2 to 10), increasing the working speed, and changing the driving method of the planting claws (from crank type to rotary type). Also, hydraulic height and rolling controls have been incorporated to keep the planting depth constant.

Rice planting is preferably performed in a straight line from the viewpoint of effective use of a field of a predetermined area, management work, and ease of harvesting. Usually, when planting rice, run for the next process while drawing the mark line on the mud. However, the mark line is not always clearly visible. In addition, although the paddy surface is flat, the cultivator on which the rice transplanter actually runs has many undulations. In addition, the tires always exert unpredictable forces due to mud resistance. For this reason, it is difficult for a skilled person to always run the rice transplanter straight in a changing situation. In addition, since there is a limit to the amount of seedlings that can be loaded onto the working part of the rice transplanter, workers must stop and replenish the mat seedlings once every few minutes. This is one factor that lowers work efficiency. In recent years, attempts have been made to increase field work efficiency by reducing the number of replenishment operations using mat seedlings that are 6 m longer than 6 m. However, in order to raise this long mat seedling, a new nursery facility and a rice transplanter are required, and it has not been widely spread.

Many researches have been conducted on the method of automatically driving a farm vehicle along a predetermined route in a farm field. Those using RTK-GPS, using geomagnetism, using an optical fiber gyro, using a total station, and using image processing. These may be used alone, but often use multiple sensors to compensate for their shortcomings. RTK-GPS and FOG systems are the most accurate and promising, but they are still expensive. In addition, there is a drawback that it takes time to survey the field in advance. The geomagnetic sensor has no drift error in terms of direction and is inexpensive, but is easily affected by the surrounding environment and cannot prevent the error. Although the system using the total station has high accuracy, it is not realistic in general farm work because the machine is expensive and preparation takes time.

Image processing has been expected to replace the human eye. Until now, both cameras and processing equipment have been expensive. However, with the widespread use of digital cameras and camera-equipped mobile phones, the price of the system is becoming cheaper. Although only the image processing cannot detect the absolute position and direction of the vehicle, it can accurately detect the distance from the crop in the field, which is an external index. This is a feature not found in other sensors. For this reason, image processing is often used in vehicles that perform management operations such as control and weeding. In the management work, the vehicle only needs to be able to travel along the plant line already existing in the field, and the travel locus does not necessarily have to be a straight line. On the other hand, when image processing is used for transplantation, the instantaneous error may be small, but the error is amplified for each stroke, and there is a great possibility that the path that was originally a straight line will be greatly curved. In particular, in paddy fields, there are many reflections of light on the surface of the water, which can cause errors. Chen (1997) and others proposed a method that uses image processing as a visual sensor for rice transplanters, but it was based on the assumption that the seedling rows to be referenced were arranged in a straight line. Also, no actual running experiment has been conducted. If the seedling line is completely in a straight line, the change in angle and distance calculated by image processing is only due to the movement of the rice transplanter. However, in the case of actual rice planting, the seedling line referred to is always curved. Therefore, it cannot be distinguished whether the change of those values is due to the curvature of the seedling row or due to the movement of the rice transplanter.

Patent Documents 1, 2, and 3 describe techniques for acquiring an image of a seedling row that has already been planted and performing steering control of the rice transplanter based on the acquired image. Patent Document 1 describes a technical means for linearly approximating a line segment connecting extracted areas extracted as areas corresponding to the colors of planted seedlings. In Patent Document 2, an image obtained by imaging a row of planted seedlings is binarized, the binarized image is linearly approximated, and the line segment obtained by the linear approximation and the traveling direction of the aircraft are set. The technical means for calculating the angle between the reference gland and the distance component between the reference line and the line segment and automatically steering the line by using the angle and distance component as steering information is described. ing. In Patent Literature 3, the imaging range of the camera provided in the traveling machine body is set so as to include a plurality of seedling rows, and only a specific seedling row is selected and displayed from the plurality of seedling rows photographed by the camera. So that the seedling row displayed on the display is displayed within a predetermined display area of the display when the traveling machine travels while maintaining an appropriate distance for seedling planting based on the existing seedlings. Technical means for setting the display area of the camera and display are described. However, all of these calculate a virtual straight line directly on the seedling row image acquired by the camera.

Conventionally, there has been a technique for detecting a straight line in an image by image processing and estimating an actual position by inverse perspective transformation. If the position and orientation of the camera with respect to a plane on which a straight line exists are known, it has been possible to estimate the position and direction of the straight line in the image as a result of the perspective transformation. However, this is only true under the condition that the straight line is completely straight. On the other hand, when fitting a straight line in a perspective-transformed image to objects that are arranged almost on a straight line on an uneven terrain, a straight line along the local column shape of the object in the vicinity of the camera is obtained. Detected. For this reason, the estimated straight line direction deviates from the ideal straight line direction. In transplanting paddy rice seedlings, the transplanter repeatedly travels following the seedling row transplanted one stroke before, so that there was a problem that the vibration of the running trajectory increased as the transplanting process progressed.

In addition, not only transplanters such as rice transplanters, but in mobile agricultural machines, a predetermined target is detected, and a vehicle (for example, a tiller, a leveling machine, a fertilizer, a seeder, a management machine, a harvester, It may be advantageous to control the steering of the livestock machinery. Furthermore, there are cases where it is desired to control the steering of the vehicle by detecting a predetermined target even in a moving vehicle other than the mobile agricultural machine.
JP 62-61509 A JP-A-2-131502 JP-A-9-154314

An object of the present invention is to define a single straight line for all rows of seedlings transplanted before one stroke in the field of view, and to run the transplanter in parallel with this.

Another object of the present invention is to apply a straight line indicating a certain direction as an initial condition to a route that is not originally a straight line using only image processing in steering control of a traveling vehicle, and a straight route toward that direction. The traveling vehicle travels along the line.

Still another object of the present invention is to perform a linear approximation to a target (for example, a seedling row) that is not a strict straight line but is to be regarded as a straight line.

The first technical means adopted by the present invention in order to achieve such a problem includes a step of capturing an already-planted seedling row by an imaging means provided in the transplanter to obtain image information of the seedling row, and the image information Obtaining a binary image in which a seedling row region is extracted from, obtaining a virtual ground plane image of the seedling row region binarized by inverse perspective transformation of the binary image, and the virtual ground plane Steering control method of transplanter having a step of approximating the seedling row region with a straight line in an image, and a step of performing steering control based on position information of the transplanter with respect to the straight line, and imaging provided in the transplanter Image input means for inputting image information of an existing seedling row imaged by the means, seedling row region extraction means for obtaining a binary image in which a seedling row region is extracted from the image information, and the binary image Temporary seedling region binarized by inverse perspective transformation And inverse perspective transformation means for obtaining a horizontal plane image, a detection device seedling row and a line extraction section approximating in the virtual ground plane image 該苗 row region in a straight line.

The second technical means adopted by the present invention includes a step of capturing an already-planted seedling row by an imaging means provided in the transplanter to obtain image information of the seedling row, and a seedling row region is extracted from the image information Obtaining a binary image; obtaining a virtual ground plane image of a seedling row region binarized by inverse perspective transformation of the binary image; and extracting the seedling row region in the virtual ground plane image A method for detecting a seedling row having a step of approximating with a straight line, an image input means for inputting image information of an existing seedling row imaged by an imaging means provided in the transplanter, and a seedling row region from the image information Seedling row region extracting means for acquiring the extracted binary image, reverse perspective conversion means for obtaining a virtual ground plane image of the seedling row region binarized by reverse perspective transformation of the binary image, Line extraction that approximates the seedling region in a virtual ground plane image with a straight line A detection device seedling row and a stage.

According to a third technical means employed by the present invention, there is provided a step of capturing a target extending in the traveling direction of the traveling vehicle by an imaging unit provided in the traveling vehicle and acquiring target image information, and a target region from the image information. Obtaining a binary image from which the binary image has been extracted, obtaining a virtual ground plane image of the target area by performing reverse perspective transformation on the binary image, and approximating the target area with a straight line in the virtual ground plane image And a steering control method for the traveling vehicle having a step of performing steering control based on position information of the traveling vehicle with respect to the straight line, and a traveling direction of the traveling vehicle imaged by an imaging means provided on the traveling vehicle. Image input means for inputting image information of a target to be output, and binary image acquisition means for acquiring a binary image obtained by extracting a target area from the image information , Based on reverse-perspective transformation of the binary image to obtain a virtual ground plane image of the target area, means for approximating the target area with a straight line in the virtual ground plane image, and position information of the traveling vehicle with respect to the straight line And a steering control device for a traveling vehicle having means for performing steering control.

According to a fourth technical means adopted by the present invention, the step of acquiring a target image information by imaging a target extending in the traveling direction of the traveling vehicle by an imaging unit provided in the traveling vehicle, and a target area from the image information Obtaining a binary image from which the binary image has been extracted, obtaining a virtual ground plane image of the target area binarized by inverse perspective transformation of the binary image, and the target area in the virtual ground plane image And a target detection method having a step of approximating a straight line, and an image input means for inputting image information of a target extending in the traveling direction of the traveling vehicle imaged by an imaging means provided on the traveling vehicle, and the image Target area extraction means for acquiring a binary image in which a target area is extracted from information, and a target that has been binarized by inverse perspective transformation of the binary image. And inverse perspective transformation means for obtaining a virtual ground plane image of the area, which is the target of the detection device and a straight line extracting means for approximating at the virtual locations, the planar image with a straight line the target area.

In the third technical means and the fourth technical means, in one aspect, the traveling vehicle is a mobile agricultural machine, the target is a crop row, a transplanter is a mobile agricultural device, and a seedling is a crop row. A column is illustrated. Further, even if the traveling vehicle is a mobile farm machine, the target is not limited to the crop line, and for example, the mobile farm machine may be steered using a straw as a target. Further, the traveling vehicle is not limited to a mobile agricultural machine, and for example, the traveling vehicle may be steered with a white line or a shoulder as a target.

In the technical means, in one preferable aspect, the imaging means images up to infinity of a target (for example, a seedling row). In this way, only one straight line can be defined for all the rows of seedlings, and the vehicle can be driven in parallel to this. Further, in order to capture an image up to infinity of the target, it is advantageous that the input image of the target is a so-called vertical image.

In the above-mentioned technical means, when the target has a main color of green as in a seedling row, in one preferred embodiment, the step of acquiring the binary image comprises converting an RGB color image of the seedling row to L ^* a ^* b ^* Convert to color system and binarize by setting a threshold to perceptual chromaticity b ^* .

In the technical means, in one preferred embodiment, the virtual horizontal plane image is set with a low horizontal resolution. By reducing the horizontal resolution of the target image, it is easy to extract an approximation of one straight line for the target even if the target is somewhat meandering or bent.

In the technical means, linear approximation of the target region (for example, seedling row region) is performed by a Hough transform or a least square method in a preferred embodiment. In addition, when performing linear approximation for a single row of seedlings transplanted one stroke before, linear approximation may be performed based on the areas of a plurality of seedling rows that have been transplanted including the seedling rows transplanted one stroke before good.

In one embodiment, the position information of the vehicle (transplanter) with respect to the straight line is an angle with respect to the straight line and / or a lateral displacement (offset). The angle of the vehicle with respect to the straight line is calculated as the angle between the reference line set in the traveling direction of the vehicle (or the imaging direction of the imaging means) and the detected straight line. The lateral shift amount (offset) is calculated as the distance between the detected straight line and the vehicle (direction orthogonal to the straight line). If the information about the angle with respect to the straight line and the amount of lateral deviation and the speed of the vehicle are known, the steering control of the vehicle can be performed.

According to the present invention, in particular, in a transplanter, a single straight line can be defined for all rows of seedlings transplanted before the one-stroke period that is in the field of view, and the vehicle can be driven in parallel with this. As a result, the disturbance of the seedling row due to the irregularity of the tiller in the previous stroke is corrected, and the steering control can be performed so as not to increase the vibration of the traveling locus but to decrease further. .

Further, in the steering control of the traveling vehicle, it is possible to apply a straight line indicating a certain direction as an initial condition to a route that is not originally a straight line, and to steer the traveling vehicle along the linear route toward the direction. it can.

Further, although it is not a strict straight line, a linear approximation can be performed on a target (for example, a seedling row) that is desired to be regarded as a straight line.

1 is a plan view of the rice transplanter, and the lower figure is a side view of the rice transplanter. The rice transplanter includes a traveling machine body 3 supported by a pair of left and right front wheels 1 and a rear wheel 2, and a front of the traveling machine body 3. This is a four-row riding rice transplanter having a power unit 4, a steering wheel 5, and a seedling supply device 6 and a planting device 7 provided behind the traveling machine body 3 via a link mechanism. An imaging means 9 is mounted on the frame of the seedling supply device 6 so as to be positioned above the left and right side bumpers 8. A driver's seat 10 is provided at the rear of the traveling machine body 3. Since the seedling supply device 6 is controlled in horizontal posture control and vertical movement position during traveling of the rice transplanter by a posture control mechanism including a float, the image pickup means 9 is provided in the seedling supply device 6 so that the image pickup means 9 is a traveling machine body. It is difficult to be affected by the attitude of 3.

The imaging means 9 is an RGB color TV camera, and is attached to both sides of the transplanting part (seedling supply device) of the rice transplanter by rotating it 90 degrees with the camera facing forward and a depression angle. By rotating the camera 90 degrees, an image of the seedling row can be obtained as vertical image information. The rice transplanter is equipped with a display (not shown), and image information captured by the camera is displayed on the display. When there is a cocoon on the left side of the vehicle, or when there is a target seedling row on the left side of the vehicle, the left camera is used, and in the opposite case, the right camera is used. The camera can be switched manually or automatically. Here, although two cameras are disposed on both sides of the seedling feeder of the rice transplanter, the number of cameras, the mounting position, and the imaging direction are not limited thereto. For example, two cameras may be disposed on the left and right sides of the rice transplanter, with one facing forward and the other facing backward.

The steering control device acquires an image input unit that inputs image information of an already-planted seedling row imaged by an imaging unit provided in the rice transplanter, and acquires a binary image in which a seedling row region is extracted from the image information. Seedling row region extracting means, reverse perspective transformation means for obtaining a virtual ground plane image of the seedling row region binarized by reverse perspective transformation of the binary image, and the seedling row region in the virtual ground plane image A straight line extracting means approximated by a straight line and a control means for performing steering control based on the position information of the rice transplanter with respect to the straight line are constituted, and these means are mainly executed by a computer.

FIG. 2 is a block diagram showing a flow of control by the steering control device of the rice transplanter. An image signal acquired by the CCD camera is input to an image processing apparatus having a computer as a main component through a signal line. This will be described in detail. One color CCD camera (Watec WAT231S) is attached to each side of the work machine. Since the seedling rows to be photographed were arranged in the vertical direction, they were rotated 90 degrees around the optical axis of the camera. Thereby, vertically long image information can be acquired, and the resolution in the vertical direction can be increased. In the experiment, the angle of view in the vertical direction was 33.4 degrees and the horizontal direction was 25.3 degrees. The depression angle of the camera was set to 15 degrees so that it could enter the screen to infinity. When there was a seedling on the left side of the vehicle, the left camera was used, and when it was on the right side, the right camera was used. Image information output from the CCD camera is recorded by a DV recorder (SONY DV1000). A personal computer (Pentium 4, 3.0 GHz, 512 MB) captures a digital image from a DV recorder via an IEEE1394 cable in real time. Direct X 8.0 was used to capture DV images.

The image processing apparatus has a seedling row region extraction means, a reverse perspective conversion means, and a straight line extraction means, and performs image processing on the image captured from the CCD camera to thereby position the rice transplanter with respect to the existing seedling rows, That is, the offset distance e and the azimuth angle θ are calculated. The seedling region extraction means is a means for generating a binary image in which the seedling row region is extracted from the image information of the existing seedling row captured by the camera. In this specification, as a preferred embodiment, as a seedling row region extracting unit, an RGB color image of a seedling row is converted into an L ^* a ^* b ^* color system, and a threshold is set to the value of the perceptual chromaticity b ^*. Although the method of performing the value conversion will be described later, any region extracting means may be used as long as it can extract the seedling row region satisfactorily. The reverse perspective conversion means is a means for generating a map of the seedling row by perspective-transforming each constituent pixel of the binary image generated by the seedling row region extracting means on a virtual ground plane constituting the farm scene. Since the reverse perspective transformation itself for converting the two-dimensional projection image acquired by the camera from the camera viewpoint coordinate system to the real plane coordinate system based on the camera parameters is known, the description of the reverse perspective transformation is omitted. The straight line extraction means is means for approximating a seedling row region in a virtual ground plane image (a seedling row map) with a straight line. Examples of the straight line extraction means include straight line detection by Hough transform and straight line detection by the least square method as a preferred embodiment, but straight lines may be extracted using other methods. If a line segment that approximates the straight line of the seedling row region is obtained, the angle θ and the offset e of the rice transplanter with respect to the line segment can be obtained. The angle and offset of the rice transplanter with respect to the detected straight line are obtained by the Hough transform or the least square method for the seedling row map. In addition, as to how to obtain the offset, other methods include obtaining from the speed and angle (dead reckoning), returning the angle obtained by the Hough transform to the map to the fluoroscopic image, and obtaining the degree of matching with the straight line drawn from the vanishing point. There is a way. In the present invention, in one preferred embodiment, an angle and an offset are obtained with respect to a seedling row planted in the immediately preceding process. You may detect a streak. In this case, from the history so far, it is determined that one is next and the offset is shifted (for example, 30 cm).

Then, the control unit performs steering control of the rice transplanter based on the position information of the rice transplanter with respect to the straight line approximated to the seedling row region. The speed v of the rice transplanter is calculated from the engine speed and the shift lever position. A photoelectric sensor is attached to the engine pulley, and the engine speed is detected based on a pulse output during one rotation of the engine pulley. In addition, a potentiometer is attached to the shift lever to detect the shift lever position. The pulse count and potentiometer voltage are measured by the PIC board, and the engine speed and shift lever position information is sent to the computer via a cable. The slip rate is fixed, and the speed of the rice transplanter is calculated by the computer using the information on the engine speed and the shift lever position.

The steering angle is determined from the position (angle θ and offset e) of the rice transplanter obtained by the image processing apparatus and the speed v of the rice transplanter. A motor is attached to the steering shaft to rotate the steering wheel. The DC servo driver controls the steering angle by driving the motor based on the command voltage from the computer. The actual steering angle is detected by a potentiometer attached to the pitman arm at the center of the front wheel. A voltage corresponding to the determined steering angle is output to the servo driver from the DA board, and the determined steering angle is realized by controlling information on the actual steering angle detected by the potentiometer.

The image input from the camera is the result of perspective transformation of the rice seedling row transplanted to the field. If the seedling row is perfectly straight, the straight line fitted in the image accurately represents the angle and distance between the camera and the seedling row. However, since the seedling rows are irregularly disordered, it is impossible in reality that this straight line does not include an error. Therefore, there is a problem of how to detect a seedling row that is not strictly a straight line as a straight line.

In the image processing apparatus, the RGB color image acquired by the camera is first converted into the L ^* a ^* b ^* color system by the seedling row region extracting means. Since the rice seedling has a high perceptual chromaticity b ^* component, the image is divided into a plurality of bands, and an arbitrary ratio of pixels from the top in each band is recognized as a seedling region. In FIG. 3, the original image as shown in the left figure is captured. The color of the original image is expressed in RGB, but this is converted to the L ^* a ^* b ^* color system. In the converted image information, the perceived chromaticity b ^* is high at the place where the seedling exists. Here, a threshold value is determined for b ^* by the p-tile method, and binarization is performed. That is, a pixel value where b ^* is equal to or greater than a threshold value is set to 1, and other pixel values are set to 0. In addition, in the area at the top and side corners of the image, there are many portions that are not related to the seedling, so the pixel value is set to zero. As a result, a binary image as shown in the center diagram is obtained. The RGB color image acquired by the imaging means is converted into the L ^* a ^* b ^* color system, and the object region is extracted by setting the perceptual chromaticity threshold in the converted image information and binarizing the image. This technique is established as a technical idea by itself, and its application is not limited to the detection of paddy rice seedling regions, and is not limited to the steering control according to the present invention. For example, in the case of detection of a crop row having a main color of green, a crop row region can be extracted by setting a threshold value to the value of perceptual chromaticity b ^* and binarizing the same as in paddy rice seedlings. . If the approximate area of the crop row area in the image is known, the threshold value can be similarly determined by the p-tile method. Even if the color of the object is not green, the object region can be extracted by appropriately setting the threshold value of the perceptual chromaticity according to the color of the object. Further, the image processing means described here is one preferable example, and the means for acquiring the binary image of the seedling row region is not limited to the one that converts to the L ^* a ^* b ^* color system, for example, In addition to HSV conversion, the seedling row region may be detected using the HSV value of each pixel.

Next, reverse perspective conversion is performed on the obtained binary image by the reverse perspective conversion means, and the seedling region in the image is back-projected onto the XY plane of the field (FIGS. 4 and 5). First, the obtained binary image is virtually projected onto the ground plane. That is, it is determined which position in the real space a certain pixel determined to be a part of the seedling as a result of binarization corresponds to. Based on the depression angle, height, and focal length of the camera, the camera focus position and binary image position are determined. And a binary image is projected on a virtual ground plane like FIG. A seedling row map is created by virtually projecting points on the binary image onto a ground plane in real space. Thereby, an angle is calculated | required more reliably. By roughening the resolution of the map, even if the seedling row is slightly bent, it can be recognized as a straight line. The angle θ of the seedling row is obtained from the obtained map using the Hough transform.

In FIG. 5, two virtual ground plane images having different horizontal resolutions were acquired from the left binary image. In the two virtual ground plane images, in the left virtual ground plane image, the size of one pixel on the ground plane is 5 cm in the vertical direction and 1 cm in the horizontal direction when projected. In the virtual ground plane image on the right, the size of one pixel is 5cm vertically and 5cm horizontally. That is, the left virtual ground plane image has a high resolution in the horizontal direction, whereas the right virtual ground plane image has a low horizontal resolution. An approximate straight line is obtained by using the Hough transform exemplified as the straight line extraction means for the image of the virtual ground plane, and the angle θ between the seedling strip (approximate straight line) and the rice transplanter is obtained. The right image was used when actually driving. In the image on the right, the image is rough in the horizontal direction, so that the angle of the stripe can be detected even if the stripe of the seedling being viewed is somewhat meandering or bent.

In this way, in the virtual ground plane image, the horizontal resolution is set low and the depth resolution is set high. Based on the shape of the seedling and the transplanting interval, it is expected that 1 cm in the horizontal direction and 20 cm in the depth direction are appropriate values for making an accurate seedling map in one pixel. Further, in one pixel, the horizontal direction is 5 cm and the depth direction is 5 cm. First, by taking a low horizontal resolution, the seedlings are placed on the same straight line even if they are slightly displaced in the horizontal direction. In addition, when a seedling row is photographed with a depression angle, the seedlings appear to be connected depending on the height of the seedling. Thus, when reverse perspective transformation is performed, the interval between the seedlings disappears. In order to detect the direction of the seedling row more accurately with respect to the map, it is desirable that the number of seedlings is large in the depth direction. Therefore, the resolution in the depth direction was increased.

Hough transform for line detection by Duda and Hart is applied to this map. On the obtained map, there are not only adjacent strips but also multiple strips. In order to run parallel to the previous stroke, it is desirable to refer to the angles obtained from more strips. Therefore, instead of simply detecting a single straight line by the Hough transform, the angle at which the most parallel straight lines exist was estimated. The sum of squares for each angle was calculated for those with values greater than half of the maximum value on the Hough plane, and the angle that gave the maximum total was adopted.

Next, detection of the lateral shift amount will be described. In the virtual ground plane image (map), since the resolution in the horizontal direction is set low, the distance between the camera and the seedling row cannot be directly estimated from this map. On the other hand, the perspective transformed image has a large angle error but a small distance error. Therefore, from the angle estimated from the map, the vanishing point of the parallel straight line group in the perspective-transformed image is obtained, and the position of the straight line passing through the seedling region most of the straight lines drawn from there is determined. The distance between the camera and the seedling row was used. This will be described with reference to FIG. When the angle between the seedling line and the rice transplanter is known, the vanishing point where the seedling line heads is required. The straight line extending from the vanishing point can be of various angles. Among them, the line that passes through the most white spots in the binary image (the straight line in the middle in FIG. 6) is adopted as the one that best represents the shoots. From this straight line, the amount of lateral deviation is obtained. For example, if this straight line is on the right side, the rice transplanter is on the left side.

The travel control is performed by calculating the expected deviation after driving an arbitrary distance by straightening the steering from the direction and distance of the seedling row with respect to the camera, and multiplying by a constant to calculate the steering angle . Since the distance error is expected to be large with respect to the direction, by dividing the distance by a constant and calculating the deviation, it is possible to perform smooth traveling with more emphasis on directionality.

I drove in the field while planting at low speed. In the first stroke of the shore, the edge of the shore was detected, and traveling and planting were performed along the edge. A prism was attached to the rice transplanter, and the locus of the rice transplanter was recorded using an automatic tracking total station. The experimental results are shown in FIGS. FIG. 7 shows the trajectory recorded by the total station, and the unit is meters for both the vertical and horizontal axes. The horizontal scale line is a straight line parallel to the ridge. From the bottom, proceed in the first, second and third strokes. Since the reflecting mirror (prism) is on the right side of the vehicle body, the distance between the trajectories in the stroke from left to right and the stroke from right to left is different. Each stroke was approximated by a straight line by the method of least squares. FIG. 8 summarizes the standard deviation of the amount of lateral deviation between the angle difference of the straight line and the approximate straight line of each data point. The angle difference tends to accumulate gradually. The standard deviation of the lateral slip did not appear to accumulate. Rather, I was able to observe the appearance of planting ignoring the small meanders in the previous stroke. As shown in the experimental results, it was possible to plant the plant almost straight at low speed.

Next, steering simulation using a so-called virtual rice transplanter will be described. Using 3D CG, we created a virtual paddy field and created a program in which a virtual rice transplanter runs. With this program, it is possible to test various image processing algorithms and control laws for various seedling row shapes. Visual C ++ 6.0 and OpenGL were used for program development. The flow of the program is shown in FIG. Before starting the run, first place the target seedling in the paddy field. The position of the seedling was changed in a sinusoidal shape. This means that in the previous stroke, the traveling direction of the rice transplanter has changed for some reason, and the position of the seedling has shifted. As already mentioned, seedlings cannot be transplanted on a complete straight line. Although the shape of the seedling row planted in the previous stroke can be arbitrarily changed, first, a sine wave was used as the first step. FIG. 11 shows a case where 12 rows of seedlings are planted in a 100 m long paddy field. The seedling spacing is 300 mm wide and 200 mm long. The period of seedling change is 40 m, and the amplitude is 10 cm. The height of the seedling is 70mm from the water surface.

A 3D scene drawn by OpenGL is captured by a virtual camera, and image processing is performed on a two-dimensional projection image in the same manner as in an outdoor experiment, and the camera's azimuth θ and offset e with respect to the seedling row are calculated. The steering angle is determined from the camera azimuth θ and the offset e. The position and direction after a certain time are calculated from the steering angle and speed, and OpenGL updates the image. The time increment was 0.1 seconds. The rice transplanter mentioned above was also processed every 0.1 seconds, so this was followed. To get closer to the actual situation, each seedling takes a random posture. Furthermore, the position is slightly deviated from the large sine wave. It also adds random disturbances to the camera azimuth. Therefore, even if the steering angle is always 0 degree, the rice transplanter does not go straight. In this program, in addition to automatic driving by a virtual controller, it is also possible to perform manual driving. It is thought that it is useful for knowing what a human being is running on a straight line with reference to.

An image processing method will be described. Using color space information such as RGB, HSV, L * a * b *, threshold, shutter speed, and aperture, it is possible to extract seedlings from captured images. Regarding the seedling region extraction means until the acquisition of the binary image of the seedling row region, the above description can be referred to, so the processing for the image after binarization is described. Here, two types of methods using the Hough transform will be described, and both methods will be compared.

The first method is a method of performing Hough transform on the projection image. The Hough transform by Duda and Hart is often used as a method to detect straight lines on images. In the Hough transform, a point r (θ) = xcos θ + ysin θ in the image is converted. If the seedling row is straight, the angle and offset between the seedling row and the camera can be obtained by returning the straight line obtained on the projection plane by reverse perspective transformation to the three-dimensional space again. FIG. 12 shows an example of detecting a straight line. In the Hough transform by Duda and Hart, a curve with a value of 1 is overlapped to detect its peak and form a straight line equation. Since the size of the peak represents the number of points on a straight line, the spatial distribution of the points is not taken into consideration. In other words, the same result can be obtained regardless of whether 10 points are concentrated and distributed on a straight line. When the seedling row is photographed, the seedling in the foreground is photographed large, and the leaves of the seedling are in a straight line shape, so there are cases where the local seedling itself is detected as the most probable straight line instead of the seedling row confirmed. In the target image, I want to detect a straight line that is longer in the vertical direction and has more points. Therefore, in addition to the curve value of 1, the value of y coordinate and the square of y coordinate were made. And the dispersion | distribution in each coordinate of a Hough plane was calculated | required from three images. Then, a linear equation was obtained from the peak by taking the linear sum of the points on the straight line and the variance.

The second method is a method of performing Hough transform on an inverse perspective transformation image. When planting rice, the operator sets a straight line with only one direction with respect to the field and travels along it. At the same time, the distance from the seedling planted in the previous process is checked from time to time to correct the driving route. The direction of the ideal route can be clearly observed from flying objects such as helicopters from above the paddy field. Therefore, reverse perspective transformation is first performed on the projection image to create a seedling row map. By performing straight line detection by Hough transform on this map, it is possible to detect the orientation of the entire seedling row that is not affected by local curves. FIG. 9 is an original image of a curved seedling row, and FIG. 10 is a map of the seedling row obtained as a result of performing reverse perspective transformation on FIG. Similarly, FIG. 13 shows the original image and the result of inverse perspective transformation. Only the rightmost seedling row in the original image was transformed. The resolution on the horizontal axis after inverse transformation is 5 cm, and the resolution in the vertical direction is 20 cm. If a Hough transform is performed on the point group in the right diagram of FIG. 13 to detect a straight line, one azimuth can be determined for one stroke. After obtaining the azimuth, the position can be estimated if the speed of the vehicle is known.

A control method will be described. According to Abe (2003), the simplest vehicle control model of a person is represented as follows if the target course is a straight line as shown in FIG.

This paper describes an automated driving experiment using a virtual rice transplanter. The test course put three sine waves along the 100m path. The amplitude of the sine wave is 400mm and the wavelength is 10m. The virtual rice transplanter travels along this seedling line. Noise was given randomly with respect to the direction angle. L was 1 m and h was 0.02. The virtual rice transplanter was run and the actual offset, angle, estimated offset, and estimated angle were recorded.

FIGS. 15 to 18 show the results of traveling by two image processes. FIGS. 15 and 17 show the results of the Hough transform performed on the perspective-transformed image, and FIGS. 16 and 18 show the results of the Hough transform performed on the image subjected to the reverse perspective transform. From FIG. 15 and FIG. 17, when trying to obtain the position in the real space after performing the Hough transform on the perspective transformation image, an accurate value for both the angle and the offset is obtained when the target seedling row is a straight line. However, it can be seen that if the seedling row is locally curved, an appropriate straight line cannot be detected, and the straight running cannot be performed. This is a fatal drawback in straight running. On the other hand, when the image subjected to reverse perspective transformation is Hough transformed, the angle of one whole seedling row can be obtained, so that the straight running can be performed regardless of local changes. However, in the case of inverse perspective transformation, since the horizontal resolution is 20 cm here, the offset cannot be obtained directly from the Hough transform, and the offset is obtained by dead reckoning. Therefore, there was a tendency for the offset to gradually increase due to intentionally applied direction noise and direction angle detection error. In this simulation, an offset error of 6 mm was seen after running 50 m.

The present invention can be used for steering control of a traveling vehicle. In a preferred example, the present invention can be used for steering control of a rice transplanter.

It is the top view and side view of a rice transplanter. It is a block diagram which shows the flow of control by a steering control apparatus. The left figure is the original image of the existing seedling rows on the field, and the middle figure is a binary image obtained by converting the original image to L ^* a ^* b ^* color system and setting a threshold value for b ^*. It is. It is explanatory drawing of the projection (reverse perspective transformation) of the binary image on the virtual ground plane. The left figure is a binary image, and the center figure and the right figure are virtual ground plane images obtained by inverse perspective transformation of the binary image. It is a figure explaining the detection of lateral deviation | shift amount. It is a figure which shows an experimental result. It is a figure which shows an experimental result. It is the image of the seedling row displayed on the display screen of the image processing device in the virtual rice transplanter. 10 is an image obtained by inverse perspective transformation of the image shown in FIG. 9. It is explanatory drawing of the steering control in a virtual rice transplanter. It is the figure which applied the Hough transform with respect to the fluoroscopic image on a display screen, and acquired the approximate straight line. The left figure is a perspective image of the seedling row, and the right figure is a map obtained by reverse perspective transformation of the left figure. It is a figure explaining the control method of a traveling vehicle. It is a running result based on what performed Hough transform to a perspective transformation image (two-dimensional projection image) inputted from a camera. It is the driving | running | working result based on having performed Hough transformation with respect to the image which performed reverse perspective transformation with respect to the perspective transformation image (two-dimensional projection image) input from the camera. It is a running result based on what performed Hough transform to a perspective transformation image (two-dimensional projection image) inputted from a camera. It is the driving | running | working result based on having performed Hough transformation with respect to the image which performed reverse perspective transformation with respect to the perspective transformation image (two-dimensional projection image) input from the camera.

Explanation of symbols

3 traveling machine body 5 steering wheel 6 seedling supply device 9 imaging means

Claims (13)

Capturing an image of an existing seedling row by an imaging means provided in the transplanter to obtain image information of the seedling row;
Obtaining a binary image in which a seedling row region is extracted from the image information;
Obtaining a virtual ground plane image of a seedling row region binarized by inverse perspective transformation of the binary image;
Approximating the seedling region with a straight line in the virtual ground plane image;
Performing steering control based on position information of the transplanter relative to the straight line;
Steering control method of transplanter having The steering control method according to claim 1, wherein the imaging unit is configured to take an image up to infinity of a seedling row. In the step of obtaining the binary image, the RGB color image of the seedling row is converted into the L ^* a ^* b ^* color system, and the threshold value is set to the value of the perceptual chromaticity b ^* to perform binarization. The steering control method according to any one of claims 1 and 2. The steering control method according to any one of claims 1 to 3, wherein a horizontal resolution is set low in the virtual ground plane image. 5. The steering control method according to claim 1, wherein the linear approximation of the seedling row region is performed by a Hough transform or a least square method. 6. The steering control method according to claim 1, wherein the position information of the transplanter with respect to the straight line is an angle with respect to the straight line and / or a lateral shift amount. An image input means for inputting image information of an already-planted seedling row imaged by an imaging means provided in the transplanter;
Seedling row region extraction means for obtaining a binary image in which the seedling row region is extracted from the image information;
Reverse perspective transformation means for obtaining a virtual ground plane image of a seedling row region binarized by performing reverse perspective transformation of the binary image;
Straight line extraction means for approximating the seedling row region with a straight line in the virtual ground plane image;
Control means for performing steering control based on the position information of the transplanter with respect to the straight line;
Steering control device of transplanter having Capturing an image of an existing seedling row by an imaging means provided in the transplanter to obtain image information of the seedling row;
Obtaining a binary image in which a seedling row region is extracted from the image information;
Obtaining a virtual ground plane image of a seedling row region binarized by inverse perspective transformation of the binary image;
Approximating the seedling region with a straight line in the virtual ground plane image;
Method for detecting seedling row having An image input means for inputting image information of an already-planted seedling row imaged by an imaging means provided in the transplanter;
Seedling row region extraction means for obtaining a binary image in which the seedling row region is extracted from the image information;
Reverse perspective transformation means for obtaining a virtual ground plane image of a seedling row region binarized by performing reverse perspective transformation of the binary image;
Straight line extraction means for approximating the seedling row region with a straight line in the virtual ground plane image;
A device for detecting a seedling row. Capturing a target extending in the traveling direction of the traveling vehicle by an imaging means provided in the traveling vehicle and acquiring image information of the target;
Obtaining a binary image in which a target area is extracted from the image information;
Obtaining a virtual ground plane image of the target region by performing reverse perspective transformation on the binary image;
Approximating a target area with a straight line in the virtual ground plane image;
Performing steering control based on position information of the traveling vehicle with respect to the straight line;
A steering control method for a traveling vehicle. Image input means for inputting image information of a target extending in the traveling direction of the traveling vehicle imaged by the imaging means provided in the traveling vehicle;
Binary image acquisition means for acquiring a binary image in which a target area is extracted from the image information;
Means for inverse perspective transformation of the binary image to obtain a virtual ground plane image of the target area;
Means for approximating the target area with a straight line in the virtual ground plane image;
Means for performing steering control based on position information of the traveling vehicle with respect to the straight line;
A steering control device for a traveling vehicle having Capturing a target extending in the traveling direction of the traveling vehicle by an imaging means provided in the traveling vehicle and acquiring image information of the target;
Obtaining a binary image in which a target area is extracted from the image information;
Obtaining a virtual ground plane image of a target area binarized by inverse perspective transformation of the binary image;
Approximating the target area with a straight line in the virtual ground plane image;
A method for detecting a target having Image input means for inputting image information of a target extending in the traveling direction of the traveling vehicle imaged by the imaging means provided in the traveling vehicle;
Target area extraction means for acquiring a binary image in which the target area is extracted from the image information;
A reverse perspective transformation means for obtaining a virtual ground plane image of a target area binarized by performing a reverse perspective transformation on the binary image;
Straight line extraction means for approximating the target area with a straight line in the virtual ground plane image;
A target detection apparatus having:

Images