JP2016173634A - Steering control device and turning state estimation method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a steering control device for calculating a turning state of a vehicle from a picked-up image with high accuracy.SOLUTION: A feature area setting unit 48 sets a feature area composed of a plurality of pixel values of a picked-up image. A feature area tracking unit 50 derives the position information of the feature area in the picked-up image which is newly picked up during vehicle traveling. A travel distance calculation unit 52 calculates a travel distance of the vehicle on the basis of a position change amount of the feature area. A target area detection unit 40 detects a target area composed of a plurality of pixel values of the picked-up image to be a target of vehicle traveling. A target area tracking unit 44 derives the position information of the target area of the picked-up image which is newly picked up during vehicle traveling. A turning angle calculation unit 46 calculates a turning angle of the vehicle on the basis of the position change amount of the target area. A turning state estimation unit 54 estimates a turning state of the vehicle on the basis of the travel distance and the turning angle.SELECTED DRAWING: Figure 4

Description

  The present invention relates to a steering control device that controls steering of a vehicle and a turning state estimation method that estimates a turning state of a vehicle.

  In operations such as sowing, transplanting, setting up, padding, and material dispersal using agricultural vehicles, in order to improve the accuracy and efficiency of the operation, and for subsequent operations such as management and harvesting, each process goes straight forward. It is important to do so in parallel. Since these operations often take a long time in a wide field and cannot be performed again, there are large physical and mental burdens on the worker, and skill is required. For this reason, there is a great need for a technology for automatically steering agricultural vehicles.

  As a technique for controlling the steering of the vehicle, there are the following techniques. For example, in Patent Document 1, a tractor is provided with a GPS device and an orientation sensor, a target steering angle is calculated based on the detection results, and the steering angle is controlled using the steering angle sensor so that the steering angle becomes the target steering angle. A technique for automatic driving is disclosed.

  In Patent Document 2, the periphery of the vehicle is photographed by a camera installed in the vehicle, a plurality of camera images acquired during movement are processed to estimate the moving speed and rotation angle (steering angle) of the vehicle, and the moving speed And a driving support system for calculating a vehicle travel guideline based on the rotation angle. Specifically, this driving support system extracts the same feature points in real space from camera images acquired at three times, and calculates two movement vectors obtained from the position coordinates of the three feature points as vehicle movement trajectories. To do. Then, it is disclosed that a camera image is converted into a bird's-eye view image, and a moving speed and a rotation angle of the vehicle are estimated based on two movement vectors on the bird's-eye view image.

  Patent Document 3 discloses a vehicle travel control device that captures a target lamp installed at a target point with a camera, detects the target lamp from the captured image, and automatically travels the vehicle straight toward the target lamp. The

JP 2002-186309 A JP 2009-17462 A JP 2010-200674 A

  In the technique of Patent Document 1, a target steering angle is calculated based on GPS information or the like, and control is performed so that the steering angle of the vehicle matches the target steering angle using a steering angle sensor. , The amount of turning of the vehicle relative to the steering angle is insufficient, and the accuracy of travel control may be reduced.

  The technique of Patent Document 2 is a method of calculating two movement vectors using a camera image obtained by photographing the periphery of a vehicle and estimating the turning radius of the vehicle from the length and angle difference between the two movement vectors. This technique is effective when traveling with a small turning radius when parked, but it is difficult to detect a large turning radius that occurs when the vehicle automatically travels straight.

  The present invention has been made in view of such a situation, and an object of the present invention is to accurately estimate the turning state of the vehicle, thereby improving the accuracy of vehicle travel control and eliminating the need for a steering angle sensor. It is to provide a technique for reducing the cost of the system.

  In order to solve the above-described problem, a steering control device according to an aspect of the present invention includes an imaging unit that captures an image of a ground on which a vehicle travels including a distant view in front of the vehicle, and an image process that processes a captured image captured by the imaging unit. And a steering control unit that controls steering of the vehicle based on the processing result of the image processing unit. The image processing unit includes a feature region setting unit that sets a feature region composed of a plurality of pixel values of the captured image, and a feature region tracking that derives position information of the feature region in the captured image newly captured while the vehicle is traveling A target area detection unit configured to detect a target area that is a target of vehicle travel, and includes a travel distance calculation unit that calculates a travel distance of the vehicle based on a position change amount of the characteristic region, and a plurality of pixel values of the captured image A target region tracking unit that derives position information of a target region of a captured image that is newly captured while the vehicle is traveling, and a turning angle calculation unit that calculates a turning angle of the vehicle based on a position change amount of the target region A turning state estimation unit that estimates the turning state of the vehicle based on the travel distance and the turning angle.

  According to this aspect, it is possible to accurately estimate the turning state of the vehicle by calculating the travel distance and the turning angle using the captured image, and calculating the turning state of the vehicle based on them.

  ADVANTAGE OF THE INVENTION According to this invention, the turning state of a vehicle can be estimated with sufficient accuracy from the captured image imaged with the imaging part provided in the vehicle, a steering angle sensor becomes unnecessary and the cost of an apparatus can be reduced.

It is explanatory drawing when the steering control apparatus which concerns on embodiment in an agricultural field is seen from the side. It is explanatory drawing when the steering control apparatus which concerns on embodiment in a farm field is seen from upper direction. It is a figure which shows the function structure of ECU which concerns on embodiment. It is a figure which shows the function structure of the image process part which concerns on embodiment. It is a figure for demonstrating the target area | region and characteristic area | region on a captured image. It is a figure which shows the captured image shown in FIG. 5 with a photograph. It is a figure which shows the captured image of the time t2 which overwrites the captured image of the time t2 with the position of the target area | region and characteristic area of the captured image of the time t1. It is a figure which shows the captured image shown in FIG. 7 with a photograph. It is a figure for demonstrating the method of calculating the movement locus | trajectory of a vehicle by tracking a feature area. It is a figure for demonstrating the motion of the vehicle which turns. It is a figure for demonstrating the image processing of the steering control apparatus of a modification. It is a figure which shows the captured image shown in FIG. 11 with a photograph. It is a figure which shows the captured image of the time t2 which overwrites the captured image of the time t2 with the position of the target area | region and characteristic area of the captured image of the time t1. It is a figure which shows the captured image shown in FIG. 13 with a photograph.

  When performing work such as setting up and sowing on a farm field with an agricultural work vehicle, the driver of the work vehicle looks back to check the work state of the work machine provided at the rear of the work vehicle, Forced to drive straight ahead. Some farms have a length of more than 500 meters, and driving this distance straight places a heavy burden on the driver. Therefore, the steering control device of the embodiment detects a target ramp that is a target of vehicle travel from an image obtained by capturing the traveling direction, and automatically steers the vehicle toward the target ramp, thereby reducing the burden on the driver. Reduce.

  A work vehicle traveling on a farm field is affected by the inclination of the ground, unevenness, and clumps, and frequently changes in the direction or traveling direction of the vehicle or skids, so return the work vehicle to the original target line. Therefore, it is necessary to frequently steer the front wheels. In a conventional general steering control method, first, a travel locus necessary for returning the vehicle, then a turning radius for obtaining the traveling locus, a steering angle of the front wheels for obtaining the turning radius (hereinafter simply referred to as a steering radius). In some cases, it may be referred to as “steering angle.”), And the steering angle is set as the target steering angle. Subsequently, the current steering angle is read from the steering angle sensor, and feedback control is performed by driving a motor or the like so that the steering angle becomes the target steering angle. At this time, the relationship between the steering angle of the vehicle and the turning radius varies depending on the hardness of the ground. For example, compared with a hard pavement or the like, the grip force of the tire is reduced on a soft ground, so that the turning radius is increased for the same rudder angle. For this reason, in order to obtain a desired turning radius, it is necessary to change the rudder angle according to the hardness of the ground or the like. For example, when calculating the rudder angle from the target turning radius, a method of multiplying a constant set in advance according to the softness of the ground and the like to increase or decrease the steering amount at a constant rate is used. However, it is difficult to set an appropriate constant, and the ground in the field is not uniform, which causes an increase in the error in the travel trajectory of the work vehicle. Therefore, the steering control device according to the embodiment is a control method in which the current turning radius is calculated from a plurality of captured images by a camera mounted on the vehicle, and the steering angle is increased or decreased so that the turning radius becomes a target turning radius. . As a result, the steering angle sensor becomes unnecessary, and the turning radius is set as a feedback control target, so that it is difficult to be affected by the hardness of the ground, and the error of the travel locus can be further reduced.

  Drawing 1 is an explanatory view when steering control device 20 concerning an embodiment in a field is seen from the side. Drawing 2 is an explanatory view when steering control device 20 concerning an embodiment in a field is seen from the upper part. In the description of the drawings, the same elements are denoted by the same reference numerals, and repeated descriptions are omitted as appropriate. The vehicle 1 includes a cockpit 3, a left front wheel 8a and a right front wheel 8b (hereinafter collectively referred to as “front wheel 8”), a left rear wheel 10a and a right rear wheel 10b (hereinafter collectively referred to as “rear wheel 10”). The working machine 16 is mounted. The steering control device 20 is mounted on the vehicle 1, and includes a left brake mechanism 6a and a right brake mechanism (not shown) (hereinafter collectively referred to as “brake mechanism 6”), a camera 11, an ECU (Electronic control unit) 12, A steering drive mechanism 14 and a steering mechanism 15 are provided.

  The vehicle 1 may be a work vehicle such as an agricultural tractor or a civil engineering work vehicle, and a work machine 16 is attached to the rear of the vehicle 1. The work machine 16 performs operations such as sowing and setting in the farm field 4, for example.

  The steering mechanism 15 includes a steering wheel (not shown), a steering shaft (not shown), and a gear device (not shown) that converts the movement of the steering shaft into the movement of the front wheels. The steering mechanism 15 converts the rotation of the steering wheel as a handle into the turning motion of the front wheels 8. The steering wheel is provided in the cockpit 3 and is turned by a driver. The steering shaft has one end connected to the steering wheel so as to rotate together with the steering wheel, and functions as a rotating shaft that transmits the rotation of the steering wheel to the gear device.

  The steering drive mechanism 14 drives a motor and gives a steering force to the steering mechanism 15. The steering drive mechanism 14 is connected to the ECU 12. The steering drive mechanism 14 may be a hydraulic type, and may drive a hydraulic pump to give a steering force to the steering mechanism 15.

  The brake mechanism 6 applies a braking force to the rear wheel 10 in accordance with an operation amount of a driver's brake pedal (not shown). The brake pedal includes a left rear wheel brake pedal and a right rear wheel brake pedal. When the driver operates each brake pedal, a braking force can be individually applied to the left rear wheel 10a and the right rear wheel 10b. For example, when turning the vehicle 1 in the right direction, the driver depresses the brake pedal for the right rear wheel to apply a braking force to the right rear wheel 10b, thereby turning the vehicle 1 around the right rear wheel 10b. Can do. The brake mechanism 6 may be connected to the ECU 12 and the braking force applied to the rear wheel 10 may be controlled by the ECU 12. The front wheel 8 of the vehicle 1 may not have a brake mechanism. Further, the brake mechanism 6 is connected to the ECU 12, and the left or right brake mechanism 6 is operated by the ECU 12 to turn the vehicle 1.

  The target lamp 2 includes a light emitting unit 21 and a housing 24 that houses the light emitting unit 21. The light emitting unit 21 emits light from the opening of the housing 24. The light emitting unit 21 may be a light emitting diode (LED), has a predetermined size, and functions as a target unit. The light emitting unit 21 may blink at a predetermined control cycle. The target unit emits light so that the ECU 12 has a known size and the ECU 12 can recognize the target unit from an image captured by the camera 11.

  A monocular camera 11 is arranged on the upper part of the cockpit 3. The camera 11 functions as an imaging unit and images the target lamp 2 that is a target of vehicle travel and the ground in the traveling direction, and images a distant view in front and a ground in front. The front distant view refers to a distant view in front of the vehicle including the horizon, for example. The direction of the camera 11 is adjusted to the front direction of the vehicle. The camera 11 outputs the captured image to the ECU 12. The camera 11 may be provided with a tilt sensor, and the roll angle and pitch angle of the vehicle may be output to the ECU 12 as the tilt angle with respect to the ground at the time of imaging. The camera 11 may be a stereo camera.

  A target travel line 26 shown in FIG. 2 is determined by the relationship between the position of the target lamp 2 included in the captured image and the initial position of the vehicle 1. In FIG. 2, the target lamp 2 is disposed at a position in a target direction for vehicle travel. The initial position of the vehicle 1 refers to the position of the vehicle 1 when the driver turns on an automatic steering start switch (not shown) connected to the ECU 12 and refers to a position at which steering control by the steering control device 20 is started. The control by the steering control device 20 is executed / stopped by turning on / off the automatic steering start switch by the driver.

  When the driver turns on the automatic steering start switch, the ECU 12 receives the captured image from the camera 11, calculates the position of the target lamp 2 from the captured image of the target lamp 2, and sets the initial position of the vehicle 1 and the target lamp 2. A line connecting the positions is defined as the target travel line 26.

  The ECU 12 functions as an image processing unit that processes the captured image, and also functions as a steering control unit that controls the steering of the vehicle 1. The ECU 12 is a nonvolatile memory such as a CPU that executes various arithmetic processes, a ROM that stores various control programs, a RAM that is used as a work area for data storage and program execution, and a backup RAM that can retain stored contents even when the engine is stopped. An A / D converter for converting an analog signal input from a memory, an input / output interface, various sensors, and the like into a digital signal and taking it in, a timer for timing, and the like are provided. Note that the image processing unit and the travel control unit may be separate units that are an image processing device and a steering control device, respectively.

  FIG. 3 shows a functional configuration of the ECU 12 according to the embodiment. The ECU 12 includes an image acquisition unit 30, an image processing unit 32, a storage unit 34, and a steering control unit 36. The image acquisition unit 30 acquires a captured image captured by the camera 11 at a predetermined imaging cycle.

  The image processing unit 32 processes the captured image received from the image acquisition unit 30. The image processing unit 32 calculates a lateral deviation with respect to the target travel line 26 and a yaw angle of the vehicle with respect to the target lamp 2 from the captured image, as described in Japanese Patent Application Laid-Open No. 2010-200674, and calculates the calculated lateral deviation and yaw angle. Based on this, the target rudder angle is calculated and supplied to the steering control unit 36. Further, the image processing unit 32 calculates the turning state of the vehicle from the captured image and supplies it to the steering control unit 36. The storage unit 34 stores captured image information.

  The yaw angle is an acute angle based on the direction of the vehicle 1 and a line connecting the target lamp 2 and the position of the vehicle 1. For example, if the front-rear direction of the vehicle 1 is straight with respect to the target lamp 2, the yaw angle may be zero. When the lateral deviation of the vehicle 1 to the left and right is calculated as plus and minus, respectively, the position where the lateral deviation is zero is the position on the target travel line 26. The turning state of the vehicle is a state quantity based on the travel distance and turning angle of the vehicle, and may be a turning radius or an estimated steering angle of the vehicle, for example. The estimated steering angle is a state quantity indicating the turning state of the vehicle calculated from the captured image, and is a numerical value converted as a steering angle for comparison with the target steering angle. The image processing unit 32 supplies the estimated steering angle to the steering control unit 36 as the turning state of the vehicle.

  The target rudder angle is calculated so that the vehicle travels on the target travel line 26 toward the target position. For example, the steering control unit 36 reduces the lateral deviation based on the lateral deviation even when the yaw angle with respect to the target ramp 2 is zero and the traveling direction of the vehicle 1 is directed straight toward the target ramp 2. The front wheel 8 is steered.

  The steering control unit 36 drives the steering drive mechanism 14 to control the steering angle of the front wheels 8. The steering control unit 36 controls the vehicle 1 to travel based on the target steering angle and the estimated steering angle received from the image processing unit 32. The steering control unit 36 calculates the steering angle correction amount by subtracting the estimated steering angle from the target steering angle, and sends the driving amount of the motor corresponding to the steering angle correction amount to the steering drive mechanism 14. For example, if the ground of the field is softer than expected and the amount of turning of the vehicle is insufficient even when steering control is performed at the target rudder angle, the estimated rudder angle is calculated to be lower than the target rudder angle, A desired turning amount can be obtained by generating a steering angle correction amount and performing additional steering control.

  When attaching the steering control device 20 of the embodiment to an existing work vehicle, it is difficult for a general user to attach a steering angle sensor to the steering shaft of the steering mechanism 15 or the like. Therefore, for example, the rudder angle sensor may be attached to the front wheel 8 which is easy to attach, but there is a problem that durability is lowered due to mud or the like. Even if the front wheel 8 is steered due to the unevenness of the field, the steering control device 20 can correct the steering angle based on the image processing result, can control the steering accurately without using the steering angle sensor, and can suppress the cost.

  FIG. 4 shows a functional configuration of the image processing unit 32 according to the embodiment. The image processing unit 32 includes a target region detection unit 40, a target rudder angle calculation unit 42, a target region tracking unit 44, a turning angle calculation unit 46, a feature region setting unit 48, a feature region tracking unit 50, a travel distance calculation unit 52, and a turning A state estimation unit 54 is included.

  The target area detection unit 40 detects the light emitting unit 21 (hereinafter referred to as “target area”) on the image in the captured image received from the image acquisition unit 30. Specifically, the target area detection unit 40 detects a set of pixels exhibiting a luminance equal to or higher than a predetermined threshold as the target lamp 2, and detects an area including the set of pixels as a target area. In the target area detection unit 40, a captured image obtained by capturing the flashing light emitting unit 21 is received, and the luminance of the captured image of the light emitting unit 21 that is turned on is compared with the luminance of the captured image of the light emitting unit 21 that is turned off. May be determined as the target lamp 2, and may be determined as the target lamp 2 if a set of pixels having a predetermined luminance difference or more satisfies a predetermined area and a predetermined aspect ratio. Furthermore, the captured image of the flashing light emitting unit 21 is received, and it is determined whether or not the target area detected by the target area detecting unit 40 satisfies a temporal change in luminance according to the flashing cycle of the light emitting unit 21. It may be determined whether the detection is valid.

  The target area detection unit 40 causes the storage unit 34 to store the captured image and the position information of the detected target area. The target area detection unit 40 supplies position information of the target area to the target rudder angle calculation unit 42 and the target area tracking unit 44. Note that the position information may be coordinates in a virtual two-dimensional space or a virtual three-dimensional space corresponding to the distance in the real space. Further, the position information may be coordinates in a virtual two-dimensional space in units of pixels of the captured image, or may be coordinates based on a predetermined position on the captured image.

  The target rudder angle calculation unit 42 receives the position information of the target area from the target area detection unit 40 and calculates the yaw angle of the vehicle 1 with respect to the target lamp 2 from the direction of the vehicle 1 and the position of the target area. The orientation of the vehicle 1 may be a preset imaging direction of the camera 11, for example, a vertical vector passing through the center position of the captured image. Further, the target rudder angle calculation unit 42 calculates a lateral deviation from the target travel line 26 from the captured image. The target rudder angle calculation unit 42 sets a target position on the target travel line 26 based on the calculated yaw angle and lateral deviation, and calculates the target rudder angle so that the vehicle 1 goes straight to the target position.

  The target area tracking unit 44 detects a target area from a captured image newly captured while the vehicle is traveling, and derives position information of the target area. The target area tracking unit 44 causes the storage unit 34 to store the position information of the target area of the newly captured image.

  The turning angle calculation unit 46 calculates the position change amount of the abscissa on the captured image from the position information of the past target area and the newly calculated position information of the target area, and turns the vehicle based on the position change amount. Calculate the corner. Since the turning angle calculation unit 46 calculates the turning angle of the vehicle by using the change in position of the target lamp 2 that is far away in the real space, the turning angle calculation unit 46 is compared with the case of using the change in position of the region located in the vicinity of the vehicle. Can be accurately calculated. The turning angle calculation unit 46 sends the calculated turning angle to the turning state estimation unit 54.

  Here, when traveling on a field with many irregularities, it is assumed that imaging is performed in a state in which the vehicle body frequently vibrates with traveling, and the calculated target area position information includes fine fluctuations in noise. . The turning angle calculation unit 46 calculates the turning angle of the vehicle from, for example, a value obtained by moving and averaging the position change amounts for the past five times with respect to captured images taken at intervals of 0.1 seconds. The position change amount calculated every 0.1 second may be smoothed by filtering. Thereby, the influence of the noise-like fine vibration can be suppressed and the turning angle can be calculated with high accuracy.

  The feature region setting unit 48 is supplied with a captured image in parallel with the target region detection unit 40 from the image acquisition unit 30. The feature region setting unit 48 detects a characteristic partial image from the captured image and sets it as a feature region composed of a plurality of pixel values of the captured image. The feature region may be a region in which a luminance gradient equal to or greater than a predetermined threshold is distributed, and may be a partial image in a window having a predetermined size.

  The feature region setting unit 48 may set the feature region at a position closer to the vehicle in real space than the target region. That is, the feature area setting unit 48 sets the feature area so as to be positioned on the lower side in the vertical direction from the target area on the captured image.

  The feature region setting unit 48 sets a new feature region when the feature region tracking unit 50 can no longer track the set feature region. The area detected as the characteristic area becomes larger and deforms as the vehicle 1 advances. Therefore, the shape of the feature region may be set to a rectangle that is short in the vertical direction. The feature region setting unit 48 causes the storage unit 34 to store pixel value information and position information using the set feature region as a template. A plurality of feature areas may be set.

  Each time a new captured image is captured, the feature region tracking unit 50 searches and tracks a region having a brightness gradient or the like similar to the template of the set feature region. The feature region tracking unit 50 searches for a range of position coordinates around the template position. If the pixel value distribution between the template and the region extracted in the window on the new captured image matches a predetermined threshold or more, the feature region tracking unit 50 displays the window on the new captured image in the real space. Are determined to be in the same area. For example, the feature region tracking unit 50 determines whether the window on the new captured image is the same region in the real space as the template using a method such as a normalized correlation method.

  If it is determined that the feature region tracking unit 50 is the same region in the real space as the template, the feature region tracking unit 50 sends the position information of the window on the new captured image to the travel distance calculation unit 52. The feature region tracking unit 50 supplies a command signal to the feature region setting unit 48 to set a new feature region if there is no region on the new captured image that is the same region in the real space as the template.

  Each time the template of the feature area is determined to be the same area, the position on the captured image is stored in the storage unit 34, and the amount of movement of the vehicle before the next imaging is estimated. A new template is generated by enlarging and deforming the feature region according to the position of the template estimated in the captured image and the amount of displacement at that position, and is stored in the storage unit 34.

  The travel distance calculation unit 52 calculates the travel distance of the vehicle from the amount of change in the position of the feature area to be tracked. The amount of change in the position tracked in the previous process and the position tracked in the current process of a certain feature area is calculated, and the travel distance of the vehicle is calculated based on the position change amount. The feature region has a greater amount of vertical position change on the captured image as the region is closer to the vehicle. By calculating the position change amount of the feature region at a position closer to the vehicle than the target region in the real space, the travel distance can be calculated with high accuracy. The travel distance calculation unit 52 sends the calculated travel distance to the turning state estimation unit 54.

  The turning state estimation unit 54 estimates the turning state of the vehicle based on the calculated travel distance and turning angle. The turning state estimation unit 54 calculates the turning radius of the vehicle based on the travel distance and the turning angle, and calculates the estimated steering angle based on the turning radius and the wheel base of the vehicle.

  The steering control unit 36 controls the steering based on the target rudder angle and the estimated rudder angle. Thus, by estimating the turning state of the vehicle from the captured image and correcting the steering angle, the actual behavior of the vehicle is reflected in the steering, and steering control can be performed with high accuracy. Image processing will be described in detail with reference to the captured image.

  FIG. 5 is a diagram for explaining a target area and a characteristic area on a captured image. A captured image 60 illustrated in FIG. 5A is captured at time t1 while the vehicle is traveling, and a captured image 62 illustrated in FIG. 5B is an image captured at time t2 after time t1. Moreover, FIG. 6 is a figure which shows the captured image shown in FIG. 5 with a photograph.

  In the picked-up image 60 and the picked-up image 62, the target lamp 2 and the ground in the traveling direction, which are targets for vehicle travel in the field, are picked up. A target area 64a shown in FIG. 5 (a) is an area including the target lamp 2, in which the flashing light emitting section 21 is turned on, and in the target area 64b shown in FIG. 5 (b), the light emitting section 21 is turned off. It is. The target area 64a is set so that the light emitting unit 21 is located at the center. In addition, since the light emitting unit 21 is turned on and off, the target area 64a is set to be wider than the target lamp 2 and include the surrounding scenery for tracking on the captured image. Accordingly, a rectangular window elongated in the vertical direction j is set. Thus, the captured image captured by the camera 11 includes not only the ground slightly ahead in the traveling direction but also a distant view ahead in the traveling direction.

  The feature area 66a, the feature area 68a, the feature area 70a, the feature area 72a, and the feature area 74a shown in FIG. 5A are images on the ground in front of the target lamp 2 that have a change in brightness that is greater than or equal to a preset threshold value. A characteristic area on the image is set, and a rectangular window elongated in the horizontal direction i is set. A plurality of feature regions shown in FIG. 5A are set apart in the vertical direction j. The feature region 66b, the feature region 68b, the feature region 70b, the feature region 72b, and the feature region 74b shown in FIG. 5B are traced by pattern matching, and the feature region 66a, the feature region 68a, the feature region 70a, and the feature region 72a. In addition, the region is determined to be the same region in real space as each of the feature regions 74a.

  FIG. 7 shows a captured image 62 at time t2, in which the positions of the target area and the feature area detected in the captured image 60 at time t1 are overwritten on the captured image 62 at time t2 for explanation. Moreover, FIG. 8 is a figure which shows the captured image shown in FIG. 7 with a photograph.

  Due to the change of the imaging position due to the traveling of the vehicle, each feature area captured at time t1 moves downward on the captured image 62 at time t2. The position change amount of each feature region is greatest in the feature region 74a and the feature region 74b on the lowermost side. The position change amount in the vertical direction j is smaller in the region farther from the vehicle, the position change amount in the vertical direction j is larger in the region closer to the vehicle, and the position change amount in the vertical direction j is calculated more accurately in the region closer to the vehicle. it can.

  The displacement in the lateral direction i between the target area 64a and the target area 64b indicates that the direction of the camera 11, that is, the traveling direction of the vehicle has changed. The amount of change in the lateral direction i on the captured image is affected by both the change in the direction of the vehicle and the movement of the vehicle in the front-rear and left-right directions. The farther away from the vehicle, the smaller the influence of forward / backward / left / right movement, and the change in the direction of the vehicle can be extracted with high accuracy. As described above, the turning angle can be calculated with high accuracy by calculating the turning angle based on the position change amount of the target area farther from the vehicle than the characteristic area.

  FIG. 9 is a plan view for explaining a method of calculating the movement trajectory of the vehicle 1 by tracking the feature region. FIG. 9A shows the positional relationship between the vehicle 1 and the feature region 66a at time t1. The characteristic region 66a is a state set on an extension line of the center line 76a of the vehicle 1. Since the camera 11 is installed on the center line 76 a of the vehicle 1, the center line of the captured image coincides with the center line 76 a of the vehicle 1.

  FIG. 9B shows the positional relationship between the vehicle 1 and the feature region 66b at time t2. The feature region 66b is the same region in the real space as the feature region 66a. At time t2, the feature region 66b approaches the vehicle 1 due to the forward movement of the vehicle 1, and is located on the right side from the center line 76b of the captured image.

  As shown in FIG. 9C, when the feature region 66b is arranged so as to be positioned at the center of the left and right in FIG. 9C, the vehicle 1 turns slightly to the left. Subsequently, since the feature region 66a and the feature region 66b are the same region in the real space, when they are overlapped, the result is as shown in FIG. 9D, and the movement of the vehicle 1 between the time t1 and the time t2 The locus 78, that is, the movement amount and the turning amount can be known. By tracking the feature region from the captured image in this way, the movement locus of the vehicle 1 can be calculated. The feature region tracking unit 50 can accurately calculate the movement locus of the vehicle 1 by tracking a plurality of feature regions as shown in FIG. 5A and averaging the calculation results.

A method for calculating the travel distance will be described in detail. The vehicle longitudinal direction coordinate Z in the real space of the ordinate jb on the captured image is calculated by the following equation (1), and the abscissa X in the real space of the abscissa ib is calculated by the following equation (2). Is done. Here, the XYZ coordinates in the actual three-dimensional space are set as follows. First, the origin directly on the ground directly below the camera 11 is defined as the Y coordinate or height H in the vertical upward direction, the vehicle longitudinal direction as the Z coordinate, and the left and right direction as the X coordinate.
Z = (CAH) / (PWV · (jv−jb)) (1)
X = Z · PWH (ib-iv) (2)
CAH is the height from the ground to the camera, PWV is the tan value of the vertical viewing angle of one pixel, jv is the ordinate of the horizon on the captured image, PWH is the tan value of the horizontal viewing angle of one pixel, iv is the abscissa of the front direction of the vehicle on the captured image. As a result, the position on the captured image can be converted to a position in the real space, and the travel distance of the vehicle can be calculated from the amount of change in the vertical direction j on the captured image using Equation (1).

A method for calculating the turning angle will be described in detail. The turning angle dS of the vehicle 1 can be calculated by the following equation (3) based on the position change amount n1 in the lateral direction i of the target area on the captured image.
dS = n1 × PWH (3)
Next, a method for calculating the estimated steering angle from the travel distance and the turning angle will be described in detail.

  FIG. 10 is a diagram for explaining the movement of a turning vehicle. FIG. 10 shows the positions of the camera 11 at the time t1 and the camera 11 at the time t2 provided in the turning vehicle. Actually, the camera 11 is provided in the vicinity of the center of the vehicle, but for the sake of simplification of explanation, it is assumed that the camera 11 is set right above the center of the rear wheel axle. A calculation method will be described.

When the vehicle turns with the turning radius Rr, the relationship between the travel distance dZ in the Z-axis direction and the turning angle dS is expressed by the following equation (4), and the following equation (5) for calculating the turning radius Rr from the equation (4) is derived. It is burned.
dZ = Rr × sin (dS) (4)
Rr = dZ / sin (dS) (5)

The relationship among the turning radius Rr, the vehicle wheel base Wb, and the steering angle Sw satisfies the following expression (6).
tan (Sw) = Wb / Rr (6)
The following equation (7) is derived from the equations (5) and (6).
tan (Sw) = Wb × sin (dS) (7)
Equation (7) shows the relationship between the turning angle dS, the travel distance dZ, and the steering angle Sw, and the steering angle Sw is calculated from the turning angle dS and the travel distance dZ. The estimated steering angle can be calculated by replacing the steering angle Sw with the estimated steering angle. Thus, the steering angle can be estimated from the turning angle dS and the travel distance dZ, and the turning state of the vehicle can be estimated.

  FIG. 11 is a diagram for explaining image processing of the steering control device 20 according to the modification. A captured image 80 illustrated in FIG. 11A is captured at time t1 while the vehicle is traveling, and a captured image 82 illustrated in FIG. 11B is an image captured at time t2 after time t1. Moreover, FIG. 12 is a figure which shows the captured image shown in FIG. 11 with a photograph.

  The modified steering control device 20 does not use the target ramp 2 but detects irregularities such as ridges and grooves formed on the farm field as a target travel line 26 that is a target of travel from the captured image, and the vehicle detects the target travel line 26. Steering is controlled to travel on the top. Therefore, the steering control device 20 of the modified example is different from the steering control device 20 of the embodiment shown in FIG. 5 in the setting method of the target area and the characteristic area.

  A target area 84 that is a target of vehicle travel shown in FIG. 11A is composed of a plurality of pixel values including a vanishing point 85. Further, the feature region 86a, the feature region 88a, and the feature region 90a are set side by side in the horizontal direction i.

  The target area detection unit 40 detects a vanishing point 85 from the captured image 80 at the start of steering control, and sets a plurality of pixel values including the vanishing point 85 as the target area 84. The ordinate j of the vanishing point 85 is on the horizontal line 87, and the abscissa i is located at the abscissa iv in the front direction of the vehicle. The detected pixel value and position information of the target area 84 are stored in the storage unit 34. The target area detection unit 40 is a characteristic area located farther in the real space than the characteristic area, and may be detected as a target area located at the center of the abscissa i of the captured image at the start of the steering control. . Further, the ordinate j of the horizontal line 87 may be calculated based on the detection result of the inclination sensor attached to the vehicle 1.

  The feature region setting unit 48 sets a plurality of regions composed of a plurality of pixel values of the captured image 80 in a horizontal line, and sets the feature region 86a, the feature region 88a, and the feature region 90a in a horizontal line. Is done. The feature region setting unit 48 is not limited to a set of feature regions in a horizontal row, and may set a plurality of sets of linear feature regions in the vertical direction.

  In FIG. 11B, the target area tracking unit 44 detects the target area 88 from the captured image 82 newly captured while the vehicle is traveling, and derives position information of the target area 88. The target area tracking unit 44 stores the position information of the target area 88 of the new captured image 82 in the storage unit 34. The target area tracking unit 44 determines that the area of the captured image 82 that most closely matches the pixel value of the target area 84 is the same area as the target area 84 in real space by pattern matching.

  In addition, each time a new captured image is acquired, the feature region tracking unit 50 has the same region in real space as the region that most closely matches the pixel values of the feature region 86a, the feature region 88a, and the feature region 90a. And the feature region 86b, the feature region 88b, and the feature region 90b are derived.

  Each characteristic region set in a straight line in a horizontal row shown in FIG. 11A is shifted in the vertical direction j in FIG. 11B. This is because the space of the real space between the camera 11 and each feature region differs due to the unevenness of the ground. This is because the feature region 88b set at the bottom of the groove among the feature region 86b, the feature region 88b, and the feature region 90b is farthest from the camera 11 in real space. The steering control device 20 according to the modified example can detect the unevenness of the ground (the ridge or groove) by changing the position of the feature region 86b, the feature region 88b, and the feature region 90b in the vertical direction j. The steering is controlled so that the vehicle travels.

  FIG. 13 is a captured image at time t2, in which the target area and the feature area of the captured image 80 at time t1 are overwritten on the captured image 82 at time t2 for explanation. The target area and feature area of the captured image 80 overwritten on the captured image 82 at time t2 is position information on the captured image 80. Moreover, FIG. 14 is a figure which shows the captured image shown in FIG. 13 with a photograph.

  The position of the target area 84a at time t1 and the position of the target area 84b at time t2 have moved, and the positions of the characteristic areas 86a, 88a, 90a at time t1 and the characteristic areas 86b, 88b, 90b at time t2 have moved. doing.

  The turning angle calculation unit 46 calculates the position change amount of the abscissa of the target area 84a at time t1 and the target area 84b at time t2, and calculates the turning angle of the vehicle based on the position change amount.

  The travel distance calculation unit 52 calculates the position change amounts of the ordinates of the feature areas 86a, 88a, 90a at the time t1 and the feature areas 86b, 88b, 90b at the time t2, and averages the position change amounts. And averaged to calculate the travel distance of the vehicle based on the position change amount.

  The turning state estimation unit 54 calculates the estimated steering angle from the turning angle and the travel distance, and estimates the turning state of the vehicle. The steering control unit 36 controls steering by reflecting the estimated steering angle on the target steering angle. Thus, in the steering control device 20 of the modified example, the steering can be controlled without using the target ramp 2, and the cost can be suppressed.

  The present invention has been described based on the embodiments. This embodiment is an exemplification, and it will be understood by those skilled in the art that various modifications can be made to combinations of the respective constituent elements, and such modifications are also within the scope of the present invention.

  In the embodiment, a mode in which the steering control device 20 automatically drives the vehicle by pressing a switch, that is, a mode in which the steering amount and the travel are controlled has been described. The steering control device 20 may control the steering amount of the vehicle 1 and the driver may control the vehicle progress with an accelerator pedal or the like.

  In the embodiment, the aspect in which the captured image is corrected based on the detection result of the tilt sensor is also shown, but the present invention is not limited to this aspect. The image processing unit 32 may detect a horizontal line from the captured image. The image processing unit 32 detects, as a horizontal line, a set of pixel values with a boundary line having a linear luminance difference on the captured image and having an inclination angle with respect to the horizontal axis within a predetermined range. The image processing unit 32 may perform correction to rotate the captured image so that the horizon is parallel to the horizontal axis.

  DESCRIPTION OF SYMBOLS 1 Vehicle, 2 Target lamp, 3 Control room, 4 Farm, 6 Brake mechanism, 8 Front wheel, 10 Rear wheel, 11 Camera, 12 ECU, 14 Steering drive mechanism, 15 Steering mechanism, 16 Working machine, 20 Steering control device, 21 Light emitting section, 26 target travel line, 30 image acquisition section, 32 image processing section, 34 storage section, 36 steering control section, 40 target area detection section, 42 target rudder angle calculation section, 44 target area tracking section, 46 turning angle calculation 48 feature region setting unit, 50 feature region tracking unit, 52 travel distance calculating unit, and 54 turning state estimating unit.

Claims (4)

An imaging unit for imaging the ground on which the vehicle travels, including a distant view in front of the vehicle;
An image processing unit that processes a captured image captured by the imaging unit;
A steering control unit for controlling steering of the vehicle based on the processing result of the image processing unit,
The image processing unit
A feature region setting unit that sets a feature region composed of a plurality of pixel values of the captured image;
A feature region tracking unit for deriving positional information of the feature region in a captured image newly captured during vehicle travel;
A mileage calculating unit that calculates the mileage of the vehicle based on the position change amount of the feature region;
A target area detection unit configured by a plurality of pixel values of the captured image and detecting a target area as a target of vehicle travel;
A target area tracking unit for deriving position information of the target area of a captured image newly captured while the vehicle is running;
A turning angle calculation unit for calculating a turning angle of the vehicle based on a position change amount of the target area;
And a turning state estimating unit that estimates a turning state of the vehicle based on the travel distance and the turning angle. The image processing unit calculates a target rudder angle based on the calculated position information of the target area;
The steering control device according to claim 1, wherein the steering control unit controls steering of the vehicle based on the target steering angle and the estimated turning state.   The steering control device according to claim 1, wherein the feature region setting unit sets the feature region at a position closer to the vehicle in real space than the target region. A turning state estimation method for estimating a turning state of a vehicle by processing a captured image captured by an imaging unit provided in the vehicle,
Setting a feature region composed of a plurality of pixel values of the captured image;
A step of detecting a target area which is composed of a plurality of pixel values of the captured image and which is a target of vehicle travel;
Deriving position information of the feature region and position information of the target region in a captured image newly captured during vehicle travel;
Calculating a mileage of the vehicle based on a position change amount of the feature region, and calculating a turning angle of the vehicle based on a position change amount of the target region;
And a step of estimating a turning state of the vehicle based on the travel distance and the turning angle.