JP7338398B2 - Vehicle image processing device - Google Patents

Description

The present embodiment relates to a vehicle image processing apparatus.

2. Description of the Related Art Conventionally, in a vehicle image processing device that performs image processing on an image obtained by capturing an image of a road surface, road marks in the image may be recognized for use in vehicle control.

WO2016/203939

In the vehicle image processing device, if the recognition accuracy of the road surface mark is low, it becomes difficult to improve the accuracy of vehicle control using the recognition result of the road surface mark. Therefore, a vehicle image processing apparatus capable of improving recognition accuracy of road marks is desired.

The vehicle image processing apparatus according to the embodiment includes, for example, a generation unit that generates an image in which a predetermined color is emphasized according to the parameters of the first color system, from the image acquired by the imaging unit that captures the road surface. a recognition unit for recognizing the road surface mark in the image in which the predetermined color is emphasized; a calculation unit for calculating parameters of the second color system for each pixel corresponding to the road surface mark; a determination unit that determines whether the calculated parameter of the second color system falls within the threshold range representing the predetermined color for the region that has been determined; and a correction unit for correcting the recognition result of the part. Therefore, according to the vehicle image processing apparatus according to the embodiment, for example, the road mark recognition results can be narrowed down step by step using the parameters of the first color system and the parameters of the second color system. Therefore, the recognition accuracy of the road mark can be improved.

In the vehicle image processing device, for example, the recognition unit recognizes a plurality of areas as the road surface mark, and the correction unit uses the calculated parameters of the second color system among the plurality of areas. cancels the recognition results of the recognizing unit for regions outside the threshold range, and maintains the recognition results of the recognizing unit for regions where the calculated parameters of the second color system fall within the threshold range. Therefore, according to the vehicle image processing apparatus according to the embodiment, for example, the recognition result of the region in which the color other than the predetermined color is emphasized can be removed from the recognition result of the recognition unit, and the recognition result of the road mark can be removed. It is possible to narrow down with high precision.

The image processing device for a vehicle further includes, for example, an extraction unit that extracts a representative area from the area of the road surface mark in the image, and the calculation unit calculates the second color system for each pixel of the representative area. Calculate parameters. Therefore, according to the vehicular image processing apparatus according to the embodiment, for example, a representative area with stable color can be extracted from the area of the road surface mark, and the parameters of the second color system can be accurately calculated for the road surface mark. can be done.

In the vehicle image processing device, for example, the first color system is the YUV color system, the second color system is the HSV color system or the HSL color system, and the calculated The parameter of the second color system is the hue parameter. Therefore, according to the vehicle image processing apparatus according to the embodiment, for example, the road surface mark can be accurately recognized from the road surface using the parameters of the first colorimetric system, and the recognition can be performed using the parameters of the second colorimetric system. It is possible to accurately remove an area in which a color other than the predetermined color is emphasized from the area of the marked road surface mark.

In the vehicle image processing device, for example, the generating unit may generate the above-described To generate an image in which a predetermined color is emphasized. Therefore, according to the vehicle image processing device according to the embodiment, for example, pixels of a predetermined color in an image obtained by imaging a road surface can be identified, and an image in which the predetermined color is emphasized can be generated.

In the vehicle image processing device, for example, the road mark is a lane marking, and the predetermined color is yellow. Therefore, according to the vehicle image processing apparatus according to the embodiment, for example, the accuracy of lane marking recognition can be increased step by step using the parameters of the first color system and the parameters of the second color system. can be done.

FIG. 1 is an exemplary perspective view showing a see-through state of a part of a cabin of a vehicle to which an ECU (vehicle image processing device) according to an embodiment is applied. FIG. 2 is an exemplary plan view (overhead view) of the vehicle to which the ECU according to the embodiment is applied. FIG. 3 is a diagram of an example of a dashboard of a vehicle to which the ECU according to the embodiment is applied, viewed from the rear of the vehicle. FIG. 4 is a diagram illustrating an example of a hardware configuration of a vehicle control system including an ECU (vehicle image processing device) according to the embodiment. FIG. 5 is a block diagram showing an example of a functional configuration of an ECU (vehicle image processing device) according to the embodiment. FIG. 6 is a diagram showing the distance from the identification plane of a predetermined color to the coordinate point of the pixel in the color space of the first color system (YUV color system) in the embodiment. FIG. 7 is a diagram showing colors to be emphasized (predetermined colors) and colors not to be emphasized in the color space of the first color system (YUV color system) in the embodiment. FIG. 8 is a diagram showing an image acquired by imaging the road surface in the embodiment. FIG. 9 is a diagram showing an image in which a predetermined color is emphasized according to the embodiment; FIG. 10 is a diagram showing a recognition result of a road mark (division line) in the embodiment. FIG. 11 is a diagram showing a representative area extracted from the road mark (division line) area in the embodiment. FIG. 12 is a diagram showing a recognition result (after correction) of a road mark (division line) in the embodiment. FIG. 13 is a flowchart showing processing for setting the target position of the vehicle in the embodiment. FIG. 14 is a flowchart showing detection processing of a road surface mark (division line) in the embodiment. FIG. 15 is a flow chart showing first color filter processing in the embodiment. FIG. 16 is a flow chart showing second color filter processing in the embodiment. FIG. 17 is a flowchart showing detection processing of a road surface mark (division line) in a modified example of the embodiment.

Hereinafter, the vehicle image processing apparatus of the present embodiment can be implemented as an ECU (Electronic Control Unit) and mounted on a vehicle. A vehicle in which the vehicle image processing device is mounted may be, for example, a vehicle 1 as shown in FIG. FIG. 1 is an exemplary perspective view showing a see-through state of a part of a cabin 2a of a vehicle 1 to which an ECU (vehicle image processing device) is applied. In this embodiment, the vehicle 1 equipped with the vehicle control device may be, for example, an automobile using an internal combustion engine (not shown) as a drive source, that is, an internal combustion engine automobile, or an automobile using an electric motor (not shown) as a drive source. That is, it may be an electric vehicle, a fuel cell vehicle, or the like. Alternatively, the vehicle 1 may be a hybrid vehicle using both an internal combustion engine and an electric motor as drive sources, or may be a vehicle equipped with other drive sources. In addition, the vehicle 1 can be equipped with various transmissions, and can be equipped with various devices, such as systems and parts, necessary for driving the internal combustion engine and the electric motor.

As illustrated in FIG. 1, the vehicle body 2 constitutes a vehicle compartment 2a in which a passenger (not shown) rides. A steering unit 4, an acceleration operation unit 5, a braking operation unit 6, a shift operation unit 7, and the like are provided in the passenger compartment 2a facing the driver's seat 2b. The steering unit 4 is, for example, a steering wheel projecting from the dashboard 24, the acceleration operation unit 5 is, for example, an accelerator pedal positioned under the driver's feet, and the braking operation unit 6 is, for example, the driver's It is a brake pedal positioned under the foot, and the shift operation unit 7 is, for example, a shift lever protruding from the center console. Note that the steering unit 4, the acceleration operation unit 5, the braking operation unit 6, the shift operation unit 7, and the like are not limited to these.

Further, a display device 8 as a display output unit and an audio output device 9 as an audio output unit are provided in the passenger compartment 2a. The display device 8 is, for example, an LCD (liquid crystal display), an OELD (organic electroluminescent display), or the like. The audio output device 9 is, for example, a speaker. Further, the display device 8 is covered with a transparent operation input section 10 such as a touch panel. A passenger can visually recognize an image displayed on the display screen of the display device 8 via the operation input unit 10 . In addition, the occupant can perform an operation input by touching, pressing, or moving the operation input unit 10 with a finger or the like at a position corresponding to the image displayed on the display screen of the display device 8. . The display device 8, the audio output device 9, the operation input unit 10, and the like are provided, for example, in a monitor device 11 positioned at the center of the dashboard 24 in the vehicle width direction, that is, in the left-right direction. The monitor device 11 can have operation input units (not shown) such as switches, dials, joysticks, and push buttons. Further, an audio output device (not shown) can be provided at another position in the vehicle interior 2a different from the monitor device 11, and audio is output from the audio output device 9 of the monitor device 11 and another audio output device. be able to. Note that the monitor device 11 can also be used, for example, as a navigation system or an audio system.

Further, a display device 12 different from the display device 8 is provided in the passenger compartment 2a. As exemplified in FIG. 3, the display device 12 is provided, for example, on the instrument panel section 25 of the dashboard 24, and is located substantially in the center of the instrument panel section 25 between the speed display section 25a and the rotation speed display section 25b. is located in The size of the screen 12 a of the display device 12 is smaller than the size of the screen 8 a of the display device 8 . The display device 12 can mainly display an image showing information related to parking assistance of the vehicle 1 . The amount of information displayed on the display device 12 may be less than the amount of information displayed on the display device 8 . The display device 12 is, for example, an LCD, an OELD, or the like. Information displayed on the display device 12 may be displayed on the display device 8 .

Further, as illustrated in FIGS. 1 and 2, the vehicle 1 is, for example, a four-wheel vehicle, and has two left and right front wheels 3F and two left and right rear wheels 3R. All of these four wheels 3 can be configured to be steerable. As illustrated in FIG. 4 , the vehicle 1 has a steering system 13 for steering at least two wheels 3 . The steering system 13 has an actuator 13a and a torque sensor 13b. The steering system 13 is electrically controlled by an ECU 14 (electronic control unit) or the like to operate an actuator 13a. The steering system 13 is, for example, an electric power steering system, an SBW (steer by wire) system, or the like. The steering system 13 supplements the steering force by adding torque, ie, assist torque, to the steering unit 4 by the actuator 13a, and steers the wheels 3 by the actuator 13a. In this case, the actuator 13 a may steer one wheel 3 or steer a plurality of wheels 3 . Further, the torque sensor 13b detects torque applied to the steering section 4 by the driver, for example.

Further, as illustrated in FIG. 2, the vehicle body 2 is provided with, for example, four imaging units 15a to 15d as a plurality of imaging units 15. As shown in FIG. The imaging unit 15 is, for example, a digital camera incorporating an imaging device such as a CCD (charge coupled device) or CIS (CMOS image sensor). The imaging unit 15 can output moving image data at a predetermined frame rate. The imaging units 15 each have a wide-angle lens or a fish-eye lens, and can take an image in a horizontal range of, for example, 140° to 190°. In addition, the optical axis of the imaging unit 15 is set obliquely downward. Therefore, the imaging unit 15 sequentially photographs the external environment around the vehicle body 2 including the road surface on which the vehicle 1 can move and the area where the vehicle 1 can be parked, and outputs the photographed image data.

The imaging unit 15a is positioned, for example, at the rear end 2e of the vehicle body 2 and provided on the wall below the door 2h of the rear trunk. The imaging unit 15b is positioned, for example, at the right end 2f of the vehicle body 2 and is provided on the right side door mirror 2g. The imaging unit 15c is positioned, for example, on the front side of the vehicle body 2, that is, at the front end portion 2c in the vehicle longitudinal direction, and is provided on the front bumper or the like. The imaging unit 15d is positioned, for example, on the left side of the vehicle body 2, that is, on the left end portion 2d in the vehicle width direction, and is provided on the door mirror 2g as a left protrusion. The ECU 14 executes arithmetic processing and image processing based on the image data obtained by the plurality of imaging units 15 to generate an image with a wider viewing angle, or to generate a virtual bird's-eye view image of the vehicle 1 viewed from above. can be generated. A bird's-eye view image can also be referred to as a plane image.

The ECU 14 also performs image processing on the image captured by the imaging unit 15 to detect road marks (for example, lane markings) indicated on the road surface around the vehicle 1 . The ECU 14 detects a target position (for example, a parking area, etc.) of the vehicle 1 based on the detected road surface mark (for example, a lane marking).

Further, as illustrated in FIGS. 1 and 2, the vehicle body 2 is provided with a plurality of distance measuring units 16 and 17, for example, four distance measuring units 16a to 16d and eight distance measuring units 17a to 17h. It is The ranging units 16 and 17 are, for example, sonars that emit ultrasonic waves and capture their reflected waves. Sonar may also be referred to as a sonar sensor or an ultrasonic detector. The ECU 14 can measure the presence or absence of an object such as an obstacle positioned around the vehicle 1 and the distance to the object based on the detection results of the distance measuring units 16 and 17 . That is, the distance measurement units 16 and 17 are examples of detection units that detect objects. Note that the distance measuring unit 17 can be used, for example, to detect relatively short-distance objects, and the distance measuring unit 16 can be used, for example, to detect relatively long-distance objects that are farther than the distance measuring unit 17 . Further, the distance measuring unit 17 can be used, for example, to detect objects in front of and behind the vehicle 1 , and the distance measuring unit 16 can be used to detect objects to the sides of the vehicle 1 .

4, the vehicle control system 100 includes an ECU 14, a monitor device 11, a steering system 13, distance measuring units 16 and 17, a brake system 18, a steering angle sensor 19, an accelerator sensor 20, and the like. , a shift sensor 21, a wheel speed sensor 22, and the like are electrically connected via an in-vehicle network 23 as an electrical communication line. The in-vehicle network 23 is configured as, for example, a CAN (controller area network). The ECU 14 can control the steering system 13, the brake system 18, etc. by sending control signals through the in-vehicle network 23. FIG. In addition, the ECU 14 detects torque sensor 13b, brake sensor 18b, steering angle sensor 19, distance measuring unit 16, distance measuring unit 17, accelerator sensor 20, shift sensor 21, wheel speed sensor 22, etc. via in-vehicle network 23. It is possible to receive results, operation signals of the operation input unit 10 and the like.

The ECU 14 includes, for example, a CPU 14a (central processing unit), a ROM 14b (read only memory), a RAM 14c (random access memory), a display control unit 14d, an audio control unit 14e, and an SSD 14f (solid state drive, flash memory). ing. The CPU 14a performs, for example, image processing related to the images displayed on the display devices 8 and 12, determination of the target position of the vehicle 1, calculation of the movement route of the vehicle 1, determination of the presence or absence of interference with an object, and processing of the vehicle 1. Various arithmetic processing and control such as automatic control and cancellation of automatic control can be executed. The CPU 14a can read a program installed and stored in a non-volatile storage device such as the ROM 14b, and execute arithmetic processing according to the program. The RAM 14c temporarily stores various data used in calculations by the CPU 14a. In addition, the display control unit 14d mainly performs image processing using image data obtained by the imaging unit 15, synthesis of image data displayed on the display device 8, and the like among the arithmetic processing performed by the ECU 14. FIG. Further, the voice control unit 14e mainly executes processing of voice data output by the voice output device 9 among the arithmetic processing in the ECU 14. FIG. The SSD 14f is a rewritable non-volatile storage unit, and can store data even when the power of the ECU 14 is turned off. The CPU 14a, ROM 14b, RAM 14c, etc. can be integrated in the same package. Further, the ECU 14 may have a configuration in which another logic operation processor such as a DSP (digital signal processor), a logic circuit, or the like is used instead of the CPU 14a. A HDD (hard disk drive) may be provided instead of the SSD 14f, or the SSD 14f and the HDD may be provided separately from the ECU 14. The ECU 14 is an example of a vehicle image processing device.

The brake system 18 includes, for example, an ABS (anti-lock brake system) that suppresses locking of the brakes, a skid prevention device (ESC: electronic stability control) that suppresses the sideslip of the vehicle 1 during cornering, and an increase in braking force ( BBW (brake by wire) and the like. The brake system 18 applies a braking force to the wheels 3 and thus to the vehicle 1 via an actuator 18a. The brake system 18 can also detect signs of brake locking, idling of the wheels 3, skidding, etc. from the rotational difference between the left and right wheels 3, and execute various controls. The brake sensor 18b is, for example, a sensor that detects the position of the movable portion of the braking operation portion 6. As shown in FIG. The brake sensor 18b can detect the position of the brake pedal as a movable part. Brake sensor 18b includes a displacement sensor.

The steering angle sensor 19 is, for example, a sensor that detects the amount of steering of the steering portion 4 such as a steering wheel. The steering angle sensor 19 is configured using, for example, a Hall element or the like. The ECU 14 acquires the steering amount of the steering unit 4 by the driver, the steering amount of each wheel 3 during automatic steering, and the like from the steering angle sensor 19, and executes various controls. A steering angle sensor 19 detects the rotation angle of a rotating portion included in the steering section 4 . The steering angle sensor 19 is an example of an angle sensor.

The accelerator sensor 20 is, for example, a sensor that detects the position of the movable portion of the acceleration operation portion 5 . The accelerator sensor 20 can detect the position of an accelerator pedal as a movable part. Accelerator sensor 20 includes a displacement sensor.

The shift sensor 21 is, for example, a sensor that detects the position of the movable portion of the shift operating portion 7 . The shift sensor 21 can detect the positions of movable parts such as levers, arms, and buttons. The shift sensor 21 may include a displacement sensor or may be configured as a switch.

The wheel speed sensor 22 is a sensor that detects the amount of rotation of the wheel 3 and the number of rotations per unit time. The wheel speed sensor 22 outputs the number of wheel speed pulses indicating the detected number of revolutions as a sensor value. The wheel speed sensor 22 can be configured using, for example, a Hall element. The ECU 14 calculates the amount of movement of the vehicle 1 based on sensor values obtained from the wheel speed sensors 22, and executes various controls. Note that the wheel speed sensor 22 may be provided in the brake system 18 in some cases. In that case, the ECU 14 acquires the detection result of the wheel speed sensor 22 via the brake system 18 .

It should be noted that the configurations and arrangements of the various sensors and actuators described above, the electrical connections, and the like are merely examples, and can be set (changed) in various ways.

FIG. 5 is a block diagram showing an example of the functional configuration of the ECU 14 according to this embodiment. As shown in FIG. 5 , the ECU 14 includes a detection section 141 , a target position determination section 142 , a route calculation section 143 , a movement control section 144 and a storage section 150 .

Each configuration of the detection unit 141, the target position determination unit 142, the path calculation unit 143, and the movement control unit 144 shown in FIG. 5 is realized by the CPU 14a executing a program stored in the ROM 14b. Note that these configurations may be configured to be realized by hardware circuits.

The detection unit 141 detects a road surface mark (for example, a lane marking) according to a peripheral image (road surface image) of the vehicle body 2 captured by the imaging unit 15 . The detection unit 141 stores data of the detected road surface mark (eg, lane marking) in the storage unit 150 . That is, the detection unit 141 manages the road surface marks (eg, lane markings, etc.). Further, the detection unit 141 accesses the storage unit 150 to refer to the data of the road surface mark, and detects a parking area around the vehicle 1 based on the road surface mark (for example, based on the road surface mark). The detection unit 141 supplies the detection result to the target position determination unit 142 .

The target position determination unit 142 determines the target parking area and the target position of the vehicle 1 based on the detection result of the detection unit 141 and the like. When the detection unit 141 detects a plurality of parking areas, the target position determining unit 142 may receive a driver's selection operation as to which parking area to use as the target parking area. For example, the target position determination unit 142 receives a driver's selection operation based on an operation signal acquired from the operation unit 14g.

The route calculation unit 143 calculates a movement route for moving the vehicle 1 from the current position to the target position when parking assistance is started. For example, the route calculation unit 143 calculates a guidance route when receiving an instruction to start parking assistance by an operation signal acquired from the operation unit 14g. The route calculation unit 143 outputs data of the calculated movement route to the storage unit 150 .

Further, although the target position determination unit 142 and the route calculation unit 143 receive the driver's operation based on the operation signal acquired from the operation unit 14g, the driver's operation input is not limited to this. For example, the above-described processing may be executed by accepting the driver's operation input from the operation input unit 10 .

The movement control unit 144 executes steering control to move the vehicle 1 based on the movement route data. Specifically, the movement control unit 144 accesses the storage unit 150 to refer to the movement route data, and controls the steering system 13 according to the position of the vehicle 1 so that the vehicle 1 moves along the movement route. It controls the actuator 13a. At this time, for example, the vehicle 1 is accelerated or decelerated (braked) according to the operation of the acceleration operation unit 5 or the braking operation unit 6 by the driver. Further, the movement control unit 144 may display guidance on the monitor device 11 or the like to instruct the driver to operate the acceleration operation unit 5 or the braking operation unit 6 .

The storage unit 150 is configured by a storage device such as the SSD 14f, for example. The storage unit 150 also stores road surface mark data and movement route data for parking assistance.

As an example of parking assistance, the movement control unit 144 performs automatic steering, and other operations are performed by the driver himself/herself. However, the present invention is not limited to this. For example, a configuration in which the movement control unit 144 automatically controls the operation of the acceleration operation unit 5 in addition to the steering may be adopted. Further, a configuration in which the operation of the shift operation unit 7 is also automatically controlled by the movement control unit 144 may be employed.

In the vehicle control system 100 including the ECU 14, for example, in order to improve the accuracy of the control of the vehicle 1 such as parking assistance using the road surface mark, it is desired to improve the accuracy of recognizing the road surface mark. When the road surface mark has a predetermined color, the detection unit 141 converts the basic color system parameters of each pixel of the image obtained by imaging the road surface into the first color system parameters. The detection unit 141 extracts pixels of a specific color from the image according to the converted parameters of the first color system, and generates an image in which a predetermined color is emphasized. The basic color system refers to the color system used for the image captured by the imaging unit 15 .

For example, when the basic color system is the RGB color system and the first color system is the YUV color system, the detection unit 141 detects the luminance, the first color difference, Each value of the second color difference is extracted. That is, the detection unit 141 converts the R value, G value, and B value in the RGB color system into a Y value (luminance), a U value (first color difference), and a V value (second color difference). The detection unit 141 substitutes the Y value, the U value, and the V value into the calculation formula for each pixel constituting the image, and calculates the number of pixels from the reference plane S in the color space of the YUV color system as shown in FIG. A distance D to the coordinate point P is calculated. The detection unit 141 newly sets the value corresponding to the calculated distance D as the Y value. The detection unit 141 generates an image in which pixels of a predetermined color (for example, yellow) are emphasized by performing this process for each pixel forming the image.

FIG. 6 is a diagram showing the distance from the discrimination plane of the predetermined color to the coordinate point of the pixel in the color space of the first color system (YUV color system). When coordinate points of pixels of various colors whose colors are known are arranged in the color space of the YUV color system, the reference plane S is a coordinate point of a pixel of a predetermined color (for example, yellow) and a predetermined color. can be determined as a plane that approximately distinguishes the coordinate points of pixels other than . A surface so determined is represented by a linear discriminant function aY+bU+cV+d=0 for a given color (eg yellow). That is, the reference plane S is an identification plane of a predetermined color (eg, yellow) in the color space of the YUV color system.

The calculation formula is a coordinate point P (Y, U, V) is a formula for calculating the distance D as shown in the following formula 1.
D=aY+bU+cV+d Expression 1

Equation 1 expresses that if the reference plane S is moved to the coordinate point P (Y, U, V) of the pixel to be calculated in the direction along the normal line from the coordinate point P to the reference plane S, the plane after movement is can also be viewed as indicating the value D of whether the coordinate point P is included in . The coefficients a, b, c, and d in Expression 1 are determined so that the reference plane S becomes a discrimination plane of a predetermined color (eg, yellow). As shown in Equation 2 below, the detection unit 141 may use a value obtained by multiplying the distance D calculated by Equation 1 by a predetermined coefficient e as the brightness Y1 of the pixel to be calculated. The coefficient e is, for example, a value greater than 1, and is determined according to the required degree of luminance enhancement.
Y1=D×e Expression 2

According to formulas 1 and 2, the farther the distance D from the reference plane S on the predetermined color side (+Y side and +V side in FIG. 6) with respect to the reference plane S in the color space of the YUV color system, emphasized. As a result, an image in which a predetermined color is emphasized can be generated using the simple calculations shown in Equations 1 and 2, so the image processing load can be reduced. In addition, since there tends to be a large difference in luminance between the road surface portion and the road surface marks in the road surface image, the road surface portion and the road surface marks can be separated with high accuracy, and an image that emphasizes the road surface mark area with high accuracy can be generated. Conceivable.

However, the processing according to formulas 1 and 2 tends to emphasize colors other than the predetermined color. For example, as shown in FIG. 7, the reference plane S at a certain luminance Y' is defined by a straight line (eg, aY'+bU+cV+d=0) on the UV plane. When the predetermined color is “yellow”, the side of the predetermined color with respect to the reference plane S is the left side in FIG. emphasized. Among the regions RG1 to RG3, the region RG1 is a region corresponding to pixels of a predetermined color (yellow) to be emphasized. On the other hand, the region RG2 is a region corresponding to pixels of a color (such as red) that is not desired to be emphasized, and the region RG3 is a region corresponding to pixels of a color (such as green) that is not desired to be emphasized.

Therefore, when emphasizing the "yellow" pixel by the formulas 1 and 2, the phenomenon occurs that the pixels of "colors other than yellow" such as red and green on the left side of the reference plane S in FIG. 7 are also emphasized. can. When the coordinate point P′ shown in FIG. 7 corresponds to a “pale yellow” pixel and the coordinate point P″ corresponds to a “dark red” pixel, the distance D from the reference plane S to the coordinate point P′ corresponds to the reference plane Since the distance D from S to the coordinate point P″ is large, the “dark red” pixels may be emphasized more than the “light yellow” pixels.

For example, the imaging unit 15 acquires a road surface image of a parking lot as shown in FIG. FIG. 8 is a diagram showing an image obtained by imaging the road surface. Although FIG. 8 is shown as a grayscale image, it is actually a color image, and is exemplified as an image in which "yellow" lane markings, "red" struts and other vehicles are mixed. At this time, the detection unit 141 performs processing according to formulas 1 and 2 to generate a grayscale image in which “yellow” pixels are emphasized as shown in FIG. As shown in FIG. 9, the intended "yellow" lane marking is highlighted, but the "red" struts and other vehicles are also highlighted. Therefore, even though it is desired to selectively detect the "yellow" lane markings, there is a possibility that the "red" struts and other vehicles may also be erroneously detected.

Therefore, in the present embodiment, in the ECU 14, the detection unit 141 further performs color discrimination using the parameters of the second color system for the area of the road mark recognized according to the parameters of the first color system. By doing so, we aim to improve the recognition accuracy of the road surface mark.

Specifically, in the ECU 14, the detection unit 141 converts the parameters of the basic color system (for example, the RGB color system) to the first color system ( For example, the YUV color system) parameters are converted. When the road mark to be detected has a predetermined color, the detection unit 141 generates an image in which the predetermined color is emphasized according to the parameters of the first color system. The detection unit 141 recognizes road marks in an image in which a predetermined color is emphasized. The detection unit 141 calculates the parameters of the second color system from the parameters of the basic color system for each pixel corresponding to each area recognized as a road surface mark. For example, when the second color system is the HSV color system or the HSL color system, the detection unit 141 converts the hue parameter (H value) from the parameters of the basic color system to the parameters of the second color system. may be calculated as The detection unit 141 determines whether or not the hue parameter (H value) falls within the threshold range for each pixel corresponding to each area recognized as a road surface mark. A threshold range is a range of hue parameters (H values) corresponding to a given color (eg, yellow). The detection unit 141 corrects the recognition result of the road mark according to the determination result. The detection unit 141 cancels the recognition results of pixels whose hue parameters fall outside the threshold range, and maintains the recognition results of pixels whose hue parameters fall within the threshold range. As a result, the road mark recognition results can be narrowed down step by step using the parameters of the first color system and the parameters of the second color system, so that the recognition accuracy of the road mark can be improved.

More specifically, the detection unit 141 has a conversion unit 1411, a generation unit 1412, a recognition unit 1413, an extraction unit 1414, a calculation unit 1415, a determination unit 1416, and a correction unit 1417, as shown in FIG.

The conversion unit 1411 acquires an image obtained by imaging the road surface (hereinafter referred to as a road surface image) from the imaging unit 15 . The conversion unit 1411 acquires, as a road surface image, for example, an image in which “yellow” lane markings, “red” pillars, and other vehicles are mixed, as shown in FIG. 8 . The conversion unit 1411 shown in FIG. 5 converts the parameters of the basic color system into the parameters of the first color system for each pixel of the road surface image. The basic color system is, for example, the RGB color system, and the first color system is, for example, the YUV color system. The conversion unit 1411 supplies the parameters of the first color system to the generation unit 1412 and the calculation unit 1415 .

The generation unit 1412 receives the parameters of the first color system from the conversion unit 1411 . The generation unit 1412 generates an image in which a predetermined color is emphasized according to the parameters of the first color system. For example, when the detection target of the detection unit 141 is a yellow road mark and the first color system is the YUV color system, the generation unit 1412 generates each pixel of the road surface image in the YUV color space. A distance D from a reference plane (for example, a yellow identification plane) S to a coordinate point P indicated by a parameter of each pixel is obtained. For each pixel of the road surface image, the generation unit 1412 substitutes the pixel parameters (Y value, U value, V value) into Equation 1 to obtain the distance from the reference plane S to the coordinate point P indicated by the parameter of each pixel. Find the distance D. As shown in Equation 2, the generation unit 1412 may use a value obtained by multiplying the distance D obtained by Equation 1 by a predetermined coefficient e for each pixel of the road surface image as the brightness Y1 of the pixel to be calculated. The generation unit 1412 generates an image in which a predetermined color is emphasized using the luminance Y1 of each pixel obtained according to the parameters of the first color system. For example, if the predetermined color is yellow, the generation unit 1412 generates a grayscale image in which the brightness (gradation value) of the "yellow" road mark area is emphasized, as shown in FIG. The generation unit 1412 supplies the generated image to the recognition unit 1413 .

The recognition unit 1413 receives an image in which a predetermined color is emphasized from the generation unit 1412 . The recognition unit 1413 performs edge extraction on the image in which the predetermined color is emphasized by the Canny method or the like, and recognizes the area of the road mark. For example, the recognition unit 1413 recognizes a plurality of areas AR1 to AR5 as shown in FIG. 10 as road mark areas in the image shown in FIG. The recognition unit 1413 generates the pixel positions of the boundaries of the areas AR1 to AR5 in the image (for example, the pixel positions of the corners of the boundaries) as a recognition result. The recognition unit 1413 supplies recognition results to the extraction unit 1414 and correction unit 1417 .

The extraction unit 1414 receives the recognition result of the recognition unit 1413 from the recognition unit 1413 . The extraction unit 1414 extracts representative regions from each region recognized by the recognition unit 1413 . The extraction unit 1414 divides each region recognized by the recognition unit 1413 into a plurality of partial regions, and selects a portion of the plurality of partial regions (for example, a region with a relatively stable color) as a representative region. may be selected as At this time, considering that color fringing occurs in the vicinity of color boundaries (edges) for areas to be determined using a color system (second color system) other than YUV, the recognition unit 1413 recognizes A region near the center of the region may be selected as the representative region. The extraction unit 1414 may, for example, equally divide the area AR1 shown in FIG. 10 vertically and horizontally to generate a plurality of partial areas. The extraction unit 1414 equally divides the area AR1 shown in FIG. 10 into three each in the vertical direction and the horizontal direction, generates a plurality of partial areas AR11 to AR19 as shown in FIG. A central partial area AR15 of AR19 may be selected as a representative area. The extraction unit 1414 generates the pixel positions on the boundary of the selected partial area AR15 (for example, the pixel positions of the corners AR15a to AR15d on the boundary) as the extraction result. The extraction unit 1414 may similarly extract representative regions for the other regions AR2 to AR5 recognized by the recognition unit 1413. FIG. The extraction unit 1414 supplies the extraction result of each representative region to the calculation unit 1415 .

Calculation unit 1415 receives the parameters of the first color system from conversion unit 1411 and the extraction result of each representative region from extraction unit 1414 . The calculation unit 1415 converts the parameters of the first color system into the parameters of the basic color system for each pixel in each representative area, and converts the parameters of the second color system from the parameters of the basic color system. calculate. For example, when the basic color system is the RGB color system, the first color system is the YUV color system, and the second color system is the HSV color system or the HSL color system, the detection unit 141 may calculate the parameters of the RGB color system from the parameters of the YUV color system, and calculate the hue parameter (H value) from the parameters of the RGB color system as a parameter of the HSV color system or the HSL color system. .

For example, the calculator 1415 calculates the hue parameter H for each pixel of the partial area AR15 (representative area of the area AR1) shown in FIG. The calculator 1415 calculates the hue parameter H in the following procedures (1) and (2).

(1) The calculation unit 1415 uses the parameters Y (luminance), U (first color difference), and V (second color difference) of the YUV color system to calculate the RGB color system using Equations 3 to 5 below. Convert to parameters R (red), G (green), B (blue).
R=Y+1.402×V Expression 3
G=Y-0.344×U-0.714×V Expression 4
B=Y+1.772×U Expression 5

(2) The calculator 1415 uses the parameters R, G, and B of the RGB color system to calculate the value of the hue parameter H using Equations 6 to 8 below. At this time, among the parameters R, G, and B of the RGB color system calculated in (1), let the maximum value be MAX and the minimum value MIN.
When R is the maximum value: H = 60 × {(GB) / (MAX-MIN)} ... Equation 6
When G is the maximum value: H = 60 × {(B-R) / (MAX-MIN)} + 120 Equation 7
When B is the maximum value: H = 60 × {(RG)/(MAX-MIN)} + 240 Equation 8

The calculation unit 1415 similarly calculates the hue parameter H for the representative areas of the other areas AR2 to AR5. The calculation unit 1415 supplies the calculated hue parameter of each pixel in each representative region to the determination unit 1416 .

The determination unit 1416 receives the hue parameter of each pixel in each representative region from the calculation unit 1415 . The determination unit 1416 determines whether the hue parameter of each pixel falls within the threshold range for each representative region. Specifically, the determination unit 1416 determines whether the ratio HR of pixels falling within the threshold range Hmin to Hmax among all pixels in the representative region exceeds the threshold ratio Rth for each representative region.

The determination unit 1416 can calculate the ratio HR of pixels falling within the threshold range Hmin to Hmax among all the pixels in the representative region using the following Equation 9.
HR=(the number of pixels satisfying (Hmin≦H≦Hmax) in the representative area)/(the total number of pixels in the representative area) Equation 9

The threshold range Hmin to Hmax is a range of hue parameter values corresponding to a predetermined color. When the detection target of the detection unit 141 is a yellow road mark, Hmin=30 and Hmax=50 may be set. The determining unit 1416 compares the ratio HR with the threshold ratio Rth for each representative region. The threshold ratio Rth is determined according to the road mark detection sensitivity required of the detection unit 141 . If the detection sensitivity of the detection unit 141 is at a standard level, Rth may be set to 50%.

If the ratio HR exceeds the threshold ratio Rth, the determination unit 1416 determines that the hue parameter of the representative region falls within the threshold range Hmin to Hmax. The fact that the hue parameter of a representative area falls within the threshold range Hmin to Hmax indicates that the color of the area represented by that representative area is generally a predetermined color (eg, yellow). If the ratio HR is equal to or less than the threshold ratio Rth, the determination unit 1416 determines that the hue parameter of the representative region does not fall within the threshold range Hmin to Hmax. The fact that the hue parameter of the representative area does not fall within the threshold range Hmin to Hmax indicates that the color of the area represented by the representative area is generally a color other than a predetermined color (eg, red, green, etc.). there is

For example, in the case of FIG. 10, in the representative area of the areas AR1 to AR3, the average value of the hue parameter is H=40, the ratio HR of pixels falling within the threshold range 30 to 50 exceeds the threshold ratio 50%, and the hue parameter falls within the threshold range of 30-50. That the hue parameter of the representative area falls within the threshold range indicates that the hue parameters of the areas AR1 to AR3 represented by the representative area fall within the threshold range. That is, it indicates that the colors of the areas AR1 to AR3 represented by the representative area are generally a predetermined color (eg, yellow). In the representative area of the areas AR4 to AR5, the average value of the hue parameter is H=16, the ratio HR of pixels falling within the threshold range of 30 to 50 is 50% or less, and the hue parameter does not fall within the threshold range. is determined. The fact that the hue parameter of the representative area does not fall within the threshold range indicates that the hue parameters of the areas AR4 to AR5 represented by the representative area do not fall within the threshold range. That is, it indicates that the colors of the areas AR4 to AR5 represented by the representative area are generally colors other than the predetermined color (eg, red, green, etc.).

The determination unit 1416 supplies the determination result to the correction unit 1417 .

The correction unit 1417 receives the recognition result from the recognition unit 1413 and the determination result from the determination unit 1416 . A correction unit 1417 corrects the recognition result of the recognition unit 1413 according to the determination result of the determination unit 1416 . Among the plurality of areas recognized as road marks, the correction unit 1417 determines that the area whose parameter of the second color system is out of the threshold range is a color other than the predetermined color, and the recognition unit 1413 Cancel the recognition result of Among the plurality of areas recognized as road marks, the correction unit 1417 determines that the color of the area whose parameter of the second color system falls within the threshold range is the predetermined color, and the recognition result of the recognition unit 1413 is determined. to maintain The correction unit 1417 supplies the corrected recognition result to the target position determination unit 142 .

For example, in the case of FIG. 10, the areas AR1 to AR3 are areas where the hue parameter falls within the threshold range, and the recognition result is maintained assuming that the color is approximately a predetermined color (eg, yellow). Areas AR4 and AR5 are areas where the hue parameter does not fall within the threshold range, and the recognition result is canceled because the color is generally a color other than the predetermined color (eg, red, green, etc.). As a result, as shown in FIG. 12, the "red" strut and other vehicle areas AR4-AR5 (see FIG. 10) are removed, leaving the "yellow" lane marking areas AR1-AR3. A later recognition result is obtained.

Next, the processing for setting the target position of the vehicle will be described with reference to FIG. 13 . FIG. 13 is a flow chart showing the process of setting the target position of the vehicle.

In the ECU 14, the detection unit 141 acquires the data of the road surface image captured by the imaging unit 15 (S1), and detects road surface marks (eg, lane markings, etc.) in the road surface image (S2). The detection unit 141 supplies the detection result to the target position determination unit 142 . The target position determination unit 142 stores the data of the road surface mark (eg, lane marking) detected in S2 in the storage unit 150, and manages the road surface mark (eg, lane marking) (S3). The detection unit 141 accesses the storage unit 150 to refer to data of the road surface mark, and detects a parking area around the vehicle 1 based on the road surface mark (for example, using the road surface mark as a reference). The detection unit 141 supplies the detection result to the target position determination unit 142 . The target position determination unit 142 determines the target parking area and the target position of the vehicle 1 based on the detection result of the detection unit 141 (S4). The route calculation unit 143 calculates a movement route for moving the vehicle 1 from the current position to the target position when parking assistance is started. The route calculation unit 143 outputs data of the calculated movement route to the storage unit 150 (S5). The movement control unit 144 accesses the storage unit 150 to refer to the movement route data, and executes steering control to move the vehicle 1 so that the vehicle 1 moves along the movement route.

Next, the details of the detection process (S2) of the road surface mark (for example, lane marking) will be described with reference to FIG. FIG. 14 is a flowchart showing detection processing of a road surface mark (division line).

After acquiring the road surface image, the detection unit 141 performs a first color filtering process (S11) to detect the road surface mark. The first color is, for example, white. If the detection of the road surface mark is successful (Yes in S12), the detection unit 141 ends the process. If the detection of the road mark fails (No in S12), the detection unit 141 performs a second color filtering process (S13) to detect the road mark. The second color is, for example, yellow.

Next, details of the first color filtering process (S11) will be described with reference to FIG. FIG. 15 is a flow chart showing the filtering process for the first color.

The detection unit 141 converts the parameters of the basic color system of each pixel of the road surface image into the parameters of the first color system (S21). The basic color system is, for example, the RGB color system, and the parameters of the basic color system include R values, G values, and B values. The first color system is, for example, the YUV color system, and the parameters of the first color system include Y values, U values, and V values. The detection unit 141 generates an image for the first color according to the conversion result of S21 (S22). The detection unit 141 uses the Y value of each pixel obtained in S21 to generate a grayscale image corresponding to the road surface image as a first color image. The detection unit 141 performs edge extraction on the image generated in S22 by the Canny method or the like (S23), and recognizes the area of the road mark (S24). The detection unit 141 supplies the recognition result of S24 to the target position determination unit 142 as the detection result of the detection unit 141 .

Next, details of the second color filtering process (S13) will be described with reference to FIG. FIG. 16 is a flow chart showing the filtering process for the second color.

The detection unit 141 converts the parameters of the basic color system of each pixel of the road surface image into the parameters of the first color system (S31). The basic color system is, for example, the RGB color system, and the parameters of the basic color system include R values, G values, and B values. The first color system is, for example, the YUV color system, and the parameters of the first color system include Y values, U values, and V values. The detection unit 141 obtains the distance D from the reference plane S of the second color (for example, the identification plane of yellow) to the coordinate point P indicated by the parameter of each pixel according to the conversion result of S31 (S32). The detection unit 141 generates an image in which the second color is emphasized according to the distance D obtained in S32 (S33). The detection unit 141 obtains a value obtained by multiplying the obtained distance D by a predetermined coefficient e as the luminance Y1 of the pixel to be calculated for each pixel, and uses the luminance Y1 of each pixel to emphasize the second color. A grayscale image is generated as a second color enhanced image. The detection unit 141 performs edge extraction on the image generated in S33 by the Canny method or the like (S34), and recognizes the area of the road mark (S35).

The detection unit 141 may recognize a plurality of areas as road mark areas. The detection unit 141 extracts a representative area from each area recognized in S35 (S36). The detection unit 141 may extract an area near the center of each area recognized in S35 as a representative area.

The detection unit 141 converts the parameters of the first color system converted in S31 into the parameters of the basic color system (S37), and converts the parameters of the second color system from the parameters of the basic color system. It is calculated as a parameter for color discrimination (S38). For example, when the basic color system is the RGB color system, the first color system is the YUV color system, and the second color system is the HSV color system or the HSL color system, the detection unit 141 may calculate the parameters of the RGB color system from the parameters of the YUV color system, and calculate the hue parameter (H value) from the parameters of the RGB color system as a parameter of the HSV color system or the HSL color system. .

For each area recognized in S36, the detection unit 141 determines whether the color discrimination parameter (eg, hue parameter) calculated in S38 is included in the threshold range (S39). Specifically, for each region recognized in S36, the detection unit 141 determines whether the ratio HR of pixels falling within the threshold range Hmin to Hmax among all pixels in the representative region exceeds the threshold ratio Rth. If the ratio HR exceeds the threshold ratio Rth for the determination target area, the detection unit 141 determines that the hue parameter of the representative area falls within the threshold range Hmin to Hmax (Yes in S39), and outputs the recognition result of S36. Maintain (S40). If the ratio HR does not exceed the threshold ratio Rth for the determination target area, the detection unit 141 determines that the hue parameter of the representative area does not fall within the threshold range Hmin to Hmax (No in S39), and recognizes in S36. Cancel the result (S41). As a result, a corrected recognition result is obtained in which the regions other than the second color are removed and the regions of the second color are left. The detection unit 141 supplies the corrected recognition result to the target position determination unit 142 as the detection result of the detection unit 141 .

As described above, in the present embodiment, in the ECU 14, the detection unit 141 further uses the parameters of the second color system for the area of the road surface mark recognized according to the parameters of the first color system. Perform color discrimination. As a result, the road mark recognition results can be narrowed down step by step using the parameters of the first color system and the parameters of the second color system, so that the recognition accuracy of the road mark can be improved.

The road surface mark to be detected by the detection unit 141 is not limited to the marking line of the road surface of the parking lot, and may be the marking line of the road surface for running such as a road.

Also, the parameters for color discrimination may include parameters other than hue parameters. For example, as a parameter for color discrimination, in addition to the hue parameter, or instead of the hue parameter, a saturation parameter (S value) of the HSV color system or the HSL color system may be used. The brightness parameter (V value) of the system may be used, or the luminance (L value) of the HSL color system may be used.

In addition, the imaging unit 15 replaces the parameters of the basic color system (eg, RGB color system) with each pixel of the image obtained by capturing the road surface using the first color system (eg, YUV color system). color system) parameters may be output. In this case, the detector 141 shown in FIG. 5 may have a configuration in which the converter 1411 is omitted. In the first color filtering process (S11) shown in FIG. 15, the conversion (S21) from the parameters of the basic color system to the parameters of the first color system may be omitted. In this case, the detection unit 141 generates an image for the first color according to the parameters of the first color system for each pixel of the road surface image (S22'). Similarly, in the second color filtering process (S21) shown in FIG. 16, the conversion (S31) from the parameters of the basic color system to the parameters of the first color system may be omitted. In this case, the detection unit 141 detects the parameter of each pixel from the second color reference plane (eg, yellow identification plane) S according to the parameter of the first color system of each pixel of the road surface image. A distance D to the coordinate point P is obtained (S32').

In the second color filtering process (S21) shown in FIG. 16, the detection unit 141 converts the parameters of the first color system converted in S31 into the second color system instead of S37 and S38. parameter may be calculated (S38'). At this time, the detection unit 141 may perform S38' using formulas obtained by substituting formulas 3 to 5 into formulas 6 to 8.

Further, the combination of the first color system used for generating an image in which a predetermined color is emphasized and the second color system used for color discrimination is the YUV color system and the HSV color system (or HSL color system). color system) may be used. For example, combinations of the first color system and the second color system include the CMYK color system, the XYZ color system, the HSV color system (or the HSL color system), and the L ^* a ^* b ^* color system. It may be two different colorimetric systems selected from among the systems.

Further, in the road mark detection process, color discrimination may be performed for a plurality of different colors. For example, as shown in FIG. 17, the detection unit 141 may perform the processes of S54 and S55 after S13. FIG. 17 is a flowchart showing detection processing of a road surface mark (division line) in a modified example of the embodiment.

The detection unit 141 detects the road surface mark by the second color filtering process (S13), and if the detection of the road surface mark is successful (Yes in S54), the process ends. If the detection of the road mark fails (No in S54), the detection unit 141 performs a third color filtering process (S55) to detect the road mark. The third color is for example red or green. As for the details of the third color filtering process (S55), the description of FIG. 16 can be similarly applied by replacing the second color in the description of FIG. 16 with the third color.

Although the embodiments of the present invention have been exemplified above, the above embodiments and modifications are merely examples, and are not intended to limit the scope of the invention. The above embodiments and modifications can be implemented in various other forms, and various omissions, replacements, combinations, and modifications can be made without departing from the scope of the invention. Also, the configuration and shape of each embodiment and each modification can be partially exchanged.

Reference Signs List 1 vehicle 14 ECU (parking support device) 141 detection unit 142 target position determination unit 143 route calculation unit 144 movement control unit 150 storage unit 1411 conversion unit 1412 generation Part 1413... Recognition part 1414... Extraction part 1415... Calculation part 1416... Judgment part 1417... Correction part.

Claims (6)

a generation unit for generating an image in which a predetermined color is emphasized according to the parameters of the first colorimetric system for the image acquired from the imaging unit that captures the road surface;
a recognition unit that recognizes road marks in the image in which the predetermined color is emphasized;
a calculation unit that calculates parameters of the second color system for each pixel corresponding to the road surface mark;
a determination unit that determines whether or not the calculated parameter of the second color system falls within the threshold range representing the predetermined color for the area recognized as the road surface mark;
a correction unit that corrects the recognition result of the recognition unit according to the determination result of the determination unit;
An image processing device for a vehicle. The recognition unit recognizes a plurality of areas as the road surface mark,
The correcting unit cancels the recognition result of the recognizing unit for a region, among the plurality of regions, in which the calculated parameters of the second color system deviate from the threshold range, and corrects the calculated second color system. 2. The vehicle image processing device according to claim 1, wherein the recognition result of said recognition unit is maintained for a region where system parameters fall within said threshold range. further comprising an extraction unit that extracts a representative area from the area of the road surface mark in the image,
The vehicle image processing device according to claim 1 or 2, wherein the calculator calculates the parameter of the second color system for each pixel of the representative area. The first color system is a YUV color system,
The second color system is the HSV color system or the HSL color system,
4. The vehicle image processing device according to any one of claims 1 to 3, wherein the calculated parameter of the second color system is a hue parameter. 5. The image according to claim 4, wherein the generating unit generates an image in which the predetermined color is emphasized according to a distance from a reference plane in a color space of the YUV color system to a coordinate point of the transformed parameter of each pixel. image processing device for vehicles. The road mark is a division line,
6. The vehicle image processing device according to claim 1, wherein said predetermined color is yellow.

Images