JP7226930B2 - Calibration device and electronic mirror system - Google Patents

Description

The present invention relates to a calibration device and an electronic mirror system.

2. Description of the Related Art When changing lanes or the like, a driver of a vehicle looks at side mirrors provided on both sides of the vehicle to face rearward, and visually confirms the image of the rear region reflected by the side mirrors.

In recent years, a so-called electronic mirror system has been developed that has a rear-facing camera instead of a side mirror and a monitor installed inside the vehicle that displays the image of the rear area captured by the camera ( For example, see Patent Document 1). According to the electronic mirror system, the aerodynamic drag of the vehicle can be reduced by replacing the camera with a camera physically smaller than the side mirror.

JP 2018-6936 A

According to the electronic mirror system, it is also possible to detect a following vehicle existing in the rear area based on the image captured by the camera. However, if the road on which you are driving is curved, you can detect the actual positional relationship (lane, distance from your vehicle) of the actual following vehicle along the road only from the position of the following vehicle in the image. difficult to do.

The present invention has been made in view of the above circumstances, and based on the image of the camera provided in the own vehicle, accurately obtains the positional relationship between the other vehicle captured in the camera image and the own vehicle in the real space. It is an object of the present invention to provide a calibration device and an electronic mirror system that can

A first aspect of the present invention is to determine the state of the curve of the road on which the vehicle is traveling, based on a plurality of images captured by a camera mounted on the vehicle at a plurality of time-series different times on the road on which the vehicle is traveling. and a correspondence between the position on the image and the position along the road in the real space of the portion captured in the image in a reference state assuming that the road is straight. A calibration device comprising: a storage unit storing relationships; and a correction unit correcting the correspondence stored in the storage unit based on the state of the curve of the road detected by the road state detection unit. is.

A second aspect of the present invention is a camera mounted on a vehicle and provided at a location where a reflecting mirror is to be provided for capturing an image of the exterior of the vehicle; An image display device provided in a cabin of the own vehicle for displaying an image that is horizontally reversed as if viewed by reflection; and a calibration device according to the present invention, wherein the image display device performs calibration This is an electronic mirror system that emphasizes an approaching vehicle based on a signal output from a notification output unit of the device.

According to the calibration device and the electronic mirror system according to the present invention, based on the image of the camera provided in the own vehicle, the positional relationship between the other vehicle captured in the image of the camera and the own vehicle in the real space can be accurately determined. can ask.

1 is a block diagram showing an electronic mirror system 1, which is an example of an electronic mirror system according to the present invention, including a camera ECU (Electronic Control Unit) 30, which is an example of a calibration device according to the present invention; FIG. A vehicle equipped with the electronic mirror system shown in FIG. It is a figure represented typically. 3 is an example of an image in which the image captured by the camera shown in FIG. 2 is horizontally reversed. FIG. 4 is a schematic diagram showing a grid for calibration stored in a storage unit; FIG. 10 is a schematic diagram showing an example when a grid for calibration is photographed by a camera and superimposed on a left-right reversed image. FIG. 4 is a diagram schematically showing a state in which the vehicle is traveling on the left lane of a horizontal road with three lanes on one side that gently curves to the right, for example, from an upper viewpoint. 7 is an example of an image in which the image captured by the camera shown in FIG. 6 is horizontally reversed. FIG. 8 is a diagram showing a state in which a grid corrected to follow the curve of the road shown in FIGS. 6 and 7 is superimposed on the image shown in FIG. 6; A view from above of the vehicle, for example, exiting a two-lane expressway on one side and exiting into a side road that branches off to the left, and is driving along a rightward curve on the side road. It is a figure represented typically. 10 is an example of an image in which the image captured by the camera shown in FIG. 9 is horizontally reversed. FIG. 11 is a diagram showing a state in which a grid corrected to follow the curve of the road shown in FIGS. 9 and 10 is superimposed on the image shown in FIG. 9; It is a schematic representation of how the end point moves from the initial position in the image as the vehicle travels.

Hereinafter, specific embodiments of an electronic mirror system according to the present invention will be described with reference to the drawings.

<Configuration of electronic mirror system>
FIG. 1 is a block diagram showing an electronic mirror system 1, which is an example of an electronic mirror system according to the present invention, including a camera ECU (Electronic Control Unit) 30, which is an example of a calibration device according to the present invention. A vehicle 200 equipped with the illustrated electronic mirror system 1 (hereinafter referred to as the own vehicle 200) is traveling on the left lane L1 of a non-curved horizontal road 400 with three lanes on each side, for example, as viewed from above. FIG. 3 is an example of an image P1 obtained by horizontally reversing the image captured by the camera 10 shown in FIG.

The electronic mirror system 1 shown in FIG. 1 includes a camera 10 , an in-vehicle network (CAN: Controller Area Network) 20 , a camera ECU 30 , and a monitor 90 .

The cameras 10 are provided at positions on both sides of the vehicle 200 to which the left and right side mirrors are to be attached, respectively, as substitutes for the left and right side mirrors. That is, one camera 10 is provided on each of the left and right sides, and each camera 10 mainly captures the rear area of the vehicle 200 on the side where it is installed. Specifically, as shown in FIG. 2, the camera 10 provided on the right side of the vehicle 200 in the direction of travel indicated by the arrow is installed toward the right rear of the vehicle 200, and the right rear camera 10 is installed. photograph the area. Although illustration is omitted, the camera 10 provided on the left side of the vehicle 200 in the direction of travel indicated by the arrow is installed toward the left rear of the vehicle 200, and the left rear region is installed. to shoot.

When the image captured by the right camera 10 shown in FIG. 2 is horizontally reversed, for example, an image P1 shown in FIG. 3 is obtained. If the vehicle 200 is equipped with a right side mirror, the driver of the vehicle 200 sees the image of the right rear side reflected by the side mirror. The image incident on the mirror is horizontally reversed.

For a driver who is accustomed to seeing the image reflected in the side mirror, if the image captured by the camera 10 is displayed on the monitor 90 as it is, the image that the driver is accustomed to seeing in the side mirror is horizontally reversed. , you may feel uncomfortable or confused.

Therefore, the camera ECU 30 subjects the image captured by the camera 10 to horizontal reversal processing, and causes the monitor 90 to display the horizontally reversed image P1 as shown in FIG.

The in-vehicle network 20 is a line for sharing various information (vehicle speed, steering angle, gear stage, etc.) of the own vehicle 200 .

<Configuration of camera ECU>
The camera ECU 30 includes a road condition detection unit 31, a storage unit 32, a correction unit 33, an object recognition unit 34, a driving lane detection unit 35, a distance calculation unit 36, and a notification output unit 37. .

Road condition detection unit 31 detects the curve condition of road 400 on which vehicle 200 is traveling. Specifically, the state of the curve is detected based on a plurality of left-right reversed images P1 captured by the camera 10 at a plurality of different time points on the road 400 on which the vehicle 200 is traveling. In this embodiment, end points 411 (end points 411 (see FIG. 2) of a white line 410 separating lanes L1 and L2 (see FIG. 2) of a road 400 in a plurality of horizontally inverted images P1 photographed by the camera 10 at a plurality of time-series different points in time. For example, the degree of curve of the road 400 is detected based on the time-series change in the position of the upper left corner point).

Note that the road state detection unit 31 may detect the state of the curve based on changes in the time-series position of the photographed object other than the end point 411 of the white line 410 .

FIG. 4 is a schematic diagram showing the calibration grid 40 stored in the storage unit 32, and FIG. 5 shows an example when the calibration grid 40 is photographed by the camera 10 and superimposed on the left-right reversed image P1. It is a schematic diagram showing.

A grid 40 for calibration shown in FIG. 4 is stored in the storage unit 32 . This grid 40 shows the positions on the image P1 and the actual space of each part on the road surface of the road 400 shown in the image P1 in a reference state assuming that the road 400 on which the vehicle 200 is traveling is straight. 4 is an example of a table (map) showing a correspondence relationship with positions along the road 400 at .

That is, when the grid 40 is superimposed on the image P1 captured by the camera 10, for example, as shown in FIG. The position of each part on the road surface of the road 400 (position in the coordinate space on the image) is associated with the position in the physical space (position in the coordinate space of the physical space).

The lattice 40 shown in FIG. 4 is set based on so-called camera parameters such as the angle of view and focal length of the camera 10, the mounting position on the vehicle 200, and the orientation. Each of the lines M1, M2, M3, M4, M5, M6, and M7 extending in the horizontal direction of the lattice 40 are equidistantly spaced rearward in the longitudinal direction of the vehicle 200 in the physical space (for example, 10 [m] in the physical space). ), and the lines N1, N2, N3, N4, N5, and N6 extending in the vertical direction are equidistant intervals in the vehicle width direction of the vehicle 200 in the physical space (for example, 1 [m] intervals in the physical space). corresponds to That is, the lattice 40 corresponds to a rectangular area of 60 [m] in the longitudinal direction and 5 [m] in the vehicle width direction of the own vehicle 200 in the physical space.

The correction unit 33 corrects the grid 40 stored in the storage unit 32 based on the curve state of the road 400 detected by the road state detection unit 31 . Specifically, for example, when the road state detection unit 31 detects that the road 400 is in a straight state with no curves, the correction unit 33 corrects the grid 40 in the reference state stored in the storage unit 32 . is superimposed on the image P1 as it is, as shown in FIG.

On the other hand, for example, when the road 400 detected by the road state detection unit 31 is curved, the correction unit 33 deforms the grid 40 in the reference state so as to follow the detected curve of the road 400. I do.

Here, FIG. 6 schematically shows a state in which the own vehicle 200 is traveling, for example, in the left lane L1 of a horizontal road with three lanes on one side that gently curves to the right, viewed from above. FIG. 7 is an example of an image P1 obtained by horizontally reversing the image P0 captured by the camera 10 shown in FIG. 6, and FIG. 7 is a diagram showing a state in which a grid 40' is superimposed on the image P1 shown in FIG. 6; FIG.

In FIG. 9, the own vehicle 200 exits the lane L1 of a side road branched to the left from a two-lane highway, for example, and travels in a place where the lane L1 of the side road curves to the right. FIG. 10 is an example of an image P1 obtained by horizontally reversing the image P0 captured by the camera 10 shown in FIG. 9, and FIG. FIG. 10 is a diagram showing a state in which a grid 40' corrected to follow the curve of the road 400 shown is superimposed on the image P1 shown in FIG. 9;

For example, when the road state detection unit 31 detects that the road 400 gently curves to the right as shown in FIGS. , is transformed into a grid 40' along the curve of the detected road 400 and superimposed on the image P1 of FIG. 6 as shown in FIG. This makes it possible to specify the positional relationship between each portion on the road surface of the road 400 and the target object in contact with the road surface and the own vehicle 200 along the curve of the road 400 in the physical space.

Similarly, when the road condition detection unit 31 detects that the road 400 branches to the left and then curves to the right as shown in FIGS. The grid 40 to be superimposed is transformed into a grid 40' along the curve of the detected road 400, and superimposed on the image P1 of FIG. 9 as shown in FIG. This makes it possible to specify the positional relationship between each portion of the road surface of road 400 and host vehicle 200 along the curve of road 400 in the physical space.

As shown in FIGS. 3, 7, 10, etc., the object recognition unit 34 recognizes following vehicles 301, 302, . . . To detect. The object recognition unit 34 stores in advance patterns such as outline shapes and patterns of front surfaces of vehicles including motorcycles in the storage unit 32, and applies the stored patterns to the image P1 by pattern matching or the like. , the following vehicle 301 and the like are detected. Further, the object recognition unit 34 detects the positions of the following vehicle 301 and the like in the image P1 by detecting the positions of the detected following vehicle 301 and the like on the road surface.

The driving lane detection unit 35 corrects the positions of the following vehicle 301 and the like on the image P1 recognized by the object recognition unit 34, and based on the lattice 40 obtained by correcting the positions, the road 400 in the real space. Then, the position along the road 400 in the physical space is identified by converting to the position along the road 400, and the lane in which the following vehicle 301 and the like are traveling in the physical space is identified.

That is, for example, as shown in FIG. 3, when the road 400 is straight and not curved, the grid 40 remains the same as the grid 40 in the reference state. , the lane is identified in the physical space of each following vehicle 301 and the like.

In this case, as shown in FIG. 5, the driving lane detection unit 35 detects that the following vehicle 301 is driving in the same lane L1 as the own vehicle 200, and the following vehicle 302 is in a lane adjacent to the right of the driving lane L1 of the own vehicle 200 ( It is specified that the following vehicle 303 is traveling in the lane (adjacent lane) L3 adjacent to the right of the adjacent lane L2.

Further, in the case shown in FIG. 8, since the lattice 40 in the reference state is deformed into the lattice 40', the traveling lane detection unit 35 detects each following vehicle 301 according to the superimposed state of the deformed lattice 40'. Identify lanes in a real space such as

In this case, as shown in FIG. 8, the traveling lane detection unit 35 detects that the following vehicle 302 travels in a lane (adjacent lane) L2 that is adjacent to the right of the traveling lane L1 of the own vehicle 200, and that the following vehicles 303, 304, and 305 specifies that the vehicle is traveling in the lane (adjacent lane) L3 adjacent to the right of the adjacent lane L2.

Similarly, in the case shown in FIG. 11, since the grid 40 in the reference state has been transformed into a grid 40', the traffic lane detection unit 35 detects each following vehicle according to the superimposed state of the transformed grid 40'. A lane is specified in real space such as 301 .

In this case, the traveling lane detection unit 35 identifies that the following vehicles 301 and 306 are traveling in the same lane L1 as the own vehicle 200, as shown in FIG.

Based on the grid 40 corrected by the correction unit 33, the distance calculation unit 36 calculates the distance along the curve of the road 400 between the position of the own vehicle 200 and the position of the following vehicle 301 in the physical space. .

The notification output unit 37 determines that the lane in which the following vehicle 301, etc., detected by the driving lane detection unit 35 is traveling is the lane L2 adjacent to the lane L1 in which the vehicle 200 is traveling, and the distance calculation unit 37 36 is becoming shorter in time series, that is, when the following vehicle traveling in the adjacent lane L2 is approaching the own vehicle 200, the signal announcing the approach of the following vehicle is monitored. output to 90.

The monitor 90 visibly displays an image P1 obtained by horizontally reversing the image P0 captured by the camera 10 by the camera ECU 30 . At this time, when the camera ECU 30 is outputting a signal notifying a following vehicle traveling faster than the own vehicle 200 so as to approach the own vehicle 200 in the adjacent lane L2, the monitor 90 displays, for example, FIG. , a vehicle detection frame 350 surrounding the detected following vehicle 302 is superimposed on the image P1.

<Action>
Next, the operation of the electronic mirror system 1 of this embodiment will be described. The camera 10 installed on the right side of the own vehicle 200 photographs the right rear area of the own vehicle 200 at predetermined intervals, and the photographed images are input to the camera ECU 30 . The camera ECU 30 converts each image periodically input from the camera 10 into a horizontally inverted image P1.

The road state detection unit 31 detects the curve state of the road 400 on which the vehicle 200 is traveling based on a plurality of horizontally inverted images P1 that are periodically input to the camera ECU 30 . The detection of the state of the curve is performed using images P1 captured at different timings in time series when the end point 411 of the specific white line 410 shown in FIG. determined by the positional transition on.

FIG. 12 schematically shows how the end point 411 moves from the initial position R1 to positions R2', R3', R4', and R5' in the image P1 as the vehicle 200 travels. It is what I did. The road state detection unit 31 detects the positions R1, R2', R3', R4', and R5' of the end points 411 that move over time on the image P1, for example, as positions on the XY orthogonal coordinates in FIG.

In FIG. 12, positions R1, R2, R3, R4, and R5 are vehicle speeds (CAN20 (input from ) represents the position where the end point 411 is predicted to move. These predicted positions R1, R2, R3, R4, and R5 use the vehicle speed of the vehicle 200 input from the in-vehicle network 20 and assume a steering angle of 0 degrees. Predicted positions R1, R2, R3, R4, and R5 in this reference state are aligned on a straight line.

However, the positions R1, R2', R3', R4', and R5' detected by the road condition detection unit 31 are not aligned, and the position R2' is shifted from the position R2 in the X-axis direction and the Y-axis direction. , and subsequent positions R3', R4', and R5' are also shifted in the X-axis direction and the Y-axis direction from the corresponding positions R3, R4, and R5, respectively. This displacement in the X-axis direction and the Y-axis direction indicates that the direction of the vehicle 200 changes in time series, that is, the road 400 on which the vehicle 200 is traveling is curved. there is

The correction unit 33 corrects the grid 40 for calibration based on the curve state of the road 400 detected by the road state detection unit 31 .

Specifically, the position R2' detected by the road condition detection unit 31 is displaced in the X-axis direction from the predicted position R2 in the reference state by the amount of deviation ΔR2x (by the X-axis in FIG. 12). The same applies hereinafter), the displacement amount ΔR2y in the Y-axis direction (described near the Y-axis in FIG. 12, the same applies hereinafter), and the position R3′ from the predicted position R3 in the reference state. , the displacement amount ΔR3x in the X-axis direction and the displacement amount ΔR3y, R4′ in the Y-axis direction are the displacements in the X-axis direction from the predicted position R4 in the reference state. The amount ΔR4x and the displacement amounts ΔR4y and R5′ in the Y-axis direction are displaced in the X-axis direction and the Y-axis direction from the predicted position R5 in the reference state. The lattice 40 shown in FIG. 2 is deformed based on the shift amount ΔR5y.

At this time, the destination positions of the positions other than the positions R2, R3, R4 and R5 in the grid 40 are the destination positions R2', R3' and R4' of the positions R2, R3, R4 and R5. , R5′ can be calculated as representative points so that the grid 40 can be deformed to follow the curved road 400. FIG.

As a result, based on the image P1 of the camera 10 provided on the own vehicle 200, the positional relationship between the following vehicle captured in the image P1 of the camera 10 and the own vehicle 200 along the curve of the road 400 in the real space. can be calculated accurately.

Note that the amount of movement of the position within the grid 40 due to the curve of the road 400 is small in the immediate rear of the vehicle 200, and increases as the distance from the vehicle 200 increases. , only a partial region P2 behind the host vehicle 200 shown in FIG. 12 may be subjected to calculation. As a result, the amount of calculation for correction can be reduced, and the calculation load can be suppressed. For the camera ECU 30 installed in a mobile object such as a vehicle with limited resources, suppression of computational load is one of the important issues.

On the other hand, the object recognition unit 34 detects the following vehicle from the image P1. The traveling lane detector 35 identifies the lane in which the following vehicle is traveling based on the corrected grid 40'.

Similarly, the distance calculator 36 obtains the distance along the curve of the road 400 from the own vehicle 200 to the following vehicle detected in the image P1 based on the corrected grid 40'.

In this way, the camera ECU 30 identifies the lane in which each detected following vehicle travels and the distance along the curve of the road 400 from the own vehicle 200 to each following vehicle. Then, the notification output unit 37 drives the lane L2 adjacent to the lane L1 in which the vehicle 200 is running, among the detected following vehicles, and the following vehicle running in the adjacent lane L2 and the vehicle When the distance along the curve of the road 400 is shortening in time series, that is, when the notification output unit 37 detects a following vehicle traveling in the adjacent lane L2 approaching the own vehicle 200, the monitor 90 to output a notification signal.

As shown in FIG. 8, the monitor 90 sets a vehicle detection frame 350 surrounding the following vehicle 302 for the following vehicle approaching the own vehicle in the adjacent lane L2, which is the target of the notification signal. It is superimposed on the image P1 and displayed.

As a result, the driver of vehicle 200 looking at monitor 90 can be warned not to change lanes to adjacent lane L2.

On the other hand, among the detected following vehicles, the notification output unit 37 outputs a following vehicle that is traveling in the lane L1 in which the host vehicle 200 is traveling or a following vehicle that is traveling in the adjacent lane L3. Even if the vehicle approaches the own vehicle 200, the notification signal is not output. Similarly, even if the following vehicle is traveling in the adjacent lane L2, the notification output unit 37 does not output the notification signal even when the following vehicle is not approaching the own vehicle 200 .

Since there is no notification signal output from the notification output unit 37, the monitor 90 displays the image P1 without adding the vehicle detection frame 350 for calling the driver's attention to any following vehicle.

As a result, the driver of the own vehicle 200 looking at the monitor 90 can be prevented from being unnecessarily alerted not to change lanes to the adjacent lane L2. Frequent occurrence of unnecessary notifications can be prevented or suppressed.

As described above, according to the camera ECU 30 and the electronic mirror system 1 of the present embodiment, based on the image P1 of the camera 10 provided in the own vehicle 200, the following vehicle 301, etc., captured in the image P1 of the camera 10, The positional relationship with the vehicle 200 along the road 400 in the real space can be obtained accurately.

In the present embodiment, only the positional relationship between the following vehicle on the right rear of the vehicle 200 and the vehicle 200 captured by the camera 10 provided on the right side of the vehicle 200 has been described. The positional relationship between the following vehicle mainly on the left rear of the own vehicle 200 and the own vehicle 200 photographed by the camera 10 provided on the left side of the vehicle 200 is similarly performed.

Also, in the present embodiment, only the positional relationship between the following vehicle existing behind the own vehicle 200 and the own vehicle 200 has been described, but the calibration device and the electronic mirror system according to the present invention can be applied not only to the following vehicle but also to the following vehicle. , and the positional relationship between the vehicle 200 and a preceding vehicle that exists in front of the vehicle 200 . In this case, a camera that faces the front of the vehicle is applied.

In the above-described embodiment, the grid 40 is deformed according to the degree of curvature of the road 400, and the position of the following vehicle in the image P1 is determined by the lane and distance along the lane represented by the deformed grid 40'. However, the image P1 is transformed into an image of another coordinate system specified by the horizontal lines M1, M2, . . . and the vertical lines N1, N2, . The position of the following vehicle (lane, distance from the own vehicle 200, etc.) may be specified on the image of another coordinate system obtained by the above.

1 electronic mirror system 10 camera 30 camera ECU
31 road condition detection unit 32 storage unit 33 correction unit 40 grid 200 vehicle (own vehicle)
400 Road L2 Adjacent lane

Claims (5)

Positions of points at the corners of lines separating lanes in a plurality of images captured by a camera mounted on the vehicle at a plurality of time-series different points in time on the road on which the vehicle is traveling. a road condition detection unit that detects the curve condition of the road based on the change in
Positions on the image of each part on the road surface of the road captured in the image and real space in a reference state assuming that the road on which the vehicle is traveling is straight a storage unit that stores correspondence relationships with positions along the road ;
The curve state of the road detected by the road state detection unit based on the time-series change in the position of the corner point of the line separating the lanes, and the state of the road on which the vehicle is traveling The position of the corner point of the line separating the lanes is compared with the position predicted to move in the reference state assuming that the road is straight, and the amount of deviation is calculated and stored in the storage unit. a correction unit that corrects the corresponding relationship according to the amount of deviation;
a target recognition unit that detects another vehicle in the image;
The real space of the other vehicle recognized by the object recognition unit based on the correspondence relationship between the coordinate position on the image obtained by the correction by the correction unit and the coordinate position in the real space. and a distance calculation unit that specifies a position along the road in the real space and calculates a distance along the road between the own vehicle and the other vehicle in the real space. The correction unit targets only a partial region in the vertical direction and the horizontal direction behind the vehicle in the image, and the region has a predetermined coordinate in the vertical direction and the horizontal direction that is larger than a minimum value. 2. The calibration device according to claim 1 , wherein said coordinate position in said vertical direction and said horizontal direction on said image is corrected within a range from a value to a predetermined value smaller than a maximum value . Based on the correspondence relationship, the position of the other vehicle recognized by the object recognition unit along the road in the physical space is specified, and the lane in which the other vehicle is traveling in the physical space is specified. 3. The calibration device according to claim 1 or 2, further comprising a driving lane detection unit for detecting a traffic lane. a driving lane detection unit that identifies the lane in which the other vehicle is driving in the physical space based on the correspondence obtained by the correction by the correction unit;
The lane in which the other vehicle is traveling detected by the traveling lane detection unit is a lane adjacent to the lane in which the own vehicle is traveling, and the distance calculated by the distance calculation unit is in time series. 3. The calibration device according to claim 1, further comprising a notification output unit for outputting a signal for notifying that the other vehicle is approaching when the time is short. a camera mounted on the own vehicle and provided at a portion where a reflecting mirror should be provided, for photographing the exterior of the own vehicle;
an image display device provided in the passenger compartment of the own vehicle for displaying an image photographed by the camera that is horizontally reversed as if it were reflected by the reflecting mirror;
A calibration device according to claim 4,
The image display device is an electronic mirror system that performs display emphasizing the approaching other vehicle based on the signal output from the notification output unit.

Images