JP2021160435A - Vehicle and image processing device - Google Patents

Abstract

To provide a vehicle and an image processing device which can properly perform image conversion in accordance with the number and arrangement of occupants in a vehicle at the time of image acquisition.SOLUTION: A vehicle 101 comprises: first wheels 3f; second wheels 3r; a vehicle body 2 which can move with the first wheels 3f and the second wheels 3r; a seat for at least one occupant which is arranged in the vehicle body 2; and an imaging part 6s which is arranged in the vehicle body 2 and captures an image including the periphery of the vehicle body 2. The image captured by the imaging part 6s includes at least one of the first wheels 3f and the second wheels 3r. The attitude of the imaging part 6s is determined on the basis of at least one grounding position of the first wheels 3f and the second wheels 3r included in the image. A parameter for image conversion is calculated on the basis of the determined attitude of the imaging part 6s. The image captured by the imaging part 6s is converted so as to be used in prescribed processing by using the calculated parameter for image conversion.SELECTED DRAWING: Figure 14

Description

The present disclosure relates to vehicles and image processing devices.

There is known a technique of capturing an image of the surroundings of a vehicle with a camera or the like attached to a vehicle body and using the captured image for various processes such as parking assistance. The image from the camera is converted and used to suit the desired processing. Image conversion is performed based on, for example, the posture of the camera at the time of calibration.

Japanese Unexamined Patent Publication No. 2016-149711 Special Table 2017-188743

However, the above calibration is performed, for example, at the time of shipment from the factory, with no vehicle occupants from the viewpoint of work efficiency. Therefore, if the number and arrangement of occupants in the vehicle change when the image to be converted is acquired, the posture of the camera may be different from that at the time of calibration. Therefore, image conversion may not be performed properly.

An object of the present disclosure is to provide a vehicle and an image processing device capable of appropriately performing image conversion according to the number and arrangement of occupants in the vehicle at the time of image acquisition.

The vehicle according to the present disclosure is arranged on the first wheel, the second wheel, the vehicle body coupled to the first wheel and the second wheel, and movable by the first wheel and the second wheel, and the vehicle body. The vehicle is provided with a seat for at least one occupant and an imaging unit arranged on the vehicle body and capturing an image including the surroundings of the vehicle body, and the image captured by the imaging unit is The posture of the imaging unit is determined and determined based on at least one of the first wheel and the second wheel, and the first wheel and the second wheel included in the image. The image conversion parameter is calculated based on the posture of the imaging unit, and the calculated image conversion parameter is used to convert the image captured by the imaging unit so as to be subjected to a predetermined process.

According to the vehicle and the image processing device according to the present disclosure, it is possible to appropriately perform image conversion according to the number and arrangement of occupants in the vehicle at the time of image acquisition.

FIG. 1 is a schematic view showing an example of a vehicle including the image processing device according to the first embodiment. FIG. 2 is a block diagram showing an example of the configuration of a vehicle including the image processing device according to the first embodiment. FIG. 3 is a diagram showing an example of the configuration of map data used by the image processing apparatus according to the first embodiment. FIG. 4 is a diagram showing an example of a definition of a posture change of a camera that captures an image to be processed by the image processing apparatus according to the first embodiment. FIG. 5 is a diagram showing an example of a posture change of a camera that captures an image to be processed by the image processing apparatus according to the first embodiment. FIG. 6 is a diagram showing an example in which the image processing device according to the first embodiment converts an image into a format that can be used for parking support processing. FIG. 7 is a diagram showing an example in which the image processing apparatus according to the first embodiment converts an image into a format that can be used to generate a bird's-eye view image. FIG. 8 is a diagram showing an example of an image displayed by the image processing apparatus according to the first embodiment at the start of the parking support process. FIG. 9 is a diagram showing an example of an image displayed by the image processing device according to the first embodiment during the continuation of the parking support process. FIG. 10 is a flow chart showing an example of an image processing procedure by the image processing apparatus according to the first embodiment. FIG. 11 is a diagram showing an example in which the image processing device according to the comparative example converts an image that can be used for parking support processing. FIG. 12 is a block diagram showing an example of the configuration of a vehicle including the image processing device according to the second embodiment. FIG. 13 is a diagram showing an captured image used for determining the height position of the camera by the image processing apparatus according to the second embodiment. FIG. 14 is a schematic view showing the relationship between the height position of the camera and the posture of the vehicle determined by the image processing device according to the second embodiment. FIG. 15 is a diagram showing an captured image used for determining the rotation angle of the camera by the image processing apparatus according to the second embodiment. FIG. 16 is a flow chart showing an example of an image processing procedure by the image processing apparatus according to the second embodiment.

Hereinafter, embodiments of the vehicle and the image processing apparatus according to the present disclosure will be described with reference to the drawings.

[Embodiment 1]
The first embodiment will be described with reference to the drawings.

(Vehicle configuration example)
FIG. 1 is a schematic view showing an example of a vehicle 1 including the image processing device 100 according to the first embodiment. FIG. 1A is a side view of the vehicle 1, and FIG. 1B is a top view of the vehicle 1.

As shown in FIG. 1, the vehicle 1 includes a vehicle body 2 and two pairs of wheels 3 (a pair of front tires 3f and a pair) arranged on the vehicle body 2 along the vehicle length direction (± Y direction) of the vehicle body 2. The rear tire 3r) is provided. The paired two front tires 3f and the paired two rear tires 3r are arranged along the vehicle width direction (± X direction orthogonal to the ± Y direction) of the vehicle body 2, respectively.

The vehicle 1 is both ends of the vehicle body 2 in the ± X direction, is closer to the front tire 3f in the ± Y direction of the vehicle body 2, and is in the vehicle height direction (± X direction and ± Z direction orthogonal to the ± Y direction). A pair of door mirrors 4 are provided at a predetermined height position.

The vehicle 1 includes a plurality of seats 5a to 5d inside the vehicle body 2. The seats 5a and 5b are arranged side by side in the ± X direction near the front tire 3f. The seats 5c and 5d are arranged side by side in the ± X direction near the rear tire 3r. The seat 5c is located behind the seat 5b and the seat 5d is located behind the seat 5a. The number and arrangement of the seats 5 included in the vehicle 1 are not limited to the example of FIG.

The vehicle 1 includes a plurality of cameras 6s, 6f, 6r on a predetermined end surface of the vehicle body 2. The cameras 6s, 6f, 6r as the imaging unit are a visible light camera, a CCD camera capable of detecting light in a wider range, a CMOS camera, and the like. The cameras 6s, 6f, 6r preferably include a wide-angle lens.

The camera 6s is arranged on each of the two door mirrors 4, for example, with the lens facing downward. The camera 6f is arranged on the end surface of the vehicle body 2 on the front tire 3f side, for example, with the lens facing diagonally downward. The camera 6r is arranged on the end surface of the vehicle body 2 on the rear tire 3r side, for example, with the lens facing diagonally downward. The cameras 6s, 6f, and 6r capture images of the surroundings of the vehicle body 2 including the road surface.

In this specification, the end surface of the vehicle body 2 on the front tire 3f side may be referred to as a front surface. Further, the end surface of the vehicle body 2 on the rear tire 3r side may be referred to as a rear surface. Further, both end faces of the vehicle body 2 in the ± X direction may be referred to as side surfaces. Further, when seated in any of the seats 5a to 5d in the vehicle 1, the side surface corresponding to the right side may be referred to as a right side surface, and the side surface corresponding to the left side may be referred to as a left side surface.

Further, in the present specification, the direction toward the left side surface of the vehicle body 2 is the + X direction, and the direction toward the right side surface is the −X direction. Further, the direction toward the front side of the vehicle body 2 is the + Y direction, and the direction toward the rear surface side is the −Y direction. Further, the upward direction of the vehicle body 2 is the + Z direction, and the downward direction (road surface side) is the −Z direction.

Further, in the present specification, when the vehicle 1 is stopped on a road surface having an ideal flat surface, the ± X direction axis (X axis) and the ± Y direction axis (Y axis) of the vehicle 1 are parallel to the road surface. It is assumed that the axis (Z axis) in the ± Z direction of the vehicle 1 is parallel to the normal line with respect to the road surface.

The vehicle 1 can travel using two pairs of wheels 3 arranged along the ± Y direction. In this case, the ± Y direction in which the two pairs of wheels 3 are arranged is the traveling direction (moving direction) of the vehicle 1, and the vehicle 1 is moved forward (traveling in the + Y direction) or backward (traveling in the −Y direction) by switching gears or the like. Can be done. You can also turn left or right by steering.

The image processing device 100 is mounted on the vehicle 1, for example, and performs predetermined conversion on the images from the cameras 6s, 6f, and 6r. The image conversion is, for example, a process of converting the viewpoint so that the image of the road surface captured by the cameras 6s, 6f, 6r from diagonally above becomes an image from directly above. As a result, the image can be used for a predetermined process such as a parking support process described later.

(Configuration example of image processing device)
FIG. 2 is a block diagram showing an example of the configuration of the vehicle 1 including the image processing device 100 according to the first embodiment. As shown in FIG. 2, the vehicle 1 includes an image processing device 100 and a parking support device 200.

The image processing device 100 includes an ECU (Electronic Control Unit) 10, an input device 31, and an output device 32. However, the image processing device 100 may include a camera 6 (cameras 6s, 6f, 6r in FIG. 1), a seatbelt attachment / detachment sensor 7, and a seating sensor 8. The parking support device 200 includes an ECU 10 and a vehicle control device 20. However, the parking support device 200 may include a camera 6, an input device 31, and an output device 32.

The ECU 10 is configured as a computer including, for example, a CPU (Central Processing Unit) 11, a RAM (Random Access Memory) 12, and a ROM (Read Only Memory) 13. The ECU 10 may include a storage device 14 composed of an HDD (Hard Disk Drive) or the like. Further, the ECU 10 includes an I / O (Input / Output) port 15 capable of transmitting and receiving detection signals and various information with various sensors and the like.

Each configuration of the RAM 12, the ROM 13, the storage device 14, and the I / O port 15 of the ECU 10 is configured to be capable of transmitting and receiving various information to and from the CPU 11 via, for example, an internal bus.

The ECU 10 controls parking support processing such as a parking area detection process and a parking route calculation process to the parking area by executing the program installed in the ROM 12 by the CPU 11. Further, the ECU 10 controls image processing such as converting an image from the above-mentioned camera 6 by executing a program installed in the ROM 12 by the CPU 11. The converted image is used, for example, in the parking zone detection process in the parking support process. Further, the ECU 10 controls a process of outputting the converted image to the output device 32 described later.

The storage device 14 stores, for example, calibration data (not shown) and map data 14a.

The calibration is performed on the camera 6 at the time of factory shipment of the vehicle 1, for example, in order to cancel the mounting error of the camera 6 that occurs at the time of manufacturing the vehicle 1. By performing calibration, even if there is a slight variation in the posture when the camera 6 is attached, the normal line to the road surface, the plane parallel to the road surface, and the height position (position in the vertical direction) of the road surface are individually determined. It is commonly determined from the viewpoint (image) of the camera 6. The calibration data includes information such as a normal to the road surface as a reference during image processing of the camera 6, a plane parallel to the road surface, and the height position of the road surface.

The map data 14a includes occupant placement information indicating the seating status of the occupants with respect to the plurality of seats 5 (for example, the seats 5a to 5d in FIG. 1) in the vehicle 1. The seating status of the occupants is, for example, the number of occupants in the vehicle 1 and the seating positions on the plurality of seats 5, that is, the arrangement of the occupants. An image conversion parameter used by the ECU 10 to convert an image from the camera 6 is associated with each of the plurality of occupant arrangement information indicating different numbers of occupants and occupant arrangements. Each image conversion parameter has a different set value depending on the number of occupants and the occupant arrangement of the occupant arrangement information associated with itself.

The vehicle control device 20 includes a steering actuator 21, a steering angle sensor 22, an accelerator sensor 23, a brake sensor 24, and a vehicle speed sensor 25. The vehicle control device 20 acquires information indicating the state of each part of the vehicle 1 from the steering angle sensor 22, the accelerator sensor 23, the brake sensor 24, the vehicle speed sensor 25, and the like. Based on this information, the vehicle control device 20 controls the steering actuator 21 while receiving operations such as accelerator and brake by the driver to provide parking support for parking the vehicle 1 in the above-mentioned parking section, for example.

The seatbelt attachment / detachment sensor 7 is a sensor that detects that the occupant is seated in the seat 5 of the vehicle 1. The seatbelt attachment / detachment sensor 7 is attached to each of the seatbelts provided in each seat 5, and detects the attachment / detachment of the seatbelt to which it is attached. The seatbelt attachment / detachment sensor 7 detects the seating of the occupant on the seat 5 by detecting the wearing of the seatbelt.

The seating sensor 8 is, for example, a load sensor that detects a load, and is attached to each seat 5 in the vehicle 1 to detect the weight applied to the seat 5 to which the seat sensor 8 is attached. That is, the seating sensor 8 can also detect that the occupant is seated in the predetermined seat 5.

The ECU 10 receives a signal from, for example, the seatbelt attachment / detachment sensor 7, and determines the number of occupants and the seating position in the vehicle 1. Further, the ECU 10 selects an occupant arrangement information that matches the determined number of occupants and the seating position from the plurality of occupant arrangement information of the map data 14a described above, and the image is associated with the selected occupant arrangement information. The conversion parameters are read from the storage device 14 and used for image conversion.

However, a signal from the seating sensor 8 may be used in place of or in addition to the seatbelt attachment / detachment sensor 7 for determining the number of occupants and the seating position in the vehicle 1.

Each configuration of the steering actuator 21, steering angle sensor 22, accelerator sensor 23, brake sensor 24, vehicle speed sensor 25, etc. included in the vehicle control device 20, as well as the camera 6, seatbelt attachment / detachment sensor 7, seating sensor 8, and ECU 10. It is connected to the I / O port 15 by a bus. The bus connecting these may be, for example, a CAN (Control Area Network) bus or the like.

The output device 32 as a display unit is, for example, a display device such as a liquid crystal monitor that is mounted in the vehicle 1 and can be visually recognized by an occupant in the vehicle 1. On the display device, for example, an image imaged by the camera 6 and converted by the ECU 10 is displayed. That is, the image processing device 100 including the output device 32 also functions as an image output device that outputs the processed image.

The input device 31 is, for example, a touch-type panel integrally molded with the liquid crystal monitor, and receives an input operation by an occupant in the vehicle 1. The occupant can instruct the image processing device 100 to display an image or instruct the parking support device 200 to start the parking support process via the input device 31.

The configurations of the input device 31 and the output device 32 are not limited to the above, and the input device 31 and the output device 32 may be separate bodies.

(Map data configuration example)
FIG. 3 is a diagram showing an example of the configuration of map data 14a used by the image processing apparatus 100 according to the first embodiment. As shown in FIG. 3, the map data 14a stored in the storage device 14 of the ECU 10 has, for example, a plurality of image conversion data D1 to D8.

Each of the image conversion data D1 to D8 is associated with a plurality of occupant arrangement information indicating various seating situations of the occupant. (A) to (D) shown in the image conversion data D1 to D8 correspond to the seats 5a to 5d, respectively, and indicate the seating of the occupant in these seats 5a to 5d.

The image conversion data D1 to D8 include image conversion parameters, respectively. For each image conversion parameter, set values corresponding to the occupant arrangement information associated with the image conversion data D1 to D8 are set.

For example, the image conversion data D1 is associated with occupant arrangement information indicating a seating situation in which one occupant is seated in the seat 5a, and includes an image conversion parameter according to such a seating situation.

Further, for example, the image conversion data D3 is associated with occupant placement information indicating the seating status in which two occupants are seated in the seats 5a and 5c, respectively, and includes image conversion parameters according to such seating status. ..

Further, for example, the image conversion data D6 is associated with the occupant arrangement information indicating the seating status in which the three occupants are seated in the seats 5a, 5c, and 5d, respectively, and the image conversion parameter according to the seating status. including.

As described above, a plurality of image conversion parameters are prepared for each pattern of the seating position of the occupant.

At the time of shipment from the factory of the vehicle 1, the calibration is performed as described above so that the image processing is uniformly performed even if the postures of the individual cameras 6 are different due to the difference in the mounting position. However, if the number of occupants and the arrangement of occupants in the vehicle 1 change, the posture of the camera 6 may be different from that at the time of calibration. The set values of the individual image conversion parameters are determined based on, for example, the posture of the camera 6 that changes according to the seating condition of the occupant.

(Example of camera posture change)
Next, a method of defining the amount of change in the posture of the camera 6 will be described with reference to FIGS. 4 and 5. FIG. 4 is a diagram showing an example of a definition of a posture change of a camera 6 that captures an image to be processed by the image processing apparatus 100 according to the first embodiment. Here, the camera 6s attached to the door mirror 4 will be described as an example. For the other cameras 6f and 6r, the change in posture can be defined by using the method described below. Further, although the camera 6s is drawn so as to have a box shape in the drawing, it may have another shape such as a cylindrical shape.

The amount of change in the posture of the camera 6s can be defined as, for example, the amount of displacement from the reference posture of the camera 6s. The reference posture of the camera 6s is determined based on, for example, the position of the camera 6s at the time of calibration.

As shown in FIG. 4A, the camera 6s is attached to the door mirror 4, for example, with the lens facing downward. The reference posture of the camera 6s can be defined based on, for example, the orientation of the lens of the camera 6s. That is, the optical axis of the camera 6s determined by the direction of the lens can be set to, for example, the Z axis, the downward direction of the lens can be set to the −Z direction, and the upward direction can be set to the + Z direction.

Further, with reference to the Z-axis of the camera 6s, an X-axis oriented in an arbitrary direction in a plane orthogonal to the Z-axis of the camera 6s and a Y oriented in a direction orthogonal to the X-axis in a plane orthogonal to the Z-axis. The axis can be set. As the above-mentioned arbitrary direction, for example, the direction substantially along the vehicle width may be the X-axis, and the direction substantially along the vehicle length may be the Y-axis.

At this time, for example, the direction toward the left side surface of the vehicle body 2 can be set to the + X direction, and the direction toward the right side surface can be set to the −X direction. Further, for example, the direction toward the front surface side of the vehicle body 2 can be set to the + Y direction, and the direction toward the rear surface side can be set to the −Y direction.

As described above, the reference posture of the camera 6s is defined by using, for example, three axes of the X-axis, the Y-axis, and the Z-axis.

It should be noted that the X-axis, Y-axis, and Z-axis set in this way for the camera 6s are not necessarily completely parallel to the X-axis, Y-axis, and Z-axis in the vehicle 1. .. This is because, for example, the optical axis of the camera 6s may not be parallel to the Z axis of the vehicle 1 due to an error in mounting the camera 6s on the vehicle 1.

As described above, when the number of occupants and the arrangement of occupants in the vehicle 1 change, the posture of the camera 6s may also change. The amount of change in the posture of the camera 6s at that time is used as the displacement value of the camera 6s on the X-axis, Y-axis, and Z-axis determined as described above with respect to the posture of the camera 6s at the time of reference calibration. Is regulated.

Specifically, as shown below, the amount of change in the posture of the camera 6s is in the ± X, ± Y, and ± Z directions from the origin, which is the intersection of the X, Y, and Z axes. It can be defined by the amount of movement of the camera 6s and the rotation angle of the camera 6s having the X-axis, the Y-axis, and the Z-axis as the rotation axes.

As shown in FIG. 4B, how much the origin position of the camera 6s deviates from the position at the time of calibration in the ± X direction can be expressed as a movement amount ΔPx in the ± X direction. The movement amount ΔPx in the + X direction of the camera 6s is represented by a value with a plus, for example, and the movement amount ΔPx in the −X direction is represented by a value with a minus, for example.

As shown in FIG. 4C, how much the origin position of the camera 6s deviates from the position at the time of calibration in the ± Y direction can be expressed as a movement amount ΔPy in the ± Y direction. The movement amount ΔPy in the + Y direction of the camera 6s is represented by a value with a plus, for example, and the movement amount ΔPy in the −Y direction is represented by a value with a minus, for example.

As shown in FIG. 4D, how much the origin position of the camera 6s deviates from the position at the time of calibration in the ± Z direction can be expressed as a movement amount ΔPz in the ± Z direction. The movement amount ΔPz in the + Z direction of the camera 6s is represented by a value with a plus, for example, and the movement amount ΔPz in the −Z direction is represented by a value with a minus, for example.

As shown in FIG. 4 (e), how much the camera 6s has rotated from the position at the time of calibration with the X axis as the rotation axis can be expressed as a rotation angle ΔRx on the X axis. In other words, the rotation angle ΔRx is the rotation angle of the camera 6s in the Y-axis and the YZ plane parallel to the Z-axis. When the minus side of the X-axis is viewed from the plus side, when the rotation direction of the camera 6s is counterclockwise, the rotation angle ΔRx is represented by a value with a plus, for example, and when the rotation direction is clockwise. The rotation angle ΔRx is represented by, for example, a value with a minus. The rotation angle ΔRx on the X-axis is also called a pitch angle.

As shown in FIG. 4 (f), how much the camera 6s has rotated from the position at the time of calibration with the Y axis as the rotation axis can be expressed as a rotation angle ΔRy on the Y axis. In other words, the rotation angle ΔRy is the rotation angle of the camera 6s in the X-axis and the XZ plane parallel to the Z-axis. When the minus side of the Y-axis is viewed from the plus side, if the rotation direction of the camera 6s is counterclockwise, the rotation angle ΔRy is represented by a value with a plus, for example, and if the rotation direction is clockwise. The rotation angle ΔRy is represented by, for example, a value with a minus. The rotation angle ΔRy on the Y-axis is also called a roll angle.

As shown in FIG. 4 (g), how much the camera 6s has rotated from the position at the time of calibration with the Z axis as the rotation axis can be expressed as a rotation angle ΔRz in the Z axis. In other words, the rotation angle ΔRz is the rotation angle of the camera 6s in the XY plane parallel to the X-axis and the Y-axis. When looking at the minus side from the plus side of the Z axis, if the rotation direction of the camera 6s is counterclockwise, the rotation angle ΔRz is represented by a value with a plus, for example, and if the rotation direction is clockwise. The rotation angle ΔRz is represented by, for example, a value with a minus. The rotation angle ΔRz on the Z axis is also called a yaw angle.

Next, a method of expressing the amount of change in the posture of the camera 6s will be described with a specific example.

FIG. 5 is a diagram showing an example of a change in the posture of the camera 6 that captures the image to be processed by the image processing device 100 according to the first embodiment. In FIG. 5, the camera 6s attached to the door mirror 4 will be described as an example. For the other cameras 6f and 6r, the amount of change in posture can be expressed by using the method described below.

FIG. 5A shows the state of the vehicle 1 and the camera 6s at the time of calibration. At the time of factory shipment of the vehicle 1, in consideration of work efficiency, for example, calibration is performed with zero occupants in the vehicle 1. Therefore, when the road surface on which the vehicle 1 stops has an ideal flat surface, the X-axis and the Y-axis of the vehicle 1 are parallel to the road surface, and the Z-axis of the vehicle 1 is It is parallel to the normal of the road surface.

On the other hand, the camera 6s is attached to the vehicle 1 in a predetermined posture as a reference. As described above, the X-axis, Y-axis, and Z-axis based on the posture of the camera 6s at this time are not necessarily parallel to the X-axis, Y-axis, and Z-axis of the vehicle 1. However, the X-axis, Y-axis, and Z-axis of the camera 6s and the X-axis, Y-axis, and Z-axis of the vehicle 1 have a predetermined positional relationship.

FIG. 5B shows the state of the vehicle 1 and the camera 6s in the seating state when one occupant is seated at the position of the seat 5a. As shown in FIG. 5B, when the occupant sits on the seat 5a, the vehicle 1 sinks downward, for example, at the position of the seat 5a. It is also conceivable that the vehicle 1 leans forward slightly and tilts toward the right side. As a result, unlike the case of calibration, the X-axis, Y-axis, and Z-axis of the vehicle 1 are tilted with respect to the road surface and the normal of the road surface.

On the other hand, the posture of the camera 6s attached to the vehicle 1 also changes according to the above-mentioned state change of the vehicle 1. However, the positional relationship between the X-axis, Y-axis, and Z-axis of the camera 6s and the X-axis, Y-axis, and Z-axis of the vehicle 1 is maintained.

Therefore, for example, as the vehicle 1 sinks downward at the position of the seat 5a, the movement amount ΔPz of the camera 6s becomes a negative value having a predetermined magnitude.

Further, for example, the movement amount ΔPy of the camera 6s becomes a positive value having a predetermined magnitude as the vehicle 1 leans forward slightly, and the rotation angle ΔRx of the camera 6s becomes a positive value having a predetermined magnitude. It becomes a value.

Further, for example, as the vehicle 1 tilts toward the right side surface, the movement amount ΔPx of the camera 6s becomes a negative value having a predetermined magnitude, and the rotation angle ΔRy of the camera 6s has a predetermined magnitude. It will be a positive value.

As described above, with respect to the seating situation of the occupant on the vehicle 1 in the example of FIG. 5B, the amount of change in the posture of the camera 6s is, for example, ΔPx (−) and ΔPy (+) having predetermined magnitudes, respectively. ), ΔPz (−), ΔRx (+), and ΔRy (+), and ΔRz (± 0).

By the way, the X-axis, Y-axis, and Z-axis in the posture of the reference camera 6s are not only with respect to the X-axis, Y-axis, and Z-axis of the vehicle 1, but also the road surface and the road surface method on which the vehicle 1 stops. It may not be parallel to the line either. Further, the relative height position between the camera 6s and the road surface may be different due to the mounting error of the camera 6s or the like.

Even in such a case, the X-axis and Y-axis parallel to the road surface and Z parallel to the normal to the road surface in the viewpoint through the camera 6s, that is, the image captured by the camera 6s by the above calibration. The axis and the height position of the road surface are determined. At this time, the directions of the X-axis and the Y-axis may be determined so as to be parallel to the X-axis and the Y-axis of the vehicle 1, for example. That is, the X-axis, Y-axis, and Z-axis at the viewpoint through the camera 6s, the X-axis, Y-axis, and Z-axis of the vehicle 1 and the road surface and the normal of the road surface are parallel to each other.

However, for example, as in the example of FIG. 5B above, the X-axis, Y-axis, and Z-axis of the vehicle 1 are no longer parallel to the road surface and the normal of the road surface, and the vehicle 1 When the height changes with respect to the road surface, the plane parallel to the road surface defined for the captured image based on the attitude of the reference camera 6s, the normal to the road surface, and the height position of the road surface become the actual road surface and the road surface. It will not match the normal and height positions.

In such a case, the above-mentioned image conversion parameters of the image conversion data D1 to D8 include a plane parallel to the road surface in the image captured by the camera 6s, a normal to the road surface, a height position of the road surface, and an actual road surface. In addition, it has a set value for matching the normal and height positions of the road surface.

Specifically, in generating the image conversion data D1 to D8, ΔPx, which indicates the amount of change in the posture of the camera 6s in each of the seating conditions of the occupants indicated by the occupant arrangement information associated with the image conversion data D1 to D8, ΔPy, ΔPz, ΔRx, ΔRy, and ΔRz are obtained. The amount of change in the posture of these cameras 6s can be obtained, for example, by simulation or actual measurement.

From the amount of change in the posture of the camera 6s, the plane parallel to the road surface defined for the captured image based on the reference posture of the camera 6s, the normal to the road surface, and the height position of the road surface are determined from time to time. It is possible to know how the actual road surface and the normal and height positions of the road surface correspond to each other. Then, the set value may be determined for each image conversion parameter based on the correspondence between them.

By performing image conversion using such image conversion parameters, the plane parallel to the road surface in the image captured by the camera 6s, the normal to the road surface, the height position of the road surface, the actual road surface, and the method of the road surface Misconversion of the image due to inconsistency between the line and height position is suppressed.

(Example of image conversion)
Next, an example of image conversion by the image processing apparatus 100 of the first embodiment will be described with reference to FIGS. 6 and 7. The image processing device 100 converts the image from the camera 6 into an image used for parking support processing by, for example, the parking support device 200. Further, the image processing device 100 converts the image from the camera 6 into a bird's-eye view image or the like displayed on the liquid crystal monitor or the like which is the output device 32.

FIG. 6 is a diagram showing an example in which the image processing device 100 according to the first embodiment converts an image into a format that can be used for parking support processing. FIG. 6A is an image captured by the camera 6s attached to the door mirror 4. FIG. 6B is an image after converting the captured image of the camera 6s.

As shown in FIG. 6A, when the vehicle 1 is placed side by side in an empty space in the parking lot, the empty space is projected in the image captured by the camera 6s. In the parking support process, the vacant space projected on the image is detected, and the size of the vacant space and the distance from the vehicle 1 to the vacant space are calculated.

However, from the mounting position of the camera 6s, an image looking down from diagonally above is captured. Therefore, it is not possible to correctly calculate the size and distance of the empty space from the image as it is. The image of FIG. 6A is, for example, an image passed through a wide-angle lens included in the camera 6s.

As shown in FIG. 6B, the image processing apparatus 100 converts the image captured by the camera 6s into an image from directly above, for example, by performing viewpoint conversion. At that time, the image processing device 100 detects, for example, the signal of the seatbelt attachment / detachment sensor 7 to determine the seating pattern in the vehicle 1, and performs image conversion using the image conversion parameters corresponding to the determined seating pattern.

The parking support device 200 calculates the size of the vacant space, the distance from the vehicle 1 to the vacant space, and the like based on the converted image. As a result, the size and distance of the empty space are properly calculated.

FIG. 7 is a diagram showing an example in which the image processing device 100 according to the first embodiment converts an image into a format that can be used to generate a bird's-eye view image 40t.

As shown in FIG. 7, the cameras 6s, 6f, and 6r included in the vehicle 1 project the sides, the front, and the rear of the vehicle 1, respectively. These side images 40m and 40h, the front image 40f, and the rear image 40r are images looking down on the surroundings from diagonally above, including, for example, the road surface in the vicinity of the vehicle 1.

The image processing device 100 performs viewpoint conversion by image conversion so that these side images 40m, 40h, front image 40f, and rear image 40r captured from diagonally above are images from directly above. At that time, the image processing device 100 detects, for example, the signal of the seatbelt attachment / detachment sensor 7 to determine the seating pattern in the vehicle 1, and performs image conversion using the image conversion parameters corresponding to the determined seating pattern.

The image processing device 100 generates a bird's-eye view image 40t centered on the vehicle icon 1ic by arranging the converted images on the sides, front, and rear of the vehicle icon 1ic. The vehicle icon 1ic is prepared in advance, for example, and is an icon representing a state in which the vehicle 1 is viewed from above. The image processing device 100 displays the generated bird's-eye view image 40t on the liquid crystal monitor or the like which is the output device 32. As a result, the occupant can recognize the situation around the vehicle 1.

(Image display example)
Next, an example of displaying an image by the image processing apparatus 100 of the first embodiment will be described with reference to FIGS. 8 and 9. The image processing device 100 can display an image on the output device 32 in various situations. Here, an example in which the image processing device 100 displays an image during the parking support processing by the parking support device 200 will be described.

FIG. 8 is a diagram showing an example of an image displayed by the image processing device 100 according to the first embodiment at the start of the parking support process. FIG. 8A is a diagram showing a state of the vehicle 1 at the start of the parking assistance process, and FIG. 8B is an example of an image displayed on the output device 32 at the start of the parking assistance process.

As shown in FIG. 8A, at the start of the parking support process, the vehicle 1 is placed side by side in, for example, an empty space 50 where parking is possible. Then, in this state, for example, the driver or the like performs an operation to start the parking support process.

As shown in FIG. 8B, when the parking support processing start operation is performed, the image processing device 100 outputs the parking support processing start screen 40a to the liquid crystal monitor or the like which is the output device 32. The parking support processing start screen 40a has, for example, a bird's-eye view image 40t generated by the image processing device 100 on the left side and an unconverted image 40h captured by the camera 6s on the right side.

At this time, the image processing device 100 displays the parking frame 50f on the bird's-eye view image 40t and the image 40h. The parking frame 50f is a frame-shaped figure showing the outer shape of the empty space 50, and when the parking support device 200 identifies the empty space 50 as a parkingable section, the bird's-eye view image 40t and the empty space in the image 40h exist. It can be displayed at the position where it is to be displayed. The parking support device 200 is attached to each parking space including, for example, the vacant space 50, and identifies the vacant space 50 by detecting a lane marking or the like that divides these parking spaces.

Further, the image processing device 100 superimposes the image 40h on the message 41 such as "Is this parking space okay?" The "Cancel" button 43 and the like are displayed. When the driver or the like presses the "Yes" button 42, the parking support process for the detected empty space 50 is continued, and when the "Cancel" button 43 is pressed, the parking support process for the empty space 50 is stopped.

When the "Yes" button 42 is pressed and the parking support process is continued, the parking support device 200 calculates the parking route from the current position of the vehicle 1 to the empty space 50. At this time, the parking route calculated by the parking support device 200 includes, for example, a turning frame and a parking frame.

FIG. 9 is a diagram showing an example of an image displayed by the image processing device 100 according to the first embodiment during the continuation of the parking support process. The left figures (a), (c), and (e) of FIG. 9 are views showing the occasional state of the vehicle 1 attempting to park in the empty space 50 with parking assistance. The right figures (b), (d), and (f) of FIG. 9 are views showing a screen displayed on the output device 32 in the case of FIGS. 9 (a), (c), and (e).

As shown in FIG. 9A, the vehicle 1 starts moving diagonally forward to the right so as to temporarily move away from the empty space 50. The vehicle control device 20 of the parking support device 100 controls the steering actuator 21 while receiving operations such as accelerator and brake by the driver to move the vehicle 1 according to a desired parking route.

As shown in FIG. 9B, when the vehicle 1 is in the state of FIG. 9A, the image processing device 100 displays a continuation screen 40b indicating that the parking support process is ongoing on the output device 32. The image processing device 100 displays a bird's-eye view image 40t on the left side of the continuation screen 40b, in which the surrounding image changes according to the movement of the vehicle 1 while keeping the vehicle icon 1ic at the center. The image processing device 100 displays, for example, unconverted images of any of the cameras 6s, 6f, 6r corresponding to the traveling direction of the vehicle 1 on the right side of the continuation screen 40b. In FIG. 9B, the image 40f of the front camera 6f corresponding to the traveling direction of the vehicle 1 is displayed.

At this time, the image processing device 100 superimposes the bird's-eye view image 40t and the image 40f on the parking route 50r to be followed by the vehicle 1. Further, the image processing device 100 superimposes the bird's-eye view image 40t on the turning frame 50t indicating the turning position of the vehicle 1. Further, the image processing device 100 superimposes the image 40f and displays a message 44 such as "Please stop according to the frame on the left screen."

As shown in FIG. 9C, when the vehicle 1 reaches the turning position, it starts moving to the left rear toward the empty space 50. Also at this time, the vehicle control device 20 controls the steering actuator 21 while receiving operations such as the accelerator and the brake by the driver to move the vehicle 1 according to a desired parking route.

As shown in FIG. 9D, when the vehicle 1 is in the state of FIG. 9C, the image processing device 100 displays the continuation screen 40c on the output device 32. The image processing device 100 displays a bird's-eye view image 40t on the left side of the continuous screen 40c, and displays an unprocessed image 40r from the rear camera 6r corresponding to the traveling direction of the vehicle 1 on the right side of the continuous screen 40c.

At this time, the image processing device 100 superimposes the bird's-eye view image 40t and the image 40r on the parking route 50r to be followed by the vehicle 1. Further, the image processing device 100 superimposes the bird's-eye view image 40t on the parking frame 50f indicating the target parking position of the vehicle 1. Further, the image processing device 100 superimposes the image 40r and displays a message 45 such as "Please stop according to the frame on the left screen."

As shown in FIG. 9E, the vehicle 1 completes parking in the empty space 50.

As shown in FIG. 9F, at this time, the image processing device 100 displays the parking support processing end screen 40d on the output device 32. The image processing device 100 displays a bird's-eye view image 40t on the left side of the end screen 40d, and displays an unprocessed image 40r from the camera 6r on the right side of the end screen 40d.

Further, the image processing device 100 superimposes the image 40r on the image 40r and displays a message 46 such as "Please park the shift."

(Processing example of image processing device)
Next, an example of image processing by the image processing apparatus 100 of the first embodiment will be described with reference to FIG. FIG. 10 is a flow chart showing an example of an image processing procedure by the image processing apparatus 100 according to the first embodiment. Hereinafter, the image processing of the image processing device 100 during the execution of the parking support processing by the parking support device 200 will be described. Therefore, FIG. 10 includes processing by the parking support device 200.

As shown in FIG. 10, the ECU 10 receives an operation instructing the start of the parking support process from the input device 31 such as a touch panel (step S101).

In response to this, the ECU 10 acquires a signal from, for example, the seatbelt attachment / detachment sensor 7 (step S102). The ECU 10 identifies the seating patterns of the occupants for the plurality of seats 5a to 5d based on the acquired signals of the seatbelt attachment / detachment sensor 7 (step S103).

The ECU 10 selects the image conversion data corresponding to the specified seating pattern from the plurality of image conversion data D1 to D8 of the map data 14a (step S104).

The ECU 10 acquires images captured by the cameras 6s, 6f, and 6r (step S105).

The ECU 10 converts images from the cameras 6s, 6f, and 6r using the image conversion parameters included in the selected image conversion data (step S106).

With reference to the examples of FIGS. 8 and 9 described above, the ECU 10 converts, for example, the image of the camera 6s into a format that can be used for parking assistance processing. Further, the ECU 10 converts the images from the cameras 6s, 6f, and 6r into a bird's-eye view image 40t format in which the occupant can recognize the surrounding situation of the vehicle 1.

The ECU 10 detects the parking frame 50f based on the converted image from the camera 6s converted into a format that can be used for the parking support process (step S107).

The ECU 10 starts outputting the bird's-eye view image 40t on which the parking frame 50f is superimposed and the unconverted image from the cameras 6s, 6f, 6r to the output device 32 (step S108).

The ECU 10 calculates the parking route 50r to the parking frame 50f based on the converted image from the camera 6s converted into a format that can be used for the parking support process (step S109).

The vehicle control device 20 controls the steering actuator 21 while receiving operations such as accelerator and brake by the driver according to the parking path 50r calculated by the ECU 10, and guides the vehicle 1 to the parking frame 50f (step S110). .. The vehicle control device 20 stops the vehicle 1 in the parking frame 50f (step S111).

The ECU 10 stops the output of the bird's-eye view image 40t and the unconverted images from the cameras 6s, 6f, 6r to the output device 32 (step S112).

The processes of steps S101 to S112 shown above do not necessarily have to be executed in this order, and the processes whose order can be changed may be executed in a different order. Further, one process may be performed in parallel with one or more other processes.

Further, the ECU 10 decides to start the image output in step S108 and stop the image output in step S112, but the image output start timing and the stop timing are other timings. There may be. The ECU 10 may continuously output the image without stopping the output of the image.

(Comparison example)
The image processing apparatus of the comparative example has only one image conversion data used for image conversion, for example. This image conversion data is set based on, for example, the posture of the camera at the time of calibration performed with zero occupants.

However, when the number of occupants in the vehicle 1 and the seating arrangement pattern of the occupants change, the posture of the camera also changes. Therefore, even if the image conversion data based on the posture of the camera at the time of calibration is used, the image may not be converted properly.

As an example, when four passengers are in the driver's seat and the vehicle 1 is sunk downward as a whole, the image of the camera attached to the door mirror (FIG. 11 (a)) is converted. In the converted image (FIG. 11B), of the lane markings Lf and Lr attached to the parking space, the lane marking Lf in front of the vehicle may shift to a position closer to the front than it actually is. be. In addition, the lane marking Lr behind the vehicle may shift to a position closer to the rear than it actually is. As a result, the target parking position also shifts toward the rear of the vehicle 1 from the proper position, and the parking space is detected wider than it actually is.

According to the image processing device 100 of the first embodiment, the ECU 10 selects an image conversion parameter based on, for example, the seating condition of the occupant acquired from the seatbelt attachment / detachment sensor 7. Further, the ECU 10 uses the selected image conversion parameters to convert the image captured by the camera 6 so as to be subjected to a predetermined process. As a result, the image can be appropriately converted according to the number and arrangement of the occupants in the vehicle 1 at the time of image acquisition.

According to the image processing device 100 of the first embodiment, the ECU 10 outputs the bird's-eye view image 40t converted using the selected image conversion parameters to the output device 32. As a result, the occupant can properly recognize the situation around the vehicle 1.

According to the image processing device 100 of the first embodiment, the ECU 10 uses the selected image conversion parameters to convert the image into a format in which the parking support device 200 can detect the parking frame 50f. As a result, the parking support device 200 can appropriately calculate the parking route 50r from the current position of the vehicle 1 to the parking frame 50f.

[Embodiment 2]
The second embodiment will be described with reference to the drawings. The configuration of the second embodiment is different from the configuration of the first embodiment in that the camera posture is determined based on the image captured by the camera and the parameters for image conversion are calculated.

(Configuration example of image processing device)
FIG. 12 is a block diagram showing an example of the configuration of the vehicle 101 including the image processing device 130 according to the second embodiment. As shown in FIG. 12, the vehicle 101 includes an image processing device 130 and a parking support device 230.

The vehicle 101 of the second embodiment also has the vehicle body 2, the wheels 3, the door mirrors 4, the seats 5a to 5d, and the cameras 6s, 6f, 6r as the imaging unit, as in the vehicle 1 of the above-described first embodiment. , At least (see FIG. 1). Further, in the following description, the same components as those in the first embodiment are designated by the same reference numerals, and the description thereof will be omitted.

The image processing device 130 includes an ECU 110, an input device 31, and an output device 32. However, the image processing device 130 may include the camera 6. The parking support device 230 includes an ECU 110 and a vehicle control device 20. However, the parking support device 230 may include a camera 6, an input device 31, and an output device 32.

The ECU 110 is configured as a computer including, for example, a CPU 111, a RAM 112, and a ROM 113. A storage device 114 composed of an HDD or the like may be built in the ECU 110. Further, the ECU 110 includes an I / O port 115 capable of transmitting and receiving detection signals and various information to and from various sensors and the like.

Each configuration of the RAM 112, ROM 113, storage device 114, and I / O port 115 of the ECU 110 is configured to be capable of transmitting and receiving various information to and from the CPU 111, for example, via an internal bus.

The ECU 110 controls parking support processing such as detection processing of an empty space in which the vehicle 101 can be parked and calculation processing of a parking route to the empty space by executing the program installed in the ROM 112 by the CPU 111.

Further, the ECU 110 controls image processing such as converting the image from the camera 6 described above by executing the program installed in the ROM 112 by the CPU 111. The converted image is used, for example, in a parking support process for detecting an empty space and for generating a bird's-eye view image. Further, the ECU 110 controls a process of outputting the converted image to the output device 32 described later.

The storage device 114 stores, for example, calibration data (not shown) and standard time image conversion data 114a. The standard time image conversion data 114a stores image conversion parameters based on the posture of the camera 6 in the standard time. The posture of the camera 6 in the standard time is, for example, the posture of the camera 6 at the time of calibration, and for example, the posture of the camera 6 when there are no occupants in the vehicle 101. The posture of the camera 6 is defined by using, for example, three axes of the X-axis, the Y-axis, and the Z-axis, as in the first embodiment described above.

Here, the X-axis, Y-axis, and Z-axis that define the posture of the camera 6 are an axis substantially along the vehicle width direction of the vehicle 101, an axis generally along the vehicle length direction of the vehicle 101, and an optical axis of the camera 6, respectively. It is an axis parallel to. The fact that the X-axis of the camera 6 "along the vehicle width direction" is caused not only when the X-axis of the camera 6 is completely parallel to the vehicle width direction, but also due to, for example, an error in mounting the camera 6 on the vehicle 101. Including the case where they are substantially parallel within the error range. The Y-axis of the camera 6 "along the vehicle length direction" is caused not only when the Y-axis of the camera 6 is completely parallel to the vehicle length direction, but also due to, for example, an error in mounting the camera 6 on the vehicle 101. Including the case where they are substantially parallel within the error range.

The vehicle control device 20 is configured in the same manner as in the first embodiment described above, for example. The input device 31 and the output device 32 as a display unit are configured in the same manner as in the first embodiment described above, for example.

(Camera posture judgment example)
Next, how the image processing device 130 of the second embodiment determines the actual posture of the camera 6 will be described with reference to FIGS. 13 to 15. The posture of the camera 6 is determined by the image processing device 130 in two stages, for example, the height position of the camera 6 (position on the Z axis) and the pitch angle of the camera 6 (rotation angle on the X axis). Will be implemented.

FIG. 13 is a diagram showing an captured image used for determining the height position of the camera 6s by the image processing device 130 according to the second embodiment. The captured image shown in FIG. 13 is an image captured by the camera 6s attached to the door mirror 4 of the vehicle 101. The right hand of the paper in FIG. 13 is the front side of the vehicle 101, and the left hand of the paper is the rear side of the vehicle 101.

FIG. 13A shows a captured image when there are no occupants in the vehicle 101.

As shown in FIG. 13A, the image captured by the camera 6s includes a part of the vehicle body 2, the front tire 3f on one side, and the ground contact position Ps between the front tire 3f and the road surface or the like. From such a captured image, it is possible to know at which position the ground contact position Ps between the front tire 3f and the road surface or the like is relative to the overall vertical width Fw of the captured image. In FIG. 13A, since there are no occupants in the vehicle 101 and the posture of the camera 6s is the captured image at the standard time, the height position in the captured image of the ground contact position Ps indicates the standard height Fs. ing.

FIG. 13B shows a captured image when there is an occupant in the vehicle 101.

As shown in FIG. 13B, in the captured image when there is an occupant in the vehicle 101, the ground contact position Pa between the front tire 3f and the road surface or the like is shifted downward in the image, and the ground contact position Ps is in the captured image. Is a height Fa lower than the above standard height Fs. This means that the vehicle body 2 has sunk due to the weight of the occupants in the vehicle 101, and the distance from the camera 6s to the road surface has become shorter.

The image processing device 130 determines the actual height of the camera 6s from the road surface as one of the postures of the camera 6s based on the height Fa obtained from the captured image of FIG. 13B.

In determining the height of the camera 6s, the image processing device 130 may use only one of the images of the pair of cameras 6s attached to both sides of the vehicle 101, or both. Images may be used.

Further, since the positions of the cameras 6s on the X-axis and the Y-axis, which can be changed depending on the occupant arrangement in the vehicle 101, are considered to be negligibly small in the image conversion process, the image processing device 130 is described as described above, for example. The height (position on the Z axis) of the camera 6s is mainly determined.

From the height of the camera 6s from the road surface determined as described above, some states can be assumed as the actual posture of the camera 6s. FIG. 14 shows some states assumed from the determined height of the camera 6s.

FIG. 14 is a schematic view showing the relationship between the height position of the camera 6s and the posture of the vehicle 101 determined by the image processing device 130 according to the second embodiment.

FIG. 14A shows the state of the vehicle 101 when the image processing device 130 determines that the camera 6s has a standard height Ds. The standard height Ds is obtained from, for example, the height Fs in the captured image of the ground contact position Ps in FIG. 13 (a) described above.

As shown in FIG. 14A, if the height from the road surface to the camera 6s is the standard height Ds, the posture of the vehicle 101 is also in the standard state, and it is considered that the number of occupants in the vehicle 101 is, for example, zero.

FIG. 14B shows an example of the state of the vehicle 101 when the image processing device 130 determines that the camera 6s has a height Da lower than the standard. Such a height Da can be obtained, for example, from the height Fa in the captured image of the ground contact position Pa in FIG. 13 (b) described above.

As shown in FIG. 14B, when the camera 6s has a height Da lower than the standard, as an example, the entire vehicle 101 is uniformly sunk, so that the distance between the road surface and the camera 6s is short. It is possible that it is in a state of being. Such a state of the vehicle 101 may occur, for example, when the same number of occupants are seated in the front and rear seats of the vehicle 101.

FIG. 14C shows another example of the state of the vehicle 101 when the image processing device 130 determines that the camera 6s has a height Da lower than the standard.

As shown in FIG. 14B, when the camera 6s has a height Da lower than the standard, as another example, the distance between the road surface and the camera 6s is short because the front part of the vehicle 101 has sunk. It is possible that it is in a state of being. That is, the pitch angle of the vehicle 101 is shifted in the positive direction in the example of the first embodiment (see FIG. 4). Such a state of the vehicle 101 is, for example, when an occupant is seated only in the front seat of the vehicle 101, or when more occupants are seated in the front seat of the vehicle 101 than in the rear seat. Can occur in.

The image processing device 130 determines the rotation angle of the camera 6s in order to grasp the posture of the camera 6s in more detail and improve the accuracy of the image conversion. The roll angle (rotation angle on the Y-axis) and yaw angle (rotation angle on the Z-axis) that can change depending on the occupant arrangement in the vehicle 101 are considered to be negligibly small in the image conversion process, and thus the image processing apparatus. The 130 mainly determines the pitch angle of the camera 6s, for example.

When the image processing device 130 determines the pitch angle, there are, for example, a plurality of lane markings for partitioning the parking space in the vicinity of the side of the vehicle 101, and these lane markings are included in the captured image of the camera 6s. It is assumed that it is.

Such a situation may occur, for example, when the driver of the vehicle 101 lays the vehicle 101 sideways in a predetermined empty space. That is, such a situation can often occur, for example, in the early stages when the driver of the vehicle 101 attempts to park the vehicle 101 in a predetermined empty space.

FIG. 15 is a diagram showing an captured image used for determining the rotation angle of the camera 6s by the image processing device 130 according to the second embodiment.

As shown in FIG. 15A, the image captured by the camera 6s includes, for example, lane markings Lf and Lr that demarcate the width direction of the parking space located near the side of the vehicle 101. The lane marking Lf is a lane marking line located on the front side of the vehicle 101. The lane marking Lr is a lane marking line located on the rear side of the vehicle 101.

As shown in FIG. 15 (b), for example, when the image of FIG. 15 (a) is converted using the image conversion parameters included in the standard time image conversion data 114a, the lane markings Lf and Lr are converted as lines parallel to each other. It shall be done. The image of FIG. 15B shows a state in which the image is converted into an image from directly above by the image conversion process.

Here, the actual lane markings Lf and Lr of the parking space should usually be arranged parallel to each other. Assuming that the actual division lines Lf and Lr are parallel to each other, the image processing device 130 assumes that the pitch angle of the camera 6s is not shifted and is in the standard position from the image of FIG. 15B above. judge.

It should be noted that the determination as to whether or not the division lines Lf and Lr in the captured image are parallel to each other is determined based on the respective angles of the division lines Lf and Lr after the image conversion. The respective angles of the lane markings Lf and Lr can be, for example, angles with respect to the X axis (axis generally along the vehicle width direction) as a virtual line defining the posture of the camera 6s. The difference or ratio of the angles θf and θr of these lane markings Lf and Lr is defined as the parallelism of these lane markings Lf and Lr, and if the parallelism is within a predetermined value, the lane markings Lf and Lr are parallel to each other. Can be determined.

As shown in FIG. 15 (c), for example, when the image of FIG. 15 (a) is converted using the image conversion parameters included in the standard time image conversion data 114a, the lane markings Lf and Lr are not converted as parallel lines. It shall be. The image of FIG. 15C shows a state in which the image is converted into an image from directly above by the image conversion process.

It is considered that the reason why the lane markings Lf and Lr are not parallel in this way is that, for example, the vehicle 101 is in the posture as shown in FIG. 14C described above. For example, there is a case where one occupant is evenly seated in the front seat of the vehicle 101 in the vehicle width direction (in the case of two passengers). When such a conversion error occurs in the captured image, the error may be larger in the lane marking Lr behind the vehicle 101 than, for example, the lane marking Lf in front of the vehicle 101 among the lane markings Lf and Lr. ..

The image processing apparatus 130 calculates the parallelism from the angles θf and θr of the converted lane markings Lf and Lr, and the lane markings Lf and Lr are not parallel to each other because the parallelism exceeds a predetermined value. Is determined. That is, in this case, it is determined that the pitch angle of the camera 6s is shifted.

Further, the image processing apparatus 130 corrects the angles θf and θr of the converted division lines Lf and Lr, and calculates the correction value required when the lines are parallel to each other. In order to calculate the correction value, for example, there is a method of actually changing the image conversion parameters in various ways and determining the parallelism of the lane markings Lf and Lr in each case. The image processing device 130 further determines the actual pitch angle of the camera 6s from the obtained correction value.

When calculating the parallelism of the lane markings Lf and Lr, the image processing device 130 does not always convert the captured image including the lane markings Lf and Lr to actually generate the converted image. The image processing device 130 can calculate the parallelism of the lane markings Lf and Lr directly from the unconverted image by arithmetic processing without generating the converted captured image.

Further, when calculating the correction value for correcting the angles θf and θr of the lane markings Lf and Lr, the image processing device 130 corrects the angles θf and θr of the lane markings Lf and Lr so that the lane markings Lf and Lr become mutually exclusive. It is not always the case that an image with parallel lines is actually generated. The image processing device 130 can calculate the correction value by arithmetic processing without correcting the captured image.

From the above, the image processing device 130 can determine the height of the camera 6s from the road surface and the actual posture of the camera 6s including at least the pitch angle of the camera 6s. The image processing device 130 calculates an image conversion parameter based on the obtained posture of the camera 6s. Further, the image processing device 130 converts the image captured by the camera 6s using the calculated image conversion parameters.

In the image conversion process by the image processing device 130, as in the case of the first embodiment, for example, in the parking support process by the parking support device 230, the vacant space is detected, and the size of the vacant space and the distance to the vacant space are determined. The process of converting the image into an image used for the above, and the process of converting the image into a bird's-eye view image displayed on the liquid crystal monitor or the like which is the output device 32 are included.

Similar to the case of the first embodiment, the image processing device 130 may display, for example, a bird's-eye view image after conversion or an image converted so that it can be used in the parking support process on the output device 32. The image processing device 130 may display the unconverted image as it is on the output device 32 as in the case of the first embodiment described above.

(Processing example of image processing device)
Next, an example of image processing by the image processing apparatus 130 of the second embodiment will be described with reference to FIG. FIG. 16 is a flow chart showing an example of an image processing procedure by the image processing apparatus 130 according to the second embodiment. Hereinafter, the image processing of the image processing device 130 during the execution of the parking support processing by the parking support device 230 will be described. Therefore, FIG. 16 includes processing by the parking support device 230.

As shown in FIG. 16, the ECU 110 receives an operation instructing the start of the parking support process from the input device 31 such as a touch panel (step S201).

The ECU 110 acquires at least an image captured by the camera 6s among the cameras 6s, 6f, and 6r (step S202).

The ECU 110 detects, for example, the ground contact position between the front tire 3f of the vehicle 101 and the road surface from the captured image of the camera 6s (step S203).

The ECU 110 calculates the height of the camera 6s based on the detected ground contact position (step S204). That is, the ECU 110 detects the height position of the detected grounding position in the captured image. Further, the ECU 110 calculates the height of the camera 6s from the road surface from the height position in the captured image of the ground contact position.

The ECU 110 detects, for example, a plurality of division lines for dividing the vehicle width direction of the parking space from the captured image of the camera 6s (step S205).

The ECU 110 calculates the parallelism of the detected lane markings (step S206). That is, the ECU 110 calculates the angle of the detected lane marking with respect to the X-axis of the camera 6s. Further, the ECU 110 calculates the parallelism of the lane markings from each angle of the lane markings calculated.

The ECU 110 determines whether or not the parallelism of the lane markings is within a predetermined value (step S207). If the parallelism of the lane markings is within a predetermined value (step S207: Yes), the ECU 110 proceeds to the process of step S210 assuming that the pitch angle of the camera 6s is the standard position.

When the parallelism of the lane markings exceeds a predetermined value (step S207: No), the ECU 110 calculates a correction value for the lane markings to be parallel (step S208). That is, the ECU 110 calculates a correction value capable of correcting the angle of the lane marking so that the parallelism of the lane marking is within a predetermined value.

The ECU 110 calculates the pitch angle of the camera 6s from the calculated correction value (step S209).

The ECU 110 calculates an image conversion parameter based on the posture of the camera 6s including the acquired height and pitch angle of the camera 6s (step S210).

The ECU 110 converts the image from the camera 6 using the calculated image conversion parameters (step S211). At this time, the captured images from the plurality of cameras 6s, 6f, 6r included in the vehicle 101 may be converted.

Subsequent parking support processing in steps S307 to S312 by the parking support device 230 is performed in the same manner as the parking support processing in steps S107 to S112 by the parking support device 200 in the first embodiment described above.

The processes of steps S201 to S211 and S307 to S312 shown above do not necessarily have to be executed in this order, and the processes whose order can be changed may be executed in a different order. Further, one process may be performed in parallel with one or more other processes.

Further, the ECU 110 decides to start the image output in step S308 and stop the image output in step S312, but the image output start timing and the stop timing are other timings. There may be. The ECU 110 may continuously output the image without stopping the output of the image.

As described above, the image processing by the image processing apparatus 130 of the second embodiment is completed.

According to the image processing device 130 of the second embodiment, the ECU 110 determines the posture of the camera 6s based on the ground contact position between the front tire 3f and the road surface included in the captured image of the camera 6s, and the image conversion parameter corresponding to the determination. Is calculated. As a result, the image can be appropriately converted according to the posture of the camera 6s that changes according to the number and arrangement of the occupants in the vehicle 101 at the time of image acquisition.

According to the image processing device 130 of the second embodiment, the ECU 110 determines the posture of the camera 6s based on the parallelism of the lane markings Lf and Lr of the parking space included in the captured image of the camera 6s, and converts the image accordingly. Calculate the parameters for. As a result, the image can be appropriately converted according to the posture of the camera 6s that changes according to the number and arrangement of the occupants in the vehicle 101 at the time of image acquisition.

According to the image processing device 130 of the second embodiment, the ECU 110 outputs a bird's-eye view image converted using the above-mentioned image conversion parameters to the output device 32, and is also converted using the above-mentioned image conversion parameters. The parking support device 230 is made to detect a parking frame or the like from the image. As a result, for example, the occupant who sees the output image can properly recognize the situation around the vehicle 101, and the accuracy of parking support by the parking support device 230 is improved.

The image processing device 130 of the second embodiment described above determines the height of the camera 6s from the ground contact position between the front tire 3f and the road surface included in the captured image of the camera 6s. However, the captured image of the camera 6s may include the ground contact position between the rear tire 3r and the road surface, and by adding the camera to an appropriate position of the vehicle, the ground contact position between the rear tire 3r and the road surface is included. The captured image may be acquired separately. In this case, the image processing device may determine the height of the camera only based on the ground contact position between the rear tire 3r and the road surface. Alternatively, the image processing device may determine the height of the camera based on both the ground contact position between the front tire 3f and the road surface and the ground contact position between the rear tire 3r and the road surface. Further, when determining the pitch angle of the camera, the image processing device determines the pitch angle from the height on the front tire 3f side and the height on the rear tire 3r side in place of or in addition to the method using the above-mentioned division line or the like. You may judge.

[Other Embodiments]
In the above-described first and second embodiments, the image processing devices 100 and 130 are used for the parking support processing, and the images displayed during the execution of the parking support processing are converted. However, the images to be processed by the image processing apparatus are not limited to these.

In the above-described first and second embodiments, the image processing devices 100 and 130 are mounted on the vehicle 1 having four wheels, but the present invention is not limited to this. The image processing device can be mounted on a moving body having two or more wheels such as a motorbike.

Although some embodiments of the present disclosure have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other embodiments, and various omissions, replacements, and changes can be made without departing from the gist of the invention. These embodiments and variations thereof are included in the scope and gist of the invention, as well as in the scope of the invention described in the claims and the equivalent scope thereof.

1,101 Vehicle 2 Body 3 Wheels 4 Door mirrors 5 (5a-5d) Seats 6 (6f, 6r, 6s) Camera 7 Seatbelt attachment / detachment sensor 8 Seating sensor 10,110 ECU
11,111 CPU
12,112 RAM
13,113 ROM
14,114 Storage device 14a Map data 15,115 I / O port 20 Vehicle control device 31 Input device 32 Output device 100, 130 Image processing device 114a Standard time image conversion data 200, 230 Parking support device

Claims (20)

With the first wheel
With the second wheel
A vehicle body coupled to the first wheel and the second wheel and movable by the first wheel and the second wheel.
Seats for at least one occupant placed on the vehicle body,
A vehicle that is arranged on the vehicle body and includes an imaging unit that captures an image including the surroundings of the vehicle body.
The image captured by the imaging unit includes at least one of the first wheel and the second wheel.
The posture of the imaging unit is determined based on at least one ground contact position of the first wheel and the second wheel included in the image.
An image conversion parameter is calculated based on the determined posture of the imaging unit, and the image conversion parameter is calculated.
Using the calculated image conversion parameters, the image captured by the imaging unit is converted so as to be subjected to a predetermined process.
vehicle. The vehicle according to claim 1.
The posture of the imaging unit to be determined includes at least the height of the imaging unit from the ground contact position.
vehicle. The vehicle according to claim 1 or 2.
When the image captured by the imaging unit includes a plurality of lines that demarcate the vehicle width direction of the parking space.
In addition to the ground contact position, the posture of the imaging unit is determined based on the parallelism of the plurality of lines that partition the vehicle width direction of the parking space.
vehicle. The vehicle according to claim 3.
The parallelism of the plurality of lines dividing the parking space in the vehicle width direction is
Obtained from the respective angles of the plurality of lines with respect to the virtual line parallel to the vehicle width direction of the vehicle body.
vehicle. The vehicle according to claim 3 or 4.
The posture of the image pickup unit to be determined includes at least the pitch angle of the image pickup unit whose rotation axis is an axis along the vehicle width direction of the vehicle body.
vehicle. The vehicle according to any one of claims 1 to 5.
The imaging unit is arranged on the side surface side of the vehicle body.
vehicle. The vehicle according to any one of claims 1 to 6.
Further equipped with a display unit that can be visually recognized by the occupant,
The predetermined process is a process of displaying the image converted into a format in which the occupant can recognize the surrounding situation of the vehicle body on the display unit.
vehicle. The vehicle according to claim 7.
The image displayed on the display unit is a bird's-eye view image including the vehicle.
vehicle. The vehicle according to any one of claims 1 to 8.
It is configured so that the route from the current position of the vehicle to a predetermined parking frame can be calculated.
The predetermined process is a process of detecting the predetermined parking frame from the image obtained by converting the parking frame into a detectable format.
vehicle. The vehicle according to claim 9.
A vehicle control device for controlling at least a steering angle based on the path is provided.
vehicle. With the first wheel
With the second wheel
A vehicle body coupled to the first wheel and the second wheel and movable by the first wheel and the second wheel.
Seats for at least one occupant placed on the vehicle body,
An image processing device that is arranged on the vehicle body and can be mounted on a vehicle including an imaging unit that captures an image including the surroundings of the vehicle body.
The image captured by the imaging unit includes at least one of the first wheel and the second wheel.
The posture of the imaging unit is determined based on at least one ground contact position of the first wheel and the second wheel included in the image.
An image conversion parameter is calculated based on the determined posture of the imaging unit, and the image conversion parameter is calculated.
Using the calculated image conversion parameters, the image captured by the imaging unit is converted so as to be subjected to a predetermined process.
Image processing device. The image processing apparatus according to claim 11.
The posture of the imaging unit to be determined includes at least the height of the imaging unit from the ground contact position.
Image processing device. The image processing apparatus according to claim 11 or 12.
When the image captured by the imaging unit includes a plurality of lines that demarcate the vehicle width direction of the parking space.
In addition to the ground contact position, the posture of the imaging unit is determined based on the parallelism of the plurality of lines that partition the vehicle width direction of the parking space.
Image processing device. The image processing apparatus according to claim 13.
The parallelism of the plurality of lines dividing the parking space in the vehicle width direction is
Obtained from the respective angles of the plurality of lines with respect to the virtual line parallel to the vehicle width direction of the vehicle body.
Image processing device. The image processing apparatus according to claim 13 or 14.
The posture of the image pickup unit to be determined includes at least the pitch angle of the image pickup unit whose rotation axis is an axis along the vehicle width direction of the vehicle body.
Image processing device. The image processing apparatus according to any one of claims 11 to 15.
The imaging unit is arranged on the side surface side of the vehicle body.
Image processing device. The image processing apparatus according to any one of claims 11 to 16.
The vehicle further includes a display unit that can be visually recognized by the occupant.
The predetermined process is a process of displaying the image converted into a format in which the occupant can recognize the surrounding situation of the vehicle body on the display unit.
Image processing device. The image processing apparatus according to claim 17.
The image displayed on the display unit is a bird's-eye view image including the vehicle.
Image processing device. The image processing apparatus according to any one of claims 11 to 18.
It is configured so that the route from the current position of the vehicle to a predetermined parking frame can be calculated.
The predetermined process is a process of detecting the predetermined parking frame from the image obtained by converting the parking frame into a detectable format.
Image processing device. The image processing apparatus according to claim 19.
The vehicle includes a vehicle control device that controls at least a steering angle based on the route.
Image processing device.

Images