JP7217090B2 - Information providing device and information providing method - Google Patents

Description

The present invention relates to technology for providing information to vehicle occupants.

Vehicles capable of parking assistance and automatic parking are already on the market. Parking assistance and automatic parking require the setting of a parking position, but when setting a parking position using an image captured by an in-vehicle camera, it is possible to detect parking position candidates when passing and indicate them on the rear camera image. , the position is difficult to understand, and it cannot be said that it is easy to select. To solve this problem, it is conceivable to display parking position candidates on the bird's-eye view image. For example, in FIG. 8, empty spaces P1 to P3 existing within a display range R1 of a bird's-eye view image generated from an image captured by an on-board camera of the vehicle V1 can be selected as parking position candidates. In FIG. 9, since the vehicle V1 has moved forward, empty spaces P3 to P4 existing within the display range R1 of the bird's-eye view image generated from the captured image captured by the vehicle-mounted camera of the vehicle V1 are selected as parking position candidates. However, the vacant spaces P1 to P2 cannot be selected as parking position candidates.

In order to select the empty spaces P1 to P2 as parking position candidates in the state shown in FIG. 9, it is conceivable to use a past image (for example, an image captured in the state shown in FIG. 8).

JP 2012-253609 A

For example, since the drive recorder disclosed in Patent Document 1 records past images, the past images can be used. However, when the past image recorded by the drive recorder is reproduced and displayed as it is, the user cannot easily grasp the positional relationship between the current position and the position where the photographed image corresponding to the past image was taken. Therefore, it was difficult to effectively use the past image after associating the past image with the current position.

SUMMARY OF THE INVENTION In view of the above problems, it is an object of the present invention to provide an information providing technique that provides a past image that allows the user to easily grasp the positional relationship between the current position and the position where the photographed image corresponding to the past image was photographed. and

An information providing device according to one aspect of the present invention stores at least one of an acquisition unit that acquires a photographed image obtained by a camera that photographs the surroundings of a vehicle, and the photographed image and a viewpoint-converted image based on the photographed image. a storage unit for generating an image representing the own vehicle; and a superimposing unit for superimposing the image representing the own vehicle on the current position of the own vehicle in at least one of the captured image and the viewpoint converted image. In addition, as an image processing process for allowing the user to visually recognize that the image was taken in the past, a transparent colored mask is applied, or an image processing process for superimposing a time going back in the past is performed on the past image. and an image processing unit that performs the processing (first configuration).

In the information providing apparatus having the first configuration, a configuration in which a past image of either the captured image or the viewpoint-transformed image and a current image of either the captured image or the viewpoint-transformed image are output as display images. (Second configuration).

In the information providing apparatus having the first or second configuration, the superimposing unit places the own vehicle at the position of the own vehicle where the photographed image corresponding to the past image in the past image of the viewpoint conversion image was photographed. A configuration (a third configuration) in which an image representing the image is superimposed may be employed.

In the information providing apparatus of the third configuration, an image representing the own vehicle superimposed on the current position of the own vehicle in the past image and the photographed image corresponding to the past image in the past image of the viewpoint conversion image are photographed. The image representing the own vehicle superimposed on the position of the own vehicle thus displayed may be displayed in a different display mode ( fourth configuration).

An information providing method according to one aspect of the present invention includes an acquiring step of acquiring a photographed image obtained by a camera that photographs the surroundings of a vehicle, and storing at least one of the photographed image and a viewpoint-converted image based on the photographed image. a generating step of generating an image representing the own vehicle; and a superimposition of superimposing the image representing the own vehicle on the current position of the own vehicle in at least one of the captured image and the viewpoint converted image. and, as an image processing process for allowing the user to visually recognize that the image was taken in the past, a transparent colored mask is applied, or an image processing process for superimposing a time going back in time is applied to the past image. and an image processing step to be applied (fifth configuration).

According to the present invention, it is possible to provide a past image that allows the user to easily grasp the positional relationship between the current position and the position where the photographed image corresponding to the past image was photographed.

A diagram showing a configuration example of an information providing device A diagram illustrating the positions where four in-vehicle cameras are arranged in a vehicle A diagram showing an example of a virtual projection plane A diagram showing an example of a display image A diagram showing an example of a display image A diagram showing an example of a display image A diagram showing an example of a display image Bird's-eye view showing the state of the parking lot Bird's-eye view showing the state of the parking lot

Exemplary embodiments of the invention are described in detail below with reference to the drawings.

<1. Configuration example of information providing device>
FIG. 1 is a diagram showing a configuration example of an information providing apparatus. An information providing device 201 shown in FIG. 1 is mounted in a vehicle such as an automobile. Hereinafter, the vehicle in which the information providing device 201 is mounted is referred to as "own vehicle". In addition, the direction from the driver's seat to the steering wheel, which is the straight traveling direction of the own vehicle, is referred to as "forward". In addition, the direction from the steering wheel to the driver's seat, which is the straight traveling direction of the own vehicle, is called "rear". Also, a direction that is perpendicular to the straight traveling direction and the vertical line of the own vehicle and that extends from the right side to the left side of the driver who is facing forward is referred to as the "left direction." Also, the direction perpendicular to the straight traveling direction and the vertical line of the own vehicle and from the left side to the right side of the driver who is facing forward is referred to as the "right direction."

The front camera 11, the back camera 12, the left side camera 13, the right side camera 14, the information providing device 201, the display device 31, the operation device 32, and the parking frame recognition device 33 shown in FIG. 1 are mounted on the own vehicle.

FIG. 2 is a diagram exemplifying the positions where four in-vehicle cameras (the front camera 11, the back camera 12, the left side camera 13, and the right side camera 14) are arranged in the host vehicle V1.

The front camera 11 is provided at the front end of the own vehicle V1. The optical axis 11a of the front camera 11 extends along the longitudinal direction of the vehicle V1 when viewed from above. The front camera 11 photographs the front direction of the own vehicle V1. The back camera 12 is provided at the rear end of the own vehicle V1. The optical axis 12a of the back camera 12 extends along the longitudinal direction of the vehicle V1 when viewed from above. The back camera 12 photographs the rear direction of the own vehicle V1. The mounting position of the front camera 11 and the rear camera 12 is preferably the left-right center of the vehicle V1, but may be slightly shifted in the left-right direction from the left-right center.

The left side camera 13 is provided on the left door mirror M1 of the own vehicle V1. The optical axis 13a of the left side camera 13 extends along the lateral direction of the vehicle V1 when viewed from above. The left side camera 13 photographs the left direction of the own vehicle V1. The right side camera 14 is provided on the right door mirror M2 of the vehicle V1. The optical axis 14a of the right side camera 14 extends along the left-right direction of the vehicle V1 when viewed from above. The right side camera 14 photographs the right direction of the own vehicle V1. When the own vehicle V1 is a so-called door mirrorless vehicle, the left side camera 13 is attached around the rotation axis (hinge portion) of the left side door without interposing the door mirror, and the right side camera 14 is attached to the right side door. It is attached around the rotating shaft (hinge part) of the door mirror without intervening the door mirror.

The horizontal angle of view θ of each vehicle-mounted camera is 180 degrees or more. Therefore, the four in-vehicle cameras (the front camera 11, the back camera 12, the left side camera 13, and the right side camera 14) can photograph the entire circumference of the vehicle V1 in the horizontal direction. In this embodiment, the number of onboard cameras is four, but the number of onboard cameras required to create a bird's-eye view image described later using the images captured by the onboard cameras is not limited to four. , multiple cameras. As an example, when the horizontal angle of view θ of each in-vehicle camera is relatively wide, a bird's-eye view image may be generated based on three captured images acquired from three cameras, which is less than four. Furthermore, as another example, when the horizontal angle of view θ of each vehicle-mounted camera is relatively narrow, a bird's-eye view image may be generated based on five captured images obtained from five cameras, which is more than four.

Returning to FIG. 1, the four in-vehicle cameras (the front camera 11, the back camera 12, the left side camera 13, and the right side camera 14) output captured images to the information providing device 201 and the parking frame recognition device 33.

The information providing device 201 processes captured images output from four in-vehicle cameras (the front camera 11, the back camera 12, the left side camera 13, and the right side camera 14) and outputs the processed images to the display device 31. do.

The display device 31 is provided at a position where the display screen of the display device 31 can be viewed by the driver of the vehicle, and displays a display image output from the information providing device 201 . Examples of the display device 31 include a display installed in the center console, a meter display installed opposite the driver's seat, and a head-up display that projects an image onto the windshield.

The operation device 32 is a device that generates operation information according to an operation performed by the driver of the own vehicle. As the operation device 32, for example, a push button, a touch panel, or the like can be used. Further, the operating device may have a microphone, recognize a voice uttered by the driver of the vehicle, and generate operation information according to the voice uttered by the driver of the vehicle. In the following description, it is assumed that the operation device 32 is a touch panel integrated with the display device 31 .

The information providing device 201 can be configured by hardware such as ASIC (application specific integrated circuit) or FPGA (field-programmable gate array), or a combination of hardware and software. When the information providing apparatus 201 is configured using software, a block diagram of a portion realized by software represents a functional block diagram of that portion. A function realized using software may be described as a program, and the function may be realized by executing the program on a program execution device. Examples of the program execution device include a computer including a CPU (Central Processing Unit), RAM (Random Access Memory), and ROM (Read Only Memory).

The information providing device 201 includes a captured image acquisition unit 21 , a display control unit 22 , a storage unit 23 and a parking position selection unit 24 .

The captured image acquisition unit 21 acquires analog or digital captured images from the four on-vehicle cameras (front camera 11, back camera 12, left side camera 13, and right side camera 14) at a predetermined cycle (for example, 1/30 second cycle). ) to acquire continuously in time. When the acquired captured image is analog, the captured image acquisition unit 21 converts (A/D converts) the analog captured image into a digital captured image. The captured image acquisition unit 21 outputs the acquired captured image or the acquired and converted captured image to the display control unit 22 .

The display control unit 22 includes a bird's-eye view image generation unit 22a, an own vehicle image generation unit 22b, a superimposition unit 22c, and an image processing unit 22d.

The bird's-eye view image generation unit 22a projects the captured image acquired by the captured image acquisition unit 21 onto a virtual projection plane and converts it into a projected image. Specifically, the overhead image generator 22a projects the image captured by the front camera 11 onto the first region R1 of the virtual projection plane 100 in the virtual three-dimensional space shown in FIG. to the first projection image. Similarly, the bird's-eye view image generating unit 22a generates images captured by the back camera 12, captured by the left side camera 13, and captured by the right side camera 14 in the second to fourth regions R2 to R4 of the virtual projection plane 100 shown in FIG. , and converts the image captured by the back camera 12, the image captured by the left side camera 13, and the image captured by the right side camera 14 into second to fourth projection images, respectively.

A virtual projection plane 100 shown in FIG. 3 has, for example, a substantially hemispherical shape (bowl shape). A center portion (bottom portion of the bowl) of the virtual projection plane 100 is defined as a position where the own vehicle V1 exists. By making the virtual projection plane 100 include a curved surface in this way, it is possible to reduce the distortion of the image of an object existing at a position distant from the host vehicle V1. Also, each of the first to fourth regions R1 to R4 has a portion that overlaps with other adjacent regions. By providing such an overlapping portion, it is possible to prevent the image of the object projected on the boundary portion of the region from disappearing from the projection image.

The bird's-eye view image generation unit 22a generates a virtual viewpoint image viewed from a virtual viewpoint based on a plurality of projection images. Specifically, the bird's-eye view image generator 22a virtually pastes the first to fourth projection images onto the first to fourth regions R1 to R4 of the virtual projection plane 100. FIG.

In addition, the host vehicle image generator 22b virtually constructs a polygonal model representing the three-dimensional shape of the host vehicle V1. This model of the own vehicle V1 is located in a position defined as the position where the own vehicle V1 exists (the central portion of the virtual projection plane 100) in a virtual three-dimensional space in which the virtual projection plane 100 is set. is on the front side, and the fourth region R4 is on the rear side.

Furthermore, the bird's-eye view image generator 22a sets a virtual viewpoint in the virtual three-dimensional space in which the virtual projection plane 100 is set. A virtual viewpoint is defined by a viewpoint position and a viewing direction. As long as at least part of the virtual projection plane 100 is within the field of view, the viewpoint position and viewing direction of the virtual viewpoint can be set to any viewpoint position and any viewing direction. In the present embodiment, the viewpoint position of the virtual viewpoint is set to be substantially directly above the own vehicle, and the visual field direction of the virtual viewpoint is set to be substantially forward and downward of the own vehicle (substantially in the direction of gravity). As a result, the virtual viewpoint image generated by the bird's-eye view image generator 22a becomes the bird's-eye view image.

The bird's-eye view image generator 22a virtually cuts out an image of a required area (area visible from the virtual viewpoint) on the virtual projection plane 100 according to the set virtual viewpoint. In addition, the own vehicle image generation unit 22b performs rendering on a polygon model according to the set virtual viewpoint, and generates a rendered image of the own vehicle V1. Then, the superimposing unit 22c generates a bird's-eye view image by superimposing the rendered image of the own vehicle V1 on the clipped image. That is, the own vehicle image generation unit 22b generates an image representing the own vehicle. A rendered image of the own vehicle V1 (an image representing the own vehicle) is superimposed on an area corresponding to the current position of the own vehicle in the current bird's-eye view image.

Note that the own vehicle image generator 22b can generate not only a rendered image of the own vehicle V1 but also a CG (Computer Graphics) image of the own vehicle V1. Also, the superimposing unit 22c can superimpose an image representing the own vehicle on an arbitrary position of the captured image or the virtual viewpoint image. Therefore, the CG image of the own vehicle V1 can be used as a vehicle illustration image that is a display indicating the current position of the own vehicle, and is displayed as the past vehicle illustration image at a position corresponding to the own vehicle position corresponding to the past image. can also

For example, when the parking support function or the automatic parking function of the own vehicle is turned on, the storage unit 23 stores captured images, which are past images, until the setting of the parking position is completed. The photographed image, which is a past image, is stored in the storage unit 23 for a maximum of a predetermined period of time before the present or a maximum of a predetermined running distance from the present. Instead of or in addition to the captured image, which is a past image, a viewpoint-converted image, which is a past image, may be stored in the storage unit 23 . The self-vehicle parking support function and the automatic parking function are automatically turned on when conditions such as the speed of the self-vehicle being equal to or less than a predetermined value are met. Therefore, for example, the information providing device 201 can perform the parking assistance function or the automatic parking function of the host vehicle based on information output from a control unit that executes the parking assistance function or the automatic parking function or information output from the vehicle speed sensor. is turned on.

The information providing device 201 causes the display device 31 to display the past image according to the output from the operation device 32 . For example, when an image is currently displayed on the display screen of the display device 31, when a swipe operation is performed at a position of the touch panel corresponding to the image representing the own vehicle or the road surface in the current image, the information providing device 201 A past image that goes back in time corresponding to the amount of swiping is included in the display image. In addition, in the first past image display, the amount of going back to the past may be automatically adjusted. When the amount of going back in the past is automatically adjusted, for example, the parking position candidate, which is the easiest to park among the plurality of parking position candidates, may be positioned in the center portion of the display image in the vertical direction. Whether or not the parking position candidate is the easiest to park may be determined by scoring the width of the parking position candidate, the amount of steering operation required for the parking operation, and the like.

The parking frame recognition device 33 performs space recognition based on a signal output from a clearance sonar (not shown) to detect an empty space in the parking lot (an empty area where the vehicle is not parked). Also, the parking frame recognition device 33 recognizes the parking frame for the empty space by recognizing the white line from the images captured by the cameras 11 to 14 . Then, the parking frame recognition device 33 detects relative position information between the parking frame and the own vehicle from the pixel position of the recognized parking frame, and stores the relative position information in its own memory as parking candidate position information.

Furthermore, the parking frame recognizing device 33 is based on the amount of movement of the own vehicle obtained from a vehicle speed sensor (not shown) and the traveling direction based on the steering angle obtained from a steering angle sensor (not shown). , including not only the parking candidate positions contained in the current image but also the parking candidate positions detected in the past), corrects the position information for the own vehicle and stores it in its own memory. Therefore, the memory in the parking frame recognition device 33 stores the latest corrected parking candidate position information including past detection information. The parking frame recognition device 33 transmits the latest parking candidate position information to the display control unit 22 each time.

The display control unit 22 causes the storage unit 23 to store the parking candidate position information received from the parking frame recognition device 33 .

Based on the parking candidate position information stored in the storage unit 23, the superimposing unit 22c superimposes the parking candidate position on an image displayed on the display device 31, such as a bird's-eye view image or a camera image. For example, the parking candidate position may be superimposed on the image displayed on the display device 31 with a frame of a specific color. The user touches the desired parking space on the screen of the display device 31 to operate the operation device 32 (touch panel) based on the image on which the parking candidate positions are superimposed.

The parking position selection unit 24 selects a parking candidate position from the user's touch position, and transmits the selected parking candidate position to the parking frame recognition device 33 .

The parking frame recognition device 33 determines the parking candidate position received from the parking position selection unit 24 as the target parking position.

<2. First example of display image>
FIG. 4 is an example of a display image including past images. The display image shown in FIG. 4 includes a current image F1 captured by the back camera 12 and a bird's-eye view image F2 that is a past image (hereinafter referred to as a past bird's-eye view image). Since the display image shown in FIG. 4 includes the current image F1, it is possible to confirm the current situation around the vehicle with the display image.

An image CV1 representing the vehicle is superimposed on the current position of the vehicle on the past bird's-eye view image F2. Thus, it is possible to provide a past image that allows the user to easily grasp the positional relationship between the current position and the position where the photographed image corresponding to the past image was taken. Alternatively, a past image may be dynamically changed by gradually going back from a bird's-eye view image that is a current image (hereinafter referred to as a current bird's-eye view image) to finally reach a past bird's-eye view image F2. The past bird's-eye view image F2 may be displayed immediately upon display. In the case of dynamically changing the past image from the current bird's-eye view image gradually going back to the past and finally reaching the past bird's-eye view image F2, the captured image of the display viewpoint based on the current position of the vehicle is displayed. , displays a photographed image in which the viewpoint is shifted in a direction including the range through which the vehicle has passed in the past using the past image stored in the storage unit 23, and shows the current position of the vehicle during the transition. Display will continue. Further, the above transition may be performed by moving the viewpoint according to the user's scrolling operation on the operation device 32, for example. Moreover, in the past image display during the transition, for example, the superimposing unit 23c may superimpose and display the parking candidate position.

Further, on the past bird's-eye view image F2, an image PV1 representing the own vehicle is superimposed at the position where the photographed image corresponding to the past bird's-eye view image F2 was taken. Thus, it is possible to provide a past image that allows the user to more easily grasp the positional relationship between the current position and the position where the photographed image corresponding to the past image was taken.

Note that, as shown in FIG. 4, an image CV1 representing the own vehicle superimposed on the current position of the own vehicle, and an image PV1 representing the own vehicle superimposed on the position where the photographed image corresponding to the previous bird's-eye view image F2 was photographed. are displayed in different modes, it becomes easier to distinguish between the image CV1 representing the own vehicle and the image PV1 representing the own vehicle.

Further, as shown in FIG. 5, the image processing unit 22d may mask the past bird's-eye view image F2 with, for example, transparent blue (in FIG. 5, transparent blue is represented by a dot pattern). Although the past image may be inconsistent with the current situation, making it clear that the image is the past image can increase the satisfaction of the inconsistency with the current situation.

Selection of parking position candidates may be performed only within the past bird's-eye view image F2, or may be performed both within the current image F1 and within the past bird's-eye view image F2. If the operation can be performed only within the past bird's-eye view image F2, the selection operation of the parking position candidate is facilitated. In addition, if it can be performed in both the current image F1 and the past bird's-eye view image F2, options for parking position candidates are expanded.

<3. Second example of display image>
FIG. 6 is an example of a display image including past images. The display image shown in FIG. 6 is a past image, and is a viewpoint conversion image F3 obtained by setting the viewpoint position to the standard position of the eyes of the driver, setting the viewing direction to the front of the vehicle, and transmitting the vehicle body portion of the vehicle. , and the current overhead image F4. Since the display image shown in FIG. 6 includes the current bird's-eye view image F4, it is possible to confirm the current situation around the vehicle in the display image.

An image CV1 representing the vehicle is superimposed on the current position of the vehicle on the viewpoint-transformed image F3 (hereinafter referred to as the past-viewpoint-transformed image), which is a past image. Thus, it is possible to provide a past image that allows the user to easily grasp the positional relationship between the current position and the position where the photographed image corresponding to the past image was taken.

The past-viewpoint-converted image F<b>3 is similar to the captured image captured by the front camera 11 , with the viewpoint position set to the standard eye position of the driver and the visual field direction set to the front of the vehicle. An image CV1 representing the own vehicle is superimposed on the current position of the own vehicle, and the past image is dynamically changed by gradually going back from the current image to make it easier to recognize the image representing the own vehicle as the own vehicle. To make it easier to grasp the positional relationship with the past image, a past image is generated in which the current position of the own vehicle passes through the image CV1 representing the own vehicle. However, when the captured image captured by the front camera 11 is used, the past image passing through the image CV1 representing the vehicle at the current position of the vehicle does not become a smooth image. For this reason, it is desirable to use the past viewpoint conversion image F3 in which the viewpoint position is the standard eye position of the driver and the viewing direction is the front of the vehicle.

Further, as shown in FIG. 7, the image processing unit 22d may mask the past-viewpoint-converted image F3 with, for example, transparent blue (transparent blue is represented by a dot pattern in FIG. 7). Although the past image may be inconsistent with the current situation, making it clear that the image is the past image can increase the satisfaction of the inconsistency with the current situation.

The parking position candidate may be selected only within the past-viewpoint-converted image F3, or may be performed both within the current image F4 and the past-viewpoint-converted image F3. If the operation can be performed only within the past-viewpoint-converted image F3, the selection operation of the parking position candidate becomes easier. In addition, if it is possible to perform the operation in both the current image F4 and the past viewpoint converted image F3, options for parking position candidates are expanded.

<4. Others>
Various modifications can be made to the various technical features disclosed in this specification without departing from the gist of the technical creation in addition to the above-described embodiments. In addition, multiple modifications shown in this specification may be combined to the extent possible.

In the above-described embodiment, the past image was a bird's-eye view image, but the past image may be a photographed image.

Alternatively, a bird's-eye view image that is a past image and a bird's-eye view image that is a current image may be synthesized to generate one bird's-eye view image.

Also, the display area in the past image may be changed by changing the viewpoint position of the virtual viewpoint image.

Further, the image processing indicating that it is a past image may be performed by using a mask other than a transparent colored mask, and for example, a past time may be superimposed on the past image.

21 camera image acquisition unit 22 display control unit 22a overhead image generation unit 22b own vehicle image generation unit 22c superimposition unit 22d image processing unit 23 storage unit 24 parking position selection unit 201 information providing device

Claims (5)

an acquisition unit that acquires a photographed image obtained by a camera that photographs the surroundings of the own vehicle;
a storage unit that stores at least one of the captured image and a viewpoint-converted image based on the captured image;
a generation unit that generates an image representing the own vehicle;
a superimposing unit that superimposes an image representing the own vehicle on a current position of the own vehicle in at least one of the captured image and the viewpoint converted image;
As an image processing process for allowing the user to visually recognize that the image was taken in the past, image processing is performed on the past image by applying a transparent colored mask or superimposing an image that goes back in time. An information providing device, comprising: a processing unit; 2. The information providing apparatus according to claim 1, wherein a past image, which is either the captured image or the viewpoint-transformed image, and a current image, which is the captured image or the viewpoint-transformed image, are output as display images. 3. The superimposing unit according to claim 1, wherein the superimposing unit superimposes an image representing the own vehicle on a position of the own vehicle at which the photographed image corresponding to the past image in the past image of the viewpoint converted image was taken. Information provider. An image representing the own vehicle superimposed on the current position of the own vehicle in the past image, and an image representing the own vehicle superimposed on the position of the own vehicle at which the photographed image corresponding to the past image in the past image of the viewpoint conversion image was photographed. 4. The information providing apparatus according to claim 3 , wherein the image representing the own vehicle is displayed in a different manner. an acquisition step of acquiring a photographed image obtained by a camera that photographs the surroundings of the own vehicle;
a storage step of storing at least one of the captured image and a viewpoint-converted image based on the captured image;
a generating step of generating an image representing the own vehicle;
a superimposing step of superimposing an image representing the own vehicle on the current position of the own vehicle in at least one of the captured image and the viewpoint converted image;
As an image processing process for allowing the user to visually recognize that the image was taken in the past, image processing is performed on the past image by applying a transparent colored mask or superimposing an image that goes back in time. A method of providing information, comprising: a processing step;

Images