KR102621703B1 - Appartus and method for displaying image of vehicle - Google Patents

Abstract

The video display device for a vehicle of the present invention includes a route search unit that searches for a route to a destination, a camera that photographs the surrounding environment, and a server that obtains similar route information based on the discovered route, and based on the similar route information It includes an image generator that generates a virtual reality image corresponding to the captured surrounding environment, and provides an environment preferred by the driver, different from the environment in which the vehicle is currently driving, so that the driver feels as if he is driving in a new environment. You can have it.

Description

Vehicle image display device and method {APPARTUS AND METHOD FOR DISPLAYING IMAGE OF VEHICLE}

The present invention relates to a vehicle image display device, and more specifically, to a vehicle image display device and method for displaying a virtual reality image corresponding to the vehicle's driving environment.

Typically, a vehicle terminal mounted on a vehicle to provide vehicle information to the driver is located in the center between the driver's seat and the passenger seat. The driver usually drives while looking ahead, but when information is provided to the vehicle terminal, the driver moves his gaze to the vehicle terminal to check the provided information.

If the driver's gaze of a vehicle traveling at 100 km/h deviates from the front for just 2 seconds, the vehicle will travel 55 meters without protection. Therefore, the vehicle can be easily exposed to unexpected situations. As the number of devices installed in vehicles, such as navigation systems and smartphones, increases, this risk may increase.

Therefore, a head-up display device that displays vehicle information in front of the driver's field of view has been proposed. Head-up display devices allow drivers to view information provided from a position in front of the driver while looking forward without changing their gaze or head direction to check the provided information, thereby lowering the risk of distraction.

Recently, Augmented Reality (AR) technology has been introduced to provide safe driving information to the driver by aligning the real-world information in front of the driver with the location of the information displayed on the head-up display device, focusing on the driver's field of view.

Meanwhile, technology that displays virtual reality images corresponding to the vehicle's current driving environment according to the driver's selection is in need of development.

The present invention seeks to provide an image display device and method for a vehicle that displays a virtual reality image corresponding to the environment in which the driver is currently driving on the windshield of the vehicle.

The video display device for a vehicle of the present invention includes a route search unit that searches for a route to a destination, a camera that photographs the surrounding environment, and a server that obtains similar route information based on the discovered route, and based on the similar route information and an image generator that generates a virtual reality image corresponding to the captured surrounding environment.

Additionally, the surrounding environment includes at least one of the lane of the road on which the vehicle is traveling, other vehicles, pedestrians, buildings around the road, and the natural environment around the road.

In addition, the similar path information includes a similar path and a similar path image obtained from the server based on the similarity between the searched path and a predetermined path retrieved from the server using the latitude and longitude of the origin and destination of the searched path. do.

And, the similarity is at least one of the distance between the predetermined route and the searched route, the curvature of the predetermined route and the searched route, the altitude of the predetermined route and the searched route, and the number of lanes of the predetermined route and the searched route. It is calculated based on

Additionally, the similar path image includes an image captured by a vehicle traveling along the similar path.

And, the virtual reality image is created by combining similar path images corresponding to the surrounding environment captured by the camera.

Then, similar path images corresponding to the surrounding environment captured by the camera are matched by extracting the surrounding environment objects from the surrounding environment captured by the camera and the similar path image, and are obtained from the server according to the number of matched objects. .

And, the surrounding environment object includes at least one of the lanes of the road, other vehicles, pedestrians, buildings around the road, and the natural environment around the road.

And, it further includes a display unit that displays the generated virtual reality image.

Additionally, the display unit includes a windshield of the vehicle.

The method of displaying an image of a vehicle according to the present invention includes the steps of searching a route, taking pictures of the surrounding environment, obtaining similar path information based on the searched path, and the captured surroundings based on the similar path information. It includes the step of generating a virtual reality image corresponding to the environment.

Additionally, the surrounding environment includes at least one of the lane of the road on which the vehicle is traveling, lane color, other vehicles, pedestrians, buildings around the road, and the natural environment around the road.

In addition, the similar path information includes a similar path and a similar path image obtained from the server based on the similarity between the searched path and a predetermined path retrieved from the server using the latitude and longitude of the origin and destination of the searched path. do.

And, the similarity is at least one of the distance between the predetermined route and the searched route, the curvature of the predetermined route and the searched route, the altitude of the predetermined route and the searched route, and the number of lanes of the predetermined route and the searched route. It is calculated based on

Additionally, the similar path image includes an image captured by a vehicle traveling along the similar path.

And, in the step of generating the virtual reality image, similar path images corresponding to the surrounding images captured by the camera are combined.

Then, similar path images corresponding to the surrounding images captured by the camera are extracted and matched with the surrounding environment objects from the surrounding images captured by the camera and the similar path images, and are obtained from the server according to the number of matched objects. .

Additionally, the surrounding environment object includes at least one of a road lane, a lane color, other vehicles, pedestrians, buildings around the road, and the natural environment around the road.

And, after the step of generating the virtual reality image, the method further includes displaying the generated virtual reality image on a display unit.

Additionally, the display unit includes a windshield of the vehicle.

The vehicle image display device and method of the present invention can display a virtual reality image corresponding to the environment in which the vehicle is currently driving to the driver, allowing the driver to feel that he or she is driving in a new environment.

1 is a configuration diagram showing a video display device for a vehicle according to the present invention.
2 to 4 are diagrams schematically showing the process of searching for a similar path according to an embodiment of the present invention.
Figure 5 is a diagram showing the surrounding environment captured by a camera according to an embodiment of the present invention.
Figure 6 is a diagram showing a similar path image corresponding to the surrounding environment captured by a camera according to an embodiment of the present invention.
Figure 7 is a flowchart showing a method for displaying an image of a vehicle according to an embodiment of the present invention.
Figure 8 is a flowchart showing a method of displaying an image of a vehicle according to another embodiment of the present invention.

Hereinafter, some embodiments of the present invention will be described in detail through illustrative drawings. When adding reference numerals to components in each drawing, it should be noted that identical components are given the same reference numerals as much as possible even if they are shown in different drawings. Additionally, when describing embodiments of the present invention, if detailed descriptions of related known configurations or functions are judged to impede understanding of the embodiments of the present invention, the detailed descriptions will be omitted.

In describing the components of the embodiment of the present invention, terms such as first, second, A, B, (a), and (b) may be used. These terms are only used to distinguish the component from other components, and the nature, sequence, or order of the component is not limited by the term. Additionally, unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by a person of ordinary skill in the technical field to which the present invention pertains. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related technology, and should not be interpreted in an ideal or excessively formal sense unless explicitly defined in the present application. No.

In embodiments of the present invention, when an autonomous vehicle autonomously drives a discovered route to a destination, a virtual reality image corresponding to the current road environment or the driver's current driving environment is displayed on the windshield of the vehicle according to the driver's selection. It relates to an image display device and method for a vehicle that displays, but is not interpreted as being limited thereto.

1 is a configuration diagram of a video display device for a vehicle according to the present invention.

As shown in FIG. 1, the image display device 100 of the present invention may include a camera 10, a path search unit 12, a display unit 16, and an image generator 18, and displays the image. Device 100 may communicate with server 200.

The camera 10 can capture the surrounding environment of the vehicle. To this end, the camera 10 may include a front ultra-wide-angle camera that captures images of the front of the vehicle, a rear ultra-wide-angle camera that captures images of the rear of the vehicle, a left ultra-wide-angle camera that captures images of the left and right sides of the vehicle, and a right ultra-wide-angle camera. As long as the camera 10 can acquire images of the front, rear, and both left and right sides of the vehicle, there is no limit to the installation location or number of cameras.

More specifically, the camera 10 can photograph the surrounding environment of the route discovered by the route search unit 12. The surrounding environment of the searched route may include at least one of the lane of the road on which the vehicle is traveling, the lane color, vehicles located around the road, pedestrians, buildings around the road, and the natural environment around the road.

The route search unit 12 may be provided with a storage means and may store map information including the number of lanes, curvature, lane conditions, altitude, etc. in the storage means. The route search unit 12 can store a precise map that includes detailed information such as the marking status of the lane in which the vehicle is being driven. The route search unit 12 can search the route to the destination based on the stored map, but is not limited to this and can also search the route to the destination by receiving a map through communication with the server 200. Routes can be searched based on shortest distance, time required, etc.

The path search unit 12 may include a location measurement unit, but may also be provided separately. The position measurement unit measures the current location of the vehicle. The location measurement unit may be implemented as a Global Positioning System (GPS) module. The GPS module calculates the current location of the vehicle using signals transmitted from three or more GPS satellites. The GPS module calculates the distance between the satellite and the GPS module using the time difference between the time the signal was transmitted from the satellite and the time the signal was received by the GPS module. The GPS module calculates the current location of the vehicle using the calculated distance between the satellite and the GPS module and the satellite location information included in the transmitted signal. At this time, the GPS module can calculate the current location of the vehicle using triangulation.

The communication unit 14 can communicate with the server 200. The communication unit 14 can receive maps and road images from the server 200 through a wireless network, and can receive similar route information. Additionally, the communication unit 14 may transmit the image captured by the camera 10 and the route searched by the route search unit 12 to the server 200.

The display unit 16 may include a head-up display, and more specifically, may include a windshield provided on the front of the vehicle. The display unit 16 can output a virtual reality image generated by the image generator 18. When the display unit 16 does not output a virtual reality image, it can display the road environment on which the vehicle is currently driving.

The image generator 18 may obtain information on a similar path from the server 200 based on the discovered path and generate a virtual reality image corresponding to the driving environment on the discovered path. Here, the server 200 can search for similar path information based on the discovered path. Similar path information may include a similar path retrieved from the server 200 and a similar path image captured by a vehicle driving along the similar path.

The server 200 searches for a predetermined route based on the latitude and longitude of the origin and destination of the searched route, calculates the similarity between the searched route and the searched predetermined route, and searches for a similar route based on the calculated similarity. can do. More details will be described with reference to FIGS. 2 to 4. 2 to 4 are diagrams schematically showing the process of searching for a similar path according to an embodiment of the present invention.

More specifically, as shown in FIG. 2, the server 200 can check the longitude and latitude of the starting point and the longitude and latitude of the destination based on the searched route. And, as shown in Figure 3, the searched route can be extracted from the map and analyzed based on the difference in longitude, difference in latitude, distance, curvature, altitude, and number of lanes between the origin and destination. According to an embodiment, the difference in latitude between the origin and destination of the searched route may be 0.037 and the difference in longitude may be 0.01.

The server 200 may search for a predetermined route having the distance of the searched route based on the analysis result. Preferably, the driver can search for a predetermined route within the preferred city.

Additionally, the server 200 can calculate the similarity between the searched path and the predetermined path. The server 200 may consider certain items to calculate similarity. Predetermined items may include distance difference, curvature difference, altitude difference, and lane difference, according to embodiments. Here, the distance difference may mean the difference between the distance of the search route and the distance of the predetermined route, the curvature difference may mean the sum of the absolute value of the difference in curvature of the search route and the difference in curvature of the predetermined route, and the altitude difference may be It can mean the sum of the absolute value of the altitude difference of the search route and the absolute value of the altitude difference of the predetermined route, and the lane difference means the difference in the number of lanes at a predetermined point in the similar route corresponding to a predetermined point in the search route. can do.

Depending on the embodiment, the server 200 may calculate the similarity by multiplying the scores obtained from the above-mentioned items by a coefficient, as shown in Tables 1 to 4. For more detailed similarity calculation formula, refer to the following formula.

Similarity (%) = a* score of item 1 + b* score of item 2 + c* score of item 3 + d* score of item 4

(Here, a, b, c, d are coefficients according to the priority of items 1, 2, 3, and 4)

distance difference 0km Within 1km Within 2km Within 3km More than 3km score 100 80 70 60 except

curvature difference 0 degrees within 3 degrees within 5 degrees within 7 degrees over 7 degrees score 100 80 70 60 except

altitude difference 0m Within 5m Within 10m over 10m score 100 80 70 except

lane difference 0 lane 1st lane 2 lanes or more score 100 70 except

According to the embodiment, the sum of the coefficients (a, b, c, d) used when calculating similarity is preferably 1, and the server 200 sorts the items considered for calculating similarity in order of highest priority. Similarity can be calculated by setting the coefficient value of the item.

For example, when the server 200 sets the priority to condition 1 as follows, the similarity can be calculated as follows. Here, condition 1 sets the priority in the order of distance difference > curvature difference > altitude difference > lane difference, and includes cases where the distance difference is less than 1km, the curvature difference is less than 5 degrees, the altitude difference is 0m, and the lane difference is 1 lane difference. You can. The value of each coefficient may vary, but as an example, a is set to 0.5, b is set to 0.35, c is set to 0.1, and d is set to 0.05.

Similarity for condition 1 = 0.5 * 80 (score for distance difference) + 0.35 * 70 (score for curvature difference) + 0.1 * 100 (score for elevation difference) + 0.05 * 100 (score for lane difference)

The server 200 can calculate the similarity for condition 1 as 79.5%.

For another example, when the server 200 sets the priority to condition 2 as follows, the similarity can be calculated as follows. Here, Condition 2 sets the priority in the order of curvature difference > lane difference > distance difference > altitude difference, and includes cases where the distance difference is less than 2km, the curvature difference is less than 0 degrees, the altitude difference is less than 5m, and the lane difference is 0 lane. You can. The value of each coefficient may vary, but as an example, a is set to 0.5, b is set to 0.35, c is set to 0.1, and d is set to 0.05.

Similarity for condition 2 = 0.5 * 100 (score for curvature difference) + 0.35 * 100 (score for lane difference) + 0.1 * 70 (score for distance difference) + 0.05 * 80 (score for elevation difference)

The server 200 can calculate the similarity for condition 2 as 94%.

According to the embodiment, the server 200 may search a predetermined path as a similar path as shown in FIG. 4 when the calculated similarity is 90% or more.

The server 200 searches for similar path images captured by a vehicle traveling along a similar path.

The server 200 determines any one of the surrounding environments among the lanes of the road, other vehicles, pedestrians, buildings around the road, and the natural environment around the road from the image of the camera 10 that captures the surrounding environment in which the vehicle is currently driving and the similar path image. Objects can be extracted.

The server 200 matches the surrounding environment objects extracted from the surrounding environment captured by the camera 10 with the surrounding environment objects extracted from the similar path image, and determines the surrounding environment captured by the camera 10 according to the number of matched objects. Corresponding similar path images can be searched. Preferably, when the number of matching objects is two or more, a similar path image corresponding to the surrounding environment captured by the camera 10 can be searched.

Meanwhile, the image generator 18 may obtain similar path images corresponding to the surrounding environment captured by the camera 10 from the server 200 and combine them to generate a virtual reality image.

For more details, refer to FIGS. 5 and 6. Figure 5 is a diagram showing the surrounding environment captured by the camera, and Figure 6 is a diagram showing a similar path image corresponding to the surrounding environment captured by the camera.

The image generator 18 may obtain similar path images corresponding to the surrounding environment captured by the camera 10 (see FIG. 5) from the server 200.

According to the embodiment, the server 200 may extract a bus and a truck in front of the vehicle and a building located on the right side of the vehicle as surrounding environment objects from the surrounding environment and similar path images captured by the camera 10. The server 200 may determine that the surrounding environment objects extracted from the similar path image matched with the surrounding environment objects extracted in FIG. 5 are buses, trucks, and buildings, and thus correspond to the surrounding environment captured by the camera 10. Similar path images (see FIG. 6) can be searched.

The image generator 18 may generate a virtual reality image by combining similar path images corresponding to the surrounding environment captured by the camera 10.

When the image generator 18 generates a virtual reality image, the control unit 20 can control the virtual reality image to be output on the front of the display unit 16. The control unit 20 can control to output a virtual reality image based on driver input. The control unit 20 may determine that a virtual reality image is output when there is a driver input. On the other hand, if there is no driver input, it can be determined that the virtual reality image is not output.

Additionally, the control unit 20 may prevent the virtual reality image from being output if the vehicle deviates from the discovered route and display the road on which the vehicle is currently traveling.

The control unit 20 can determine whether to output a virtual reality image even when the route to the destination is not found. If the control unit 20 determines that a virtual reality image will be output even if the route to the destination is not found, it guides the driver to set the destination. Afterwards, the destination is set, and the control unit 20 searches for a route according to the destination. If a searched route exists, similar path information is obtained based on the searched path, and the camera 10 takes pictures based on the similar path information. A virtual reality image corresponding to the surrounding environment can be output through the display unit 16.

Figure 7 is a flowchart showing a method of displaying an image of a vehicle according to an embodiment of the present invention. As shown in FIG. 7, a route is searched and the discovered route is photographed (S110). In S110, the route search unit 12 searches for a route to the destination when the destination is input. Additionally, in S110, the camera 10 photographs the surrounding environment within the discovered path. For example, the camera 10 can photograph the lane of the road on which the vehicle is traveling, the color of the lane, vehicles located around the road, pedestrians, buildings around the road, and the natural environment around the road.

The path search unit 12 obtains similar path information from the server 200 (S120). In SS120, the path search unit 12 can obtain similar path information searched by the server 200. The server 200 may search for similar path information based on the discovered path. Similar path information may include similar paths and similar path images. The server 200 can search for a predetermined route based on the latitude and longitude of the origin and destination of the searched route, calculate the similarity between the searched route and the predetermined route, and search for a similar route based on the similarity. Additionally, the server 200 can search images captured by a vehicle traveling along a similar path as similar path images.

The control unit 20 determines whether to output a virtual reality image through the display unit 16 (S130). In S130, the control unit 20 may determine whether to output a virtual reality image based on driver input. The control unit 20 may determine that a virtual reality image is output when there is a driver input. On the other hand, if there is no driver input, it can be determined that the virtual reality image is not output.

If the control unit 20 determines that a virtual reality image will be output in S130 (Yes), the image generator 18 generates a virtual reality image based on the similar path information (S140). In S140, the image generator 18 obtains similar path images corresponding to the surrounding environment captured by the camera 10 from the server 200, and generates a virtual reality image by combining them. Similar path images corresponding to the surrounding environment captured by the camera 10 extract and match surrounding environment objects from the surrounding environment and similar path images captured by the camera 10, and are sent to the server 200 according to the number of matched objects. It can be searched from .

If the control unit 20 determines that the virtual reality image will not be output in S130 (No), it controls the display unit 16 to display the road currently being driven (S170).

The control unit 20 controls the virtual reality image to be output on the front of the display unit 16 (S150).

The control unit 20 checks whether the vehicle has deviated from the discovered route (S160). If it is confirmed that the route has deviated, the control is performed to display the road currently being driven without outputting the virtual reality image (S170).

Figure 8 is a flowchart showing a method of displaying an image of a vehicle according to another embodiment of the present invention.

As shown in FIG. 8, it is confirmed that there is no searched route (S200). In S200, the route search unit 12 can confirm that the destination has not been entered if there is no route found.

The control unit 20 determines whether to output a virtual reality image through the display unit 16 (S210). In S210, the control unit 20 can determine whether to output a virtual reality image based on the driver's input. The control unit 20 may determine that a virtual reality image is output when there is a driver input. On the other hand, if there is no driver input, it can be determined that the driver is not outputting the virtual reality image. In this case, perform S270.

In S210, if the control unit 20 determines that the virtual reality image is output through the display unit 16 (Yes), the route search unit 12 guides the destination input, searches the route to the input destination, and detects the camera. (10) photographs the surrounding environment of the searched route (S220). In S220, the camera 10 photographs at least one of the lane of the road on which the vehicle is traveling, other vehicles, pedestrians, buildings around the road, and the natural environment around the road within the searched route.

The path search unit 12 obtains similar path information from the server 200 (S230). S230 is the same as S120, so refer to the description of S120.

The image generator 18 generates a virtual reality image based on the similar path information (S240). In S240, the image generator 18 obtains similar path images corresponding to the surrounding environment captured by the camera 10 from the server 200, and generates a virtual reality image by combining them. Similar path images corresponding to the surrounding environment captured by the camera 10 extract and match surrounding environment objects from the surrounding environment and similar path images captured by the camera 10, and are sent to the server 200 according to the number of matched objects. It can be searched from .

The control unit 20 controls the virtual reality image to be output on the front of the display unit 16 (S250).

The control unit 20 checks whether the vehicle has deviated from the discovered route (S260). If it is confirmed that the route has deviated, the road currently being driven is displayed instead of outputting the virtual reality image (S270).

The above description is merely an illustrative explanation of the technical idea of the present invention, and various modifications and variations will be possible to those skilled in the art without departing from the essential characteristics of the present invention.

Accordingly, the embodiments disclosed in the present invention are not intended to limit the technical idea of the present invention, but are for illustrative purposes, and the scope of the technical idea of the present invention is not limited by these embodiments. The scope of protection of the present invention should be interpreted in accordance with the claims below, and all technical ideas within the equivalent scope should be construed as being included in the scope of rights of the present invention.

camera 10
Route search unit 12
Department of Communications 14
Display unit 16
Image creation unit 18
Control unit 20
Vehicle video display device 100
server 200

Claims (20)

a route search unit that searches the route to the destination;
A camera that captures the surrounding environment; and
An image generator that obtains similar path information from a server based on the discovered path and generates a virtual reality image corresponding to the captured surrounding environment based on the similar path information,
The above similar path information is
An image display device for a vehicle including a similar path and a similar path image obtained from the server based on the similarity between the predetermined path searched from the server using the latitude and longitude of the origin and destination of the searched path and the searched path. In claim 1,
The surrounding environment is
A video display device for a vehicle that includes at least one of the lane of the road on which the vehicle is traveling, other vehicles, pedestrians, buildings around the road, and the natural environment around the road. delete In claim 1,
The similarity is
A vehicle calculated based on at least one of the distance between the predetermined route and the searched route, the curvature of the predetermined route and the searched route, the altitude of the predetermined route and the searched route, and the number of lanes of the predetermined route and the searched route. video display device. In claim 1,
The similar path video is
A vehicle image display device including images captured by a vehicle traveling along the similar path. In claim 1,
The virtual reality video is
An image display device for a vehicle that is generated by combining similar path images corresponding to the surrounding environment captured by the camera. In claim 6,
Similar path images corresponding to the surrounding environment captured by the camera are
An image display device for a vehicle in which surrounding environment objects are extracted and matched from the surrounding environment captured by the camera and the similar path image, and obtained from the server according to the number of matched objects. In claim 7,
The surrounding environment objects are
A vehicle image display device that includes at least one of the road lanes, other vehicles, pedestrians, buildings around the road, and the natural environment around the road. In claim 1,
An image display device for a vehicle further comprising a display unit that displays the generated virtual reality image. In claim 9,
The display unit
A visual display device for a vehicle, including the vehicle's windshield. finding a route;
Photographing the surrounding environment;
Obtaining similar path information based on the discovered path; and
Generating a virtual reality image corresponding to the captured surrounding environment based on the similar path information,
The above similar path information is
A method for displaying images of a vehicle including a similar path and a similar path image obtained from the server based on the similarity between the searched path and a predetermined path retrieved from the server using the latitude and longitude of the origin and destination of the searched path. In claim 11,
The surrounding environment is
A method of displaying an image of a vehicle including at least one of the lane of the road on which the vehicle is traveling, lane color, other vehicles, pedestrians, buildings around the road, and the natural environment around the road. delete In claim 11,
The similarity is
A vehicle calculated based on at least one of the distance between the predetermined route and the searched route, the curvature of the predetermined route and the searched route, the altitude of the predetermined route and the searched route, and the number of lanes of the predetermined route and the searched route. Video display method. In claim 11,
The similar path video is
A method of displaying images of a vehicle including images captured by a vehicle traveling along the similar path. In claim 11,
The step of generating the virtual reality image is
A vehicle image display method that combines similar path images corresponding to surrounding images captured by a camera. In claim 16,
Similar path images corresponding to the surrounding images captured by the camera are
A method for displaying images of a vehicle in which the surrounding environment objects are extracted and matched from the surrounding images captured by the camera and the similar path images, and obtained from the server according to the number of matched objects. In claim 17,
The surrounding environment objects are
A method of displaying an image of a vehicle including at least one of the lanes of a road, lane colors, other vehicles, pedestrians, buildings around the road, and the natural environment around the road. In claim 11,
After the step of generating the virtual reality image
An image display method for a vehicle further comprising displaying the generated virtual reality image on a display unit. In claim 19,
The display unit
A method of displaying video of a vehicle, including the windshield of the vehicle.

Images