JP7464230B1 - Display device - Google Patents

Abstract

[Problem] To provide a display device that supports a vehicle so that it can travel safely on roads even in bad weather. [Solution] A display device having an imaging unit that captures an image of a road and the conditions around the road, a position information acquisition unit that acquires position information that represents the position of the vehicle when the image was captured, a storage unit that stores a first image that represents the image captured by the imaging unit when the weather is good, a display unit, and a control unit that causes the display unit to simultaneously display a second image that represents the image captured by the imaging unit when the weather is bad and the first image according to the position of the vehicle. [Selected Figure] Figure 1

Description

The present invention relates to a display device.

Conventionally, there are navigation devices that search for vehicle routes. One such navigation device is a route search system that estimates future weather conditions and searches for roads by prioritizing roads in areas where the estimated weather is better than the reference weather over roads in areas where the weather is worse than the reference weather.

JP 2018-36178 A

However, for example, in heavy snow conditions where roads are buried, even if a navigation device searches for roads, it is difficult for a vehicle to travel on the roads. In particular, when a vehicle is traveling at night, it may be difficult to know the surrounding conditions at all.

Therefore, the present disclosure aims to provide a display device that supports vehicles so that they can travel safely on roads even in bad weather.

One aspect of the present disclosure is a display device having an imaging unit that captures an image of a road and the conditions around the road, a location information acquisition unit that acquires location information representing the location of the vehicle when the image is captured, a storage unit that stores a first image representing an image captured by the imaging unit when the weather is good, a display unit, and a control unit that causes the display unit to simultaneously display a second image representing an image captured by the imaging unit when the weather is bad and the first image according to the location of the vehicle.

According to one aspect, a display device can be provided that supports a vehicle so that it can travel safely on roads even in bad weather.

FIG. 1 is a diagram illustrating an example of the configuration of a display device according to a first embodiment. 2A and 2B are diagrams illustrating a display example according to the first embodiment. FIG. 3 is a diagram illustrating an example of an operation according to the first embodiment. 4A and 4B are diagrams illustrating a display example according to the second embodiment. 5A and 5B are diagrams illustrating a display example according to the second embodiment.

Hereinafter, an embodiment of the present disclosure will be described in detail with reference to the drawings. In this specification and the drawings, elements that can be described in the same way may be omitted from repeated description by using the terms "similar" or "similar."

[First embodiment]
First, the first embodiment will be described.

Figure 1 shows an example of the configuration of a display device 100 according to the first embodiment.

The display device 100 is attached to a vehicle. The display device 100 may be provided, for example, in a navigation device. The display device 100 may display an image in response to an instruction from the navigation device.

As shown in FIG. 1, the display device 100 has an imaging unit 110, a position information acquisition unit 120, a memory unit 130, a display unit 140, and a control unit 150.

The imaging unit 110 captures images under the control of the control unit 150. The imaging unit 110 captures images in front of the vehicle, in particular, images of the road and the conditions around the road. The imaging unit 110 captures images when the weather is good. The imaging unit 110 also captures images when the weather is bad. Below, an image captured when the weather is good may be referred to as the first image. Below, an image captured when the weather is bad may be referred to as the first image. The imaging unit 110 outputs an image signal of the captured image to the control unit 150. The imaging unit 110 is, for example, a camera.

Good weather means clear skies. Good weather may be better than the reference weather. Bad weather means snow or rain. Bad weather may be worse than the reference weather.

The position information acquisition unit 120 acquires position information indicating the position of the vehicle when the image is captured by the imaging unit 110. The position information acquisition unit 120 outputs the acquired position information to the control unit 150. For example, the position information acquisition unit 120 acquires the position information of the vehicle in conjunction with the time when the image is captured by the imaging unit 110. That is, the position information acquisition unit 120 acquires the position information when the first image is captured by the imaging unit 110 in good weather. The position information acquired by the position information acquisition unit 120 when the first image is captured by the imaging unit 110 in good weather may be referred to as the first position information below. The position information acquisition unit 120 also acquires the position information indicating that the second image is captured by the imaging unit 110 in bad weather. The position information acquired by the position information acquisition unit 120 when the second image is captured by the imaging unit 110 in bad weather may be referred to as the second position information below.

The location information acquisition unit 120 may be, for example, a Global Navigation Satellite System (GNSS) receiving unit. The GNSS receiving unit may receive a GNSS reception signal from a satellite and acquire location information (for example, latitude, longitude, and/or altitude) from the received GNSS reception signal.

The storage unit 130 can store data or information and read out the stored data or information under the control of the control unit 150. The storage unit 130 stores the image signal of the first image and the first position information output from the control unit 150. The storage unit 130 may store the image signal of the first image and the first position information in association with each other. The storage unit 130 can appropriately read out the stored image signal of the first image and the first position information by the control unit 150. The storage unit 130 may be a memory.

The display unit 140 displays an image under the control of the control unit 150. The display unit 140 may also display text information and/or map information together with the image.

The control unit 150 controls each block of the display device 100. The control unit 150 outputs the first image (the image signal) and the first position information from the images output from the imaging unit 110 to the storage unit 130, and stores this information in the storage unit 130.

When the control unit 150 receives the image signal of the second image from the imaging unit 110 and the second position information from the position information acquisition unit 120, the control unit 150 performs the following process. That is, the control unit 150 reads out the image signal of the first image of the first position information corresponding to the second position information from the storage unit 130. That is, for example, when the imaging unit 110 is capturing a second image of poor weather, the control unit 150 reads out the image signal of the first image having the same position information (first position information) as the second position information acquired when the second image was captured from the storage unit 130. As a result, while capturing the second image with the imaging unit 110, the control unit 150 can read out the image signal of the first image at the same position as the position of the vehicle when the second image was captured from the storage unit 130. Then, the control unit 150 can output the image signal of the first image and the image signal of the second image to the display unit 140, thereby displaying the first image and the second image simultaneously on the display unit 140 according to the position of the vehicle.

Figures 2(A) and 2(B) are diagrams showing examples of display screens displayed on the display unit 140. Figure 2(A) shows an example of the display screen when the vehicle is traveling in the center of the road, and Figure 2(B) shows an example of the display screen when the vehicle is traveling in the left lane of the road.

As shown in Figures 2(A) and 2(B), the display unit 140 has a first display area 141 in which a first image is displayed, and a second display area 142 in which a second image is displayed. In the example shown in Figures 2(A) and 2(B), the second display area 142 has a larger display area than the first display area 141. The second image displayed in the second display area 142 is an image that represents the current situation, so the display area of the first display area 141 is larger.

The second display area 142 displays the second image currently captured by the imaging unit 110. The first display area 141 also displays the first image captured in good weather at the same position as the second image was captured.

As shown in Fig. 2(A) and Fig. 2(B), the first image displayed in the first display area 141 shows the lanes 1411 of the road on which the vehicle is traveling, along with guardrails 1410 provided on the side of the road on which the vehicle is traveling. On the other hand, the second display area 142 shows an image of a situation in which the weather is bad, the road is covered with snow, and it is unclear where the road is.

Let's assume that the vehicle is a snowplow. When the driver of the snowplow looks at the image displayed in the second display area 142 as the current image, he or she cannot even tell if the vehicle is traveling on a road.

However, because an image of the same location in good weather is displayed in the first display area 141, the driver can determine whether or not he or she is currently traveling on a road by checking the image displayed in the first display area 141.

Moreover, the first image displayed in the first display area 141 is an image according to the current position of the vehicle. In FIG. 2(A), the vehicle is currently traveling in the center of the road, so the first image displayed in the first display area 141 shows the vehicle traveling in the center of the road. Also, as shown in FIG. 2(B), when the vehicle is currently traveling in the left lane of the road, the first image displayed in the first display area 141 shows the vehicle traveling in the left lane of the road.

In this way, the display unit 140 displays the first image in the first display area 141 in conjunction with the second image displayed in the second display area 142. Furthermore, the display unit 140 displays the first image according to the position of the road contained in the second image. In this way, an image of good weather according to the current position of the snowplow is displayed on the display unit 140. The driver of the snowplow can know the current position on the road, so he or she can safely drive the snowplow and safely perform snow removal work.

For example, when the weather is good, the vehicle is driven in multiple directions on the road (e.g., the right lane of the road, the center of the road, the left lane of the road), and multiple images corresponding to the multiple directions are captured by the imaging unit 110 and stored in the memory unit 130. Then, the control unit 150 reads out an image (an image signal) having position information corresponding to the current position of the vehicle (i.e., the position when the weather is bad) from the memory unit 130 and displays it on the display unit 140. As a result, as described above, the first image corresponding to the position of the road included in the second image is displayed.

Furthermore, the display unit 140 can display the first image according to the direction of rotation of the steering wheel of the snowplow. For example, as shown in FIG. 2(A), when the vehicle is traveling in the center of the road and the steering wheel is rotated counterclockwise, the first image can be displayed in which the vehicle is shifted to the left side of the road, as shown in FIG. 2(B). Also, as shown in FIG. 2(A), when the vehicle is traveling in the center of the road and the steering wheel is rotated clockwise, the first image can be displayed in which the vehicle is shifted to the right side of the road. This makes it possible to display a first image that matches the driver's sense in the first display area 141.

The first display area 141 displays time information when the first image was captured. In the example shown in FIG. 2(A) and FIG. 2(B), the time information is displayed in the upper right corner of the first display area 141, but is not limited to this and may be displayed at any position in the first display area 141. In addition to the date when the first image was captured, the time information may also display the time when the first image was captured (hour, minute, second). The time may be displayed in minutes (hour, minute, second, frame), or may be displayed in image frame units (hour, minute, second, frame). When capturing the first image, the control unit 150 stores the time information obtained from the timer together with the video signal output from the imaging unit 110 in the storage unit 130. Then, when the control unit 150 displays the first image on the display unit 140, the control unit 150 reads out the time information from the storage unit 130 together with the video signal, thereby displaying the first image shown in FIG. 2(A). The time information may also be displayed in the second display area 142. In this case, the time information is the current time. The time information may include the date.

It is desirable for the time information to be as close as possible to the current time (i.e., when the weather is not good). In other words, it is desirable for the image of good weather to be as close as possible to the current time. This is because if the time when the image of good weather is taken is more than a threshold time away from the current time, road conditions may change significantly due to natural disasters, construction work, etc.

(Operation example according to the first embodiment)
Next, an operation example according to the first embodiment will be described.

Figure 3 shows an example of operation according to the first embodiment.

As shown in FIG. 3, in step S10, the control unit 150 starts processing.

In step S11, the control unit 150 stores in the memory unit 130 an image (first image) (image signal) taken when the weather is good, and location information (first location information) indicating the location of the vehicle when the image was captured.

In step S12, the control unit 150 acquires from the imaging unit 110 an image (second image) (image signal) taken when the weather is bad, and acquires from the position information acquisition unit 120 position information (second position information) indicating the position of the vehicle when the image was captured.

In step S13, the control unit 150 reads from the memory unit 130 an image (first image) (image signal) of the location information (first location information) corresponding to the location information (second location information) obtained when the weather was bad.

In step S14, the control unit 150 outputs image signals of the image acquired from the imaging unit 110 (second image) and the image read from the memory unit 130 (first image) to the display unit 140. The display unit 140 displays the first image in the first display area 141 and the second image in the second display area 142.

Then, in step S15, the control unit 150 ends the series of processes.

In this way, in the display device 100 of the first embodiment, the first image captured by the imaging unit 110 when the weather is good and the second image captured by the imaging unit 110 when the weather is bad can be displayed on the display unit 140 according to the position of the vehicle. As a result, even when the current weather is bad, for example, the first image showing the vehicle traveling in the same position as when the weather is good is displayed, so the driver of the vehicle can drive the vehicle on the road while looking at the first image. Therefore, it is possible to provide a display device that supports the vehicle so that it can travel safely on the road even when the weather is bad.

(Another Example of the First Embodiment)
In the first embodiment, a snowplow is used as an example, but the vehicle is not limited to this. For example, the vehicle may be an emergency vehicle. An emergency vehicle is, for example, an emergency automobile such as an ambulance, a fire engine, or a police vehicle.

In the first embodiment, an example of a display screen displayed on the display unit 140 has been described in which the display area of the second display area 142 is larger than the display area of the first display area 141, but this is not limiting. For example, the display area of the first display area 141 may be larger than the display area of the second display area 142. The display device 100 can also switch the size of the area displayed on the display unit 140 for the first image and the second image. The display unit 140 may switch the area in accordance with an instruction from the control unit 150. Alternatively, the first display area 141 and the second display area 142 may have the same display area.

Furthermore, the road described in the first embodiment may be a specific road. For example, the road from which the snowplow removes snow may be a road with multiple lanes or a road with heavy traffic, and a specific road is the road that is the target of snow removal. For example, snowplows may be used as a commissioned task by a city or prefecture, and a specific road is the road that is the target of snow removal at the request of the city or prefecture.

Furthermore, the first image described in the first embodiment only needs to capture an image of the road on which the vehicle is traveling and the conditions around the road. Therefore, it is desirable for the imaging unit 110 to be directed forward of the vehicle.

Furthermore, in the first image described in the first embodiment, guardrails and lanes are used as examples of things representing the conditions around the road, but the present invention is not limited to this. For example, things representing the conditions around the road may be gutters (or ditches) on the side of the road, or curbs on the side of the road.

[Second embodiment]
Next, a second embodiment will be described.

The second embodiment is an example in which a line for locating a vehicle is displayed in a current image (second image) in which the weather is bad.

Figures 4(A) and 4(B) are diagrams showing examples of display screens displayed on the display unit 140. Figure 4(A) shows an example of the display screen when the vehicle is traveling in the left lane of the road, and Figure 4(B) shows an example of the display screen when the vehicle is traveling in the center of the road.

As shown in FIG. 4(A) and FIG. 4(B), a positioning line 1421 for positioning the vehicle is displayed in the second display area 142. The positioning line 1421 is not an image of what actually exists on the road captured by the imaging unit 110 in bad weather, but is, for example, a line displayed superimposed on the second image as a line representing the edge of the road. The positioning line 1421 may correspond to the white line 1412 drawn on the edge of the road captured in good weather. Alternatively, the positioning line 1421 may correspond to the edge of the road captured in good weather. The edge of the road represents, for example, a boundary line between a road formed of asphalt and soil.

In this way, because the positioning line 1421 is displayed in the second display area 142 in the image (second image) when the weather is bad, the driver of the snowplow can predict the position of the blade that will remove the snow by looking at the positioning line 1421. This allows the snowplow to properly remove snow that has accumulated up to the left edge of the road. In addition, because the positioning line 1421 represents the left edge of the road, it also serves as a guide for the driver to drive the snowplow safely on the road. Therefore, the positioning line 1421 enables the snowplow to properly remove snow and also enables the snowplow to be driven safely on the road.

The display position of the positioning line 1421 is also displayed according to the direction of rotation of the snowplow's steering wheel, as in the first embodiment. For example, as shown in FIG. 4(A), when the snowplow is traveling in the center of the road and the driver turns the steering wheel counterclockwise, the display position of the positioning line 1421 is also displayed in accordance with the snowplow traveling in the left lane of the road, as shown in FIG. 4(B). Conversely, when the snowplow is traveling in the left lane of the road and the driver turns the steering wheel clockwise, as shown in FIG. 4(B), the display position of the positioning line 1421 is also displayed in accordance with the snowplow traveling in the center of the road, as shown in FIG. 4(A).

The positioning line 1421 may be displayed based on the first image taken when the weather is good.

First, the positioning line 1421 may be displayed based on the white line of the road included in the first image. For example, the control unit 150 may read the image signal of the first image from the storage unit 130 and analyze the first image (for example, image analysis using machine learning may be used) to recognize the white line included in the first image. Then, the control unit 150 stores the image signal representing the white line in the storage unit 130 together with the image signal and position information of the first image. When the vehicle is traveling in bad weather, the control unit 150 reads from the storage unit 130 the image signal of the first image having position information that matches the position information acquired from the position information acquisition unit 120, and reads from the storage unit 130 the image signal of the white line having position information that matches the position information acquired from the position information acquisition unit 120. The control unit 150 can display the second image including the white line on the display unit 140 by outputting the image signal of the second image captured by the imaging unit 110 and the image signal of the white line to the display unit 140. At the same time, the control unit 150 outputs an image signal of the first image read from the storage unit 130 to the display unit 140, thereby causing the display unit 140 to display the first image together with the second image. This allows the display unit 140 to display, for example, the image shown in FIG. 4(A).

Secondly, the positioning line 1421 may be displayed using a pole provided on the vehicle. For example, a pole having a predetermined length in the vertical direction is attached to the front left side of the vehicle. When the weather is good, the imaging unit 110 is used to capture the pole along with the surrounding road conditions. The control unit 150 stores a predetermined part of the pole, such as the lower end, center, or upper end of the pole, when the imaging unit 110 captures the pole and the road surroundings, as an image of the edge of the road (or the part corresponding to the white line) (hereinafter, sometimes referred to as an "edge image") in the memory unit 130 together with the image signal and position information of the first image. Then, when the vehicle is traveling in bad weather, the control unit 150 reads out from the memory unit 130 the image signal of the first image having position information that matches the position information acquired from the position information acquisition unit 120, and reads out the image signal of the edge image having position information that matches the position information acquired from the position information acquisition unit 120. The control unit 150 can display the second image including the positioning line 1421 on the display unit 140 by outputting the image signal of the end image to the display unit 140 together with the image signal of the second image captured by the imaging unit 110. At the same time, the control unit 150 can display the first image together with the second image on the display unit 140 by outputting the image signal of the first image read from the storage unit 130 to the display unit 140. The display unit 140 may adjust the display area so that the image of the pole included in the first image is not displayed. This allows the display unit 140 to display, for example, the image shown in FIG. 4(A).

Thirdly, the positioning line 1421 may be displayed using laser light. For example, the vehicle has an irradiation unit that irradiates laser light toward the road in front of it. The irradiation angle of the irradiation unit may be fixed. The laser light irradiated from the irradiation unit is irradiated to a predetermined area of the road, and the imaging unit 110 can capture the reflected light as the first image. The reflected light portion included in the first image can become part of the positioning line 1421. A series of multiple images forms an image for the positioning line. The image signal of the image for the positioning line is stored in the memory unit 130 together with the first image and the position information. Then, when the vehicle is traveling in bad weather, the control unit 150 reads out from the memory unit 130 the image signal of the first image having position information that matches the position information acquired from the position information acquisition unit 120, and reads out the image signal of the image for the positioning line having position information that matches the position information acquired from the position information acquisition unit 120. The control unit 150 outputs an image signal of an image for the positioning line together with an image signal of the second image captured by the imaging unit 110 to the display unit 140, thereby causing the display unit 140 to display the second image including the positioning line 1421. At the same time, the control unit 150 outputs an image signal of the first image read from the storage unit 130 to the display unit 140, thereby causing the display unit 140 to display the first image together with the second image. This makes it possible to display, for example, the image shown in FIG. 4(A) on the display unit 140.

The positioning line 1421 may be a lane marked on a road.

First, the lane may be a center line that represents a median strip displayed in the center of a road. Figures 5(A) and 5(B) are diagrams showing an example in which the lane 1411 representing the center line is the positioning line 1422. In this case, the positioning line 1422 can be displayed in the same manner as when the white line 1412 described above is the positioning line 1421. As shown in Figures 5(A) and 5(B), even if the lane 1411 representing the center line is the positioning line 1422, the positioning line 1422 is displayed in the second image in bad weather, so that the driver of the snowplow can drive the snowplow using the positioning line 1422 as a guide. Therefore, the positioning line 1422 makes it possible to drive the snowplow safely.

Secondly, the lane may be a dividing line that separates the travel lanes of a road. For example, the positioning line 1422 shown in Figures 5(A) and 5(B) may represent a dividing line. In this case, too, it can be implemented in the same way as in the case of the white lines described above.

(Another example of the second embodiment)
The other examples described in the first embodiment are also applicable to the second embodiment.

In the second embodiment, an example has been described in which the positioning lines are a positioning line 1421 (e.g., FIG. 4(A) and FIG. 4(B)) displayed at the end of the road and a positioning line 1422 (e.g., FIG. 5(A) and FIG. 5(B)) displayed in the center of the road (or separating the driving lanes), but this is not limiting. The two positioning lines 1421 and 1422 may be displayed simultaneously in the second image. Alternatively, the two positioning lines 1421 and 1422 may be displayed alternately in the second image.

In the second embodiment, an example in which the image processing for analyzing the white line from the first image is performed in the control unit 150 has been described, but the present invention is not limited to this. For example, the image processing may be performed in an external image processing device. In this case, the display device 100 may have an external interface that enables external connection. Under the control of the control unit 150, an image signal of the first image is read from the storage unit 130 and transmitted to the image processing device via the external interface. In the image processing device, an image signal of the white line contained in the first image is recognized, and the image signal is stored in the storage unit 130 via the external interface under the control of the control unit 150. The subsequent steps can be implemented in the same manner as in the second embodiment. Communication between the external interface and the image processing device may be wired or wireless.

[Other embodiments]
In the first and second embodiments, examples of a snowplow or an emergency vehicle with a driver have been described as a vehicle, but the present invention is not limited thereto. For example, the vehicle may be an autonomous vehicle or an unmanned vehicle without a driver. Alternatively, instead of a vehicle, an unmanned aerial vehicle such as a drone may be used. In this case, the display device 100 may be attached to the unmanned aerial vehicle, and the display device 100 may include a wireless communication unit. Image signals of the first image and the second image captured by the imaging unit 110 of the display device 100 may be transmitted to the image processing device via the wireless communication unit. While viewing the first image displayed on the display unit of the image processing device, the operator can confirm that an image of good weather is being captured. In addition, while viewing the second image and the first image (for example, the images shown in FIG. 4A and FIG. 4B) displayed on the display unit of the image processing device, the operator can also confirm that the unmanned aerial vehicle is flying on the road. The vehicle may be a patrol car that discovers abnormalities on the road and grasps the road conditions.

Also, for example, a method including the operation of one or more components of the device described in this specification may be provided, and a program for causing a computer to execute the operation of the above components may be provided. The program may be recorded on a computer-readable medium. Using the computer-readable medium, it is possible to install the program on a computer. Here, the computer-readable medium on which the program is recorded may be a non-transient recording medium. The non-transient recording medium is not particularly limited, and may be, for example, a recording medium such as a CD-ROM or a DVD-ROM. An example of such a recording medium is the storage unit 130 described above.

In addition, the circuits that execute the various processes performed by the display device 100 may be integrated, and at least a portion of the display device 100 may be configured as a semiconductor integrated circuit (chip set, SoC).

Although one embodiment has been described in detail above with reference to the drawings, the specific configuration is not limited to the above, and various design changes can be made without departing from the gist of the invention. In addition, it is also possible to combine all or part of each embodiment, process, and step as long as there are no contradictions.

100: Display device 110: Imaging unit 120: Position information acquisition unit 130: Storage unit 140: Display unit 150: Control unit

Claims (6)

An imaging unit that captures an image of a road and a situation around the road;
a position information acquisition unit that acquires position information representing a position of a vehicle when the image is captured;
a storage unit configured to store a first image representing the image captured by the imaging unit when the weather is good;
A display unit;
a control unit that causes the display unit to simultaneously display a second image representing the image captured by the imaging unit when the weather is bad and the first image in accordance with a position of the vehicle,
The control unit causes the display unit to display the second image including a positioning line indicating a line that serves as a guide for safely driving the vehicle on a road;
the positioning line corresponds to at least any one of a line representing an edge of the road in the first image, a center line representing a median strip displayed in the center of the road in the first image, and a dividing line separating driving lanes of the road in the first image,
The control unit causes the second image to be displayed on the display unit so as to superimpose the positioning line on the second image.
Display device. the storage unit stores first position information representing position information acquired from the position information acquisition unit when the first image is captured;
the control unit reads out from the storage unit the first image having the first position information corresponding to second position information representing position information acquired from the position information acquisition unit when the second image is captured by the imaging unit, and displays the first image on the display unit.
The display device according to claim 1. The display device according to claim 1 , wherein the display unit displays the first image in conjunction with the second image. The display device according to claim 2 , wherein the control unit causes the display unit to display the first image of the first position information having the same position information as the second position information. The display device according to claim 1 , wherein the display unit displays the first image in accordance with a position of the road included in the second image. The display device according to claim 1 , wherein the control unit causes the display unit to display the positioning line in accordance with a rotation direction of a steering wheel of the vehicle.

Images