WO2021029016A1

WO2021029016A1 - Guide display method

Info

Publication number: WO2021029016A1
Application number: PCT/JP2019/031863
Authority: WO
Inventors: 山岡　大祐; 田中　一彦; 祐瀧口; 瞳 ▲濱▼村
Original assignee: 本田技研工業株式会社
Priority date: 2019-08-13
Filing date: 2019-08-13
Publication date: 2021-02-18
Also published as: JPWO2021029016A1; JP7212788B2

Abstract

Provided is a design that appropriately displays a guide for traveling in a post-snowfall state. Photograph data representing a post-snowfall state in a prescribed position is acquired, and on the basis of the acquired photograph data, image data corresponding to the post-snowfall state represented by the photograph data is acquired from a storage means that stores image data representing a pre-snowfall state. Display data is generated on the basis of the acquired photograph data and image data, the display data being for displaying a guide for traveling in the post-snowfall state represented by the photograph data.

Description

Guide display method

The present invention relates to a guide display.

In recent years, a display technology known as AR (Augmented Reality) has been used in various fields. Patent Document 1 describes an agricultural work machine 1 which is a self-propelled boom sprayer. In Patent Document 1, the captured image A2 taken by the camera 32 attached to the vehicle 10 (agricultural work machine 1) is superimposed on the guide image A1 by the arithmetic unit 35b of the computer 35 and displayed as AR on the display 36. It is stated that. Further, since the guide image A1 and the actual field shot image A2 are superimposed and the guide image A3 is displayed, the information shown in the guide image A1 (for example, the guide line L, the traveling locus X, the work area S, etc.) ) And the position of crop C, the structure, etc. can be grasped by the worker.

JP-A-2015-164001

In Patent Document 1, the guide image A1 is corrected based on the captured image A2 of the grid pattern drawn on the ground of the field which is the moving region of the vehicle 10, and the corrected guide image A1 is superimposed on the captured image A2. To.

However, when the vehicle travels on the snow surface, the ground is covered with snow, and objects such as irrigation canals and curbs that hinder the travel may exist under the snow surface. Therefore, further improvement is required for the guide display in the state after snowfall.

An object of the present invention is to provide a mechanism for appropriately displaying a guide for traveling in a state after snowfall.

The guide display method according to the present invention includes a first acquisition step of acquiring photographing data representing a state after snow accumulation at a predetermined position, and snow accumulation at the predetermined position based on the photographing data acquired in the first acquisition step. A second acquisition step of acquiring image data corresponding to the state after snow accumulation represented by the shooting data from a first storage means for storing image data representing the previous state, and the shooting acquired in the first acquisition step. Based on the data and the image data acquired in the second acquisition step, a generation step of generating display data for displaying a guide for traveling in the state after snowfall represented by the shooting data, and a generation step. It is characterized by having.

The guide display method according to the present invention is a guide display method executed in a guide display system including a server and a vehicle capable of communicating with the server, and the server represents a state after snowfall at a predetermined position. Based on the first receiving step of receiving the shooting data from the vehicle and the shooting data received in the first receiving step, the shooting is performed from a storage means for storing image data representing the state before snow accumulation at the predetermined position. Based on the acquisition step of acquiring the image data corresponding to the state after snow accumulation represented by the data, the shooting data received in the first reception step, and the image data acquired in the acquisition step, the said A generation step of generating display data for displaying a guide for traveling in the state after snowfall represented by the shooting data, and a first transmission step of transmitting the display data generated in the generation step to the vehicle. , The second transmission step of transmitting the photographed data photographed by the photographing means to the server, the second receiving step of receiving the display data from the server, and the second receiving process of the vehicle. It is characterized by having a display step of displaying a screen on which the guide is superimposed on the state after snow accumulation based on the display data.

The guide for driving in the state after snowfall can be displayed appropriately.

Other features and advantages of the present invention will be clarified by the following description with reference to the accompanying drawings. In the attached drawings, the same or similar configurations are given the same reference numbers.

The accompanying drawings are included in the specification, form a part thereof, show an embodiment of the present invention, and are used together with the description to explain the principle of the present invention.
It is a figure which shows the structure of the guide display system. It is a figure which shows the structure of a snowplow. It is a block diagram which shows the structure of a snow blower. It is a block diagram which shows the structure of a four-wheeled vehicle. It is a block diagram which shows the structure of a server. It is a figure for demonstrating the generation of AR display data. It is a flowchart which shows the processing on the vehicle side. It is a flowchart which shows the process on the server side. It is a flowchart which shows the process on the server side. It is a flowchart which shows the process on the server side. It is a figure for demonstrating the deviation of a direction. It is a flowchart which shows the process on the server side. It is a flowchart which shows the process on the server side. It is a flowchart which shows the process on the server side. It is a flowchart which shows the process on the server side. It is a figure which shows the image of the shooting data. It is a figure which shows the image of the image data before snow cover. It is a figure which shows the AR object. It is a figure which shows the image which AR object superposed on the photographed data. It is a figure which shows the work line object. It is a figure which shows the snow cover information object. It is a figure which shows the warning object. It is a figure which shows the warning object.

FIG. 1 is a diagram showing a configuration of a guide display system 100 according to the present embodiment. The guide display system 100 is a system that displays guide information indicating a travel path on a photographed image taken by a vehicle traveling on a snow surface in AR (Augmented Reality) and provides the passengers of the vehicle. As shown in FIG. 1, the guide display system 100 includes a server 101,

vehicles

104, 105, 106, and a portable terminal 107. The server 101 is a server for providing the above-mentioned guide display service. Although the server 101 is shown as one device in FIG. 1, it may be composed of a plurality of devices. That is, the server 101 in the present embodiment includes a configuration in which one device provides a server function and a configuration in which a plurality of devices cooperate to provide a server function. In the present embodiment, the "vehicle" is not limited to that form, and includes a four-wheeled vehicle, a saddle-riding two-wheeled vehicle, and a work vehicle as shown by

vehicles

104, 105, and 106 in FIG. Including. The work vehicle is a vehicle for special purposes, such as a snowplow. In the guide display system 100, the passengers of the

vehicles

104, 105, 106 can enjoy the guide display service. In addition, a portable terminal 107 such as a smartphone can also be connected to the guide display system 100, and a migrant who holds the portable terminal 107 and moves on a snow surface enjoys the guide display service of the guide display system 100. be able to. In the present embodiment, the passengers of the vehicle include the driver of the vehicle and other occupants.

The wireless base station 103 is, for example, a wireless base station provided in an area where the server 101 can provide a guide display service, and can communicate with the

vehicles

104, 105, 106, and the portable terminal 107. Further, the server 101 is configured to be able to communicate with the radio base station 103 via the network 102. With such a configuration, for example, the

vehicles

104, 105, 106 and the portable terminal 107 can transmit vehicle information such as position information or terminal information to the server 101, and the server 101 can send the

vehicles

104, 105, 106, Display data can be transmitted to the portable terminal 107. In FIG. 1, the network 102 is shown as a single network, but a plurality of networks may be configured by a router or the like. Further, the server 101, the

vehicles

104, 105, 106, and the portable terminal 107 can be connected to a network other than the network 102 shown in FIG. 1, and can be connected to, for example, the Internet.

The communication device 108 is, for example, a roadside device provided along the road, and for example, communication with at least one of the

vehicles

104, 105, 106, and the portable terminal 107 by DSRC (Dedicated Short Range Communication) (road-to-vehicle communication, etc.). Is possible. For example, the communication device 108 is used to transmit the vehicle information of the vehicle 104 to the server 101 and to transmit the road surface condition information (snow cover state, frozen state, etc.) to the server 101.

The guide display system 100 is not limited to the configuration shown in FIG. 1, and may include other devices as appropriate. For example, a database server for map information, traffic information, or weather information may be included, and the server 101 may be configured to have access to those database servers.

FIG. 2 is a diagram showing the configuration of the vehicle 106, which is a snowplow. In the present embodiment, the vehicle 106 is a crawler type snow removal vehicle that collects snow by an auger and projects snow from a shooter. A part of the snow removal mechanism of the vehicle 106 will be described. The vehicle 106 has a snow removal unit 201, which includes an auger 203, a blower 204, a sled 210, and a shooter 205 driven by an engine 221. The power of the engine 221 is transmitted in the order of the small diameter pulley 207, the belt 208, the large diameter pulley 209, the drive shaft 206, and the auger shaft 202 to rotate the auger 203. By rotating, the auger 203 collects the snow on the road in the front and back directions of the drawing and sends it to the blower 204, and the centrifugal force of the blower 204 projects the snow to the outside through the shooter 205. The vehicle 106 also has an engine cover 220, an engine air-cooled fan 222, an air-cooled fan shaft 223, and an output pulley connected to the drive wheels 217.

The part of the traveling mechanism of the vehicle 106 will be described. The vehicle 106 has a body frame 219 connected to the rear part of the crawler frame 212, which is provided with a floating wheel 211 at the front and three

lower wheels

213, 214, 215 at the bottom, respectively, and is connected to the front part of the body frame 219. The snow removal unit 201 is configured in. The pivot shaft 218 connects the vehicle body frame 219 to the crawler frame 212. A drive wheel 217 is configured at the rear of the body frame 219, and a crawler belt 216 is wound between the drive wheel 217 and the idler wheel 211, whereby the vehicle 106 is configured as a crawler type snow removal vehicle. Further, the vehicle 106 is equipped with a headlamp 225 and a sub headlamp 226 capable of illuminating the front. Further, the operation unit 224 is a human-machine interface capable of receiving instructions for controlling each unit related to the traveling mechanism, the snow removal mechanism, and the like of the vehicle 106.

In the present embodiment, the vehicle 106 further includes a control unit 229. The control unit 229 acquires the detection signal from the

detection units

227 and 228 for recognizing the external environment attached to the vehicle 106. The

detection units

227 and 228 are, for example, cameras and sensors.

In the present embodiment, the vehicle 106 may be a type on which the operator is boarded or a hand-push type by the operator. Further, the vehicle 106 may be a vehicle capable of fully automatic driving in the absence of a driver. If the type is for the operator to board, a boarding unit (not shown) including a seat, a control unit such as a steering wheel and pedals, and an operation unit 224 is provided at the rear of the engine cover 220.

FIG. 3 is a block diagram showing the configuration of the vehicle 106, which is a snowplow. The control unit 300 includes a main processor 301 and a memory 302. For example, the processor 301 reads the program stored in the storage unit 315 into the memory 302 and executes it, so that the control unit 300 controls the vehicle 106 in an integrated manner. The control unit 300 corresponds to the control unit 229 in FIG. The external world recognition camera 305 and the external world recognition sensor 306 of FIG. 3 correspond to the

detection units

227 and 228 of FIG. The operation unit 314 of FIG. 3 corresponds to the operation unit 224 of FIG. The drive control unit 303 of FIG. 3 corresponds to the engine 221 of FIG. 2, a mechanism control unit (not shown) that controls the mechanism of each part of the vehicle 106, and an electric system control unit (not shown). The traveling mechanism 310 and the snow removing mechanism 311 correspond to the respective parts of the traveling mechanism and the snow removing mechanism described in FIG. The lighting unit 312 includes a notification mechanism such as a headlamp 225 and a sub headlamp 226, and a direction indicator (not shown). The display control unit 304 controls the display on the display unit 313 based on the display data.

The storage unit 315 stores various parameters, data, etc. in addition to the above program. The operation unit 314 includes a steering wheel and pedal for receiving a maneuvering operation from a passenger, an instrument panel, a panel (display unit 313) for displaying various user interface screens, and an input unit for receiving a setting operation. The communication interface (I / F) 309 includes an antenna for communicating with the radio base station 103 and a signal format conversion unit. The GPS sensor 307 is a GPS (Global Positioning System) sensor, and detects the current position of the vehicle 106. The gyro sensor 308 detects the posture of the vehicle 106. The posture of the vehicle 106 is, for example, a roll angle, a pitch angle, and a yaw angle.

The external world recognition camera 305 is a camera that images the external environment, and is attached so that it can image the front, side, and rear of the vehicle 106. The external world recognition sensor 306 is, for example, an ultrasonic sensor, and detects the state of the snow-covered surface, for example, the snow-covered surface is hill-shaped by detecting the reflected wave of the ultrasonic wave emitted in front of the vehicle 106. It is possible. Further, the external world recognition sensor 306 is, for example, an infrared sensor provided above the shooter 205, for example, at the position of the detection unit 227 in FIG. 2, and detects the snow depth in front by projecting infrared rays onto the snow surface. It is possible. Further, for example, it is possible to detect, for example, whether the snow quality is high and the snow surface is hard, or the density is low and the snow surface is soft, based on the intensity of the reflected wave. The outside world recognition camera 305 and the outside world recognition sensor 306 are not limited to the positions of the

detection units

227 and 228 in FIG. For example, it may be on the side surface of the vehicle 106, near the headlamp 225, or near the tip of the shooter 205. For example, by attaching it to a portion close to the tip of the shooter 205, even when the snow depth is about 2 m, it is possible to take an image of a distance in front of the passenger of the vehicle 106, which is a blind spot, with a camera.

In FIGS. 2 and 3, the configuration of the vehicle 106, which is a snowplow, has been described. However, if it is provided with a configuration for recognizing the outside world (camera, etc.), a configuration for recognizing position information and attitude information (GPS sensor, gyro sensor, etc.), a display unit, and a communication I / F, the vehicle 106 The configuration is not limited to that of a four-wheeled vehicle such as the vehicle 104, a two-wheeled vehicle such as the vehicle 105, and a portable terminal 107. Further, in FIGS. 2 and 3, an example in which the vehicle 106 has a configuration for recognizing the outside world has been described, but the configuration for recognizing the outside world with respect to the vehicle 106 may be removable. .. For example, a smartphone equipped with a camera or GPS may be attached to the vehicle 106 by an attachment or the like. Further, the vehicle 106 is not limited to a snowplow as long as it is a vehicle for special purposes, and may be, for example, an agricultural work machine. Hereinafter, a configuration example of a four-wheeled vehicle will be described as a vehicle other than the vehicle 106.

FIG. 4 is a block diagram showing an example of the configuration of the vehicle 104, which is a four-wheeled vehicle. The control unit 400 includes a main processor 401 and a memory 402. For example, the processor 401 reads the program stored in the storage unit into the memory 402 and executes it, so that the control unit 400 controls the vehicle 104 in an integrated manner. Further, the control unit 400 includes an outside world recognition unit 403, an action planning unit 404, a drive control unit 405, and a device control unit 406. Each block is realized by one ECU or a plurality of ECUs.

The outside world recognition unit 403 recognizes the outside world information of the vehicle 104 based on the signals from the outside world recognition camera 407 and the outside world recognition sensor 408. Here, the external world recognition camera 407 is, for example, a camera that photographs the front of the vehicle 104, and is a camera that is attached to the vehicle interior side of the front window at the front of the roof. The external world recognition sensor 408 is, for example, a LIDAR (Light Detection and Ringing) or a millimeter-wave radar that detects a target around the vehicle 104 and measures the distance from the target. The outside world recognition unit 403 recognizes, for example, free spaces such as intersections, railroad crossings, tunnels, and shoulders, and the behavior (speed and traveling direction) of other vehicles based on the signals from the outside world recognition camera 407 and the outside world recognition sensor 408. To do. The GPS sensor 409 detects the current position of the vehicle 104, and the gyro sensor 410 detects the posture of the vehicle 104. The posture of the vehicle 104 is, for example, a roll angle, a pitch angle, and a yaw angle.

The action planning unit 404 plans the actions of the vehicle 104 such as the optimum route and the risk avoidance route based on the position information detected by the external world recognition unit 403 and the GPS sensor 409. The action planning unit 404 performs, for example, an approach determination based on the start point and the end point of an intersection or a railroad crossing, and an action plan based on the behavior prediction of another vehicle. The drive control unit 405 controls the drive force output mechanism 412, the steering mechanism 413, and the brake mechanism 414 based on the action plan by the action planning unit 404. Here, the driving force output mechanism 412 is, for example, a power plant, the steering mechanism 413 is, for example, an electric power steering device, and the brake mechanism 414 is, for example, a disc brake device.

The device control unit 406 controls the device connected to the control unit 400. For example, the device control unit 406 controls the speaker 415 to output a predetermined voice message such as a warning or a message for navigation. Further, for example, the device control unit 406 controls the display device 416 to display various interface screens. Further, for example, the device control unit 406 controls the navigation device 417 and acquires the setting information in the navigation device 417. The communication interface (I / F) 411 includes an antenna for communicating with the radio base station 103 and a signal format conversion unit.

The control unit 400 may appropriately include functional blocks other than those shown in FIG. 4, and may include, for example, an optimum route calculation unit that calculates the optimum route to the destination based on the map information. Further, the control unit 400 may acquire the outside world information from other than the outside world recognition camera 407 and the outside world recognition sensor 408. For example, the control unit 400 may acquire the outside world information via another vehicle. Further, the control unit 400 receives detection signals from not only the GPS sensor 409 and the gyro sensor 410 but also various sensors provided in the vehicle 104. For example, the control unit 400 receives the detection signals of the door open / close sensor and the door lock mechanism sensor provided on the door portion of the vehicle 104 via the ECU configured in the door portion. As a result, the control unit 400 can detect the unlocking of the door and the opening / closing operation of the door.

FIG. 5 is a block diagram showing the configuration of the server 101. The control unit 500 includes a main processor 505, which is a CPU and GPU, and a memory 506, which is a ROM and RAM, and controls the server 101 in an integrated manner. For example, the operation of the present embodiment is realized by the processor 505 reading the program stored in the storage unit 501 into the memory 506 and executing the program. The server 101 can be a computer that implements the present invention relating to a program stored in a storage medium.

The pre-snow state acquisition unit 507 acquires pre-snow image data representing the pre-snow state at a predetermined position. The pre-snow cover state acquisition unit 507 acquires, for example, by the user taking a picture of a state at a predetermined position without snow cover. Alternatively, the pre-snow state acquisition unit 507 acquires street view data taken before snow cover as pre-snow image data by a drone or the like. The pre-snow state acquisition unit 507 stores the acquired pre-snow image data in the storage unit 501 as pre-snow image data 514. The pre-snow state acquisition unit 507 periodically acquires pre-snow image data and updates the pre-snow image data 514.

The shooting data acquisition unit 508 acquires shooting data shot by the

vehicles

104, 105, 106, and the portable terminal 107. In the present embodiment, the shooting data acquired by the shooting data acquisition unit 508 is shooting data obtained by shooting the snow cover state in front of the traveling directions of the

vehicles

104, 105, 106 and the moving direction of the holder of the portable terminal 107. .. The shooting data acquisition unit 508 stores the acquired shooting data in the storage unit 501 as shooting data 517.

The vehicle information acquisition unit 509 acquires vehicle information from the

vehicles

104, 105, and 106. Here, the vehicle information is, for example, vehicle position information, vehicle behavior information, vehicle attitude information, and vehicle attribute information. The position information of the vehicle is, for example, latitude and longitude information, and the information regarding the behavior of the vehicle is, for example, speed and acceleration information, driving information of each part of the vehicle, and detection information such as a camera and a sensor. The information regarding the posture of the vehicle is, for example, tilt information of the vehicle having a roll angle, a pitch angle, and a yaw angle. The vehicle attribute information is, for example, information regarding the size (small, medium, or large) of the vehicle 106 of the snowplow. Further, the vehicle information acquisition unit 509 acquires terminal information from the portable terminal 107. Here, the terminal information includes, for example, position information of the portable terminal and information regarding the behavior of the portable terminal. The position information of the portable terminal is, for example, latitude and longitude information, and the information regarding the behavior of the portable terminal is, for example, speed and acceleration information, detection information of a camera, a sensor, and the like. In the present embodiment, the vehicle 106 will be described as a representative example of the

vehicles

104, 105, 106, and the portable terminal 107. Therefore, the vehicle information in the explanation can be replaced as terminal information. The vehicle information acquisition unit 509 stores the acquired vehicle information in the storage unit 501 as vehicle information 518 (or terminal information).

The image recognition unit 510 analyzes the shooting data acquired by the shooting data acquisition unit 508 and performs image recognition. The image recognition unit 510 extracts, for example, an object from the image represented by the shooting data. The extraction of objects by the image recognition unit 510 will be described later.

The AR object generation unit 511 generates an AR object based on the pre-snow image data acquired by the pre-snow state acquisition unit 507. The generation of the AR object by the AR object generation unit 511 will be described later. The display data generation unit 512 generates display data for AR display based on the shooting data acquired by the shooting data acquisition unit 508 and the AR object generated by the AR object generation unit 511. The generation of display data by the display data generation unit 512 will be described later.

The environmental information acquisition unit 513 acquires environmental information regarding the environment of the vehicle's driving path. The environmental information is, for example, information on snow cover, information on moving objects (pedestrians, etc.) around the vehicle's travel path, slope information on the travel path, and width information. The environmental information acquisition unit 513 acquires environmental information from, for example, a communication device 108 installed near the road, a road surface condition sensor, or an external database server. In addition, the environmental information acquisition unit 513 may acquire information on snow cover from the vehicle 106. The information on snow cover is, for example, information on snow depth and snow quality (hardness, etc.). The environmental information acquisition unit 513 stores the acquired environmental information in the storage material 501 as the environmental information 515. In addition, the environmental information acquisition unit 513 periodically acquires the environmental information and updates the environmental information 515.

The storage unit 501 stores programs, various parameters, data, and the like. Further, as described above, the storage unit 501 stores the image data 514 before snowfall, the environmental information 515, the shooting data 517, and the vehicle information 518. Since the shooting data 517 and the vehicle information 518 are information acquired from each vehicle, they are stored in association with each other as information 516 for each vehicle. In addition, the information 516 for each vehicle also includes snow removal amount information 519, which will be described later.

The display unit 502 is, for example, a display and displays various user interface screens. The operation unit 503 is, for example, a keyboard or a pointing device, and accepts user operations. The communication interface (I / F) 504 is an interface for enabling communication with the network 102, and has a configuration according to the medium of the network 102.

The operation of this embodiment will be described below. In the present embodiment, for example, it is assumed that the four-wheeled vehicle 104 travels on a snow-covered road and the snowplow 106 is snow-removing on a snow-covered road. When the snow depth is several centimeters to several tens of centimeters, the road is completely covered with snow, and it becomes difficult to recognize the boundary between the road and the non-road. At the boundary between the running road and the non-running road, for example, a gutter, an irrigation canal, a curb, a flower bed, a tree planting, or the like may exist. In such cases, the occupants of the vehicle may unintentionally travel over them. If there is no snow, for example, the outside world recognition camera 407 and the outside world recognition sensor 408 of the four-wheeled vehicle can recognize the boundary between the traveling road and the non-traveling road. However, in the above situation, it is difficult to recognize the boundary between the traveling road and the non-traveling road even if these devices are used.

Therefore, in the present embodiment, an AR object is generated from the pre-snow image data previously photographed by the user himself or in Street View or the like at a place corresponding to the traveling position of the vehicle, and the vehicle photographs the AR object. The AR is displayed by superimposing it on the shooting data. Hereinafter, the vehicle 106 will be described as a representative example of the

vehicles

104, 105, 106, and the portable terminal 107.

FIG. 6 is a diagram for explaining the generation of AR display data in the present embodiment. The shooting data 601 is shooting data taken by the vehicle 106. FIG. 16 is a diagram showing an example of shooting data 601. The shooting data 601 is shooting data obtained by shooting the front of the vehicle 106, and as shown in FIG. 16, one surface is covered with snow, and the traveling path cannot be recognized.

The pre-snow image data 602 is data stored in the storage unit 501 of the server 101, and is, for example, street view data before snow cover corresponding to the position of the photographed image of the photographed data 601. FIG. 17 is a diagram showing an example of pre-snow image data 602. As shown in FIG. 17, the shooting data 601 of FIG. 16 shows the state before snowfall, and the traveling path and the gutter can be recognized.

In the present embodiment, the AR object 603 is generated from the pre-snow cover image data 602, and the generated AR object 603 is superimposed on the shooting data 601 to generate the AR display data 604. FIG. 18 is a diagram showing an example of the AR object 603, and FIG. 19 is a diagram showing an example of the AR display data 604. As shown in FIG. 19, in the AR display data 604, the AR object 603 of FIG. 18 is superimposed on the shooting data 601 of FIG. Then, the AR display data 604 of FIG. 19 is displayed on the display unit 313 of the vehicle 106.

In the present embodiment, since the AR display for guiding the traveling path is performed as described above, the passenger of the vehicle 106 can easily recognize which part of the snow surface is the traveling path and which is not the traveling path. Can be done.

FIG. 7 is a flowchart showing a process of requesting the server 101 to display AR, which is executed in the vehicle 106. The process of FIG. 7 is realized, for example, by the processor 301 loading the program stored in the storage unit 315 into the memory 302 and executing the program.

In S101, the control unit 300 determines whether or not an instruction to perform AR display has been received. This determination may be determined, for example, depending on whether or not a menu item (such as an "AR display implementation" item) on the user interface screen displayed on the display unit 313 of the vehicle 106 is selected by the operation unit 314. .. The above user interface screen is, for example, a screen displayed by starting an application provided by the server 101. If it is determined that the instruction to display the AR is accepted, the process proceeds to S102, and if it is determined that the instruction to display the AR is not accepted, the process of FIG. 7 is terminated.

In S102, the control unit 300 acquires shooting data in front of the vehicle 106 in the traveling direction. At that time, the shooting data may be still image data or moving image data. In S103, the control unit 300 transmits a request for AR display data to the server 101 together with the identification information of the vehicle 106.

In S104, the control unit 300 transmits the vehicle information and the shooting data acquired in S102 to the server 101 together with the identification information of the vehicle 106. Here, the vehicle information is, for example, position information detected by the GPS sensor 307. Further, the vehicle information may include other information, for example, information on the posture and behavior of the vehicle 106 detected by the gyro sensor 308, and information on snow cover detected by the external world recognition sensor 306.

Then, in S105, the control unit 300 receives the AR display data from the server 101, and in S106, AR displays based on the received AR display data. In S107, the control unit 300 determines whether or not to continue the AR display. For example, when the control unit 300 detects that the vehicle 106 has stopped and the instruction to end the AR display has been received, the control unit 300 determines that the AR display will not be continued. When it is determined that the AR display is to be continued, the process from S102 is repeated, and when it is determined that the AR display is not to be continued, the process of FIG. 7 is terminated.

FIG. 8 is a flowchart showing a process of generating AR display data executed on the server 101. The process of FIG. 8 is realized, for example, by the processor 505 loading the program stored in the storage unit 501 into the memory 506 and executing the program.

In S201, the control unit 500 determines whether or not the request for AR display data has been received. If it is determined that the request for the AR display data has been received, the process proceeds to S202, and if it is determined that the request for the AR display data has not been received, the process of S201 is repeated.

In S202, the control unit 500 acquires the vehicle information and stores it as the vehicle information 518 in the storage unit 501 together with the identification information of the vehicle 106. In S203, the control unit 500 acquires the shooting data and stores the shooting data 517 in the storage unit 501 together with the identification information of the vehicle 106. In S204, the control unit 500 acquires the pre-snow image data.

FIG. 9 is a flowchart showing a process of acquiring pre-snow image data of S204. In S301, the control unit 500 performs image recognition on the shooting data acquired in S203, and extracts an object in S302. As a result of image recognition, a plurality of objects such as trees, buildings, and utility poles can be recognized. In S302, the control unit 500 extracts at least two objects in the image of the shooting data. For example, in the case of the shooting data of FIG. 16, the object 801 and the object 802 are extracted. By extracting in this way, as will be described later, it is possible to detect a directional deviation between the photographed data and the pre-snow image data.

In S303, the control unit 500 determines whether or not the pre-snow image data including the object extracted in S302 exists. In S303, the control unit 500 targets the pre-snow image data corresponding to the position of the vehicle 106 among the pre-snow image data 514 stored in the storage unit 501, and the pre-snow image data including the object extracted in S302 is generated. Determine if it exists. When it is determined that the pre-snow image data including the object extracted in S302 exists, the control unit 500 acquires the pre-snow image data in S306. After S306, the process of FIG. 9 is completed, and the process proceeds to S205 of FIG. On the other hand, if it is determined that the pre-snow image data including the object extracted in S303 does not exist, the process proceeds to S304. For example, the pre-snow cover image data of FIG. 17 includes

objects

901 and 902 corresponding to the

objects

801 and 802 extracted from the shooting data of FIG. In that case, it is determined that the pre-snow image data including the object extracted in S302 exists.

In S304, the control unit 500 determines whether or not the determination of S303 has been completed for all the extracted objects as a result of image recognition of the shooting data. If it is determined that all the extracted objects are not finished, the process returns to S302 and other objects are extracted. On the other hand, when it is determined that all the extracted objects have been completed, the AR object in S205 of FIG. 8 cannot be generated, so the notification message data indicating that the AR cannot be displayed is sent to the control unit 300 of the vehicle 106. Send. After S305, the processing of FIGS. 9 and 8 is completed. The control unit 300 of the vehicle 106 that has received the notification message data displays the notification message on the display unit 313. The notification message data may be display data or voice output data.

When the process of S204 of FIG. 8 is performed as described above, the pre-snow image data corresponding to the position of the vehicle 106 is acquired. After S204, in S205, the control unit 500 generates an AR object based on the pre-snow image data acquired in S204.

FIG. 10 is a flowchart showing a process of generating the AR object of S205. In S401, the control unit 500 compares the shooting data acquired in S203 with the pre-snow cover image data acquired in S306. In S402, as a result of comparison, does the control unit 500 have a directional deviation between the shooting data acquired in S203 and the pre-snow image data acquired in S306 due to a difference in shooting position and shooting angle? Judge whether or not.

For example, the judgment of the deviation of the direction is performed as follows. As shown in FIG. 11, when two

objects

701 and 702 are photographed from the direction A, it is assumed that the distance between the objects in the photographed image is the distance 703. On the other hand, when shooting from the direction B rotated by an angle θ1 minutes, the distance between the objects in the captured image is 704, and when shooting from the direction C rotated by an angle θ2 minutes, the distance between the objects in the captured image is 704. The distance is 705. As shown in FIG. 11, as the rotation angle increases, the distance between objects in the captured image becomes shorter. In other words, it can be said that there is a correlation between the directionality and the distance between objects in the captured image. Therefore, as a result of comparison in S401, if there is a difference of a predetermined distance or more between the shooting data and the pre-snow image data with respect to the distances between the plurality of objects extracted in S302, it is determined that there is a directional deviation. If it is determined in S402 that there is a directional deviation, the process proceeds to S403, and if it is determined in S402 that there is no directional deviation, the process proceeds to S404.

In S403, the control unit 500 corrects the pre-snow image data according to the deviation of the directionality. The correction here is a correction for matching the pre-snow image data acquired in S306 with the directionality of the shooting data acquired in S203, that is, eliminating the deviation determined in S402, for example. It is a rotation of an angle θ1 minute in FIG. After the rotation is performed based on the shooting coordinates of the pre-snow image data and the position of the vehicle 106, the movement in the shooting direction (for example, direction B) (approaches the

objects

701 and 702 or moves away from the

objects

701 and 702). ) May be performed.

In S404, the control unit 500 acquires the AR display mode specified by the passenger of the vehicle 106. The AR display mode is information about whether or not to display an additional object described later. In the present embodiment, the AR display that enables the distinction between the traveling road and the non-traveling road as shown in FIG. 19 is set as the standard AR display mode. Further, in the present embodiment, in addition to the standard AR display, a display mode for warning the inclination of the vehicle 106, the presence of a moving object such as a person, or the like by the AR display is possible. The selection information of those display modes is transmitted together with the request for AR display data in S103 of FIG. 7, for example. Here, it is described that only the standard AR display mode is selected.

In S405, the control unit 500 generates an AR object for track identification using the image recognition result by the image recognition unit 510.

FIG. 18 is a diagram showing an example of an AR object. For example, the image recognition unit 510 of the server 101 performs image recognition on the pre-snow image data 602 of FIG. 17 and extracts an object. The object extracted here is an object for recognizing the boundary between the traveling path and the non-traveling path, and is, for example, a gutter, an irrigation canal, a curb, a flower bed, or a tree planting. These objects to be image-recognized are learned in advance by deep learning or the like on the server 101, and the image recognition unit 510 extracts those objects from the pre-snow image data 602 using the trained model. ..

For example, the image recognition unit 510 recognizes the gutter 905, the bollard 903, and the road end 904 as a result of the image recognition of the pre-snow image data 602 in FIG. Then, the image recognition unit 510 generates the AR objects 1001, 1002, 1003, 1004, 1005 of FIG. Here, the AR object 1001 corresponds to the gutter 905, and the AR object 1002 corresponds to the roadside edge 904. Further, the AR object 1003 corresponds to the bollard 903. Further, the AR object 1004 is a warning mark indicating that the vehicle cannot travel (cannot enter), and is arranged in the vicinity of the AR objects 1001, 1003, 1002 corresponding to the gutter 905, the bollard 903, and the road end 904. Further, the AR object 1005 is an AR object for representing a travelable area. Although the AR object 1005 has the shape of an arrow pointing in the traveling direction, it does not have to have the shape shown in FIG. 18 as long as it indicates that the AR object 1005 can travel. Further, the AR object 1005 may be dynamically displayed so that the colors are sequentially blinked toward the back side of the screen.

In S406, the control unit 500 determines whether or not to generate an additional AR object based on the AR display mode acquired in S404. Here, if it is determined that the additional AR object is to be generated, the process proceeds to S407, and the additional AR object is generated. The processing of S407 will be described later. On the other hand, if it is determined that the additional AR object is not generated, the process of FIG. 10 is terminated and the process proceeds to S206 of FIG. When only the standard AR display mode is selected as described above, it is determined in S406 that no additional AR object is generated.

In S206 of FIG. 8, the control unit 500 generates AR display data. As described with reference to FIG. 6, the control unit 500 generates AR display data by superimposing the AR object generated in S205 on the shooting data acquired in S203.

In S207, the control unit 500 transmits the AR display data generated in S206 to the vehicle 106 via the communication I / F 504. The control unit 300 of the vehicle 106 causes the display unit 313 to display the AR display screen based on the received AR display data. On the display unit 313, for example, an AR display screen in which an AR object is superimposed on shooting data as shown in FIG. 19 is displayed. After S207, the process of FIG. 8 ends. The description of the AR object may be displayed according to an operation such as matching or clicking an icon with each AR object on the screen by the passenger of the vehicle 106. For example, by matching the icon to the AR object 1003, a balloon display such as "It is a car stop. You cannot pass due to a dead end." May be displayed. Further, the description of the icon may be presented by another method such as in the user interface screen of the application provided in advance from the server 101.

Hereinafter, the processing when it is determined in S406 of FIG. 10 that an additional AR object will be generated will be described.

FIG. 12 is a flowchart showing a process of generating the additional AR object of S407. Subsequent determinations of S501, S503, S505, and S507 are performed based on, for example, the selection state of a menu item on the user interface screen when receiving an instruction to perform AR display. For example, on the user interface screen, a submenu item called "additional AR display" is further provided in addition to the menu item "implementation of AR display", and the detailed items are "warning", "work line", "snow cover information", and "snow removal". It is displayed so that the necessity of each display of "amount" can be selected. The selection information is received, for example, in S201 together with the request for AR display data.

In S501, the control unit 500 determines whether or not to display the warning object based on the above selection information. If it is determined that the warning object is to be displayed, the process proceeds to S502 and the warning object is generated. On the other hand, if it is determined that the warning object is not displayed, the warning object is not generated and the process proceeds to S503.

An example of generating a warning object will be explained. For example, the warning object is an AR object for giving a warning notification by AR display when the inclination of the vehicle 106 becomes equal to or higher than a predetermined value. When traveling on a snow-covered road, the vehicle may tilt depending on the degree of snow depth and the quality of the snow, and if the degree of tilt is large, it may cause a fall or the like. Therefore, in the present embodiment, the control unit 500 monitors the tilted state of the vehicle 106, and when the amount exceeds a predetermined value, causes the vehicle 106 to display a warning object.

When it is determined in S501 that the warning object is to be displayed, in S502, the control unit 500 monitors the inclination of the vehicle 106 based on the vehicle information of the vehicle 106. For example, the control unit 500 acquires information on the posture from the vehicle 106 at predetermined time intervals. The process of FIG. 12 proceeds from S502 to S503 in parallel with the monitoring of the inclination of the vehicle 106 being performed.

FIG. 13 is a flowchart showing the process of monitoring the inclination of the vehicle 106. In S601, the control unit 500 acquires information on the inclination (posture) from the vehicle 106. Then, in S602, it is determined whether or not the information regarding the inclination acquired in S601, for example, the roll angle or the pitch angle is equal to or greater than the threshold value. If it is determined that the threshold value is not equal to or higher than the threshold value, the process of S601 is repeated. On the other hand, if it is determined that the threshold value is equal to or higher than the threshold value, the control unit 500 generates a warning object in S603.

FIG. 22 is a diagram showing an example of a warning object that warns of the inclination of the vehicle 106. As shown in FIG. 22, the warning object 1301 is displayed. In FIG. 22, the warning object 1301 is dynamically displayed so as to repeatedly tilt in the arrow direction, but the display form is not limited as long as it warns of the tilt of the vehicle 106. Other display forms such as blinking display and text display may be used. After S603, if the process of FIG. 13 is completed and the process of FIG. 12 is restarted, or if the process of FIG. 12 is completed, the process proceeds to S206 of FIG.

Another example of generating the warning object of S502 will be described. For example, the warning object is an AR object for giving a warning when there is a moving object such as a person at a position that is a blind spot from the passenger of the vehicle 106. During extremely heavy snowfall, the snow depth is several meters, and even if there is a person on the other side of the walled snow, it may not be visible to the passengers of vehicle 106. If the passenger removes snow without being aware of the existence of such a person, the snow projected from the shooter 205 may be dropped onto the person's head. Therefore, in the present embodiment, when it is estimated that a moving object exists in the blind spot from the passenger of the vehicle 106, the control unit 500 displays a warning object to that effect in AR.

FIG. 14 is a flowchart showing a process of generating the warning object of S502. In S701, the control unit 500 acquires environmental information around the position of the vehicle 106. The environmental information is, for example, information regarding the existence of a moving body around the position of the vehicle 106. Here, the moving body is, for example, a pedestrian, a bicycle, or a vehicle. The control unit 500 may be acquired from a communication device 108 such as a roadside unit, or may be acquired from an external database server. Alternatively, the control unit 500 may acquire information regarding the existence of the moving body from the vehicle information. For example, the control unit 500 may acquire image data of a camera attached to a portion of the vehicle 106 near the tip of the shooter 205.

In S702, the control unit 500 determines whether or not the moving body is recognized based on the environmental information acquired in S701. Here, if it is determined that the moving body has been recognized, the process proceeds to S703. On the other hand, when it is determined that the moving object is not recognized, it is not necessary to generate the warning object, so the process of FIG. 14 is terminated and the process proceeds to S503 of FIG.

In S703, the control unit 500 performs image recognition of the shooting data acquired in S203, and determines in S704 whether or not the moving object is recognized on the shooting data. When it is determined that the moving object is recognized in S704, a warning object is generated in S705. The case of proceeding to S705 is, for example, a state in which it is recognized from the environmental information that a pedestrian exists in the right front in the traveling direction of the vehicle 106, and is also recognized in the shooting data by the vehicle 106. Therefore, in S705, the control unit 500 generates a warning object so that the pedestrian object recognized on the shooting data can be easily identified by the passenger, such as surrounding the pedestrian object with a quadrangle. After S705, the process of FIG. 14 is completed, and the process proceeds to S503 of FIG.

On the other hand, if it is determined in S704 that the moving object is not recognized, a warning object is generated in S706. The case of proceeding to S706 is, for example, a state in which it is recognized from the environmental information that a pedestrian exists in the right front in the traveling direction of the vehicle 106, but is not recognized in the shooting data data of the vehicle 106. This may be the case when the snow depth on the right front of the vehicle 106 is several meters and the pedestrian on the other side of the snow is invisible to the passengers, as shown in FIG. Therefore, in S706, a warning object is generated as in the warning object 1401 of FIG. 23, and a message such as "There is a person on the other side of the snow!" Is displayed in the vicinity thereof. By such a warning display, it is possible to make the passenger recognize the moving body existing at the position where the passenger of the vehicle 106 becomes a blind spot. Further, the warning message 1401 may be dynamically displayed by blinking or the like to enhance the recognition effect. After S706, the process of FIG. 14 is completed, and the process proceeds to S503 of FIG.

Refer to FIG. 12 again. In S503, the control unit 500 determines whether or not to display the work line based on the selection information. If it is determined that the work line is to be displayed, the process proceeds to S504 and the work line object is generated. On the other hand, if it is determined that the work line is not displayed, the work line object is not generated and the process proceeds to S505.

An example of generating a work line object will be explained. For example, the work line object is a snow removal work line of the vehicle 106, which is a snow remover. When the vehicle 106 is a small snowplow and the travel path width is wide, the work is performed while reciprocating a certain distance. Further, in the case of a hand-push type snowplow, it may be difficult to drive in a straight line depending on the snow condition and the proficiency of the passenger's work. Therefore, in the present embodiment, in addition to displaying the travel path and the non-travel path in an distinguishable manner, for example, as shown in FIG. 20, the control unit 500 AR-displays the work line as in the work line object 1101. Let me. At that time, the control unit 500 generates a work line object according to the vehicle information acquired from the vehicle 106 and the environmental information. For example, a medium-sized snowplow and a small-sized snowplow have different numbers of round trips for the same road width. Therefore, in S504, the control unit 500 determines the number of round trips based on the size of the snowplow obtained from the vehicle information and the travel path width obtained from the environmental information, and generates a work line object according to the number of round trips. .. For example, according to the work line object 1101 of FIG. 20, after making three round trips, the user is guided to exit the traveling path. In this way, the passenger of the vehicle 106 can be easily made to recognize the route going out of the traveling path from the portion without the gutter 905.

The display of the work line object 1101 may be changed according to the traveling of the vehicle 106. For example, when the vehicle 106 is traveling on a line that greatly deviates from the work line of the work line object 1101, the color of the work line object 1101 is blinked in red, and "off the line" or the like is displayed. You may want to display a message. After S504, the process proceeds to S505.

In S505, the control unit 500 determines whether or not to display the information regarding the snow cover based on the selection information. If it is determined that the information regarding the snow cover is to be displayed, the process proceeds to S506, and the snow cover information object is generated. On the other hand, if it is determined that the information regarding the snow cover is not displayed, the snow cover information object is not generated and the process proceeds to S507.

An example of generating a snow cover information object will be described. Snow quality, such as the density of snow, may vary depending on the location of the snow. Such a difference in snow quality may affect not only the running of the vehicle 106 but also the snow removal work of the snowplow. Therefore, in the present embodiment, in addition to displaying the traveling road and the non-traveling road in an identifiable manner, for example, as shown in FIG. 21, the control unit 500, like the snow information objects 1201, 1202, 1203, snow. AR display the difference in quality. The control unit 500 generates snow information objects 1201, 1202, 1203 according to the snow quality information. For example, the snow information objects 1201, 1202, and 1203 indicate that the snow density increases (hardness increases) in that order. The snow quality information may be acquired from the vehicle information (sensor information) from the vehicle 106 or from the environmental information. The display as shown in FIG. 21 makes it possible for the passengers of the vehicle 106 to easily recognize that the snow quality becomes harder as the distance from the traveling path increases. Although the display of the object 1005 is omitted in FIG. 21 for the purpose of explaining this display, the object 1005 may be displayed together.

Information on snow cover is not limited to snow quality, but may be other information. For example, it may be snow depth. In that case, the control unit 500 displays the snow depth information in AR in addition to displaying the traveling road and the non-traveling road in an distinguishable manner. As a form of the AR display, for example, in the shooting data, a numerical value such as "1 m" may be displayed beside an object recognized as a snowy mountain covered with snow in a hill shape. Alternatively, contour lines (50 cm line, 1 m line, etc.) may be superimposed on the object recognized as the snowy mountain. After S506, proceed to S507.

In S507, the control unit 500 determines whether or not to display the amount of snow removal based on the selection information. If it is determined that the snow removal amount is to be displayed, the process proceeds to S508 and the snow removal amount object is generated. On the other hand, if it is determined that the snow removal amount is not displayed, the snow removal amount object is not generated, the process of FIG. 12 is terminated, and the process proceeds to S206 of FIG.

An example of generating a snow removal amount object will be described. FIG. 15 is a flowchart showing an example of a process for generating a snow removal amount object. In S801, the control unit 500 determines whether or not snow is being removed based on the vehicle information from the vehicle 106. If it is determined that snow is being removed, the process proceeds to S802. On the other hand, when it is determined that the snow is not being removed, such as when the snow is stopped, the processing of FIGS. 15 and 12 is terminated, and the process proceeds to S206 of FIG.

In S802, the control unit 500 acquires snow removal amount information. For example, the control unit 500 may acquire the engine torque as the snow removal amount information from the vehicle information from the vehicle 106. Then, in S803, the control unit 500 estimates the amount of snow removal based on the acquired engine torque.

In S804, the control unit 500 creates a snow removal amount object according to the snow removal amount estimated in S803. The snow removal amount object is an object that indicates a numerical value, and is an object for displaying a numerical display such as "10 t / h" at the edge of the screen like a score. With such a display, the passenger can be made to feel like a game, and can be used for income conversion associated with snow removal work. After S804, the process of S801 is repeated. In parallel with the execution of the processes S801 to S804, the processes after S206 in FIG. 8 are executed.

The control unit 500 may store the snow removal amount information acquired in S802 in the storage unit 501 as the snow removal amount information 519 and use it for maintenance notification. For example, the control unit 500 may notify the vehicle 106 that it is time to replace consumables such as battery replacement based on the accumulated snow removal amount information 519.

As described above, according to the present embodiment, when the vehicle travels on a snow surface, the passenger can easily recognize the traveling road and the non-traveling road. Further, in the present embodiment, as shown in FIG. 1, a mode in which the server 101 is configured as a separate device from the

vehicles

104, 105, 106 and the portable terminal 107 has been described, but at least a part of the server 101 is the vehicle 104. , 105, 106, and the portable terminal 107, and the real-time property from the acquisition of the shooting data to the generation of the AR display data and the AR display may be improved.

<Summary of Embodiment>
The guide display method of the above embodiment is based on the first acquisition step (S203) of acquiring shooting data representing the state after snowfall at a predetermined position and the shooting data acquired in the first acquisition step. The second acquisition step (S204) of acquiring the image data corresponding to the state after snowfall represented by the shooting data from the first storage means (514) that stores the image data representing the state before snowfall at the position, and the first. 1 To display a guide for traveling in the state after snowfall represented by the shooting data, based on the shooting data acquired in the acquisition step and the image data acquired in the second acquisition step. It is characterized by having a generation step (S205) for generating display data.

With such a configuration, when the vehicle travels on a snow surface, the passenger can easily recognize the traveling road and the non-traveling road.

Further, the guide is characterized by including an object representing a traveling path and an object representing a non-traveling path. Further, the object representing the non-traveling road is characterized by including a gutter.

With such a configuration, for example, the passenger can easily recognize the gutter.

Further, the second acquisition step is characterized in that the image data including the object recognized by the shooting data acquired in the first acquisition step is acquired from the first storage means (FIG. 9).

With such a configuration, it is possible to acquire image data representing the state before snowfall based on the object.

Further, the guide display method is a determination step (determining whether or not there is a directional deviation between the captured data acquired in the first acquisition step and the image data acquired from the first storage means (). S402) and the correction step (S402), which corrects the image data acquired from the first storage means so as to eliminate the directional deviation when it is determined in the determination step that there is the directional deviation. S403), and the second acquisition step is characterized in that the image data corrected in the correction step is acquired.

With such a configuration, even if the directionality of the image data representing the state before snowfall deviates from the directionality of the shooting data, it can be used for generating display data.

Further, the guide display method includes a display control step (S206, S207) of displaying a screen on which the guide is superimposed on the state after snow accumulation on the display device based on the display data generated in the generation step. It is further characterized by having. Further, the display device (313) is characterized in that it is configured as a moving body.

With such a configuration, it is possible to display the screen on which the guide is superimposed on the state after snowfall on the display device.

Further, the guide display method further includes a third acquisition step (FIG. 13) for acquiring information on the inclination of the moving object, and the generation step notifies that the inclination of the moving object is equal to or higher than a predetermined value. It is characterized by further creating an object for the purpose.

With such a configuration, it is possible to notify when the inclination of the moving body exceeds a predetermined value.

Further, the guide display method is characterized in that the moving body is a snowplow. Further, the guide display method further includes a fourth acquisition step (FIG. 15) for acquiring information on the amount of snow removed by the snowplow, and the generation step further includes an object for notifying the information on the amount of snow removal. It is characterized by generating. Further, the guide display method is further characterized by further including a storage step of storing the snow removal amount information acquired in the fourth acquisition step in the second storage means (519).

With such a configuration, it is possible to notify the vehicle, which is a snowplow, of information on the amount of snow removed.

The present invention is not limited to the above embodiments, and various modifications and modifications can be made without departing from the spirit and scope of the present invention. Therefore, in order to make the scope of the present invention public, the following claims are attached.

100 guide display system: 101 server: 104, 105, 106 vehicle: 107 portable terminal: 313 display unit: 416 display device: 500 control unit

Claims

The first acquisition step of acquiring shooting data representing the state after snowfall at a predetermined position,
Based on the shooting data acquired in the first acquisition step, an image corresponding to the state after snowfall represented by the shooting data from the first storage means for storing image data representing the state before snowfall at the predetermined position. The second acquisition process to acquire data and
Based on the shooting data acquired in the first acquisition step and the image data acquired in the second acquisition step, a guide for traveling in the state after snowfall represented by the shooting data is displayed. Generation process to generate display data for
A guide display method characterized by having.
The guide display method according to claim 1, wherein the guide includes an object representing a traveling path and an object representing a non-traveling path.
The guide display method according to claim 2, wherein the object representing the non-traveling road includes a gutter.
Any of claims 1 to 3, wherein the second acquisition step acquires the image data including an object recognized by the shooting data acquired in the first acquisition step from the first storage means. The guide display method described in item 1.
A determination step of determining whether or not there is a directional deviation between the captured data acquired in the first acquisition step and the image data acquired from the first storage means.
When it is determined in the determination step that there is a deviation in the directionality, there is further a correction step for correcting the image data acquired from the first storage means so as to eliminate the deviation in the directionality. And
The guide display method according to any one of claims 1 to 4, wherein the second acquisition step acquires image data corrected in the correction step.
Claims 1 to 5 further include a display control step of displaying a screen on which the guide is superimposed on the state after snow accumulation on a display device based on the display data generated in the generation step. The guide display method according to any one of the above.
The guide display method according to claim 6, wherein the display device is configured as a mobile body.
Further, it has a third acquisition step of acquiring information on the inclination of the moving body.
The guide display method according to claim 7, wherein the generation step further generates an object for notifying that the inclination of the moving body is equal to or higher than a predetermined value.
The guide display method according to claim 7 or 8, wherein the moving body is a snowplow.
It further has a fourth acquisition step of acquiring information on the amount of snow removed by the snowplow.
The guide display method according to claim 9, wherein the generation step further generates an object for notifying the snow removal amount information.
The guide display method according to claim 10, further comprising a storage step of storing the snow removal amount information acquired in the fourth acquisition step in the second storage means.
A guide display method executed in a guide display system including a server and a vehicle capable of communicating with the server.
The server
A first receiving step of receiving shooting data representing a state after snowfall at a predetermined position from the vehicle, and
Based on the shooting data received in the first receiving step, image data corresponding to the state after snowfall represented by the shooting data is stored from a storage means that stores image data representing the state before snowfall at the predetermined position. Acquisition process to acquire and
Based on the shooting data received in the first receiving step and the image data acquired in the acquisition step, a guide for displaying the guide for traveling in the state after snowfall represented by the shooting data is displayed. The generation process to generate display data and
A first transmission step of transmitting the display data generated in the generation step to the vehicle, and
The vehicle
A second transmission step of transmitting the photographed data photographed by the photographing means to the server, and
A second receiving step of receiving the display data from the server, and
A display step of displaying a screen on which the guide is superimposed on the state after snow accumulation based on the display data received in the second reception step.
A guide display method characterized by having.