JP6997156B2 - Driving support system, driving support method and program - Google Patents

Description

The present invention relates to a driving support system, a driving support method and a program for supporting driving.

Patent Document 1 describes the areas of the left side portion and the right side portion of the steering wheel of the small electric vehicle in the captured image of the small electric vehicle traveling or stopped in the direction opposite to the direction of the own vehicle in front of the own vehicle. A configuration is described that determines whether or not the small electric vehicle has started moving to the right or left based on the change. Patent Document 1 describes that by recognizing the behavior of such a small electric vehicle, contact with the small electric vehicle or sudden approach can be prevented.

Japanese Unexamined Patent Publication No. 2019-091376

In the case where the own vehicle is parallel to another vehicle in front, it may be difficult for the passengers of the own vehicle to recognize the road condition in front of the other vehicle depending on the inter-vehicle distance and the surrounding environment. possible. In such a case, it is required to estimate the road condition in the traveling direction based on the behavior of other vehicles.

An object of the present invention is to provide a travel support system, a travel support method, and a program for estimating a road condition in a traveling direction based on the behavior of another vehicle.

The travel support system according to the present invention is a travel support system that supports the travel of a vehicle, and is an acquisition means for acquiring image data of a peripheral vehicle located in the traveling direction of the vehicle, which is imaged by the image pickup means, and the above-mentioned acquisition means. The traveling direction of the vehicle based on the first estimation means for estimating the behavior of the peripheral vehicle based on the imaging data acquired by the acquisition means and the behavior of the peripheral vehicle estimated by the first estimation means. The first estimation means includes a second estimation means for estimating the road condition of the vehicle and a notification means for notifying the road condition in the traveling direction of the vehicle estimated by the second estimation means . It is characterized in that the behavior of the peripheral vehicle is estimated based on the change of the image different from the vehicle image of the peripheral vehicle among the captured images .

The travel support method according to the present invention is a travel support method executed in a travel support system that supports the travel of the vehicle, and captures image data of peripheral vehicles located in the traveling direction of the vehicle, which are imaged by the image pickup means. Based on the acquisition process to be acquired, the first estimation process for estimating the behavior of the peripheral vehicle based on the imaging data acquired in the acquisition process, and the behavior of the peripheral vehicle estimated in the first estimation process. The first estimation process includes a second estimation step of estimating the road condition in the traveling direction of the vehicle, and a notification step of notifying the road condition of the traveling direction of the vehicle estimated in the second estimation process . The estimation step is characterized in that the behavior of the peripheral vehicle is estimated based on the change of the image different from the vehicle image of the peripheral vehicle among the captured images of the captured data .

The program according to the present invention is based on the acquisition means for acquiring the image pickup data of the peripheral vehicle located in the traveling direction of the vehicle, which is imaged by the image pickup means, and the image pickup data acquired by the acquisition means. The first estimation means for estimating the behavior of peripheral vehicles, the second estimation means for estimating the road condition in the traveling direction of the vehicle based on the behavior of the peripheral vehicles estimated by the first estimation means, and the second estimation. It functions as a notification means for notifying the road condition in the traveling direction of the vehicle estimated by the means, and the first estimation means changes an image of the captured image of the captured data different from the vehicle image of the peripheral vehicle. Based on, the behavior of the peripheral vehicle is estimated .

According to the present invention, it is possible to estimate the road condition in the traveling direction based on the behavior of another vehicle.

It is a figure which shows the structure of the driving support system. It is a figure which shows the structure of a vehicle. It is a figure which shows the block composition of a portable terminal. It is a figure which shows the block composition of a server. It is a figure for demonstrating the operation of this embodiment. It is a figure for demonstrating the change of a vehicle image. It is a figure for demonstrating the change of a vehicle image. It is a figure for demonstrating the change of a vehicle image. It is a flowchart which shows the process executed in a vehicle. It is a flowchart which shows the process which is executed in a portable terminal. It is a flowchart which shows the process executed in a server. It is a flowchart which shows the process of behavior estimation of S307. It is a flowchart which shows the process of behavior estimation of S307. It is a flowchart which shows the process of road condition estimation of S308. It is a figure which shows the display screen which is displayed by the notification control data. It is a flowchart which shows the process of behavior estimation of S307 in this embodiment. It is a figure which shows a state that a vehicle goes around the back side of an obstacle and turns left.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. It should be noted that the following embodiments do not limit the invention according to the claims, and not all combinations of features described in the embodiments are essential to the invention. Two or more of the plurality of features described in the embodiments may be arbitrarily combined. In addition, the same or similar configuration will be given the same reference number, and duplicated explanations will be omitted.

[First Embodiment]
FIG. 1 is a diagram showing a configuration of a traveling support system according to the present embodiment. The travel support system 100 includes a server 101 that provides a travel support service, and a base station 103 that is connected to the server 101 via a network 102. The network 102 is a wireless communication network, a wired communication network, or a network including both. The travel support system 100 includes vehicles 105 and 107. In the present embodiment, the vehicle 105 is a four-wheeled vehicle, and the vehicle 107 is a saddle-riding type two-wheeled vehicle. However, the vehicles 105 and 107 are not limited to the vehicle types shown in FIG. For example, both vehicles may be four-wheeled vehicles or saddle-riding two-wheeled vehicles. Alternatively, it may be a vehicle for special purposes. Further, the vehicle 107 is not limited to a two-wheeled vehicle, and may be, for example, a three-wheeled saddle-riding type vehicle. The portable terminal 106 is a device having a portable terminal function including an image pickup function such as a camera. In the present embodiment, the passenger (driver) of the vehicle 107 holds, for example, a smartphone as a portable terminal 106, and the smartphone is attached to the vehicle 107 by an attachment or the like. Further, the portable terminal function realized by the portable terminal 106 may be included in the vehicle 107. The vehicles 105, 107, and the portable terminal 106 can communicate with the server 101 via the wireless communication 104 with the base station 103 and the network 102. Further, the vehicle 107 and the portable terminal 106 can communicate with each other. For example, the vehicle 107 can communicate with the server 101 via the portable terminal 106. In this embodiment, the wireless communication 104 between the vehicles 105 and 107 and the base station 103 does not necessarily have to be performed.

In the present embodiment, it is assumed that the vehicle 107 is traveling in a column behind the vehicle 105. In such a case, the portable terminal 106 transmits the image pickup data of the vehicle 105 in front to the server 101 at any time. The server 101 estimates the road condition in front of the vehicle 105 based on the received image pickup data, and notifies the portable terminal 106 of the estimation result. Here, the road condition is information such as, for example, an uphill slope or severe unevenness of the road surface. Upon receiving the notification of the estimation result from the server 101, the portable terminal 106 transmits the notification control data to the vehicle 107. The operation of this embodiment will be described later.

FIG. 2 is a diagram showing a block configuration of the vehicle 107. As described above, in the present embodiment, the vehicle 107 is a saddle-riding type two-wheeled vehicle. The ECU (Electronic Control Unit) 210 comprehensively controls each part of the vehicle 107. The vehicle 107 has, for example, a configuration in which traveling control is performed by a throttle-by-wire method, and the ECU 210 receives an electric signal of the accelerator opening of the steering unit 213 to open and close the throttle valve to open and close the engine. The drive unit 211 including the drive unit 211 is controlled. The braking unit 212 includes a braking mechanism. The sensor group 214 includes various sensors such as a speed sensor, a throttle sensor, and an atmospheric pressure sensor, and the ECU 210 receives the detection signals thereof. The notification unit 215 includes a lamp, an LED, and the like, and for example, in the present embodiment, the notification unit 215 displays the direction of the road in front of the vehicle 105 (left direction, right direction, etc.). The image pickup unit 216 is, for example, a camera, and images the front of the vehicle 107. The image pickup unit 216, for example, takes an image of the vehicle 105 traveling in front of the vehicle 107. Further, the vehicle 107 may attach a portable terminal 106 having an image pickup function to the vehicle 107 by an attachment or the like to realize the image pickup function instead of the image pickup unit 216.

The control unit 200 is a block capable of executing wireless communication with the portable terminal 106 via the communication I / F 205. The communication I / F 205 may be configured to enable wireless communication with the base station 103. The control unit 200 is a computer system including a processor 201, a memory 202, a notification control unit 203, and an image pickup data transmission unit 204 that collectively control the operation in the control unit 200. The operation of the vehicle 107 in this embodiment is realized, for example, by the processor 201 executing a program on the memory 202. The notification control unit 203 requests the ECU 210 to control the notification unit 215 based on the notification control data transmitted from the portable terminal 106. For example, the notification control unit 203 converts the notification control data transmitted from the portable terminal 106 into a signal that can be processed by the ECU 210, and transmits the signal to the ECU 210. The image pickup data transmission unit 204 transmits the image pickup data captured by the image pickup unit 216 to the portable terminal 106 via the communication I / F 205. Further, the control unit 200 can be connected to a storage unit 217 such as USB. For example, the application provided by the server 101 is stored in the storage unit 217. Further, for example, each block of the control unit 200 is realized by the application, and registration to the server 101 and pairing with the control unit 300 of the portable terminal 106 are performed.

The vehicle 107 is not limited to the block configuration shown in FIG. 2, and includes, for example, functional blocks corresponding to the functions of the vehicle 107 as appropriate. Further, the image pickup data transmission unit 204 and the image pickup unit 216 may not be provided. For example, as described above, when the portable terminal 106 is used in place of the image pickup unit 216, the image pickup data transmission unit 204 and the image pickup unit 216 may not be provided.

FIG. 3 is a diagram showing a block configuration of the portable terminal 106. The control unit 300 comprehensively controls the portable terminal 106. The control unit 300 is a computer system including a processor 301, a memory 302, an image pickup data transmission unit 303, a GPS information transmission unit 304, a display control unit 305, and a notification control data transmission unit 306 that collectively control the operation in the control unit 300. Is. The operation of the portable terminal 106 in this embodiment is realized, for example, by the processor 301 executing a program on the memory 302. The image pickup data transmission unit 303 transmits, for example, the image pickup data captured by the image pickup unit 315, which is a camera, or the image pickup data captured by the image pickup unit 216 of the vehicle 107 to the outside via the communication I / F 313. The GPS information transmission unit 304 transmits the GPS information acquired by the GPS 314 to the outside via the communication I / F 313. The display control unit 305 controls the display on the display unit 311. The notification control data transmission unit 306 generates notification control data used in the vehicle 107 based on the notification control data transmitted from the server 101, and transmits the notification control data to the vehicle 107 via the communication I / F 313.

The storage unit 310 stores programs and data for operating the portable terminal 106, and programs and data necessary for the operation of the present embodiment. For example, the application provided by the server 101 is stored in the storage unit 310. For example, the user of the portable terminal 106 starts the application, registers it with the server 101, and performs pairing with the control unit 200 of the vehicle 107 on the setting screen. The display unit 311 is, for example, a touch panel, and displays various user interface screens for the user of the portable terminal 106. In this embodiment, the user of the portable terminal 106 corresponds to the passenger of the vehicle 107. The operation unit 312 can accept operations from the user, and includes hard keys and soft keys displayed on the touch panel. The GPS (Global Positioning System) 314 is a position detection mechanism for detecting the current position of the portable terminal 106. The communication I / F 313 enables wireless communication 104 with the outside. The communication I / F 313 may be configured to support a plurality of wireless communications having different communication distances. The portable terminal 106 is not limited to the block configuration shown in FIG. 3, and includes, for example, functional blocks corresponding to the functions of the portable terminal 106 as appropriate.

In the present embodiment, the vehicle 107 is described as a saddle-riding type two-wheeled vehicle, but other types of vehicles, for example, a four-wheeled vehicle may be used. In that case, the control unit 200 of FIG. 2 and the control unit 300 of FIG. 3 may be integrally configured in the four-wheeled vehicle.

FIG. 4 is a diagram showing a block configuration of the server 101. The control unit 400 controls the server 101 in an integrated manner. The control unit 400 includes a processor 401 such as a CPU or GPU that comprehensively controls the operation in the control unit 400, a memory 402, an imaging data receiving unit 403, an image processing unit 404, an image recognition unit 405, an image analysis unit 406, and behavior. It is a computer system including an estimation unit 407, a road condition estimation unit 408, and a notification control data transmission unit 409. The operation of the server 101 in this embodiment is realized, for example, by the processor 401 executing a program on the memory 402. The image pickup data receiving unit 403 receives the image pickup data transmitted from the portable terminal 106. In the present embodiment, the vehicle 105 is imaged in this image pickup data. The image processing unit 404 performs preprocessing for executing image recognition on the imaged data, and as the preprocessing, for example, correction of shaking of the imaged data is performed. The image recognition unit 405 recognizes an object on the image pickup data received by the image pickup data reception unit 403 by using a technique such as classification. The image analysis unit 406 analyzes the object recognized by the image recognition unit 405. For example, the image analysis unit 406 analyzes changes in relative positions between objects. The behavior estimation unit 407 estimates the behavior of the vehicle 105 based on the analysis result by the image analysis unit 406. The road condition estimation unit 408 estimates the road condition on which the vehicle 107 travels based on the estimation result by the behavior estimation unit 407. The notification control data transmission unit 409 generates notification control data, which will be described later, based on the estimation result by the road condition estimation unit 408, and transmits it to the portable terminal 106.

The storage unit 410 stores programs and data for operating the server 101, and programs and data necessary for the operation of the present embodiment. The display unit 411 is, for example, a display, and displays various user interface screens to the user of the server 101. The operation unit 412 can receive an operation from the user, and is, for example, a keyboard or a pointing device. The communication I / F 413 enables communication between the server 101 and the network 102. The server 101 is not limited to the block configuration shown in FIG. 4, and includes, for example, functional blocks corresponding to the functions of the server 101 as appropriate. Although the server 101 is shown as one device in FIG. 4, it may be composed of a plurality of devices. That is, the server 101 in the present embodiment includes a configuration in which one device provides a server function, and a configuration in which a plurality of devices cooperate to provide a server function.

Hereinafter, the operation of the present embodiment will be described with reference to FIGS. 5 (a) and 5 (b). As shown in FIG. 5A, the vehicle 107 is traveling behind the vehicle 105 on the road 500. Here, the vehicle 107 is located at position 501 and the vehicle 105 is located at position 502. Further, the passenger of the vehicle 107 possesses the portable terminal 106. Alternatively, the portable terminal 106 is attached to the vehicle 107. The road 500 is a straight road, and the vehicle 105 turns to the left as it moves forward as in positions 503 and 504. Here, if the visibility from the passengers of the vehicle 501 is not good due to trees on both sides of the road 500, there may be a case where the passengers of the vehicle 501 do not know that the road 500 turns to the left in front. In such a case, the image pickup unit 216 of the vehicle 107 or the image pickup unit 315 of the portable terminal 106 images the vehicle 105 in front at predetermined time intervals, and transmits the image pickup data to the server 101.

FIG. 5B shows changes in the vehicle image of the captured data, and the lower vehicle image 510 shows the rear part of the vehicle 105 in FIG. 5A. When the vehicle 105 advances to the position 503, the captured data thereof changes as shown in the vehicle image 511 in the middle of FIG. 5 (b). Then, when the vehicle 105 further advances to the position 504, the captured image data changes as shown in the vehicle image 512 in the upper part of FIG. 5B. The server 101 estimates, for example, a left turn as the behavior of the vehicle 105 based on the change in the vehicle image as shown in FIG. 5B, and estimates that the road 500 is turning to the left in front.

Here, changes in the vehicle image will be described. In the present embodiment, the server 101 estimates the behavior of the vehicle 105 based on the change in the outer shape of the vehicle image of the vehicle 105. FIG. 6 is a diagram for explaining the concept of change in the vehicle image.

The image pickup data 600 of FIG. 6 includes a vehicle image 601. After the acquisition of the imaging data 600, it is assumed that the vehicle image 601 changes like the vehicle image 602 when the imaging data 610 is acquired. This corresponds to the change from the vehicle image 510 in FIG. 5B to the vehicle image 511 or 512. In this way, when the vehicle makes a left turn or a right turn, the outer shape of the vehicle image changes so as to be stretched horizontally with respect to the reference line 620. If the server 101 recognizes that the stretched direction is on the left side, the behavior of the vehicle 105 is presumed to be a left turn, and if it is recognized as a right side, the behavior of the vehicle 105 is a right turn. I presume.

Further, it is assumed that the vehicle image 601 is changed like the vehicle image 603 when the image pickup data 611 is acquired after the acquisition of the image pickup data 600. This corresponds to the case where the road is uphill.

7 (a) and 7 (b) are diagrams for explaining changes in the vehicle image when the road is uphill. FIG. 7A shows that the vehicle 107 is located at position 701 on the road 700 and the vehicle 105 is located at position 702. As shown in FIG. 7A, the road 700 is an uphill from the middle. FIG. 7B shows changes in the vehicle image of the captured data, and the lower vehicle image 710 shows the rear part of the vehicle 105 in FIG. 7A. When the vehicle 105 advances to the position 703, the vehicle image 710 changes as shown in the vehicle image 711 in the upper part of FIG. 7B. That is, as the vehicle 105 travels uphill, the outer shape of the vehicle image changes from the vehicle image 710 to the vehicle image 711, that is, the change from the vehicle image 601 to the vehicle image 603 in FIG. , It changes so as to be stretched in the direction perpendicular to the reference line 621. The server 101 estimates that the behavior of the vehicle 105 is uphill, based on its stretched direction being the vertical direction.

Further, it is assumed that the vehicle image 601 of FIG. 6 changes as shown in the vehicle image 604 when the image pickup data 612 is acquired after the acquisition of the image pickup data 600. This corresponds to the case where the road is downhill. 8 (a) and 8 (b) are diagrams for explaining changes in the vehicle image when the road is downhill. FIG. 8A shows that the vehicle 107 is located at position 721 on the road 720 and the vehicle 105 is located at position 722. As shown in FIG. 8A, the road 720 is downhill from the middle. FIG. 8B shows changes in the vehicle image of the captured data, and the lower vehicle image 730 shows the rear part of the vehicle 105 in FIG. 8A. When the vehicle 105 advances to the position 723, the vehicle image 730 changes as shown in the vehicle image 731 in the upper part of FIG. 8B. That is, when the road is a downhill, the front part of the vehicle sinks downward, so that the outer shape of the vehicle image changes so as to shrink in the vertical direction when viewed from the following vehicle. That is, as the vehicle travels downhill, the outer shape of the vehicle image changes so as to shrink in the direction perpendicular to the reference line 622, as in the change from the vehicle image 601 to the vehicle image 604 in FIG. .. The server 101 estimates that the behavior of the vehicle 105 is downhill, based on the fact that its contraction direction is vertical.

As described above, the server 101 can estimate the behavior of the vehicle 105 by changing the outer shape of the vehicle image of the vehicle 105. Further, the server 101 can estimate the behavior of the vehicle 105 based on the analysis result of the object of the vehicle image of the vehicle 105.

The server 101 recognizes, for example, objects 513, 516, 516, 522 in the vehicle image 510 by the image recognition unit 405. In the case of FIG. 5B, the object 513 is the lower region including the tire, the object 516 is the license plate, the object 519 is the left mirror, and the object 522 is the right mirror. When the vehicle 105 makes a left turn, each object changes as shown in FIG. 5 (b). For example, the object 513 becomes larger in the vertical and horizontal directions because the upper part of the tire and the front wheels are more visible. Further, the object 516 changes from a rectangle to a trapezoidal shape having a longer side on the turning direction side. Further, since the object 522 is hidden behind the vehicle body of the vehicle 105, it disappears on the image pickup data. The server 101 estimates the behavior of the vehicle 105 (turning left or turning right in this example) based on changes in each such object.

In the case of FIG. 7B, the object 712 is an antenna and the object 713 is the hood or roof of the vehicle 105. When the vehicle 105 is in the climbing state, the front part of the vehicle body is lifted when viewed from the following vehicle, so that the object 713 appears on the imaging data. Further, when the vehicle 105 goes downhill, as shown in FIG. 8B, the front part of the vehicle body sinks, so that the object 732, which is an antenna, disappears on the imaging data. The server 101 estimates the behavior of the vehicle 105 (uphill or downhill in this example) based on the changes in each of these objects.

Further, the server 101 may estimate the behavior of the vehicle 105 by changing the relative position between the objects. For example, in the lower part of FIG. 5B, the object 516 is located almost in the center with respect to the horizontal length of the object 513, whereas in the middle part to the upper part of FIG. 5B, the object 516 is located. Move toward the right end of object 513 like objects 517 and 518. In this way, the server 101 may estimate the behavior of the vehicle 105 based on the relative position between the objects. Alternatively, the above methods may be combined to improve the accuracy of estimating the behavior of the vehicle 105.

FIG. 9 is a flowchart showing the processing executed in the vehicle 107. The process of FIG. 9 is realized, for example, by the processor 201 executing a program on the memory 202.

In S101, the processor 201 transmits the image pickup data captured by the image pickup unit 216 to the portable terminal 106 via the communication I / F 205 by the image pickup data transmission unit 204. When the portable terminal 106 attached to the vehicle 105 is used instead of the image pickup unit 216, the processing of S101 may not be performed.

In S102, the processor 201 determines whether or not the notification control data has been received from the portable terminal 106 by the notification control unit 203. Here, if it is determined that the notification control data has not been received, the process from S101 is repeated. On the other hand, when it is determined that the notification control data has been received, in S103, the processor 201 requests the ECU 210 to notify the notification unit 215 based on the notification control data by the notification control unit 203. For example, the data representing the "curve to the left" transmitted from the portable terminal 106 is converted into a signal that can be processed by the ECU 210 and transmitted to the ECU 210. The content notified here is the estimation result of the road condition in front of the vehicle 105. The content of the notification will be described later. After S103, the process from S101 is repeated.

FIG. 10 is a flowchart showing a process executed by the portable terminal 106. The process of FIG. 10 is realized, for example, by the processor 301 executing a program on the memory 302.

In S201, the processor 301 transmits the image pickup data to the server 101 by the image pickup data transmission unit 303. Here, the image pickup data transmitted is the image pickup data captured by the image pickup unit 216 of the vehicle 107 or the image pickup data captured by the image pickup unit 315 of the portable terminal 106.

In S202, the processor 301 determines whether or not the notification control data has been received from the server 101 by the notification control data transmission unit 306. Here, if it is determined that the notification control data has not been received, the process from S201 is repeated. On the other hand, if it is determined that the notification control data has been received, the process proceeds to S203.

Here, the notification control data will be described. As described above, the notification control data is an estimation result of the road condition in front of the vehicle 105, and is received as display data on, for example, a panel or the like. FIG. 15 is a diagram showing an example of the display screen 1400 displayed by the notification control data. The portable terminal 106 can display the display screen 1400 on the display unit 311 based on the notification control data received from the server 101.

On the display screen 1400, the arrow marks 1402 and 1403 are displayed, and the passenger of the vehicle 107 can be made to recognize in which direction the road in front of the vehicle 105 turns on the left side or the right side. In FIG. 15, the arrow mark 1402 is lit, the arrow mark 1403 is not lit, and it is shown that the road in front of the vehicle 105 turns to the left.

Further, in the display area 1401, it is graphically displayed that the road turns to the left. If the server 101 determines that the road in front of the vehicle 105 is uphill to the left and the road surface is extremely uneven, as in the display area 1404, "uphill to the left". A message such as "Please be careful about unevenness." Is displayed. At that time, the warning mark 1405 is displayed in the display area 1401. In this way, the combination of the plurality of display modes makes it possible for the passenger of the vehicle 107 to more easily recognize the road condition in front of the vehicle 105.

In S203, the processor 301 generates notification control data that can be notified by the notification unit 215 of the vehicle 107 from the notification control data received from the server 101 by the notification control data transmission unit 306, and transmits the notification control data to the vehicle 107. For example, if the notification unit 215 of the vehicle 107 is configured to notify either the left direction or the right direction by an LED, the processor 301 uses an arrow from the notification control data received from the server 101 by the notification control data transmission unit 306. The lighting signal corresponding to the marks 1402 and 1403 is transmitted to the vehicle 107 as notification control data.

In S204, the processor 301 causes the display unit 311 to display the notification control data received from the server 101 by the display control unit 305. In S204, for example, the display screen 1400 is displayed. After S204, the process from S201 is repeated.

In the above, it has been described that the processor 301 in S203 generates the notification control data used for the notification control in the vehicle 107 from the notification control data received from the server 101. However, the processor 301 may transfer the notification control data received from the server 101 to the vehicle 107. In that case, the notification control unit 203 of the vehicle 107 generates notification control data that can be notified by the notification unit 215 from the notification control data received from the portable terminal 106.

Further, although it has been described above that the display is performed in S204, the processing from S201 may be repeated after S203 without performing the processing in S204. Further, whether or not to display in S204 may be set by the passenger of the vehicle 107 on the user interface screen displayed on the display unit 311 of the portable terminal 106. With such a configuration, for example, when the portable terminal 106 is attached to the vehicle 107 by an attachment, the display is performed by S204, and the portable terminal 106 is stored in the bag or the like of the passenger of the vehicle 107. In this case, it is possible to control the display such that the display is not performed in S204.

FIG. 11 is a flowchart showing a process executed by the server 101. The process of FIG. 11 is realized, for example, by the processor 401 executing a program on the memory 402.

In S301, the processor 401 receives the image pickup data from the portable terminal 106 by the image pickup data receiving unit 403. Then, in S302, the processor 401 executes preprocessing on the image pickup data received in S301 by the image processing unit 404. Here, as a preprocessing, the fluctuation of the imaging data is corrected. The processor 401 stores the correction amount at that time in a storage area such as the memory 402.

In S303, the processor 401 performs image recognition by the image recognition unit 405, and in S304, determines whether or not the vehicle 105 is in front. If it is determined in S304 that there is no vehicle 105 in front, the process from S301 is repeated. On the other hand, if it is determined that the vehicle 105 is in front, the process proceeds to S305.

In S305, the processor 401 determines whether or not there is a change in the vehicle image of the captured data. For example, when the vehicle 105 and the vehicle 107 are parallel and straight, no change appears in the vehicle image, so it is determined in S305 that there is no change in the vehicle image, and the process from S301 is repeated. On the other hand, if it is determined that there is a change in the vehicle image, the process proceeds to S306.

In the determination of S305, the following criteria may be set. For example, even when parallel parking is performed, the vehicle image is reduced when the distance between the vehicles 105 and 107 is long, and the vehicle image is enlarged when the distance between the vehicles is short. As a result, the vehicle image is displayed in S305. It will be determined that there is a change. Therefore, even if the vehicle image changes, if the change in the aspect ratio is within a predetermined range, it is determined that there is no change in the vehicle image. With such a configuration, it is possible to prevent erroneous determination due to fluctuations in the inter-vehicle distance.

In S306, the processor 401 analyzes changes in the vehicle image by the image analysis unit 406. For example, the processor 401 analyzes changes in the outer shape of the vehicle image as described in FIG. Further, for example, the processor 401 analyzes the deformation and movement of the object as described with reference to FIGS. 5 (b) and 7 (b).

In S307, the processor 401 estimates the behavior of the vehicle 105 by the behavior estimation unit 407. The processing of S307 will be described later. In S308, the processor 401 estimates the road condition in front of the vehicle 105 by the road condition estimation unit 408. The processing of S308 will be described later.

In S309, the processor 401 generates notification control data based on the estimation result in S308. The notification control data generated in S309 is, for example, display data for displaying the display screen 1400 of FIG. In S310, the processor 401 transmits the notification control data generated in S309 to the portable terminal 106 by the notification control data transmission unit 409. After S310, the process of FIG. 11 is terminated.

FIG. 12 is a flowchart showing the process of estimating the behavior of S307. In S401, the processor 401 detects the amount of change in the horizontal direction from the analysis results in S306, and in S402, determines whether or not the amount of change in the horizontal direction satisfies a condition that can be presumed to be a left turn. For example, regarding the outer shape of the vehicle image of the vehicle 105, as shown in the vehicle image 602 of FIG. 6, the degree of deformation to the left is the degree of deformation to the right with respect to the reference line 620 corresponding to the side wall of the vehicle image 601. If it is larger than, it is determined that the condition that can be presumed to be a left turn is satisfied. Further, for example, when the relative positions of the objects 516 and 513 of the vehicle image 510 are moved toward the right end of the object 513 as shown in the corresponding objects 518 and 515 in the upper part of FIG. 5B. , It is determined that the condition that can be presumed to be a left turn is satisfied.

If it is determined in S402 that the condition that can be estimated to be a left turn is satisfied, the process proceeds to S403, and if it is determined that the condition that can be estimated to be a left turn is not satisfied, the process proceeds to S421 in FIG.

In S403, the processor 401 detects the amount of change in the vertical direction from the analysis results in S306, and in S404, determines whether or not the amount of change in the vertical direction satisfies the condition that can be presumed to be climbing. For example, when the outer shape of the vehicle image of the vehicle 105 includes the reference line 621 corresponding to the bottom of the vehicle image 601 and the degree of upward deformation is larger than the threshold value, as shown in the vehicle image 603 of FIG. Determines that the condition that can be presumed to be climbing is satisfied. Further, for example, as shown in the upper part of FIG. 7B, when the object 713 appears, it is determined that the condition that can be presumed to be climbing is satisfied.

When it is determined in S404 that the condition that can be estimated as climbing is satisfied, in S406, the processor 401 estimates that the vehicle 105 is in a state of climbing to the left, and stores the estimation result in a storage area such as a memory 402. do. After that, the process of FIG. 12 is terminated. On the other hand, if it is determined in S404 that the condition that can be presumed to be climbing is not satisfied, the process proceeds to S405.

In S405, the processor 401 determines whether or not the amount of change in the vertical direction satisfies a condition that can be presumed to be a downhill. For example, when the outer shape of the vehicle image of the vehicle 105 includes the reference line 622 corresponding to the bottom of the vehicle image 601 and the degree of downward deformation is larger than the threshold value, as shown in the vehicle image 604 of FIG. Determines that the condition that can be presumed to be a downhill condition is satisfied. Further, for example, when the object 712 of FIG. 7A disappears, it is determined that the condition that can be presumed to be a downhill condition is satisfied.

When it is determined in S405 that the condition that can be estimated to be downhill is satisfied, in S407, the processor 401 estimates that the vehicle 105 is in a downhill state to the left, and the estimation result is stored in a storage area such as a memory 402. Store in. After that, the process of FIG. 12 is terminated. On the other hand, when it is determined in S405 that the condition that can be estimated as a downhill is not satisfied, in S408, the processor 401 estimates that the vehicle 105 is in a left turn state, and the estimation result is stored in a storage area such as a memory 402. Store in. After that, the process of FIG. 12 is terminated.

When it is determined in S402 that the condition that can be estimated to be a left turn is not satisfied, in S421 of FIG. 13, the processor 401 sets the condition that the amount of change in the horizontal direction in the analysis result in S306 can be estimated to be a right turn. Determine if it meets or not. For example, regarding the outer shape of the vehicle image of the vehicle 105, when the degree of deformation to the right is larger than the degree of deformation to the left with respect to the reference line 620 corresponding to the side wall of the vehicle image 601 of FIG. It is determined that the condition that can be presumed to be a right turn is satisfied. Further, for example, with respect to the relative positions of the objects 516 and 513 of the vehicle image 510, when the object 516 approaches the left end of the object 513, it is determined that the condition that can be presumed to be a right turn is satisfied.

If it is determined in S421 that the condition that can be estimated to be a right turn is satisfied, the process proceeds to S422, and if it is determined that the condition that can be estimated to be a right turn is not satisfied, the process proceeds to S428.

In S422, the processor 401 detects the amount of change in the vertical direction from the analysis results in S306, and in S423, as in S405, determines whether or not the amount of change in the vertical direction satisfies the condition that can be estimated to be climbing. do.

When it is determined in S423 that the condition that can be presumed to be climbing is satisfied, in S425, the processor 401 estimates that the vehicle 105 is climbing to the right, and stores the estimation result in a storage area such as a memory 402. do. After that, the process of FIG. 13 is terminated. On the other hand, if it is determined in S423 that the condition that can be estimated as climbing is not satisfied, in S424, whether or not the amount of change in the vertical direction satisfies the condition that can be estimated as descending, as in S406. To judge.

When it is determined in S424 that the condition for descending a slope is satisfied, in S426, the processor 401 estimates that the vehicle 105 is in a state of descending to the right, and stores the estimation result in a storage area such as a memory 402. .. After that, the process of FIG. 13 is terminated. On the other hand, when it is determined in S424 that the condition that can be estimated as a downhill is not satisfied, in S427, the processor 401 estimates that the vehicle 105 is in a right-turning state, and stores the estimation result in the memory 402 or the like. Store in the area. After that, the process of FIG. 13 is terminated.

When it is determined in S421 that the condition that can be presumed to be a right turn is not satisfied, in S428, the processor 401 detects the amount of change in the vertical direction among the analysis results in S306, and in S429, similarly to S405, is vertical. It is determined whether or not the amount of change in direction satisfies the condition that can be presumed to be climbing.

When it is determined in S429 that the condition for climbing a slope is satisfied, in S430, the processor 401 estimates that the vehicle 105 is in a climbing state, and stores the estimation result in a storage area such as a memory 402. After that, the process of FIG. 13 is terminated. On the other hand, when it is determined in S429 that the condition that can be estimated to be uphill is not satisfied, in S431, the processor 401 estimates that the vehicle 105 is in a downhill state, and the estimation result is stored in the storage area such as the memory 402. Store. After that, the process of FIG. 13 is terminated.

FIG. 14 is a flowchart showing the process of estimating the road condition of S308. In S501, the processor 401 acquires the result of the behavior estimation of S307 from the storage area such as the memory 402.

In S502, the processor 401 estimates the direction of the road ahead of the vehicle 105 based on the result of the behavior estimation. For example, when it is estimated in S406 that the vehicle 105 is climbing to the left, the direction of the road in front of the vehicle 105 is estimated to be uphill to the left. Further, when it is estimated in S407 that the vehicle 105 is in a state of descending to the left, the direction of the road in front of the vehicle 105 is estimated to be a downward slope to the left. Further, when it is estimated in S408 that the vehicle 105 is in a state of turning left, the direction of the road in front of the vehicle 105 is estimated to be the left direction (no uphill / downhill). Further, when it is estimated in S425 that the vehicle 105 is in a state of climbing to the right, the direction of the road in front of the vehicle 105 is estimated to be an uphill to the right. Further, when it is estimated in S426 that the vehicle 105 is in a state of descending to the right, the direction of the road in front of the vehicle 105 is estimated to be descending to the right. Further, when it is estimated in S427 that the vehicle 105 is in a right-turning state, the direction of the road in front of the vehicle 105 is estimated to be the right direction (no uphill / downhill). Further, when it is estimated in S430 that the vehicle 105 is in an uphill state, the direction of the road in front of the vehicle 105 is estimated to be an uphill (straight ahead). Further, when it is estimated in S431 that the vehicle 105 is in a downhill state, the direction of the road in front of the vehicle 105 is estimated to be downhill (straight ahead). The processor 401 stores the estimation result in S502 in a storage area such as a memory 402.

In S503, the processor 401 acquires the correction amount of the shaking of the image pickup data stored in the storage area such as the memory 402 in S302, estimates the shape of the road based on the correction amount, and uses the estimation result as the memory 402 or the like. Store in the storage area of. For example, the processor 401 may classify the correction amount into a plurality of stages and classify the uneven state of the road surface into levels. In that case, in the message about the shape of the road in the display area 1404 of FIG. 15, a message corresponding to each level-divided uneven state of the road surface may be displayed. At that time, as for the warning mark 1405, the mark corresponding to each level-divided uneven state of the road surface may be displayed.

As described above, according to the present embodiment, it is possible to estimate the state of the road in front of the vehicle based on the captured image of the vehicle traveling ahead. With such a configuration, for example, even in an area where the navigation function does not function sufficiently, the road condition can be notified, and the passenger of the vehicle 107 can have a margin. Further, for example, in an area where the road is not sufficiently paved, such as a gravel road or a natural road, when the vehicle 105 and the vehicle 107 are traveling in a straight line on a straight road, a large crack occurs on the road in front of the vehicle 105. Suppose you are. In such a case, the vehicle 105 is assumed to travel while turning, for example, to the left in order to avoid cracks. In that case, according to the present embodiment, although the actual road is straight, in S103 and S204, it is notified that the direction of the road is to the left based on the behavior of the vehicle 105. Therefore, by following the notification, the vehicle 107 can travel on the route where the vehicle 105 can actually travel, and can avoid cracks.

In the above, it has been described that only one of the left and right is estimated as the turning direction of the vehicle 105. For example, the roll angle is estimated from the inclination of the objects 513, 514, and 515, and the roll angle and the vehicle image of the vehicle 105 are estimated. The degree of R of the road may be estimated and included in the notification content based on the horizontal change in the outer shape. Further, in the above description, only the uphill or downhill of the vehicle 105 is estimated, but for example, the pitch angle is estimated based on the appearance of the object 713 or the disappearance of the object 712, and the pitch angle and the pitch angle are used. The degree of inclination of the road may be estimated and included in the notification content based on the vertical change of the outer shape of the vehicle 105 vehicle image.

In the present embodiment, the configuration in which the notification control unit 203 and the notification unit 215 are provided in the vehicle 107 has been described, but other configurations may be used. For example, when the helmet of the passenger of the vehicle 107 has a communication module and a head-up display, the helmet may be provided with a configuration corresponding to the control unit 200. In that case, the notification control data transmission unit 306 of the portable terminal 106 transmits the notification control data to the notification control unit of the helmet, and the information corresponding to the display screen 1400 of FIG. 15 is displayed on the head-up display as the notification unit. It may be displayed. Further, when the camera function is mounted on the helmet, the camera function of the helmet may be used instead of the image pickup unit 216 and the image pickup unit 315.

[Second Embodiment]
In the first embodiment, it has been described that the behavior of the vehicle is estimated based on the change of the outer shape of the vehicle image of the vehicle 105 and the change of the object of the vehicle image of the vehicle 105. However, there may be a case where the outer shape or the object of the vehicle image of the vehicle 105 cannot be recognized because the obstacle overlaps in front of the vehicle 105. Hereinafter, the second embodiment will be described with respect to the differences from the first embodiment.

FIG. 17 is a diagram showing a vehicle 1601 turning left around the back side of the tree 1602. The vehicle 1601 corresponds to the vehicle 105 in FIG. In the middle row of FIG. 17, the objects of the left mirror and the tire are hidden by the tree 1602, and in the middle row and the upper row of FIG. 17, the outer shape of the vehicle image of the vehicle 1602 is missing. That is, in the case shown in FIG. 17, changes in the vehicle image cannot be properly recognized. Further, in the case of the saddle-riding type two-wheeled vehicle 107, the passenger may have a narrow field of view due to the helmet, and it may be difficult for the passenger to see the approaching tree 1602.

In the present embodiment, if there is an object whose size is changing other than the vehicle image of the vehicle 105, the object is recognized as an obstacle, and the behavior of the vehicle 105 is based on the overlap with the vehicle image of the vehicle 105. To estimate. As a result, even if the vehicle 105 goes around to the back of the obstacle, the passenger of the vehicle 107 can be notified of the traveling direction of the road.

FIG. 16 is a flowchart showing the process of estimating the behavior of S307 in the present embodiment. In S601, the processor 401 determines whether or not an object other than the vehicle image of the vehicle 105 is recognized. The object other than the vehicle image here is an object whose size changes greatly on the captured data, and corresponds to, for example, a tree on the side of a road. If it is determined that the object other than the vehicle image is not recognized, the process proceeds to S401 in FIG. On the other hand, if it is determined that the object other than the vehicle image is recognized, the process proceeds to S602.

In S602, the processor 401 determines whether or not the overlap between the object recognized in S601 (hereinafter referred to as an obstacle object) and the vehicle image of the vehicle 105 is detected. If it is determined that no overlap has been detected, the process proceeds to S401 in FIG. However, the determination process of S602 is executed in parallel even while the process of FIGS. 12 and 13 is being performed in the process of proceeding to S401.

When it is determined in S602 that the overlap is detected, in S603, the processor 401 estimates the behavior of the vehicle 105. For example, when the vehicle image is superimposed on the obstacle object, the vehicle image can be recognized, so that the behavior of the vehicle 105 is estimated by the processing of FIGS. 12 and 13. On the other hand, when the obstacle object overlaps the vehicle image, it is estimated that the vehicle 105 has turned in the direction of the overlapping obstacle object, and the estimation result is stored in a storage area such as a memory 402. .. After that, the process of FIG. 16 is terminated.

When the notification control data is generated in S309 via S603, a message or an image notifying the existence of the obstacle object may be displayed. With such a configuration, it is possible to notify the passenger of the saddle-riding type two-wheeled vehicle 107 wearing a helmet of the presence of an approaching obstacle. Further, if the uphill state or the downhill state is estimated in S430 or S431 before S603, the notification may be made in combination with the estimation result. In the above S603, the uphill and downhill states of the vehicle 105 are not estimated, but by combining with the previous estimation results, it is possible to estimate the uphill and downhill states.

In each of the above embodiments, the system configuration of the vehicle 107, the portable terminal 106, and the server 101 has been described, but other configurations may be used. For example, it may be integrally configured in the four-wheeled vehicle 107 to improve real-time performance.

<Summary of embodiments>
The travel support system of the above embodiment is a travel support system that supports the travel of the vehicle, and acquires image data of peripheral vehicles located in the traveling direction of the vehicle, which are imaged by the image pickup means (216, 315). The acquisition means (S301), the first estimation means (S307) for estimating the behavior of the peripheral vehicle based on the imaging data acquired by the acquisition means, and the peripheral vehicle estimated by the first estimation means. A second estimation means (S308) that estimates the road condition in the traveling direction of the vehicle based on the behavior of the vehicle, and a notification means (S310) that notifies the road condition in the traveling direction of the vehicle estimated by the second estimation means. ) And.

With such a configuration, it is possible to estimate the road condition in the traveling direction based on the behavior of peripheral vehicles located in the traveling direction of the vehicle.

Further, the behavior of the peripheral vehicle is characterized by including right turn and left turn (S402, S421).

With such a configuration, the direction of the road can be estimated based on the turning direction of the behavior of the vehicle.

Further, the behavior of the peripheral vehicle is characterized by including uphill and downhill (S404, S405, S423, S424, S429).

With such a configuration, the direction of the road can be estimated based on the uphill and downhill of the vehicle.

Further, the first estimation means has the behavior of the peripheral vehicle based on the change of the object of the vehicle image of the peripheral vehicle in the captured image of the captured data (FIGS. 5 (b) and 7 (b)). Is characterized by estimating. Further, the first estimation means is characterized in that the behavior of the peripheral vehicle is estimated based on the change in the aspect ratio of the object of the vehicle image of the peripheral vehicle in the captured image of the captured data. Further, the object is characterized by including at least one of a tire, a mirror, and a license plate of the peripheral vehicle. Further, the first estimation means is characterized in that, among the captured images of the captured data, the behavior of the peripheral vehicle is estimated based on the change in the outer shape of the vehicle image of the peripheral vehicle (FIG. 6). ..

With such a configuration, the direction of the road can be estimated based on, for example, changes in the horizontal direction and the vertical direction of the vehicle image. Further, the direction of the road can be estimated even for a simple image change such as a change in aspect ratio. Further, for example, the direction of the road can be estimated based on the change of the tire, the mirror, and the license plate on the vehicle image.

Further, the first estimation means is characterized in that the behavior of the peripheral vehicle is estimated based on the change of the image (1602) different from the vehicle image of the peripheral vehicle among the captured images of the captured data.

With such a configuration, for example, the turning direction of the vehicle can be estimated based on the overlap between the image of the tree on the side of the road and the image of the vehicle.

Further, the image pickup means is characterized in that it is configured as a terminal that moves with the vehicle.

With such a configuration, for example, the behavior of a peripheral vehicle can be estimated based on the image pickup data captured by the camera of the portable terminal held by the passenger of the vehicle.

Further, the vehicle is characterized by being a saddle-riding type vehicle. With such a configuration, the operation of the travel support system can be applied even when the vehicle is a saddle-riding vehicle.

The invention is not limited to the above embodiment, and various modifications and changes can be made within the scope of the gist of the invention.

100 Driving support system: 101 Server: 105, 107 Vehicle: 106 Portable terminal: 200, 300, 400 Control unit: 201, 301, 401 Processor: 202, 302, 402 Memory

Claims (12)

It is a driving support system that supports the driving of vehicles.
An acquisition means for acquiring image pickup data of a peripheral vehicle located in the traveling direction of the vehicle, which is captured by the image pickup means, and an acquisition means.
A first estimation means for estimating the behavior of the surrounding vehicle based on the imaging data acquired by the acquisition means, and a first estimation means.
A second estimation means for estimating the road condition in the traveling direction of the vehicle based on the behavior of the peripheral vehicle estimated by the first estimation means, and a second estimation means.
A notification means for notifying the road condition in the traveling direction of the vehicle estimated by the second estimation means is provided.
The first estimation means estimates the behavior of the peripheral vehicle based on the change of the image different from the vehicle image of the peripheral vehicle in the captured image of the captured data.
A driving support system characterized by this. The traveling support system according to claim 1, wherein the change in the image different from the vehicle image of the peripheral vehicle is an overlap of the vehicle image and the image different from the vehicle image. The traveling support system according to claim 1 or 2 , wherein the behavior of the peripheral vehicle includes a right turn and a left turn. The traveling support system according to any one of claims 1 to 3, wherein the behavior of the peripheral vehicle includes uphill and downhill. The first estimation means is any of claims 1 to 4 , wherein the first estimation means estimates the behavior of the peripheral vehicle based on the change of the object of the vehicle image of the peripheral vehicle in the captured image of the captured data. The driving support system described in item 1. The first estimation means is characterized in that, among the captured images of the captured data, the behavior of the peripheral vehicle is estimated based on the change in the aspect ratio of the object of the vehicle image of the peripheral vehicle. The described driving support system. The travel support system according to claim 5 or 6 , wherein the object includes at least one of a tire, a mirror, and a license plate of the peripheral vehicle. The first estimation means is characterized in that, among the captured images of the captured data, the behavior of the peripheral vehicle is estimated based on the change in the outer shape of the vehicle image of the peripheral vehicle. The driving support system according to any one of the items. The traveling support system according to any one of claims 1 to 8, wherein the image pickup means is configured in a terminal that moves with the vehicle. The traveling support system according to any one of claims 1 to 9, wherein the vehicle is a saddle-riding type vehicle. It is a driving support method executed in a driving support system that supports the driving of a vehicle.
The acquisition process of acquiring the image pickup data of the peripheral vehicle located in the traveling direction of the vehicle, which is imaged by the image pickup means, and the acquisition process.
A first estimation step of estimating the behavior of the peripheral vehicle based on the imaging data acquired in the acquisition step, and a first estimation step.
A second estimation step of estimating the road condition in the traveling direction of the vehicle based on the behavior of the peripheral vehicle estimated in the first estimation step, and a second estimation step.
It has a notification step of notifying the road condition in the traveling direction of the vehicle estimated in the second estimation step.
In the first estimation step, the behavior of the peripheral vehicle is estimated based on the change of the image different from the vehicle image of the peripheral vehicle among the captured images of the captured data.
A driving support method characterized by having. Computer,
An acquisition means for acquiring image pickup data of a peripheral vehicle located in the traveling direction of the vehicle, which is captured by the image pickup means.
A first estimation means for estimating the behavior of the peripheral vehicle based on the imaging data acquired by the acquisition means,
A second estimation means for estimating the road condition in the traveling direction of the vehicle based on the behavior of the peripheral vehicle estimated by the first estimation means.
It functions as a notification means for notifying the road condition in the traveling direction of the vehicle estimated by the second estimation means.
The first estimation means estimates the behavior of the peripheral vehicle based on the change of the image different from the vehicle image of the peripheral vehicle in the captured image of the captured data.
A program to function as.

Images