WO2024202015A1 - 制御装置、車両、制御方法、記憶媒体及びプログラム - Google Patents

制御装置、車両、制御方法、記憶媒体及びプログラム Download PDF

Info

Publication number
WO2024202015A1
WO2024202015A1 PCT/JP2023/013592 JP2023013592W WO2024202015A1 WO 2024202015 A1 WO2024202015 A1 WO 2024202015A1 JP 2023013592 W JP2023013592 W JP 2023013592W WO 2024202015 A1 WO2024202015 A1 WO 2024202015A1
Authority
WO
WIPO (PCT)
Prior art keywords
notification
risk
vehicle
target
user
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2023/013592
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
裕司 安井
健太郎 山田
亮太 三橋
稔之 田中
さち恵 坂田
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Honda Motor Co Ltd
Original Assignee
Honda Motor Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Honda Motor Co Ltd filed Critical Honda Motor Co Ltd
Priority to PCT/JP2023/013592 priority Critical patent/WO2024202015A1/ja
Priority to EP24779044.7A priority patent/EP4693253A1/en
Priority to CN202480022810.9A priority patent/CN121058049A/zh
Priority to JP2025510058A priority patent/JPWO2024202905A1/ja
Priority to PCT/JP2024/007567 priority patent/WO2024202905A1/ja
Publication of WO2024202015A1 publication Critical patent/WO2024202015A1/ja
Priority to US19/340,987 priority patent/US20260021823A1/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W50/00—Details of control systems for road vehicle drive control not related to the control of a particular sub-unit, e.g. process diagnostic or vehicle driver interfaces
    • B60W50/08—Interaction between the driver and the control system
    • B60W50/14—Means for informing the driver, warning the driver or prompting a driver intervention
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W30/00—Purposes of road vehicle drive control systems not related to the control of a particular sub-unit, e.g. of systems using conjoint control of vehicle sub-units
    • B60W30/08—Active safety systems predicting or avoiding probable or impending collision or attempting to minimise its consequences
    • B60W30/09—Taking automatic action to avoid collision, e.g. braking and steering
    • G—PHYSICS
    • G08—SIGNALLING
    • G08G—TRAFFIC CONTROL SYSTEMS
    • G08G1/00—Traffic control systems for road vehicles
    • G08G1/16—Anti-collision systems
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W50/00—Details of control systems for road vehicle drive control not related to the control of a particular sub-unit, e.g. process diagnostic or vehicle driver interfaces
    • B60W50/08—Interaction between the driver and the control system
    • B60W50/14—Means for informing the driver, warning the driver or prompting a driver intervention
    • B60W2050/143—Alarm means
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W2420/00—Indexing codes relating to the type of sensors based on the principle of their operation
    • B60W2420/40—Photo, light or radio wave sensitive means, e.g. infrared sensors
    • B60W2420/403—Image sensing, e.g. optical camera
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W2554/00—Input parameters relating to objects
    • B60W2554/40—Dynamic objects, e.g. animals, windblown objects
    • B60W2554/404—Characteristics
    • B60W2554/4041—Position
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
    • G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L13/00—Speech synthesis; Text to speech systems
    • G10L13/02—Methods for producing synthetic speech; Speech synthesisers
    • G10L13/027—Concept to speech synthesisers; Generation of natural phrases from machine-based concepts

Definitions

  • the present invention relates to a control device, a vehicle, a control method, a storage medium, and a program.
  • Patent Document 1 Conventionally, there is known a technology that issues a warning to the driver and controls the brakes when it is predicted that a moving vehicle may collide with a pedestrian or other passerby (Patent Document 1). Patent Document 1 also discloses that information to alert the driver (information pointing in the direction of a user with a high risk score or a message displaying a warning) is projected onto the windshield of the vehicle, regardless of whether a collision is predicted to occur.
  • an audible warning is issued when a pedestrian is approaching the vicinity of a moving vehicle
  • the driver of the vehicle can identify the object with a high probability of collision and the operation to avoid the collision (such as decelerating) without any additional information.
  • the driver may not be able to immediately understand what the warning is about, so it is useful to display visual information (attention alert information) on the windshield to notify the driver of the object of the warning.
  • visual information attention alert information
  • the driver may feel uncomfortable or confused about what to do.
  • the present invention was made in consideration of the above problems, and its purpose is to realize technology that can provide appropriate notifications depending on the situation.
  • a control device disposed in a vehicle equipped with an imaging device, an image acquisition means for acquiring an image of the outside of the vehicle by the imaging device; A recognition means for recognizing a target outside the vehicle based on the image; A means for identifying a risk object that has a risk of being in proximity to the vehicle among the recognized targets; A notification control means for notifying a user of a voice notification using a natural language including an expression representing a recognized target,
  • a control device is provided, characterized in that the notification control means sends a first notification including either a direct notification that notifies the user of at least the existence of a risk posed by a risk object and an indirect notification that notifies the user of information suggesting the risk object based on the result of identifying the risk object and the degree of proximity between the risk object and the vehicle.
  • the present invention makes it possible to provide appropriate notifications depending on the situation.
  • FIG. 1 is a diagram showing an example of the configuration of a vehicle according to an embodiment
  • FIG. 2 is a block diagram showing an example of a functional configuration of a control device according to an embodiment
  • FIG. 1 is a diagram illustrating an example of a notification output by a notification control unit according to an embodiment.
  • FIG. 1 is a diagram for explaining the relationship between the state of a risk object and a notification mode according to an embodiment
  • FIG. 1 is a diagram for explaining an example of a case where a risk target is not specified according to an embodiment
  • FIG. 1 is a diagram for explaining an example in which a risk target is specified according to an embodiment
  • FIG. 13 is a diagram for explaining another example in which a risk target according to the embodiment is specified
  • a flowchart showing a series of operations in a notification process according to an embodiment A flowchart showing a series of operations of a notification control process according to an embodiment.
  • FIG. 13 is a diagram for explaining an example of a notification depending on whether a risk object is visually recognized according to the embodiment;
  • Fig. 1 is a block diagram of a vehicle 1 according to an embodiment of the present invention.
  • the vehicle 1 is shown in a schematic plan view and a side view.
  • the vehicle 1 is a four-wheeled passenger car, but it may be a two-wheeled vehicle or another type of vehicle.
  • the vehicle 1 includes a vehicle control device 2 (hereinafter simply referred to as the control device 2) that controls the vehicle 1.
  • the control device 2 includes a plurality of ECUs (Electronic Control Units) 20-29 that are communicatively connected via an in-vehicle network.
  • Each ECU includes a processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), a memory such as a semiconductor memory, an interface with an external device, etc.
  • the memory stores programs executed by the processor and data used by the processor for processing, etc.
  • Each ECU may include a plurality of processors, memories, interfaces, etc.
  • the ECU 20 includes a processor 20a and a memory 20b.
  • the ECU 20 executes processing by having the processor 20a execute instructions included in a program stored in the memory 20b.
  • the ECU 20 may include a dedicated integrated circuit such as an ASIC (Application Specific Integrated Circuit) for executing processing by the ECU 20. The same applies to the other ECUs.
  • ASIC Application Specific Integrated Circuit
  • each ECU 20-29 is responsible for The functions etc. that each ECU 20-29 is responsible for are described below. Note that the number of ECUs and the functions they are responsible for can be designed as appropriate, and they can be subdivided or integrated more than in this embodiment. For example, one ECU (e.g. ECU 22) may also have the functions of other ECUs.
  • the ECU 20 executes control related to manual driving and automatic driving of the vehicle 1.
  • automatic driving at least one of the steering and acceleration/deceleration of the vehicle 1 is automatically controlled.
  • automatic driving by the ECU 20 may include automatic driving that does not require driving operation by the driver (also called automatic driving), and automatic driving that assists the driver in driving operation (also called driving assistance).
  • the control of driving by the ECU 20 may include, for example, control to automatically stop or steer the vehicle to avoid a collision in place of driving by the driver.
  • the ECU 21 controls the electric power steering device 3.
  • the electric power steering device 3 includes a mechanism for steering the front wheels in response to the driver's operation (steering operation) on the steering wheel 31.
  • the electric power steering device 3 also includes a motor that generates driving force to assist the steering operation and automatically steer the front wheels, a sensor that detects the steering angle, etc.
  • the ECU 21 automatically controls the electric power steering device 3 in response to instructions from the ECU 20, and controls the traveling direction of the vehicle 1.
  • Vehicle 1 includes, for example, one standard camera 40 and four fisheye cameras 41 to 44 as detection units that detect the vehicle's surroundings.
  • Standard camera 40 and fisheye cameras 42 and 44 are connected to ECU 22.
  • Fisheye cameras 41 and 43 are connected to ECU 23.
  • ECUs 22 and 23 can recognize the type and movement trajectory of targets in the images, lane areas on the road, and dividing lines (white lines, etc.) by analyzing images captured by standard camera 40 and fisheye cameras 41 to 44.
  • the type, number, and mounting positions of the cameras possessed by vehicle 1 are not limited to the example of this embodiment and may be other configurations.
  • the standard camera 40 is attached to the center of the front of the vehicle 1 and captures the surroundings in front of the vehicle 1.
  • the fisheye camera 41 is attached to the center of the front of the vehicle 1 and captures the surroundings in front of the vehicle 1.
  • the standard camera 40 and the fisheye camera 41 are shown aligned horizontally.
  • the arrangement of the standard camera 40 and the fisheye camera 41 is not limited to this, and for example, they may be aligned vertically.
  • at least one of the standard camera 40 and the fisheye camera 41 may be attached to the front of the roof of the vehicle 1 (for example, the inside of the passenger compartment of the front window).
  • the fisheye camera 42 is attached to the center of the right side of the vehicle 1 and captures the surroundings to the right of the vehicle 1.
  • the fisheye camera 43 is attached to the center of the rear of the vehicle 1 and captures the surroundings to the rear of the vehicle 1.
  • the fisheye camera 44 is attached to the center of the left side of the vehicle 1 and captures the surroundings to the left of the vehicle 1.
  • Vehicle 1 may include a lidar (Light Detection and Ranging) or millimeter wave radar as a detection unit for detecting targets around vehicle 1 and measuring the distance to the targets.
  • lidar Light Detection and Ranging
  • millimeter wave radar millimeter wave radar
  • ECU 22 controls standard camera 40 and fisheye cameras 42 and 44 and processes information on the detection results.
  • ECU 23 controls fisheye cameras 41 and 43 and processes information on the detection results. By dividing the detection unit that detects the vehicle's surroundings into two systems, the reliability of the detection results can be improved.
  • ECU 22 can detect the driver's head direction and line of sight using an image of the driver captured by a fisheye camera (not shown) installed inside the vehicle cabin.
  • the ECU 24 controls the gyro sensor 5, GPS sensor 24b, and communication device 24c, and processes information on the detection results or communication results.
  • the gyro sensor 5 detects the rotational motion of the vehicle 1.
  • the path of the vehicle 1 can be determined based on the detection results of the gyro sensor 5, the wheel speed, etc.
  • the GPS sensor 24b detects the current position of the vehicle 1.
  • the communication device 24c acquires this information through wireless communication with a server that provides map information and traffic information.
  • the ECU 24 can access a database 24a of map information constructed in memory, and performs operations such as searching for a route from the current location to the destination.
  • the ECU 24, map database 24a, and GPS sensor 24b make up what is known as a navigation device.
  • the ECU 25 includes a communication device 25a for vehicle-to-vehicle communication.
  • the communication device 25a for example, performs wireless communication with other vehicles in the vicinity and exchanges information between the vehicles.
  • the ECU 26 controls the power plant 6.
  • the power plant 6 is a mechanism that outputs driving force to rotate the drive wheels of the vehicle 1, and includes, for example, an engine and a transmission.
  • the ECU 26 controls the output of the engine in response to the driver's driving operation (accelerator operation or acceleration operation) detected by an operation detection sensor 7a provided on the accelerator pedal 7A, for example, and switches the gears of the transmission based on information such as the vehicle speed detected by the vehicle speed sensor 7c.
  • the ECU 27 controls lighting devices (headlights, taillights, etc.) including turn signals 8 (blinkers).
  • the turn signals 8 are provided at the front, door mirrors, and rear of the vehicle 1.
  • the ECU 28 controls the input/output device 9.
  • the input/output device 9 outputs information to the driver and accepts information input from the driver.
  • the audio output device 91 notifies the driver of information by audio, including speech, for example.
  • the notification content is generated, for example, by the ECU 22 performing a notification control process described below, and output by transmitting it to the ECU 28.
  • the display device 92 notifies the driver of information by displaying an image.
  • the display device 92 is arranged, for example, on the surface of the driver's seat, and constitutes an instrument panel or the like. Note that, although audio and display are exemplified here, information may be notified by vibration or light. Information may also be notified by a combination of audio, display, vibration, or light.
  • the input device 93 is a group of switches arranged in a position operable by the driver to give instructions to the vehicle 1, but may also include an audio input device.
  • the ECU 29 controls the brake device 10 and the parking brake (not shown).
  • the brake device 10 is, for example, a disc brake device provided on each wheel of the vehicle 1, and applies resistance to the rotation of the wheel to slow down or stop the vehicle 1.
  • the ECU 29 controls the operation of the brake device 10 in response to the driver's driving operation (brake operation) detected, for example, by an operation detection sensor 7b provided on the brake pedal 7B.
  • the driving state of the vehicle 1 is automatic driving
  • the ECU 29 automatically controls the brake device 10 in response to an instruction from the ECU 20, and controls the deceleration and stopping of the vehicle 1.
  • the brake device 10 and the parking brake can also be operated to keep the vehicle 1 stopped. Also, if the transmission of the power plant 6 is equipped with a parking lock mechanism, this can also be operated to keep the vehicle 1 stopped.
  • FIG. 2 shows an example of a functional configuration realized by the ECU 22 executing a program stored in an internal memory.
  • the functional configuration example shown in Fig. 2 focuses on a configuration related to a notification process, which will be described later. Therefore, the functions realized in the ECU 22 are not limited to those shown in Fig. 2 and may include other functions.
  • the image acquisition unit 201 acquires images of the outside of the vehicle 1 captured by the standard camera 40. It is also possible to acquire images captured by the standard camera 40 and the fisheye cameras 42 and 44.
  • the target recognition unit 202 recognizes targets outside the vehicle 1 based on the image acquired by the image acquisition unit 201.
  • Targets include, for example, pedestrians (pedestrians and cyclists) traveling on the road.
  • the target recognition unit 202 may recognize, for example, the type of target in the image, lane areas and dividing lines (white lines, etc.) on the road, for example, by inputting the image into one or more neural networks.
  • the risk assessment unit 203 identifies, from among the recognized targets (e.g., pedestrians), targets that pose a risk of coming close to the vehicle 1 as risk targets.
  • the risk assessment unit 203 estimates the trajectory of the pedestrian's movement based on the image acquired by the image acquisition unit 201.
  • the trajectory of the pedestrian's movement in the image may be estimated, for example, by inputting the image into one or more neural networks.
  • a known technique can be used as a method for estimating the trajectory of the pedestrian's movement in the image based on the image.
  • the trajectory of the pedestrian's movement may be estimated, for example, based on the direction of the pedestrian's body and face estimated from the image.
  • the risk assessment unit 203 calculates the predicted length of time until a collision between the vehicle 1 and the pedestrian, and the distance of the pedestrian from the trajectory on which the vehicle 1 is traveling. Then, based on the calculated length of time and the distance between the trajectory and the pedestrian, it calculates the risk of the pedestrian and the vehicle 1 approaching each other (for example, the risk of a collision).
  • the risk of approach may be a numerical value between 0 and 1, for example, or may be expressed in a stepwise manner divided into several stages (zero, low, medium, high, etc.).
  • the risk assessment unit 203 may identify, as a risk target, a target (for example, a pedestrian) whose risk of approach is greater than a predetermined risk threshold. Furthermore, if a risk target is not identified, the risk assessment unit 203 may determine that the presence of a risk target is unknown.
  • a target for example, a pedestrian
  • the risk assessment unit 203 may, for example, identify a recognized target object present in the image as a risk object, not just when the approaching risk is higher than the risk threshold. Furthermore, if it is determined that a target object such as a pedestrian is not present in the image, and further if a scene is recognized in the image in which a target object that creates a blind spot, such as a parked vehicle, is present (i.e., a predetermined scene in which there is statistically a risk of an object jumping out, etc.), it may determine that the presence of a risk object is unknown.
  • FIG. 5 shows a schematic diagram of a pedestrian 503 walking along the sidewalk as a vehicle 501 travels along a road 502.
  • the pedestrian 503 is walking at a distance from the vehicle 1.
  • the risk assessment unit 203 may determine, for example, that the risk of the vehicle 1 and the pedestrian 503 coming into close proximity is equal to or lower than a predetermined threshold, and may determine that the presence of a risk object is unknown.
  • FIG. 6 also shows a schematic diagram of a situation in which a pedestrian 603 is walking on the sidewalk as a vehicle 601 travels along a road 602, and an obstacle may cause the pedestrian's predicted trajectory to come close to the vehicle's travel trajectory.
  • the pedestrian 603 is walking at a distance from the vehicle 1 (distance 604 is large).
  • the risk assessment unit 203 may determine, for example, that the risk of the vehicle 601 and the pedestrian 603 coming close to each other is greater than a predetermined threshold, and determine that the pedestrian 603 is a risk target.
  • FIG. 7 shows a schematic diagram of a situation in which a pedestrian 703 is walking on the sidewalk as a vehicle 701 travels along a road 702, and an obstacle may cause the pedestrian's predicted trajectory to come close to the vehicle's travel trajectory.
  • the pedestrian 703 is walking close to the vehicle 701 (distance 704 is small). Therefore, the risk of the vehicle 701 and the pedestrian 703 coming close to each other is higher than the risk of the vehicle 601 and the pedestrian 603 coming close to each other. Therefore, the risk assessment unit 203 determines, for example, that the risk of the vehicle 701 and the pedestrian 703 coming close to each other is higher than a predetermined threshold, and determines the pedestrian 703 to be a risk target.
  • the gaze estimation unit 204 estimates whether the driver is viewing a pedestrian using an image captured by a camera inside the vehicle and including the driver (for convenience, referred to as a rear-view image).
  • the gaze estimation unit 204 estimates the direction of the driver's face and gaze using, for example, the rear-view image.
  • the direction of the driver's face and gaze may be estimated, for example, by inputting the rear-view image into one or more neural networks.
  • the gaze estimation unit 204 estimates whether the driver is viewing a pedestrian based on, for example, the trajectory of the pedestrian (position as seen from the vehicle 1) obtained by the risk assessment unit 203 and the direction of the driver's face and gaze.
  • the notification control unit 205 notifies the driver of a voice notification using natural language (including an expression expressing the recognized target) based on the result of identifying the risk target and the degree of proximity between the risk target and the vehicle 1.
  • the notification control unit 205 outputs either a direct notification or an indirect notification as a voice notification using natural language to the driver.
  • the notification control unit 205 outputs one of the indirect notification 301, the direct notification 302, and the alarm sound 303, depending on the degree of proximity between the vehicle and the risk object (e.g., a pedestrian) (e.g., the time until the predicted proximity occurs, or the distance from the vehicle 1 to the position where the predicted proximity occurs).
  • the example shown in FIG. 3 shows that the alarm sound 303, the direct notification 302, and the indirect notification 301 are output in order of the shortest time until the risk object and the vehicle come into proximity.
  • the time range satisfying the condition for outputting the indirect notification 301 may be wider than the time range satisfying the condition for outputting the direct notification 302.
  • FIG. 4 also shows the relationship between the state of a risk object and the notification mode. For example, an alarm sound 303 is output when the risk object is about to be approached. If the risk object is identified, a direct notification 302 is output (under certain conditions, such as the time until the approach occurs). For example, in the situation of the risk object shown in FIG. 7, a direct notification 302 is output. Furthermore, an indirect notification 301 is issued when the risk object is not identified (for example, the situation shown in FIG. 5) and under certain conditions when the risk object is identified (for example, the situation shown in FIG. 6, such as the time until the approach occurs being long). If the notification control unit 205 determines that no risk exists, such as when there is no object in the image, it does not output a notification.
  • the notification control unit 205 determines that no risk exists, such as when there is no object in the image, it does not output a notification.
  • the notification control unit 205 notifies the driver of the direct notification 302.
  • the direct notification includes, for example, an expression representing a target and an expression for causing the driver to start risk avoidance.
  • the direct notification includes an expression such as "Pedestrian in the left front. He may come here.
  • the direct notification may include an expression representing the position of the risk object (e.g., the left front), an expression representing the type of the target object that is the risk object (e.g., a pedestrian), and an expression for causing the driver to start risk avoidance (e.g., let's go slowly, etc.).
  • the expression for causing the driver to start risk avoidance may include an expression of a behavior that the driver should cause the vehicle to generate (e.g., slow down, etc.).
  • the direct notification may include at least one of an expression representing the behavior of the target predicted in the future (e.g., he may come here) and an expression representing the reason for starting risk avoidance (e.g., a pedestrian is approaching from the left front).
  • the direct notification has been described as including, as an example, an expression representing a target and an expression that causes the driver to start risk avoidance.
  • the direct notification may include various expressions as long as it notifies the user of the presence of a risk due to a specific target.
  • the direct notification may include an expression indicating the presence of a risk due to a risk object (specific target), such as "There is a pedestrian crossing the road.”
  • the direct notification may include an expression representing the position of the risk object, such as "There is a pedestrian approaching from the left front,” and an expression indicating the presence of a risk due to the risk object.
  • the direct notification may include an expression indicating the presence of a risk due to a risk object, such as "There is a pedestrian crossing the road. Let's go slowly.”
  • the notification control unit 205 outputs an alarm sound 303 if the time until the identified risk object and the vehicle 1 approach each other is equal to or shorter than a second time that is shorter than the first time (i.e., the state in which the vehicle 1 is about to approach the risk object).
  • the alarm sound 303 does not have to include natural language, for example. In other words, if the vehicle is about to approach a pedestrian who is a risk object (e.g., 2 seconds before), the driver can understand by hearing the output alarm sound and looking in the direction of travel that there is a possibility that the vehicle will collide with the pedestrian and that the vehicle should be slowed down or steered (i.e., the risk is self-evident). Therefore, by generating an alarm sound 303, the driver can take action to avoid the risk.
  • the notification control unit 205 notifies the driver of an indirect notification 301 when (a risk object has been identified) the time until the vehicle comes into close proximity with the identified risk object is longer than a predetermined first time.
  • the notification control unit 205 also notifies the driver of an indirect notification 301 when a risk object has not been identified in the risk assessment results of the risk assessment unit 203.
  • the risk state for outputting such an indirect notification is, for example, the state described above in Figures 5 and 6.
  • the indirect notification includes, for example, an expression that indicates a target but does not include an expression that causes the driver to start risk avoidance.
  • the indirect notification includes, for example, an expression such as "There are a lot of bicycles making deliveries these days.”
  • the indirect notification includes an expression that indicates the presence of a target.
  • the indirect notification does not include an expression that indicates the position of the target like the direct notification, and may simply indicate the presence of the target.
  • the indirect notification does not include an expression that identifies the individual target like the direct notification, and may include a general expression that does not identify the individual target.
  • the indirect notification does not include an expression that indicates the position of the target or an expression that causes the driver to start risk avoidance
  • the indirect notification does not cause the driver to perform excessive risk avoidance operations by simply informing the driver of the presence of the target.
  • the driver may not intuitively understand the risk content and may feel uncomfortable with the alarm sound that sounds frequently.
  • the driver can continue driving while understanding the potential risk.
  • the indirect notification has been described as including an expression representing a target but not including an expression that causes the driver to start risk avoidance.
  • the indirect notification may be an expression including information suggesting a risk target without identifying a specific risk target (individual), such as "There are a lot of bicycles making deliveries these days" or "There are a lot of children going to school.”
  • the information suggesting a risk target may include, for example, at least one of the characteristics of a risk target (target) for which an individual is not identified, information that can identify a risk target, and information other than the content of a risk related to the risk target.
  • the information other than the content of a risk related to a risk target may be information associated with current events or trending topics.
  • the ECU 22 can acquire and use information such as news and SNS published on the Internet via the communication device 24c.
  • the indirect notification may be a notification including information associated with a current events expression, such as "It seems that there has been an increase in children running out recently," and not identifying an individual child.
  • the notification control unit 205 may include an expression showing empathy with the user or an expression accepting the user in the evaluation notification 304 after notifying the driver of a direct or indirect notification.
  • the expression showing empathy with the user may include, for example, an expression such as "That kind of behavior is annoying, isn't it?".
  • the notification control unit 205 can increase the user's receptivity to the notification by notifying the user of a direct or indirect notification and including an expression showing empathy with the user or an expression accepting the user.
  • the expression for increasing the user's receptivity may include an expression praising the user and an expression giving advice to the user.
  • the notification control unit 205 may notify the driver of the evaluation notification 304 including an expression evaluating the user's behavior in accordance with the user operation (such as deceleration or steering) received after notifying the driver of the direct or indirect notification.
  • the notification control unit 205 may include in the evaluation notification 304 a representation praising the user or a representation giving advice to the user, depending on the time from when the direct or indirect notification is notified until when the user operation is accepted. For example, if a user operation is accepted within a predetermined time or less after the direct or indirect notification is notified, a representation praising the driver may be included in the notification, and if not, a representation giving advice to the user may be included in the notification.
  • the notification control unit 205 generates a direct notification utterance sentence by inputting, as an example, the type and position of the recognized risk object, the relative distance from the vehicle 1, etc., into a trained utterance generation algorithm.
  • the utterance generation algorithm is trained with direct notification data, which is a set of various risk objects, relative distances, etc., collected in advance, and utterance examples for direct notification, as learning data.
  • the notification control unit 205 also generates an indirect notification utterance sentence by inputting, as an example, the presence or absence of a risk object, the type and position of the risk object, the relative distance from the vehicle 1, etc., into a trained utterance generation algorithm.
  • the utterance generation algorithm is trained with indirect notification data, which is a set of various risk objects, relative distances, the presence or absence of a risk object, etc., collected in advance, and utterance examples for indirect notification, as learning data.
  • indirect notification data is a set of various risk objects, relative distances, the presence or absence of a risk object, etc., collected in advance, and utterance examples for indirect notification, as learning data.
  • a direct notification or indirect notification may be output using a single utterance generation algorithm.
  • the notification control unit 205 similarly generates notifications including expressions that praise the user or give advice to the user by using an utterance generation model trained using the praising utterance data and the advising utterance data.
  • the image acquisition unit 201 acquires an image of the outside of the vehicle 1 captured by, for example, the standard camera 40.
  • the target recognition unit 202 recognizes targets outside the vehicle 1 based on the image acquired by the image acquisition unit 201.
  • the risk assessment unit 203 identifies, from among the recognized targets (e.g., pedestrians), targets that pose a risk of being in close proximity to the vehicle 1 as risk targets.
  • the notification control unit 205 performs a notification control process, which will be described later. When the notification control unit 205 completes the notification control process, it ends this series of operations.
  • the notification control process described below is executed by the notification control unit 205.
  • the notification control unit 205 determines whether a risk exists. For example, the notification control unit 205 may determine that no risk exists when the target object in the image is not recognized by the target object recognition unit 202 (i.e., the target object does not exist). If the notification control unit 205 determines that no risk exists, it returns the process to the caller, and if not, it proceeds to S902.
  • the notification control unit 205 determines whether a risk target has been identified. For example, the notification control unit 205 determines that a risk target has been identified when a risk target has been identified by the risk assessment unit 203 as described above. If the notification control unit 205 determines that a risk target has been identified, the process proceeds to S903, and if not, the process proceeds to S906.
  • the notification control unit 205 determines whether the time until the vehicle 1 and the risk target come into close proximity is longer than a time threshold (first time). At this time, if the time until the vehicle 1 and the risk target come into close proximity is longer than the time threshold (first time), the notification control unit 205 may determine that the risk target and vehicle 1 satisfy a predetermined proximity condition. If the notification control unit 205 determines that the time until the vehicle 1 and the risk target come into close proximity is longer than the time threshold (first time), the process proceeds to S906, and if not, the process proceeds to S904.
  • the notification control unit 205 determines whether the driver is viewing the risk target. For example, the notification control unit 205 determines whether the driver is viewing the risk target based on the estimation result by the gaze estimation unit 204 as to whether the driver is viewing the risk target.
  • FIG. 10 shows a state in which a pedestrian 1003 is crossing a road while a vehicle 1001 is traveling on the road 1002.
  • the driver's line of sight 1010 is directed at the pedestrian 1003, and the driver is visually checking the pedestrian 1003 crossing the road 1002.
  • the notification control unit 205 determines that the driver is visually checking the risk target.
  • the notification control unit 205 can prevent the driver from outputting a notification when the driver is visually checking the risk target. By preventing the driver from notifying the driver even when the driver is visually checking the risk target, unnecessary notifications to the driver can be suppressed.
  • the notification control unit 205 determines that the driver is not visually checking the risk target.
  • the notification control unit 205 outputs a direct notification when the driver does not visually recognize the risk object (i.e., when there is a high possibility that the driver is unaware of the risk object). If the notification control unit 205 determines that the driver visually recognizes the risk object, it returns the process to the caller, and if not, it advances the process to S905.
  • the notification control unit 205 In S905, the notification control unit 205 generates an utterance sentence for direct notification using the utterance generation model trained with the above-mentioned direct notification data, and notifies the driver (user). In S906, the notification control unit 205 generates an utterance sentence for indirect notification using the utterance generation model trained with the above-mentioned indirect notification data, and notifies the driver (user). After outputting the notification, the notification control unit 205 returns the process to the caller.
  • the notification control unit 205 issues a direct or indirect notification based on the result of identifying the risk target and the degree of proximity between the risk target and the vehicle.
  • a direct notification includes an expression representing the target and an expression that causes the user to start risk avoidance
  • an indirect notification includes an expression representing the target but does not include an expression that causes the user to start risk avoidance. In this way, it becomes possible to provide an appropriate notification depending on the situation.
  • the control device (e.g., 2) of the above embodiment is A control device disposed in a vehicle (e.g., 1) equipped with an imaging device (e.g., 40), An image acquisition means (e.g., 201) for acquiring an image of the outside of the vehicle by the imaging device; A recognition means (e.g., 202) for recognizing a target outside the vehicle based on the image; A means for identifying a risk object (e.g., 203) that has a risk of being in close proximity to the vehicle among the recognized targets; A notification control means (e.g., 205) for notifying a user of a voice notification using a natural language including an expression representing the recognized target object, The notification control means sends a first notification including either a direct notification (e.g., 302) that notifies the user of at least the existence of a risk posed by a risk object, or an indirect notification (e.g., 301) that notifies the user of information suggesting the risk object, based on the result of identifying the risk
  • This embodiment makes it possible to provide appropriate notifications depending on the situation.
  • the notification control means includes the indirect notification in the first notification depending on the degree of proximity between the identified risk object and the vehicle.
  • the notification control means includes the direct notification in the first notification when a risk target is identified in the risk target identification result and the degree of proximity between the identified risk target and the vehicle satisfies a predetermined proximity condition.
  • the notification control means includes the direct notification in the first notification when a risk target is identified in the risk target identification result and the degree of proximity between the identified risk target and the vehicle satisfies a predetermined proximity condition, and when it is further estimated that the user does not visually recognize the risk target.
  • a notification can be output appropriately when there is a high possibility that the risk subject is unaware.
  • the notification control means controls so as not to output the first notification when a risk target is identified in the risk target identification result and the degree of proximity between the identified risk target and the vehicle satisfies a predetermined proximity condition, and when it is further estimated that the user is viewing the risk target.
  • unnecessary notifications to the user can be suppressed by not notifying the user even when the user is viewing a risk target.
  • the notification control means includes the indirect notification in the first notification when the time until the identified risk object and the vehicle come into close proximity is longer than a first time, and includes the direct notification in the first notification when the time until the identified risk object and the vehicle come into close proximity is equal to or less than the first time.
  • the indirect notification allows the user to continue driving while being aware of the potential risk, and if the time until the risk object and the vehicle come close to each other is short, the direct notification allows the user to pay attention to the specific risk object.
  • the notification control means outputs an alarm sound instead of the first notification when the time until the identified risk object and the vehicle come into close proximity is equal to or shorter than a second time that is shorter than the first time.
  • the user can be intuitively notified of risks in situations where they can understand the situation and the actions they should take by looking in the direction of travel.
  • the notification control means includes the indirect notification in the first notification.
  • the direct notification includes an expression representing the position of the target that is the risk object, an expression representing the type of the target that is the risk object, and an expression that causes the user to initiate risk avoidance.
  • the user can specifically identify areas that require attention and take the necessary action.
  • the direct notification further includes at least one of an expression representing a predicted future behavior of the target and an expression representing a reason for initiating the risk avoidance.
  • the user can easily understand what the situation will be if he or she continues driving.
  • the indirect notification includes, as an expression representing a target, an expression representing the presence of a target.
  • driving can be continued while being aware of potential risks.
  • the notification control means after notifying the user of the direct notification or the indirect notification, further outputs a second notification including an expression evaluating the user's behavior.
  • feedback on risk avoidance can be given to the user.
  • the notification control means After notifying the direct notification or the indirect notification, causes the second notification to include an expression showing sympathy with the user or an expression of acceptance of the user.
  • This embodiment can increase user acceptance of notifications.
  • a vehicle is provided that includes the control device described above.
  • a vehicle is provided that is capable of providing appropriate notifications depending on the situation.

Landscapes

  • Engineering & Computer Science (AREA)
  • Automation & Control Theory (AREA)
  • Transportation (AREA)
  • Mechanical Engineering (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Multimedia (AREA)
  • Theoretical Computer Science (AREA)
  • Traffic Control Systems (AREA)
PCT/JP2023/013592 2023-03-31 2023-03-31 制御装置、車両、制御方法、記憶媒体及びプログラム Ceased WO2024202015A1 (ja)

Priority Applications (6)

Application Number Priority Date Filing Date Title
PCT/JP2023/013592 WO2024202015A1 (ja) 2023-03-31 2023-03-31 制御装置、車両、制御方法、記憶媒体及びプログラム
EP24779044.7A EP4693253A1 (en) 2023-03-31 2024-02-29 Control device and control method
CN202480022810.9A CN121058049A (zh) 2023-03-31 2024-02-29 控制装置、以及控制方法
JP2025510058A JPWO2024202905A1 (https=) 2023-03-31 2024-02-29
PCT/JP2024/007567 WO2024202905A1 (ja) 2023-03-31 2024-02-29 制御装置、及び制御方法
US19/340,987 US20260021823A1 (en) 2023-03-31 2025-09-26 Control apparatus and control method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2023/013592 WO2024202015A1 (ja) 2023-03-31 2023-03-31 制御装置、車両、制御方法、記憶媒体及びプログラム

Publications (1)

Publication Number Publication Date
WO2024202015A1 true WO2024202015A1 (ja) 2024-10-03

Family

ID=92904544

Family Applications (2)

Application Number Title Priority Date Filing Date
PCT/JP2023/013592 Ceased WO2024202015A1 (ja) 2023-03-31 2023-03-31 制御装置、車両、制御方法、記憶媒体及びプログラム
PCT/JP2024/007567 Ceased WO2024202905A1 (ja) 2023-03-31 2024-02-29 制御装置、及び制御方法

Family Applications After (1)

Application Number Title Priority Date Filing Date
PCT/JP2024/007567 Ceased WO2024202905A1 (ja) 2023-03-31 2024-02-29 制御装置、及び制御方法

Country Status (5)

Country Link
US (1) US20260021823A1 (https=)
EP (1) EP4693253A1 (https=)
JP (1) JPWO2024202905A1 (https=)
CN (1) CN121058049A (https=)
WO (2) WO2024202015A1 (https=)

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2007051973A (ja) * 2005-08-19 2007-03-01 Denso Corp 危険箇所情報表示装置
JP2010182085A (ja) * 2009-02-05 2010-08-19 Fuji Heavy Ind Ltd 運転行動誘導システム
JP2016184200A (ja) * 2015-03-25 2016-10-20 住友電気工業株式会社 歩行者接近報知装置、歩行者接近報知システム、コンピュータプログラム及び歩行者接近報知方法
JP2020129289A (ja) * 2019-02-08 2020-08-27 スズキ株式会社 運転支援装置
JP2020194499A (ja) * 2019-05-30 2020-12-03 株式会社デンソー 情報提示装置、情報提示方法および情報提示プログラム
JP2021026720A (ja) * 2019-08-08 2021-02-22 本田技研工業株式会社 運転支援装置、車両の制御方法、およびプログラム
JP2021105994A (ja) * 2019-12-26 2021-07-26 パナソニックＩｐマネジメント株式会社 表示制御装置、表示システム、表示制御方法

Family Cites Families (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2009251814A (ja) * 2008-04-03 2009-10-29 Xanavi Informatics Corp 車載用運転評価装置、運転評価方法
US9043133B2 (en) * 2011-12-29 2015-05-26 Intel Corporation Navigation systems and associated methods
JP2015219830A (ja) * 2014-05-20 2015-12-07 トヨタ自動車株式会社 運転支援装置
JP6860553B2 (ja) * 2016-03-30 2021-04-14 川崎重工業株式会社 鞍乗型車両の情報出力装置
WO2020003392A1 (ja) * 2018-06-27 2020-01-02 三菱電機株式会社 運転支援装置および運転モード判断モデル生成装置
US11635299B2 (en) * 2020-02-06 2023-04-25 Mitsubishi Electric Research Laboratories, Inc. Method and system for scene-aware interaction
WO2022239327A1 (ja) 2021-05-14 2022-11-17 ソニーグループ株式会社 情報処理装置および情報処理方法、並びに情報処理システム
US11861321B1 (en) * 2023-06-29 2024-01-02 Casetext, Inc. Systems and methods for structure discovery and structure-based analysis in natural language processing models
JP7430437B1 (ja) * 2023-09-25 2024-02-13 株式会社Zisedai 印刷物に印刷された文字情報を収集する方法、プログラム及び情報処理装置

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2007051973A (ja) * 2005-08-19 2007-03-01 Denso Corp 危険箇所情報表示装置
JP2010182085A (ja) * 2009-02-05 2010-08-19 Fuji Heavy Ind Ltd 運転行動誘導システム
JP2016184200A (ja) * 2015-03-25 2016-10-20 住友電気工業株式会社 歩行者接近報知装置、歩行者接近報知システム、コンピュータプログラム及び歩行者接近報知方法
JP2020129289A (ja) * 2019-02-08 2020-08-27 スズキ株式会社 運転支援装置
JP2020194499A (ja) * 2019-05-30 2020-12-03 株式会社デンソー 情報提示装置、情報提示方法および情報提示プログラム
JP2021026720A (ja) * 2019-08-08 2021-02-22 本田技研工業株式会社 運転支援装置、車両の制御方法、およびプログラム
JP2021105994A (ja) * 2019-12-26 2021-07-26 パナソニックＩｐマネジメント株式会社 表示制御装置、表示システム、表示制御方法

Also Published As

Publication number Publication date
US20260021823A1 (en) 2026-01-22
JPWO2024202905A1 (https=) 2024-10-03
WO2024202905A1 (ja) 2024-10-03
EP4693253A1 (en) 2026-02-11
CN121058049A (zh) 2025-12-02

Similar Documents

Publication Publication Date Title
JP7115270B2 (ja) 自動運転システム
JP7158352B2 (ja) 運転支援装置、車両の制御方法、およびプログラム
JP6515814B2 (ja) 運転支援装置
CN111443708B (zh) 自动驾驶系统
JP6693354B2 (ja) 車両用情報提示装置
JP4604691B2 (ja) 車両用警報装置、車両周囲状況の警報方法
JP5401961B2 (ja) 操舵支援装置
JP7411706B2 (ja) 運転支援装置
WO2019155778A1 (ja) 衝突回避支援装置、プログラム、衝突回避支援方法
CN114148336A (zh) 车辆控制方法及装置
CN115140062B (zh) 驾驶辅助装置
US11403948B2 (en) Warning device of vehicle and warning method thereof
WO2024202015A1 (ja) 制御装置、車両、制御方法、記憶媒体及びプログラム
JP7775125B2 (ja) 運転支援装置及び運転支援システム
CN111959519B (zh) 一种辅助驾驶功能设置方法、装置、设备及介质
JP7637238B2 (ja) 変化した周辺環境に対して検知および警告する方法
KR20230106207A (ko) 적어도 하나 이상의 모드에 따라, 제어되는 차량 및 방법
CN115709726B (zh) 驾驶辅助装置
CN115703461B (zh) 驾驶辅助装置
US11932167B2 (en) System and method of alerting pedestrians to vehicles
US20250304042A1 (en) Moving body control system, control method thereof, and storage medium
US20250100443A1 (en) Apparatus and method for rear communication display
CN121246807A (zh) 一种车辆控制方法、装置、电子设备和车辆
CN114872692A (zh) 驾驶辅助装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23930650

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 23930650

Country of ref document: EP

Kind code of ref document: A1