WO2022124164A1

WO2022124164A1 - Attention object sharing device, and attention object sharing method

Info

Publication number: WO2022124164A1
Application number: PCT/JP2021/044133
Authority: WO
Inventors: 加奈子金澤; 好文伊藤; 大地八木; 昊舟李; 友英西野; 裕子中村
Original assignee: 株式会社デンソー
Priority date: 2020-12-11
Filing date: 2021-12-01
Publication date: 2022-06-16
Also published as: CN116547729A; JP7537259B2; JP2022093012A

Abstract

When an interest reaction by a child is detected based on input data from a child camera (25) and a child sensor, an HCU (1) identifies an attention object that is something outside a vehicle that a child shows interest in based on the child's line of sight information and an image of a vehicle-external camera at the time of that detection. The name, type, characteristics, etc., of the attention object are acquired from a server (5) and presented to the driver and the child. Information about the attention object is saved in a server, smartphone, etc., so that it can be referenced later.

Description

Attention target sharing device, attention target sharing method

Cross-reference of related applications

This application is based on Patent Application No. 2020-206044 filed in Japan on December 11, 2020, and the contents of the basic application are incorporated by reference as a whole.

This disclosure relates to a technique in which the driver shares information about things outside the vehicle that the person sitting in the backseat is interested in.

Patent Document 1 describes an out-of-vehicle camera based on the fact that a passenger who is a occupant other than the driver makes a specific facial expression or action, emits a voice expressing a specific emotion, or performs a predetermined instruction operation. The technique of displaying the captured image of the above on a display visible to the driver is disclosed. The image displayed on the display is an image of a subject existing in the direction of the passenger's line of sight outside the vehicle interior. As described above, in Patent Document 1, in the configuration disclosed in Patent Document 1, an image including a subject that the passenger may be interested in is displayed on the display.

Japanese Unexamined Patent Publication No. 2014-96632

As a vehicle usage pattern, a child may be seated in a child's seat provided in the back seat, and a guardian (for example, a parent) may be seated in the driver's seat. Children can be more interested in a variety of objects than adults. And children often ask their parents as drivers about things they are interested in and tell them about their existence.

However, parents as drivers need to concentrate on driving, so they cannot always respond politely to children's questions. Also, it is difficult for the driver to know what the child sitting in the back seat is looking at. Therefore, even when the driving load is light, it is difficult for the driver to respond closely to the child's interest reaction. It becomes even more difficult, especially when the child is still unable to verbalize things well, or when the vocabulary is low.

To solve such a problem, according to the technique disclosed in Patent Document 1, an image including a subject that may be of interest to a child is displayed on the display, so that the driver can see the image. You can have some idea of what your child is interested in. However, in the configuration disclosed in Patent Document 1, only an image is displayed. Therefore, in order for parents to respond / sympathize with the child's reaction to things outside the vehicle interior, it is necessary to identify the subject that the child would have focused on based on the displayed image, verbalize it, and speak. Yes, the burden on the driver is still large. The driver here refers to a driver's seat occupant who is a person seated in the driver's seat.

The present disclosure is based on this circumstance, and the purpose of the present disclosure is to make it easier for the driver's seat occupant to respond to the reaction of the child to things outside the vehicle interior. To provide a method.

The focus sharing device for achieving that purpose is a focus sharing device used in a vehicle provided with a child seat, which is a seat for a child to sit on, and is seated on the child seat. As information indicating the state of the child, a child information acquisition unit that acquires the line-of-sight direction of the child based on an image of a vehicle interior camera that includes at least the face of the child in the imaging range, a child's biological information, a child's voice, and Attached to the vehicle so as to image the line-of-sight direction acquired by the child information acquisition unit and the interest reaction detection unit that detects the child's interest reaction to things outside the vehicle interior based on at least one of the line-of-sight directions. Attention is paid from the attention target detection unit that detects the attention target that is the target that the child is interested in, and the database arranged inside or outside the vehicle, based on the captured image of the vehicle outside camera (28). The driver's seat occupant and the driver's seat occupant and the information acquired by the target information acquisition unit using at least one of the display of the text corresponding to the information and the voice output of the target information acquisition unit for acquiring the verbalized information about the target. It is provided with a notification processing unit that notifies at least one of the children.

In the above configuration, based on the child's line-of-sight information, the thing that the child is paying attention to is specified as the object of interest, and at least the child or the driver's seat occupant obtains the verbalized information about the object of interest. Provide to one side. Here, according to the configuration of notifying the guardian as a driver's seat occupant of information about the object of interest, not only the image but also the verbalized information is notified, so that the guardian pays attention to what the child pays attention to. It is easy to recognize if you are doing it. Therefore, it becomes easier to have a conversation with the child on the theme of the thing that the child is paying attention to.

In addition, according to the configuration for notifying the child of the information about the object of interest, the verbalized information about the object of interest is also notified, and based on the information, the driver's seat occupant knows what he / she is paying attention to. It will be easier to tell. As a result, it becomes easier for the driver's seat occupant to respond to the child's interest reaction.

Also, the focus sharing method to achieve the above objectives is at least one processor for parents to share things that the child is interested in sitting in the child's seat preset in the vehicle. A method of sharing attention that is performed by the child's gaze direction and acquisition of the child's line-of-sight direction based on the image of the vehicle interior camera that includes at least the face of the child sitting on the child's seat in the imaging range. To detect the child's interest reaction to things outside the vehicle interior based on at least one of biometric information, the child's voice, and the line-of-sight direction, and to image the acquired child's line-of-sight direction and the outside of the vehicle. Based on the image taken by the outside camera attached to the vehicle, the child can detect the object of interest that is of interest to the child, and the language of the object of interest from the database placed inside or outside the vehicle. At least one of the driver's seat occupant and the child, using at least one of the display of the text corresponding to the information and the voice output of the acquired information about the acquired information and the acquired information. (S110, S111, S113) and the like.

The above method is a method executed by the above-mentioned attention target sharing device. According to the above method, the same effect can be obtained by the same operation as that of the target sharing device of interest.

The reference numerals in parentheses described in the claims indicate the correspondence with the specific means described in the embodiment described later as one embodiment, and do not limit the technical scope of the present disclosure. do not have.

It is a block diagram which shows an example of the whole structure of the attention target sharing system. It is a figure which shows an example of the structure of a window display. It is a functional block diagram of HCU. It is a flowchart for demonstrating operation of HCU. It is a figure which shows an example of the notification image about the object of interest (dog). It is a figure which shows an example of the notification image about the object of interest (car). It is a figure which shows an example of the notification image about the object of interest (building). It is a figure which shows an example of the notification image about the object of interest (rainbow).

Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. FIG. 1 is a diagram showing an example of a schematic configuration of a attention target sharing system Sys to which the attention target sharing device according to the present disclosure is applied. Some or all of the elements constituting the attention target sharing system Sys of the present disclosure are mounted on the vehicle. Some of the functional elements included in the attention target sharing system Sys may be provided outside the vehicle. In addition, the attention target sharing system Sys can be understood as a system that supports communication between a child and a guardian as a driver in one aspect. Therefore, the attention target sharing system Sys can also be called a communication support system.

<Introduction>
The driver in the present disclosure refers to a person sitting in the driver's seat, that is, a driver's seat occupant. The expression "driver" is not limited to a person who actually performs a part or all of the driving operation. The description of a driver refers to a person who should receive the authority of driving operation from the automatic driving system during automatic driving. Here, as an example, it is assumed that a child is in a child seat installed in the back seat and a guardian of the child sitting in the back seat is seated in the driver's seat as a driver. The operation of the attention target sharing system Sys will be described. The concept of a guardian includes the mother of the child, the father, and relatives such as grandparents. Babysitters may also be included in parents according to local customs or laws in which the system is used.

A child seat is fixed to a predetermined position among the rear seats of the vehicle, such as a seat located behind the driver's seat, by a mechanism such as a seat belt or ISOFIX. ISOFIX has a configuration defined in ISO13216-1: 1999, and is also referred to as LATCH (Lower Anchors and Tethers for Children). The child seat here refers to a occupant protection device or a restraint device for children in a broad sense, and may include a booster seat, a junior seat, and the like for adjusting the height of the seating surface. Seats where children are seated, such as seats with child seats, are hereinafter referred to as children's seats. If the condition that the installation of the restraint device can be omitted is satisfied, for example, when the child's body is sufficiently large, the child seat may not be installed on the child seat. Children's seats need only be seats that children can sit in, and are not necessarily exclusively for children. Children's seats may sometimes be configured as seats that adults can also sit in.

Further, the level pointed to by "automatic driving" in the present disclosure may be, for example, equivalent to level 3 defined by the American Society of Automotive Engineers of Japan (SAE International), or may be level 4 or higher. Level 3 refers to the level at which the system executes all operation tasks within the operational design domain (ODD), while the operation authority is transferred from the system to the user in an emergency. The ODD defines conditions under which automatic driving can be executed, such as the traveling position being in a highway. At level 3, the driver is required to be able to respond promptly when there is a request for a change of operation from the system. Level 3 corresponds to so-called conditional automated driving. Level 4 is a level at which the system can perform all driving tasks except under specific circumstances such as unresponsive roads and extreme environments. Level 5 is the level at which the system can perform all driving tasks in any environment. Level 4 or higher automated driving refers to the level at which the automated driving device performs all driving tasks, that is, the automated level at which the driver's seat occupants are allowed to sleep. It should be noted that level 2 or lower corresponds to a driving support level in which the driver executes at least a part of driving tasks of steering and acceleration / deceleration. The system here refers to an in-vehicle system including the attention target sharing system Sys.

<About the configuration of the attention target sharing system Sys>
As shown in FIG. 1, the attention target sharing system Sys includes an HCU (HMI Control Unit) 1 that controls the operation of an HMI (Human Machine Interface) such as a display. HCU1 corresponds to a shared device of interest.

HCU1 is used by being connected to various in-vehicle devices such as a driver status monitor (hereinafter, DSM: Driver Status Monitor) 21. For example, the HCU 1 is connected to a DSM 21, a driver sensor 22, a driver microphone 23, an input device 24, a child camera 25, a child sensor 26, a child microphone 27, an outside camera 28, and a locator 29. The HCU 1 is also connected to a communication device 31, a meter display 32, a center display 33, a head-up display (HUD) 34, a rear seat display 35, a window display 36, a dialogue device 37, a speaker 38, and the like.

In addition, the HCU1 is also connected to various sensors / devices (not shown in FIG. 1) via the in-vehicle network Nw, which is a communication network constructed in the vehicle. For example, the HCU 1 is configured to be capable of intercommunication with a computer that controls the running of the vehicle such as an automatic driving device via the in-vehicle network Nw, and is a signal indicating whether or not the current vehicle is in the automatic driving mode. Can be entered. Further, the HCU1 is input with the detection results of various in-vehicle sensors and the like via the in-vehicle network Nw. Examples of the in-vehicle sensor include sensors that detect vehicle speed, acceleration, steering angle, shift position, accelerator depression amount, brake depression amount, and the like. In-vehicle sensors also include sensors / switches that detect the operating state of the parking brake and the power state of the vehicle.

The HCU1 and various devices may be connected by a dedicated line or may be connected via the in-vehicle network Nw. Further, an ECU (Electronic Control Unit) may be interposed between the HCU 1 and the in-vehicle device.

The DSM21 is a device that sequentially detects the user's state based on the user's face image. The DSM 21 includes, for example, a near-infrared light source, a near-infrared camera, and a control module for controlling them. The DSM 21 is installed in a posture in which the near-infrared camera faces the direction in which the headrest of the driver's seat is present, for example, on the upper surface of the steering column portion, the upper surface of the instrument panel, or the like. The DSM 21 uses a near-infrared camera to photograph the head of the driver irradiated with near-infrared light by a near-infrared light source. The image captured by the near-infrared camera is image-analyzed by the control module. The control module extracts driver state information, which is information indicating the driver's state, such as the opening of the driver's eyes, from the captured image input from the near-infrared camera. The camera constituting the DSM 21 may be a visible light camera. The DSM 21 outputs the driver status information extracted from the driver's face image to the HCU1.

The driver status information includes, for example, the direction of the driver's face, the direction of the line of sight, the opening of the eyelids, the opening of the pupil, the opening of the mouth, the posture, and the like. Further, the DSM 21 may be configured to estimate the facial expression, emotion, etc. of the driver based on the distribution of facial feature points and the like. As a method for detecting facial expressions based on image analysis, various methods such as a method using a distribution pattern of facial feature points and a method using facial muscle movement can be used, and therefore detailed description thereof is omitted here. As an example, multiple types of Action units (hereinafter referred to as AU) defined as units of movement corresponding to each muscle of the face based on anatomical knowledge are detected, and the detection amount (score) pattern for each AU is detected. It is possible to adopt a method of estimating facial expressions based on. Examples of the AU include lowering the eyebrows, raising the cheeks, raising the corners of the mouth, and raising the upper lip.

In addition, the DSM21 may estimate the degree of driver tension based on the driver's facial expression and the like. In addition, the health condition of the driver may be estimated based on the color information of the driver's face. The driver's status information can include facial expressions, emotions, tension, health status, and the like. The HCU1 may have a function of detecting the state of the driver based on image analysis. In that case, the DSM 21 may be configured to be able to output the driver's face image to the HCU1. The functional arrangement between DSM21 and HCU1 can be changed as appropriate.

The driver sensor 22 is a biosensor that senses the biometric information of the driver. The driver sensor 22 is, for example, a pulse wave sensor that senses a pulse wave. The driver sensor 22 includes blood pressure, electrocardiogram, heart rate, sweating amount, body temperature, heat dissipation from the human body, breathing rhythm, breathing depth, exhaled breath component, body composition, posture, body movement, skin electrical activity, and facial muscles. The sensor may be a sensor whose detection target is at least one of the activity potential of the body and the peripheral blood pressure. Peripheral blood flow refers to blood flow in the peripheral part such as a fingertip. Biological sensors include temperature sensors, pulse wave sensors, humidity sensors, heart rate sensors and the like. In addition, the concept of biometric information can include various state quantities as described above. A plurality of types of driver sensors 22 having different biological information to be detected may be connected to the HCU 1. The above-mentioned DSM 21 can also be included in the driver sensor 22 in a broad sense.

The driver sensor 22 may be built in the backrest of the driver's seat or the headrest, or may be provided in the steering. In addition, by transmitting and receiving millimeter waves as exploration waves toward the driver's seat, millimeter wave radars that detect the driver's heart rate, body movement, and posture can also be included in the biosensor. The driver sensor 22 may be a thermography camera. As the driver sensor 22, sensors having various detection principles such as a radio wave type sensor and a driver sensor 22 using infrared rays can be adopted.

In addition, some or all of the various driver sensors 22 may be wearable devices that are worn and used on the driver, for example, the wrist. As the wearable device, various shapes such as a wristband type, a wristwatch type, a ring type, a glasses type, and an earphone type can be adopted. The wearable device as the driver sensor 22 is configured to be capable of intercommunication with the HCU 1 via a communication device 31 mounted on the vehicle. The connection mode between the wearable device and the communication device 31 may be a wired connection or a wireless connection. As a wireless connection method, short-range wireless communication standards such as Bluetooth (registered trademark) and Wi-Fi (registered trademark) can be adopted. In addition, as the biological sensor, a skin electrical activity (EDA) sensor that detects changes in skin conductance (skin surface potential) derived from sympathetic nervous system activity can also be adopted.

The driver microphone 23 is a device that converts ambient sounds such as voices spoken by passengers in the front seats into electrical signals and inputs them to the HCU 1. The driver microphone 23 is arranged, for example, on the upper surface of the steering column cover, the steering wheel, the central portion of the instrument panel, or the like so as to easily collect the voice spoken by the driver.

The input device 24 is an operating member for receiving a driver's instruction to the HCU1. The input device 24 may be a mechanical switch (so-called steer switch) provided on the spoke portion of the steering wheel, or may be a voice input device that recognizes the utterance content of the driver. Further, the input device 24 may be a touch panel laminated on the display panel of the display provided on the instrument panel, for example, the center display 33. Further, the input device 24 may be a driver's smartphone. For example, the touch panel and display of the smartphone possessed by the driver can be used as the input device 24.

The child camera 25 is a camera that captures the face of a child sitting on a child seat. For example, the child camera 25 is attached to the back of the front seat, which is located in front of the child's seat. The child camera 25 may be arranged on the ceiling or the like in the vehicle interior so that the face of the child sitting in the rear seat can be imaged. A camera that includes the entire rear seat in the imaging range may be used as the child camera 25. For example, the child camera 25 may be attached to the central portion of the ceiling portion, the upper end portion of the windshield, the overhead console, or the like. The image data captured by the child camera 25 is output toward the HCU 1. It should be noted that the concept of image data here can also include a video signal. The child camera 25 corresponds to a vehicle interior camera.

The child sensor 26 is a biological sensor that senses biological information of an occupant (that is, a child) sitting on a child's seat. Like the driver sensor 22, the child sensor 26 is a sensor that detects at least one of various state quantities such as blood pressure, heart rate, heart rate, sweating amount, and body temperature. For example, the child sensor 26 is a pulse wave sensor that senses a pulse wave. A plurality of types of child sensors 26 having different biological information to be detected may be connected to the HCU 1. The biometric information of a child can include various items as well as the biometric information of a driver.

The child sensor 26 is built in, for example, a child seat. The child sensor 26 may be a non-contact type sensor that acquires various vital information using millimeter waves, infrared rays, or the like. The child sensor 26 may be a thermography camera. In addition, the child sensor 26 may be a wearable device that is worn and used on, for example, a wrist of a child. The wearable device as the child sensor 26 may be configured to be capable of intercommunication with the HCU 1 via a communication device 31 mounted on the vehicle.

The child's microphone 27 is a device that converts the voice spoken by a occupant in the back seat, particularly a child sitting in the child's seat, into an electric signal and inputs it to the HCU1. The children's microphone 27 is, for example, the rear portion of the front seat located in front of the children's seat or the central portion of the vehicle interior ceiling so that the voice of the child sitting on the children's seat can be easily collected. It is placed in such as. The child's microphone 27 may be provided near the headrest of the child seat.

The out-of-vehicle camera 28 is an in-vehicle camera that photographs the surroundings of the own vehicle and outputs the data of the captured image to the HCU 1. The out-of-vehicle camera 28 includes at least a lens and an image sensor, and electronically acquires an image showing the periphery of the own vehicle. The number of external cameras 28 may be one or a plurality.

For example, the front camera, the rear camera, the left side camera, and the right side camera are connected to the HCU 1 as the outside camera 28. The front camera is a camera that captures an image of the front of the vehicle at a predetermined angle of view, and is attached to the front end of the own vehicle such as a front grill. The rear camera is a camera that captures the rear of the vehicle at a predetermined angle of view, and is arranged at a predetermined position on the rear surface of the vehicle body, for example, near the rear license plate or the rear window. The left side camera is a camera that captures the left side of the vehicle and is attached to the left side mirror. The right side camera is a camera that captures the right side of the vehicle and is attached to the right side mirror. A wide-angle lens such as a fisheye lens is adopted as the lens of these out-of-vehicle cameras 28, and each out-of-vehicle camera 28 has an angle of view of 180 degrees or more. Therefore, by using the four cameras 2, it is possible to take a picture of the entire circumference (that is, 360 °) of the own vehicle.

The mounting position of each camera 2 described above can be changed as appropriate. The front camera may be attached to a rearview mirror, the upper end of the windshield, or the like. The left and right side cameras may be arranged near the bases of the A pillar and the B pillar. The external camera 28 may be mounted on the roof or may be mounted on the ceiling in the vehicle interior. The partial or all exterior cameras 28 may be cameras retrofitted, for example, on the roof, on the dashboard, near the window frame of the door for the rear seats, and the like. The out-of-vehicle camera 28 may be a compound-eye camera that includes a plurality of sets of lenses and image pickup elements and can take 360 ° images with one unit. Further, the attention target sharing system Sys may include a plurality of cameras having different imaging distance ranges as the outside camera 28. For example, the attention target sharing system Sys may include a short-distance camera for taking a short-distance image and a telephoto camera for taking a relatively long-distance image as the out-of-vehicle camera 28.

The locator 29 is a device for positioning the current position of the own vehicle. The locator 29 is realized by using, for example, a GNSS receiver, an inertial sensor, and a map database (hereinafter, DB). A GNSS receiver is a device that sequentially (for example, every 100 milliseconds) detects the current position of the GNSS receiver by receiving a navigation signal transmitted from a positioning satellite that constitutes a GNSS (Global Navigation Satellite System). be. The locator 29 sequentially positions the position of its own vehicle by combining the positioning result of the GNSS receiver and the measurement result of the inertial sensor. The positioned vehicle position is output toward HCU1. Further, the locator 29 reads out the map data in a predetermined range determined based on the current position from the map DB and provides it to the HCU 1. The map DB may be stored locally in the vehicle or may be located on the cloud.

The communication device 31 is a device for wirelessly or wired communication with an external device. For example, the communication device 31 carries out data communication with a smartphone 4 or a wearable device brought into the vehicle interior in accordance with a standard such as Bluetooth (registered trademark). The smartphone 4 is mainly a smartphone owned by the driver. Further, the communication device 31 is configured to be capable of performing wireless communication conforming to standards such as LTE (Long Term Evolution) and 4G, 5G, and performs data communication with a predetermined server 5.

The server 5 stores data about an object of interest, which is an object outside the vehicle interior that the child is interested in, as will be described later. Further, the server 5 includes a database and the like in which explanatory information about various objects is registered. The server 5 is configured to be able to transmit explanatory information about the child's object of interest detected by the HCU1 based on the request from the vehicle. The server 5 may have a function of acquiring information on the Internet and returning the information on the object of interest transmitted from the vehicle to the vehicle. The server 5 may be a Web server. The data about the object of interest transmitted from the vehicle may be image data, and in that case, the server 5 analyzes the received image to use text data in which the object of interest is verbalized as a search word. It suffices if it is configured so that it can be acquired.

The meter display 32 is a display arranged in an area located in front of the driver's seat on the instrument panel. As the display, a liquid crystal display, an organic EL display, or the like can be adopted. The center display 33 is a display arranged near the center in the vehicle width direction on the instrument panel. The HUD 34 is a device that projects a virtual image that can be perceived by the user by projecting image light onto a predetermined area of the windshield based on control signals and video data input from the HCU 1 or a navigation device. The HUD 34 displays an image superimposed on the landscape in front of the vehicle. Each of the meter display 32, the center display 33, and the HUD 34 displays an image corresponding to the signal input from the HCU 1. The meter display 32, the center display 33, and the HUD 34 correspond to displays for drivers. Of course, the center display 33 can also be visually recognized by the passenger in the passenger seat. Therefore, the center display 33 can be understood as a display for front seat occupants including the passenger seat occupant.

The rear seat display 35 is a display for passengers in the rear seats, mainly passengers seated in children's seats. The rear seat display 35 is also referred to as a rear monitor. The rear seat display 35 is arranged, for example, on the back portion of the front seat located in front of the child seat, the ceiling portion in the vehicle interior, or the like. The rear seat display 35 also operates based on the input signal from the HCU 1. Here, as an example, the rear seat display 35 is provided with a touch panel, and the rear seat display 35 is configured to be able to receive an instruction operation by a child on the display screen.

The window display 36 is a device that displays an image on a window glass on the side of a vehicle, particularly a side window adjacent to a child's seat, by irradiating the window glass with display light. The window display 36 includes, for example, as shown in FIG. 2, a projector 361 for emitting image light and a screen 362. The screen 362 has a film-like structure for reflecting the image light toward the vehicle interior, and is attached to the surface of the window glass 61 to be irradiated with the image light on the vehicle interior side. A mirror 363 for magnified reflection may be interposed between the projector 361 and the screen 362. The mirror 363 is preferably a concave mirror, but may be a plane mirror. The projector 361 and the mirror 363 are arranged on the surface of the roof portion 62 on the vehicle interior side, that is, on the vehicle interior ceiling portion.

The dialogue device 37 is a device that interacts with various occupants such as a child and a driver sitting on a child's seat. The dialogue device 37 recognizes, for example, the child's voice data acquired by the child's microphone 27, creates an answer to the child's input voice, and outputs the answer by voice. The dialogue device 37 is configured to, for example, use artificial intelligence to recognize the utterance content of the occupant and generate an answer. The process of recognizing the utterance content and the process of generating an answer may be configured to be performed by the server 5 via the communication device 31. The dialogue device 37 may also have a function of displaying an image of a predetermined agent on the rear seat display 35. An agent is, for example, a character such as a fictitious person or an anthropomorphic animal. The agent may be a driver's avatar or the like preset by the driver.

When the dialogue device 37 outputs a predetermined message by voice, the dialogue device 37 can display an animation that moves as if the agent is speaking on the rear seat display 35. The operation of the dialogue device 37, including the display of the animation using the agent, is controlled by the HCU1. Further, the display destination of the animation using the agent is not limited to the rear seat display 35, and may be the window display 36.

The speaker 38 generates sound in the passenger compartment of the vehicle. Types of voice output include voice messages that read a predetermined text, music, and alarms. The expression with voice includes mere sound. The vehicle is provided with a speaker 38A for a driver and a speaker 38B for a child as the speaker 38. The speaker 38A for the driver is provided, for example, on the instrument panel, the headrest of the driver's seat, or the like. The child speaker 38B is built in the child seat. The child speaker 38B may be provided on the side wall portion or the ceiling portion near the child seat.

<About the configuration of HCU1>
The HCU 1 is a computer that comprehensively controls the presentation of information to a user using a display or the like. The HCU 1 is configured as a computer including a processor 11, a RAM (Random Access Memory) 12, a storage 13, a communication interface 14 (I / O in the figure), a bus line connecting these configurations, and the like.

The processor 11 is, for example, an arithmetic core such as a CPU (Central Processing Unit). The processor 11 executes various processes by accessing the RAM 12. The RAM 12 is a volatile memory. The communication interface 14 is a circuit for the HCU 1 to communicate with another device. The communication interface 14 may be realized by using an analog circuit element, an IC, or the like.

The storage 13 has a configuration including a non-volatile storage medium such as a flash memory. In the storage 13, a attention target sharing program, which is a program for making the computer function as the HCU 1, is stored. Executing the attention target sharing program by the processor 11 corresponds to executing the attention target sharing method which is a method corresponding to the attention target sharing program. Further, data indicating the installation position of the external camera 28 in the vehicle, the position of the child's seat, the installation position of the child's camera 25, and the like are registered in the storage 13.

The HCU 1 provides each functional unit shown in FIG. 3 by the processor 11 executing the attention target sharing program stored in the storage 13. That is, the HCU 1 includes a child information acquisition unit F1, an outside vehicle information acquisition unit F2, a driver information acquisition unit F3, and a vehicle information acquisition unit F4. Further, the HCU 1 includes an interest reaction detection unit F5, an object identification unit F6, an explanatory information acquisition unit F7, a driving load estimation unit F8, a notification control unit F9, a recording processing unit FA, and an interest target management unit FB as functional units. .. The notification control unit F9 includes a timing arbitration unit F91 and a target control unit F92 as finer functional units. The notification control unit F9 corresponds to the notification processing unit.

The child information acquisition unit F1 acquires various information regarding the condition of the child sitting on the child seat from the child camera 25 and the child sensor 26. For example, the child information acquisition unit F1 analyzes an image provided by the child camera 25 to determine the direction of the child's face, the direction of the line of sight, the opening of the eyelids, the opening of the pupil, the opening of the mouth, the posture, and the like. Estimate at least part of the body movement. Body movements also include the behavior of pointing a finger or hand out of the window. The child information acquisition unit F1 may be configured to estimate the facial expression, emotion, etc. of the child based on the distribution of facial feature points included in the image from the child camera 25. In addition, it may be configured to detect the presence or absence of utterances in the child based on the time-series change pattern of the distribution of feature points around the mouth. Furthermore, the rhythm of breathing may be estimated from the pattern of changes in body movements, mainly the position of the chest or abdomen.

The child camera 25 may have a function / processing module for estimating the state of the child by analyzing the captured image of the child camera 25. Further, the above functions may be provided in the external server. In that case, the communication device 31 transmits the image data of the child to the server 5, analyzes the received image on the server 5, and returns the result to the vehicle. As such, the various functions for realizing the configuration of the present disclosure may be distributed at the edge and the cloud. The arrangement mode of various functions can be changed as appropriate.

Further, the child information acquisition unit F1 acquires detection results such as pulse wave information from the child sensor 26. Of course, the child sensor 26 can detect pulse rate, blood pressure, electrocardiogram, heart rate, sweating amount, body temperature, heat dissipation amount, breathing rhythm, breathing depth, exhalation component, body composition, posture, body movement, etc. When configured, the child information acquisition unit F1 can also acquire such information.

The child status information acquired by the child information acquisition unit F1 is stored in the RAM 12 with a time stamp indicating the acquisition time. The information on the state amount of the child acquired by the child information acquisition unit F1 is classified for each information type and is stored in, for example, the RAM 12 for a certain period of time. Data with different acquisition times can be sorted and saved in order of acquisition time so that the data with the latest acquisition time comes first. The data retention period can be, for example, 2 minutes or 5 minutes.

In addition, the child information acquisition unit F1 calculates the normal values of the pulse rate, the opening of the eyes, the heart rate, the body temperature, the skin surface potential, etc., based on the detection results of the latest predetermined time. The normal value can be, for example, the average value or the median value of the observed values within the latest predetermined time. For example, the child information acquisition unit F1 calculates the normal value of each state quantity by averaging the observed values within the last 1 minute for each of the pulse rate, the opening of the eyes, and the direction of the face. These normal values can be used as criteria for detecting that a child is interested in something and becomes excited.

In addition, when the child speaks, the child information acquisition unit F1 detects that the child has spoken based on the input signal from the child microphone 27. Further, more preferably, when the child utters some words, the child information acquisition unit F1 acquires information indicating the utterance content and the loudness of the voice. The content of the utterance can be specified by voice recognition processing. Various information acquired or detected by the child information acquisition unit F1 is used by the interest reaction detection unit F5 and the object identification unit F6.

The out-of-vehicle information acquisition unit F2 is configured to acquire out-of-vehicle information from an out-of-vehicle camera 28, a locator 29, or the like. For example, the vehicle exterior information acquisition unit F2 sequentially acquires image data captured by the vehicle exterior camera 28 and temporarily stores the image data in a RAM 12 or the like. The image data storage area can be configured as a ring buffer. That is, when the conserved quantity reaches a certain upper limit, the oldest data is sequentially deleted and new data is saved.

The vehicle outside information acquisition unit F2 stores the image data input from the vehicle outside camera 28 in the RAM 12 in association with the vehicle position information and the time information at the time of shooting. The vehicle exterior information acquisition unit F2 may be configured to specify the position and type of the subject included in the image by analyzing the image input from the vehicle exterior camera 28. For example, CNN (Convolutional Neural Network) or DNN (Deep Neural Network) technology to which deep learning is applied can be used for identifying an object.

In addition, the out-of-vehicle information acquisition unit F2 acquires map information around the vehicle from the locator 29. That is, information such as facilities and landmarks existing around the vehicle is acquired. The map information also includes information related to the road structure such as the type of road currently being traveled. The type of road indicates, for example, whether it is a general road or an expressway. In addition, the information related to the road structure includes the curvature of the road, the branch / confluence points existing in front of the vehicle, the intersection, the remaining distance to the traffic light, and the like.

The driver information acquisition unit F3 acquires various information regarding the driver status from the DSM 21 and the driver sensor 22. The driver information acquisition unit F3 corresponds to the driver's seat occupant information acquisition unit. For example, the driver information acquisition unit F3 acquires at least a part of the driver's face direction, line-of-sight direction, eyelid opening degree, pupil opening degree, mouth opening degree, posture, and the like from DSM21. Further, when the DSM 21 is configured so that the facial expression of the driver, the degree of tension, and the like can be estimated, the driver information acquisition unit F3 can also acquire such information. In addition, it may be configured to detect the presence or absence of utterance of the driver based on the time-series change pattern of the distribution of the feature points around the mouth. In addition, the driver's respiratory rhythm may be estimated from body movements, primarily patterns of changes in chest or abdominal position. The rhythm of breathing can be an estimate of the degree of tension and driving load. As described above, the function included in the DSM 21, that is, the function / processing module for estimating the state of the driver by analyzing the image of the driver may be provided in the driver information acquisition unit F3 or the server 5.

The vehicle information acquisition unit F4 also acquires the state (on / off) of the driving power source, vehicle speed, acceleration, steering angle, accelerator depression amount, brake depression amount, and the like. The traveling power source is a power source for the vehicle to travel, and refers to an ignition power source when the vehicle is a gasoline-powered vehicle. When the vehicle is an electric vehicle such as an electric vehicle or a hybrid vehicle, the driving power source refers to the system main relay. Further, the vehicle information acquisition unit F4 acquires information indicating the current position of the vehicle from the locator 29. The position information of the own vehicle can be expressed by latitude, longitude, altitude, and the like.

The interest reaction detection unit F5 determines whether or not the child is interested in something outside the vehicle interior based on the child status information acquired by the child information acquisition unit F1. The fact that the child is interested in something outside the car room is also described here as an interest reaction. Determining that an interest reaction has occurred corresponds to detecting an interest reaction. In addition to stationary objects such as buildings and signboards, various things such as moving objects such as pedestrians and animals, and scenery are assumed to be of interest to children. The expression of things here includes not only things such as stationary objects and moving objects, but also situations, states, landscapes, experiences, and the like.

For example, the interest reaction detection unit F5 determines that there is an interest reaction when the child sitting on the child's seat keeps looking at the same thing for a predetermined time or more. If the child keeps looking at the same thing for a predetermined time or more, the line-of-sight direction is constant for a predetermined time, and the line-of-sight direction is opposite to the traveling direction of the vehicle (for example, rearward) so as to follow the object of interest with the eyes. ) Can also be included.

Further, the interest reaction detection unit F5 may determine that there is an interest reaction when the child has a facial expression expressing a specific emotion. The specific emotion here includes admiration, marvel, and more specifically, surprise, admiration, smile, and the like. Further, the interest reaction detection unit F5 may determine that there is an interest reaction when it detects that the child has spoken with his / her face turned to the outside of the vehicle interior. At that time, the height of interest may be evaluated according to the loudness of the child's voice. Further, it may be determined that the child shows a high interest reaction based on the fact that the same word is repeatedly spoken a predetermined number of times or more with the child facing the outside of the vehicle interior.

In addition, it may be judged that there was an interest reaction based on the child's phrase such as "look" or "what is that?" That asks the driver. In addition, an interest reaction may be detected based on the onomatopoeia, mimetic word, or onomatopoeia (so-called onomatopoeia) indicating a specific animal or state. In addition, it may be determined that the child has an interest reaction based on the action of pointing a finger or hand out of the window, that is, the action of pointing at something.

In addition, the interest reaction detection unit F5 may detect the child's interest reaction based on biological information such as the child's pulse, heartbeat, and eye opening. For example, it may be determined that there is an interest reaction based on the fact that the pulse becomes faster than the normal value by a predetermined threshold value or more. Further, it may be determined that there is an interest reaction based on the fact that the opening degree of the eyes is increased by a predetermined value or more from the normal value. As a material for detecting an interest reaction, changes in body temperature, respiratory conditions, body composition, and the like can also be used. The state of respiration includes the speed of respiration, the exhaled breath component, and the like. In this way, the interest reaction detection unit F5 may detect that the child is in an unusual state or is in an excited state and is interested in something.

The object identification unit F6 is an object in which the child shows interest based on the line-of-sight direction of the child within the most recent predetermined retroactive time from that time when the interest reaction detection unit F5 detects the child's interest reaction. Identify the object of interest. The concept of the object of interest here includes facilities, landmarks, stationary objects such as signs, moving objects such as pedestrians, animals, situations (events), landscapes, and the like. For example, unfamiliar buildings, people wearing whimsy, emergency vehicles such as dogs, cats, and fire trucks, commercial signs with characters popular with children, and corporate signs can be of interest. The object identification unit F6 corresponds to the object detection unit of interest.

The object identification unit F6 identifies a subject existing in the line-of-sight direction of the child as an object of interest in the image captured by the external camera 28, for example, at the time when the interest reaction is detected or within the retroactive time. The retroactive time here can be 200 milliseconds, 500 milliseconds, 1 second, or the like.

The direction of the child's line of sight with respect to the outside of the vehicle interior is based on the position of the child's eyes in the vehicle interior and the direction of the line of sight starting from the position of the child's eyes, which is specified based on the image of the child camera 25. Can be calculated. The position of the child's eyes in the vehicle interior can be calculated based on the installation position and posture of the child camera 25 and the position of the child's eyes in the image. Further, by combining information such as the azimuth angle at which the vehicle body is facing with the direction of the child's line of sight with respect to the outside of the vehicle interior, the absolute direction in which the child is looking can be calculated. The absolute direction here is a direction corresponding to a predetermined azimuth angle such as north, south, east, and west.

By collating the image of the outside camera 28 acquired and saved at the time when the interest reaction is detected or immediately before that with the line-of-sight direction of the child, the object that the child is paying attention to is automatically identified. Can be done.

Here, as an example, the object identification unit F6 acquires the object of interest as an image. Of course, as another aspect, the object specifying unit F6 may specify the type and name of the object of interest by analyzing the image of the object of interest, which is an image of the subject as the specified object of interest. .. The attention target image, which is an image of the attention target acquired by the object identification unit F6, is output to the explanatory information acquisition unit F7. The object specifying unit F6 also specifies the direction in which the object of interest exists when viewed from the vehicle or the driver's seat in the process of specifying the object of interest. Information about the direction in which the object of interest exists is also output to the explanatory information acquisition unit F7 and the notification control unit F9.

The explanatory information acquisition unit F7 acquires explanatory information about the object of interest specified by the object specifying unit F6 from the dictionary database arranged inside or outside the vehicle. The dictionary database is a database in which explanatory information about various objects is registered. The dictionary database may be installed in the vehicle or may be located in the cloud. The dictionary database may be a Web server or the like. The explanatory information acquisition unit F7 corresponds to the target information acquisition unit.

For example, the explanatory information acquisition unit F7 acquires explanatory information about the object of interest from the server 5 by transmitting an image of the object of interest to the server 5. The explanatory information here includes at least one such as a target category (major classification), a role, a background, and similar things, in addition to the target name. For example, if the object of interest is a building, its proper name / general name, role, height, year of construction, etc. are included in the explanatory information. In addition, if the object of interest is an animal, the explanatory information includes a large classification such as dogs, cats, birds, etc., and more detailed names, sizes, places of origin, characteristics (characters), and the like. If the object of interest is an event such as a festival, the explanatory information includes the name, historical background, and time of the event. When the object of interest is a natural phenomenon such as a rainbow, its name, generation principle, etc. can be explanatory information. When the object of interest is a vehicle such as an ambulance or a construction vehicle, its name, services, features, etc. are included in the explanatory information. When the object of interest is a signboard of a company or shop, the explanatory information may include the type of service provided, the year of establishment, the representative product, etc., in addition to the name of the company / shop. The explanatory information can be mainly information published in pictorial books, dictionaries, guidebooks, and the like.

The server 5 identifies the name of the object of interest by analyzing the image of the object of interest based on the inquiry from the explanatory information acquisition unit F7, and also acquires information other than the name related to the object of interest. Information other than the name may be acquired by searching the Internet using the name as a search key, or by referring to the dictionary database owned by the server 5 itself. Then, the server 5 returns the information collected using the name of the object of interest as the search key to the vehicle as explanatory information. According to such a configuration, the explanatory information acquisition unit F7 can acquire information about various objects of interest even if the vehicle does not have a database containing a huge amount of data.

The explanatory information acquisition unit F7 may specify the type and name of the object of interest by analyzing the image of the object of interest. In that case, the name of the object of interest may be used as a search key, and supplementary information such as the service and background of the object of interest may be acquired from a database arranged inside or outside the vehicle. The information acquired by the explanatory information acquisition unit F7 is temporarily stored in the RAM 12 or the like in association with the object image. The data set is referred to by the notification control unit F9 and the recording processing unit FA.

The driving load estimation unit F8 determines whether or not the driver's driving load is high based on at least one of the driver's state information, the driving environment of the vehicle, and whether or not automatic driving is in progress. As the driver status information, the information acquired / detected by the driver information acquisition unit F3 is used. As the traveling environment information, the information around the vehicle acquired by the vehicle information acquisition unit F4 or the outside information acquisition unit F2 can be used. Whether it is in automatic driving or not can be input from the automatic driving device via the in-vehicle network Nw. A signal indicating the level of automatic driving, that is, any of levels 0 to 5, may be input from the automatic driving device.

The driving load estimation unit F8 determines that the driving load is not high, for example, when the automatic operation of level 3 or higher is in progress and the remaining time until the handover is equal to or longer than a predetermined threshold value. The handover corresponds to transferring the authority of the driving operation from the system to the driver's seat occupant. On the other hand, even during automatic operation, it can be determined that the driver's driving load is high when the system requests the driver to perform a handover.

In addition, the driving load estimation unit F8 determines whether or not the driving load is high based on biological information such as the driver's pulse, breathing interval and depth, skin electrical activity, facial muscle action potential, and peripheral blood flow. You may. For example, when the driver is in a tense state, such as when the pulse or respiration is faster than the normal value by a predetermined value or more, it can be determined that the driving load is high. The tension state of the driver can be estimated from various biological information such as the strength of gripping the handle, the posture, the electrical activity of the skin, the interval between blinks, and the peripheral blood flow. Various methods can be used to determine the tension state and driving load.

In addition, the driving load estimation unit F8 may determine that the driving load is high based on the fact that the vehicle is traveling near a branch / confluence point or an intersection of an expressway or that the lane is being changed. The vicinity of the intersection can be, for example, a section where the remaining distance to the intersection is within 50 m. The vicinity of the intersection also includes the inside of the intersection. In addition, it may be determined that the driving load is high based on the fact that the vehicle is traveling at an intersection with poor visibility, a road with many pedestrians jumping out, or a section where traffic accidents occur frequently. Information such as intersections with poor visibility, roads with many pedestrians popping out, and sections with frequent traffic accidents can be acquired from the locator 29 or the map server as map data.

Whether or not you are driving near a branch / confluence or intersection of an expressway, whether or not you are planning to change lanes, etc. correspond to indicators of the safety level of the driving environment. The above configuration corresponds to a configuration in which the driver's driving load is evaluated according to the safety level of the driving environment, in other words, the magnitude of the potential risk determined according to the driving environment. Various information / conditions can be adopted as the judgment material / judgment condition for judging that the operating load is high.

The notification control unit F9 is configured to control the notification related to the information about the object of interest acquired by the explanatory information acquisition unit F7. For example, the notification control unit F9 determines the determination result of the driving load estimation unit F8 regarding the person / seat to be notified, the display timing of the image related to the object of interest, the timing of voice output, the output destination when displaying the image, and the like. Control in an integrated manner based on. The functional unit that arbitrates the timing of displaying the image and the audio output related to the object of interest corresponds to the timing arbitration unit F91. In addition, adjusting the display destination and audio output destination of the image related to the object of interest corresponds to selecting the occupant to be notified of the information related to the object of interest. Excluding the display for the driver such as the meter display 32 and the HUD 34 as the display destination of the image related to the object of interest is equivalent to excluding the driver from the display target of the image information. The functional unit that adjusts the display destination and audio output destination of the image related to the object of interest corresponds to the target control unit F92. As one control mode, the notification control unit F9 is configured to display an icon image indicating that the child has shown an interest reaction on the HUD352 when the interest reaction of the child is detected by the interest reaction detection unit F5. May be. The details of the notification control unit F9 will be described later.

The recording processing unit FA stores the data about the object of interest acquired by the explanatory information acquisition unit F7 in a predetermined storage medium in association with the position information at the time when the interest reaction is shown. The device (so-called storage device) to store the data may be an external device such as the driver's smartphone 4 or the server 5, or an internal device such as the storage 13. The recorded data, which is the data to be recorded, preferably includes the image data of the object of interest. The image data included in the recorded data may be a still image or a video. Further, the recorded data may include text data of explanatory information determined by analyzing the image. In the recorded data, not only the position information of the vehicle at the time when the interest reaction is detected, but also the direction in which the object of interest exists, the position information of the object of interest, and the detection time information are stored in association with each other. good. Further, the recorded data may include image data of a child when looking at an object of interest. In addition, the recorded data may include voice data in the vehicle within a predetermined time determined based on the time when the interest reaction is detected.

When the storage destination of the above data is the server 5, it is preferable that the data stored in the server 5 can be referred to from a smartphone or the like owned by a driver, grandparents, or the like. This configuration allows children to share things that interest them with their families living apart. When the data storage triggered by the detection of the interest reaction occurs, the HCU 1 or the server 5 may notify the device registered in advance that the recorded data has been updated. The above sharing process may be realized by cooperation with a social networking service (SNS: Social Networking Service).

As described above, the recording processing unit FA saves the data about the object of interest in association with various information, so that the data of the object to be recorded can be referred to later. As a result, parents are more likely to talk to their child about what they have seen, for example during a driving break or after returning home, referring to recorded data about the object of interest. It also has the advantage that parents can feel the growth of their child by seeing the things that the child has paid attention to later.

The interest target management unit FB is configured to specify the interest target category, which is the category of things that the child is interested in, based on the information such as the type of the attention object detected in the past. Vehicles, animals, buildings, signboards, plants, characters, fashion, etc. are assumed as the categories of interest. Characters may be classified in more detail by the name of the animation or the like. Vehicles may also be subdivided into four-wheeled vehicles, two-wheeled vehicles, trains, and the like. Fashion may also be subdivided into clothes and hairstyles. Animals may be subdivided into dogs, cats, birds and the like.

The HCU1 detects the child's interest reaction when the interest category, which is the category in which the child is interested, is configured to be identifiable, and when things belonging to the interest category are captured by the outside camera 28. The threshold value of may be adjusted. For example, when things belonging to the interest category are captured by the outside camera 28, the threshold value for detecting the interest reaction of the child may be lowered.

<About the operation example of HCU1>
Here, the communication support process executed by the HCU 1 will be described with reference to FIG. Here, as an example, the communication support process includes steps S101 to S115. Of course, the number of steps constituting the communication support process and the process order can be changed as appropriate.

The communication support process shown in FIG. 4 is started when a predetermined start event occurs. As the start event, for example, the ignition of the vehicle is turned on, the running of the vehicle is started, the start instruction by the driver is input, and the like can be adopted. In addition, the communication support process may be started with the detection of the interest reaction of the child sitting on the child's seat as a trigger by the interest reaction detection unit F5. In that case, steps S101 to S104 can be sequentially executed as a process independent of the communication support process.

First, in step S101, HCU1 acquires information necessary for processing from various devices connected to HCU1 and moves to step S102. For example, the child information acquisition unit F1 acquires the biological information of the child from the child sensor 26 or the like. The out-of-vehicle information acquisition unit F2 acquires information on the environment outside the vehicle from the out-of-vehicle camera 28 and the like. The driver information acquisition unit F3 acquires the biological information of the driver from the DSM21, the driver sensor 22, and the like. Further, the vehicle information acquisition unit F4 acquires information on the current position and traveling speed of the vehicle from the in-vehicle network Nw, the locator 29, and the like. The various information acquired in step S101 is stored in a predetermined memory such as the RAM 12 together with the information indicating the acquisition time. Such step S101 can be called an information acquisition step and an information acquisition process. It should be noted that the information acquisition step can be sequentially executed in a predetermined cycle such as 100 milliseconds, 200 milliseconds, and 500 milliseconds even after step S102.

In step S102, the driving load estimation unit F8 determines whether or not the driver's driving load is high based on the driver's biological information acquired by the driver information acquisition unit F3, for example, by the above-mentioned algorithm, and in step S103. Move. The determination result relating to the driver's operating load may be expressed by a level value of a plurality of stages. In that case, the state in which the determination value is equal to or higher than the predetermined threshold value corresponds to the state in which the operating load is high. Further, whether or not the driver's operating load is high may be managed by a flag. For example, when the operating load is high, the operating load flag may be set to 1 (on), while when the operating load is not high, the operating load flag may be set to 0 (off). When step S102 is completed, the process proceeds to step S103. Such step S102 can be called a driver state estimation step.

In step S103, the interest reaction detection unit F5 determines whether or not the child has shown an interest reaction based on the biological information of the child within the latest predetermined time acquired by the child information acquisition unit F1. If there is an interest reaction, affirmative determination is made in step S104, and the process proceeds to step S105. On the other hand, if there is no interest reaction, it is determined in step S115 whether or not the predetermined end condition is satisfied. The processing of steps S103 to S104 can be referred to as an interest reaction detection step.

In step S105, the object identification unit F6 estimates the direction in which the object of interest is present when viewed from the vehicle, based on the direction of the child's line of sight and the position of the child's eyes. Then, from the subject included in the image data of the external camera 28, the thing existing in the estimation direction of the object of interest is extracted as the object of interest, and the process proceeds to step S106. Such step S105 can be referred to as an object specifying step of interest.

In step S106, the object identification unit F6 extracts the image portion in which the object of interest is captured from the image captured by the camera outside the vehicle 28 as the image of the object of interest, and moves to step S107. Such step S106 can be called an object image acquisition step of interest.

In step S107, the explanatory information acquisition unit F7 cooperates with the communication device 31 to access the server 5 or the like, the Internet, or the like to acquire explanatory information about the object of interest and move to step S108. The explanatory information is data in which the features, services, names, etc. of the object of interest are verbalized by text or voice data. Such step S107 can be called an explanatory information acquisition step.

In step S108, it is determined whether or not it is determined that the operating load is high. As the information related to the operating load used in step S108, the determination result in step S102 can be diverted. If the operating load is high, step S108 is positively determined and the process proceeds to step S109. On the other hand, if the operating load is not high, step S108 is negatively determined and the process proceeds to step S111. As the operation mode of the HCU 1, a proxy response mode in which the system automatically responds instead of the driver may be provided. When the HCU1 is set to the proxy response mode by the operation of the driver, it may be configured to move to step S109 regardless of the determination value of the operating load.

In step S109, the notification control unit F9 excludes the driver from the notification target of the information related to the object of interest. For example, the notification target of the information related to the object of interest is set only for children. If an occupant is also in the passenger seat, the notification target of the information related to the object of interest may be set to the child and the passenger seat occupant. Whether or not a person is in the passenger seat can be specified from the detection signal of the seating sensor provided in the passenger seat.

The information related to the object of interest here is an image or explanatory information of the object of interest. As described above, the explanatory information may be output as an image such as text or an icon, or may be output as a voice message. The image of the object of interest includes not only an image of the object of interest itself, but also a text image of explanatory information. Information about the object of interest can be realized using at least one of image display and audio output.

It should be noted that including the child as the notification target corresponds to adopting at least one of the rear seat display 35, the window display 36, and the speaker 38B as the notification device. The notification device here refers to a device that outputs information related to an object of interest as an image or voice. Further, including the driver in the notification target corresponds to adopting at least one of the meter display 32, the center display 33, the HUD 34, and the speaker 38A as the notification device. Including the passenger seat occupant in the notification target corresponds to adopting at least one of the center display 33 and the speaker 38A as the notification device. When the attention target sharing system Sys is provided with a passenger seat display which is a display provided in front of the passenger seat, the passenger seat display can also be adopted as a notification device for the passenger seat occupant. The passenger display may be part of a display that is continuously provided from the right edge to the left edge of the instrument panel.

Of course, the notification target for image display and the notification target for voice output may be set separately. The child may be notified by both image and voice, while the driver may be controlled to notify only by voice. When the selection process of the notification target in step S109 is completed, the process proceeds to step S110.

In step S110, the notification control unit F9 displays an image of the object of interest and explanatory information thereof on at least one of the rear seat display 35 and the window display 36, and outputs the voice corresponding to the explanatory information from the speaker 38B. .. If the content of the interest reaction detected in step S104 is of the type in which the child asks the driver about the object of interest, the system response as step S110 is a process in which the system responds on behalf of the driver. Corresponds to a certain proxy response process. The proxy response process can also be understood as a process of controlling the operation of the dialogue device 37 in one aspect.

When the driver's driving load is high in this way, the system responds promptly on behalf of the driver to provide explanatory information about the object of interest before the child loses interest in the object of interest. can. In addition, as a driver, the need to immediately respond to a child's question or the like is reduced, so that the driver can concentrate on the driving operation.

Note that in step S110, the dialogue device 37 may be activated to build a system state capable of responding to additional questions of the child. With such a configuration, the system will continue to be able to respond to children's questions about the presented explanatory information. The interaction between the dialogue device 37 and the child can be performed using agent images. When step S110 is completed, the process proceeds to step S114.

In step S111, information about the object of interest is displayed on the display for the driver's seat and the display for children, and the process proceeds to step S112. When displaying the image on various displays, a predetermined sound effect may be output from each speaker 38 so that the driver or the child can easily notice that the image is displayed.

In step S112, it is determined whether or not the driver has spoken within a predetermined response waiting time from the image display in step S111. The step S112 corresponds to a process of determining whether or not the driver has made any response to the child's interest reaction. The response waiting time can be, for example, 4 seconds or 6 seconds. If the driver's utterance is detected within the response waiting time, it is considered that the driver has responded to the child's interest reaction, the voice output is omitted, and the process proceeds to step S114. On the other hand, if the driver's utterance is not detected even after the response waiting time has elapsed from the start of the image display in step S111, it is considered that the driver has difficulty in responding to the interest reaction of the child, and the process proceeds to step S113. .. In step S113, explanatory information about the object of interest is output by voice and the process proceeds to step S114. Steps S110, S111, and S113 can be called a notification processing step because they present at least one of the driver and the child with verbalized information about the object of interest.

In step S114, the recording processing unit FA saves the image data of the object of interest in a predetermined recording device in association with the position information, the voice data in the vehicle interior, the time information, and the like, and moves to step S115.

In step S115, it is determined whether or not the predetermined end condition is satisfied. The termination conditions include, for example, that the vehicle's running power has been turned off, that the vehicle has arrived at the destination, that the driver has given an termination instruction, and the like. If the end condition is satisfied (S115 YES), this process ends.

<Display example of notification image about the object of interest>
Here, an example of a display image showing information about an object of interest will be described with reference to FIGS. 5 to 8.

In the notification image about the object of interest provided to the driver, the object of interest, such as the direction in which the object identification unit F6 has detected the object of interest and whether or not the object of interest can still be captured by the external camera 28. It is preferable to include the target position information image Pd indicating the position of. The target position information image Pd may be composed of text, an icon, or the like. By presenting the target position information image Pd, the driver can easily see and confirm the object of interest with his / her own eyes. Further, by presenting information indicating whether or not the image can be taken by the external camera 28, it is possible to suppress the search for an object that has already disappeared due to the movement of at least one of the vehicle and the object of interest.

FIG. 5 is a diagram simulating the notification image Px when the object of interest is, for example, a dog walking on a sidewalk. As shown in FIG. 5, the notification image Px when the object of interest is a dog includes a text image Pt such as a name (breed), a country of origin, and a character, in addition to the image Pi of the object of interest itself. Further, FIG. 6 is a diagram simulating the notification image Px when the object of interest is another vehicle traveling in the adjacent lane. As shown in FIG. 6, the notification image Px when the object of interest is a car is a text image Pt of the name (vehicle type), the manufacturer (so-called manufacturer), features, etc., in addition to the image Pi of the object of interest itself. including. The characteristics of a vehicle refer to whether it is an electric vehicle or an engine vehicle, size, running performance, and the like.

As shown in FIG. 7, the notification image Px when the object of interest is a building such as a tower shows the name, role, year of construction, height, etc. of the building in addition to the image Pi of the object of interest itself. Includes text image Pt. The role of a building includes, for example, a radio tower that transmits broadcast waves such as television and radio, a commercial complex, government offices, factories, schools, and houses. Roles can also be rephrased as provided services and attributes. The notification image Px of the building as a landmark may display a photograph from another angle or an external photograph Pj at a time zone different from the present, such as at night, in a touch-selectable manner.

FIG. 8 is a diagram simulating the notification image Px when the object of interest is a rainbow. As shown in FIG. 8, the notification image Px when the object of interest is a natural phenomenon such as a rainbow includes, in addition to the image Pi of the object of interest itself, a text image Pt showing the reason for occurrence, features, and the like. The characteristic of the rainbow includes information about the color gradation, for example, the outermost colors are red and purple.

In addition, the notification control unit F9 may display the video content related to the object of interest or a list thereof as the notification image Px. For example, when the object of interest is a character of a certain animation, the content such as a moving image of the animation may be displayed.

<Effect>
In the above configuration, the HCU 1 presents to the driver not only the image of the object of interest that the child is interested in, but also explanatory information in which the name of the object of interest is verbalized. With such a configuration, the driver can easily recognize what the child has focused on. As a result, the driver is more likely to respond closely to the child's reaction to the child's interest reaction.

Further, as an example in the above, when it is determined that the driver's driving load is high, the system answers the child's question or the like without waiting for the driver's response to the child's interest reaction. According to this configuration, the driver tends to concentrate on driving. Also, the child can get the information he wants to know from the system. In addition, the system responds to what the child is interested in, reducing the risk of traveling time being boring.

Further, in the above configuration, as an example, even when the driver is notified of the information about the object of interest by image and voice, the explanatory information is based on the elapse of a predetermined response waiting time after displaying the image. Audio output. By providing a time difference between the image display and the audio output in this way, the guardian himself as a driver can easily respond to the interest reaction of the child. This can be expected to have the effect of activating communication between the guardian as a driver and the child. Further, in the above configuration, if the driver's utterance cannot be detected even after a predetermined response waiting time has elapsed after displaying the image on the driver's display, the system responds on behalf of the driver. According to the above configuration, the child can easily enjoy the traveling time because at least one of the guardian and the system responds to his / her interest.

In addition, in the above configuration, things that show the child's interest are recorded in association with location information and the like. According to this, what the child is interested in can be specified later even if the driver cannot be specified immediately after the interest reaction is detected. For example, even after passing the object of interest, by looking at the image of the object of interest, it becomes possible to identify things that the child is interested in. Further, for example, it is possible to look back at the image of the object of interest even after the operation is completed or after returning home. In addition, the data of the object of interest is configured so that the family living apart in cooperation with the server 5 or the like can also refer to it. Therefore, it becomes easier for family members and relatives other than the driver to share things that the child is interested in.

In the above configuration, as an example, the driver is notified of the type of the object of interest and its existence direction by image or voice. According to this configuration, the driver can easily see the object that the child is paying attention to outside the vehicle interior even while driving. Further, according to the above configuration, a guardian such as a driver can easily notice the growth of the child from the change of the object of interest to the child. Therefore, according to the above configuration, the travel time can also be used as an opportunity for parents to know the growth of their children.

Furthermore, things that are detected as objects of interest are not limited to map elements such as facilities and landmarks that are registered in map data. Various objects such as pedestrians, automobiles, trains, animals, and rainbows are detected as objects of interest. Then, the explanatory information about the detected object of interest is acquired from the Internet or the like and presented to at least one of the child and the driver. With such a configuration, the child can actively obtain information about various things that he / she is interested in, so that he / she can learn various things during the traveling time.

In addition, according to the above configuration, the child can learn by specifically linking the knowledge on the database with the real world. The knowledge and memories that you actually get with your own eyes are easier to remember than the information that you get from textbooks. Therefore, according to the above configuration, it is possible to efficiently acquire knowledge by using the travel time to a cram school or school. In other words, travel time can be used as an opportunity to acquire knowledge. In particular, when a child attends multiple cram schools, it may be important for the child and the guardian who accompanies the child to make effective use of the travel time. In response to such circumstances, the configuration of the present disclosure has an advantage that the travel time can be easily utilized as an opportunity for knowledge acquisition and communication between parents and children.

In addition, in the above configuration, as an example, the image of the object of interest and the voice data showing the conversation in the car are also saved. According to this configuration, the conversation in the car about the object of interest can be reproduced at a later date.

According to the above configuration, by supporting communication between the child and the guardian as a driver while driving, it is possible to make the traveling time in the vehicle more enjoyable. Also, small children can sometimes yell or cry for lack of parental response. When a child has a tantrum, the driver's driving load can increase as a result. According to the configuration of the present disclosure, by facilitating communication between the child and the guardian while driving, the risk of the child causing tantrum can be reduced, and as a result, the effect of suppressing an increase in the driving load can be expected.

Although the embodiments of the present disclosure have been described above, the present disclosure is not limited to the above-described embodiments, and various supplements and modifications described below are also included in the technical scope of the present disclosure, and further described below. In addition to this, various changes can be made within the range that does not deviate from the gist. For example, various modifications can be appropriately combined and carried out within a range that does not cause a technical contradiction. The members having the same functions as those described in the above-described embodiment are designated by the same reference numerals, and the description thereof will be omitted. Further, when only a part of the configuration is referred to, the configuration of the embodiment described above can be applied to the other parts.

<Example of controlling the content of notification to children>
The amount, type, and expression method of information to be notified to the child may be changed based on the child's age, knowledge level, and ability level. For example, if the child is still unable to read the characters in his / her mother tongue, the text and voice in his / her mother tongue may be output as a set. According to this structure, the effect of accelerating the acquisition of characters in the native language can be expected. In addition, if the child is around the age of being able to read the characters in his / her native language to some extent, the text in his / her native language and the translated text in another language may be output as a set. According to this structure, the effect of learning a language other than the mother tongue can be expected.

In addition, the amount, type, expression method, etc. of information to be notified to the child may be changed based on the arousal level, posture, boarding time, etc. of the child while riding. When the child is sleepy, the amount of information may be less than when the child is awake. Alternatively, the amount of characters may be reduced to increase the number of images. According to this configuration, it is possible to reduce the risk of causing annoyance to the child due to excessive information. Also, if the child's riding posture is out of order, it is highly likely that the child is tired. Therefore, when the riding posture of the child is broken, the amount of information to be presented may be smaller than that when the riding posture of the child is not broken. In addition, the longer the elapsed time (that is, the boarding time) after boarding, the more information may be required to be reduced. In addition, in the configuration of presenting the list of video contents related to the object of interest to the child, the content is configured to be presented to the child only for the content whose playback time is shorter than the remaining time until the destination is reached. Is also good.

The amount and type of information to be notified to the child via the rear seat display 35 or the like may be configured so that the driver can set it via a predetermined setting screen. Information such as the age of the child may be manually input by the driver via the setting screen or the like, or may be estimated by the HCU 1 based on an image of the child camera 25 or the like.

In addition, the amount of information of the content of notification to the child depends on whether or not the interest reaction is detected based on the fact that the child utters a compound word phrase that is a phrase that is a combination of multiple words such as "what is that?" You may change such things. Since it can be expected that the compound word phrase can be uttered and the corresponding word is memorized, more detailed explanatory information may be notified. On the other hand, when an interest reaction is detected based on an utterance other than a compound word phrase, the content of the notification may be relatively simple information. In this way, the amount of information notified to the child may be increased or decreased according to the content of the utterance used as the detection trigger of the interest reaction.

Further, the information notified to the child may be controlled depending on whether or not the interest reaction is detected by using the spoken voice of the child. For example, information about an object of interest detected in the absence of a child's utterance may be limited to an image display. This is because there is a relatively high risk of erroneous detection when the child does not speak, and there is a risk of causing trouble to users including children if voice output is performed. According to the above configuration, when the risk of erroneous detection is relatively high, the risk of causing trouble to the user can be reduced by omitting the audio output.

The HCU1 evaluates the degree of attention based on the length of time while gazing at the same object, the content of speech, the degree of excitement indicated by biological information such as heartbeat, and the like, and when the degree of attention is equal to or less than a predetermined threshold value. , The audio output for the object of interest may be omitted. Even with this configuration, it is possible to reduce the risk of voice output of information about things that are not of interest or that are of little interest. Further, as a result, it is possible to reduce the risk of causing trouble to the user.

<Example of controlling the content of notification to the driver>
Even when no interest reaction is detected, the HCU 1 may display status information indicating the child's body temperature, drowsiness, etc. on a display for a driver such as HUD34 at all times or at predetermined notification intervals. Such a configuration makes it easier to recognize the child's appearance, such as whether the child is sleepy or hot. In addition, it becomes easier to make a voice call according to the child's condition.

<Supplement to notification control to the driver>
In the above-described embodiment, a control example is disclosed in which the mode of presenting information about the object of interest to the driver is changed according to the determination result of whether or not the driving load is high, but the mode of presenting information to the driver is switched. The parameters for this are not limited to the height of the operating load. For example, the HCU 1 may change its operation when presenting information about an object of interest to the driver, depending on whether the vehicle is running or not. For example, when the vehicle is stopped, both the image display and the audio output may be performed, while when the vehicle is running, the image display may not be performed and only the audio output may be performed.

HCU1 may change the operation when presenting information about the object of interest to the driver depending on whether or not the automatic operation of level 3 or higher is in progress. For example, when the automatic operation of level 3 or higher is performed, both the image display and the audio output may be performed, while when the automatic operation is not performed, the image display may not be performed and only the audio output may be performed.

The HCU1 may change the combination of image display destinations for the object of interest by using not only the running state of the vehicle but also whether or not another occupant is in the passenger seat. For example, if an occupant is also seated in the passenger seat, information about the object of interest is displayed only on the passenger seat display and the child display during automatic driving, while the driver display is also displayed during automatic driving. An information image may be displayed.

In addition, even if it is determined that the driver's operating load is not high, if the transfer of authority from the system to the driver is near, the presentation of information about the object of interest to the driver is omitted. You may be. If the time remaining until the transfer of authority is less than the predetermined value, by refraining from presenting information to the object of interest to the driver, it becomes easier for the driver to concentrate on preparing to resume the driving operation.

Further, the HCU 1 may change the place where information about the object of interest is displayed and the notification mode according to the driving state of the driver. The HCU 1 may be configured so that the timing of displaying an image can be adjusted based on the driver's situation and the driver's instruction. The driver's instruction operation can be accepted by voice, touch, gesture, or the like. As display timing selection candidates, immediately, 5 minutes later, when the vehicle is temporarily stopped, when the vehicle is parked, when the driving load is reduced, when the automatic operation is started, and the like can be adopted.

<Use of past detection results>
The HCU1 may adjust the threshold value for detecting the interest reaction of the child based on the past detection results of the object of interest. For example, the interest reaction detection unit F5 may lower the threshold value for detecting the interest reaction of a child when traveling near an object that has shown an interest reaction in the past. In addition, the object of interest management unit FB may register an object that has shown an interest reaction a plurality of times in the past as a favorite object that is an object of particular interest. The HCU 1 may notify the child of the existence of the favorite object when the remaining distance to the favorite object is less than a predetermined distance or when the favorite object is imaged by the outside camera 28. good. The HCU1 may be configured to learn an object that the child is interested in based on past detection results.

In addition, when the HCU1 is configured to manage the interest target category and detects that the child is interested in an object belonging to the existing interest target category, a mode different from the normal mode is used. You may notify the driver with. The normal time here refers to the case where the child is interested in an object belonging to an existing interest category. With this configuration, the driver may be able to know that the child has begun to take an interest in new things that are different from the past, in other words, the change of interests as he grows up. Further, according to the above configuration, in one aspect, the driver can know the interest target of the child that the driver himself did not know. In other words, the management of the interest category by the HCU1 can help to know the unexpected side of the child.

<Use of preset children's objects of interest>
In the above, the mode in which the HCU1 learns things that the child is interested in based on the past detection results is mentioned, but the present invention is not limited to this. Information on things and preferences that the child likes via a predetermined setting screen may be registered in advance in the HCU 1 or the server 5.

When data related to things that the child likes is registered by manual registration or automatic learning, the object identification unit F6 may be configured to detect the object of interest based on the data. That is, it may be configured to preferentially extract a subject related to a child's favorite thing as an object of interest from the camera image corresponding to the line-of-sight direction of the child. According to this configuration, it is possible to reduce the possibility that things that the child is not interested in / thin will be extracted as objects of interest. In other words, it is possible to suppress the malfunction of the system. Further, the HCU 1 may change the amount of information to be notified to the child, the expression method, and the like based on the child's preference information. If the object of interest is a child's favorite thing, it may display more / more detailed information than it would otherwise. Also, if the object of interest is a child's favorite thing, an image related to the object of interest is displayed with a notification sound, while if the object of interest is not a thing that the child likes, the object of interest is not a notification sound. An image related to an object may be displayed.

<Supplement of communication method>
The driver and the child may be configured to be able to talk through a camera and a microphone. For example, when the conversation function is set to on, the image of the child camera 25 is displayed on the meter display 32, and the voice acquired by the child microphone 27 is output from the speaker 38A. Further, the image of the DSM 21 is displayed on the rear seat display 35, and the sound acquired by the driver microphone 23 is output from the speaker 38B. According to this configuration, communication is possible even when the driver's seat and the child's seat are separated, for example, when the seat configuration has three or more rows and the child is seated in the third and subsequent rows from the front. It will be easier.

<About controlling the imaging range of a child's camera>
The HCU 1 may be configured so that the image pickup direction and the enlargement ratio of the child camera 25 can be changed based on the voice command of the driver acquired via the driver microphone 23. The imaging direction of the child camera 25 is expressed by a pitch angle, a roll angle, a yaw angle, and the like. Changing the imaging direction of the child camera 25 corresponds to changing the posture angle. The imaging direction of the child camera 25 can be realized, for example, by controlling a motor that controls the posture of the child camera 25.

According to the above configuration, the driver can adjust the imaging range of the child camera 25 by voice, and it becomes easy to check the facial expression of the child. The control of the imaging range of the child camera 25 is not limited to voice input, and may be configured to be executable via an operating member such as a haptic device.

<Addition>
The devices, systems, and methods thereof described herein may be implemented by a dedicated computer constituting a processor programmed to perform one or more functions embodied by a computer program. .. Further, the apparatus and the method thereof described in the present disclosure may be realized by using a dedicated hardware logic circuit. Further, the apparatus and method thereof described in the present disclosure may be realized by one or more dedicated computers configured by a combination of a processor for executing a computer program and one or more hardware logic circuits. Further, the computer program may be stored in a computer-readable non-transitional tangible recording medium as an instruction executed by the computer. That is, the means and / or functions provided by HCU1 and the like can be provided by software recorded in a substantive memory device and a computer, software only, hardware only, or a combination thereof that execute the software. For example, some or all of the functions included in the HCU 1 may be realized as hardware. A mode in which a certain function is realized as hardware includes a mode in which one or more ICs are used. The HCU 1 may be realized by using an MPU, a GPU, or a DFP (Data Flow Processor) instead of the CPU. The HCU 1 may be realized by combining a plurality of types of arithmetic processing devices such as a CPU, an MPU, and a GPU. HCU1 may be realized as a system-on-chip (SoC: System-on-Chip). Further, various processing units may be realized by using FPGA (Field-Programmable Gate Array) or ASIC (Application Specific Integrated Circuit). Various programs may be stored in a non-transitionary tangible storage medium. As a program storage medium, various storage media such as HDD (Hard-disk Drive), SSD (Solid State Drive), flash memory, and SD (Secure Digital) card can be adopted. The non-transitional substantive recording medium also includes a ROM such as EPROM (Erasable Programmable Read Only Memory).

The plurality of functions possessed by one component in the above embodiment may be realized by a plurality of components, or one function possessed by one component may be realized by a plurality of components. Further, a plurality of functions possessed by the plurality of components may be realized by one component, or one function realized by the plurality of components may be realized by one component. In addition, a part of the configuration of the above embodiment may be omitted. Further, at least a part of the configuration of the above embodiment may be added or replaced with the configuration of the other above embodiment.

Various forms such as the system having the above-mentioned attention target sharing device as a component are also included in the scope of the present disclosure. For example, the scope of the present disclosure also includes a program for making a computer function as a shared device of interest, a non-transitional actual recording medium such as a semiconductor memory in which this program is recorded, and the like.

Claims

A shared device of interest used in vehicles equipped with children's seats, which are seats for children to sit on.
As information representing the state of the child sitting on the child's seat, the child information acquisition for acquiring the line-of-sight direction of the child based on the image of the vehicle interior camera (25) including at least the face of the child in the imaging range. Part (F1) and
An interest reaction detection unit (F5) that detects the child's interest reaction to things outside the vehicle interior based on the child's biological information, the child's voice, and at least one of the line-of-sight directions.
Based on the line-of-sight direction acquired by the child information acquisition unit and the captured image of the vehicle outside camera (28) attached to the vehicle so as to image the outside of the vehicle, the child is interested in the object. The attention target detection unit (F6) that detects the attention target, and
A target information acquisition unit (F7) that acquires verbalized information about the attention target from a database arranged inside or outside the vehicle, and a target information acquisition unit (F7).
A notification processing unit that notifies at least one of the driver's seat occupant and the child by using at least one of the display of the text corresponding to the information and the voice output of the information acquired by the target information acquisition unit ( F9) and a shared device of interest.
The shared device of interest according to claim 1.
The target information acquisition unit acquires the type of the attention target as information about the attention target, and obtains the type of the attention target.
The attention target detection unit acquires the direction in which the attention target exists with respect to the vehicle, and obtains the direction in which the attention target exists.
The notification processing unit is an attention target sharing device configured to notify the driver's seat occupant of the type of the attention target and the direction in which the attention target exists.
The shared device of interest according to claim 1 or 2.
The attention target detection unit is configured to be capable of detecting at least one of a pedestrian, another vehicle, an animal, an event, a natural phenomenon, and a picture of a predetermined character as the attention target.
The target information acquisition unit has at least one of the name, characteristic, manufacturer, country of origin, reason for occurrence, size, year of construction, role, and historical background of the target of interest as information about the target of interest. To get,
The notification processing unit is an attention target sharing device configured to display a text corresponding to the information acquired by the target information acquisition unit on a predetermined display together with an image of the attention target.
The shared device of interest according to any one of claims 1 to 3.
It is equipped with an interest target management unit (FB) that manages the detection result of the attention target.
The interest target management unit identifies the interest target category, which is the target category in which the child is interested, based on the history of the detection results of the attention target in the past, and also,
When the newly detected attention target does not belong to the existing interest target category, the newly detected attention target is assigned to the driver's seat occupant in a manner different from the case where the newly detected attention target belongs to the existing interest target category. A focus sharing device that is configured to make notifications.
The shared device of interest according to any one of claims 1 to 4.
A recording processing unit (FA) that records the object of interest in a predetermined storage device in association with location information is provided.
The notification processing unit is configured to notify at least one of the child or the driver's seat occupant of the existence of the attention target based on the approach to the attention target detected in the past within a predetermined distance. Attention target sharing device that has been.
The shared device of interest according to any one of claims 1 to 5.
The driver's seat occupant information acquisition unit (F3) for acquiring information indicating the state of the driver's seat occupant, including the voice emitted by the driver's seat occupant, is provided.
In the notification processing unit, even if a predetermined response waiting time elapses after the interest reaction detection unit detects the child's interest reaction, the driver's seat occupant information acquisition unit still hears the voice of the driver's seat occupant. An attention target sharing device configured to output the information acquired by the target information acquisition unit by voice based on the fact that the information was not acquired.
The shared device of interest according to claim 6.
The driver's seat occupant information acquisition unit is configured to be able to acquire information indicating the driving load of the driver's seat occupant.
The notification processing unit
If the driver's seat occupant information acquisition unit has not acquired information indicating that the driver's seat occupant has a high driving load, the response waiting time may elapse after the interest reaction is detected. On the condition that the driver's seat occupant information acquisition unit does not acquire the voice of the driver's seat occupant, the target information acquisition unit outputs the acquired information by voice.
When the driver's seat occupant information acquisition unit has acquired information indicating that the driver's seat occupant has a high driving load, the target information acquisition unit has acquired the information without waiting for the response waiting time to elapse. An object sharing device of interest configured to output the information by voice from a speaker provided in the vehicle.
The shared device of interest according to claim 7.
Based on the fact that the interest reaction detection unit detects the interest reaction, the notification processing unit allows the driver's seat occupant to visually recognize information about the attention target including the information acquired by the target information acquisition unit. Display on each of the display and the display that can be seen by the child.
When the driver's seat occupant information acquisition unit has not acquired the information indicating that the driver's seat occupant has a high driving load, the response standby time after displaying the information about the attention target on the display. Is configured to output the information acquired by the target information acquisition unit by voice, provided that the driver's seat occupant information acquisition unit does not acquire the voice of the driver's seat occupant even after the lapse of time. Attention target sharing device.
The shared device of interest according to any one of claims 1 to 8.
The notification processing unit notifies the child of the information about the attention target based on at least one of the child's age, preference, elapsed time from boarding, and remaining time to the destination. A focus sharing device that is configured to change.
A focus sharing method performed by at least one processor for parents to share things that a child sitting in a child's seat preset in the vehicle is interested in.
Acquiring the line-of-sight direction of the child based on the image of the vehicle interior camera (25) including at least the face of the child sitting on the child's seat in the imaging range (S101).
Detecting the child's interest reaction to things outside the vehicle interior based on the child's biological information, the child's voice, and at least one of the line-of-sight directions (S104).
Based on the acquired direction of the line of sight of the child and the captured image of the vehicle outside camera (28) attached to the vehicle so as to image the outside of the vehicle, the object of interest that the child is interested in is selected. To detect (S105) and
Obtaining verbalized information about the object of interest from a database arranged inside or outside the vehicle (S107), and
Notifying at least one of the driver's seat occupant and the child by using at least one of the display of the text corresponding to the information and the voice output of the acquired information about the attention target (S110, S111). , S113), and a method of sharing attention targets.