CN116547729A

CN116547729A - Attention object sharing device and attention object sharing method

Info

Publication number: CN116547729A
Application number: CN202180081745.3A
Authority: CN
Inventors: 金泽加奈子; 伊藤好文; 八木大地; 李昊舟; 西野友英; 中村裕子
Original assignee: Denso Corp
Current assignee: Denso Corp
Priority date: 2020-12-11
Filing date: 2021-12-01
Publication date: 2023-08-04
Also published as: WO2022124164A1; JP2022093012A; JP7537259B2

Abstract

The invention relates to an attention object sharing device and an attention object sharing method. When an interest reaction of a child is detected based on input data from a child camera (25) and a child sensor, the HCU (1) identifies an object of interest, which is an object outside the vehicle where the child is interested, based on the line-of-sight information of the child at the detection time and an image of the camera outside the vehicle. Then, the name, type, feature, etc. of the object of interest is acquired from the server (5), and presented to the driver and the child. The information of the object of interest can be stored in a server, a smart phone, or the like with reference to the information.

Description

Attention object sharing device and attention object sharing method

Cross Reference to Related Applications

The present application is based on japanese patent application No. 2020-206064 of the japanese application at 12/11/2020, the content of which is incorporated by reference in its entirety.

Technical Field

The present disclosure relates to a technique in which a driver shares information about things outside a vehicle that are of interest to a person sitting in a rear seat.

Background

Patent document 1 discloses a technique of displaying a captured image of an off-vehicle camera on a display that can be visually confirmed by a driver, based on a specific expression or motion of an occupant other than the driver, that is, a passenger, making a voice indicating a specific emotion, or performing a predetermined instruction operation. The image displayed on the display is an image of an object existing outside the vehicle cabin in the direction of the sight of the occupant. As described above, in patent document 1, in the configuration disclosed in patent document 1, an image including a subject that a co-passenger may be interested in is displayed on a display.

Patent document 1: japanese patent laid-open No. 2014-96632

As a vehicle usage system, there is a case where a child sits on a child seat provided in a rear seat and a parent (for example, a parent) sits on a driver's seat. Children may be interested in a wide variety of objects compared to adults. Moreover, children often ask a guardian, who is the driver, for an object of interest, or tell the guardian about the existence of the object.

However, since a guardian as a driver needs to concentrate on driving, a child's inquiry cannot always be treated carefully. In addition, it is difficult for the driver to grasp what the child sitting in the rear seat is looking at. Therefore, even when the driving load is light, it is difficult for the driver to deal with the reaction of interest close to the child. In particular, it is more difficult for children to be at an age at which things cannot be well uttered, or at an age at which the vocabulary is small.

In order to solve such a problem, according to the technique disclosed in patent document 1, since an image including a subject that may be of interest to a child is displayed on a display, a driver can estimate what the child is interested in to a certain extent by viewing the image. However, in the structure disclosed in patent document 1, only an image is displayed. Therefore, in order for parents to respond to and co-feel the child's reaction to things outside the vehicle, it is necessary to specify an object that the child may look at based on the display image, and speak the object in language, and the burden on the driver is still large. Here, the driver refers to a driver seat occupant as a person sitting in the driver seat.

Disclosure of Invention

The present disclosure has been made in view of the above circumstances, and an object of the present disclosure is to provide an object-of-interest sharing device and an object-of-interest sharing method in which an occupant of a driver seat can easily respond to a child's reaction to an object outside a vehicle.

An object of interest sharing device for achieving the object is an object of interest sharing device for use in a vehicle provided with a seat for a child to sit on, that is, a child seat, and includes: a child information acquisition unit that acquires, as information indicating a state of a child sitting in the child seat, a direction of a line of sight of the child from an image of an in-vehicle camera that includes at least a face of the child in a shooting range; an interest response detection unit that detects an interest response of the child to an object outside the vehicle based on at least one of biological information of the child, voice of the child, and a visual line direction; an object-of-interest detection unit that detects an object of interest, which is an object of interest of a child, from the line-of-sight direction acquired by the child information acquisition unit and a captured image of an off-vehicle camera (28) that is mounted on the vehicle so as to capture the outside of the vehicle; an object information acquisition unit that acquires information, which is linguistic for an object of interest, from a database disposed inside or outside the vehicle; and a notification processing unit that notifies at least one of the driver seat occupant and the child of the information acquired by the target information acquisition unit using at least one of a display of a text and a voice output corresponding to the information.

In the above configuration, the object of interest of the child is determined as the object of interest based on the line-of-sight information of the child, and the information which is linguistic for the object of interest is acquired and provided to at least one of the child and the driver seat occupant. Here, according to the structure in which the information on the object of interest is notified to the guardian who is the driver seat occupant, the guardian can easily recognize what the child is paying attention to, since not only the notification image but also the information after the linguistic is notified. Thus, it is easy to talk with children on the subject of things that the children are interested in.

In addition, according to the configuration in which the child is notified of the information on the object of interest, the information which is linguistic with respect to the object of interest is also notified, and therefore it is easy to convey what the child is paying attention to the driver's seat occupant based on the information. As a result, the driver seat occupant can easily respond to the child's interest response.

In addition, an object of interest sharing method for achieving the above object is an object of interest sharing method executed by at least one processor for a guardian to share an object of interest exhibited by a child seated in a child seat set in advance in a vehicle, comprising: acquiring a line-of-sight direction of a child from an image of an in-vehicle camera including at least a face of the child seated in the child seat in a photographing range; detecting an interest reaction of the child to things outside the vehicle according to at least any one of biological information of the child, voice of the child, and a sight line direction; detecting an object of interest of the child, i.e., an object of interest, from the acquired line-of-sight direction of the child and a captured image of an off-vehicle camera mounted on the vehicle so as to capture the outside of the vehicle; acquiring information which is linguistic for the object of interest from a database disposed inside or outside the vehicle; and notifying at least one of the driver seat occupant and the child using at least one of a display of text and a voice output corresponding to the acquired information on the object of interest (S110, S111, S113).

The method is a method executed by the object of interest sharing apparatus. According to the above method, the same effect can be obtained by the same action as the object of interest sharing apparatus.

The reference numerals in brackets described in the claims indicate correspondence with specific mechanisms described in the embodiments described below as one embodiment, and do not limit the technical scope of the present disclosure.

Drawings

Fig. 1 is a block diagram showing an example of the overall structure of an object of interest sharing system.

Fig. 2 is a diagram showing an example of the structure of the window display.

Fig. 3 is a functional block diagram of an HCU.

Fig. 4 is a flowchart for explaining the operation of the HCU.

Fig. 5 is a diagram showing an example of a notification image concerning an object of interest (dog).

Fig. 6 is a diagram showing an example of a notification image of an object of interest (vehicle).

Fig. 7 is a diagram showing an example of a notification image concerning an object of interest (building).

Fig. 8 is a diagram showing an example of a notification image concerning an object of interest (rainbow).

Detailed Description

Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Fig. 1 is a diagram showing an example of a schematic configuration of an object of interest sharing system Sys to which the object of interest sharing apparatus according to the present disclosure is applied. Some or all of the elements constituting the attention object sharing system Sys of the present disclosure are mounted on the vehicle. A part of the functional elements included in the target sharing system Sys may be provided outside the vehicle. In addition, the object of interest sharing system Sys can also be understood as a system that assists communication of children with guardians who are drivers in one aspect. Therefore, the attention sharing system Sys can also be referred to as an exchange support system.

< preamble >

The driver in the present disclosure refers to a person sitting in the driver's seat, i.e., a driver seat occupant. The expression of the driver is not limited to a person who actually performs part or all of the driving operation. The term "driver" refers to a person who should receive permission to drive from the automated driving system during automated driving. Here, as an example, the operation of the attention sharing system Sys will be described assuming that a child sits on a child seat provided in a rear seat and a guardian of the child sitting on the rear seat sits on a driver seat as a driver. In addition to the mother and father of the child, the parent and father of the child are included in the concept of the guardian. The caregivers may also be included in guardians according to custom or law of the territory in which the system is used.

For example, the child seat is fixed to a predetermined position in a rear seat provided in a vehicle such as a seat located at the rear side of a driver's seat by a mechanism such as a seat belt or ISOFIX. ISOFIX is according to ISO13216-1:1999, also known as LATCH (Lower Anchors and Tethers for Children: child-use lower fastener and tether). The child seat herein refers to a passenger protection device or restraint device for children in a broad sense, and may include a booster seat, a child seat, and the like for adjusting the height of a seat surface. Hereinafter, a seat on which a child such as a seat provided with a child seat sits will be referred to as a child seat. In addition, for example, when the child is sufficiently large, and the condition that the restraint device can be omitted is satisfied, the child seat may not be provided to the child seat. The child seat is a child-seating seat, and need not be child-specific. Child seats may also be configured as seats that allow adults to sit.

The "autopilot" instruction of the present disclosure may be rated, for example, as rating 3 defined by the american society of automotive engineers (SAE International), or may be rated 4 or more. Rank 3 refers to a rank at which the system performs all driving tasks in the operation design area (ODD: operational Design Domain), and on the other hand, the operation authority is transferred from the system to the user in an emergency. The ODD is, for example, an area in which the driving position is defined as an inside of a highway or the like, and conditions for automatic driving can be executed. At level 3, it is required that the driver can cope with the request from the system for driving alternation promptly. Grade 3 corresponds to so-called conditional automatic driving. The level 4 is a level at which the system can perform all driving tasks except for specific conditions such as an unresponsive road and a limited environment. Level 5 is a level at which the system can perform all driving tasks under all circumstances. The automatic driving of the class 4 or more is a class in which the automatic driving apparatus performs all driving tasks, that is, an automation class in which the sleeping of the driver seat occupant is permitted. The level 2 or less corresponds to a driving assistance level at which the driver performs at least a part of the driving task of steering and acceleration/deceleration. The system herein refers to an in-vehicle system including an object of interest sharing system Sys.

Structure of object of interest sharing System Sys

As shown in fig. 1, the target sharing system Sys is configured as an HCU (HMI Control Unit) 1 including an HMI (Human Machine Interface: human-machine interface) for controlling the operation of the display or the like. The HCU1 corresponds to an object of interest sharing apparatus.

The HCU1 is used in connection with various in-vehicle devices such as a driver status monitor (hereinafter, DSM: driver Status Monitor) 21. For example, the HCU1 is connected to the DSM21, the driver sensor 22, the driver microphone 23, the input device 24, the child camera 25, the child sensor 26, the child microphone 27, the vehicle exterior camera 28, and the positioner 29. The HCU1 is also connected to a communication device 31, a meter display 32, a center display 33, a head-up display (HUD) 34, a rear seat display 35, a window display 36, a dialogue device 37, a speaker 38, and the like.

The HCU1 is also connected to various sensors and devices, not shown in fig. 1, via an in-vehicle network Nw, which is a communication network built in the vehicle. For example, the HCU1 is configured to be able to communicate with a computer that controls the travel of a vehicle such as an automatic driving device via the in-vehicle network Nw, and a signal indicating whether or not the current vehicle is in an automatic driving mode may be input. The HCU1 inputs various detection results of the in-vehicle sensors via the in-vehicle network Nw. Examples of the in-vehicle sensor include sensors that detect a vehicle speed, acceleration, steering angle, shift position, accelerator depression amount, brake depression amount, and the like. The in-vehicle sensor includes a sensor/switch for detecting an operating state of the parking brake and a power supply state of the vehicle.

The HCU1 and various devices may be connected by dedicated lines or may be connected via an in-vehicle network Nw. An ECU (Electronic Control Unit: electronic control unit) may be interposed between the HCU1 and the in-vehicle apparatus.

The DSM21 is a device that sequentially detects the status of a user from the face image of the user. The DSM21 contains, for example, a near infrared light source, a near infrared camera, and a control module that controls them. The DSM21 is provided on, for example, an upper surface of the steering column portion or an upper surface of the instrument panel in a posture in which the near infrared camera is directed in a direction in which the headrest of the driver's seat exists. The DSM21 photographs the head of the driver irradiated with near infrared light by the near infrared light source through the near infrared camera. The captured image captured by the near infrared camera is subjected to image analysis by the control module. The control module extracts, for example, driver state information, which is information indicating the state of the driver, such as the opening degree of the eyes of the driver, from the captured image input from the near infrared camera. In addition, the camera constituting the DSM21 may be a visible light camera. The DSM21 outputs the driver state information extracted from the face image of the driver to the HCU1.

The driver state information includes, for example, a direction of a face of the driver, a line of sight direction, an opening degree of an eyelid, an opening degree of a pupil, an opening degree of a mouth, a posture, and the like. The DSM21 may be configured to estimate the expression, emotion, or the like of the driver from the distribution of feature points of the face, or the like. As a method of detecting an expression based on image analysis, various methods such as a method of using a distribution pattern of feature points of a face, a method of movement of muscles of a face, and the like can be applied, and therefore, a detailed description thereof will be omitted. As an example, a method or the like of detecting a plurality of kinds of Action units (activity units, hereinafter, AUs) defined as units of motion corresponding to respective muscles of a face according to anatomical insights and estimating expressions from a pattern of detected amounts (scores) of each AU can be employed. Examples of AU include lowering eyebrows, raising cheeks, raising corners of the mouth, raising upper lips, and the like.

In addition, the DSM21 may estimate the tension of the driver based on the expression of the driver, etc. In addition, the health state of the driver may also be estimated from the color information of the face of the driver. The state information of the driver can include expression, emotion, tension, health state, and the like. The HCU1 may have a function of detecting the state of the driver based on image analysis. In this case, the DSM21 may be configured to be able to output a face image of the driver to the HCU 1. The functional configuration between the DSM21 and the HCU1 can be changed as appropriate.

The driver sensor 22 is a biometric sensor that senses biometric information of the driver. The driver sensor 22 is, for example, a pulse sensor that senses a pulse. The driver sensor 22 may be a sensor that detects at least one of blood pressure, cardiac potential, heart rate, perspiration amount, body temperature, heat dissipation amount from a human body, breathing rhythm, breathing depth, exhalation component, body component, posture, body movement, skin electric activity, facial muscle activity potential, and peripheral blood flow. The peripheral blood flow is, for example, the blood flow of a peripheral portion such as a fingertip. The biosensor includes a temperature sensor, a pulse sensor, a humidity sensor, a heart rate sensor, and the like. In addition, the concept of the biological information may include various state amounts as described above. The HCU1 may be connected to a plurality of types of driver sensors 22 whose biological information to be detected is different. The DSM21 described above may be included in the driver sensor 22 in a broad sense.

The driver sensor 22 may be incorporated in a back portion or a headrest of the driver's seat, or may be provided in a steering wheel. Further, millimeter wave radar that detects the heart rate, body motion, and posture of the driver by transmitting and receiving millimeter waves as detection waves to and from the driver's seat can also be included in the biosensor. The driver sensor 22 may also be a thermal imaging camera. As the driver sensor 22, a variety of detection principle sensors such as a radio wave type sensor and an infrared ray type sensor can be used.

In addition, some or all of the various driver sensors 22 may be wearable devices that are worn by the driver, such as an arm. The wearable device can adopt various shapes of devices such as wristband type, wristwatch type, finger-ring type, glasses type, earphone type, and the like. The wearable device as the driver sensor 22 is configured to be capable of communicating with the HCU1 via the communication device 31 mounted on the vehicle. The wearable device and the communication device 31 may be connected by a wired connection or a wireless connection. As a wireless connection method, a short-range wireless communication standard such as Bluetooth (registered trademark) and Wi-Fi (registered trademark) can be used. Further, as the biosensor, an electrodermal activity (EDA) sensor or the like that detects a change in skin conductance (skin surface potential) caused by a sympathetic nervous system activity can also be employed.

The driver microphone 23 is a device that converts ambient sounds such as voices generated by the front seat occupant into electrical signals and inputs the electrical signals to the HCU 1. The driver microphone 23 is disposed on the upper surface of the steering column cover, the steering wheel, the center portion of the instrument panel, and the like, for example, so as to facilitate the sound pickup of the speech of the driver.

The input device 24 is an operation member for receiving an instruction from the driver to the HCU 1. The input device 24 may be a mechanical switch (so-called steering switch) provided in a spoke portion of the steering wheel, or may be a voice input device for recognizing the content of the speech of the driver. The input device 24 may be, for example, a touch panel laminated on a display panel of a display provided on the instrument panel, such as the center display 33. The input device 24 may be a smartphone of the driver. For example, a touch panel and a display of a smartphone owned by the driver can be applied as the input device 24.

The child camera 25 is a camera that photographs the face of a child seated in the child seat. For example, the child camera 25 is attached to the back of a front seat located on the front of the child seat. The child camera 25 may be disposed at a ceiling portion of a vehicle interior or the like so as to be able to capture a face of a child sitting in a rear seat. The camera including the entire rear seat in the imaging range may be used as the child camera 25. For example, the child camera 25 may be attached to a center portion of the ceiling, an upper end portion of a windshield, a high-position dashboard, or the like. The image data captured by the child camera 25 is output to the HCU 1. The concept of image data herein can also include video signals. The child camera 25 corresponds to an in-vehicle camera.

The child sensor 26 is a biosensor that senses biological information of an occupant sitting in the child seat (i.e., a child). The child sensor 26 is a sensor that is a detection target for at least one of various state amounts such as blood pressure, cardiac potential, heart rate, perspiration amount, and body temperature, similarly to the driver sensor 22. For example, the child sensor 26 is a pulse sensor that senses pulses. The HCU1 may be connected to a plurality of types of child sensors 26 having different biological information as detection targets. Similar to the biological information of the driver, the biological information of the child may include various items.

The child sensor 26 is built in a child seat, for example. The child sensor 26 may be a noncontact sensor that acquires various pieces of key information using millimeter waves, infrared rays, or the like. The child sensor 26 may also be a thermal imaging camera. The child sensor 26 may be a wearable device worn by a child, for example, an arm. The wearable device as the child sensor 26 may be configured to be capable of communicating with the HCU1 via a communication device 31 mounted on the vehicle.

The child microphone 27 is a device that converts voice uttered by an occupant of the rear seat, particularly a child sitting in the child seat, into an electric signal and inputs the electric signal to the HCU 1. The child microphone 27 is disposed, for example, on a rear surface portion of a front seat located on a front surface of the child seat, a center portion of a roof portion in a vehicle cabin, or the like, so as to facilitate sound reception of a speech of a child sitting on the child seat. The child microphone 27 may be provided near a headrest of the child seat.

The off-vehicle camera 28 is an on-vehicle camera that photographs the surroundings of the vehicle and outputs the data of the photographed image to the HCU 1. The vehicle exterior camera 28 includes at least a lens and an image sensor, and electronically acquires an image representing the periphery of the vehicle. The number of cameras 28 may be one or more.

For example, the HCU1 is connected with a front camera, a rear camera, a left side camera, and a right side camera as the vehicle exterior camera 28. The front camera is a camera that photographs the front of the vehicle at a predetermined angle of view, and is mounted on the front end of the vehicle such as a front grille. The rear camera is a camera that photographs the rear of the vehicle at a predetermined angle of view, and is disposed at a predetermined position on the rear surface of the vehicle body, for example, near the rear license plate or near the rear window. The left side camera is a camera for photographing the left side of the vehicle, and is mounted on the left side mirror. The right camera is a camera for photographing the right side of the vehicle, and is mounted on the right rearview mirror. As lenses of these off-vehicle cameras 28, wide-angle lenses such as fish-eye lenses are used, and each of the off-vehicle cameras 28 has a viewing angle of 180 degrees or more. Therefore, by using four cameras 2, the entire periphery (i.e., 360 °) of the host vehicle can be photographed.

The mounting positions of the cameras 2 can be changed as appropriate. The front camera may be attached to an inside mirror, an upper end portion of a windshield, or the like. The left and right side cameras may be disposed near the root of the a-pillar or the B-pillar. The off-board camera 28 may be mounted on the roof of the vehicle or may be mounted on the roof of a booth in the vehicle. Some or all of the off-vehicle cameras 28 may be cameras that are mounted on the roof, on the dashboard, near the window frame of the rear seat door, or the like, for example. The off-vehicle camera 28 may be a compound eye camera that includes a plurality of lenses and imaging elements, and is capable of performing 360 ° imaging by one camera. The attention sharing system Sys may include a plurality of cameras having different imaging distance ranges as the off-vehicle camera 28. For example, the attention sharing system Sys may include a near-field camera for capturing a near field and a telephoto camera for capturing a relatively distant field as the off-vehicle camera 28.

The positioner 29 is a device that positions the current position of the host vehicle. The locator 29 is implemented using, for example, a GNSS receiver, an inertial sensor, and a map database (hereinafter, DB). A GNSS receiver is a device that detects the current position of the GNSS receiver in sequence (for example, every 100 milliseconds) by receiving navigation signals transmitted from positioning satellites constituting a GNSS (Global Navigation Satellite System: global navigation satellite system). The locator 29 sequentially locates the position of the own vehicle by combining the positioning result of the GNSS receiver and the measurement result in the inertial sensor. The located vehicle position is output to the HCU 1. The locator 29 reads map data of a predetermined range specified with reference to the current position from the map DB, and supplies the map data to the HCU 1. The map DB may be stored locally in the vehicle or may be disposed in the cloud.

The communication device 31 is a device for wireless or wired communication with an external device. For example, the communication device 31 performs data communication with the smart phone 4 and the wearable device that are brought into the vehicle room in accordance with standards such as Bluetooth (registered trademark). The smartphone 4 is mainly a smartphone owned by the driver. The communication device 31 is configured to be capable of performing wireless communication according to standards such as LTE (Long Term Evolution: long term evolution), 4G, 5G, and the like, and is configured to perform data communication with a predetermined server 5.

For example, as will be described later, the server 5 stores data concerning an object of interest, which is an object outside the vehicle cabin in which children are interested. The server 5 includes a database or the like in which description information about various objects is registered. The server 5 is configured to be able to transmit, in response to a request from the vehicle, explanatory information about an object of interest of the child detected by the HCU 1. The server 5 may have a function of acquiring information on the internet for the object of interest transmitted from the vehicle and returning the acquired information to the vehicle. The server 5 may also be a Web server. The data on the object of interest transmitted from the vehicle may be image data, in which case the server 5 may be configured to acquire text data in which the object of interest is linguistic as a search term by analyzing the received image.

The meter display 32 is a display disposed in a region located on the front surface of the driver's seat in the instrument panel. As the display, a liquid crystal display, an organic EL display, or the like can be employed. The center display 33 is a display disposed near the center in the vehicle width direction in the dashboard. The HUD34 is a device that projects image light to a predetermined area of a windshield in accordance with control signals and video data input from the HCU1, a navigation device, and the like, thereby displaying a virtual image that can be perceived by a user. The HUD34 displays an image overlapping with a landscape in front of the vehicle. The meter display 32, the center display 33, and the HUD34 display images corresponding to signals input from the HCU1, respectively. The meter display 32, the center display 33, and the HUD34 correspond to a display facing the driver. Of course, the passenger of the passenger seat can also visually confirm the central display 33. Thus, the center display 33 can also be understood as a display facing an occupant of the front seat containing the passenger seat occupant.

The rear seat display 35 is a display facing the occupant of the rear seat, mainly facing the occupant seated in the child seat. The rear seat display 35 is also referred to as a rear monitor. The rear seat display 35 is disposed on, for example, a rear surface portion of a front seat located on the front surface of the child seat, a roof portion in the vehicle cabin, and the like. The rear seat display 35 also operates according to the input signal from the HCU 1. Further, here, as an example, the rear seat display 35 is provided with a touch panel, and is configured to be able to receive an instruction operation of the child on the display screen on the rear seat display 35.

The window display 36 is a device that irradiates display light to a window glass on a side of the vehicle, particularly to a side window adjacent to the child seat, to display an image on the window glass. For example, as shown in fig. 2, the window display 36 includes a projector 361 for emitting image light, and a screen 362. The screen 362 is a sheet-like structure for reflecting the image light into the vehicle interior, and is attached to the vehicle interior side surface of the window glass 61 that is the irradiation target of the image light. A mirror 363 for magnifying reflection may be interposed between the projector 361 and the screen 362. Mirror 363 is preferably a concave mirror, but may also be a flat mirror. The projector 361 and the reflecting mirror 363 are disposed on the vehicle interior side surface of the roof portion 62, that is, on the roof portion of the vehicle interior booth.

The dialogue device 37 is a device for performing a dialogue with various kinds of passengers such as a child sitting in a child seat and a driver. The dialogue device 37 performs, for example, the following operations: the voice data of the child acquired by the child microphone 27 is recognized, a response to the input voice of the child is made, and a voice output response is made. The dialogue device 37 is configured to recognize the content of the speech of the occupant and generate the response using artificial intelligence, for example. The server 5 may be configured to perform the recognition processing of the content of the utterance and the generation processing of the response via the communication device 31. The dialogue device 37 may have a function of displaying an image of a predetermined Agent (Agent) on the rear-seat display 35. Agents are roles such as aerial characters, anthropomorphic animals, etc. The agent may be an avatar (avatar) of the driver set in advance by the driver, or the like.

The dialogue device 37 may display an animation of the agent acting as if speaking on the rear seat display 35 in the case of a voice output of a prescribed message. Further, the operation of the dialogue device 37 including the display of the animation using the agent is controlled by the HCU 1. The display destination of the animation using the agent is not limited to the rear seat display 35, and may be the window display 36.

The speaker 38 causes speech to be generated in the cabin of the vehicle. As the type of speech output, there are a speech message, music, an alarm, and the like, in which a predetermined text is read. The expression speech also includes a simple sound. The vehicle is provided with a speaker 38A for the driver and a speaker 38B for the child as speakers 38. The driver speaker 38A is provided, for example, in an instrument panel, a headrest of a driver seat, or the like. The speaker 38B for children is incorporated in the seat for children. The speaker 38B for children may be provided near the side wall portion and the ceiling portion of the child seat.

Structure related to HCU1

The HCU1 is a computer for comprehensively controlling information presentation to a user using a display or the like. The HCU1 is configured as a computer including a processor 11, a RAM (Random Access Memory: random access memory) 12, a storage 13, a communication interface 14 (I/O in the figure), a bus connecting these structures, and the like.

The processor 11 is an operation core such as a CPU (Central Processing Unit: central processing unit). The processor 11 performs various processes by accessing the RAM 12. The RAM12 is a volatile memory. The communication interface 14 is a circuit for the HCU1 to communicate with other devices. The communication interface 14 may be implemented using analog circuit elements, ICs, or the like.

The storage 13 is a structure including a nonvolatile storage medium such as a flash memory. The memory 13 stores a target sharing program, which is a program for causing a computer to function as the HCU 1. The processor 11 executes the attention object sharing program corresponds to a method corresponding to the attention object sharing program, that is, an attention object sharing method. In addition, data indicating the installation position of the vehicle exterior camera 28, the position of the child seat, the installation position of the child camera 25, and the like are registered in the memory 13.

The HCU1 executes the target sharing program stored in the memory 13 by the processor 11, thereby providing the respective functional units shown in fig. 3. That is, the HCU1 includes a child information acquisition unit F1, an outside vehicle information acquisition unit F2, a driver information acquisition unit F3, and a vehicle information acquisition unit F4. The HCU1 includes, as functional units, an interest response detection unit F5, an object determination unit F6, a description information acquisition unit F7, a driving load estimation unit F8, a notification control unit F9, a recording processing unit FA, and an interest object management unit FB. The notification control unit F9 includes a timing adjustment unit F91 and a target control unit F92 as more detailed functional units. The notification control unit F9 corresponds to a notification processing unit.

The child information acquisition unit F1 acquires various pieces of information about the state of the child sitting in the child seat from the child camera 25 and the child sensor 26. For example, the child information acquisition unit F1 analyzes the image supplied from the child camera 25 to estimate at least a part of the face direction, the line of sight direction, the eyelid opening, the pupil opening, the mouth opening, the posture, the body movement, and the like of the child. The body motion also includes a behavior of directing a finger or hand out of the window. The child information acquisition unit F1 may be configured to estimate the expression, emotion, and the like of the child from the distribution of feature points of the face included in the image from the child camera 25. Further, the presence or absence of a child speaking may be detected from a time-series change pattern of the distribution of feature points around the mouth. The breathing rhythm may be estimated from the body motion, mainly from the change pattern of the chest or abdomen position.

The child camera 25 may include a function/processing module that analyzes the captured image of the child camera 25 to estimate the state of the child. The external server may have the above-described function. In this case, the communication device 31 transmits the image data of the child to the server 5, and analyzes the received image at the server 5 and returns the result thereof to the vehicle. As such, various functions for implementing the structures of the present disclosure may also be distributed across edges and clouds. The arrangement of the various functions can be changed as appropriate.

The child information acquisition unit F1 acquires detection results such as pulse information from the child sensor 26. Of course, when the child sensor 26 is configured to be able to detect the pulse rate, blood pressure, cardiac potential, heart rate, perspiration amount, body temperature, heat dissipation amount, breathing rhythm, breathing depth, exhalation component, body component, posture, body movement, and the like, the child information acquisition unit F1 may acquire these pieces of information.

The state information of the child acquired by the child information acquisition unit F1 is given a time stamp indicating the acquisition time and stored in the RAM12. The information of the state quantity of the child acquired by the child information acquisition unit F1 is classified for each information type, and is stored in the RAM12 for a certain period of time, for example. The data with different acquisition time can be sorted and stored according to the sequence of the acquisition time, so that the data with the latest acquisition time is the front end. The data storage period can be set to, for example, 2 minutes, 5 minutes, or the like.

The child information acquisition unit F1 calculates the normal values of the pulse rate, the eye opening, the heart rate, the body temperature, the skin surface potential, and the like from the detection results of the last predetermined time. The normal value may be, for example, an average value or a median value of observed values within a latest predetermined time. For example, the child information acquisition unit F1 calculates a normal value of each state quantity by averaging observation values within 1 minute recently for each of the pulse rate, the eye opening, and the face direction. These normal values can be used as a judgment reference for detecting that a child is interested in something and becomes excited.

When the child speaks, the child information acquisition unit F1 detects that the child speaks based on the input signal from the child microphone 27. Further, it is more preferable that the child information acquiring unit F1 acquires information indicating the content of the utterance and the size of the sound when the child utters some words. The content of the utterance may be determined by a speech recognition process. The child information acquisition unit F1 acquires or detects various information, and uses the information in the interest response detection unit F5 and the object determination unit F6.

The outside-vehicle information acquiring unit F2 acquires outside-vehicle information from the outside-vehicle camera 28, the positioner 29, and the like. For example, the outside-vehicle information acquiring unit F2 sequentially acquires image data captured by the outside-vehicle camera 28, and temporarily stores the image data in the RAM12 or the like. The save area of the image data may be configured as a ring buffer. That is, when the storage amount reaches a certain upper limit value, new data is deleted and stored in order from the earlier data.

The outside-vehicle information acquiring unit F2 associates the image data input from the outside-vehicle camera 28 with the position information and the time information of the vehicle at the time of shooting, and stores the same in the RAM12. The outside-vehicle information acquiring unit F2 may be configured to analyze an image input from the outside-vehicle camera 28 to determine the position and type of the subject included in the image. For example, CNN (Convolutional Neural Network: convolutional neural network) and DNN (Deep Neural Network: deep neural network) techniques to which deep learning is applied can be used for recognition of an object.

In addition, the outside-vehicle information acquiring unit F2 acquires map information of the surroundings of the vehicle from the locator 29. That is, information of facilities, landmarks, and the like existing in the periphery of the vehicle is acquired. The map information also includes information related to a road structure such as the type of the road currently traveling. The type of road indicates, for example, whether it is a general road or an expressway. The information related to the road structure includes a curvature of the road, a bifurcation/confluence point existing in front of the vehicle, an intersection, a remaining distance to the traffic light, and the like.

The driver information acquisition unit F3 acquires various information on the state of the driver from the DSM21 and the driver sensor 22. The driver information acquisition unit F3 corresponds to a driver seat occupant information acquisition unit. For example, the driver information acquisition unit F3 acquires at least a part of the direction of the face, the line of sight direction, the opening of the eyelid, the opening of the pupil, the opening of the mouth, the posture, and the like of the driver from the DSM 21. In addition, when the DSM21 is configured to be able to estimate the expression, the tension, and the like of the driver, the driver information acquisition unit F3 may acquire these pieces of information. Further, the presence or absence of a speech of the driver may be detected from a time-series change pattern of the distribution of feature points around the mouth. The breathing rhythm of the driver may be estimated from the body motion, mainly from the change pattern of the chest or abdomen position. The rhythm of breathing can take the tension and driving load as estimation materials. As described above, the driver information acquisition unit F3 and the server 5 may have functions of the DSM21, that is, functions and processing modules for estimating the state of the driver by analyzing the image of the driver.

The vehicle information acquisition unit F4 also acquires the state (on/off) of the power supply for running, the vehicle speed, the acceleration, the steering angle, the accelerator depression amount, the brake depression amount, and the like. The power source for running is a power source for running a vehicle, and in the case where the vehicle is a gasoline vehicle, it is referred to as an ignition power source. In the case where the vehicle is an electric vehicle such as an electric vehicle or a hybrid vehicle, the power source for running is a system main relay. In addition, the vehicle information acquisition unit F4 acquires information indicating the current position of the vehicle from the locator 29. The position information of the own vehicle can be expressed by latitude, longitude, altitude, and the like.

The interest response detecting unit F5 determines whether or not the child has an interest in something outside the vehicle based on the state information of the child acquired by the child information acquiring unit F1. Here, the expression of an interest of a child in something outside the vehicle is also described as an interest reaction. The determination that the generation of the response of interest corresponds to the detection of the response of interest. As objects of interest for children, in addition to stationary objects such as buildings and signs, various things such as moving objects such as pedestrians and animals, and sceneries are assumed. The expression "article" here includes not only articles such as stationary articles and moving articles but also conditions, states, scenery, experiences, and the like.

For example, the interest response detecting unit F5 determines that an interest response exists when a child sitting in the child seat continues to observe the same object for a predetermined time or longer. In the case where the child continues to observe the same object for a predetermined time or longer, in addition to the case where the line of sight direction is a constant direction for a predetermined time, the child includes a case where the line of sight direction is moved to the opposite side (for example, the rear) of the traveling direction of the vehicle so as to chase the object of interest by eyes.

The interest response detecting unit F5 may determine that an interest response exists when the child is an expression indicating a specific emotion. Specific emotions herein include exclamation, and the like, and more specifically, convulsion, jersey, smiling face, and the like. Further, the interest reaction detection unit F5 may determine that an interest reaction exists when detecting that the child speaks in a state in which the child faces the outside of the vehicle. In this case, the level of interest may be evaluated based on the sound level of the child. In addition, it may be determined that the child has exhibited a high interest response by repeatedly speaking the same word a predetermined number of times or more while the child is facing the outside of the vehicle.

Furthermore, the child may speak "look ahead", "what is that? "and the like ask the driver for a phrase, and determine that an interesting reaction exists. Further, the interest response may be detected by speaking a speech-like, a mimicry, a speech-like (so-called pictographic) or a speech-like (so-called pictographic) representing a specific animal or state. Further, it may be determined that an interesting reaction exists based on the child performing an operation of directing a finger or a hand to the outside of the window, that is, performing an operation of instructing something.

The interest response detection unit F5 may detect the interest response of the child based on biological information such as the pulse, heart rate, and eye opening of the child. For example, the presence of the interesting response may be determined based on the pulse becoming equal to or more than a threshold value defined faster than the normal value. Further, it may be determined that there is an interesting reaction based on the opening degree of the eye being increased by a predetermined value or more from the normal value. As the detection material for the reaction of interest, a change in body temperature, a state of respiration, a body composition, or the like can also be used. In addition, in the state of breathing, the speed of breathing, the expiratory component, and the like are included. In this way, the interest response detection unit F5 may detect that something shows interest in the child in a state different from the normal state or in an excited state.

When the interest response detection unit F5 detects an interest response of the child, the object specification unit F6 specifies an object of interest, which is an object of interest that the child has expressed an interest, based on the line of sight direction of the child within a predetermined trace-back time period from the point in time. The concept of the object of interest here includes stationary objects such as facilities, landmarks, signs, and the like, moving objects such as pedestrians, animals, conditions (events), scenes, and the like. For example, unfamiliar buildings, people wearing doll clothes, emergency vehicles such as dogs, cats, fire fighting vehicles, etc., commercial signs using children's welcome roles, corporate signs, etc., may be objects of interest. The object specifying unit F6 corresponds to an object of interest detecting unit.

The object specifying unit F6 specifies an object existing in the line of sight of the child as a target of interest in an image captured by the off-vehicle camera 28, for example, at a time when the interest reaction is detected or within a trace-back time. The trace back time here can be set to 200 ms, 500 ms, 1 second, or the like.

Further, the direction of the line of sight for the child outside the vehicle cabin may be calculated based on the position of the eyes of the child in the vehicle cabin determined from the image of the child camera 25 and the direction of the line of sight starting from the position of the eyes of the child. The position of the eyes of the child in the vehicle interior can be calculated from the set position and posture of the child camera 25 and the position of the eyes of the child in the image. In addition, by combining information such as the direction of the line of sight for the child outside the vehicle and the azimuth angle in which the vehicle body is oriented, the absolute direction in which the child is oriented, and the like can be calculated. The absolute direction herein is, for example, a direction corresponding to a predetermined azimuth angle such as north-south.

In this way, by comparing the image of the off-vehicle camera 28 acquired and saved at or immediately before the time when the interest reaction is detected with the line-of-sight direction of the child, the object of interest to the child can be automatically determined.

Here, as an example, the object specifying unit F6 acquires the object of interest as an image. Of course, as another aspect, the object specifying unit F6 may also specify the type, name, and the like of the object of interest by analyzing the object image, which is an image of the object of interest specified. The object image, which is an image of the object of interest acquired by the object determining unit F6, is output to the description information acquiring unit F7. The object specifying unit F6 also obtains the direction in which the object of interest exists from the perspective of the vehicle or the driver's seat in the process of specifying the object of interest. Information about the direction in which the object of interest exists is also output to the description information acquisition unit F7 and the notification control unit F9.

The description information acquiring unit F7 acquires description information about the object of interest specified by the object specifying unit F6 from a dictionary database disposed inside or outside the vehicle. The dictionary database is a database in which explanatory information about various objects is registered. The dictionary database may be mounted on a vehicle or on a cloud. The dictionary database may be a Web server or the like. The description information acquisition unit F7 corresponds to an object information acquisition unit.

For example, the description information acquiring unit F7 transmits the object image to the server 5, thereby acquiring description information about the object from the server 5. The description information here includes at least one of the category (large category), action, background, and the like of the object in addition to the name of the object. For example, if the object of interest is a building, the proper name, general name, effect, height, year of construction, and the like thereof are included in the description information. If the object of interest is an animal, a large class such as a dog, cat, or bird, and a name, size, origin, character (character), or the like of a more detailed category are included in the description information. If the object of interest is an event such as a ceremony, the name, history background, holding time, and the like are included in the description information. When the object of interest is a natural phenomenon such as rainbow, the name, generation principle, and the like thereof can be used as explanatory information. When the object of interest is a vehicle such as an ambulance or a construction vehicle, the name, responsibility, feature, or the like thereof is included in the description information. In the case where the object of interest is a sign of a business or a store, the type of service provided, the year of creation, the representative commodity, or the like may be included in the description information in addition to the name of the business or store. The explanatory information can be mainly information published on a drawing, dictionary, manual, or the like.

The server 5 analyzes the object image to determine the name of the object of interest based on the inquiry from the description information acquiring unit F7, and also acquires information other than the name related to the object of interest. The information other than the name may be obtained by performing internet search using the name as a search keyword, or may be obtained by referring to a dictionary database stored in the server 5 itself. The server 5 then returns information collected by using the name of the object of interest as a search keyword to the vehicle as explanatory information. According to this configuration, even if the vehicle does not include a database in which huge data is recorded, the description information acquiring unit F7 can acquire information on various objects of interest.

The description information acquiring unit F7 may analyze the object image to determine the type, name, and the like of the object of interest. In this case, the name of the object of interest is used as a search keyword, and the supplementary information such as responsibility and background of the object of interest is acquired from a database disposed in or outside the vehicle. The information acquired by the description information acquiring unit F7 is associated with the object image and temporarily stored in the RAM12 or the like. The data set is notified to the control unit F9 and the recording processing unit FA for reference.

The driving load estimating unit F8 determines whether the driving load of the driver is high or not based on at least any one of the state information of the driver, the running environment of the vehicle, and whether or not the vehicle is being driven automatically. As the state information of the driver, information acquired/detected by the driver information acquisition portion F3 may be used. As the running environment information, information on the surroundings of the vehicle acquired by the vehicle information acquisition unit F4 or the outside-vehicle information acquisition unit F2 may be used. Whether or not in the automatic driving can be input from the automatic driving apparatus via the in-vehicle network Nw. Further, a signal indicating the level of automatic driving, that is, indicating which of the levels 0 to 5 corresponds to may be input from the automatic driving device.

The driving load estimating unit F8 determines that the driving load is not high when, for example, the vehicle is being driven automatically at a level of 3 or more and the remaining time until the handover is equal to or more than a predetermined threshold value. In addition, the transfer corresponds to permission of the system to transfer driving operations to the driver seat occupant. On the other hand, even in the case of automatic driving, in the case where the system requests a handover from the driver or the like, it can be determined that the driving load of the driver is high.

The driving load estimating unit F8 may determine whether the driving load is high or not based on biological information such as the pulse, the interval and depth of breathing, the skin electric activity, the activity potential of the facial muscles, and the peripheral blood flow rate of the driver. For example, when the driver is in a tense state, such as when the pulse or the respiration is a predetermined value or more faster than the normal value, it can be determined that the driving load is high. The tension of the driver can be estimated from various pieces of biological information such as the strength of holding the steering wheel, the posture, the skin electric activity, the interval between blinks, and the peripheral blood flow. As a method for determining the state of tension and the driving load, various methods can be applied.

The driving load estimating unit F8 may determine that the driving load is high based on the bifurcation/junction point of the expressway, the traveling near the intersection, or the lane change. The vicinity of the intersection may be, for example, a section having a remaining distance to the intersection of 50m or less. The vicinity of the intersection also includes the interior of the intersection. Further, the driving load may be determined to be high based on a section where the vehicle is traveling at an intersection where visibility is poor, a road where many pedestrians suddenly appear, or a traffic accident is frequent. The map data may be obtained from the locator 29 or the map server as information such as intersections with poor visibility, roads with a large number of sudden occurrences of pedestrians, and sections with frequent traffic accidents.

Further, an index of safety level corresponding to a running environment such as whether or not the vehicle is running at a bifurcation/junction point of a highway, in the vicinity of an intersection, or whether or not the vehicle is in a planned lane change. The above-described configuration corresponds to a configuration in which the driving load of the driver is evaluated based on the safety level of the running environment, in other words, based on the magnitude of the risk potential determined according to the running environment. As the judgment material/judgment condition for judging that the driving load is high, various information/conditions can be employed.

The notification control unit F9 is configured to control the notification related to the information on the object of interest acquired by the description information acquisition unit F7. For example, the notification control unit F9 comprehensively controls the person/seat to be notified, the timing of displaying the image related to the target object, the timing of outputting the voice, the output destination in the case of displaying the image, and the like, based on the determination result of the driving load estimating unit F8. The function unit that mediates the timing of the display of the image and the speech output of the object of interest corresponds to the timing mediation unit F91. The display destination and the voice output destination of the image of the target object are adjusted to correspond to the occupant who has selected the notification target as the information of the target object. This is because the removal of the display for the driver such as the meter display 32 and the HUD34 as the display destination of the image of the object of interest corresponds to the removal of the driver from the presentation object of the image information. The function unit for adjusting the display destination and the voice output destination of the image of the object of interest corresponds to the object control unit F92. As one control method, the notification control unit F9 may be configured to display an icon image indicating that the child has expressed an interest response on the HUD352 when the interest response detection unit F5 detects the interest response of the child. The details of the notification control unit F9 will be described later.

The recording processing unit FA associates the data on the object of interest acquired by the description information acquiring unit F7 with the position information at the time point when the interest reaction is expressed, and stores the data in a predetermined storage medium. The device to be the storage destination of the data (so-called storage device) may be an external device such as the driver's smartphone 4 or the server 5, or may be an internal device such as the storage 13. The recording data, which is the data to be recorded, preferably includes image data of the object of interest. The image data included in the recorded data may be still images or video. The recorded data may include text data including explanatory information specified by analyzing the image. In the recorded data, the direction in which the object of interest exists, the position information of the object of interest, and the detection time information may be stored in association with each other, not with the position information of the vehicle at the time when the reaction of interest is detected. The recorded data may include image data of a child when the object of interest is observed. The recorded data may include voice data in the vehicle within a predetermined time period determined based on the time point at which the interest response is detected.

In the case where the storage destination of the data is the server 5, it is preferable that the data stored in the server 5 can be referred to from a smartphone or the like owned by the driver, the grandparent or the like. According to this structure, the child-interested article can be shared with the family members living separately. In the case of generating data save that triggers the detection of an interest reaction, the HCU1 or the server 5 may notify a device registered in advance that update of the record data has been generated. The sharing process described above may also be implemented through cooperation with a social networking service (SNS: social Networking Service).

As described above, the recording processing unit FA associates and stores data on the object of interest with various information, and can refer to the data on the object of interest later. As a result, for example, when the guardian returns to his home while driving at rest, the guardian can easily refer to the recorded data on the object of interest and talk with what the child has seen for the child. In addition, there is an advantage in that the guardian can also feel the growth of the child by observing the thing of interest to the child later.

The interest object management unit FB is a structure for specifying an interest object category, which is a category of an object of interest to the child, based on information such as a type of an object of interest detected in the past. As the object of interest category, a vehicle, an animal, a building, a signboard, a plant, a character, a fashion, or the like is assumed. Characters can also be distinguished in more detail by the name of the animation, etc. The vehicles and the like may be subdivided by four-wheelers, two-wheelers, or electric vehicles. Regarding fashion, it can also be subdivided into clothing, hairstyles, etc. Animals can also be subdivided into dogs, cats, birds, and the like.

In the case where the HCU1 is configured to be able to determine the class of interest, that is, the class of interest object, the threshold value for detecting the interest response of the child may be adjusted even when the object belonging to the class of interest object is captured by the off-vehicle camera 28. For example, in the case where things belonging to the object of interest category are captured by the off-board camera 28, the threshold for detecting the child's interest reaction may also be lowered.

< working example on HCU1 >

Here, the ac assist process performed by the HCU1 will be described with reference to fig. 4. Here, as an example, the ac support process includes steps S101 to S115. Of course, the number of steps and the processing order constituting the ac auxiliary processing can be appropriately changed.

The ac assist process shown in fig. 4 is started when a predetermined start event is generated. As the start event, for example, an ignition switch of the vehicle is turned on, a running of the vehicle is started, a start instruction from a driver is input, or the like can be employed. The communication support process may be started by triggering an interest response of the child sitting in the child seat detected by the interest response detecting unit F5. In this case, steps S101 to S104 may be sequentially executed as a process independent of the ac auxiliary process.

First, in step S101, the HCU1 acquires information necessary for processing from various devices connected to the HCU1, and the process proceeds to step S102. For example, the child information acquisition unit F1 acquires biological information of the child from the child sensor 26 or the like. The outside-vehicle information acquiring unit F2 acquires information on the environment outside the vehicle from the outside-vehicle camera 28 or the like. The driver information acquisition unit F3 acquires biometric information of the driver from the DSM21, the driver sensor 22, and the like. The vehicle information acquiring unit F4 acquires information on the current position, the running speed, and the like of the vehicle from the in-vehicle network Nw, the positioner 29, and the like. The various pieces of information acquired in step S101 are stored in a predetermined memory such as the RAM12 together with information indicating the acquisition timing. Such step S101 can be referred to as an information acquisition step and an information acquisition process. The information acquisition step may be sequentially performed at step S102 and thereafter, for example, at a predetermined cycle of 100 ms, 200 ms, 500 ms, or the like.

In step S102, the driving load estimating unit F8 determines whether or not the driving load of the driver is high based on the biological information of the driver acquired by the driver information acquiring unit F3, for example, by the algorithm described above, and the flow proceeds to step S103. The determination result concerning the driving load of the driver may be expressed by a plurality of stage rank values. In this case, a state in which the determination value is equal to or greater than the predetermined threshold corresponds to a state in which the driving load is high. Whether or not the driving load of the driver is high may be managed by a flag. For example, the driving load flag may be set to 1 (ON) when the driving load is high, and may be set to 0 (OFF) when the driving load is not high. If step S102 is completed, the process proceeds to step S103. Such step S102 can be referred to as a driver state estimation step.

In step S103, the interest response detection unit F5 determines whether or not the child has exhibited an interest response based on the biological information of the child within the latest predetermined time acquired by the child information acquisition unit F1. An affirmative determination is made in step S104 in the case where there is an interest reaction and the process moves to step S105. On the other hand, if there is no interest in the reaction, it is determined in step S115 whether or not a predetermined end condition is satisfied. The processing of steps S103 to S104 can be referred to as an interest reaction detection step.

In step S105, the object specifying unit F6 estimates the direction in which the object of interest exists when viewed from the vehicle, based on the direction of the line of sight of the child and the position of the eyes of the child. Then, from among the objects included in the image data of the off-vehicle camera 28, the object existing in the estimated direction of the object of interest is extracted as the object of interest, and the flow proceeds to step S106. Such step S105 can be referred to as an object of interest determination step.

In step S106, the object determination unit F6 extracts an image portion in which the object of interest is captured as an object image of interest from the captured image of the off-vehicle camera 28, and moves to step S107. Such step S106 can be referred to as an object-of-interest image acquisition step.

In step S107, the description information acquiring unit F7 cooperates with the communication device 31 to acquire description information about the object of interest by accessing the server 5 or the like, the internet, or the like, and the flow advances to step S108. The explanatory information is data obtained by linguistic processing of characteristics, responsibilities, names, and the like of the object of interest by text, voice data, or the like. Such step S107 can be referred to as a description information acquisition step.

In step S108, it is determined whether or not the driving load is determined to be high. The information on the driving load used in step S108 can override the determination result in step S102. If the driving load is high, the process makes an affirmative determination in step S108, and the process proceeds to step S109. On the other hand, if the driving load is not in the high state, a negative determination is made in step S108, and the process proceeds to step S111. Further, as the operation mode of the HCU1, a proxy response mode may be provided in which the system automatically responds in place of the driver. The HCU1 may be configured to move to step S109 regardless of the determination value of the driving load when the proxy response mode is set by the operation of the driver.

In step S109, the notification control unit F9 removes the driver from the notification target of the information related to the object of interest. For example, the notification target of the information related to the object of interest is set to be only children. When the passenger is also seated in the passenger seat, the notification target of the information related to the object of interest may be set as a child or passenger seat. Whether or not the passenger sits on the passenger seat can be determined based on a detection signal of a sitting sensor provided to the passenger seat.

The information related to the object of interest here is an image or description information of the object of interest. The description information may be output as an image such as text or an icon, or may be output as a voice message. The image of the object of interest includes a text image of explanatory information in addition to the image of the object of interest itself. The information related to the object of interest can be realized using at least one of image display and voice output.

The inclusion of the child in the notification object corresponds to the use of at least one of the rear seat display 35, the window display 36, and the speaker 38B as the notification device. The notification device here refers to a device that outputs information related to the object of interest by image or voice. The driver is included in the notification object, and at least one of the meter display 32, the center display 33, the HUD34, and the speaker 38A is used as the notification device. The inclusion of the passenger seat occupant in the notification target corresponds to the use of at least one of the center display 33 and the speaker 38A as the notification means. In addition, in the case where the attention sharing system Sys includes a display for the passenger seat, which is a display provided on the front surface of the passenger seat, the passenger seat display can be used as a notification device for the passenger of the passenger seat. The display for the passenger seat may be a part of a display provided continuously from the right end to the left end of the instrument panel.

Of course, the notification target for image display and the notification target for voice output may be set separately. The control may be performed so as to notify the child through both the image and the voice, and on the other hand, so as to notify the driver through only the voice. When the notification object selection processing in step S109 is completed, the process proceeds to step S110.

In step S110, the notification control unit F9 displays an image of the object of interest and description information thereof on at least one of the rear seat display 35 and the window display 36, and causes the speaker 38B to output a voice corresponding to the description information. If the content of the interest response detected in step S104 is the type of question that the child makes to the driver about the object of interest, the system response in step S110 corresponds to the proxy response process that is the process that the system responds instead of the driver. Proxy response processing can also be understood in one aspect as processing to control the actions of the dialog device 37.

In this way, when the driving load of the driver is high, the system responds promptly in place of the driver, and it is possible to present the description information about the object of interest before the child loses the interest in the object of interest. In addition, as the driver, since the necessity of immediately responding to the inquiry of the child or the like is reduced, the driving operation can be focused.

In step S110, the dialogue device 37 may be started to build a system state that can respond to the added question of the child. With this configuration, the system can continue responding to questions and the like of the instruction information of the child. Interaction of the dialog device 37 with the child may be implemented using the agent image. If step S110 is completed, the process proceeds to step S114.

In step S111, information on the object of interest is displayed on the driver seat-facing display and the child-facing display, and the process proceeds to step S112. In addition, when displayed on various displays, predetermined effect sounds may be outputted from the speakers 38 so that the driver and the child can easily notice that the image is displayed.

In step S112, it is determined whether or not the driver has uttered within the predetermined response waiting time from the image display in step S111. This step S112 corresponds to a process of determining whether or not the driver has performed some sort of response to the child' S interest reaction. The response waiting time can be, for example, 4 seconds, 6 seconds, or the like. In the case where the speech of the driver is detected within the response standby time, it is regarded that the driver responds to the child' S interest reaction, the voice output is omitted, and the process proceeds to step S114. On the other hand, if the driver 'S speech is not detected even after the lapse of the response waiting time from the start of the image display in step S111, it is regarded that the driver has difficulty responding to the child' S interest reaction, and the process proceeds to step S113. In step S113, the description information on the object of interest is output in voice, and the process proceeds to step S114. Steps S110, S111, and S113 can be referred to as a notification processing step because the information which is linguistic for the attention object is presented to at least one of the driver and the child.

In step S114, the recording processing unit FA associates and stores the image data on the object of interest with the position information, the voice data in the vehicle interior, the time information, and the like in a predetermined recording device, and the process proceeds to step S115.

In step S115, it is determined whether or not a predetermined end condition is satisfied. Examples of the end condition include turning off the power supply for running the vehicle, reaching the destination, and giving an end instruction to the driver. When the end condition is satisfied (yes in S115), the present process ends.

< display example of notification image concerning object of interest >)

Here, an example of a display image showing information about an object of interest will be described with reference to fig. 5 to 8.

The notification image of the object of interest provided to the driver preferably includes an object position information image Pd indicating the position of the object of interest, such as whether the object of interest is detected by the object determination unit F6 or not, and the object position information image can be captured by the off-vehicle camera 28. The object position information image Pd may be composed of text, an icon, or the like. The driver can easily observe the object of interest through his eyes by presenting the object position information image Pd. Further, by presenting information indicating whether or not the vehicle-outside camera 28 is still able to capture, it is possible to suppress the search for an object that is not visible due to the movement of at least one of the vehicle and the object of interest.

Fig. 5 is a diagram of a notification image Px simulating a case where the object of interest is, for example, a dog walking on a sidewalk. As shown in fig. 5, the notification image Px of the case where the object of interest is a dog includes, in addition to the image Pi of the object of interest itself, a text image Pt of the name (canine species), country of origin, character, and the like. Fig. 6 is a diagram of a notification image Px simulating a case where the object of interest is another vehicle traveling in an adjacent lane. As shown in fig. 6, the notification image Px of the case where the object of interest is a car includes a text image Pt of a name (model), a manufacturer (so-called manufacturer), a feature, and the like, in addition to the image Pi of the object of interest itself. The characteristics of the vehicle are electric vehicles or locomotives, dimensions, running performance, and the like.

As shown in fig. 7, the notification image Px of the case where the object of interest is a building such as a tower includes a text image Pt of the name, effect, year of construction, height, and the like of the building in addition to the image Pi of the object of interest itself. The functions of a building include, for example, a radio tower for transmitting broadcast waves such as television and radio, a composite commercial facility, a government agency, a factory, a school, a house, and the like. Roles can also be referred to as providing services, attributes. The notification image Px of the building as the landmark may be displayed by enabling touch selection of a photograph from another angle, for example, an appearance photograph Pj of a different time zone from the current time zone such as the night.

Fig. 8 is a diagram of a notification image Px simulating a case where the object of interest is a rainbow. As shown in fig. 8, the notification image Px of the case where the object of interest is a natural phenomenon such as rainbow includes a text image Pt indicating the cause, characteristics, and the like of the generation in addition to the image Pi of the object of interest itself. In addition, the characteristics of the rainbow contain information about the gradation of colors, such as red and violet, as the outermost colors.

The notification control unit F9 may display, as the notification image Px, video contents or a list of video contents related to the object of interest. For example, when the object of interest is a character of a certain animation, a content such as a video of the animation may be displayed.

< Effect >

In the above configuration, the HCU1 presents not only the image of the object of interest that the child is interested in, but also the description information in which the name of the object of interest is linguistically expressed, etc., to the driver. According to such a structure, the driver can easily recognize what the child is paying attention to. As a result, the driver can easily respond to the child's interest, and can respond to the child's response.

In the above, when it is determined that the driving load of the driver is high, the system responds to a question or the like for the child without waiting for the response of the driver to the response of interest of the child, as an example. According to this structure, the driver easily concentrates on driving. In addition, the child can obtain information from the system that he wants to know. In addition, for children, since the system responds to things that the system is interested in, the fear of boring the moving time can be reduced.

In the above configuration, even when the driver is notified of information about the object of interest by an image or voice, the voice output of the explanatory information is performed based on the elapse of a predetermined response waiting time from the display of the image, as an example. Thus, by setting a time difference between the image display and the voice output, the guardian as the driver easily responds to the interest reaction of the child himself. This can expect an effect that communication between the guardian as the driver and the child becomes active. In the above configuration, the system responds in a proxy manner when the driver's speech has not been detected for a predetermined response waiting time from the display for the driver to display the image. According to the above structure, since at least one of the guardian and the system responds to the own interest, the child can easily enjoy the moving time.

In addition, in the above-described structure, things showing interests of the child are associated with position information and the like and recorded. Thus, even in the case where the driver cannot determine what the child is interested in immediately after the detection of the interest reaction, it can be determined later. For example, after passing through the object of interest, the object of interest can be identified by observing the image of the object of interest. Further, for example, after the driving is completed, after the home is returned, the image of the object of interest can be seen back. The data of the object of interest may be referred to by a family living separately in cooperation with the server 5 or the like. Therefore, family members and relatives other than the driver easily share something interesting to children.

As an example of the above-described configuration, the type of the object of interest and the existence direction thereof are notified to the driver by an image or voice. According to this configuration, the driver can easily visually confirm the object of interest to the child outside the vehicle even during driving. In addition, according to the above configuration, the guardian such as the driver easily notices the growth of the child according to the change of the object of interest of the child. Therefore, according to the above configuration, the movement time can be used as an opportunity for the guardian to know the growth of the child.

The object detected as the object of interest is not limited to, for example, a facility, a landmark, or other map element registered in the map data. Various things such as pedestrians, automobiles, trains, animals, and rainbows are also detected as objects of interest. Then, the instruction information on the detected object of interest is acquired from the internet or the like and presented to at least one of the child and the driver. According to this configuration, since information on various objects of interest to the child can be positively acquired, the child can learn various objects even during the movement time.

In addition, according to the above structure, the child can specifically combine and learn knowledge on the database with the real world. Knowledge actually obtained by observation of the eyes of the user and memory thereof are easier to memorize than information obtained from textbooks and the like. Therefore, according to the above configuration, knowledge can be obtained efficiently using the travel time to the review class or school. That is, the movement time can be utilized as an opportunity to acquire knowledge. In particular, in the case of multiple complementary shifts on a child, effective utilization of movement time may be important to the child, as well as to the guardian accompanying the child. In this case, the structure according to the present disclosure has an advantage that it is easy to use the movement time as a chance of knowledge acquisition and parent-child communication.

In the above configuration, as an example, voice data representing a conversation in a vehicle and the like are stored together with an image of an object of interest. According to this configuration, it is possible to play a conversation in the vehicle about the object of interest at a later date.

According to the above configuration, by assisting communication between the child during driving and the guardian who is the driver, the movement time in the vehicle can be made more pleasant. In addition, smaller children may sometimes make a loud sound or cry because they do not get a response from the parent. If the child develops spleen qi, as a result, the driving load of the driver may increase. According to the structure of the present disclosure, by making communication between the child and the guardian smooth during driving, the concern of the child developing spleen qi can be reduced, and as a result, an effect of suppressing an increase in driving load can be expected.

While the embodiments of the present disclosure have been described above, the present disclosure is not limited to the above embodiments, and various supplementary and modified examples described below are included in the technical scope of the present disclosure, and various modifications and implementations are possible within the scope of the present disclosure other than the following. For example, various modifications can be appropriately combined and implemented within a range where technical contradiction does not occur. In addition, the components having the same functions as those described in the above-described embodiments are denoted by the same reference numerals, and the description thereof is omitted. In addition, in the case where only a part of the structure is mentioned, the structure of the embodiment described previously can be applied with respect to the other part.

Control example of Notification Contents to child

The amount, type, and expression method of information to be notified to the child may be changed according to the age, knowledge level, and ability level of the child. For example, if the child is of an age at which the child cannot read the text of the native language, the text and the voice of the native language may be output as a set. With this configuration, the effect of accelerating learning of characters in a native language can be expected. In addition, when the child is of an age at which the child can read the text of the native language to some extent, the text of the native language and the translated text of the other language may be output as a set. According to this configuration, the effect of learning the language other than the native language can be determined.

The amount, type, and expression method of information to be notified to the child may be changed according to the degree of waking up, posture, and riding time of the child while riding. The information amount can also be reduced in a state where the child is drowsy compared to a state of waking up. Alternatively, the amount of text may be reduced and the image may be increased. According to this configuration, the concern that the child is bothered by the excessive information can be reduced. In addition, when the riding posture of the child is not correct, the child is highly tired. Therefore, even when the riding posture of the child is not correct, the amount of information to be presented can be reduced as compared with when the riding posture of the child is not correct. Further, the information amount may be reduced as the elapsed time from the riding (that is, riding time) is longer. In addition, in the configuration of the list of video contents related to the object of interest on the assumption that the child is presented with the list, the child may be presented with the content whose playback time is shorter than the remaining time until the destination is reached.

The amount and type of information that the driver can notify the child may be set by the rear seat display 35 or the like through a predetermined setting screen. The information such as the age of the child may be manually input by the driver via a setting screen or the like, or may be estimated by the HCU1 from an image of the child camera 25 or the like.

In addition, it is also possible to base on whether "what is that? "such a phrase composed of a plurality of words, that is, a compound phrase, detects an interest response, and changes the information amount of the notification content to the child. Since being able to speak a compound phrase means being able to expect to remember the corresponding word, more detailed explanatory information can also be notified. On the other hand, in the case where an interest reaction is detected from a speech other than the compound phrase, the notification content may also be kept as relatively simple information. In this way, the amount of information to be notified to the child may be increased or decreased according to the content of the utterance used as the detection trigger of the interest response.

In addition, the information to be notified to the child may be controlled according to whether or not the interest response is detected using the speech of the child. For example, information on the object of interest detected in a state where the child is not speaking may be retained as the image display. This is because there is a relatively high concern of false detection without accompanying the speech of the child, and there is a concern that the user including the child is bothered by performing the voice output. According to the above-described structure, in the case where the concern of false detection is relatively high, the concern of annoying the user can be reduced by omitting the voice output.

The HCU1 may evaluate the attention level based on the length of time for which the same object is focused, the excitation level shown by biological information such as the content of speech and heart rate, and the like, and may omit the speech output to the object of interest when the attention level is equal to or less than a predetermined threshold. According to this structure, it is also possible to reduce the concern of voice output of information on an object of no interest or an object of weak interest. In addition, as a result, the concern that the user is bothered can be reduced.

Control example of notification content to driver

The HCU1 may display status information indicating the body temperature, drowsiness, etc. of the child on a display for the driver such as the HUD34 at all times or at predetermined notification intervals when no interest reaction is detected. According to such a configuration, it is easy to recognize the state of the child such as drowsiness and athermal. In addition, a call corresponding to the state of the child is easily made.

< supplement of Notification control to driver >

In the above-described embodiment, a control example has been disclosed in which the presentation method of the information on the object of interest to the driver is changed based on the determination result of whether the driving load is high or not, but the parameter for switching the presentation method of the information to the driver is not limited to the level of the driving load. For example, the HCU1 may change the operation when information on the object of interest is presented to the driver according to whether the vehicle is traveling. For example, both the image display and the voice output may be performed during the stop, whereas only the voice output may be retained without performing the image display when the vehicle is traveling.

The HCU1 may change the operation when presenting information on the object of interest to the driver, depending on whether or not the vehicle is being driven automatically at level 3 or more. For example, both the image display and the voice output may be performed when the vehicle is in automatic driving of level 3 or more, while only the voice output may be retained without performing the image display when the vehicle is not in automatic driving.

The HCU1 may change the combination of the display destinations of the images of the object of interest by using, in addition to the running state of the vehicle, whether or not the passenger seat is occupied by another passenger. For example, when the passenger is seated in the passenger seat, information on the object of interest may be displayed only on the passenger seat display and the child display during the automatic driving, and the information image may be displayed on the driver display during the automatic driving.

In addition, even in a state where it is determined that the driving load of the driver is not high, when the timing of the permission transition from the system to the driver is close, the information presentation to the driver regarding the object of interest may be omitted. When the remaining time until the permission transfer is less than the predetermined value, information presentation to the driver of the object of interest is avoided, and the driver is likely to concentrate on preparation for resuming the driving operation.

The HCU1 may change the location and notification mode of displaying information on the object of interest according to the driving state of the driver. The HCU1 may be configured to be able to adjust the timing of image display in accordance with the situation of the driver and the instruction of the driver. The instruction operation of the driver can be received by voice, touch, gesture, or the like. As selection candidates for the display timing, immediate, after 5 minutes, when temporarily stopped, when the driving load is reduced, when automatic driving is started, and the like can be employed.

< utilization of past detection results >

The HCU1 may adjust a threshold value for detecting the interest response of the child based on the past detection result of the object of interest. For example, the interest response detection unit F5 may decrease the threshold value for detecting the interest response of the child when the child travels in the vicinity of the object that has previously exhibited the interest response. The object of interest management unit FB may register an object that has previously exhibited a plurality of interest reactions as a desired object that is an object of particular interest. The HCU1 may notify the child of the presence of the intended object when the remaining distance from the intended object is less than the predetermined distance or when the intended object is captured by the off-vehicle camera 28. The HCU1 may also be configured to learn objects of interest to the child based on past detection results.

In the case where the HCU1 is configured to be able to manage the interest object category, if it is detected that the child is interested in an object belonging to the existing interest object category, the HCU may notify the driver in a manner different from that in the normal case. The term "usual time" as used herein refers to a case where a child has an interest in an object belonging to an existing interest object category. According to this structure, the driver can understand that the child starts to be interested in a new thing different from the previous one, in other words, can understand the transition of the object of interest accompanying the growth. In addition, according to the above-described structure, in one aspect, the driver can understand the object of interest of the child that the driver does not grasp by himself. In other words, the management of the object of interest categories based on the HCU1 described above may help solve the unexpected side of children.

Use of object of interest for preset child

In the above, the manner in which the HCU1 learns the child's interest from the past detection results is mentioned, but is not limited thereto. Information on favorite things and hobbies of the child may be registered in advance in the HCU1 and the server 5 via a predetermined setting screen.

When data related to a favorite object of a child is registered by manual registration or automatic learning, the object specifying unit F6 may be configured to detect the object of interest based on the data. That is, the object related to the favorite thing of the child may be preferentially extracted from the camera image corresponding to the line of sight of the child as the object of interest. According to this structure, the concern of extracting an object of interest that is not interesting/of weak interest to the child can be reduced. In other words, erroneous operation of the system can be suppressed. The HCU1 may change the amount of information to be notified to the child, the expression method, and the like, according to the preference information of the child. More/detailed information may be displayed when the object of interest is a favorite of the child than when it is not. In addition, when the object of interest is a favorite object of a child, a notification sound may be generated and an image related to the object of interest may be displayed, whereas when the object of interest is not a favorite object of a child, an image related to the object of interest may be displayed without a notification sound.

< supplement of communication method >)

The driver and the child may be configured to talk through the camera and the microphone. For example, when the talk function is set to on, an image of the child camera 25 is displayed on the meter display 32 and the voice acquired by the child microphone 27 is output from the speaker 38A. In addition, an image of the DSM21 is displayed on the rear seat display 35 and the voice acquired by the driver microphone 23 is output from the speaker 38B. According to this configuration, even when the child is seated in the third and subsequent rows of seats, for example, and the driver's seat is away from the child seat, communication is easily achieved.

< control of shooting Range of Camera for Children)

The HCU1 may be configured to be able to change the shooting direction and the magnification of the child camera 25 in accordance with the voice command of the driver acquired via the driver microphone 23. The shooting direction of the child camera 25 is expressed by a pitch angle, a roll angle, a yaw angle, or the like. Changing the shooting direction of the child camera 25 corresponds to changing the attitude angle. The shooting direction of the child camera 25 can be achieved by controlling, for example, a motor that controls the posture of the child camera 25.

According to the above configuration, the driver can easily confirm the expression of the child, etc., by adjusting the shooting range of the child camera 25 by voice. The control of the photographing range of the child camera 25 is not limited to voice input, and may be executed via an operation member such as a haptic device.

< appendix >

The apparatus, systems, and methods described in this disclosure may be implemented by special purpose computers that constitute a processor programmed to perform one or more functions embodied by a computer program. In addition, the apparatus described in the present disclosure and the method thereof may also be implemented using dedicated hardware logic circuits. Moreover, the apparatus described in this disclosure, as well as the methods thereof, may also be implemented by one or more special purpose computers, which are comprised of a combination of one or more hardware logic circuits and a processor for executing a computer program. In addition, a computer program as instructions to be executed by a computer may be stored in a computer-readable non-transitory tangible storage medium. That is, the devices and/or functions provided by the HCU1, etc. may be provided by software, software only, hardware only, or a combination thereof stored in the tangible storage device and computer for executing them. For example, part or all of the functions provided by the HCU1 may be implemented as hardware. Aspects in which a particular function is implemented by hardware include aspects in which the function is implemented using one or more ICs or the like. HCU1 may also be implemented using an MPU, GPU, or DFP (Data Flow Processor: data stream processor) instead of a CPU. The HCU1 may be implemented by combining a plurality of types of arithmetic processing devices such as CPU, MPU, GPU. The HCU1 may also be implemented as a System-on-Chip (SoC). The various processing units may be realized by using an FPGA (Field-Programmable Gate Array: field programmable gate array) or an ASIC (Application Specific Integrated Circuit: application specific integrated circuit). Various programs are stored on a non-transitory physical recording medium (non-transitory tangible storage medium). As a storage medium for the program, various storage media such as HDD (Hard-disk Drive), SSD (Solid State Drive) and flash memory, SD (Secure Digital) card can be used. The non-transitory physical recording medium also includes a ROM such as EPROM (Erasable Programmable Read Only Memory: erasable programmable read only memory).

The functions of one component in the above embodiments may be realized by a plurality of components, or one function of one component may be realized by a plurality of components. In addition, a plurality of functions of a plurality of components may be realized by one component, or one function realized by a plurality of components may be realized by one component. In addition, a part of the structure of the above embodiment may be omitted. At least a part of the structure of the above embodiment may be added or replaced with the other structure of the above embodiment.

Various modes such as a system in which the above-described object of interest sharing apparatus is used as a constituent element are also included in the scope of the present disclosure. For example, a program for causing a computer to function as an object of interest sharing apparatus, a non-transitory entity recording medium such as a semiconductor memory in which the program is recorded, and the like are also included in the scope of the present disclosure.

Claims

1. An object of interest sharing device for use in a vehicle provided with a seat for a child to sit on, that is, a child seat, comprising:

A child information acquisition unit (F1) that acquires, as information indicating the state of a child sitting in the child seat, the direction of the line of sight of the child from an image of an in-vehicle camera (25) that includes at least the face of the child in a shooting range;

an interest response detection unit (F5) that detects an interest response of the child to an object outside the vehicle compartment based on at least one of the biological information of the child, the voice of the child, and the visual line direction;

an object-of-interest detection unit (F6) that detects an object of interest, which is the object of interest of the child, from the line-of-sight direction acquired by the child information acquisition unit and a captured image of an off-vehicle camera (28) that is mounted on the vehicle so as to capture the outside of the vehicle;

an object information acquisition unit (F7) that acquires information that has been linguistic for the object of interest from a database that is disposed inside or outside the vehicle; and

and a notification processing unit (F9) for notifying at least one of the child and the driver's seat occupant, using at least one of a display of text and a voice output corresponding to the information, with respect to the information acquired by the object information acquisition unit.

2. The object of interest sharing apparatus as defined in claim 1, wherein,

the object information acquiring unit acquires the type of the object of interest as information on the object of interest,

the object-of-interest detecting unit acquires a direction in which the object of interest exists with respect to the vehicle,

the notification processing unit is configured to notify the driver seat occupant of the type of the object of interest and the direction in which the object of interest exists.

3. The object of interest sharing apparatus as defined in claim 1 or 2, wherein,

the object of interest detection unit is configured to be able to detect at least one of a pedestrian, another vehicle, an animal, an event, a natural phenomenon, and a drawing of a predetermined character as the object of interest,

the object information acquiring unit acquires at least any one of a name, a feature, a manufacturer, a country of origin, a reason for generation, a size, a year of construction, an effect, and a history background of the object of interest as information on the object of interest,

the notification processing unit is configured to display a text corresponding to the information acquired by the object information acquisition unit on a predetermined display together with the image of the object of interest.

4. The object of interest sharing apparatus according to any one of claims 1 to 3, comprising:

an object of interest management unit (FB) for managing a detection result of the object of interest,

the object of interest management unit is configured to:

based on past histories of detection results of the object of interest, an object of interest class, which is a class of the object of interest to the child, is determined, and

when the newly detected object of interest does not belong to the existing object of interest category, the driver seat occupant is notified in a manner different from the case where the newly detected object of interest belongs to the existing object of interest category.

5. The object of interest sharing apparatus according to any one of claims 1 to 4, comprising:

a recording processing unit (FA) for associating the object of interest with the position information and recording the same in a predetermined storage device,

the notification processing unit is configured to notify at least one of the child and the driver seat occupant of the presence of the target object based on the approaching or approaching of the target object within a predetermined distance from the target object detected in the past.

6. The object of interest sharing apparatus according to any one of claims 1 to 5, comprising:

A driver seat occupant information acquisition unit (F3) that acquires information indicating the state of the driver seat occupant, including a voice uttered by the driver seat occupant,

the notification processing unit is configured to output the information acquired by the target information acquisition unit in a voice manner based on the fact that the driver seat occupant has not acquired the speech of the driver seat occupant after a predetermined waiting time has elapsed since the driver seat occupant information acquisition unit detected the interest response of the child.

7. The object of interest sharing apparatus as defined in claim 6, wherein,

the driver-seat occupant information acquisition unit is configured to be able to acquire information indicating a driving load of the driver-seat occupant,

the notification processing unit is configured to:

when the driver-seat-occupant-information acquiring unit does not acquire information indicating that the driving load of the driver-seat occupant is high, the driver-seat-occupant-information acquiring unit outputs the information acquired by the target-information acquiring unit in voice under the condition that the response waiting time has not elapsed since the detection of the interest reaction has not acquired the speech of the driver-seat occupant,

On the other hand, when the driver-seat occupant information acquisition unit acquires information indicating that the driving load of the driver-seat occupant is high, the information acquired by the target-information acquisition unit is output in a voice from a speaker provided in the vehicle without waiting for the elapse of the response waiting time.

8. The object of interest sharing apparatus as defined in claim 7, wherein,

the notification processing unit is configured to:

according to the interest reaction detected by the interest reaction detecting unit, information on the object of interest including the information acquired by the object information acquiring unit is displayed on a display visually identifiable by the driver's seat occupant and on a display visually identifiable by the child, respectively,

when the driver-seat occupant information acquisition unit does not acquire the information indicating the high driving load of the driver-seat occupant, the information acquired by the target-information acquisition unit is output in voice on the condition that the driver-seat occupant information acquisition unit has not acquired the speech of the driver-seat occupant after the response waiting time has elapsed since the display of the information on the target of interest was displayed on the display.

9. The object of interest sharing apparatus according to any one of claims 1 to 8, wherein,

the notification processing unit is configured to change a notification mode of the information on the object of interest to the child based on at least one of an age, a preference, an elapsed time from a ride of the child, and a remaining time from a destination of the child.

10. An object of interest sharing method performed by at least one processor for a guardian to share an object of interest exhibited by a child seated in a child seat previously set in a vehicle, comprising:

acquiring a line-of-sight direction of a child seated in the child seat based on an image of an in-vehicle camera (25) including at least a face of the child in a photographing range (S101);

detecting an interest reaction of the child to an object outside the vehicle compartment based on at least one of the biological information of the child, the voice of the child, and the visual line direction (S104);

detecting an object of interest, which is an object of interest of the child, based on the acquired line-of-sight direction of the child and a captured image of an off-vehicle camera (28) mounted on the vehicle so as to capture the outside of the vehicle (S105);

Acquiring information linguistically related to the object of interest from a database disposed inside or outside the vehicle (S107); and

for the acquired information on the object of interest, at least one of the driver seat occupant and the child is notified using at least one of a display of text and a voice output corresponding to the information (S110, S111, S113).