WO2022270379A1

WO2022270379A1 - In-vehicle device, notification method for object, program, and system for vehicle

Info

Publication number: WO2022270379A1
Application number: PCT/JP2022/023959
Authority: WO
Inventors: 浩一郎竹内; 正也伊藤
Original assignee: 株式会社デンソー
Priority date: 2021-06-24
Filing date: 2022-06-15
Publication date: 2022-12-29
Also published as: JP2023003665A; JP7439797B2

Abstract

An in-vehicle device (3) comprises an image acquisition unit, a selection factor acquisition unit, a model selection unit, and an object recognition unit. The selection factor acquisition unit acquires a selection factor including at least one of a vehicle environment and a relative location of an object with respect to a camera (15). The model selection unit selects an image recognition model according to the selection factor. The object recognition unit performs a process for recognizing an object by using the selected image recognition model to perform image recognition. If an object is recognized, a notification is sent to a cloud (5) via a communication unit.

Description

In-vehicle device, object notification method, program, and vehicle system

Cross-reference to related applications

This international application claims priority based on Japanese Patent Application No. 2021-104882 filed with the Japan Patent Office on June 24, 2021, and Japanese Patent Application No. 2021-104882 The entire contents are incorporated by reference into this international application.

The present disclosure relates to an in-vehicle device, an object notification method, a program, and a vehicle system.

A vehicle external environment recognition device is disclosed in Patent Document 1. A vehicle external environment recognition device photographs the surroundings of the vehicle and generates an image. A vehicle external environment recognition device performs image recognition on an image to recognize an object. The vehicle's external environment recognition device switches the control parameters of the camera according to the vehicle's external environment.

JP 2014-178836 A

It is conceivable to perform image recognition on the images obtained by photographing the interior of the vehicle with a camera, and to recognize objects inside the vehicle. The following problems that arise in this case were found as a result of detailed studies by the inventors. The accuracy of image recognition is affected by the environment inside the vehicle, the distance from the camera to the object, and the like. Therefore, even if the control parameters of the camera are switched as in the technology described in Patent Document 1, the accuracy of image recognition may be low.

In one aspect of the present disclosure, it is preferable to provide an in-vehicle device, an object notification method, a program, and a vehicle system capable of recognizing an object in a vehicle interior with high accuracy.

One aspect of the present disclosure is an in-vehicle device. The in-vehicle device is configured to communicate between the user's mobile terminal and the communicable cloud via the communication unit. The in-vehicle device includes an image acquisition unit configured to acquire an image of the interior of the vehicle captured by a camera, a vehicle environment that is the environment of the vehicle, and an object in the interior of the vehicle based on the camera. a selection factor acquisition unit configured to acquire a selection factor including at least one of the relative positions of the two; and a model configured to select an image recognition model according to the selection factor acquired by the selection factor acquisition unit. a selection unit, and an object recognition configured to perform image recognition using the image recognition model selected by the model selection unit on the image acquired by the image acquisition unit, and to perform a process of recognizing the object. a unit; When the object recognition unit recognizes the object that satisfies a predetermined condition, it notifies the cloud via the communication unit of information on the recognition result of the object that leads to notification to the portable terminal.

An in-vehicle device, which is one aspect of the present disclosure, can select an appropriate image recognition model and perform image recognition according to the vehicle environment and the relative positions of objects. Therefore, the in-vehicle device can recognize objects in the vehicle interior with high accuracy. As a result, the in-vehicle device can accurately notify the mobile terminal.

Another aspect of the present disclosure is to acquire an image of the interior of a vehicle with a camera, and obtain at least the vehicle environment, which is the environment of the vehicle, and the relative position of an object in the interior of the vehicle with respect to the camera. Acquire a selection factor including one, select an image recognition model according to the selection factor, perform image recognition on the image using the selected image recognition model, perform processing to recognize the object, Notifying a cloud that can communicate with the mobile terminal, via a communication unit, of information about a recognition result of the object that will lead to notification to the user's mobile terminal when the object that satisfies a predetermined condition is recognized. The method.

According to the object notification method, which is another aspect of the present disclosure, it is possible to select an appropriate image recognition model and perform image recognition according to the vehicle environment and the relative position of the object. Therefore, objects in the vehicle interior can be recognized with high accuracy. As a result, it is possible to accurately notify the mobile terminal.

Another aspect of the present disclosure is a vehicle system comprising a cloud capable of communicating with a user's mobile terminal, and an in-vehicle device configured to communicate with the cloud via a communication unit.

The in-vehicle device includes an image acquisition unit configured to acquire an image of an interior of a vehicle captured by a camera, a vehicle environment that is the environment of the vehicle, and an object in the interior of the vehicle as a reference to the camera. a selection factor acquisition unit configured to acquire a selection factor including at least one of the relative positions of an object configured to perform image recognition on the image acquired by the model selection unit and the image acquired by the image acquisition unit using the image recognition model selected by the model selection unit, and to perform processing to recognize the object. a recognition unit;

When the object recognition unit recognizes the object that satisfies a predetermined condition, the in-vehicle device notifies the cloud of information regarding the recognition result of the object via the communication unit.

The cloud is configured to notify the portable terminal based on the information about the recognition result of the object notified from the in-vehicle device.

A vehicle system, which is another aspect of the present disclosure, can select an appropriate image recognition model and perform image recognition according to the vehicle environment and relative positions of objects. Therefore, the vehicle system can recognize objects in the vehicle interior with high accuracy. As a result, the vehicle system can accurately notify the mobile terminal.

1 is a block diagram showing the configuration of a mobility IoT system; FIG. 3 is a block diagram showing a functional configuration of a controller of the in-vehicle device; FIG. It is explanatory drawing showing arrangement|positioning of the sensor in a vehicle interior, a camera, and an object. It is a flow chart showing processing which an in-vehicle device performs.

Exemplary embodiments of the present disclosure are described with reference to the drawings.
<First embodiment>
1. Configuration of Mobility IoT System 1 The configuration of the mobility IoT system 1 will be described based on FIG. IoT is an abbreviation for Internet of Things. The mobility IoT system 1 includes an in-vehicle device 3 , a cloud 5 and a service providing server 7 . Although only one in-vehicle device 3 is shown in FIG. 1 for convenience, the mobility IoT system 1 includes a plurality of in-vehicle devices 3, for example. The multiple in-vehicle devices 3 are mounted in different vehicles 9, respectively. A combination of the in-vehicle device 3 and the cloud 5 corresponds to a vehicle system.

The in-vehicle device 3 can communicate with the cloud 5 via a communication device 19 mounted on the vehicle 9. Detailed configurations of the in-vehicle device 3 and the vehicle 9 will be described later.

The cloud 5 can communicate with the in-vehicle device 3, the service providing server 7, and the mobile terminal 23. The mobile terminal 23 is, for example, a mobile terminal owned by the user of the vehicle 9 . Examples of the mobile terminal 23 include a smart phone, a tablet terminal, a notebook PC, and the like.

The cloud 5 includes a control unit 25, a communication unit 27, and a storage unit 29. The control unit 25 includes a CPU 31 and a semiconductor memory such as RAM or ROM (hereinafter referred to as memory 33). The functions of the control unit 25 are implemented by the CPU 31 executing programs stored in the memory 33 . Also, by executing this program, a method corresponding to the program is executed. The communication unit 27 can perform wireless communication with the communication device 19 and the mobile terminal 23 . The storage unit 29 can record information.

The service providing server 7 can communicate with the cloud 5. The service providing server 7 is, for example, a server installed to provide a service for managing operation of the vehicle 9 . Note that the mobility IoT system 1 may include a plurality of service providing servers 7 having different service contents.

The cloud 5 collects vehicle 9 data transmitted from each of the multiple in-vehicle devices 3 via the communication device 19 . The cloud 5 stores the collected data in the storage unit 29 for each vehicle 9 .

The cloud 5 creates a digital twin based on the data of the vehicle 9 stored in the storage unit 29. A digital twin is normalized index data. The service providing server 7 can acquire the data of the predetermined vehicle stored in the storage unit 29 using the index data acquired from the digital twin. The service providing server 7 determines control details of the vehicle 9 and transmits instructions corresponding to the control details to the cloud 5 . The cloud 5 transmits control contents to the vehicle 9 based on the instruction.

2. Configurations of In-vehicle Device 3 and Vehicle 9 Configurations of the in-vehicle device 3 and vehicle 9 will be described with reference to FIGS. 1 to 3. FIG. As shown in FIG. 1, the in-vehicle device 3 includes a control unit 35, a storage unit 37, and a clock 38. The control unit 35 includes a CPU 39 and a semiconductor memory such as RAM or ROM (hereinafter referred to as memory 41). The functions of the control unit 35 are realized by the CPU 39 executing programs stored in the memory 41 . Also, by executing this program, a method corresponding to the program is executed.

The storage unit 37 can store information. The storage unit 37 stores a plurality of image recognition models. The image recognition model is used to recognize the object 57 by performing image recognition on an image generated using the camera 15 mounted on the vehicle 9 . The clock 38 transmits time information representing time to the control unit 35 .

As shown in FIG. 2, the functional configuration of the control unit 35 includes an image acquisition unit 43, a communication unit 45, a selection factor acquisition unit 47, a model selection unit 49, an object recognition unit 51, and a vehicle state acquisition unit. a unit 53;

The image acquisition unit 43 uses the camera 15 to photograph the inside of the cabin 55 of the vehicle 9 and acquires the image. The communication unit 45 communicates with the cloud 5 .

The selection factor acquisition unit 47 acquires selection factors. The selection factor is at least one of the vehicle environment, which is the environment of the vehicle 9, and the relative position of the object 57 in the cabin 55 of the vehicle 9 with respect to the camera 15 (hereinafter referred to as the relative position of the object). include. The selection factor affects object recognition accuracy when image recognition is performed on an image captured by the camera 15 .

The vehicle environment is the vehicle environment that affects the recognition accuracy when image recognition is performed on images captured by the camera 15 . The vehicle environment is, for example, the vehicle environment that affects the brightness in the vehicle interior 55, the brightness around the vehicle 9, the direction in which light enters the vehicle interior 55, and the like. The vehicle environment includes, for example, the position of the vehicle 9, the direction or direction of travel of the vehicle 9, the current time, the weather, and the state of the room lamps 17, and the like.

The position of the vehicle 9 and the traveling direction of the vehicle 9 affect the brightness in the vehicle interior 55 and the direction in which light enters the vehicle interior 55 . The current time affects the brightness in the vehicle interior 55 and the direction in which light enters the vehicle interior 55 . The weather affects the brightness inside the vehicle compartment 55 . Weather includes, for example, fine weather, cloudy weather, rainy weather, fog, and the like. The state of the room lamp 17 affects the brightness in the passenger compartment 55 . The state of the room lamp 17 includes, for example, a lit state, an extinguished state, a state in which the amount of light is within a specific range, and the like. When there are a plurality of room lamps 17, the state of each room lamp 17 corresponds to the vehicle environment.

The relative position of the object consists of the distance from the camera 15 to the object 57 and the direction of the object 57 with respect to the camera 15. The relative position of the object affects the size of object 57 in the image produced by camera 15 . The farther the object 57 is from the camera 15, the smaller the object 57 is in the image produced by the camera 15. FIG. In addition to the relative positions of the objects, if the direction of light incident on the vehicle interior 55 is known as the vehicle environment, for example, the state in which the light is irradiated to the object 57 and the state in which the light is irradiated from behind the object 57 can be determined. , backlit conditions, etc. can be seen. The relative position of the object affects the recognition accuracy when image recognition is performed on an image captured by the camera 15 .

A plurality of image recognition models stored in the storage unit 37 are each associated with a selection factor. The image recognition model is, for example, an image recognition model obtained by learning in the selection factor associated therewith. Using the image recognition model in its associated selection factors, the object 57 can be recognized with high accuracy.

For example, if the image recognition model is used in the vehicle environment included in the selection factors associated with it, the object 57 can be recognized with high accuracy. Also, for example, if the image recognition model is used in the relative position of the object included in the selection factors associated with it, the object 57 can be recognized with high accuracy.

The model selection unit 49 selects an image recognition model according to the selection factors acquired by the selection factor acquisition unit 47 .

For example, the model selection unit 49 selects an image recognition model according to the vehicle environment. Specifically, when the vehicle interior 55 is dark at night, the model selection unit 49 selects an image recognition model for dark camera images. In addition, the model selection unit 49 selects the following when the interior of the vehicle interior 55 is bright during the day, when the interior of the vehicle interior 55 is bright due to sunset, and when the interior of the vehicle interior 55 is bright due to lighting of the room lamp 17 at night. Select the appropriate image recognition model for each case.

Also, for example, the model selection unit 49 selects an image recognition model according to the relative position of the object. Specifically, when the distance from the camera 15 to the object 57 is long, the model selection unit 49 selects an image recognition model for camera images in which the object 57 is small. Also, when the distance from the camera 15 to the object 57 is short, the model selection unit 49 selects an image recognition model for camera images in which the object 57 is captured large. In addition, the model selection unit 49 selects an image recognition model suitable for each case, such as when the object 57 is in front of the camera 15 and when the object 57 is oblique or close to the side of the camera 15. . By selecting the image recognition model using the vehicle environment and the relative position of the object as selection factors, the object can be recognized with higher accuracy.

The object recognition unit 51 performs image recognition on the image acquired by the image acquisition unit 43 using the image recognition model selected by the model selection unit 49, and performs processing for recognizing the object 57.

The vehicle state acquisition unit 53 acquires a signal representing the state of the vehicle 9 from the vehicle ECU 11. The state of the vehicle 9 includes, for example, engine on, start of running, stop, key lock, and the like.

As shown in FIG. 1, the vehicle 9 includes a vehicle ECU 11, a sensor 13, a camera 15, a room lamp 17, a communication device 19, and a navigation system 21 in addition to the in-vehicle device 3. The vehicle-mounted device 3 can communicate with each of the vehicle ECU 11 , the sensor 13 , the camera 15 , the room lamp 17 , the communication device 19 , and the navigation 21 .

The vehicle ECU 11 detects the state of the vehicle 9 and transmits a signal representing the detected state to the vehicle state acquisition unit 53 . For example, the vehicle ECU 11 detects the vehicle speed of the vehicle 9 . When the vehicle speed of the vehicle 9 changes from 0 Km/h to a threshold value or more, the vehicle ECU 11 transmits a signal indicating start of running to the vehicle state acquisition unit 53 . Further, when the vehicle speed of the vehicle 9 changes from the threshold value or higher to 0 Km/h, the vehicle ECU 11 transmits a signal indicating that the vehicle is stopped to the vehicle state acquisition unit 53 .

The vehicle ECU 11 detects, for example, the shift position. When the shift position changes to parking, the vehicle ECU 11 transmits a signal indicating that the vehicle is stopped to the vehicle state acquisition unit 53 .

　As shown in FIG. 3, the sensor 13 and the camera 15 are installed inside the vehicle compartment 55 of the vehicle 9, respectively. The position of sensor 13 is close to the position of camera 15 . For example, the sensor 13 and camera 15 are installed above the windshield or near the rearview mirror. Also, the relative position of the camera 15 with respect to the position of the sensor 13 is constant.

The sensor 13 is, for example, a millimeter wave radar. The sensor 13 can detect the object 57 in the vehicle interior 55 by comparing the detection result of the sensor 13 with the reference data. The reference data is the detection result of the sensor 13 when the object 57 does not exist inside the vehicle compartment 55 . Objects 57 include, for example, things (excluding people and animals), people, animals, and the like. The object is, for example, an object that can be carried by the user of the vehicle 9 . Humans include, for example, infants. The sensor 13 can calculate the presence or absence of an object and the relative position of the object.

The camera 15 is installed inside the vehicle compartment 55 . The photographing range of the camera 15 includes a range within the vehicle interior 55 where the object 57 is likely to be placed. The photographing range of the camera 15 includes, for example, part or all of the driver's seat, front passenger's seat, rear seat, and dashboard.

One or more room lamps 17 are installed in the vehicle interior 55 . The room lamp 17 can be turned on and off. Also, the room lamp 17 can be adjusted in light quantity. The room lamp 17 transmits a signal representing the state of the room lamp 17 to the controller 35 .

The communication device 19 can communicate with the communication unit 27 of the cloud 5. Navi 21 has the functions of a normal navigation system. The navigation 21 can acquire the position of the vehicle 9 and the traveling direction of the vehicle 9 based on the data received from the positioning satellites.

3. Processing Executed by In-vehicle Device 3 Processing executed by the in-vehicle device 3 will be described with reference to FIG. The process shown in FIG. 4 is executed, for example, when the in-vehicle device 3 in sleep mode is activated. The in-vehicle device 3 is activated, for example, as follows. When the door of the vehicle 9 is unlocked, the vehicle ECU 11 is activated. The activated vehicle ECU 11 activates the in-vehicle device 3 . Note that the in-vehicle device 3 enters a sleep state when the processing shown in FIG. 4 is completed.

In step 1 of FIG. 4, the vehicle state acquisition unit 53 determines whether the engine of the vehicle 9 has been turned on and whether the vehicle 9 has started running, based on the signal acquired from the vehicle ECU 11. If it is determined that the engine of the vehicle 9 has been turned on and that the vehicle 9 has started running, the process proceeds to step 2 . If it is determined that the engine of the vehicle 9 is not on, or if it is determined that the vehicle 9 has not started running, the process returns to step 1 .

In step 2, the selection factor acquisition unit 47 uses the sensor 13 to acquire the relative position of the object.

In step 3, the selection factor acquisition unit 47 acquires the vehicle environment. For example, the selection factor acquisition unit 47 acquires the position of the vehicle 9 and the traveling direction of the vehicle 9 using the navigation system 21 . The selection factor acquisition unit 47 also acquires the current time using the clock 38, for example. Also, the selection factor acquisition unit 47 acquires the weather by communicating with the cloud 5 via the communication device 19, for example. The weather is, for example, the current weather at the location of the vehicle 9 . Further, the selection factor acquisition unit 47 acquires the state of the room lamp 17 from the room lamp 17, for example. Further, the selection factor acquisition unit 47 may, for example, photograph the interior of the vehicle interior 55 with the camera 15 , analyze the photographed image, and detect the brightness within the vehicle interior 55 . Note that the navigation 21, the clock 38, and the room lamp 17 correspond to devices mounted on the vehicle 9. FIG. Cloud 5 corresponds to the exterior of vehicle 9 . Note that the selection factor acquisition unit 47 may perform only one of the processes of steps 2 and 3.

In step 4, the model selection unit 49 selects an image recognition model according to selection factors. The selection factors include both the relative position of the object obtained in step 2 above and the vehicle environment obtained in step 3 above.

As described above, the plurality of image recognition models stored in the storage unit 37 are each associated with a selection factor. A model selection unit 49 selects an image recognition model associated with a selection factor from among a plurality of image recognition models stored in the storage unit 37 .

In step 5, the vehicle state acquisition unit 53 determines whether or not the vehicle 9 has finished traveling based on the signal acquired from the vehicle ECU 11 . If it is determined that the vehicle 9 has finished traveling, the process proceeds to step 6 . If it is determined that the vehicle 9 has not finished running, the process proceeds to step 2 .

In step 6, the vehicle state acquisition unit 53 determines whether the vehicle 9 has stopped and whether the key lock has been turned on, based on the signal obtained from the vehicle ECU 11 . If it is determined that the vehicle 9 has stopped and that the key lock has been turned on, the process proceeds to step 7 . If it is determined that the vehicle 9 is not stopped or if it is determined that the key lock is not turned on, the process returns to before step 6 .

In step 7, the image acquisition unit 43 uses the camera 15 to photograph the inside of the vehicle interior 55 and acquires the image.

In step 8, the object recognition unit 51 performs image recognition on the image acquired in step 7 using the image recognition model selected in step 4, and performs processing to recognize the object 57.

In step 9, the object recognition unit 51 determines whether or not the specific object 57 has been recognized in the process of step 8. If it is determined that the specific object 57 has been recognized, the process proceeds to step 10 . Specific objects 57 are, for example, children, animals, and baggage. If it is determined that the specific object 57 has not been recognized, this process ends. A specific object 57 corresponds to an object that satisfies a predetermined condition.

In step 10, the communication unit 45 transmits information about the specific object 57 to the cloud 5 in order to notify the mobile terminal 23 from the cloud 5. The processing from step 1 to this point corresponds to the notification method. Upon receiving the information, the cloud 5 notifies the portable terminal 23 that a child, pet, or baggage remains in the passenger compartment 55 . The information transmitted by the communication unit 45 corresponds to information regarding the recognition result of the object 57 leading to notification to the mobile terminal 23 . The mobile terminal 23 displays, for example, a notification image, and generates sound, vibration, or the like. A user of the vehicle 9 can know that there is an object 57 in the vehicle interior 55 from the notification image, sound, vibration, or the like. The communication unit 45 transmits, for example, the camera image obtained in step 7 or an image obtained by processing it to the cloud 5 . Cloud 5 transmits those images to mobile terminal 23 . The mobile terminal 23 displays the camera image obtained in step 7 or an image obtained by processing it. A user of the vehicle 9 can know that the object 57 is inside the vehicle compartment 55 and what the object 57 is by looking at the displayed image.

4. Effects of In-vehicle Device 3 (1A) The in-vehicle device 3 uses the camera 15 to photograph the inside of the vehicle interior 55 to obtain an image. The in-vehicle device 3 acquires the selection factor. The in-vehicle device 3 selects an image recognition model according to selection factors. The in-vehicle device 3 performs image recognition on the acquired image using the selected image recognition model, and performs processing for recognizing the object 57 . When the in-vehicle device 3 recognizes the object 57 , the in-vehicle device 3 notifies the portable terminal 23 via the cloud 5 .

The in-vehicle device 3 can select an appropriate image recognition model and perform image recognition according to the vehicle environment and the relative positions of objects. Therefore, the in-vehicle device 3 can recognize the object 57 with high accuracy. As a result, the in-vehicle device 3 can accurately notify the mobile terminal 23 .

(1B) The in-vehicle device 3 acquires selection factors in steps 2 and 3 before acquiring an image in step 7 above. Also, the in-vehicle device 3 selects an image recognition model in step 4 before acquiring an image in step 7 .

Therefore, the time from acquiring an image to completing image recognition can be shortened compared to acquiring selection factors after acquiring an image or selecting an image recognition model after acquiring an image.

(1C) Selection factors include the position of the vehicle 9, the traveling direction of the vehicle 9, the current time, and the weather. Therefore, the in-vehicle device 3 can select an appropriate image recognition model according to the position of the vehicle 9, the traveling direction of the vehicle 9, the time of day, and the weather.

(1D) Selection factors include the state of the room lamp 17 . Therefore, the in-vehicle device 3 can select an appropriate image recognition model according to the state of the room lamp 17 .

(1E) The in-vehicle device 3 acquires the vehicle environment from outside the vehicle 9 . The outside of the vehicle 9 is the cloud 5, for example. Therefore, the in-vehicle device 3 can acquire a vehicle environment that is difficult to acquire in the vehicle 9 .

(1F) The in-vehicle device 3 acquires the vehicle environment from a device mounted in the vehicle 9. The devices mounted on the vehicle 9 are, for example, the on-vehicle device 3, the room lamp 17, the navigation 21, the clock 38, and the like. Therefore, the in-vehicle device 3 can acquire various vehicle environments.

(1G) The in-vehicle device 3 acquires the relative position of the object using the sensor 13 . Therefore, the in-vehicle device 3 can acquire the relative position of the object accurately and easily.
<Other embodiments>
Although the embodiments of the present disclosure have been described above, the present disclosure is not limited to the above-described embodiments, and various modifications can be made.

(1) In the first embodiment, selection factors included both the vehicle environment and the relative positions of objects. Selection factors may include the vehicle environment, but not the relative positions of objects. Also in this case, the in-vehicle device 3 can select an appropriate image recognition model according to the vehicle environment and perform image recognition.

Also, the selection factor includes the relative position of the object, but does not have to include the vehicle environment. Also in this case, the in-vehicle device 3 can select an appropriate image recognition model according to the relative position of the object and perform image recognition.

The vehicle environment does not have to include one or more of the position of the vehicle 9, the traveling direction of the vehicle 9, the time of day, and the weather. The vehicle environment may further include other elements.

(2) In the first embodiment, the in-vehicle device 3 acquires an image when the vehicle 9 stops and the key lock is turned on. The timing of acquiring the image may be another timing. For example, the in-vehicle device 3 may acquire images before the engine is turned on, when the vehicle is idling, during driving, during a period from when the vehicle is stopped until the key lock is turned on, and the like. Also in this case, the same effects as in the first embodiment can be obtained.

(3) In the first embodiment, the in-vehicle device 3 acquires selection factors in steps 2 and 3 before acquiring an image in step 7, and selects an image recognition model in step 4. did.

After acquiring the image, the in-vehicle device 3 may acquire selection factors and select an image recognition model. Also, the in-vehicle device 3 may perform processing in the order of acquisition of selection factors, acquisition of images, and selection of an image recognition model. Even in these cases, the in-vehicle device 3 can achieve the effects (1A), (1C) to (1G) of the first embodiment.

(4) In the first embodiment, the sensor 13 is used to obtain the relative position of the object. Sensor 13 was a millimeter wave radar. The sensor 13 may be a sensor other than the millimeter wave radar. Also, the method of acquiring the relative position of the object may be another method. For example, based on the image of the camera 15, the relative position of the object may be obtained.

(5) In step 10, the in-vehicle device 3 may perform other processing in addition to notifying the portable terminal 23. Other processing includes, for example, processing for sounding the horn of the vehicle 9 . If the object 57 is a human infant, the in-vehicle device 3 operates the air conditioner of the vehicle 9 to lower the temperature of the passenger compartment 55, opens the windows, unlocks the doors, etc. in step 10. It can be performed.

(6) The in-vehicle device 3 may have one or more functions of the sensor 13, the camera 15, the communication device 19, and the navigation 21.

(7) The selection factor acquisition unit 47 may acquire selection factors while the vehicle 9 is running. The model selection unit 49 may select an image recognition model while the vehicle 9 is running. After the vehicle 9 has traveled, the object recognition unit 51 may perform the process of recognizing the object while the vehicle 9 is parked. In this case, a notification can be given if an object 57 is left behind in the parked vehicle 9 .

(8) The controller 35 and its techniques described in the present disclosure may be implemented by a dedicated computer provided by configuring a processor with one or more dedicated hardware logic circuits. Alternatively, the controller 35 and techniques described in this disclosure are a combination of a processor and memory programmed to perform one or more functions and a processor configured by one or more hardware logic circuits. may be implemented by one or more dedicated computers configured by The method of realizing the function of each unit included in the control unit 35 does not necessarily include software, and all the functions may be realized using one or more pieces of hardware.

(9) A plurality of functions possessed by one component in the above embodiment may be realized by a plurality of components, or a function possessed by one component may be realized by a plurality of components. . Also, a plurality of functions possessed by a plurality of components may be realized by a single component, or a function realized by a plurality of components may be realized by a single component. Also, part of the configuration of the above embodiment may be omitted. Moreover, at least part of the configuration of the above embodiment may be added or replaced with respect to the configuration of the other above embodiment.

(10) In addition to the above-described in-vehicle device 3, a system having the in-vehicle device 3 as a component, a program for causing a computer to function as the control unit 35 of the in-vehicle device 3, a non-transition of a semiconductor memory storing this program, etc. The present disclosure can also be implemented in various forms such as a physical recording medium, an object recognition method, and the like.

Claims

An in-vehicle device (3) configured to communicate between a user's mobile terminal (23) and a communicable cloud (5) via a communication unit (45),
an image acquisition unit (43) configured to acquire an image of the interior (55) of the vehicle (9) captured by the camera (15);
A selection factor obtaining unit (47) configured to obtain a selection factor including at least one of a vehicle environment, which is the environment of the vehicle, and a position of an object (57) in the vehicle interior relative to the camera. )When,
a model selection unit (49) configured to select an image recognition model according to the selection factor obtained by the selection factor obtaining unit;
An object recognition unit (51) configured to perform image recognition on the image acquired by the image acquisition unit using the image recognition model selected by the model selection unit, and to perform processing for recognizing the object. When,
with
When the object recognition unit recognizes the object that satisfies a predetermined condition, notifies the cloud via the communication unit of information regarding the recognition result of the object that leads to notification to the mobile terminal.
In-vehicle device.
The in-vehicle device according to claim 1,
the selection factor obtaining unit is configured to obtain the selection factor before the image obtaining unit obtains the image;
wherein the model selection unit is configured to select the image recognition model before the image acquisition unit acquires the image;
In-vehicle device.
The in-vehicle device according to claim 1,
The selection factor acquisition unit is configured to acquire the selection factor while the vehicle is running;
the model selection unit is configured to select the image recognition model while the vehicle is running;
The object recognition unit is configured to perform a process of recognizing the object when the vehicle is parked after running the vehicle.
In-vehicle device.
The in-vehicle device according to any one of claims 1 to 3,
The selection factor includes the vehicle environment including one or more selected from the group consisting of the position of the vehicle, the direction of travel of the vehicle, the current time, and the weather.
In-vehicle device.
The in-vehicle device according to any one of claims 1 to 4,
The selection factor includes the vehicle environment including the state of a room lamp (17) provided in the vehicle compartment,
In-vehicle device.
The in-vehicle device according to any one of claims 1 to 5,
said selection factor obtaining unit is adapted to obtain said vehicle environment from outside (5) said vehicle;
In-vehicle device.
The in-vehicle device according to any one of claims 1 to 6,
wherein the selection factor acquisition unit is configured to acquire the vehicle environment from a device (3, 17, 21) onboard the vehicle;
In-vehicle device.
The in-vehicle device according to any one of claims 1 to 7,
wherein the selection factor acquisition unit is configured to acquire the relative position using a sensor (13);
In-vehicle device.
Acquiring an image of the interior (55) of the vehicle (9) captured by the camera (15),
obtaining a selection factor including at least one of a vehicle environment, which is the environment of the vehicle, and a relative position of an object (57) in the vehicle interior with respect to the camera;
selecting an image recognition model according to the selection factor;
performing image recognition on the image using the selected image recognition model to perform processing for recognizing the object;
When the object that satisfies a predetermined condition is recognized, information on the recognition result of the object, which leads to notification to the mobile terminal (23) of the user, is sent to the cloud (5) that can communicate with the mobile terminal via the communication unit. Notice,
Object notification method.
A program that causes a computer to function as a control unit of the in-vehicle device according to any one of claims 1 to 8.
For a vehicle comprising a cloud (5) communicable with a user's portable terminal (23) and an in-vehicle device (3) configured to communicate with the cloud via a communication unit (45) a system,
The in-vehicle device
an image acquisition unit (43) configured to acquire an image of the interior (55) of the vehicle (9) captured by the camera (15);
A selection factor obtaining unit (47) configured to obtain a selection factor including at least one of a vehicle environment, which is the environment of the vehicle, and a position of an object (57) in the vehicle interior relative to the camera. )When,
a model selection unit (49) configured to select an image recognition model according to the selection factor obtained by the selection factor obtaining unit;
An object recognition unit (51) configured to perform image recognition on the image acquired by the image acquisition unit using the image recognition model selected by the model selection unit, and to perform processing for recognizing the object. When,
with
when the object recognition unit recognizes the object that satisfies a predetermined condition, notifies the cloud of information about the recognition result of the object via the communication unit;
The cloud is
configured to notify the mobile terminal based on the information about the recognition result of the object notified from the in-vehicle device,
vehicle system.