JP2020201745A - Object identification device, method, and system - Google Patents

Abstract

To allow for an effective use of limited memory capacity and more accurate and faster object identification by preparing multiple types of dedicated identification models optimized for identification in advance for each object that may become an obstacle for driving and selectively downloading to each vehicle only some dedicated identification models according to the surrounding environment of each vehicle for object identification.SOLUTION: An object identification device includes: a plurality of identification models 107a, 107b, 107c; acquisition units 101, 102, 103, and 104 that acquire camera images and sensor outputs specific to the driving environment of a connected car; an ambient environment identification unit 105 that identifies features of the ambient environment based on the camera images and sensor outputs; a driving environment estimation unit 106 that estimates the driving environment based on the features of the ambient environment; an identification model selection unit 106 that selects a part of the identification models based on the estimation result of the driving environment; and an object identification unit 109 that identifies an object by applying the images of an in-vehicle camera to the selected identification model.SELECTED DRAWING: Figure 1

Description

The present invention relates to object identification devices, methods and systems, and more particularly to object identification devices, methods and systems suitable for automated driving of connected cars.

In recent years, various sensors identify objects existing in the traveling direction of the vehicle, and if the result of object identification is an obstacle, the steering wheel is automatically steered to avoid this, or the driver or a remote location A technique for notifying the operator of the identification result has been proposed. On the other hand, as a typical example of the IoT (Internet of Things), a connected car equipped with a function of always connecting to the Internet is drawing attention.

In Patent Document 1, the recognition conditions of the recognition dictionary used for object identification are changed based on the light distribution state of the headlight of the own vehicle, and the object recognition is performed from the image taken in front of the own vehicle based on the changed recognition dictionary. The technology to be performed is disclosed.

Patent Document 2 discloses a technique for detecting an object that is difficult to recognize with a camera by another detection means when the traveling environment suddenly changes. Specifically, the disappearance of the object detected in the previous frame is detected, the laser is irradiated in the direction of the object, and the distance information of the object is output.

Patent Document 3 discloses a technique for reducing the amount of processing related to object recognition by changing the object recognition range in an image according to the image processing apparatus to be adopted.

JP-A-2017-165345 Japanese Patent No. 6471528 Japanese Patent No. 5777389

There are a wide variety of objects that can be obstacles to vehicle travel, and a learning model for identifying objects (discriminative model) needs to be trained so that any object can be identified. However, in such a universal type identification model, if the capacities are the same, the identification rate is lower than that of the dedicated identification model in which the identification target is limited.

On the other hand, if a dedicated identification model is prepared for each object that may become an obstacle, a large amount of storage capacity is consumed to hold the identification model, and if all the identification models are operated at the same time, computer resources are insufficient.

In Patent Document 1, while the recognition accuracy can be improved by changing the recognition parameter from the light distribution state of the headlight, changes in the environment such as a shopping street, a general highway, and a general road in the suburbs under the same light source state. The identification target cannot be changed according to the above.

In Patent Document 2, while it is possible to improve the recognition accuracy when the environment changes, it is difficult to reduce the processing load of the server and the amount of data stored because the objects to be recognized are the same.

In Patent Document 3, it is difficult to switch the identification method according to the surrounding environment.

An object of the present invention is to solve the above technical problems, prepare a plurality of types of dedicated models optimized for identifying each object that may become a traveling obstacle in advance, and prepare the surrounding environment of each vehicle in advance. Object identification that enables more accurate and high-speed object identification by effectively utilizing the limited memory capacity by selectively downloading only some dedicated models according to the above to each vehicle and using them for object identification. To provide equipment, methods and systems.

In order to achieve the above object, the present invention is characterized in that it has the following configurations in an object identification device that identifies an object by applying an image of an in-vehicle camera to an identification model.

(1) Means for storing multiple types of identification models, means for identifying the characteristics of the surrounding environment based on the image of the in-vehicle camera, means for estimating the driving environment based on the characteristics of the surrounding environment, and estimation of the driving environment. A means for selecting a part of the identification model based on the result and a means for applying the image of the in-vehicle camera to the selected identification model to perform object identification are provided.

(2) The means for acquiring the sensor output peculiar to the surrounding environment of the vehicle is further provided, and the means for identifying the characteristics of the surrounding environment is to identify the characteristics of the surrounding environment based on the image of the in-vehicle camera and the sensor output. I made it.

(3) As means for identifying the characteristics of the surrounding environment, a first environmental dictionary suitable for identifying moving objects, a second environmental dictionary suitable for identifying real estate, and a third environmental dictionary suitable for identifying traffic / road signs are used. Used together to identify moving objects and real estate.

According to the present invention, the following effects are achieved.

(1) For each object that may become a driving obstacle, prepare multiple types of dedicated identification models optimized for identification in advance, and some dedicated identification models according to the surrounding environment of each vehicle. Only are selectively downloaded to each vehicle and used for object identification. Therefore, it is not necessary to always implement all the dedicated models in each vehicle, so that the limited memory capacity can be effectively used. In addition, as compared with the case of implementing a universal identification model in which the identification target is not limited, if the model sizes are the same, more accurate and high-speed object identification can be performed while using the same computer resources.

(2) Since the surrounding environment of the vehicle is estimated based on images and sensors, shape identification and size identification that are unsuitable for image-based estimation at night, rainfall, heavy fog, etc., or sensor-based estimation, etc. The identification accuracy can be complemented with each other. Therefore, high identification accuracy can always be obtained regardless of the weather, the situation, the shape of the object, and the like.

(3) When estimating the surrounding environment of a vehicle, pay attention to moving objects such as people and animals, real estate such as buildings and houses, traffic signs and traffic lights as the estimation parameters, and the first environment suitable for identifying moving objects. A dictionary, a second environmental dictionary suitable for identifying real estate, and a third environmental dictionary suitable for identifying traffic / road signs are used in combination to preferentially identify moving objects and real estate, so that the surrounding environment can be estimated accurately. Become.

It is a functional block diagram of the object identification apparatus which concerns on one Embodiment of this invention. It is a figure which showed the structure of the network which applied the object identification apparatus of this invention. It is a functional block diagram which showed the 1st example which distributed implementation of each function of an object identification apparatus to a connected car and a server. It is a sequence flow of object identification in the 1st implementation example. It is a functional block diagram which showed the 2nd example which distributed implementation of each function of the object identification apparatus in a connected car and a server. It is a sequence flow of object identification in the second implementation example. It is a figure which showed the relationship between a driving environment and an identification model.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a functional block diagram of an object identification device 1 according to an embodiment of the present invention, based on an image taken by a camera mounted on a vehicle such as a connected car (hereinafter, collectively referred to as an in-vehicle camera). Identify people, animals, and other objects that hinder driving (hereinafter referred to as "objects").

The camera image acquisition unit 102 acquires and outputs a camera image from the vehicle-mounted camera 101. The environment sensor 103 measures parameters specific to the surrounding environment such as distance to each object, noise, temperature, humidity, and brightness. Each sensor output is output from the sensor interface (I / F) 104.

The surrounding environment identification unit 105 is a first environment dictionary 105a constructed especially for identifying moving objects such as the number of vehicles and people, and a second environment particularly constructed for identifying real estate such as the number and height of buildings. It comprises a plurality of dictionaries including a third environmental dictionary 105b constructed for identifying traffic / road signs such as dictionaries 105b, in particular guidance signs, traffic signs, traffic lights and pedestrian crossings, and identifies at least one camera image and sensor output. By applying it to the dictionary, it identifies the number of vehicles, the number of people, the number of buildings and their height, and the characteristics of the surrounding environment such as road signs.

The traveling environment estimation unit 106 includes the environment identification model 106a, and estimates the traveling environment of the vehicle based on the identification result output by the surrounding environment identification unit 105. In the present embodiment, it is estimated whether the driving environment is one of a large number of driving environment candidates including a "shopping street" or a similar environment, a "suburb" or a similar environment, a "highway" or a similar environment. Will be done. For example, if a large number of people are identified, it is likely to be presumed to be a "shopping street" or similar environment. In addition, if a guide sign or a speed limit sign peculiar to an expressway is identified without identifying a pedestrian crossing or a traffic light, it is highly likely that the environment is presumed to be an "expressway" or a similar environment.

The discrimination model database (DB) 107 includes a temporary figurine identification model 107a tuned for identifying temporary figurines, a sign identification model 107b tuned for tag identification, and pedestrians tuned for pedestrian number identification. A large number of dedicated discriminative models including the number discriminative model 107c are accumulated.

The discriminative model selection unit 108 selects one or a plurality of discriminative models from the discriminative model DB 107 based on the estimation result of the running environment by the running environment estimation unit 106. In the present embodiment, as shown in FIG. 7, when the traveling environment is estimated to be a “shopping street”, the temporary figurine identification model 107a and the pedestrian number identification model 107c are preferentially selected. When the driving environment is estimated to be "suburbs", the temporary figurine identification model 107a and the sign identification model 107b are preferentially selected. When the driving environment is presumed to be a "highway", the temporary figurine identification model 107a is preferentially selected.

The object identification unit 109 applies the selected identification model (109a, 109b) to the image captured by the vehicle-mounted camera 101 to execute object identification. The result of the object identification is reflected in the steering wheel operation, the accelerator operation or the braking operation in the automatic driving, or is reflected in the message provided to the driver (operator) who directly or remotely controls the vehicle.

Such an object identification device 1 can be configured by mounting an application (program) that realizes each function described later on a general-purpose computer or server. Alternatively, a part of the application can be configured as a dedicated machine or a single-purpose machine that is made into hardware or ROM.

FIG. 2 is a diagram showing a network configuration to which the object identification device 1 is applied, and is a server 3 on which the main functions of the object identification device 1 are implemented, and the server 3 and the network NW and the radio base station BS. It is composed of a large number of connected cars 2 that are wirelessly connected by relaying the above, and the in-vehicle camera 101 and the environment sensor 103 are mounted on each connected car 2.

FIG. 3 is a functional block diagram showing a first example in which each function of the object identification device 1 is distributedly mounted on the connected car 2 and the server 3. Here, the in-vehicle camera 101, the camera image acquisition unit 102, and the environment sensor are shown. The 103, the sensor I / F 104, and the object identification unit 109 are mounted on the connected car 2, and other functions are mounted on the server 3.

FIG. 4 shows a sequence flow of object identification in the first implementation example, and in the connected car 2, shooting by the in-vehicle camera 101 is started at time t1. At time t2, the ambient environment is measured by the ambient environment sensor 103. At time t3, the in-vehicle camera image and the output of each sensor are uploaded to the server 3 via the network.

In the server 3, at time t4, the surrounding environment identification unit 105 applies the video and sensor output of the in-vehicle camera to the plurality of environment dictionaries 105a and 105b, so that the number of vehicles, the number of people, and the number of people are the characteristics of the surrounding environment of the own vehicle. Identify the number of buildings and their height.

At time t5, the identification result is applied to the environment estimation model 106a of the driving environment estimation unit 106, and the driving environment of the connected car 2 is any of "shopping street", "suburbs", "expressway" or "other". It is estimated that there is. At time t6, the discriminative model selection unit 108 selects the optimum discriminative model from the discriminative model DB 107 based on the estimation result of the traveling environment. At time t7, the selected discriminative model is delivered to the connected car 2.

When the identification model is delivered to the connected car 2, the object identification unit 109 installs the current identification model in place of the current identification model at time t8. At time t9, object identification is executed by applying the camera image to the identification model. At time t10, the result of the object identification is reflected in the automatic driving control of the own vehicle and the message notified to the driver and the like.

FIG. 5 is a diagram showing a second example in which each function of the object identification device 1 is distributedly mounted on the connected car 2 and the server 3, and here, the in-vehicle camera 101, the camera image acquisition unit 102, the environment sensor 103, In addition to the sensor I / F 104 and the object identification unit 109, the ambient environment identification unit 105 is mounted on the connected car 2, and other functions are mounted on the server 3.

FIG. 6 is a sequence flow of object identification in the second implementation example, and in the connected car 2, shooting by the in-vehicle camera 101 is started at time t1. At time t2, the ambient environment is measured by the ambient environment sensor 103. At time t3, the surrounding environment identification unit 105 applies the in-vehicle camera image and sensor output to the plurality of environment dictionaries 105a and 105b to obtain the number of vehicles, the number of people, the number of buildings and their heights around the own vehicle. Identify. At time t4, the identification result of the surrounding environment is uploaded to the server 3.

At time t5, the uploaded identification result is applied to the environment estimation model 106a of the driving environment estimation unit 106, and the driving environment of the connected car 2 is either "suburbs", "in Tokyo", "shopping street" or "others". Is presumed to be. At time t6, the discriminative model selection unit 108 selects the optimum discriminative model from the discriminative model DB 107 based on the estimation result of the traveling environment. At time t7, the selected discriminative model is delivered to the connected car 2.

When the identification model is delivered to the connected car 2, the object identification unit 109 installs the current identification model in place of the current identification model at time t8. At time t9, object identification is executed by applying the camera image to the identification model. At time t10, the result of the object identification is reflected in the automatic driving control of the own vehicle and the message notified to the driver and the like.

1 ... object identification device, 2 ... connected car, 3 ... server, 101 ... in-vehicle camera, 102 ... camera image acquisition unit, 103 ... environment sensor, 104 ... sensor interface (I / F), 105 ... Surrounding environment identification unit, 105a ... First environment dictionary, 105b ... Second environment dictionary, 105c ... Third environment dictionary, 106 ... Driving environment estimation unit , 106a ... environment identification model, 107 ... identification model DB, 107a ... temporary figurine identification model, 107b ... marker identification model, 107c ... pedestrian number identification model, 108 ... identification model Selection unit, 109 ... Object identification unit

Claims (14)

In an object identification device that identifies an object by applying the image of an in-vehicle camera to an identification model
A means of storing multiple discriminative models and
A means of identifying the characteristics of the surrounding environment based on the image of the in-vehicle camera,
A means of estimating the driving environment based on the characteristics of the surrounding environment,
A means of selecting some discriminative models based on the estimation results of the driving environment,
An object identification device including means for performing object identification by applying an image of an in-vehicle camera to the selected identification model. Further equipped with a means for acquiring the sensor output peculiar to the surrounding environment of the vehicle,
The object identification device according to claim 1, wherein the means for identifying the characteristics of the surrounding environment is to identify the characteristics of the surrounding environment based on the image of the vehicle-mounted camera and the sensor output. The means for identifying the characteristics of the surrounding environment is at least one of a first environmental dictionary suitable for identifying moving objects, a second environmental dictionary suitable for identifying real estate, and a third environmental dictionary suitable for identifying traffic / road signs. The object identification device according to claim 1 or 2, wherein the moving object and the real estate are identified by using one. The means for estimating the driving environment estimates whether the driving environment is a plurality of driving environment candidates including a shopping district or a similar environment, a suburb or a similar environment, a highway or a similar environment. The object identification device according to any one of claims 1 to 3, wherein the object identification device is characterized. The object identification device according to any one of claims 1 to 4, wherein one of the identification models is for identifying the number of pedestrians. The object identification device according to any one of claims 1 to 5, wherein one of the identification models is for tag identification. The object identification device according to any one of claims 1 to 6, wherein one of the identification models is for identifying a temporary figurine. In an object identification method in which a computer applies an image from an in-vehicle camera to an identification model to identify an object.
The procedure for identifying the characteristics of the surrounding environment based on the image of the in-vehicle camera,
Procedures for estimating the driving environment based on the characteristics of the surrounding environment,
The procedure for selecting a part of multiple discriminative models based on the estimation result of the driving environment,
An object identification method comprising a procedure of applying an image of an in-vehicle camera to the selected identification model to perform object identification. Including the procedure to acquire the sensor output specific to the surrounding environment of the vehicle,
The object identification method according to claim 8, wherein the procedure for identifying the characteristics of the surrounding environment is characterized by identifying the characteristics of the surrounding environment based on the image of the vehicle-mounted camera and the sensor output. In the procedure for identifying the characteristics of the surrounding environment, at least one of a first environmental dictionary suitable for identifying moving objects, a second environmental dictionary suitable for identifying real estate, and a third environmental dictionary suitable for identifying traffic / road signs. The object identification method according to claim 8 or 9, wherein the moving object and the real estate are identified by using one. In an object identification system in which a server applies an image from an in-vehicle camera of a vehicle to an identification model to identify an object.
The vehicle
Equipped with a means to upload the image of the in-vehicle camera to the server,
The server
A means of storing multiple discriminative models and
A means of identifying the characteristics of the surrounding environment based on the uploaded in-vehicle camera image,
A means of estimating the driving environment based on the characteristics of the surrounding environment,
A means of selecting some discriminative models based on the estimation results of the driving environment,
An object identification system including means for performing object identification by applying an image of an in-vehicle camera to the selected identification model. The vehicle
Further equipped with a means for acquiring the sensor output peculiar to the surrounding environment of the vehicle,
The uploading means uploads the video and sensor output of the in-vehicle camera.
The object identification system according to claim 11, wherein in the server, the means for identifying the characteristics of the surrounding environment is to identify the characteristics of the surrounding environment based on the uploaded image of the vehicle-mounted camera and the sensor output. In an object identification system in which a server applies an image from an in-vehicle camera of a vehicle to an identification model to identify an object.
The vehicle
A means of identifying the characteristics of the surrounding environment based on the image of the in-vehicle camera,
It has a means to upload the image of the in-vehicle camera and the characteristics of the surrounding environment to the server.
The server
Multiple discriminative models and
A means of estimating the driving environment based on the characteristics of the uploaded surrounding environment,
A means of selecting some discriminative models based on the estimation results of the driving environment,
An object identification system including means for performing object identification by applying an uploaded image of an in-vehicle camera to the selected identification model. The vehicle further comprises means for acquiring sensor outputs specific to the vehicle's surrounding environment.
The object identification system according to claim 13, wherein the means for identifying the characteristics of the surrounding environment is to identify the characteristics of the surrounding environment based on the image of the vehicle-mounted camera and the sensor output.

Images