JP7226368B2 - Object state identification device - Google Patents

Description

The present invention relates to an object state identification device that identifies the state of an object represented in an image.

Techniques for detecting objects represented by sensor information, such as images obtained by cameras, are being researched. In recent years, techniques have been proposed to improve detection accuracy by using machine learning techniques such as so-called deep neural networks (hereinafter simply referred to as DNNs) to detect objects.

In addition, in order to track objects represented in images or detect anomalies, techniques have been proposed that utilize multiple time-series images or feature values obtained from those images as inputs to a neural network. (See, for example, Patent Documents 1 and 2).

For example, in the object tracking method disclosed in Patent Document 1, two or more images that are consecutive in time series are input to a neural network. This object tracking method compares feature amounts extracted by a neural network, which are feature amounts of each of the two or more images, and checks for similarity. Then, this object tracking method matches one or more objects that are tracking candidates appearing in the previous image in chronological order based on the matching result, and finds one or more objects appearing in the following image in chronological order relative to the previous image. Identification information and location information are output as identification results. Also, the neural network used includes two or more identical structures with one or more fully connected layers and zero or more convolutional layers, and shares parameters in corresponding layers between identical structures.

In addition, the anomaly monitoring system disclosed in Patent Document 2 extracts an image portion that has changed from an image to be monitored, inputs the changed image portion to a convolutional neural network, extracts a feature amount, and extracts The resulting feature amount is input to a recursive neural network to generate an image description that outlines the image.

Japanese Patent Application Laid-Open No. 2018-26108 JP 2018-101317 A

Even with the techniques described above, it may not be possible to accurately identify the state of the object represented in the image.

SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to provide an object state identification device capable of identifying the state of an object represented in an image.

According to one embodiment, an object state identification device is provided. This object state identification device inputs a series of images obtained in time series to a first classifier that has been pre-learned to detect a predetermined object. and an object detection unit that detects an object region that includes a predetermined object and has a predetermined shape, and a set of pixels in which the predetermined object is represented in pixel units for each of the series of images. By inputting it to a second discriminator that has been pre-learned to discriminate, each of the series of images is divided into a region of interest, which is a set of pixels representing a predetermined object, and other regions. and a third discriminator having a recursive structure in chronological order for features included in both the object region and the region of interest, among features obtained from pixel values in the object region detected in each of the series of images. and a state identification unit that identifies a state of a predetermined object that accompanies a time-series change in appearance by inputting to .

The object state identification device according to the present invention has the effect of being able to identify the state of an object represented in an image.

1 is a schematic configuration diagram of a vehicle control system in which an object state identification device is mounted; FIG. 1 is a hardware configuration diagram of an electronic control device that is an embodiment of an object state identification device; FIG. FIG. 3 is a functional block diagram of a processor of an electronic control unit relating to vehicle control processing including object state identification processing; FIG. 4 is a diagram showing an example of the configuration of a DNN used as a first discriminator; FIG. FIG. 3 is a diagram showing an example of a detection target object and an object region represented on an image; FIG. 3 is a diagram showing an example of an object region and a region of interest; 4 is a timing chart of processing of each unit related to state identification processing; FIG. 10 is a diagram showing an example of a detected object list; 4 is an operation flowchart of vehicle control processing including object state identification processing;

The object state identification device will be described below with reference to the drawings. This object state identification device identifies a state of an object to be detected (hereinafter, sometimes referred to as a detection target object) represented by a series of images obtained in time series, accompanied by time-series changes in appearance. . For this purpose, this object state identification apparatus inputs the series of images to a first classifier pre-trained to detect the detection target object, and for each image, the detection target object and having a predetermined shape (hereinafter sometimes referred to as an object region) is detected. By tracking the detection target object detected from each image, the object state identification apparatus associates object regions in which the same detection target object is represented in each image. In addition, this object state identification device inputs a series of images to a second discriminator trained in advance to discriminate a set of pixels representing a detection target object from other sets of pixels. , each image is divided into a region of interest, which is a set of pixels representing a detection target object, and other regions (hereinafter sometimes referred to as mask regions). Further, the object state identification apparatus, among features obtained from pixel values in an object region representing the same detection target object in a series of images, detects features included in both the object region and the target region at a time. By inputting to the third discriminator having a recursive structure in sequential order, the state of the object to be detected is discriminated.

For example, assume that the object to be detected is a vehicle. The vehicle flashes the turn signals when turning left or right. In addition, the vehicle turns on the brake lamps when decelerating, and blinks the hazard lamps when stopping the vehicle. The lighting or blinking of these signals or lamps is accompanied by changes in the appearance of the vehicle over time, and represents the state of behavior of the vehicle. However, individual images showing turn signals, brake lamps, or hazard lamps do not show time-series changes in blinking of those lamps, so whether the turn signals or hazard lamps are blinking, and whether the brake lamps are on or off cannot be determined. It is difficult to accurately identify whether the lights are on or off from individual images showing turn signals, brake lights, or hazard lights. Therefore, as described above, this object state identification device inputs the features included in both the object region and the region of interest of each of the series of time-series images into the third classifier having a recursive structure. Thus, it is possible to accurately identify whether the turn signals or hazard lamps are blinking, and whether the brake lamps are on or off.

An example in which the object state identification device is applied to a vehicle control system will be described below. In this example, the object state identification device performs object state identification processing on a series of time-series images obtained by a camera mounted on a vehicle, and detects objects existing around the vehicle as objects to be detected. Detect other vehicles. Then, the object state identification device determines whether or not left or right turn signals or hazard lamps are blinking, whether the brake lamps are on or off, as the other vehicle's detected state accompanied by a change in appearance. identify the state of

FIG. 1 is a schematic configuration diagram of a vehicle control system in which an object state identification device is installed. FIG. 2 is a hardware configuration diagram of an electronic control unit, which is one embodiment of the object state identification device. In this embodiment, a vehicle control system 1 mounted on a vehicle 10 and controlling the vehicle 10 includes a camera 2 for capturing an image of the surroundings of the vehicle 10 and an electronic control unit (ECU) which is an example of an object state identification device. ) and 3. The camera 2 and the ECU 3 are communicably connected via an in-vehicle network 4 conforming to a standard such as a controller area network. Note that the vehicle control system 1 may further include a storage device that stores maps used for automatic driving control of the vehicle 10 . Further, the vehicle control system 1 includes a distance measuring sensor such as LiDAR or radar, a receiver such as a GPS receiver for measuring the self-position of the vehicle 10 in compliance with the satellite positioning system, and a receiver for wirelessly communicating with other devices. A wireless terminal, a navigation device for searching for a planned travel route of the vehicle 10, and the like may be provided.

The camera 2 is an example of an imaging unit that is a sensor for detecting an object existing within a predetermined detection range, and is composed of an array of photoelectric conversion elements sensitive to visible light, such as a CCD or C-MOS. It has a two-dimensional detector and an imaging optical system that forms an image of an area to be photographed on the two-dimensional detector. The camera 2 is installed, for example, in the vehicle interior of the vehicle 10 so as to face the front of the vehicle 10 . Then, the camera 2 photographs the area in front of the vehicle 10 at predetermined photographing intervals (for example, 1/30 second to 1/10 second) and generates an image showing the area in front of the vehicle. The image obtained by camera 2 is preferably a color image. Note that the vehicle 10 may be provided with a plurality of cameras having different photographing directions or focal lengths.

The camera 2 outputs the generated image to the ECU 3 via the in-vehicle network 4 each time it generates an image.

The ECU 3 controls the vehicle 10 . In this embodiment, the ECU 3 controls the vehicle 10 to automatically drive the vehicle 10 based on an object detected from a series of time-series images obtained by the camera 2 . Therefore, the ECU 3 has a communication interface 21 , a memory 22 and a processor 23 .

The communication interface 21 is an example of a communication section and has an interface circuit for connecting the ECU 3 to the in-vehicle network 4 . That is, the communication interface 21 is connected to the camera 2 via the in-vehicle network 4 . Then, the communication interface 21 passes the received image to the processor 23 every time it receives an image from the camera 2 .

The memory 22 is an example of a storage unit, and has, for example, a volatile semiconductor memory and a nonvolatile semiconductor memory. Note that the memory 22 may have a dedicated memory circuit for each arithmetic unit when the processor 23 has a plurality of arithmetic units as will be described later. The memory 22 specifies various data and parameters used in the object state identification processing executed by the processor 23 of the ECU 3, such as images received from the camera 2 and classifiers used in the object state identification processing. It stores various parameters for the determination, confidence thresholds for each type of object, and the like. Furthermore, the memory 22 stores various data generated during the object state identification process, such as a detected object list representing information about detected objects, for a certain period of time. Furthermore, the memory 22 may store information used for driving control of the vehicle 10, such as map information.

The processor 23 is an example of a control unit, and has one or more CPUs (Central Processing Units) and their peripheral circuits. The processor 23 may further comprise other arithmetic circuits such as a logic operation unit, a numerical operation unit or a graphics processing unit (GPU). Each time an image is received from the camera 2 while the vehicle 10 is running, the processor 23 executes vehicle control processing including object state identification processing on the received image. The processor 23 then controls the vehicle 10 to automatically drive the vehicle 10 based on the detected objects around the vehicle 10 .

FIG. 3 is a functional block diagram of the processor 23 of the ECU 3 regarding vehicle control processing including object state identification processing. The processor 23 has an object detection unit 31 , a tracking unit 32 , an area division unit 33 , a state identification unit 34 , an operation planning unit 35 and a vehicle control unit 36 . These units of the processor 23 are, for example, functional modules implemented by computer programs running on the processor 23 . Alternatively, each of these units of processor 23 may be a dedicated arithmetic circuit provided in processor 23 . Among these units of the processor 23, the object detection unit 31, the tracking unit 32, the area division unit 33, and the state identification unit 34 execute object state identification processing. In addition, when a plurality of cameras are provided in the vehicle 10, the processor 23 may perform the object state identification processing for each camera based on the image obtained by the camera.

Every time an image is received from the camera 2, the object detection unit 31 inputs the received latest image to the first discriminator for object detection, so that the object to be detected (that is, other vehicles) and having a predetermined shape (that is, an object region), and specifies the type of the detection target object.

In this embodiment, the object detection unit 31 detects an object region including a detection target object represented in an image as a first discriminator, and is pre-trained to identify the type of the detection target object. Use DNNs. The DNN used by the object detection unit 31 is, for example, a Single Shot MultiBox Detector (SSD) or a DNN with a convolutional neural network (hereinafter simply referred to as CNN) type architecture such as Faster R-CNN. can be done.

FIG. 4 is a diagram showing an example of the configuration of a DNN used as the first discriminator. The DNN 400 has a main body 401 provided on the input side to which an image is input, and a position detection section 402 and type estimation section 403 provided on the output side of the main body 401 . Based on the output from the main body 401, the position detection unit 402 outputs the circumscribed rectangle of the detection target object represented on the image as the object region. Note that the shape of the object region is not limited to a rectangular shape, and may be, for example, a circular shape, an elliptical shape, or a polygonal shape with pentagons or more. Based on the output from the main body 401 , the type estimating section 403 calculates a certainty factor for each type of detection target object represented in the object region detected by the position detecting section 402 . Note that the position detection unit 402 and the type estimation unit 403 may be integrally formed.

The main trunk 401 can be, for example, a CNN with multiple layers connected in series from the input side to the output side. The multiple layers include two or more convolutional layers. Furthermore, the plurality of layers included in main trunk 401 may include pooling layers provided for each of one or more convolution layers. Furthermore, the multiple layers of main trunk 401 may include one or more fully bonded layers. For example, the main trunk 401 can be configured similar to the base layer of an SSD. Alternatively, main trunk 401 may be configured according to other CNN architectures such as VGG-19, AlexNet or Network-In-Network.

When an image is input, the main body 401 outputs a feature map calculated from the image by executing operations in each layer on the image. Note that the main body 401 may output a plurality of feature maps with different resolutions. For example, the main body 401 may output a feature map having the same resolution as the input image and one or more feature maps having a lower resolution than the input image.

The feature map output from the main body 401 is input to the position detection unit 402 and the type estimation unit 403, respectively. Each of the position detection unit 402 and the type estimation unit 403 can be, for example, a CNN having multiple layers connected in series from the input side to the output side. For each of the position detection unit 402 and the type estimation unit 403, the multiple layers of the CNN include two or more convolution layers. Further, for each of the position detection unit 402 and the type estimation unit 403, the multiple layers of the CNN may include a pooling layer provided for each one or multiple convolution layers. Note that the convolution layer and pooling layer of the CNN may be shared between the position detection unit 402 and the type estimation unit 403 . Furthermore, for each of the position detection unit 402 and the type estimation unit 403, the plurality of layers may include one or more fully connected layers. In this case, the fully connected layer is preferably provided on the output side of each convolutional layer. Alternatively, the output from each convolutional layer may be directly input to the fully connected layer. The output layer of the type estimating unit 403 may be a softmax layer that calculates the confidence of each type of the detection target object according to the softmax function, or a softmax layer that calculates the confidence of each type of the detection target object according to the sigmoid function. It may be a sigmoid layer to be calculated.

The position detecting unit 402 and the type estimating unit 403 are trained to output respective degrees of certainty of the type of the detection target object, for example, for regions of various positions on the image, various sizes, and various aspect ratios. be. Therefore, when an image is input, the discriminator 400 outputs respective confidence factors of the detection target object type for each region of various positions, various sizes, and various aspect ratios on the image. Then, the position detecting unit 402 and the type estimating unit 403 detect an area where the certainty of any type of detection target object is equal to or higher than a predetermined certainty threshold as an object area representing that type of detection target object. do.

An image (teacher image) included in the training data used for learning of the discriminator 400 represents, for example, the type of detection target object (eg, ordinary passenger car, bus, truck, motorcycle, etc.) and the detection target object. The circumscribing rectangle of the object to be detected, which is the object region that is detected, is tagged.

The discriminator 400 is trained using a large number of teacher images as described above, for example, according to a learning technique such as error backpropagation. By using the classifier 400 learned in this way, the processor 23 can accurately detect the object to be detected from the image.

Note that the object detection unit 31 may detect objects other than other vehicles around the vehicle 10 that affect the travel control of the vehicle 10 . Such objects include, for example, people, road signs, traffic lights, road markings such as lane markings, and other objects on the road. In this case, the first discriminator may be trained in advance so as to detect these objects as well. The object detection unit 31 can also detect these objects by inputting the image to the first discriminator.

The object detection unit 31 further performs non-maximum suppression (NMS) processing to detect object regions estimated to represent the same object, out of two or more object regions that at least partially overlap. You can choose one from

The object detection unit 31 registers the position and range of each object area on the image and the type of object included in the object area in a detected object list. The object detection unit 31 then stores the detected object list in the memory 22 . Furthermore, for each object region, the object detection unit 31 stores the feature map, which is calculated by the main part of the first discriminator from each pixel included in the object region and is output to the state discrimination unit 34, in the memory 22. Remember. Note that the feature map output to the state identification unit 34 can have the same resolution as the resolution of the image input to the first classifier. In addition, when a feature map with a resolution lower than the resolution of the input image is calculated by a pooling layer of the main body of the first discriminator, the feature map with the lower resolution is used by the state discriminator. 34. Furthermore, a plurality of feature maps having different resolutions calculated by the main part of the first discriminator may be output to the state discriminating section 34 .

For each object region detected from the latest image, the tracking unit 32 refers to the detected object list and associates the detection target object represented in the object region with the detection target object detected from the previous image. , to track the object to be detected represented in the object region. Further, when there are more than a predetermined number (for example, 5 to 10) of detection target objects being tracked, the tracking unit 32 selects a predetermined number of detection target objects from among the detection target objects being tracked, Select as an object to be identified.

For example, the tracking unit 32 applies tracking processing based on optical flow, such as the Lucas-Kanade method, to the object region of interest in the latest image and the object region in the previous image, thereby displaying the object region. track the detected object. Therefore, the tracking unit 32 extracts a plurality of feature points from the object region by applying a feature point extraction filter such as SIFT or Harris operator to the object region of interest. Then, the tracking unit 32 may calculate the optical flow by specifying corresponding points in the object region in the past image for each of the plurality of feature points according to the applied tracking method. Alternatively, the tracking unit 32 applies another tracking method applied to tracking a moving object detected from an image to the object region of interest in the latest image and the object region in the previous image, A detection target object represented in the object region may be tracked.

The tracking unit 32 selects, among the detection target objects detected from the latest image, the detection target objects that have not been associated with the detection target objects represented in the past images as new tracking targets for other tracking. An identification number different from that of the object to be detected is assigned, and the assigned identification number is registered in the detection object list. On the other hand, the tracking unit 32 detects the detection target object associated with the detection target object represented in the previous image, that is, the detection target object being tracked, among the detection target objects detected from the latest image. The same identification number as the identification number assigned to the object being tracked is associated.

As described above, when there are more than a predetermined number of detection target objects being tracked, the tracking unit 32 selects a predetermined number of detection target objects from among the detection target objects being tracked. Select as

For example, since a detection target object closer to the vehicle 10 has a greater influence on the operation control of the vehicle 10, the tracking unit 32 selects a predetermined number of detection target objects from among the detection target objects being tracked in order from the closest to the vehicle 10. select. For example, the larger the object area where the detection target object is represented on the image, the closer the distance from the vehicle 10 to the detection target object is estimated. Therefore, the tracking unit 32 selects a predetermined number of detection target objects in descending order of the size of the object region on the latest image, for example.

Alternatively, the tracking unit 32 may select a predetermined number of detection target objects based on the positions of the lower ends of the object areas of the detection target objects being tracked on the image. When the object to be detected is traveling on the same road as the road on which the vehicle 10 is traveling, the position of the lower end of the object area in which the object to be detected on the image is represented is the position of the object to be detected. Estimated position on the road surface. The closer the detection target object is to the vehicle 10, the lower the direction from the camera 2 to the position on the road surface where the detection target object is located. get closer. Therefore, it is estimated that the closer the position of the lower end of the object region is to the edge of the image, the closer the distance from the vehicle 10 to the detection target object represented in the object region is. Therefore, the tracking unit 32 may select a predetermined number of detection target objects from among the detection target objects being tracked in order from the lower end of the object region closer to the lower end of the image in the latest image.

Alternatively, for each of the detection target objects being tracked, the tracking unit 32 determines the size (e.g., width) of the object region in which the detection target object is represented and the reference object of the same type as the detection target object from the vehicle 10. The distance from the vehicle 10 to the object to be detected may be estimated based on the ratio to the reference size assuming that the object is located at a predetermined distance. Alternatively, if the vehicle control system 1 has a ranging sensor (not shown) such as LiDAR or radar, the ranging sensor may measure the distance to each detection target object being tracked. In this case, for example, the distance in the azimuth from the ranging sensor corresponding to the azimuth from the camera 2 corresponding to the center of gravity of the object area in which the detection target object is represented on the image is from the vehicle 10 to the detection target object. measured as the distance between Then, the tracking unit 32 may select a predetermined number of detection target objects in descending order of estimated or measured distance from the vehicle 10 .

Alternatively, the tracking unit 32 may select a predetermined number of detection target objects for each lane from the detection target objects being tracked. For example, the tracking unit 32 selects a detection target object estimated to be closest to the vehicle 10 from among the detection target objects traveling in the same lane as the vehicle 10 is traveling. Furthermore, the tracking unit 32 tracks each of the lanes adjacent to the left and right of the lane in which the vehicle 10 is traveling, and the lanes further adjacent to the adjacent lanes (that is, each of the left and right lanes centered on the lane in which the vehicle 10 is traveling). A detection target object estimated to be closest to the vehicle 10 is selected from each of the two lanes). In this case, for example, when the object detection unit 31 detects lane markings from the latest image, or when the localization processing unit (not shown) detects lane markings from the latest image, In other words, the tracking unit 32 may identify the lane in which each detection target object is traveling based on the positional relationship between the lane markings and the object area. For example, the tracking unit 32 determines that a detection target object of interest is located on a lane sandwiched between two lane markings located on both sides of the lower end of the object area containing the detection target object. should be determined. Further, the tracking unit 32 selects the detection target object closest to the vehicle 10 from among the detection target objects traveling in the lane by executing the same processing as the detection target object selection for each lane. do it. Note that the tracking unit 32 may select two or more detection target objects in order from the closest to the vehicle 10 for each lane.

According to a variant, the tracking unit 32 may select all detected objects that are being tracked as objects for state identification.

The tracking unit 32 notifies the state identification unit 34 of the identification number of the detection target object whose state is to be identified. In addition, the tracking unit 32 updates the value of the index indicating the detection target object that is the target of state identification in the detection object list based on the determination result regarding the target of state identification.

Each time an image is received from the camera 2, the region dividing unit 33 divides the image into a second pixel group that has been learned in advance so as to distinguish a group of pixels representing an object to be detected from other groups of pixels. , the image is divided into a region of interest, which is a set of pixels representing the object to be detected, and other mask regions.

As described in the object detection unit 31, the object area is set as a circumscribed rectangle of the object to be detected or an area having a predetermined shape. On the other hand, the external shape of the detection target object as viewed from the vehicle 10 is not limited to a rectangle. In addition, objects other than the detection target object may be visible from the vehicle 10 through a transparent portion (for example, a rear window) of the detection target object.

FIG. 5 is a diagram showing an example of a detection target object and an object region displayed on an image. An image 500 includes a plurality of vehicles, which are examples of detection target objects, and each vehicle is detected as a detection target object. Of these vehicles 501 , vehicles 502 and 503 located in front of the vehicle 501 appear to overlap with the vehicle 501 . contains some of the Specifically, in this example, the brake lights and turn signals of vehicle 502 and vehicle 503 are included within object region 511 . Therefore, if all the features in the object region 511 are used to identify the state of the vehicle 501, the state of the vehicle 501 will be affected by the lighting state of the brake lights or the blinking state of the turn signals of the vehicle 502 or 503. The state may not be accurately identified.

Similarly, image 500 shows a portion of vehicle 505 located ahead of vehicle 504 through the front and rear windows of vehicle 504 . Therefore, part of the vehicle 505 is included in the object region 514 set for the vehicle 504 . Therefore, if all the features in the object region 514 are used to identify the state of the vehicle 504, the state of the vehicle 504 can be determined by the state of the brake lights of the vehicle 505, the blinking state of the turn signal, or the like. It may not be identified correctly.

Therefore, the region dividing unit 33 divides each image into a region of interest and a mask region. By segmenting each image into regions in this manner, the state identification unit 34, which will be described later, can replace or reduce features representing objects other than the detection target object.

For example, the region dividing unit 33 can use a DNN having a CNN type architecture such as Fully Convolutional Network (FCN), SegNet, or U-Net as the second discriminator.

Alternatively, the region dividing unit 33 uses a DNN for instance segmentation that can divide regions for each set of pixels representing different objects (that is, for each instance) even for objects of the same type as the second discriminator. may The region dividing unit 33 can use, for example, Mask-RCNN or Instance FCN as DNN for such instance segmentation. By using a classifier for instance segmentation as the second classifier, the region dividing unit 33 can detect that a plurality of detection target objects of the same type are represented on the image, and that the plurality of detection target objects are partially overlapped, a region of interest can be set for each detection target object. Therefore, for example, even if the brake lamps or turn signals of other vehicles other than the vehicle of interest are included in the object region of the vehicle of interest, the brake lamps or turn signals of the other vehicles are not of the vehicle of interest. Affecting the identification result of the state (for example, whether the brake lamp is on or the turn signal is blinking) is suppressed.

Also, the region dividing unit 33 may use, as the second classifier, a classifier for semantic segmentation based on a method other than a neural network, for example, a classifier for semantic segmentation based on a method such as random forest.

Preferably, the second discriminator divides each image into a region of interest and a mask region on a pixel-by-pixel basis. As a result, the shape of the region of interest can more accurately represent the shape of the detection target object. Input to the discriminator becomes difficult. However, for each image, the second discriminator divides the region of interest and the mask region into units of pixel groups smaller than the minimum size of the assumed object region (for example, units of 2×2 pixels or units of 4×4 pixels). It is possible to divide the area into This reduces the amount of computation by the second discriminator.

Note that the information representing the region division result is represented as a bitmap having the same size as the image obtained by the camera 2 and having different values for each region, for example.

The region division unit 33 passes information representing the region division result to the state identification unit 34 .

Each time an image is obtained from the camera 2 for each detection target object to be subjected to state identification among the detection target objects being tracked, the state identification unit 34 identifies pixels in the object region including the detection target object. Among the features obtained from the values, the features included in both the object region and the region of interest are extracted by referring to the information representing the segmentation result. The state identification unit 34 then inputs the extracted features to a third classifier having a recursive structure, thereby identifying the state of the detection target object that accompanies changes in appearance over time.

The state identification unit 34 uses, for example, the feature map calculated by the main part of the first discriminator as the feature obtained from the pixel values in the object region representing the detection target object. feature can be used. This makes it possible to use not only the characteristics of the detection target object itself, but also the characteristics of the environment around the detection target object for state identification. In this embodiment, the effect of the relative positional relationship between the vehicle that is the object to be detected and the other vehicle, such as the fact that part of the turn signal of the vehicle that is the object to be detected is hidden behind other vehicles In consideration of the situation, the third discriminator can discriminate the state of the vehicle, which is the object to be detected. For example, when the resolution of the feature map is the same as the resolution of the image input to the first discriminator, each feature included in the region on the feature map corresponding to the object region on the image is the object It is a feature obtained from the pixel values in the region. Further, when the resolution of the feature map is lower than the resolution of the image input to the first discriminator, the position and The range is the area on the feature map that corresponds to the object area. For example, the upper left edge position and upper right edge position of the object area on the input image are (tlX, tlY) and (brX, brY), respectively, and 1/N (N is 2 or more) for the input image. ) is downsized and the feature map is calculated. In this case, the upper left end position and lower right end position of the region on the feature map corresponding to the object region on the image are (tlX/N, tlY/N) and (brX/N, brY/N), respectively.

Further, for each of the detection target objects to be subjected to state identification, the state identification unit 34 further determines, from among the object regions containing the detection target object, the information indicating the region division result of the same type of object as the detection target object. The region of interest is defined as a region of interest within this object region, and the other region is defined as a mask region. Note that when a DNN for instance segmentation is used as the second discriminator, and a region of interest is individually set for each object, the state identification unit 34 determines the type of detection target object included in the object region and Among the objects of the same type, the region of interest of the largest object within the object region in the information indicating the result of region division is set as the region of interest within the object region.

FIG. 6 is a diagram showing an example of an object region and a region of interest. In the region segmentation result 610 corresponding to the object region 600 shown in FIG. is divided into a mask region 612 which is . Therefore, a region 620, which is the intersection of the object region 600 and the region of interest 611, is a feature extraction target.

The state identification unit 34 replaces the features included in the mask region with 0 among the features included in the object region so that the features included in the mask region do not affect the identification of the state of the detection target object, or Attenuates the features contained in the mask area by multiplying them by a factor less than 1. As a result, features included in both the object region and the region of interest are extracted.

According to the modified example, the state identifying unit 34 inputs the value itself of each pixel in the object region representing the detection target object on the image input to the first classifier to the third classifier. , may be a feature obtained from pixel values in an object region representing the object to be detected. Alternatively, the state identification unit 34 inputs a value obtained by performing a predetermined filtering process such as a convolution operation on each pixel in the object region to the third classifier. It may also be a feature obtained from pixel values in the object region. In this case as well, the state identification unit 34 selects the value of each pixel included in the object region or the filtered value of each pixel so that the features included in the mask region do not affect the identification of the state of the object to be detected. , replaces the values contained in the mask region with 0, or attenuates the values contained in the mask region by multiplying them by a factor of less than 1.

The state identification unit 34 resizes each object region to a predetermined size (eg, 32×32) by down-sampling or up-sampling the extracted features. As a result, even if the relative distance between the vehicle 10 and the detection target object changes during tracking of the detection target object and the size of the detection target object on the image changes, the third discriminator is not input. The configuration of the third discriminator is simplified because the feature can be treated as a constant size.

The state identification unit 34 has a recursive structure such as a Recurrent Neural Network (RNN), Long Short Term Memory (LSTM), or Gated Recurrent Unit (GRU) as a third classifier having a recursive structure. A neural network can be used. Since the third discriminator only needs to process features included in both the object region and the region of interest, the size of the input layer and the intermediate layer can be smaller than the first discriminator, and the weight A small number of parameters, such as coefficients, are required to define the third discriminator. Therefore, the third discriminator has a smaller amount of calculation than the first discriminator, and can reduce the calculation load on the processor 23 . Furthermore, the amount of computation required for learning the third discriminator is also reduced. In addition, when the first discriminator, the second discriminator, and the third discriminator are each configured as a neural network, these neural networks are integrally integrated by the error backpropagation method using common teacher data. may be learned. During the learning, the weight coefficient of the kernel of each convolutional layer of the second discriminator may be fixed. Thereby, the features input to the third discriminator include not only the features of the detection target object itself but also the features of the environment around the detection target object that are necessary for improving the accuracy of state identification. Each discriminator is trained so as not to include those that reduce the accuracy of state discrimination.

Since the third discriminator has a recursive structure, it updates its internal state each time features are sequentially input in time series. Thereby, the third discriminator can discriminate the state of the detection target object based on the time-series change in the appearance of the detection target object being tracked. In the present embodiment, as described above, the state identification unit 34 determines whether the state of the object to be detected (that is, another vehicle around the vehicle 10) is the left or right turn signal or whether the hazard lamp is blinking. , to identify whether the brake lamp is on or off. Therefore, for example, a sigmoid function is used as the activation function of the output layer of the third discriminator. This allows the third discriminator to output the confidence of each state. Then, the state identification unit 34 compares the certainty of each state with the corresponding threshold, and determines that the state of the detection target object has the certainty greater than or equal to the corresponding threshold. For example, suppose the confidence factor for the left turn signal flashing condition of the detected object is 0.8, while the confidence factor for the left turn signal non-flashing condition is 0.2. Assuming that the threshold value is 0.5, the state identification unit 34 determines that the state of the object to be detected is a state in which the left turn signal is blinking.

Alternatively, a softmax function may be used as the activation function of the output layer of the third discriminator. In this case, the third discriminator indicates that the state of the object to be detected is blinking of the left turn signal, blinking of the right turn signal, blinking of the hazard lamp, blinking of the brake lamp, or none of these. Output the judgment result. Therefore, the state identification unit 34 may set the state represented by the determination result output from the third classifier as the state of the detection target object.

The state identification unit 34 notifies the operation planning unit 35 while registering the state identification results of the individual detection target objects to be subjected to state identification in the detection object list.

FIG. 7 is a timing chart of processing of each unit related to state identification processing. The processing of each part of the processor 23 is managed, for example, by a scheduler (not shown) operating on the processor 23, and executed according to the timing chart shown in FIG. In FIG. 7, the horizontal axis represents time. In FIG. 7, each block indicates that the processing shown in that block is executed, and each arrow indicates the transfer of data (image, segmentation result, feature, etc.) between each processing. represents For example, when the ECU 3 receives an image from the camera 2 at time t1, the GPU of the processor 23 performs detection processing of a detection target object by the object detection unit 31 and region division processing by the region division unit 33 for the image. are executed in parallel. Note that preprocessing such as contrast correction or color conversion may be performed on the image before the detection processing of the detection target object and the region division processing are performed.

When the detection target object detection processing is performed, the CPU of the processor 23 performs object detection post-processing such as registering the type of the detected object and the object area in the detection object list. A tracking process by the unit 32 is executed. After the tracking process, the state identification unit 34 extracts features input to the third classifier for each object region, resizes the extracted features, and performs state identification processing using the third classifier. executed. As described above, since the amount of computation by the third discriminator is relatively small, the computation time required for state identification processing of each detection target object can be reduced. Then, the obtained result of state identification of the object to be detected is used for the processing of the operation planning unit 35 and the vehicle control unit 36 . In order to minimize task switching cost and memory transfer amount between processing by CPU and processing by GPU, region division processing, feature extraction processing for each detection target object, state identification processing, and state identification processing results is preferably executed collectively as a batch process.

FIG. 8 is a diagram showing an example of a detected object list. In the detected object list 800, the memory 22 stores an index indicating whether or not the object is a state identification target, an identification number assigned to the object, and information about the object for each object to be detected being tracked. , and the number of times the state has been identified by the state identification unit 34 (that is, the number of times the features in the object region corresponding to the third classifier have been input) are stored. Further, the detected object list 800 stores information (not shown) representing the position and range of the object region and information (not shown) representing the type of the detected object for each of the detected objects being tracked. be done. Further, in a storage area 801 on the memory 22 indicated by a pointer for each detection target object, the feature input to the third discriminator, the intermediate layer state of the third discriminator, the third The output results from the discriminators of are stored.

The operation planning unit 35 refers to the detected object list to generate one or more planned travel routes (trajectories) for the vehicle 10 so that the vehicle 10 does not collide with objects existing around the vehicle 10 . The planned travel route is represented, for example, as a set of target positions of the vehicle 10 at each time from the current time to a predetermined time ahead. For example, the driving planning unit 35 refers to the detected object list and executes viewpoint conversion processing using information such as the mounting position of the camera 2 on the vehicle 10, thereby obtaining the in-image coordinates of the object in the detected object list. to coordinates on the bird's-eye image (bird's-eye coordinates). Then, the operation planning unit 35 tracks objects registered in the detected object list by executing tracking processing using a Kalman filter, a particle filter, or the like on a series of bird's-eye coordinates. From the trajectory obtained, the predicted trajectory of each object up to a predetermined time ahead is estimated. At that time, the operation planning unit 35 uses the state identification result of the detection target object to estimate the predicted trajectory. For example, when the state of the detection target object of interest is a state in which the left turn signal is blinking, the detection target object is highly likely to change lanes to the left or turn left. Therefore, the driving planning unit 35 estimates a predicted trajectory for changing lanes to the left or turning left for the object to be detected. Further, when the state of the detection target object of interest is a state in which the brake lamps are on or a state in which the hazard lamps are blinking, there is a high possibility that the detection target object will decelerate. Therefore, the operation planning unit 35 estimates a predicted trajectory for the object to be detected that decelerates from the current time. Furthermore, when the state of the detection target object of interest is a state in which none of the left and right turn signals and hazard lamps are blinking and the brake lights are off, the detection target object does not decelerate. likely to go straight. Therefore, the operation planning unit 35 estimates a predicted trajectory for the object to be detected so that the object goes straight without deceleration.

Based on the predicted trajectory of each object being tracked and the position, speed, and attitude of the vehicle 10, the operation planning unit 35 calculates the distance between each object being tracked and the vehicle 10 up to a predetermined time ahead for any object. A planned travel route for the vehicle 10 is generated so that the predicted value of is equal to or greater than a predetermined distance. The operation planning unit 35 determines the position, speed, and attitude of the vehicle 10 based on current position information indicating the current position of the vehicle 10 obtained from a GPS receiver (not shown) mounted on the vehicle 10, for example. can be estimated. Alternatively, each time an image is obtained by the camera 2, a localization processing unit (not shown) detects the left and right lane markings of the vehicle 10 from the image, and the detected lane markings and the detected lane markings are stored in the memory 22. The position, speed, and attitude of the vehicle 10 may be estimated by matching with the map information in the vehicle 10 . Further, the operation planning unit 35 may refer to the current position information of the vehicle 10 and the map information stored in the memory 22 to confirm the number of lanes in which the vehicle 10 can travel. If there are a plurality of lanes on which the vehicle 10 can travel, the operation planning unit 35 may generate the planned travel route so as to change the lane on which the vehicle 10 travels.
Note that the operation planning unit 35 may generate a plurality of planned travel routes. In this case, the operation planning unit 35 may select the route that minimizes the sum of the absolute values of the acceleration of the vehicle 10 from among the plurality of planned travel routes.

The operation planning unit 35 notifies the vehicle control unit 36 of the generated planned travel route.

The vehicle control unit 36 controls each part of the vehicle 10 so that the vehicle 10 travels along the notified planned travel route. For example, the vehicle control unit 36 obtains the acceleration of the vehicle 10 according to the notified planned travel route and the current vehicle speed of the vehicle 10 measured by a vehicle speed sensor (not shown), and adjusts the accelerator to achieve that acceleration. Set the opening or braking amount. The vehicle control unit 36 obtains the fuel injection amount according to the set accelerator opening, and outputs a control signal corresponding to the fuel injection amount to the fuel injection device of the engine of the vehicle 10 . Alternatively, the vehicle control unit 36 outputs a control signal corresponding to the set brake amount to the brake of the vehicle 10 .

Further, when the vehicle 10 changes its course so that the vehicle 10 travels along the planned travel route, the vehicle control unit 36 obtains the steering angle of the vehicle 10 according to the planned travel route, and determines the steering angle according to the steering angle. This control signal is output to an actuator (not shown) that controls the steered wheels of the vehicle 10 .

FIG. 9 is an operation flowchart of vehicle control processing including object state identification processing executed by the processor 23 . Each time the processor 23 receives an image from the camera 2, it executes vehicle control processing according to the operation flowchart shown in FIG. In the operation flowchart shown below, the processing of steps S101 to S106 corresponds to the object state identification processing.

The object detection unit 31 of the processor 23 inputs the latest image obtained from the camera 2 to the first discriminator, and detects the detection target object represented in the image. That is, the object detection unit 31 detects an object region having a predetermined shape including the detection target object on the image (step S101). Furthermore, the object detection unit 31 identifies the type of the object to be detected. Then, the object detection unit 31 registers the detected object in the detected object list.

For each object region containing the object to be detected in the latest image, the tracking unit 32 of the processor 23 displays the object region in the latest image based on the object region and the object regions in the past images. The object to be detected is tracked (step S102). Further, the tracking unit 32 selects a predetermined number of detection target objects from among the detection target objects being tracked as detection target objects to be subjected to state identification (step S103).

In addition, the region dividing unit 33 of the processor 23 inputs the latest image to the second discriminator, and divides the image into a region of interest representing each detection target object and a mask region other than that. (step S104).

The state identification unit 34 of the processor 23 determines, for each of the selected detection target objects to be subjected to state identification, the object region and the focus, among the features obtained from the pixel values in the object region representing the detection target object. Features included in both regions are extracted (step S105). Then, the state identification unit 34 identifies the state of each detection target object by inputting the extracted features to a third classifier having a recursive structure ( step S106).

The operation planning unit 35 of the processor 23 refers to the detected object list, and for each detection target object registered in the detected object list, determines the predicted trajectory and predetermined distance of the object estimated by referring to the state identification result. As described above, the planned travel route for the vehicle 10 is generated (step S107). Then, the vehicle control unit 36 of the processor 23 controls the vehicle 10 so that the vehicle 10 travels along the planned travel route (step S108). The processor 23 then terminates the vehicle control process.

As described above, this object state identification apparatus inputs a series of images obtained in time series to the first classifier, respectively. , the object area is detected. In addition, the object state identification device inputs each image to the second classifier, and divides the image into a set of pixels representing the object to be detected and a set of pixels representing other objects. , using the region segmentation result, features included in the region of interest representing the object to be detected are extracted from among the features in the object region. This object state identification device identifies the state of the object to be detected by inputting the extracted features into a third classifier having a recursive structure in chronological order. As a result, the object state identification apparatus can capture time-series changes in the appearance of the detection target object represented in the image as time-series changes in features used for state identification determination. Furthermore, this object state identification device can suppress the influence of information other than the detection target object represented in the image on the identification of the state of the detection target object. Therefore, this object state identification device can accurately identify the state of the object to be detected. Furthermore, this object state identification device uses the first classifier for detecting the object from each image, and extracts features to be input to the third classifier from each of the time-series images. , the overall amount of computation can be reduced compared to inputting the entire image to a discriminator having a recursive structure to discriminate the state of the object. Images used for learning the first classifier and the second classifier may be still images. On the other hand, learning for the third classifier requires a moving image. The size of each image included in may be smaller than the size of the images used for training the first and second classifiers. Therefore, this object state identification device reduces the cost required for learning each classifier (for example, the cost required for annotation of teacher images, the cost required for collecting teacher images, etc.), and the cost required for learning each classifier. A calculation amount and calculation time can be reduced.

Note that, according to a modification, the object detection unit 31 may detect a detection target object from an image using a discriminator other than DNN. For example, the object detection unit 31 receives as input a feature amount (for example, HOG) calculated from a window set on the image as a first discriminator, and is confident that the object to be detected is represented in that window. A support vector machine (SVM) pre-trained to output degrees may also be used. The object detection unit 31 calculates the feature amount from the window while variously changing the position, size, and aspect ratio of the window set on the image, and inputs the calculated feature amount to the SVM, thereby obtaining information about the window. Seek certainty. Then, the object detection unit 31 determines that the detection target object is represented in a window in which the certainty of any type of detection target object is equal to or higher than a predetermined certainty threshold, and determines that the window is an object region. And it is sufficient. Note that an SVM may be prepared for each type of object to be detected. In this case, the object detection unit 31 may calculate the certainty factor for each type of object by inputting the feature amount calculated from each window into each SVM. In this case, the feature of the object region input to the third discriminator of the state identification unit 34 is extracted from the window (i.e., the object region) determined to represent the detection target object and input to the SVM. A feature quantity such as HOG can be used.

The object state identification device according to the above embodiment or modification may be mounted in a device other than an in-vehicle device. For example, the object state identification device according to the above embodiment or modification detects an object from an image generated by a surveillance camera installed to photograph a predetermined outdoor or indoor area at predetermined intervals, and detects the detected object. It may be configured to identify the state of the object. Then, when an object is detected for a certain period of time, the object state identification device displays a message indicating that the object has been detected and the identification result of the state of the object on a display connected to the object state identification device. good.

In addition, the computer program that realizes the function of each part of the processor 23 of the object state identification device according to the above embodiment or modification can be stored in a computer-readable portable recording medium such as a semiconductor memory, a magnetic recording medium, or an optical recording medium. may be provided in recorded form.

As described above, those skilled in the art can make various modifications within the scope of the present invention according to the embodiment.

1 vehicle control system 2 camera 3 electronic control device (object state identification device)
4 in-vehicle network 21 communication interface 22 memory 23 processor 31 object detector 32 tracking unit 33 area dividing unit 34 state identifying unit 35 operation planning unit 36 vehicle control unit

Claims (1)

By inputting a series of images obtained in time series to a first discriminator pre-trained to detect a predetermined object, each of the series of images includes the object on the image, and an object detection unit that detects an object region having a predetermined shape;
each of the series of images by inputting each of the series of images into a second discriminator pretrained to distinguish sets of pixels representing the object from other sets of pixels; a region division unit that divides into a region of interest, which is a set of pixels representing the object, and other regions,
A third method having a recursive structure in which, among features obtained from pixel values in the object region detected in each of the series of images, the features included in both the object region and the region of interest are arranged in chronological order. A state identification unit that identifies the state of the object that accompanies a time-series appearance change by inputting to the classifier;
An object state identification device comprising:

Images