JPWO2019106789A1 - Processing apparatus and processing method - Google Patents

Abstract

The camera 1 includes a memory 12 that stores a moving image of the camera 1 that captures a moving image around the own vehicle, and a processing unit 13 that performs processing for determining a route of the own vehicle from the stored moving images. In order to determine the behavior prediction using one point of the moving image, the categories of the surrounding vehicles and the person are identified from the section earlier than the temporary point, and the own vehicle and the surroundings are determined according to the identified surrounding vehicle and the category of the person. By determining the behavior prediction of the vehicle and the person, the behavior is predicted using the behavior rules according to the surrounding vehicles and the person categorized in advance, so that more reliable prediction can be performed.

Description

  The present invention relates to a processing apparatus and a processing method that contribute to automatic driving or automatic driving support.

  By combining the own vehicle model that predicts the behavior of the own vehicle according to the operation of the own vehicle and the other vehicle model that predicts the behavior of the other vehicle according to the behavior of the vehicle that satisfies the predetermined positional relationship, A series of recommended operations from the current time to the future time is presented by predicting a plurality of future behaviors that occur in the vehicle group consisting of the host vehicle and other surrounding vehicles under the influence of the vehicle. (See Patent Document 1)

  Coexistence with people is a major issue in the automatic driving of vehicles. For example, in the case of automatic driving of a car, even if the same motion control is performed according to the characteristics of the peripheral driver, there may be situations where the results are different. As an example, when a direction indicator winker is issued to change lanes while driving on an expressway, the behavior of an automobile traveling behind the lane to be changed varies greatly depending on individual differences of drivers. The same phenomenon occurs in distribution centers where forklifts and AGVs coexist. In order to realize optimum automatic driving in such an environment where people and automatic trolleys coexist, a mechanism for predicting the behavior of objects / people themselves operated by surrounding people is required. In the prior art document 1, although a plurality of predictions from a certain point in time are performed, there is a time lag until it is determined that the prediction is different from the most likely prediction, and the probability that the prediction is different from the certain prediction is not small.

JP 2003-228800 A

  The present invention has been made to solve the above-described problems, and an object of the present invention is to make a more reliable prediction.

  The processing device according to the present invention includes a memory that stores a moving image of a camera that captures a moving image around the own vehicle, and a processing unit that performs a process for determining the route of the own vehicle from the stored moving image. The processing unit identifies a category of the surrounding vehicle and the person from the section past the temporary point in order to determine a behavior prediction using a point in time of the moving image, and according to the identified category of the surrounding vehicle and the person To determine the behavior prediction of the vehicle, surrounding vehicles and people.

  According to this invention, by determining the behavior prediction of the own vehicle, the surrounding vehicle and the person according to the identified surrounding vehicle and the category of the person, the behavior rule corresponding to the surrounding vehicle and the person categorized in advance is used. Therefore, a more reliable prediction can be made.

FIG. 2 is a hardware configuration diagram of the processing apparatus according to the first embodiment. 3 is a flowchart of a processing unit according to the first embodiment. FIG. 3 is a supplementary explanatory diagram of the flowchart of FIG. 2. An example of the template for the white line detection in Embodiment 1. FIG. 3 is an example of an overhead view in Embodiment 1. FIG. The simulation result in Embodiment 1, the example which performed lane change. The simulation result in Embodiment 1, the example which did not perform lane change. The example which shows the state St of the own vehicle at the time of performing a simulation using CNN, a surrounding vehicle, and a person with an image. An example of an output diagram in which a simulation is performed using the CNN in the first embodiment. The image after a predetermined frame from FIG.

Embodiment 1 FIG.
Coexistence with people is an important factor in controlling a vehicle (automobile, AGV, self-propelled forklift, etc.). According to the present invention, human compatibility and efficiency are improved in an environment in which a person, a vehicle operated by a person, a forklift, a bicycle, or other traveling object (peripheral vehicle) operated by the person and the vehicle to be controlled coexist. The goal is to develop a good vehicle control. To achieve this, we estimate the operating characteristics of moving objects, including those in the vicinity, and simulate the interaction between the operating characteristics and the movement of the vehicle's direction indication display and moving direction. By examining the situation, the most appropriate control method at each time is realized.
Embodiments of the present invention will be described below.

  In the first embodiment, with respect to the own vehicle that is a traveling cart that self-propels using power such as an automobile, an AGV, and a self-propelled forklift, a person who coexists in the vicinity recognizes the surrounding environment using a sensor. Realize the control of the vehicle or the entire system to improve the efficiency of the entire system, taking into account the interaction between the movement of the surrounding vehicles (including motorcycles and bicycles) and the movement of people, which is the traveling carriage to be operated By observing surrounding vehicles, the operation category of each vehicle is estimated, optimal vehicle control of the own vehicle is realized based on the category, and traveling categories are classified for surrounding vehicles. Make predictions based on categories. At that time, a description will be given of preparing several types of control patterns of the own vehicle, comparing the respective prediction results, and selecting the control pattern that provides the best situation in the future. An example of an automobile will be described as an example.

FIG. 1 is a hardware configuration diagram of the processing apparatus according to the first embodiment.
The configuration will be described. The own vehicle has a camera 1. The camera 1 is basically installed so that the front, side, and back can be imaged. Therefore, a plurality of cameras 1 may be installed in the vehicle so that the front, side, and rear can be imaged, and the front, side, and rear can be imaged with one camera. May be.
The distance sensor 2 has a device that can measure the distance between the surrounding vehicle and the own vehicle. That is, it has a function of measuring the distance between the surrounding vehicle and the host vehicle for the front, the side, and the rear. As a method, a method using a laser, a millimeter wave, or the like is common. In calculating the distance, even a stereo-type camera can cover the function, so if it is possible to measure the distance between the surrounding moving body and the vehicle by installing multiple cameras 1, the distance as hardware The sensor 2 becomes unnecessary.

The vehicle speed sensor 3 is a sensor 3 that measures the vehicle speed of the host vehicle. In general, the vehicle speed is often measured from the rotation of the engine and motor of the automobile, the drive shaft, etc., but this also provides the function of the vehicle speed sensor 3 depending on the function of the camera 1 and the installation method. The vehicle speed sensor 3 as hardware is not necessary.
The yaw rate sensor 4 is a sensor for detecting a rotational angular velocity (yaw rate) around the vertical axis of the vehicle of the host vehicle. Also in this case, the function of the yaw rate sensor 4 can be provided by the function of the camera 1 (moving images are taken and the difference of motion vectors for each frame is measured). In this case, the yaw rate sensor 4 as hardware is not necessary.

The rudder angle sensor 5 is a sensor for measuring the rotation angle of the steering wheel of the own vehicle, and the rotation speed of the own vehicle can be calculated from the value of this sensor.
The accelerator / brake sensor 6 is for measuring information on the accelerator opening and the brake opening of the host vehicle. When a person drives, an accelerator sensor and a brake sensor are often mounted separately. Here, a description will be given from the viewpoint of measuring the acceleration and deceleration of a car assuming future automatic driving. When there is a brake opening, the tail lamp of the own vehicle emits light.

  The driver switch 7 corresponds to a switch of a device (direction indicator, hazard lamp, front light) for the vehicle to give information to surrounding vehicles and people.

The information of the camera 1, the sensors 2 to 6, and the driver switch 7 described above is first received by the input interface 11 of the processing device 10 and sent to the memory 12. The processing unit 13 performs processing based on information in the memory 12, finally selects a base route on which the vehicle will travel in the future, and outputs it via the output interface 14. Based on the output information, the vehicle controls the steering wheel, accelerator, brake, driver switches for automatic driving, or the base route to the driver for semi-automatic driving, etc. (not shown) ) To present.
When the processing circuit is hardware, the processing circuit is, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC, an FPGA, or a combination thereof. Is applicable.
Furthermore, although this processing circuit is described for the purpose of processing inside, it is configured so that all or part of the processing circuit is processed on the cloud side via the communication line and the result is received via the communication line. May be.

Next, processing performed by the processing unit 13 in FIG. 1 will be described.
FIG. 2 is a flowchart of the processing unit according to the first embodiment. FIG. 3 is a supplementary explanatory diagram of the flowchart of FIG.
In S201, the camera 1 is used to photograph a moving image having a predetermined frame rate around the vehicle for a predetermined time (for example, about several seconds to several tens of seconds). At that time, data of the distance sensor 2, vehicle speed sensor 3, yaw rate sensor 4, rudder angle sensor 5, accelerator / brake sensor 6, and driver switch 7 other than the camera 1 may be acquired. When the video of this step is shot, the vehicle controls some action, that is, information that the driver controls or automatically controls the steering wheel, accelerator, brake, and driver switches (hereinafter referred to as vehicle control information), Thereafter, data of the camera 1, the distance sensor 2, the vehicle speed sensor 3, the yaw rate sensor 4, the steering angle sensor 5, the accelerator / brake sensor 6, and the driver switch 7 may be acquired for several seconds to several tens of seconds. The reason will be described later.

In step S202, an obstacle, a lane, a surrounding vehicle, and a person are specified.
The obstacle is detected by detecting a nearby obstacle using the distance sensor 2, for example. For example, it may be obtained by irradiating a plurality of laser beams in a part of the vehicle or all around the vehicle, or it can be detected from vector transition information that can be measured from the image of the camera 1. Alternatively, although not shown, it can be acquired and used as input signal processing information in step S201 from a map image or the like used in a navigation system or the like. Further, template matching with respect to camera images may be performed for roadside steps, guardrails, signals, and the like.

  The lane can be detected by detecting a white line from the image of the camera 1. Further, for example, FIG. 4 shows an example of a template for white line detection in the first embodiment, which is performed by matching the template and the camera image as shown in FIGS. Is called. When matching, it is necessary to perform either processing after deforming the camera image so as to match the template viewed from above based on the position of the camera 1 or by deforming the template so as to match the camera image. There is. The map image described above may be used instead of the template.

The detection of surrounding vehicles is mainly performed by the camera 1 or the distance sensor 2 or both.
In the case of a camera, after performing edge detection, there is a method of detecting by using an optical flow (a motion vector in a frame unit for a part of an image) in a relatively short distance. However, the optical flow may not be effective in front of 100 meters or more, and the DoG (Difference of Gaussian) filter that easily extracts pixel changes is small because the area on the road beyond 100 meters that the camera can capture is small. A method may be used in which edges are extracted by using and template matching is performed only on an area on a road 100 meters or more ahead. Although the above is described as 100 meters or more, it may be linked with the limit on the front side of the surrounding vehicle that can be captured by the optical flow.

  For example, there is a method of performing template rate matching on an area recognized from the image of the camera 1 on the road (an area recognized as a step or white line on the road or the inside of the guard rail) from the image of the camera 1.

Next, in step S203, the attributes of the detected surrounding vehicle and person are specified.
The identification of the attributes of surrounding vehicles and persons categorizes the behavior of the surrounding vehicles and persons. For example, in the case of surrounding vehicles, Category A: Rough car, Category B: General car, Category C : A gentle car. These categories are merely naming labels, and numerical parameters capable of specifying vehicle characteristics such as the maximum vehicle speed, vehicle acceleration, inter-vehicle distance, and lane change frequency are associated with each category. In the case of a person, if the speed is estimated to be running or if the person is estimated to be a child, the person is categorized as category A because the possibility of jumping out on the road or ignoring the signal is high. In other cases, it is categorized as category B because the possibility of jumping out on the road or ignoring the signal is small.
Furthermore, after detecting the surrounding vehicle and before projecting it onto the bird's-eye view image, the attribute of the vehicle can be specified from the image information of the vehicle, the feature amount, and the latest operation. “Features”, “vehicle color”, “number plate”, and the like can be given as feature quantities to be input in addition to image information, but there is no problem even if these are not used depending on the system configuration.

  In order to categorize the behavior of the surrounding vehicles, the latest behavior information (moving image) is used. As one method of obtaining, there is generally a technique combining CNN (Convolutional Neural Network) and RNN (Recurent Neural Network). CNN is mainly suitable for image recognition such as template matching, and is first used for identification to surrounding vehicles. Next, RNN is good at time series analysis that handles time series data. Therefore, by combining CNN and RNN, by inputting the still image for each frame obtained by the moving image, first, the surrounding vehicle in S202 is recognized, and the moving image (for example, 5 seconds) is observed more accurately. The behavior of surrounding vehicles can be categorized. For example, when the surrounding vehicle has a larger speed difference or more lane changes than the own vehicle, Category A: it is easy to be categorized as a rough car, and vice versa, Category C: a calm car.

  In the same way as described above, two-wheeled vehicles (belonging to surrounding vehicles), bicycles (belonging to surrounding vehicles), people, and the like can also prepare respective categories and categorize moving objects existing in the vicinity. In these, for example, the speed on the sidewalk and the size of the meander can be factors that determine the category.

Next, an overhead view is generated in S204.
FIG. 5 is an example of an overhead view in the first embodiment. There are walls on both sides of the two lanes, and the vehicle is on the left rear. There are surrounding vehicles on the right side and the front side when viewed from the own vehicle, and both are traveling forward (upward in the figure).
The bird's-eye view is created based on the information processed in steps S202 and S203 by deforming the camera image based on the position of the camera 1 to fit the image viewed from above.

Next, in S205, a simulation is performed based on the overhead view created based on step S204, and the reward amount Q is calculated.
The simulation is based on the identified obstacle, lane, surrounding vehicle (position, vector, acceleration vector), person (position, vector, acceleration vector), own vehicle control information, and the model used for the simulation, Future positions of lanes, surrounding vehicles (position, vector, acceleration vector), and person (position, vector, acceleration vector) are simulated. When simulating with a neural network, for example, an obstacle of a specific frame, a lane, a surrounding vehicle (position, vector, acceleration vector), a person (position, vector, acceleration vector) as an input layer, an obstacle of the next frame, Lanes, surrounding vehicles (position, vector, acceleration vector), and person (position, vector, acceleration vector) are output layers.

As shown in FIG. 3, the action a (peripheral vehicle, person, own vehicle control information) is given to the state St (obstacle, lane, surrounding vehicle, person, own vehicle control information) of the frame for starting the simulation. Then, an overhead view of the next frame (including a frame after a predetermined frame) is generated, and the state of the next frame is St + 1 (obstacle, lane, surrounding vehicle, person, and own vehicle control information). This is repeated for several seconds to several tens of seconds. Furthermore, this repetition is performed for the number of action rules. The behavior rule may change only the own vehicle control information, or the behavior of surrounding vehicles or people changes (according to surrounding vehicles, acceleration, deceleration, lane change, if a person jumps out of a pedestrian crossing, In the case of crossing on a busy road, etc.) both cases are included.
For example, two simulated examples are shown in FIGS. FIG. 6 shows an example in which the simulation result and lane change are made in the first embodiment, and FIG. 7 shows an example in which the lane change is not made and the simulation result in the first embodiment.

FIG. 6 shows (a) to (d) from the left, and this shows the transition for each predetermined number of frames with reference to (a).
FIG. 7 shows from (a) to (d) from the left, and this shows a transition for each predetermined number of frames with reference to (a).

  Although it is necessary to perform as many simulation action rules as possible, it is based on information captured on a moving image basis, that is, a state St having a relative speed with respect to the own vehicle, and is thus restricted to the movement of the own vehicle, surrounding vehicles, and people. Is born, and the number of action rules can be reduced.

  In addition, in the state St, the behavior of surrounding vehicles and persons is categorized in advance in step S203. Therefore, the rule conditions can be narrowed down to action rules in accordance with the categorization. By predicting the surrounding vehicles (acceleration, deceleration, lane change) in advance, it is possible to predict with better action rules, and to improve the prediction accuracy.

Furthermore, the timing of the frame state St (obstacles, lanes, surrounding vehicles, people, and own vehicle control information) at which the simulation is started is also an important factor in reducing behavior rules and predicting behavior described later. . In other words, the most difficult thing to predict is that there has been a change in the vehicle control information, for example, a blinker, a brake, an accelerator, or a lane change is started (for neighboring vehicles and people). When sending information), there are many cases in which the behavior of surrounding vehicles and people does not take plausible behavior. That is the point.
In addition, by setting at least 1 second after the vehicle control information has changed as the frame state St for starting the simulation, the surrounding vehicle (acceleration, deceleration, lane change), person (jumping on the pedestrian crossing) It is also possible to observe the initial motion of a crossing on a road, etc., and to narrow down the action rules categorized according to the initial motion.

  Further, in performing the above simulation, it is necessary to determine a control method for the own vehicle, surrounding vehicles, and persons. In addition to the case where rules prepared in advance are used as this method, it is conceivable to determine control using CNN or the like. As an implementation method, for example, a method in which all images are given as CNN inputs and the vehicle control information is output can be considered.

  Further, as described above, the simulation itself may be performed using the CNN in addition to the rule determination. As an input channel of the first layer of CNN, a method of explicitly extracting and inputting an image as shown in FIG. FIG. 8 shows an example of the state St of the host vehicle, surrounding vehicles, and persons when performing a simulation using CNN.

  On the upper left side, a surrounding vehicle that has stopped with a hazard lamp indicated by both the left and right arrows is shown. In the surrounding vehicle, the right front door is opened and closed, and a line from the right front to the right diagonal rear is drawn to indicate that. A person indicated by a round side is detected on the right side of the left upper peripheral vehicle, and 5 in the circle indicates that the vehicle is moving at 5 km / h. In addition, two arrows on the lower side of the circle indicate the rearward direction of the vehicle. For example, the right side indicates a direction and the left side indicates an acceleration vector. That is, the person detected as a circle indicates that the person is moving while accelerating toward the vehicle rear side at a speed of 5 km / h.

Similarly, the host vehicle shown on the lower side shows a direction indication indicated by a right arrow, and also proceeds to the right front direction at a speed of 20 km / h, and also shows an acceleration vector to the right rear. That is, it is shown that the own vehicle is further decelerated from 20 km / h with the steering wheel turned to the right. In addition, there is a right arrow on the left side of the vehicle, which indicates that the lane has changed.
It is shown that the surrounding vehicle on the upper right side accelerates further in the traveling direction side with the lower side as the traveling direction at 40 km / h.
By performing such input processing, visualization of CNN processing (driving support) can be expected. It is possible to select a combination of the two methods or one of them while looking at the balance with the calculation cost.

The output layer of the image input to the input layer of the CNN can also be output as an image similar to the input layer. FIG. 9 is an example of an output diagram in which a simulation is performed using the CNN in the first embodiment. The contents shown in FIG. 9 are the same as those in FIG.
In FIG. 9, blind spots formed on the oncoming vehicle and the opposite side of the wall as seen from the own vehicle are formed. It is also possible to enter it explicitly. This can be formed as an output image by inputting blind spot image information as a filter image with respect to the input image to the CNN. When calculating a reward amount Q described later, a route (distance close to the blind spot image information) is considered. If the speed is small), a high reward amount can be given.
The blind spot image information varies depending on the position of the own vehicle, the surrounding vehicle, and the person. FIG. 10 shows an image after a predetermined frame from FIG. Therefore, blind spot image information corresponding to an image after a predetermined frame can be explicitly entered.

A reward amount Q for each rule is calculated for the simulation for each determined rule.
Various methods can be considered as a method of calculating the evaluation value. For example, an idea of using the amount of movement as a reward amount, an idea of using safety as a reward amount based on the inter-vehicle distance from other vehicles, or a total value of both may be considered. As described above, it is also conceivable to give a high reward amount to a route that takes into account the blind spot image information (the speed is small when the distance is short). Separately from the reward amount, the simulation completion time (in this example, for example, the situation around the vehicle after 5 seconds) is evaluated, and the total of the reward amount and the environmental evaluation value up to that time can be considered as the evaluation index. it can. As another realization method, a method of calculating a reward amount on the assumption that a person's operation log is correct may be considered (reverse reinforcement learning). Of the above methods, any method may be used as long as the operation pattern can be evaluated.

In step S206 of FIG. 2, the behavior prediction is determined from the reward amount Q.
Action prediction, which is a simulation result with the highest reward amount Q from each rule, is set as a provisional base route.

  In step S207, the provisional base route is directly input from the input information to another CNN information predictor to have a route for generating an action. That is, a moving image of a predetermined time (for example, about several seconds to several tens of seconds) having a predetermined frame rate around the own vehicle using the camera 1, a distance sensor 2, a vehicle speed sensor 3, a yaw rate sensor 4, a rudder angle sensor 5, an accelerator / The data of the brake sensor 6 and the driver switch 7 are directly input to the input layer of the neural network including CNN, and the route information including the vehicle control information is output to the output layer.

  Since the provisional base route output from Step S202 to Step S206 described so far has built a model, it can be expected to operate with high accuracy in the assumed environment. In a situation that the designer does not assume, it may not work because the system is built. Therefore, in such a situation, without performing any of these pre-processing, it is input in a data format that can be input to a neural network including CNN, and the route information including the own vehicle control information that is output is used as it is. It is possible to improve safety with respect to operation control. In each process of step S202 to step S206, if any error or the minimum threshold value for determination cannot be exceeded, the route information output using only step S207 is output as the base route. To do. Further, if there is no error or the minimum threshold value for determination is exceeded, the provisional base route output from step S202 to step S206 is determined as the base route.

  The feature of this embodiment is that the simulation is performed by categorizing surrounding vehicles and persons using a moving image of time before the simulation time point. In particular, the base route can be selected with higher accuracy by performing simulation after looking at the behavior of surrounding vehicles and persons one second after the operation information is provided from the own vehicle.

  Therefore, in the first embodiment, the memory 12 that stores the moving image of the camera 1 that captures the moving image around the own vehicle, and the processing unit that performs the process for determining the route of the own vehicle from the stored moving image. 13, the processing unit identifies the category of the surrounding vehicle and the person from the section past the temporary point in order to determine the behavior prediction using one time point of the moving image, and the identified surrounding vehicle and the person By predicting the behavior of the vehicle, surrounding vehicles, and people according to the category, the behavior is predicted using behavior rules according to the surrounding vehicles and people that are categorized in advance. Can do.

  Further, after a predetermined time from the end of the past section, information transmitted from the own vehicle to the surrounding vehicle or person during the past section, and the own vehicle and the surrounding vehicle according to the identified surrounding vehicle and person category By determining the person's behavior prediction, the behavior is predicted using the pre-categorized surrounding vehicle and person, the initial action according to the transmitted information and the action rule according to the initial action, so a more probable prediction Can be done.

  Further, by classifying the categories of the surrounding vehicle and the person from the moving image section using image recognition and time series analysis, it is possible to categorize the behavior of the surrounding vehicle with higher accuracy. Furthermore, CNN can be used for image recognition, and RNN can be used for time series analysis.

  In addition, the behavior rules for performing behavior prediction can be limited to behavior rules according to pre-categorized surrounding vehicles and persons, and initial actions according to information transmitted from the vehicle. It is possible to shorten the processing time until the process is completed.

  In addition, in order to predict behavior prediction for each behavior rule using a neural network, the input layer is an image with state information added to the own vehicle, surrounding vehicles, and the movement of the person, and the output layer is also predicted by the own vehicle, By making an image to which information on the state of surrounding vehicles and people's movements is added, processing can be visualized or driving assistance can be achieved.

  In addition, the neural network is used to predict behavior for each behavior rule, and the blind spot information from the own vehicle is added to the image with the information about the state of the movement of the own vehicle, surrounding vehicles, and people. Can predict behavior.

  Although the action prediction with the highest evaluation value is selected as described above, when the control is started based on the result, there may be a situation in which the behavior of the surrounding vehicle is greatly different. For example, when the control of FIG. 6 is selected, when the right turn signal is issued, the surrounding vehicle maintains the speed in the prediction, but after actually observing the initial movement, the right lane as shown in FIG. Suppose your car started to accelerate. Since such prediction and actual behavior correspond to changes depending on the behavior of surrounding vehicles, it is necessary to rotate steps S201 to S207 in a predetermined cycle, change the category of surrounding vehicles, and perform behavior prediction according to the category. is there. Therefore, a mechanism that always checks the predicted value and the actual behavior is included. And if it has a function of recognizing obstacles, lanes, surrounding vehicles, people, and own vehicle control information in real time in S202, the specific change of the attributes of surrounding vehicles and people from step S203 is also changed immediately. Is possible.

  If any error or the minimum threshold value for judgment cannot be exceeded, the data is input in the data format that can be input to the neural network including CNN without any preprocessing, and the output is The case where the route information including the vehicle control information is used as it is can be considered as follows. The feature of this embodiment is that the simulation is performed by categorizing surrounding vehicles and persons using a moving image of time before the simulation time point. In particular, there is a merit that the base route can be selected more accurately by performing simulation after seeing the behavior of surrounding vehicles and people, for example, after about 1 second after providing operation information from the own vehicle, For example, there is a demerit that proper control is not in time if the person jumps out, for example, after waiting for about one second or more. Therefore, when the obstacle is recognized in front of the own vehicle, the surrounding vehicle, or the person and it is recognized that the speed difference from the own vehicle is large, the route information output directly from step S201 to step S207. Is output as the base route.

Accordingly, a storage unit that stores a moving image of the camera 1 that captures a moving image around the host vehicle and a processing unit 13 that performs a process for determining the route of the host vehicle from the stored moving image are provided. The section has a section for identifying the category of the surrounding vehicle and the person from the section of the moving image, and the process of determining the behavior prediction of the own vehicle, the surrounding vehicle and the person according to the identified surrounding vehicle and the category of the person. By having a process that can quickly determine the behavior prediction, it is possible to appropriately output the behavior prediction even when a surrounding vehicle or a person jumps out.

1: Camera 2: Distance sensor 3: Vehicle speed sensor,
4: Yaw rate sensor
5: Rudder angle sensor 6: Accelerator / brake sensor 7: Driver switch 10: Processing device 11: Input interface 12: Memory 13: Processing unit 14: Output interface 301: Input / output interface

The processing device according to the present invention includes a memory that stores a moving image of a camera that captures a moving image around the own vehicle, and a processing unit that performs a process for determining the route of the own vehicle from the stored moving image. The processing unit identifies a category of behavior using the speed of each of the surrounding vehicle and the person from the section past the temporary point in order to determine a behavior prediction using a point in time of the moving image, and identifies the surrounding Depending on the category of the vehicle and the person, the behavior prediction of the own vehicle, the surrounding vehicles and the person is determined.

The processing device according to the present invention includes a memory that stores a moving image of a camera that captures a moving image around the own vehicle, and a processing unit that performs a process for determining the route of the own vehicle from the stored moving image. The processing unit identifies a category of behavior using the speed of each of the surrounding vehicle and the person from the section past the temporary point in order to determine a behavior prediction using a point in time of the moving image, and identifies the surrounding A first process for determining the behavior prediction of the own vehicle, the surrounding vehicles and the person according to the category of the vehicle and the person, the behavior prediction can be determined earlier than the first process, and a neural network directly from the moving image The second process processing is selected when the behavior prediction is determined using the, and when the speed difference from the own vehicle is large .

Claims (8)

  A memory that stores a moving image of a camera that captures a moving image around the own vehicle, and a processing unit that performs a process for determining the route of the own vehicle from the stored moving image, In order to determine a behavior prediction using a point in time of a moving image, a category of a surrounding vehicle and a person is identified from a section before the temporary point, and the own vehicle is determined according to the identified surrounding vehicle and the category of the person A processing device for determining behavior predictions of the surrounding vehicle and the person.   The processing unit, after a predetermined time from the end of the past section, information transmitted from the own vehicle to the surrounding vehicle and the person during the past section, and the identified surrounding vehicle and the person The processing device according to claim 1, wherein behavior prediction of the own vehicle, the surrounding vehicle, and the person is determined according to a category.   The processing device according to claim 1, wherein the processing unit classifies the category using image recognition and time series analysis when classifying the category of the surrounding vehicle and the person from the section of the moving image.   The processing unit also limits action rules for predicting actions to action rules according to the surrounding vehicle, the person, and the initial action according to information transmitted from the own vehicle. The processing apparatus according to claim 1.   In order to predict behavior prediction for each behavior rule using a neural network, the processing unit uses the input layer as an image to which state information is added regarding the movement of the host vehicle, the surrounding vehicle, and the person, and an output layer The processing apparatus according to claim 4, wherein a state information is added to the predicted movement of the host vehicle, the surrounding vehicle, and the person.   The processing unit adds blind spot information from the own vehicle to an image in which information on the state of the movement of the own vehicle, surrounding vehicles, and people is added to the input layer using a neural network for the behavior prediction for each behavior rule. The processing apparatus according to claim 5.   The processing apparatus according to claim 1, further comprising a process capable of determining the behavior prediction earlier than the process of determining the behavior prediction according to claim 1. Stores the moving image of the camera that captures the moving image around the vehicle,
A processing method for performing a process for determining a route of the own vehicle from the stored moving image, wherein a motion prediction is determined using a point in time of the moving image from a section earlier than the temporary point. The processing method of identifying a category of a surrounding vehicle and a person, and determining a behavior prediction of the own vehicle, the surrounding vehicle and the person according to the identified category of the surrounding vehicle and the person.

Images