WO2020110857A1 - Serveur de traitement d'informations routières, procédé de traitement d'informations routières et programme informatique - Google Patents

Serveur de traitement d'informations routières, procédé de traitement d'informations routières et programme informatique Download PDF

Info

Publication number
WO2020110857A1
WO2020110857A1 PCT/JP2019/045414 JP2019045414W WO2020110857A1 WO 2020110857 A1 WO2020110857 A1 WO 2020110857A1 JP 2019045414 W JP2019045414 W JP 2019045414W WO 2020110857 A1 WO2020110857 A1 WO 2020110857A1
Authority
WO
WIPO (PCT)
Prior art keywords
analysis
unit
image
vehicle
inference
Prior art date
Application number
PCT/JP2019/045414
Other languages
English (en)
Japanese (ja)
Inventor
崇弘 黒瀬
明紘 小川
Original Assignee
住友電気工業株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 住友電気工業株式会社 filed Critical 住友電気工業株式会社
Publication of WO2020110857A1 publication Critical patent/WO2020110857A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G—PHYSICS
    • G08—SIGNALLING
    • G08G—TRAFFIC CONTROL SYSTEMS
    • G08G1/00—Traffic control systems for road vehicles
    • G08G1/01—Detecting movement of traffic to be counted or controlled
    • G08G1/04—Detecting movement of traffic to be counted or controlled using optical or ultrasonic detectors
    • G—PHYSICS
    • G08—SIGNALLING
    • G08G—TRAFFIC CONTROL SYSTEMS
    • G08G1/00—Traffic control systems for road vehicles
    • G08G1/09—Arrangements for giving variable traffic instructions
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Definitions

  • This disclosure relates to a traffic information processing server, a traffic information processing method, and a computer program.
  • This application claims priority based on Japanese Patent Application No. 2018-2199112 filed on Nov. 26, 2018, and all the contents described in the above Japanese application are incorporated herein by reference.
  • infrastructure sensor such as a street surveillance camera (hereinafter simply referred to as “camera”)
  • server computer hereinafter simply referred to as “server”
  • sensor information from a fixedly installed sensor
  • infrastructure sensor such as a street surveillance camera (hereinafter simply referred to as “camera”
  • server computer hereinafter simply referred to as “server”
  • sensor information from a server is uploaded to the server
  • sensor information from the infrastructure sensor is uploaded.
  • Patent Document 1 listed below discloses a system for transmitting images and camera parameters obtained from a plurality of cameras from a server to a vehicle.
  • the car navigation system installed in the vehicle generates an image of an arrow indicating the traveling direction of the own vehicle in the image from the image, the camera parameter, and the traveling direction of the own vehicle, and displays the image in combination with the image. It is said that the driver can easily determine the positional relationship between the congested vehicle and the own vehicle, the degree of congestion, and the like because the driver can determine the traveling direction of the own vehicle even when looking at the image at a distant position.
  • the traffic information processing server includes an image receiving unit that receives a real-time image, a vehicle information receiving unit that receives vehicle information including a position of the vehicle from a vehicle, and an image receiving unit for each image.
  • An image analysis unit that includes multiple analysis units that perform analysis with different processing times and that outputs analysis results, and dynamic information that stores multiple analysis results output by the image analysis unit, distinguishing each other according to the processing time required for the analysis.
  • the image analysis unit detects at least the geographical position of the object existing in the image by analysis, and the vehicle information and the geographical position of the object and the geographical position of the object at each of the predetermined timings repeatedly specified.
  • a distribution unit that selects one or a plurality of analysis results stored in the dynamic information storage unit for each vehicle based on the time required for the analysis and distributes the result to the vehicle.
  • a computer receives a real-time image
  • a computer receives vehicle information including a position of the vehicle from a vehicle
  • the computer Depending on the step of executing a plurality of analysis processes that analyze the images with different processing times and outputting the analysis results, and the processing time required for the computer to analyze the plurality of analysis results output by the plurality of analysis processes.
  • At least the geographical position of the object existing in the image is detected by the step of storing the objects in the storage device separately from each other, and the computer detects the vehicle information and the object at each of the predetermined timings periodically repeated.
  • the analysis results stored in the storage device for each vehicle on the basis of the geographical position and the time required for the analysis, and delivering the analysis result to the vehicle.
  • a computer program includes a computer that can wirelessly communicate with the outside via a wireless communication device, an image receiving unit that receives a real-time image, and a vehicle from the vehicle via the wireless communication device.
  • a vehicle information receiving unit that receives vehicle information including a position, an image analysis unit that includes a plurality of analysis units that perform analysis of images with different processing times, and outputs the results, and a plurality of images output by the image analysis unit.
  • the image analysis unit detects at least the geographical position of an object existing in the image by analyzing, and makes the periodic analysis function function as a dynamic information storage unit that distinguishes and stores the analysis result according to the processing time required for the analysis.
  • the analysis result is selected and the analysis result is caused to function as a distribution unit that distributes the analysis result to the vehicle.
  • FIG. 1 is a schematic configuration diagram of a traffic infrastructure system according to a first embodiment of this disclosure.
  • FIG. 2 is a schematic diagram for explaining the function of the simple classification inference unit shown in FIG.
  • FIG. 3 is a block diagram showing the configuration of the simple classification inference unit.
  • FIG. 4 is a schematic diagram for explaining the function of the attribute inference unit shown in FIG.
  • FIG. 5 is a block diagram showing the configuration of the attribute inference unit.
  • FIG. 6 is a schematic diagram for explaining the function of the detailed attribute inference unit shown in FIG.
  • FIG. 7 is a block diagram showing the configuration of the detailed attribute inference unit.
  • FIG. 8 is a schematic diagram for explaining the function of the action prediction and inference unit shown in FIG. FIG.
  • FIG. 9 is a block diagram showing the configuration of the behavior prediction and inference unit.
  • FIG. 10 is a block diagram showing a hardware configuration of the traffic information processing server shown in FIG.
  • FIG. 11 is a flowchart showing a control structure of a program executed by the traffic information processing server shown in FIG. 1 for distributing traffic information.
  • FIG. 12 is a flowchart showing the control structure of a program that realizes the function of the simple classification and inference unit.
  • FIG. 13 is a flowchart showing the control structure of a program that realizes the function of the attribute inference unit.
  • FIG. 14 is a flowchart showing the control structure of a program that realizes the function of the detailed attribute inference unit.
  • FIG. 10 is a block diagram showing a hardware configuration of the traffic information processing server shown in FIG.
  • FIG. 11 is a flowchart showing a control structure of a program executed by the traffic information processing server shown in FIG. 1 for distributing traffic information.
  • FIG. 12 is a flowchart showing
  • FIG. 15 is a flowchart showing the control structure of a program that realizes the function of the behavior prediction and inference unit.
  • FIG. 16 is a flowchart showing the control structure of a program that realizes the function of the distribution processing unit.
  • FIG. 17 is a flowchart showing the control structure of a program that executes the distribution process shown in FIG.
  • FIG. 18 is a schematic configuration diagram of a traffic information processing server of the traffic infrastructure system according to the second embodiment of the present disclosure.
  • FIG. 19 is a block diagram showing the configuration of a simplified classification inference unit using a reduced image.
  • FIG. 20 is a flowchart showing the control structure of a program that realizes a simplified classification inference unit using reduced images in the second embodiment.
  • FIG. 21 is a flowchart showing the control structure of the program that realizes the simple classification inference unit in the second embodiment.
  • FIG. 22 is a flowchart showing the control structure of the program that realizes the distribution processing unit in the second embodiment.
  • FIG. 23 is a flowchart showing a control structure of a program that realizes a process of generating a distribution processing thread in the second embodiment.
  • FIG. 24 is a schematic configuration diagram of a traffic information processing server used in the traffic infrastructure system according to the third embodiment of the present disclosure.
  • FIG. 25 is a flowchart showing the control structure of a program that realizes the distribution processing unit of the traffic information processing server according to the third embodiment.
  • FIG. 26 is a schematic configuration diagram of the traffic information processing server of the traffic infrastructure system according to the fourth embodiment of the present disclosure.
  • Patent Document 1 The system of Patent Document 1 described above has a problem in that the driver needs to look at the image displayed on the monitor and interpret the meaning of the image. In particular, when displaying dynamic information on a monitor, it is necessary to look at the screen for a certain period of time, and there is a problem that it is not possible to concentrate on driving.
  • an object of this disclosure is to provide a traffic information processing server, a traffic information processing method, and a computer program that can supply appropriate information for driving assistance to a driver at an appropriate timing by using a dynamic image. Is to provide.
  • a traffic information processing server includes an image receiving unit that receives a time series of real-time images, that is, an image receiving unit that receives real-time images in time series, and a position of the vehicle from the vehicle.
  • the vehicle information receiving unit that receives the vehicle information including, and the image analysis unit including a plurality of analysis units that analyze the images with different processing times and output the analysis results, and the analysis results of the image analysis unit
  • a dynamic information storage unit that stores the information separately according to the processing time required for the analysis, and the image analysis unit detects at least the geographical position of the object existing in the image by the analysis, and the timing that is repeatedly specified, that is, Based on the vehicle information, the geographical position of the object, and the time required for the analysis at each of the predetermined timings that are cyclically repeated, one of the analysis results stored in the dynamic information storage unit for each vehicle, that is, one of Alternatively, a distribution unit that selects a plurality of analysis results and distributes them to the vehicle is included.
  • the vehicle information may further include
  • the dynamic image can be used to supply the driver with appropriate information for driving assistance at an appropriate timing.
  • the distribution unit determines at least the vehicle information and the geographical position of the object among the plurality of analysis results stored in the dynamic information storage unit for each vehicle based on the vehicle information and the geographical position of the object.
  • a cumulative distribution unit that cumulatively selects and distributes the analysis result obtained in a processing time shorter than the threshold value determined based on the vehicle to the vehicle may be included.
  • the analysis result with a shorter processing time than the analysis result is also delivered to the vehicle.
  • the analysis result in which the processing time is shorter than that can also be effectively used similarly or more.
  • the dynamic image can be used to supply the driver with appropriate information for driving assistance at an appropriate timing.
  • the multiple analysis results have a hierarchical structure in which the hierarchy increases from the analysis result with the shortest processing time to the analysis result with the longer processing time, and the dynamic information storage unit stores the multiple analysis results as a hierarchical structure. Then, at each timing, the cumulative distribution unit selects, for each vehicle, one or a plurality of hierarchies including the analysis result obtained in the processing time shorter than the threshold value in the hierarchical structure, and selects the hierarchies of the selected hierarchies.
  • a hierarchical distribution unit that distributes the analysis result to the vehicle may be included.
  • the analysis results of layers below a certain level are obtained in a processing time shorter than the processing time required for the analysis of that layer, and are obtained from the image closer to the present.
  • the analysis result by the analysis unit whose processing time is shorter than that of the analysis unit will be delivered to the vehicle.
  • an analysis result that has a shorter processing time than that, that is, an analysis result obtained from an image that is closer to the present time can be used similarly or more effectively. ..
  • the dynamic image can be used to supply the driver with appropriate information for driving assistance at an appropriate timing.
  • the distribution unit selects a target vehicle selection unit that selects a vehicle moving toward the object detected from the image as a target vehicle for distribution of the analysis result among the vehicles specified by the vehicle information, and the target vehicle.
  • the dynamic information storage unit stores each vehicle of the target vehicle based on the vehicle information, the geographical position of the detected object, and the time required for the analysis.
  • a second distribution unit that selects any one of the analyzed results, that is, one or a plurality of analyzed results and distributes the selected one to the vehicle may be included.
  • the second distribution unit responds to the selection of the target vehicle by the target vehicle selection unit based on the vehicle information, the geographical position of the detected object, and the time required for the analysis at each timing.
  • an analysis result in which the processing time for obtaining the analysis result is shorter than a predetermined threshold is cumulatively selected and distributed to the vehicle. 3 delivery units may be included.
  • the analysis result obtained in a processing time shorter than the analysis result is also delivered to the vehicle. It In the case of a vehicle in which the analysis result that requires a certain processing time can be effectively used, the analysis result in which the processing time is shorter than that can also be effectively used similarly or more. As a result, the dynamic image can be used to supply the driver with appropriate information for driving assistance at an appropriate timing.
  • the plurality of analysis units includes at least a first analysis unit having a shortest processing time among the plurality of analysis units and a second analysis unit having a longer processing time than the first analysis unit, and the distribution unit is a vehicle.
  • a first comparing section that compares the distance from the object detected from the image from the vehicle with a first threshold value, and a comparison by the first comparing section
  • a process of reading the analysis result of the first analysis unit from the dynamic information storage unit and delivering it to the vehicle and the analysis results of the first analysis unit and the second analysis unit are stored in the dynamic information storage unit. It may also include a delivery execution unit that selectively executes the process of reading out from the device and delivering it to the vehicle.
  • the time for the vehicle to reach the position of the object is relatively short.
  • Information close to the current state at the detected position of the object can be distributed to the vehicle by distributing the analysis result of the first analysis unit having a short processing time to such a vehicle. Further, it is useless to distribute the analysis result obtained from the old image to the vehicle. Therefore, the vehicle can effectively support the driver by using the distributed information without wasting the communication band.
  • the distance between the vehicle and the object is larger than the first threshold value, the time for the vehicle to move to the position of the object is relatively long. Therefore, it is possible to effectively utilize not only the current state but also information before that.
  • the analysis result obtained in a processing time shorter than the analysis result is also delivered to the vehicle.
  • the analysis result in which the processing time is shorter than that can also be effectively used similarly or more.
  • the dynamic image can be used to supply the driver with appropriate information for driving assistance at an appropriate timing.
  • the first analysis unit includes a classification analysis unit that receives an image received by the image reception unit as an input, and analyzes an object existing in the image, a class to which the object belongs, and a geographical position of the object.
  • the second analysis unit may include an attribute analysis unit that analyzes the attribute of the object analyzed by the first analysis unit based on the output of the first analysis unit.
  • the second analysis unit analyzes the attribute of the object based on the processing result of the first analysis unit. Therefore, the analysis result of the second analysis unit is obtained after the processing result of the first analysis unit.
  • the analysis result of the first analysis unit and the accumulation of the analysis results of the first and second analysis units are delivered to the vehicle according to the distance between the vehicle and the position where the image was obtained.
  • the communication band can be effectively used by not transmitting the analysis result of the second analysis unit to the vehicle that cannot be effectively used even if the analysis result of the second analysis unit is transmitted.
  • both the analysis result of the first analysis unit and the analysis result of the second analysis unit can be effectively used.
  • the dynamic image can be used to supply the driver with appropriate information for driving assistance at an appropriate timing.
  • the second analysis unit may use the output of the first analysis unit to analyze the image.
  • the second analysis unit uses the processing result of the first analysis unit. Therefore, the analysis result of the second analysis unit is obtained after the processing result of the first analysis unit.
  • the analysis result of the first analysis unit and the accumulated analysis result of the first and second analysis units are delivered to the vehicle according to the distance between the vehicle and the position of the object.
  • the communication band can be effectively used by not transmitting the analysis result of the second analysis unit to the vehicle that cannot be effectively used even if the analysis result of the second analysis unit is transmitted.
  • both the analysis result of the first analysis unit and the analysis result of the second analysis unit can be effectively used.
  • the dynamic image can be used to supply the driver with appropriate information for driving assistance at an appropriate timing.
  • the second analysis unit may include a single image analysis unit that executes an analysis on one image at a time, and the first analysis unit is configured to execute a single image analysis unit when analysis on a certain image is completed.
  • a determination unit that determines whether or not another image is being analyzed, based on the determination result of the determination unit, a process of skipping analysis of a single image analysis unit for an image, and a single image analysis unit
  • a selective execution unit that selectively executes the process of starting the execution of the analysis on a certain image may be included.
  • the second analysis unit uses the output of the first analysis unit, if the second analysis unit is still processing at the end of the processing of the first analysis unit for the new image, the selective execution unit Due to the function, the second analysis unit does not process this new image.
  • the calculation resources are not sufficient, the whole operation can be maintained by thinning out the processing of the second analysis unit in this way. If there are analysis units that utilize the output of the second analysis unit, they will not process this new image. Therefore, further calculation resources can be saved.
  • Each of the plurality of analysis units may operate without receiving an output from any other analysis unit of the plurality of analysis units as an input, at the same time when the image is input, starting the image analysis with the image as an input. ..
  • each analysis unit operates independently, the processing of each analysis unit is not affected by the processing results of other analysis units. As a result, it is possible to efficiently analyze each image.
  • Each of the plurality of analysis units may start an analysis process for processing the image each time the image is input, execute the analysis processes in parallel, and output the analysis result for each image.
  • each analysis unit starts an analysis process for processing the image. Since all the analysis processing is performed on all the images, the time interval between the obtained analysis results is the same as the interval at which the original image is received in any processing. Therefore, no matter what timing the analysis result is delivered to the vehicle, the latest analysis result available at that time can be delivered to the vehicle by the output from each analysis unit.
  • a traffic information processing method includes a step in which a computer receives a time series of real-time images, that is, a step in which real-time images are received in time series;
  • the analysis results of the analysis processing that is, the plurality of analysis results output by the plurality of analysis processing are distinguished from each other according to the processing time required for the analysis and stored in the storage device; and the object existing in the image by the analysis of the image.
  • the computer repeats, based on the vehicle information, the geographical position of the object, and the time required for the analysis, at each of the repeatedly specified timings, that is, the predetermined repeated cyclical timings. Selecting one of the analysis results stored in the storage device for the vehicle, that is, one or a plurality of analysis results, and delivering the analysis result to the vehicle.
  • the vehicle information may further include a moving direction and a moving speed.
  • the results are stored in the storage device separately from each other according to the processing time.
  • one of the analysis results stored in the storage device that is, one or more analysis results is selected and delivered to the vehicle based on the vehicle information and the geographical position of the object detected from the image. Since the output of the analysis process is selected based on the geographical position of the vehicle and the object, the output of the analysis process of the processing time determined in relation to the positions of the vehicle and the object is delivered to the vehicle. Therefore, the dynamic image can be used to supply the driver with appropriate information for driving assistance at an appropriate timing.
  • a computer program is an image that receives a time series of real-time images, that is, an image that receives real-time images in time series by a computer capable of wireless communication with the outside via a wireless communication device.
  • a receiving unit that is, an image receiving unit that receives real-time images in time series, a vehicle information receiving unit that receives vehicle information including the position of the vehicle from the vehicle via a wireless communication device, and processes the images.
  • An image analysis unit including a plurality of analysis units that performs different analysis times and outputs a plurality of analysis results, and a storage unit that stores the plurality of analysis results output by the image analysis unit separately from each other according to the processing time required for the analysis.
  • the image analysis unit detects at least the geographical position of an object existing in the image by analysis, and further, the timing is specified repeatedly, that is, the predetermined timing is repeated periodically.
  • the timing is specified repeatedly, that is, the predetermined timing is repeated periodically.
  • one of the analysis results stored in the dynamic information storage unit for each vehicle that is, one or more
  • the analysis result is selected and the analysis result is caused to function as a distribution unit that distributes the analysis result to the vehicle.
  • the vehicle information may further include a moving direction and a moving speed.
  • the dynamic information storage unit may be configured to store a plurality of analysis results output by the image analysis unit in the storage device, distinguishing them from each other according to the processing time required for the analysis.
  • the plurality of analysis results which are the results, are stored in the dynamic information storage unit separately from each other according to the processing time.
  • the distribution unit selects one of the analysis results, that is, one or more analysis results, and distributes it to the vehicle based on the vehicle information and the geographical position of the object detected from the image. Since the analysis result is selected based on the vehicle and the geographical position of the detected object, the analysis result of the processing time determined in relation to the position of the vehicle and the object is distributed to the vehicle. Therefore, the dynamic image can be used to supply the driver with appropriate information for driving assistance at an appropriate timing.
  • a traffic information processing server As described above, according to the present disclosure, a traffic information processing server, a traffic information processing method, and a computer, which are capable of supplying appropriate information for driving assistance to a driver at an appropriate timing using a dynamic image. Can provide programs.
  • a traffic infrastructure system 100 is provided with an intersection 110 or other signal 110, and an infrastructure camera 114 that continuously captures and outputs images of a certain area including the intersection.
  • the time series of the images output from the infrastructure camera 114 are received and analyzed, and the objects such as the pedestrian 112 near the intersection and the drivers such as vehicles 118, 120, 122, and 124 on the road to be noted
  • a traffic information processing server 116 that distributes information to each vehicle.
  • the traffic information processing server 116 performs a predetermined image analysis on the time series of images received by the camera image receiving unit 150 and the camera image receiving unit 150 that receives the time series of images from the infrastructure camera 114 in real time,
  • the image analysis unit 152 that outputs the analysis result and the dynamic information storage unit 154 that stores and accumulates the image analysis result output by the image analysis unit 152 are included.
  • the image analysis unit 152 performs a plurality of inference processes with different processing times on the image and outputs the position, attribute, etc. of an object such as the pedestrian 112 as an analysis result.
  • the time required for the plurality of inference processes differs depending on the complexity.
  • the dynamic information storage unit 154 distinguishes the results of a plurality of inference processings having different processing times from each other, and those having a short processing time form a lower hierarchy, and those having a long processing time form an upper hierarchy.
  • the traffic information server 116 further receives, from the vehicles 118, 120, 122, 124 and the like, vehicle information receiving unit 156 that receives vehicle information including the position of each vehicle, the moving direction, the moving speed, and the like by wireless communication. Then, a vehicle information storage unit 158 that stores this vehicle information and a target vehicle to which the analysis result is to be transmitted are selected from the vehicles specified by the vehicle information stored in the vehicle information storage unit 158 at regular intervals. , Based on the vehicle information of the target vehicle, the position of the detected object obtained from the image captured by the infrastructure camera 114, and the length of processing time required for the inference processing in which the analysis result is output in the image analysis unit 152. And a distribution processing unit 160 that selects and distributes the analysis result stored in the dynamic information storage unit 154.
  • the image analysis unit 152 infers, for each image received by the camera image reception unit 150, a rough position in the image of an object in the image, its classification, and position coordinates indicating its geographical position.
  • a simple classification inference unit 180 an attribute inference unit 182 that infers an attribute of the object by extracting an image of the object from the image based on the rough position in the image of the object inferred by the simple classification inference unit 180,
  • the inference unit 182 performs a process of increasing the resolution of the image extracted from the original image, and the detailed attribute inference unit 184 that infers the detailed attribute of the object from the obtained high-resolution image, the simple classification inference unit 180, and the attribute It includes an inference unit 182, an inference result of the detailed attribute inference unit 184, and an action prediction inference unit 186 that predicts future actions of the object detected by the simple classification inference unit 180 based on these past inference results. ..
  • the distribution processing unit 160 is based on the vehicle information stored in the vehicle information storage unit 158 and the distances between the detected object detected from the image captured by the infrastructure camera 114 and the vehicles 118, 120, 122, 124. , Has a function of selecting and transmitting the information stored in the dynamic information storage unit 154 to each vehicle.
  • the information to be transmitted to each vehicle is obtained by comparing the distance between each vehicle and the detected object with the first to third threshold values D1 to D3, and the result and the moving speed of each vehicle. Select according to. A method of selecting information by the distribution processing unit 160 will be described later with reference to FIGS. 16 and 17.
  • the simple classification inference unit 180 shown in FIG. 1 infers a rough position of an object in the input image 200 to which the driver should pay attention, and a rough classification thereof.
  • the simple classification inference unit 180 outputs a bounding box (frame 210) that is a quadrangle surrounding the image of a pedestrian in the image 200 and an inference result that the pedestrian is a pedestrian.
  • Such processing can be performed at high speed by using a convolutional neural network trained in advance with teacher data.
  • a convolutional neural network trained in advance with teacher data As a program for this purpose, there is, for example, tinyYOLO (https://pjreddie.com/darknet/yolo/) that operates at high speed.
  • the bounding box of the object in the moving image and the class to which the object belongs are output as the inference result.
  • the inference result is also output along with the reliability. Therefore, the convolutional neural network is trained in advance by using the image including the class of the object to be noticed by the driver and the image including the other objects as the teacher data, and selecting the class of the object to be noticed. It is possible to obtain the inference result of the position of the target object and its classification by, for example, selecting the belonging object whose reliability is equal to or higher than a predetermined value.
  • tinyYOLO you can also adjust the accuracy.
  • a tool with higher accuracy has been announced by tinyYOLO.
  • the accuracy increases, the number of inputs to the neural network, the number of hidden layers, and the number of neurons in the hidden layers increase, and it goes without saying that the processing time becomes long.
  • the simple classification and inference unit 180 detects a frame 210 of an image of an object belonging to a class to which the driver should pay attention, that is, an image of an object such as a pedestrian, in each of the images 200 by the above-described processing. And outputs the image 202, and outputs the class of the object (pedestrian, bicycle, car, animal,...) As the result of the simple classification inference.
  • the simple classification and inference unit 180 also estimates and outputs the geographical position coordinates of the target object based on the position of the target object in the image and its class. The results of these simple classification inferences are stored in the dynamic information storage unit 154 together with the time stamp when the image was captured.
  • the simple classification inference unit 180 receives an image normalization processing unit 230 that normalizes the image 200 into an image of a predetermined size, and an image normalized by the image normalization processing unit 230.
  • the position of the object and its class are detected, and for each object, the coordinates of the frame surrounding the object, the class identifier to which the object belongs, the reliability of class classification, and the geographical position coordinates of the object.
  • a pre-learned convolutional neural network 232 that outputs simple classification inference results 234, 236, 238 and the like.
  • the substance of the convolutional neural network 232 is a program such as tinyYOLO described above in this embodiment.
  • the normalization referred to here is to divide an image into a predetermined number of blocks in each of the vertical and horizontal directions to divide the image into a predetermined number of blocks.
  • the reason for normalizing the image in this way is that the number of inputs of the convolutional neural network 232 must be a predetermined number.
  • learning of the convolutional neural network 232 can be performed by artificially synthesizing a large number of learning images based on the specifications of the camera, the installation position and orientation of the camera, and information regarding the imaging target area.
  • the process of inferring the attribute of an object such as a pedestrian performed by the attribute inference unit 182 cuts out each image of the object from each frame such as the frame 210 in the image 202, and cuts the image. From the image 250 of the target object in the issued image, attributes such as the body orientation of the target object and the distinction between adults and children in the case of a person are inferred, and the attribute inference result 274 is stored in the dynamic information storage unit 154. It has a function to output with the time stamp of the original image.
  • the attribute inference unit 182 basically performs the same processing as the simple classification inference unit 180 for a longer time, and the image 250 cut out from the image 202 is displayed.
  • the image normalization processing unit 270 that performs normalization processing with a finer block (more blocks) than the normalization by the simple classification inference unit 180, and the image normalized by the image normalization processing unit 270 are input.
  • a convolutional neural network 272 that infers whether the object is a human body and is an old man, an adult, or a child, and outputs it as an attribute inference result 274.
  • the configuration of the convolutional neural network 272 is similar to that of the convolutional neural network 232 shown in FIG.
  • the image normalization processing unit 270 shown in FIG. 5 divides the image into finer blocks than the image normalization processing unit 230 and normalizes the image into an image composed of a larger number of blocks as a whole. Therefore, the number of inputs to the convolutional neural network 272 is larger than the number of inputs to the convolutional neural network 232.
  • the inference by the image normalization processing unit 270 can make more detailed inference about the attribute than the inference by the convolutional neural network 232.
  • the time required for the processing becomes long.
  • the attribute inference result 274 includes the probabilities that, assuming that the class of the object is a human, the orientation of the body is in each of the eight surrounding directions. These directions are eight directions that are rotated 45 degrees clockwise about the human axis with reference to the posture of the human facing the camera. That is, this information includes eight probability values, and the sum of them is 1.
  • the attribute inference result 274 further outputs the probabilities indicating that the human being, which is the target object, belongs to the age group of an elderly person, an adult, and a child. These include a total of 3 probability values, the sum of which is 1.
  • the detailed attribute inference unit 184 has a function of inferring more detailed attributes than the attribute inference unit 182. That is, the detailed attribute inference unit 184 performs a process called super-resolution processing on the clipped image 250 to generate a high resolution image 290, and infers detailed attributes for the high resolution image 290. , To the dynamic information storage unit 154.
  • the detailed attributes here are the orientation and posture of the face.
  • the face orientation includes the probability that the face faces in each of the eight surrounding directions, and the probability that the face faces in each of the upper and lower five directions.
  • the posture is represented as a three-dimensional skeleton coordinate including a set of characteristic points of the human skeleton.
  • a coordinate candidate based on the position of the face is estimated with a probability, and the one with the highest probability among those combinations is set as the three-dimensional skeleton coordinate.
  • the detailed attribute inference unit 184 performs a normalization process for dividing the cut-out image 250 into a predetermined number of blocks and outputs a normalized image.
  • a super-resolution processing unit 302 for performing super-resolution processing on this normalized image to generate and output a high-resolution image 290; and inputting the high-resolution image 290, outputting a detailed attribute 306.
  • a pre-trained convolutional neural network 304 A pre-trained convolutional neural network 304.
  • the normalization process by the image normalization processing unit 300 divides the image 250 into a larger number of blocks than the normalization process by the image normalization processing unit 270.
  • the super-resolution processing unit 302 applies a super-resolution restoration model (not shown) prepared in advance to each of these blocks to perform processing for increasing the resolution of the low-resolution portion.
  • the super-resolution restoration model can be considered as a database that stores a large number of pairs of high-resolution images and reduced-resolution images of the images. For each part of the low-resolution image, the most similar low-resolution image is found in the database and replaced with the high-resolution image paired with the low-resolution image to increase the resolution of that part of the image.
  • the high-resolution image 290 Since the number of pixels forming the high-resolution image 290 increases due to the processing by the super-resolution processing unit 302, the number of inputs to the convolutional neural network 304 also increases, and the time required for the inference processing by the convolutional neural network 304 becomes long. .. However, if the high-resolution image 290 is an image close to the original image, it is possible to estimate the detailed attributes (face orientation and orientation) of the target object (for example, person).
  • the behavior prediction inference unit 186 predicts the behavior of the target object using the dynamic information acquired in the past stored in the dynamic information storage unit 154, and moves the behavior prediction result 310. It is output to the specific information storage unit 154. Since it is necessary to use past information in addition to the latest information, the action prediction by the action prediction and inference unit 186 requires a long time compared with any other process.
  • the behavior prediction inference unit 186 stores the most recently inferred simple classification, attribute, and detailed attribute of a certain object stored in the dynamic information storage unit 154. And a predicted recurrent neural network 320 for receiving the behavior prediction together with those time stamps as an input and outputting information representing the behavior prediction (behavior type and predicted position) of the object after a fixed time.
  • the time stamp here, the difference between the time stamp added to the latest simple classification prediction and the time stamp added to each analysis result is used. It is possible to prevent the training data from becoming sparse by using the difference as the time stamp.
  • time stamp in addition to the difference from the time stamp of the simple classification prediction, a time stamp of only time or a time stamp reset in a cycle shorter than 1 hour (for example, every 2 minutes) may be used. This is because the pedestrian's behavior may differ depending on the time, but the learning data may be insufficient if the time stamp is used in a unit that is too long.
  • the use of the time stamp is the same in other embodiments.
  • the recurrent neural network 320 Since the recurrent neural network 320 is used, only the latest information needs to be given to the recurrent neural network 320 as an input. Information based on the previous input is stored in the recurrent neural network 320, and the information is reflected in the subsequent processing. That is, the past input information is reflected in the output of the neural network. These properties of the recurrent neural network are optimal for behavior prediction as in this embodiment. The same applies to the LSTM (Long Short-Term Memory) network and the like. In this embodiment, the action types are classified into some predetermined action types (stop, start, run, reverse, etc.), and the recurrent neural network 320 in the form of probability for each of them. Get output from.
  • the output is obtained from the recurrent neural network 320 in the form of a probability that the target object is divided into a certain number of sections around the current position of the target object and moves to these sections after a certain time.
  • the combinations of the action type and the predicted position the one with the highest probability is adopted as the action type and the predicted position.
  • the traffic information processing server 116 shown in FIG. 1 can be realized by computer hardware and a program executed by the computer hardware.
  • a computer system 330 that realizes the traffic information processing server 116 includes a computer 340 that can be connected to a network 342 such as the Internet and that can communicate with a vehicle or the like by wireless communication using an antenna 346.
  • the computer 340 is a central processing unit (CPU) 350, a bus 352 that is connected to the CPU 350 and that provides a communication path between the CPU 350 and each other functional unit and a communication path between each functional unit, and a bus 352.
  • a GPU Graphics Processing Unit
  • ROM Read-Only Memory
  • RAM Random Access Memory
  • a control program connected to the bus 352 and executed by the CPU 350, information necessary for executing the control program, image time series, vehicle information, parameters of each neural network, a program for implementing the present disclosure, dynamic information, and the like.
  • a hard disk drive 360 for storing the. Since the GPU 354 is present, it is possible to reduce the time required for the learning that needs to perform the operation related to the neural network, particularly the operation for a large amount of data.
  • the computer 340 is further connected to the bus 352, is connected to the wireless communication device 362 having an antenna 346 for providing the CPU 350 with wireless communication with a vehicle, and the bus 352, and connects the CPU 350 to the network 342.
  • a network interface (I/F) 364 that provides a connection to the I/F 366 and an input/output I/F 366 connected to the bus 352, which is provided for inputting and outputting a signal 344 from an external device.
  • An SSD Solid State Drive
  • a control program executed by the CPU 350 information necessary for executing the control program, image time series, vehicle information, parameters of each neural network, a program for implementing the present disclosure, dynamic information, and the like.
  • the program for implementing the present disclosure is provided from an external storage medium, such as a DVD (Digital Versatile Disc), a USB (Universal Serial Bus) memory, etc., which is separate from the main body of the computer 340, and which is distributed independently of the computer. It may be installed in any of the non-volatile storage devices of the computer 340 via a reading device, then loaded into the RAM 358 and executed by the CPU 350. Further, the program code may be directly downloaded from the program server on the network 342 to the RAM 358 of the computer 340 via the network I/F 364, expanded, and executed by the CPU 350.
  • an external storage medium such as a DVD (Digital Versatile Disc), a USB (Universal Serial Bus) memory, etc.
  • the image analysis unit 152 shown in FIG. 1 can be realized by executing some programs on the computer 340.
  • the simple classification inference unit 180, the attribute inference unit 182, the detailed attribute inference unit 184, and the behavior prediction inference unit 186 are all realized by different programs.
  • the program shown in FIG. 11 is a flowchart showing the control structure of the program for activating the program that realizes the simple classification inference unit 180 among these programs.
  • This program is started in response to the camera image receiving unit 150 receiving a new image, determines whether or not the simple classification inference in-execution flag is set, and executes the control flow according to the determination result. Step 390 of branching is included.
  • the simple classification inference in-execution flag is set when the simple classification inference process is started for an image having a program (hereinafter, referred to as “simplified classification inference routine”) that implements the simple classification inference unit 180, and when it is completed. It is a flag to be reset.
  • the simple classification inference unit 180 can process only one image at a time, it is necessary to determine whether or not the simple classification inference unit 180 is executing. In this embodiment, if the simple classification inference unit 180 is executing, the simple classification inference unit 180 cannot process the next image. Such a state is called a busy state. The time when the simple classification inference unit 180 is not performing any processing is called an idle state.
  • the program further includes a step 392 for instructing the simple classification inference routine to start and ending the process when the determination in step 390 is negative.
  • a step 392 for instructing the simple classification inference routine to start and ending the process when the determination in step 390 is negative.
  • this process ends without further processing. That is, according to this program, if the simple classification inference routine is busy when the image is received, nothing is processed for the image.
  • the program is started only when the simple classification inference routine is idle, and the simple classification inference process for the input is executed.
  • the simple classification inference routine started in step 392 of FIG. 11 includes a step 410 of setting a simple classification inference execution flag and a step 412 of starting simple classification inference subsequent to step 410. Including.
  • the input image is normalized and then the normalized image is input to the convolutional neural network 232. , 236, 238, etc. from convolutional neural network 232.
  • the simple classification inference result is output by the number of objects of a specific class (person or the like) detected by the convolutional neural network 232.
  • This program further stores step 414 in which the inference result obtained as a result of the processing of step 412 is stored in the dynamic information storage unit 154 together with the time stamp of the original image, and whether or not the attribute inference execution flag is set.
  • the attribute inference executing flag is set when the attribute inference routine is started to execute the attribute inferring process on an image, and is reset when the attribute inferring routine is executed, like the simple classification inference executing flag. If this flag is set, the attribute inference routine is too busy to start processing a new image.
  • step 416 When the determination in step 416 is negative, this program further passes the image for which the simple classification inference is completed to the attribute inference routine, and instructs step 418 to start the image, and after completion of step 418, the simple classification inference execution flag. And reset the process to end the process 420.
  • the determination in step 416 is affirmative, the control proceeds to step 420 without executing the process of step 418.
  • the attribute inference routine started in step 418 of FIG. 12 sets the attribute inference in progress flag for step 440 and for each of the object frames specified by the processing by the simple classification inference unit 180.
  • a step 446 of determining whether a detailed attribute inference executing flag indicating whether or not the realized detailed attribute inference routine is busy is set and branching the control flow according to the result.
  • step 446 When the determination in step 446 is negative, the program further passes step 448 in which the image of the target object is passed to the detailed attribute inference routine to instruct the start of the detailed attribute inference processing, and the attribute inference executing flag is reset. And 450 to end the process.
  • step 448 When the determination in step 446 is affirmative, step 448 is not executed, and the control directly proceeds to step 450.
  • the detailed attribute inference routine started in step 448 of FIG. 13 sets a detailed attribute inference in-execution flag, step 470, starts detailed attribute inference, and extracts the inference result from the original image. Is stored in the dynamic information storage unit 154 together with the time stamp when the image was captured.
  • the process of step 472 is as shown in FIG. 7, and the image 250 cut out is subjected to image normalization and super-resolution processing to obtain a high-resolution image 290.
  • the detailed attribute 306 is obtained at the output of the convolutional neural network 304.
  • This program further determines whether or not the action prediction inference execution flag is set and branches the control flow according to the result.
  • the action prediction inference routine is executed. It includes a step 478 of starting and a step 480 of ending the processing by resetting the detailed attribute inference in-execution flag.
  • the process of step 478 is not executed, and the control directly proceeds to step 480.
  • the action prediction inference in-execution flag is a flag that is set at the start of execution of the program corresponding to the action prediction inference unit 186 and reset at the end of execution. According to the program shown in FIG. 14, the behavior prediction inference processing is not executed when this flag is set.
  • the action prediction inference routine activated in step 478 of FIG. 14 sets step 500 for setting the action prediction inference in progress flag, the result of simple classification inference, the result of attribute inference, and the detailed attribute inference.
  • the latest one is read from the dynamic information storage unit 154, these are input to the recurrent neural network 320 shown in FIG. 9 to obtain the action prediction result 310, and the inference result of the action prediction is activated.
  • the step 504 of storing in the dynamic information storage unit 154 and the step 506 of resetting the action prediction inference in-execution flag and ending the processing are included.
  • the program that realizes distribution processing unit 160 shown in FIG. 1 includes vehicle information stored in vehicle information storage unit 158 and a detected object detected by simple classification inference unit 180 shown in FIG. Step 530 of performing the following process 532 for all combinations of
  • a process 532 determines whether or not the traveling direction of the vehicle approaches the detected object of the process target vehicle, and if the determination is negative, the process for the combination of the vehicle and the detected object is terminated.
  • Step 540 to perform, and when the determination result of Step 540 is affirmative, Step 542 of executing the processing of distributing the information regarding the detected object to be processed to the target vehicle and ending the processing is included.
  • step 542 shown in FIG. 16 relates to step 560 of calculating a distance D from the detected object to be processed to the vehicle to be processed, the calculated distance D, and the prepared distance. 562 comparing three thresholds D1, D2 and D3 (where D1 ⁇ D2 ⁇ D3) and branching the flow of control accordingly. In the following process, three threshold values V1, V2, and V3 regarding speed, which are compared with the speed of the target vehicle (where V3 ⁇ V2 ⁇ V1) are used.
  • step 564 the latest simple classification and inference result regarding the detected object to be processed stored in the dynamic information storage unit 154, that is, the lowest layer information of the hierarchical structure is delivered to the target vehicle.
  • threshold D1 ⁇ distance D ⁇ threshold D2 the control proceeds to step 566.
  • step 566 it is determined whether the speed of the target vehicle is greater than the threshold value V1. If the determination is positive, the control proceeds to step 564, and if the determination is negative, the control proceeds to step 568.
  • step 568 the latest simple classification inference result and attribute inference result regarding the detected object to be processed read out from the dynamic information storage unit 154 are delivered to the target vehicle. That is, in this case, the information of the lowest layer of the hierarchical structure and the information of the next layer are delivered to the target vehicle.
  • step 570 it is determined whether or not the speed of the target vehicle is greater than the threshold value V2. If the determination is positive, the control proceeds to step 568, and if the determination is negative, the control proceeds to step 572.
  • step 572 the latest simple classification inference result, attribute inference result, and detailed attribute inference result regarding the detected object to be processed read from the dynamic information storage unit 154 are distributed to the target vehicle. That is, in this case, the information of the lower three layers of the hierarchical structure is distributed to the target vehicle.
  • step 574 it is determined whether or not the speed of the target vehicle is greater than the threshold value V3.
  • step 572 it is determined whether or not the speed of the target vehicle is greater than the threshold value V3.
  • step 576 the latest simple classification inference result, attribute inference result, detailed attribute inference result, and action prediction inference result regarding the detected object to be processed read from the dynamic information storage unit 154 are distributed to the target vehicle. .. That is, in this case, information on all the layers of the hierarchical structure is distributed to the target vehicle.
  • the transportation infrastructure system 100 having the configuration described above with reference to FIGS. 1 to 17 operates as follows.
  • the infrastructure camera 114 captures an image of a region to be imaged (in the vicinity of the signal 110) at a predetermined frame rate, and transmits an image to the camera image receiving unit 150 each time the image is captured.
  • the camera image receiving unit 150 Upon receiving the new image, the camera image receiving unit 150 stores it in the storage device, and determines whether the simple classification inference unit 180 is busy (step 390 in FIG. 11). If the simple classification inference unit 180 is busy, nothing is processed for this image. If the simple classification inference unit 180 is not busy (in the idle state), the simple classification inference unit 180 is activated (step 392 in FIG. 11), and the process ends.
  • a simple classification inference execution flag is set in step 410.
  • simple classification inference is started.
  • the input image is normalized and then the normalized image is input to the convolutional neural network 232. , 236, 238, etc. from convolutional neural network 232.
  • the simple classification inference result is output by the number of objects of a specific class (person or the like) detected by the convolutional neural network 232 in the image normalization processing unit 230.
  • a frame 210 of an image of a person or the like class information of an object in the frame 210, its reliability, and geographical position coordinates of the object are detected.
  • step 414 of FIG. 12 the inference result obtained as a result of the process of step 412 is stored in the dynamic information storage unit 154 together with the time stamp of the original image. Further, in step 416, it is determined whether or not the attribute inference executing flag is set. If the determination is negative, in step 418 the image for which the simple classification inference is completed is passed to the attribute inference routine, and its start is instructed. After step 418 ends, in step 420, the simple classification inference in-execution flag is reset and the process ends. Normally, such processing is executed. When the determination in step 416 is affirmative, the control proceeds to step 420 without executing the process of step 418, the simple classification inference in-execution flag is reset, and the process ends. That is, the execution of the attribute inference routine and the skip of the attribute inference routine are selectively executed based on whether the determination result of step 416 is affirmative or negative.
  • step 440 when the attribute inference routine is activated in step 418 of FIG. 12, the attribute inference executing flag is set in step 440.
  • step 442 for each of the object frames specified by the processing by the simple classification and inference unit 180, attribute inference is started for the image of the object in the frame.
  • the inference result obtained as a result is stored in the dynamic information storage unit 154 together with the time stamp of the original image in step 444.
  • step 446 it is determined whether the detailed attribute inference in progress flag is set. When the determination in step 446 is negative, the image of the target object is passed to the detailed attribute inference routine in step 448 to instruct the start of the detailed attribute inference processing.
  • step 450 When the detailed attribute inference routine starts executing, the attribute inference in-execution flag is reset in step 450, and the process ends.
  • step 446 If the determination in step 446 is affirmative, step 448 is not executed and control directly proceeds to step 450 to reset the attribute inference executing flag and end the processing.
  • step 470 when the detailed attribute inference routine is activated in step 448 of FIG. 13, the detailed attribute inference executing flag is set in step 470.
  • step 472 detailed attribute inference is started.
  • step 474 the obtained inference result is stored in the dynamic information storage unit 154 together with the time stamp when the original image was captured.
  • step 476 it is determined whether or not the action prediction inference in-execution flag is set. When the determination in step 476 is negative, that is, when the behavior prediction inference processing is idle, the behavior prediction inference routine is started in step 478. Then, in step 480, the detailed attribute inference in-execution flag is reset, and the process ends.
  • step 476 When the determination in step 476 is affirmative, that is, when the behavior prediction inference processing is busy, the processing in step 478 is not executed, control directly proceeds to step 480, the detailed attribute inference executing flag is reset, and the detailed attribute inference processing is performed. Ends.
  • step 500 when the action prediction inference routine is started in step 478 of FIG. 14, the action prediction inference in-execution flag is set in step 500.
  • step 502 the latest of the simple classification inference result, the attribute inference result, and the detailed attribute inference result is read from the dynamic information storage unit 154, and these are input to the recurrent neural network 320 shown in FIG. And the action prediction result 310 is obtained.
  • step 504 the inference result of the behavior prediction is stored in the dynamic information storage unit 154.
  • step 506 the action prediction inference in-execution flag is reset and the process ends.
  • the process shown in FIG. 15 uses a recurrent neural network for prediction, it is not necessary to trace back to past information and input it to the neural network, although past information is required. Therefore, the process itself is completed in a relatively short time, but all of the simple classification inference, the attribute inference, and the detailed attribute inference must be completed for this process. Therefore, the time until the final result is obtained is the longest as compared with other processes.
  • the dynamic information storage unit 154 shown in FIG. 1 has the simple classification inference unit 180, the attribute inference unit 182, the detailed attribute inference unit 184, and the behavior prediction inference unit 186 for the image from the infrastructure camera 114.
  • the analysis results obtained by are distinguished by their processing time and stored together with the time stamp of the original image.
  • the processing time of the attribute inference unit 182 is longer than the processing time of the simple classification inference unit 180
  • the processing time of the detailed attribute inference unit 184 is the time required for the processing of the attribute inference unit 182. That is, the processing time of the action prediction and inference unit 186 is longer than that of any other processing.
  • the inference result by the simple classification inference unit 180 is stored in the dynamic information storage unit 154 earliest, followed by the inference result by the attribute inference unit 182, and then by the detailed attribute inference unit 184.
  • the inference result is finally stored in the dynamic information storage unit 154 in the order of the inference result by the behavior prediction inference unit 186.
  • an image processed by the simple classification inference unit 180 may not be processed by the attribute inference unit 182 or later, and an image processed by the simple classification inference unit 180 or the attribute inference unit 182 may be used. Even if there are some, they are not processed by the detailed attribute inference unit 184 and later.
  • the delivery processing unit 160 delivers the dynamic information stored in the dynamic information storage unit 154 to the vehicle as follows.
  • the distribution processing by the distribution processing unit 160 is performed at regular intervals.
  • step 530 distribution processing unit 160 determines all combinations of vehicle information stored in vehicle information storage unit 158 and detected objects stored in dynamic information storage unit 154.
  • the following processing is executed. That is, it is determined in step 540 whether the traveling direction of the vehicle of the combination to be processed is the direction of the detected object of the combination to be processed, and if the determination is negative, the process for this combination is terminated and the process for the next combination is performed. Move. If the determination in step 540 is affirmative, the distribution process is executed for this vehicle in step 542, the process for this combination is terminated, and the process proceeds to the next combination. In this way, the information about the detected object is distributed only to the vehicle traveling in the direction of the detected object, that is, the information about the state of the detected object is distributed to the vehicle moving in the direction away from the detected object. However, it has little utility value.
  • step 560 of step 542 shown in FIG. 16 a distance D from the detected object to be processed to the target vehicle is calculated for the target vehicle.
  • the distance D calculated in step 560 is compared with the three threshold values D1, D2 and D3 relating to the distance.
  • step 564 when the distance D ⁇ threshold value D1, in step 564, the latest simple classification inference result regarding the detected object stored in the dynamic information storage unit 154 is delivered to the target vehicle. That is, the information of the lowest layer of the hierarchical structure is delivered to the target vehicle.
  • step 566 it is determined in step 566 whether the speed of the target vehicle is greater than threshold V1. If the determination is affirmative, the process proceeds to step 564, and the latest simple classification inference result of the information of the detected object to be processed is distributed to the target vehicle. If the determination is negative, in step 568, the latest simple classification inference result and attribute inference result of the information of the detected object to be processed read from the dynamic information storage unit 154 are delivered to the target vehicle. That is, the information of the lower two layers of the hierarchical structure is delivered to the target vehicle.
  • step 570 it is determined in step 570 whether the speed of the target vehicle is greater than threshold V2. If the determination is affirmative, the control advances to step 568 to deliver the simple classification inference result and the attribute inference result of the information of the detected object to be processed to the target vehicle. If the determination is negative, in step 572, the latest simple classification inference result, attribute inference result, and detailed attribute inference result of the information of the detected object to be processed read from the dynamic information storage unit 154 is delivered to the target vehicle. To do. That is, the information of the lower three layers of the hierarchical structure is delivered to the target vehicle.
  • step 574 it is determined in step 574 whether the speed of the target vehicle is greater than the threshold value V3. If the determination is affirmative, in step 572, the control reads the latest simple classification inference result, attribute inference result, and detailed attribute inference result from the information of the detected object to be processed from the dynamic information storage unit 154 and distributes them to the target vehicle. To do. When the determination is negative, the latest simple classification inference result, attribute inference result, detailed attribute inference result, and action for the target vehicle read out from the dynamic information storage unit 154 for the target vehicle in step 576. Deliver predictive inference results. That is, the information of all the layers of the hierarchical structure regarding the information of the detected object to be processed is distributed to the target vehicle.
  • a distance D from a vehicle to a detected object for example, pedestrian 112 detected from an image of infrastructure camera 114 is shorter than threshold value D1 (for example, vehicle 118), this vehicle is a pedestrian.
  • D1 for example, vehicle 118
  • the time to reach the position 112 is shorter than that of the other vehicles 120 and the like. Therefore, even if the inference result regarding the pedestrian 112 by the behavior prediction and inference unit 186 is transmitted to the vehicle 118, it is not useful for the vehicle 118.
  • the information that reaches the vehicle 118 as the action prediction result is the information obtained from the old image, during which the state of the pedestrian 112 has changed considerably. This is because it is not helpful for the driver of the vehicle 118 located in the immediate vicinity of the pedestrian 112. On the contrary, such information is not necessary for the driver of the vehicle 118 and wastes the communication band, which is not preferable. Therefore, in the above-described embodiment, only the latest information on the pedestrian 112 (that is, the inference result by the simple classification inference unit 180) is delivered to a vehicle such as the vehicle 118 that is close to the imaging area of the pedestrian 112.
  • the processing time is relatively short among the inference results regarding the pedestrian 112.
  • the inference result by the certain simple classification inference unit 180 and the attribute inference unit 182 is distributed.
  • the information including the inference result by the detailed attribute inference unit 184, which is information having a longer processing time, is also distributed. To do.
  • the distribution processing by the distribution processing unit 160 may be performed every predetermined time as described above, or may be performed every time the camera image receiving unit 150 receives a predetermined number of images. Alternatively, in addition to these, it may be performed when the vehicle information receiving unit 156 receives vehicle information from a new vehicle, or may be performed when a request for information transmission is received from any vehicle.
  • the distribution interval may be adjusted according to the number of vehicle information stored in the vehicle information storage unit 158, or the interval may be shortened when the image received by the camera image receiving unit 150 has some movement, and may be lengthened otherwise. Good. In short, the repetitive distribution process may be performed at the timing determined by some algorithm.
  • the attribute inference unit 182 uses the processing result of the simple classification inference unit 180. This is effective in the sense that the above function can be realized even when the traffic information processing server 116 has insufficient computing resources.
  • the time until the inference result of the attribute inference unit 182 is obtained is the processing time of the simple classification inference unit 180 plus the time required for the processing of the attribute inference unit 182 itself.
  • the processing time is accumulated. Therefore, there is a problem that the final inference result by the behavior prediction/inference unit 186 becomes long.
  • the second embodiment is intended to shorten this time to some extent.
  • the traffic information processing server 600 is different from the traffic information processing server 116 shown in FIG. 1 in that it receives a camera image before the simplified classification inference unit 180 shown in FIG.
  • the image is received from the unit 150 and reduced, and the same process as the simple classification inference unit 180 is performed on the reduced image in parallel to detect the position (cut out) of the object in the image in a shorter time.
  • This is a point that includes a simplified classification inference unit 620 based on a reduced image, which performs classification inference and stores the result in the dynamic information storage unit 154.
  • the reduced image simple classification inference unit 620 and the simplified classification inference unit 180 simultaneously start processing for the same image, the time required for the reduced image simple classification inference unit 620 to output an inference result is shorter. Therefore, it is possible to provide effective information even to a vehicle located so close to the detected object that it cannot be processed in time by the simple classification and inference unit 180.
  • the reduced image simple classification inference unit 620, the simple classification inference unit 180, the attribute inference unit 182, the detailed attribute inference unit 184, and the behavior prediction inference unit 186 are executed in parallel in one program. It is realized by threads.
  • the traffic information processing server 600 of FIG. 1 also includes a distribution processing unit 624 that distributes necessary information to a plurality of target vehicles in parallel processing instead of the distribution processing unit 160 of FIG. 1. Different from 116.
  • information is sequentially delivered one combination at a time for the combination of the target vehicle and the detected object. The time required for this distribution process may not be negligible if the number of combinations of the target vehicle and the detected object increases.
  • the distribution processing unit 624 of the traffic information processing server 600 executes the distribution processing to the target vehicle by parallel processing.
  • the reduced image simple classification and inference unit 620 receives the image 200 from the camera image receiving unit 150, reduces the image, and divides the image into a predetermined number of blocks, thereby normalizing the image.
  • the image reduction normalization processing unit 640 to be converted and the image reduced by the image reduction normalization processing unit 640 are subjected to the same processing as the convolutional neural network 232 shown in FIG. , And 648, and a convolutional neural network 642 that outputs the number of detected objects.
  • the convolutional neural network 642 is configured to perform the same processing as the convolutional neural network 232, and is trained with the same teacher data. However, since the input image of the convolutional neural network 642 is a reduced image, the number of inputs to the convolutional neural network 642 is small, and the number of hidden layers may be smaller than that of the convolutional neural network 232. Since the image is two-dimensional, assuming that the size of the block when normalizing the image is constant, the number of blocks is 1/4 when the image is reduced to 1/2, which is inversely proportional to the square of the reduction ratio. It becomes less in the form. As a result, when the reduction ratio of the image is reduced (so that the image becomes smaller), the time required for the inference of the convolutional neural network 642 is accelerated.
  • the camera image receiving unit 150 Upon receiving the image, the camera image receiving unit 150 activates the simplified classification inference unit 620 and the simplified classification inference unit 180 using the reduced image. That is, the camera image receiving unit 150 passes the image to the simplified classification inference unit 620 based on the reduced image to generate an image simple classification processing thread based on the reduced image, and passes the image to the simple classification inference unit 180 to perform simple processing. Create a classification inference processing thread.
  • a simple classification inference processing thread and a simple classification inference processing thread based on a reduced image are generated, and both are operated in parallel. Further, the simplified classification inference processing thread by the reduced image is generated before the simple classification inference processing thread.
  • the program executed by the thread of the reduced image simple classification inference unit 620 shown in FIG. 19 executes the step 690 of reducing the received image and the simple classification inference for the reduced image. It includes step 692 and step 694 of storing the inference result obtained as a result of step 692 in the dynamic information storage unit 154.
  • the program executed by the thread of the simple classification inference unit 180 stores the result of step 412 of executing the simple classification inference in the dynamic information storage unit 154 together with the time stamp of the image. Storing and ending the process 414. All of these processes are the same as those shown in FIG.
  • the process executed by the thread that realizes the simple classification inference unit 180 is the same as that shown in FIG. 20 except that the process of step 690 does not exist and that the detailed attribute inference thread is generated as the next process of step 694. It has a similar configuration.
  • the processing executed by the thread that realizes the behavior prediction and inference unit 186 is the same as that in FIG. 20 except that the processing of step 690 does not exist and that the thread of the behavior prediction and inference is started in the processing subsequent to step 694. Have a configuration.
  • the program executed by distribution processing unit 624 shown in FIG. 18 is the vehicle specified by the vehicle information stored in dynamic information storage unit 154 and the detection detected from the image of infrastructure camera 114. It includes a step 530 of performing the following process 702 for all combinations with objects.
  • a process 702 determines whether or not the traveling direction of the vehicle to be processed is a direction toward the detected object to be processed, and if the determination is negative, the process on the vehicle to be processed is terminated, and the determination of step 540. Is positive, a step 712 of generating a thread of distribution processing for distributing information regarding the detected object to be processed to the vehicle.
  • step 712 is a step 560 of calculating a distance D from the processing target detected object to the processing target vehicle, the calculated distance D, and four threshold values D1 and D2 relating to the distance. 722 comparing D3 and D4 (where D4 ⁇ D1 ⁇ D2 ⁇ D3) and branching the flow of control accordingly.
  • four threshold values V1, V2, V3, and V4 (provided that V3 ⁇ V2 ⁇ V1 ⁇ V4) relating to the speed, which are compared with the speed of the target vehicle, are used.
  • step 726 the simplified classification inference result obtained by the simplified classification inference unit 620 based on the reduced image, that is, the information of the lowest hierarchy in the hierarchical structure is transmitted to the target vehicle.
  • step 724 it is determined whether the speed of the target vehicle is faster than the threshold value V4. If the determination is positive, the control proceeds to step 726, and if the determination is negative, the control proceeds to step 728.
  • step 728 regarding the detected object to be processed stored in the dynamic information storage unit 154, the simplified classification inference result obtained from the reduced image and the latest simplified classification inference result obtained from the normal image, that is, the hierarchy Information of the lowest two layers of the structure is delivered to the target vehicle.
  • step 566 it is determined whether the speed of the target vehicle is greater than the threshold value V1. If the determination is positive, the control proceeds to step 728, and if the determination is negative, the control proceeds to step 730.
  • step 730 the latest simple classification inference result obtained from the reduced image of the detected object to be processed read out from the dynamic information storage unit 154, the latest simple classification inference result obtained from a normal image, and the attribute inference result. Will be delivered to the target vehicle. That is, in this case, the information of the lowest three layers of the hierarchical structure is delivered to the target vehicle.
  • step 570 it is determined whether or not the speed of the target vehicle is greater than the threshold value V2. If the determination is positive, the control proceeds to step 730, and if the determination is negative, the control proceeds to step 732.
  • step 732 for the target vehicle, the latest simple classification inference result obtained from the reduced image of the detected object to be processed read out from the dynamic information storage unit 154, the latest simple classification inference obtained from the normal image. Results, attribute inference results, and detailed attribute inference results are delivered. That is, in this case, information of the lower four layers of the hierarchical structure is distributed to the target vehicle.
  • step 574 it is determined whether or not the speed of the target vehicle is greater than the threshold value V3. If the determination is positive, the control proceeds to step 732, and if the determination is negative, the control proceeds to step 734.
  • step 734 for the target vehicle, the latest simple classification inference result obtained from the reduced image regarding the detected object to be processed read from the dynamic information storage unit 154, the latest simple classification inference obtained from the normal image. The result, the attribute inference result, the detailed attribute inference result, and the behavior prediction inference result are distributed. That is, in this case, information on all the layers of the hierarchical structure is distributed to the target vehicle.
  • the distance from the vehicle to the detected object to be processed is the same, only the information of the lower hierarchy of the hierarchical structure is transmitted to the vehicle when the moving speed of the vehicle is higher, and the information of the upper hierarchy is transmitted when the speed is slower. Is added and sent.
  • this embodiment compared to the first embodiment, it is possible to send the simple classification inference result by the reduced image to a vehicle whose distance to the target detection object is short, and the distance to the detection object is very large. Even if it is short, effective information can be delivered to the vehicle.
  • the above embodiment is premised on the abundance of computational resources.
  • all of the simple classification inference unit 620, the simple classification inference unit 180, the attribute inference unit 182, the detailed attribute inference unit 184, and the behavior prediction inference unit 186 using reduced images can be threaded. If all of these are threaded, each time the camera image receiving unit 150 receives an image, the reduced image simple classification inference unit 620, the simple classification inference unit 180, the attribute inference unit 182, the detailed attribute inference unit 184, and the behavior prediction inference are performed. New threads are created for all of the units 186 and executed in parallel. In the first embodiment, when the subsequent process is busy, the execution of the subsequent process is not started, and therefore the analysis result may not be obtained for some images. On the other hand, when each process is threaded in the second embodiment, all the analysis results are obtained for each image, and it is possible to avoid the occurrence of a blank period in some information.
  • the distribution processing by the distribution processing unit 624 may be performed, for example, every predetermined time, as in the first embodiment, or may be performed every time the camera image receiving unit 150 receives a predetermined number of images. Good. Or in addition to these. It may be performed when the vehicle information receiving unit 156 receives vehicle information from a new vehicle, or may be performed when receiving a request for information transmission from any vehicle.
  • the distribution interval may be adjusted according to the number of vehicle information stored in the vehicle information storage unit 158, or the interval may be shortened when the image received by the camera image receiving unit 150 has some movement, and may be lengthened otherwise. Good. In short, the repetitive distribution process may be performed at the timing determined by some algorithm.
  • the camera image receiving unit 150 when the camera image receiving unit 150 receives an image, the camera image receiving unit 150 generates a thread of the simplified classification inference unit 620 and a thread of the simplified classification inference unit 180 based on the reduced image.
  • the thread of the simplified classification inference unit 620 based on the reduced image ends the processing earlier than the thread of the simplified classification inference unit 180, and the simple classification inference result is stored in the dynamic information storage unit 154.
  • the simple classification inference unit 180 ends the processing, and stores the simple classification inference result by the normal image in the dynamic information storage unit 154.
  • a thread of the attribute inference unit 182 is generated and attribute inference is started.
  • the attribute inference unit 182 stores the inference result in the dynamic information storage unit 154, and creates the thread of the detailed attribute inference unit 184.
  • the detailed attribute inference unit 184 stores the inference result in the dynamic information storage unit 154 and generates the thread of the behavior prediction inference unit 186.
  • the thread of the action prediction and inference unit 186 performs action prediction and inference based on the latest information stored in the dynamic information storage unit 154, and stores the inference result in the dynamic information storage unit 154.
  • the camera image receiving unit 150 When the camera image receiving unit 150 receives the next image at an arbitrary point during these processes, the camera image receiving unit 150 both the reduced image simple classification inference unit 620 and the normal image simple classification inference unit 180. Create a new thread for. As a result, threads of the attribute inference unit 182, the detailed attribute inference unit 184, and the behavior prediction inference unit 186 are derived from them.
  • the simplified classification inference unit 620 based on the reduced image, the simple classification inference unit 180 based on a normal image, the attribute inference unit 182, the detailed attribute inference unit 184, and the behavior prediction inference unit 186.
  • a plurality of threads operate in parallel for each, and the result is stored in the dynamic information storage unit 154 each time the processing ends. Therefore, in the dynamic information storage unit 154, for each image that arrives, a simplified classification inference unit 620 based on a reduced image, a simple classification inference unit 180 based on a normal image, an attribute inference unit 182, a detailed attribute inference unit 184, and a behavior prediction inference are performed. All the inference results of the unit 186 are saved/stored.
  • the delivery processing unit 624 shown in FIG. 18 executes the following processing at regular time intervals. 22 and 23, in step 530 of FIG. 22, all of the vehicles whose vehicle information is stored in the dynamic information storage unit 154 and the detected objects stored in the dynamic information storage unit 154.
  • a process 702 is executed for the combination. In process 702, it is determined whether or not the traveling direction of the vehicle to be processed is the direction toward the detected object to be processed. If the determination is negative, the process for this combination ends. That is, when the traveling direction of the vehicle is not the direction of the detected object, the information regarding the detected object is not transmitted to the vehicle.
  • step 712 in FIG. 22 is executed. That is, in step 560 of FIG. 23, the distance D between the vehicle to be processed and the detected object to be processed is calculated. In the following step 722, this distance D is compared with four threshold values D1, D2, D3 and D4 (where D4 ⁇ D1 ⁇ D2 ⁇ D3).
  • step 726 is executed and the process ends.
  • step 726 a thread for transmitting the simplified classification inference result obtained by the simplified classification inference unit 620 based on the reduced image, that is, the lowest layer information in the hierarchical structure to the target vehicle is generated.
  • step 724 it is determined in step 724 whether the speed of the target vehicle is faster than threshold V4. If the determination is positive, control proceeds to step 726. If the determination in step 724 is negative, the simplified classification inference result obtained from the reduced image and the latest classification inference obtained from the normal image regarding the detected object to be processed stored in the dynamic information storage unit 154 in step 728. A thread for delivering the result of the simple classification inference, that is, the information on the lowermost layer of the hierarchical structure and the second layer from the bottom to the target vehicle is generated.
  • step 566 it is determined in step 566 whether the speed of the target vehicle is greater than threshold V1. If the determination is positive, the process proceeds to step 728. If the determination in step 566 is negative, the latest simple classification inference result obtained from the reduced image relating to the detected object to be processed read from the dynamic information storage unit 154 in step 730, the latest simple classification inference obtained from the normal image. A thread for distributing the classification inference result and the attribute inference result to the target vehicle is generated. That is, in this case, the information of the lowermost three layers of the hierarchical structure is distributed to the target vehicle.
  • step 570 it is determined in step 570 whether the speed of the target vehicle is greater than threshold V2. If the determination is positive, control proceeds to step 730. If the determination in step 570 is negative, the latest simple classification inference result obtained from the reduced image regarding the detected object of the processing target read from the dynamic information storage unit 154 for the target vehicle in step 732 is obtained from the normal image. A thread for delivering the latest simplified classification inference result, attribute inference result, and detailed attribute inference result is generated. That is, in this case, information of the lower four layers of the hierarchical structure is distributed to the target vehicle.
  • step 574 it is determined whether or not the speed of the target vehicle is greater than the threshold value V3. If the determination is positive, the control proceeds to step 732, and if the determination is negative, the control proceeds to step 734.
  • step 734 for the target vehicle, the latest simple classification inference result obtained from the reduced image regarding the detected object to be processed read from the dynamic information storage unit 154, the latest simple classification inference obtained from the normal image.
  • a thread for distributing the result, the attribute inference result, the detailed attribute inference result, and the behavior prediction inference result is generated. That is, in this case, information on all the layers of the hierarchical structure is distributed to the target vehicle.
  • the threads of the reduced image simple classification inference unit 620, the simple classification inference unit 180, the attribute inference unit 182, the detailed attribute inference unit 184, and the behavior prediction inference unit 186 are executed. They are generated one by one. Since one thread is generated for each image, a plurality of threads may be simultaneously executed depending on the processing. Then, for each image, all the inference results of the simplified classification inference unit 620, the simplified classification inference unit 180, the attribute inference unit 182, the detailed attribute inference unit 184, and the behavior prediction inference unit 186 using the reduced image are obtained. However, since the time required for each process is different, even if these inference results are the latest, the time of the original image is different.
  • the inference result is for a new image, whereas when the processing time is long, it is an inference result for a relatively old image (at least before the processing time). .. Even so, the inference result obtained from each processing unit is added every time an image is input, and it is possible to avoid a situation in which the processing result for some images is not reflected in the inference result.
  • distribution of dynamic information to each vehicle is executed in parallel. It is possible to avoid the problem that the delivery of information to some vehicles is delayed due to the delivery time.
  • each analysis unit may use the output of other analysis units.
  • this disclosure is not limited to such an embodiment. As it is easy to understand, if each analysis unit includes all the processing units necessary for the function realized by itself, each analysis unit does not use the processing of other analysis units and independently analyzes the results. Can be output.
  • both the traffic information processing server 116 and the traffic information processing server 600 process only information from one camera.
  • this disclosure is not limited to such an embodiment, and a single server may process images from multiple cameras.
  • the third embodiment relates to a traffic infrastructure system including such a traffic information processing server.
  • a traffic infrastructure system 750 is an image from a plurality of cameras such as infrastructure camera 114 instead of traffic information processing server 116 in traffic infrastructure system 100 shown in FIG. It includes a traffic information processing server 760 that receives the image information, analyzes the image, and distributes the analysis result to each vehicle.
  • the traffic information processing server 760 includes a camera image receiving unit 780 that receives images in real time from a plurality of cameras including the infrastructure camera 114, a camera information storage unit 782 that stores camera information such as the position of each camera and the imaging range.
  • the image analysis unit 786 that executes the same processing as the image analysis unit 610 of the traffic information processing server 600 shown in FIG. , 786, the information about the camera that has transmitted the image data is read from the camera information storage unit 782, and the image analysis unit 784,..., 786 that corresponds to the camera analyzes the image of the image.
  • a camera image distribution unit 788 that distributes processing.
  • each of the image analysis units 784,..., 786 is a separate process executed in parallel, and in each process, the simplified classification inference unit 620 and the simplified classification inference unit using the reduced image shown in FIG. 180, an attribute inference unit 182, a detailed attribute inference unit 184, a behavior prediction inference unit 186, and the like are generated and executed in parallel to perform image analysis.
  • the analysis results of the image analysis units 784,..., 786 include the time stamp of the original image and the identifier of the camera that has transmitted the original image. ..
  • the traffic information processing server 760 further includes a dynamic information storage unit 790 that stores an inference result that is a result of image analysis output by each of the image analysis units 784,..., 786, and a vehicle information reception unit that receives vehicle information from a vehicle. 792, based on the vehicle information storage unit 794 that stores the vehicle information received by the vehicle information receiving unit 792, the camera information stored in the camera information storage unit 782, and the vehicle information stored in the vehicle information storage unit 794, A distribution processing unit 796 that distributes the dynamic information stored in the dynamic information storage unit 790 regarding the detected object detected from the image of each camera to the vehicle moving toward each camera.
  • the distribution processing unit 796 determines, for each of the cameras, the vehicle and the detected object for the vehicle that is moving toward the detected object detected within the predetermined distance from the camera. Depending on the distance between them and the speed of movement of the vehicle, information of an appropriate layer is selected and distributed in the layered structure of the dynamic information obtained from the image of the camera. Since there are multiple cameras, transmitting information from all cameras to all vehicles may confuse the driver of each vehicle. In addition, the amount of communication will be excessive. Therefore, in this embodiment, the information transmitted to each vehicle is limited.
  • the program that realizes distribution processing unit 796 is read in steps 800 and 800 in which the camera information of all cameras stored in camera information storage unit 782 is read from camera information storage unit 782. 802, which executes the following processing 804 for each camera based on the information obtained.
  • a process 804 executes step 810 for reading information about each detected object obtained from the image of the camera to be processed from the dynamic information storage unit 790, and process 814 for each detected object read in step 810. And step 812.
  • the process 814 reads out from the dynamic information storage unit 790 vehicle information regarding a vehicle that is within a predetermined distance from the processing target detection object and has a moving direction that is the direction of the processing target detection object, and step 816. 23. For each of the vehicles read in 816, step 712 of executing the process of generating the distribution process thread shown in FIG. 23.
  • step 712 includes a process of generating a new thread for distributing the dynamic information obtained from the image of the specific camera to the specific vehicle. Each thread ends its execution when the distribution of the dynamic information obtained from the image of the specific camera to the target vehicle is completed.
  • ⁇ motion> 24 and 25 when the camera image receiving unit 780 first receives an image from any of a plurality of cameras, each process of image analysis corresponding to the camera that has transmitted the image. Is launched. After that, the process continues to operate until, for example, the termination condition that no image is received from the camera for a predetermined time is satisfied.
  • a simplified classification inference unit 620 based on a reduced image similar to that shown in FIG. 18, a simple classification inference unit 180 based on normal images, an attribute inference unit 182, a detailed attribute inference unit 184, and a behavior prediction inference unit 186 are provided. Threads similar to those to be realized are generated, and the outputs of these threads are stored in the dynamic information storage unit 790 together with the camera identifier and the image time stamp.
  • the distribution processing unit 796 reads out all the camera information stored in the camera information storage unit 782 at regular time intervals (step 800 in FIG. 25), and reads the detected object detected from the image of the camera for each camera. (Step 810). A vehicle existing in a predetermined range and moving toward the camera from each of the read detected objects is extracted as a target vehicle (step 816). The distribution processing unit 796 reads the latest dynamic information to which the identifier of the camera is attached from the dynamic information storage unit 790 for each extracted vehicle, and determines the distance between the vehicle and the camera of the vehicle. A distribution processing thread that distributes only dynamic information according to the speed of movement in the direction to each vehicle is generated (steps 818 and 712).
  • the information to be distributed to each vehicle is selected by an algorithm as shown in FIG. 17 or 23, as in the first and second embodiments.
  • images from a plurality of cameras are distributed to a plurality of vehicles.
  • overlapping information will be detected from the images of the cameras capturing the mutually overlapping image capturing areas. Distributing them as separate items to each vehicle risks driver confusion. Therefore, when the detected objects (for example, passersby) detected as a result of the analysis are the same, unifying them makes the driver's assistance more effective.
  • it is determined by using a neural network whether all the combinations of the detected objects detected by the simple classification inference from the images of the two cameras whose imaging areas overlap each other are the same objects or not. To be done. It should be noted that whether or not the image pickup areas of the two cameras overlap may be automatically determined from the camera information, or may be manually set in advance as an initial setting.
  • the traffic information processing server 830 of the traffic infrastructure system 820 has an imaging area stored in the dynamic information storage unit 790 in addition to the traffic information processing server 760 shown in FIG. Determines whether the detected object detected in the image of one camera is the same as the detected object detected in the image of the other camera, based on the simple classification inference result obtained from the latest images of the two overlapping cameras
  • the same object unifying unit 840 having a function of integrating the detected objects determined to be the same into one object in the dynamic information storage unit 790 is included.
  • the attribute inference result, the detailed attribute inference result, the action prediction inference result, and the like in the dynamic information storage unit 790 are also integrated. At this time, both the identifiers of the cameras that have captured the object are maintained.
  • the simplified classification inference unit 620 based on the reduced image is adopted, the inference result is also integrated.
  • the same object unifying unit 840 is realized by a recurrent neural network in this embodiment.
  • This neural network is a vector obtained by concatenating the latest simple classification inference results regarding two detected objects to be compared with a predetermined number of latest simple classification inference results existing in the vicinity of the detected objects in each image. Is input, and the probability that these two detected objects are the same and the probability that they are different detected objects are output. Since the recurrent neural network is used, past information is also reflected in the determination result. When it is determined that the two are highly likely to be the same, a predetermined calculation is performed on the simple classification inference results of both (for example, in the case of a position, the average of the two is calculated), and the two are integrated. Whether or not to integrate other inference results also depends on the result of the judgment for this simple classification inference. If it is determined that the two are the same, the attributes and the like of other inference results are also integrated using the results obtained by the respective predetermined calculations.
  • the same effects as those of the first to third embodiments can be obtained.
  • the information of one integrated object is distributed to each vehicle for the detected objects determined to be the same.
  • the information of the same object will not be delivered in a layered manner, so that the driver's support can be ensured and the communication band can be effectively used.
  • the image analysis is mainly performed by the neural network.
  • this disclosure is not limited to such an embodiment, and image analysis may be performed by any means.
  • which analysis result is to be distributed is determined according to the distance and speed between the vehicle and the geographical position of the object detected from the image.
  • this disclosure is not limited to such embodiments.
  • the structure is such that the moving speed of an object can be estimated by the earliest processing, and the geographical position of the detected object and the relative speed between the vehicle and the detected object (relative speed component in the traveling direction of the vehicle ) And which analysis result to deliver may be determined.
  • the geographical position of the object detected from the image is near the camera and the vehicle is far from the camera (for example, the distance between the position of the camera and the geographical position of the detected object is smaller than D1/2).
  • the analysis result to be distributed may be determined based on the distance between the position of the camera and the vehicle and the speed of the vehicle.

Landscapes

  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Theoretical Computer Science (AREA)
  • Traffic Control Systems (AREA)

Abstract

Ce serveur de traitement d'informations routières comprend : une unité de réception d'image qui reçoit une image en temps réel ; une unité de réception d'informations de véhicule qui reçoit, en provenance d'un véhicule, des informations de véhicule comprenant au moins la position du véhicule ; une unité d'analyse d'image comprenant une pluralité d'unités d'analyse qui effectuent une analyse avec différents temps de traitement sur les images et délivrent des résultats d'analyse ; une unité de stockage d'informations dynamiques qui stocke une pluralité de résultats d'analyse délivrés par l'unité d'analyse d'image séparément les uns des autres selon un temps de traitement requis pour l'analyse, l'unité d'analyse d'image détectant au moins la position géographique d'un objet présent dans l'image par analyse ; et une unité de distribution qui, à chacun des moments prédéterminés qui sont périodiquement répétés sur la base des informations de véhicule, de la position géographique de l'objet et du temps nécessaire à l'analyse, sélectionne un ou plusieurs résultats d'analyse stockés dans l'unité de stockage d'informations dynamiques pour chaque véhicule et distribue les résultats d'analyse au véhicule.
PCT/JP2019/045414 2018-11-26 2019-11-20 Serveur de traitement d'informations routières, procédé de traitement d'informations routières et programme informatique WO2020110857A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2018-219912 2018-11-26
JP2018219912A JP2022034571A (ja) 2018-11-26 2018-11-26 交通情報処理サーバ、交通情報の処理方法、及びコンピュータプログラム

Publications (1)

Publication Number Publication Date
WO2020110857A1 true WO2020110857A1 (fr) 2020-06-04

Family

ID=70852099

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2019/045414 WO2020110857A1 (fr) 2018-11-26 2019-11-20 Serveur de traitement d'informations routières, procédé de traitement d'informations routières et programme informatique

Country Status (2)

Country Link
JP (1) JP2022034571A (fr)
WO (1) WO2020110857A1 (fr)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2022070426A1 (fr) * 2020-10-02 2022-04-07 日本電気株式会社 Dispositif de traitement d'informations, système de transmission d'image et procédé de traitement d'informations
CN114549613A (zh) * 2022-01-30 2022-05-27 清华大学 基于深度超分辨率网络的结构位移测量方法及装置
JP2022170649A (ja) * 2021-04-28 2022-11-10 緯創資通股▲ふん▼有限公司 人体の方向を検出する検出装置、および、人体の方向を検出する検出方法
WO2022244377A1 (fr) * 2021-05-18 2022-11-24 株式会社日立製作所 Système de commande

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH1139589A (ja) * 1997-07-18 1999-02-12 Fuji Electric Co Ltd 交通監視装置および交通監視方法
JP2001169275A (ja) * 1999-12-08 2001-06-22 Nagoya Electric Works Co Ltd マルチタスクスケジューリングによる画像処理の統合方法
JP2009017540A (ja) * 2007-05-31 2009-01-22 Panasonic Corp 画像撮影装置、付加情報提供サーバ及び付加情報フィルタリングシステム
JP2012221253A (ja) * 2011-04-08 2012-11-12 Toshiba Corp 情報提供方法、情報提供プログラム、情報提供システム、管理サーバ、移動端末、記憶媒体
JP2017161998A (ja) * 2016-03-07 2017-09-14 トヨタ自動車株式会社 車両用画像生成装置
WO2019188429A1 (fr) * 2018-03-29 2019-10-03 住友電気工業株式会社 Dispositif, système et procédé de gestion de corps mobile, et programme d'ordinateur

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH1139589A (ja) * 1997-07-18 1999-02-12 Fuji Electric Co Ltd 交通監視装置および交通監視方法
JP2001169275A (ja) * 1999-12-08 2001-06-22 Nagoya Electric Works Co Ltd マルチタスクスケジューリングによる画像処理の統合方法
JP2009017540A (ja) * 2007-05-31 2009-01-22 Panasonic Corp 画像撮影装置、付加情報提供サーバ及び付加情報フィルタリングシステム
JP2012221253A (ja) * 2011-04-08 2012-11-12 Toshiba Corp 情報提供方法、情報提供プログラム、情報提供システム、管理サーバ、移動端末、記憶媒体
JP2017161998A (ja) * 2016-03-07 2017-09-14 トヨタ自動車株式会社 車両用画像生成装置
WO2019188429A1 (fr) * 2018-03-29 2019-10-03 住友電気工業株式会社 Dispositif, système et procédé de gestion de corps mobile, et programme d'ordinateur

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2022070426A1 (fr) * 2020-10-02 2022-04-07 日本電気株式会社 Dispositif de traitement d'informations, système de transmission d'image et procédé de traitement d'informations
JP2022170649A (ja) * 2021-04-28 2022-11-10 緯創資通股▲ふん▼有限公司 人体の方向を検出する検出装置、および、人体の方向を検出する検出方法
JP7235826B2 (ja) 2021-04-28 2023-03-08 緯創資通股▲ふん▼有限公司 人体の方向を検出する検出装置、および、人体の方向を検出する検出方法
WO2022244377A1 (fr) * 2021-05-18 2022-11-24 株式会社日立製作所 Système de commande
CN114549613A (zh) * 2022-01-30 2022-05-27 清华大学 基于深度超分辨率网络的结构位移测量方法及装置

Also Published As

Publication number Publication date
JP2022034571A (ja) 2022-03-04

Similar Documents

Publication Publication Date Title
WO2020110857A1 (fr) Serveur de traitement d'informations routières, procédé de traitement d'informations routières et programme informatique
JP6693587B2 (ja) 情報処理方法および情報処理システム
US11816585B2 (en) Machine learning models operating at different frequencies for autonomous vehicles
JP6741107B2 (ja) 交通状況のリアルタイム検出
US10740658B2 (en) Object recognition and classification using multiple sensor modalities
US10916135B2 (en) Similarity learning and association between observations of multiple connected vehicles
US10849543B2 (en) Focus-based tagging of sensor data
KR20180113447A (ko) 전자 장치, 그의 경고 메시지 제공 방법 및 비일시적 컴퓨터 판독가능 기록매체
JP5278419B2 (ja) 運転シーンの遷移予測装置及び車両用推奨運転操作提示装置
US10994749B2 (en) Vehicle control method, related device, and computer storage medium
JP7278484B2 (ja) 車両交通のアテンションベースの制御
JP6418574B2 (ja) 危険度推定装置、危険度推定方法及び危険度推定用コンピュータプログラム
US11615266B2 (en) Adaptive sampling of stimuli for training of machine learning based models for predicting hidden context of traffic entities for navigating autonomous vehicles
JP7376992B2 (ja) 情報処理装置、情報処理方法、及び、プログラム
US11537819B1 (en) Learned state covariances
US20200249677A1 (en) Probabilistic neural network for predicting hidden context of traffic entities for autonomous vehicles
CN114080629A (zh) 点云中的对象检测
US11624831B2 (en) Obstacle detection method and apparatus and storage medium
US20230048304A1 (en) Environmentally aware prediction of human behaviors
US11501538B2 (en) Systems and methods for detecting vehicle tailgating
US20220343637A1 (en) Traffic flow machine-learning modeling system and method applied to vehicles
CN114488072A (zh) 障碍物检测方法、装置及存储介质
EP3989031A1 (fr) Systèmes et procédés de fusion des données sur la friction des routes pour améliorer les manoeuvres de véhicule
US20230159023A1 (en) Method and electronic apparatus for predicting path based on object interaction relationship
US20240062386A1 (en) High throughput point cloud processing

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19889369

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19889369

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: JP