WO2022195929A1 - 物体認識装置 - Google Patents

物体認識装置 Download PDF

Info

Publication number
WO2022195929A1
WO2022195929A1 PCT/JP2021/035525 JP2021035525W WO2022195929A1 WO 2022195929 A1 WO2022195929 A1 WO 2022195929A1 JP 2021035525 W JP2021035525 W JP 2021035525W WO 2022195929 A1 WO2022195929 A1 WO 2022195929A1
Authority
WO
WIPO (PCT)
Prior art keywords
object recognition
stage
recognition device
data
processing
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2021/035525
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
昂亮 坂田
昌芳 黒田
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Astemo Ltd
Original Assignee
Hitachi Astemo Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hitachi Astemo Ltd filed Critical Hitachi Astemo Ltd
Priority to JP2023506720A priority Critical patent/JP7478900B2/ja
Priority to US18/252,779 priority patent/US12530221B2/en
Publication of WO2022195929A1 publication Critical patent/WO2022195929A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/48—Program initiating; Program switching, e.g. by interrupt
    • G06F9/4806—Task transfer initiation or dispatching
    • G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
    • G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/48—Program initiating; Program switching, e.g. by interrupt
    • G06F9/4806—Task transfer initiation or dispatching
    • G06F9/4812—Task transfer initiation or dispatching by interrupt, e.g. masked
    • G06F9/4831—Task transfer initiation or dispatching by interrupt, e.g. masked with variable priority
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
    • G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/60—Type of objects
    • G06V20/64—Three-dimensional [3D] objects
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
    • G06V2201/12—Acquisition of 3D measurements of objects

Definitions

  • the present invention relates to an object recognition device.
  • a LiDAR sensor is a sensor that measures the distance to an object by observing the scattered reflection of laser pulses emitted toward the outside world. , we can obtain point cloud-like data as a set of very dense observations.
  • point cloud data is a set of very dense observed values, the amount of data is large, and a large processing load or a long processing time is generated in the external world recognition device that processes the point cloud data.
  • the manufacturing cost of the external recognition device increases.
  • the delay time from updating the point cloud data to outputting the recognition results will be long, and the recognition speed for sudden changes in the external world will decrease, degrading the performance of the external recognition system. put away.
  • Patent Document 1 in a system configured with a plurality of signal processing devices for external recognition sensors, the signal processing devices for external recognition sensors are operated in parallel to obtain external recognition sensor information. Improves processing performance.
  • Patent Document 1 uses a signal processing device for an external world recognition sensor provided for each of a plurality of external world recognition sensors to process the signal of each external world recognition sensor. It improves the processing performance per hit and shortens the processing time, but it does not distribute the large amount of data emitted from a single sensor such as a LiDAR sensor.
  • the input signal unit, the processing unit applied to the input signal, or the processing It is necessary to decentralize the execution timing, etc., and measures to reduce the processing amount itself, and the above invention does not take them into consideration.
  • the present invention was created to solve the above problems, and its purpose is to provide an object recognition device that can reduce the processing load in object recognition processing using point cloud sensor data as input.
  • the object recognition apparatus of the present invention includes a processor that inputs sensor data of a point cloud sensed by a sensor into a software pipeline consisting of a plurality of stages to execute object recognition processing; , wherein the processor has a plurality of processing units that perform processing constituting the object recognition processing assigned to each of the stages, and the memory temporarily stores output data of each of the stages. having a plurality of task queue areas for storage, wherein the processor does not store the output data in each task queue area when the output data of each stage is not valid at the input of the subsequent stage reject.
  • FIG. 1 is a block diagram showing an example of the configuration of an object recognition device according to Example 1 of the present invention; FIG. It is a figure for demonstrating the usage form of the object recognition apparatus which concerns on Example 1 of this invention.
  • FIG. 4 is a diagram for explaining an example of a sensing form of the external world recognition sensor according to Example 1 of the present invention;
  • FIG. 4 is a diagram for explaining an example of a sensing form of the external world recognition sensor according to Example 1 of the present invention;
  • FIG. 4 is a diagram for explaining the time-series relationship between sensing by the external world recognition sensor according to the first embodiment of the present invention and execution timing of a conventional recognition algorithm;
  • FIG. 5 is a diagram for explaining the time-series relationship between the sensing of the external world recognition sensor according to the first embodiment of the present invention and the execution timing of the partial recognition algorithm in the object recognition device to which the present invention is applied;
  • FIG. 4 is a diagram for explaining a processing procedure of a coordinate transformation algorithm as an example of a partial recognition algorithm according to Example 1 of the present invention;
  • FIG. 4 is a diagram for explaining a processing procedure of a clustering algorithm as an example of a partial recognition algorithm according to Example 1 of the present invention;
  • FIG. 4 is a diagram for explaining a processing procedure of a gridding algorithm as an example of a partial recognition algorithm according to Example 1 of the present invention;
  • FIG. 4 is a diagram for explaining a processing procedure of a grouping algorithm as an example of a partial recognition algorithm according to Example 1 of the present invention; It is a figure for demonstrating an example of operation
  • FIG. 4 is a flowchart for explaining a processing procedure of a coordinate conversion algorithm as an example of a partial recognition algorithm according to Example 1 of the present invention;
  • FIG. 4 is a flow chart for explaining a processing procedure of a clustering algorithm as an example of a partial recognition algorithm according to Example 1 of the present invention;
  • FIG. 4 is a diagram for explaining a processing procedure of a grouping algorithm as an example of a partial recognition algorithm according to Example 1 of the present invention. It is a figure for demonstrating an example of operation
  • FIG. 4 is
  • FIG. 4 is a flowchart for explaining a processing procedure of a gridding algorithm as an example of a partial recognition algorithm according to Example 1 of the present invention
  • FIG. 4 is a flowchart for explaining a processing procedure of a grouping algorithm as an example of a partial recognition algorithm according to Example 1 of the present invention
  • FIG. 4 is a flowchart for explaining a processing procedure of a grouping algorithm as an example of a partial recognition algorithm according to Example 1 of the present invention
  • An object recognition device is a processing device that is mounted on a mobile object such as a vehicle and detects an object using a set of observation points obtained as a result of observation by an external recognition sensor such as a LiDAR sensor.
  • Example 1 An example of an embodiment of the present invention will be described below with reference to the accompanying drawings. Also, the correspondence between the description in the explanation and the processing shown in the attached flow chart is shown by writing together the symbols (S1, S2, . . . ) given in the flow chart in the description.
  • FIG. 1 is a block diagram showing an example of the configuration of the object recognition device 3 according to the first embodiment.
  • each process itself may be described as the subject of action, but the actual subject of action of each process is the processor 4 .
  • FIG. 2 is a schematic diagram showing the form of use of the object recognition device 3 according to this application.
  • An object recognition device 3 shown in FIG. 2 is installed in a vehicle and includes an external recognition sensor 1 (such as a LiDAR sensor).
  • an external recognition sensor 1 such as a LiDAR sensor
  • one external world recognition sensor 1 in FIG. 2 is installed at the height of the bumper of the vehicle, the installation location and the number of installation of the external world recognition sensor 1 may be designed arbitrarily, and are limited to the form shown in FIG. not a thing
  • the external world recognition sensor 1 radially and discretely irradiates a plurality of laser beams within a vertical recognizable angular range (range indicated by reference numerals 12 to 13 in FIG. can be obtained.
  • observation points 11 in a plurality of angular directions can be obtained by irradiating a plurality of laser beams.
  • point cloud data as a set of a plurality of observation points 11 having three-dimensional coordinate information can be obtained from the external world recognition sensor 1 .
  • the external world recognition sensor 1 continuously sweeps the irradiation direction of the laser light from the start end 14 toward the end 15 within the recognizable angle range in the horizontal direction.
  • point cloud data is generated as a set of a plurality of observation points 11 having three-dimensional coordinate information.
  • a plurality of laser light irradiation devices installed at different elevation angles in the vertical direction as in the example shown in FIG.
  • a method of sweeping in the horizontal direction is conceivable.
  • a single laser beam irradiation device is swept back and forth from the horizontal starting end 14 to the terminal end 15 within the recognizable angular range, and at the same time, the vertical lower end within the recognizable angular range.
  • point cloud data is generated as a set of a plurality of observation points 11 having three-dimensional coordinate information.
  • the point cloud data representation method is not limited to the coordinate format in the three-dimensional space. The present invention is applicable even when it is realized.
  • the point group data will be treated as information in the form of coordinates (x, y, z) in a three-dimensional space.
  • Fig. 5 shows a typical processing procedure by a conventional object recognition device using point cloud data on the time axis.
  • recognition processing is performed using point cloud data having all three-dimensional coordinate information within the recognizable angle range obtained by the sweeping. is converted into coordinate information, and then recognition processing A and recognition processing B are performed in order to detect an object from the point cloud data, and the result is output.
  • FIG. 6 shows a method for shortening the processing time of the conventional object recognition device shown in FIG.
  • the point cloud data obtained by processing the input data from the external recognition sensor is divided by the number of fixed observation points 11 or the angle range of the sweep, and the partial point cloud data included in each divided section is used. It shows that the process A, the process B, and the process C are partially performed repeatedly.
  • partial recognition algorithm A From the input data processing from the external world recognition sensor, partial recognition algorithm A, partial recognition algorithm B, and partial recognition algorithm C are in a software pipeline, and each processing result is an input for the subsequent processing. are placed.
  • the result of input data processing becomes the input for partial recognition algorithm A
  • the result of partial recognition algorithm A becomes the input to partial recognition algorithm B
  • partial recognition algorithm C also has similar dependencies. be.
  • part of the recognition processing can be partially performed in parallel without waiting for the completion of the sweep of the laser beam irradiation device.
  • Recognition processing for the point cloud data included in the point cloud data section has been completed, and the remaining point cloud data will be processed and the results will be output. Therefore, compared with the processing time of the conventional object recognition device illustrated in FIG. 5, the processing time of the object recognition device when the present invention illustrated in FIG. 6 is applied is shortened.
  • a recognition algorithm for recognizing an object using point cloud data consists of a plurality of partial recognition algorithms. Each partial recognition algorithm is arranged in a software pipeline, and has a dependency relationship such that the processing result of the upper partial recognition algorithm becomes the input to the lower partial recognition algorithm.
  • the partial recognition algorithm A is the processing positioned at the top of the software pipeline, and the processing result data is processed as a task in the task queue A area in the memory through the determination processing in the enqueue determination unit A. is written to
  • the partial recognition algorithm B always checks the task queue A area on the memory, and if there is an unprocessed task, it takes it out and uses it as input data to the partial recognition algorithm B to execute the algorithm.
  • the processing result data is written as a task in the task queue B area in the memory after being judged by the enqueue judging section B, as described above. Also, the task taken out from the task queue and processed is deleted from the memory.
  • the behavior based on the partial recognition algorithm C is also the same as described above.
  • the subject that implements each partial recognition algorithm is the processing unit 5. If there are multiple processing units 5, assignment of partial recognition algorithms to the processing units 5 is determined by the task priority determination unit.
  • the task priority determination unit compares the total number of tasks written in each task queue area in memory, sets a high priority for task queues with many unprocessed tasks, Assign low priority to task queues with few tasks.
  • the processing unit 5 successively confirms the priority set in the task queue, and preferentially executes the partial recognition algorithm corresponding to the untreated task in the task queue set with high priority.
  • the purpose of the task priority determining section is to determine which partial recognition algorithm is to be assigned to each processing unit 5 when the processor 4 to which the present invention is applied has a plurality of processing units 5 .
  • the allocation of processing units 5 is determined by the priority of task queues processed by respective partial recognition algorithms determined by the task priority determination unit.
  • the priority of the task queue is represented by numbers, and the smaller the number, the higher the priority.
  • the plurality of processing units are assigned to execute the highest priority partial recognition algorithm according to the priority set in the task queue corresponding to each partial recognition algorithm.
  • the priority is determined by comparing the amount of unprocessed tasks written to each task queue, and giving priority to the partial recognition algorithm that inputs the task queue with the largest amount of unprocessed tasks. do.
  • task queues A, B, and C have the largest amount of unprocessed tasks in that order, so the partial recognition algorithms corresponding to them are given higher priority in the order A, B, and C. is shown.
  • a coordinate transformation algorithm is shown in FIG. 7 as an example of a partial recognition algorithm.
  • the coordinate conversion algorithm receives data transmitted from the external recognition sensor 1 such as LiDAR (S1), decodes the data according to the data format unique to the sensor (S2), and converts it into coordinate format data recognizable by the partial recognition algorithm in the latter stage. 30 conversion processing is performed (S3).
  • FIG. 7 illustrates the coordinate format data 30 using a three-dimensional orthogonal coordinate system. It can also be in a form suitable for the algorithm.
  • (x, y, z) is used as expression data using a three-dimensional orthogonal coordinate system with the installation position of the external recognition sensor as the origin for consistent explanation.
  • the scope of application of the present invention is not limited to the coordinate format data 30 alone.
  • the coordinate format data 30 is input to the enqueue determination unit 21 as the output of the coordinate transformation algorithm to determine whether it should be written to the task queue area 22 for the coordinate transformation algorithm.
  • the enqueue judgment of the coordinate transformation algorithm it is judged whether or not the data can be used as an input to the partial recognition algorithm in the subsequent stage, and the data that has passed the judgment is written as a task in the task queue area 22 for the coordinate transformation algorithm (S6).
  • the data rejected by the judgment is not written in the task queue area 22 described above, but erased on the spot or written in another memory area as rejection data.
  • a specific determination method in the enqueue determination unit 21 is based on whether the coordinate format data 30 is included in the predetermined space.
  • upper and lower limits are set for the coordinate values in the X-axis direction, the Y-axis direction, and the Z-axis direction, respectively, and coordinate format data contained within a rectangular parallelepiped space defined by the upper and lower limit values 30 is accepted, and the coordinate format data 30 located outside the space is rejected (S4).
  • the distance information to the coordinate format data 30 is used to reject the coordinate format data 30 closer than a predetermined threshold, or to reject the coordinate format data 30 farther than a certain threshold, or both.
  • the information contained in the sensor-specific data format indicates that the specific coordinate format data 30 is an invalid value, it may be rejected (S5).
  • the strength, reliability, accuracy, etc. of the coordinate format data 30 are indicated as information contained in the data format unique to the sensor, it may be rejected if the value is equal to or less than a predetermined threshold (S5).
  • a clustering algorithm is shown in FIG. 8 as an example of a partial recognition algorithm.
  • the clustering algorithm takes the coordinate format data 30 as input (S11) and determines whether the plurality of coordinate format data 30 are sufficiently close or related to each other so as to determine that the plurality of coordinate format data 30 are detecting the same object. If the property is recognized, the plurality of coordinate format data 30 are converted into cluster data 31 added with label information for indicating that the same object is detected.
  • the cluster data 31 is input to the enqueue determination unit 23 as the output of the clustering algorithm to determine whether it should be written to the task queue area 24 for the clustering algorithm.
  • the enqueue determination unit 23 of the clustering algorithm determines whether the data can be used as an input to the partial recognition algorithm in the subsequent stage, and the data that has passed the determination is written as a task in the task queue area 24 for the clustering algorithm (S18).
  • the data rejected by the judgment is not written to the task queue area 24 described above, but is erased on the spot or written to another memory area as rejection data.
  • a specific determination method in the enqueue determination unit 23 is based on whether the coordinate format data 30 belongs to the cluster data 31 that holds a certain number of coordinate format data 30 or more. If the relevant coordinate format data 30 does not belong to any cluster data 31 or if the number of coordinate format data 30 held by the cluster data 31 belonging to it is less than a certain number, those coordinate format data 30 are treated as noise and passed. (S16).
  • the cluster data 31 composed of the coordinate format data 30 of a certain number or more is judged to be observing some kind of object, and the enqueue judging section 23 judges it as passing. Since the number of constituent points of the cluster data 31 is updated as the processing of the clustering algorithm progresses, even if the judgment at a certain timing does not pass, there is a possibility that the acceptance condition will be satisfied later, so it is not immediately rejected.
  • the upper and lower limits are set for the coordinate values in the X-axis direction, the Y-axis direction, and the Z-axis direction in the three-dimensional orthogonal coordinate system in FIG. 31 may be passed, and the cluster data 31 located outside the space may be rejected (S17).
  • a gridding algorithm is shown in FIG. 9 as an example of a partial recognition algorithm.
  • the gridding algorithm receives the cluster data 31 as input (S21), and projectively transforms the coordinate information indicated by the cluster data 31 onto the XY plane defined by the X axis and the Y axis shown in FIG. 9 (S22). It is determined which section (X, Y) the coordinate information indicated by the projectively transformed cluster data 31 belongs to among a plurality of rectangular sections arranged in a lattice map pattern provided on the XY plane. After the section 32 on the grid map to which certain cluster data 31 belongs is determined, an operation is performed to update the statistical data linked to the section 32 and held in the memory (S23).
  • Details of the specific statistical data managed in association with a certain section 32 include the number of at least one or more cluster data 31 associated with a certain section 32, the maximum and minimum values of each element of the cluster data 31. value, each variance value of each element of the observation point, each average value of each element of the observation point, the total number of times the cluster data 31 have been associated up to now based on a certain time, and the number of consecutive times of association etc., and it is conceivable to set various kinds of statistical information as the present statistical data.
  • the index information (X, Y) representing the address on the grid map of the partition 32 whose statistical data has been updated is enqueued to determine whether to write it to the task queue area 26 for the gridding algorithm as the output of the gridding algorithm. It is input to the part 25 .
  • the enqueue determination unit 25 of the gridding algorithm determines whether the data can be used as an input to the partial recognition algorithm in the subsequent stage, and the data that has passed the determination is written as a task in the task queue area 26 for the gridding algorithm (S25 ).
  • the data rejected by the judgment is not written in the task queue area 26 described above, but erased on the spot or written in another memory area as rejection data.
  • a specific determination method in the enqueue determination unit 25 is that the index information indicating a certain section 32 whose statistical data has been updated in a certain sensing period is determined to be the address of the same section 32 within a certain time interval defined by the sensing period, for example. has not already been judged by the enqueue judging section 25, it is judged to pass. is regarded as information that causes redundant operations in the partial recognition algorithm of , and is rejected (S24).
  • a grouping algorithm is shown in FIG. 10 as an example of a partial recognition algorithm.
  • the grouping algorithm takes as input (S31) index information representing the divisions on the grid map, and the divisions are close enough to determine that the same object has been detected in a plurality of divisions, or there is a correlation between divisions. If so, it is converted into group information 33 in which label information is added to indicate that the same object is detected in the plurality of sections.
  • the evaluation target partition 32 is assumed to be a partition forming the same group as that partition, and the adjacent partition possesses the label value. If so, the same label value is assigned (S36), and if the adjacent partition does not have a label value, a label value that does not duplicate the label value assigned to the existing group is assigned to each partition as an index corresponding to each partition. It is added to the information (S35).
  • the evaluation target section 32 is assigned to the existing group as a section forming a new group.
  • a label value that does not overlap with the label value may be assigned (S33).
  • New index information is added and updated to the group information 33 corresponding to the label value as an existing or new group (S37).
  • any of the enqueue determination units data that has not been rejected by the determination is written in the task queue area for each partial recognition algorithm as a task to be processed by the subsequent partial recognition algorithm.
  • the data are stored in a first-in, first-out (FIFO) order, and when the subsequent partial recognition algorithm acquires a task from this task queue area, it acquires the oldest data first.
  • FIFO first-in, first-out
  • the object recognition device 3 includes a processor 4 that inputs point cloud sensor data sensed by a sensor (external world recognition sensor 1) into a software pipeline consisting of a plurality of stages to execute object recognition processing, and a memory. .
  • a sensor external world recognition sensor 1
  • the processor 4 includes a plurality of processing units 5 ( processor core).
  • the memory has a plurality of task queue areas (task queues A to C) that temporarily store the output data of each stage. If the output data of each stage is not valid for the input of the subsequent stage, the processor 4 discards the output data without storing it in the respective task queue area. As a result, the processing load of the subsequent stages can be reduced.
  • the processor 4 comprises a plurality of enqueue determination units A to C that determine whether or not to store the output data of each stage A to C in each task queue area (task queues A to C).
  • Each enqueue determination unit discards the output data without storing it in the respective task queue area when the output data of each stage is not valid for the input of the subsequent stage.
  • each processing unit 5 is the subject that performs the processing of each of the enqueue determination units A to C.
  • each enqueue determination unit A to C outputs to each task queue area (task queues A to C) only when the output data of each stage A to C is valid for the input of the subsequent stage.
  • Store data For example, processor and memory resources can be effectively utilized for each stage.
  • each processing unit 5 performs an operation in each stage A to C using sensor data or output data stored in the task queue areas (task queues A to C) of the preceding stage.
  • the sensor data is stream data that is output from a sensor (external recognition sensor 1 such as a LiDAR sensor) and sequentially stored in a memory. Since the sensor data is sequentially processed by the software pipeline, the time required for object recognition processing for one scan is shorter than in the conventional example.
  • processor 4 shown in FIG. It has a task priority determination unit that determines priority. For example, the task priority determining unit assigns higher priority to the task queue areas in descending order of the number of unprocessed output data (tasks) stored in the task queue areas (task queues A to C) of each stage. .
  • the processor 4 dynamically assigns a plurality of processing units 5 to subsequent stages that process the output data stored in the task queue areas (task queues A to C) assigned the highest priority. As a result, the processing load for each stage is leveled, and the throughput can be improved.
  • the processor 4 executes object recognition processing using a software pipeline for each sensor data corresponding to a predetermined range of azimuth angles sensed by the sensor (external world recognition sensor 1). As a result, the time required for object recognition processing for one scan is shorter than in the conventional example.
  • the processing unit 5 in the first stage outputs the output data (coordinate format data 30) including the information indicating invalidity.
  • the enqueue determination unit 21 of the first stage determines that the output data of the first stage is not valid for the input of the subsequent stage when the output data includes information indicating invalidity.
  • the enqueue determination unit 21 of the first stage does not store the output data in the task queue area 22 of the first stage if the output data (coordinate format data 30) of the first stage is not valid for the input of the subsequent stage. As a result, the output data is excluded when the observed value cannot be obtained, and the processing load of the subsequent stage can be reduced.
  • the enqueue determination unit 21 of at least one stage determines that, when the output data of that stage is within the range of the three-dimensional space that can be sensed by the sensor (external recognition sensor 1), the output data of that stage is the output data of the subsequent stage. Determine that the input is valid. This makes it possible to determine whether the output data is valid from the performance (specifications) of the sensor.
  • the enqueue determination unit 25 of at least one stage determines that the second and subsequent overlapping output data among the output data (X, Y) of that stage are not valid. This makes it possible to further reduce the processing load of the subsequent stages.
  • Object recognition processing consists of a first step of decoding sensor data and converting the decoded sensor data into coordinate format data 30 in a first stage, as shown in FIG.
  • the cluster data 31 are generated by grouping the coordinate format data 30 that are close to each other in the second stage, and as shown in FIG.
  • multiple software pipelines are provided, and sensor data corresponding to a predetermined range of azimuth angles sensed by the sensor (external world recognition sensor 1) is input to the corresponding software pipeline.
  • Processors 4 perform object recognition processing through respective software pipelines.
  • the sensor data corresponding to each predetermined range of azimuth angles are sequentially processed by each software pipeline, so that the latency of object recognition processing can be further shortened compared to the conventional example.
  • each processing unit 5 executes processing according to partial recognition algorithms A to C, which indicate processing algorithms constituting object recognition processing assigned to each stage. Thereby, the processing assigned to each stage can be performed by software.
  • the object recognition device 3 to which the present invention is applied can also use a Radar sensor as the external recognition sensor 1 serving as an input to the device.
  • the object recognition device 3 to which the present invention shown in the first embodiment is applied can also be configured for point cloud data as a set of observation points acquired using a Radar sensor.
  • the object recognition device 3 to which the present invention shown in the first embodiment is applied can be similarly configured for point cloud data as a set of observation points extracted from image data acquired using a camera sensor. .
  • the object recognition device 3 to which the present invention is applied is not limited to one type of external recognition sensor 1 that serves as an input to the device. is not limited to one, and the object recognition device 3 to which the present invention shown in the first and second embodiments is applied can be configured using a plurality of sensors.
  • each of the above configurations, functions, etc. may be realized by hardware, for example, by designing a part or all of them with an integrated circuit.
  • each of the above configurations, functions, etc. may be realized by software by a processor interpreting and executing a program for realizing each function.
  • Information such as programs, tables, and files that implement each function can be stored in a recording device such as a memory, a hard disk, an SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.
  • An object based on an object recognition algorithm composed of a memory for sequentially receiving and storing the point cloud data of the external world acquired by sensing of the external world recognition sensor, and the point cloud data stored in the memory and a plurality of partial recognition algorithms.
  • an arithmetic processing unit that executes recognition processing, the arithmetic processing unit comprising a software pipeline consisting of a plurality of stages that perform arithmetic based on each of the partial recognition algorithms, and the memory comprising: and a plurality of task queue areas for temporarily storing input/output data to each stage.
  • Each of the stages performs an operation using the input data stored in the task queue area or the processing result data of the previous stage, and the operation unit receives the processing result data of the previous stage.
  • a plurality of enqueue determination units for determining whether or not to input to the subsequent stage wherein the enqueue determination unit determines whether the processing result data of the previous stage is a valid value for input to the subsequent stage and if it is determined that the value is not valid, the object recognition device discards the data without adding it to the corresponding task queue area.
  • the computing unit further includes a task priority determining unit that sets processing priority of each task for the plurality of task queue areas, and the task priority determining unit The object recognition device, wherein the unit assigns a higher priority to the task queue areas in descending order of the number of unprocessed data items held in the plurality of task queue areas in each stage.
  • the image processing includes decoding data acquired from the external recognition sensor and converting it into coordinate values.
  • the computing unit includes a task priority determining unit that sets the processing priority of each task in the plurality of task queue areas, and the task priority determining unit determines the processing priority of the plurality of tasks in each stage. Giving high priority to the task queue area in descending order of the number of unprocessed data held in the queue area, and giving priority to the stages of the task queue area to which the high priority is assigned computational resources.
  • the point cloud data is sequentially transmitted to the memory each time a predetermined observation point is acquired in sensing, and is sequentially processed by the arithmetic unit to reduce the load on the arithmetic processing units that operate in parallel. It can be leveled and processing efficiency can be improved.
  • the observation value cannot be obtained by the external recognition sensor, and when the observation value cannot be obtained, data indicating invalidity is output, and the observation value is obtained. If invalidated and not the valid value, the observed value is not output to the task queue area, and the data that is the valid value is sensing data within the observation range of the three-dimensional space to be recognized by the object recognition device.
  • the image processing includes a first step of decoding data acquired from the external recognition sensor and converting it into coordinate values; a second step of grouping objects existing at close positions on the coordinates among the coordinate values obtained in the first step and recognizing them as observation points; Independent image processing is parallelized by including a third step of corresponding to the XY plane and a fourth step of grouping and recognizing the information associated in the third step by grouping the observation points that are close to each other. processing becomes possible, and the processing efficiency is improved.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Software Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • General Engineering & Computer Science (AREA)
  • Artificial Intelligence (AREA)
  • Health & Medical Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computing Systems (AREA)
  • Databases & Information Systems (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Image Processing (AREA)
PCT/JP2021/035525 2021-03-17 2021-09-28 物体認識装置 Ceased WO2022195929A1 (ja)

Priority Applications (2)

Application Number Priority Date Filing Date Title
JP2023506720A JP7478900B2 (ja) 2021-03-17 2021-09-28 物体認識装置
US18/252,779 US12530221B2 (en) 2021-03-17 2021-09-28 Object recognition device

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2021043669 2021-03-17
JP2021-043669 2021-03-17

Publications (1)

Publication Number Publication Date
WO2022195929A1 true WO2022195929A1 (ja) 2022-09-22

Family

ID=83320025

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2021/035525 Ceased WO2022195929A1 (ja) 2021-03-17 2021-09-28 物体認識装置

Country Status (3)

Country Link
US (1) US12530221B2 (https=)
JP (1) JP7478900B2 (https=)
WO (1) WO2022195929A1 (https=)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116434185A (zh) * 2023-03-10 2023-07-14 嬴彻星创智能科技(上海)有限公司 使用hdl实现点云聚类方法、装置、电子设备及储存介质
WO2024180708A1 (ja) * 2023-03-01 2024-09-06 日立Astemo株式会社 物標認識装置及び物標認識方法

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2016112724A (ja) * 2014-12-11 2016-06-23 キヤノン株式会社 画像処理装置、画像処理装置の制御方法、及びプログラム
JP2017092899A (ja) * 2015-11-17 2017-05-25 ソニー株式会社 画像処理装置、画像処理方法、及び、プログラム
JP2020148513A (ja) * 2019-03-11 2020-09-17 三菱重工業株式会社 距離推定装置及び距離推定システム
JP2020154830A (ja) * 2019-03-20 2020-09-24 オムロン株式会社 パイプライン演算装置、プログラマブルロジックコントローラ、及び、パイプライン処理の実行方法

Family Cites Families (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10417817B2 (en) * 2014-11-13 2019-09-17 Nvidia Corporation Supersampling for spatially distributed and disjoined large-scale data
US11449061B2 (en) * 2016-02-29 2022-09-20 AI Incorporated Obstacle recognition method for autonomous robots
CN108701029A (zh) * 2016-02-29 2018-10-23 奥林巴斯株式会社 图像处理装置
JP6722278B2 (ja) * 2016-04-11 2020-07-15 オリンパス株式会社 画像処理装置
US10602242B2 (en) * 2017-06-14 2020-03-24 GM Global Technology Operations LLC Apparatus, method and system for multi-mode fusion processing of data of multiple different formats sensed from heterogeneous devices
US11004202B2 (en) * 2017-10-09 2021-05-11 The Board Of Trustees Of The Leland Stanford Junior University Systems and methods for semantic segmentation of 3D point clouds
JP6932664B2 (ja) 2018-03-08 2021-09-08 日立Astemo株式会社 信号処理システム、及びその評価システム、並びにその信号処理システムに用いられる信号処理装置
US10922832B2 (en) * 2018-07-31 2021-02-16 Intel Corporation Removal of projection noise and point-based rendering
US20220126864A1 (en) * 2019-03-29 2022-04-28 Intel Corporation Autonomous vehicle system
US11062454B1 (en) * 2019-04-16 2021-07-13 Zoox, Inc. Multi-modal sensor data association architecture
US11694314B2 (en) * 2019-09-25 2023-07-04 The Regents Of The University Of Michigan Digital foveation for machine vision
US10964089B1 (en) * 2019-10-07 2021-03-30 Sony Corporation Method and apparatus for coding view-dependent texture attributes of points in a 3D point cloud
EP3828704B1 (en) * 2019-11-29 2023-06-28 My Virtual Reality Software AS Real-time data acquisition and visualization for point-cloud recording
US11210261B2 (en) * 2019-12-23 2021-12-28 Verizon Patent And Licensing Inc. Systems and methods for synchronizing frame processing within a multi-stage modular architecture
CA3174535A1 (en) * 2020-03-04 2021-09-10 Geopipe, Inc. Systems and methods for inferring object from aerial imagery
US12416726B2 (en) * 2020-09-18 2025-09-16 Denso International America, Inc. Systems and methods for enhancing outputs of a LiDAR
US11637998B1 (en) * 2020-12-11 2023-04-25 Nvidia Corporation Determination of luminance values using image signal processing pipeline
US11978181B1 (en) * 2020-12-11 2024-05-07 Nvidia Corporation Training a neural network using luminance

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2016112724A (ja) * 2014-12-11 2016-06-23 キヤノン株式会社 画像処理装置、画像処理装置の制御方法、及びプログラム
JP2017092899A (ja) * 2015-11-17 2017-05-25 ソニー株式会社 画像処理装置、画像処理方法、及び、プログラム
JP2020148513A (ja) * 2019-03-11 2020-09-17 三菱重工業株式会社 距離推定装置及び距離推定システム
JP2020154830A (ja) * 2019-03-20 2020-09-24 オムロン株式会社 パイプライン演算装置、プログラマブルロジックコントローラ、及び、パイプライン処理の実行方法

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2024180708A1 (ja) * 2023-03-01 2024-09-06 日立Astemo株式会社 物標認識装置及び物標認識方法
JPWO2024180708A1 (https=) * 2023-03-01 2024-09-06
CN116434185A (zh) * 2023-03-10 2023-07-14 嬴彻星创智能科技(上海)有限公司 使用hdl实现点云聚类方法、装置、电子设备及储存介质

Also Published As

Publication number Publication date
US20230418659A1 (en) 2023-12-28
US12530221B2 (en) 2026-01-20
JPWO2022195929A1 (https=) 2022-09-22
JP7478900B2 (ja) 2024-05-07

Similar Documents

Publication Publication Date Title
JP6843203B2 (ja) 地面検出方法、装置、電子機器、車両及び記憶媒体
CN111721470B (zh) 使用深度学习的力估计
CN110083547B (zh) 异构计算系统及其操作方法
JP5425541B2 (ja) マルチプロセッサ・システム上でデータ・セットを区分化およびソートするための方法および装置
CN104237852B (zh) 用于处理雷达信号的方法、装置和系统
WO2022195929A1 (ja) 物体認識装置
US20140325151A1 (en) Method and system for dynamically managing big data in hierarchical cloud storage classes to improve data storing and processing cost efficiency
KR20140057641A (ko) 데이터센터 리소스 할당
CN111832343B (zh) 跟踪方法和装置、存储介质
CN113454592A (zh) 存储器管理系统
CN112256599B (zh) 一种数据预取方法、装置及存储设备
WO2022017136A1 (zh) 点云数据处理的方法及装置、电子设备及存储介质
US20180046577A1 (en) Thread block managing method, warp managing method and non-transitory computer readable recording medium can perform the methods
de Oliveira et al. Towards optimizing the execution of spark scientific workflows using machine learning‐based parameter tuning
CN116681767B (zh) 一种点云搜索方法、装置及终端设备
CN114066739B (zh) 背景点云滤除方法、装置、计算机设备和存储介质
JP2018132873A (ja) 情報処理装置
KR102057219B1 (ko) 데이터 입출력 포화 검출 장치 및 방법
CN118655544A (zh) 一种定位方法、装置、设备及介质
US20240070326A1 (en) Methods and apparatus to translate hash-based signature signals for machine learning applications
US11682188B2 (en) Image processing apparatus and computer-readable storage medium for storing screen processing program
US20140149716A1 (en) Systems and method for assigning executable functions to available processors in a multiprocessing environment
CN118918187B (zh) 一种基于不变矩的激光slam全局定位方法及装置
CN117370471B (zh) 基于修剪平均的全局预测方法、装置、设备及存储介质
CN114677284A (zh) 地图构建方法、装置、电子设备及存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21931672

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2023506720

Country of ref document: JP

Kind code of ref document: A

WWE Wipo information: entry into national phase

Ref document number: 18252779

Country of ref document: US

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21931672

Country of ref document: EP

Kind code of ref document: A1

WWG Wipo information: grant in national office

Ref document number: 18252779

Country of ref document: US