KR102267040B1 - Objection recognition device, autonomous travel system including the same, and objection recognizing method using the objection recognition device - Google Patents

Abstract

The present invention relates to an object recognition apparatus, an autonomous driving system including the same, and an object recognition method using the object recognition apparatus. An object recognition apparatus according to an embodiment of the present invention includes an object frame information generator, a frame analyzer, an object priority calculator, a frame complexity calculator, and a mode controller. The object frame information generator generates object frame information based on the mode control signal. The frame analyzer generates object tracking information based on the object frame information. The object priority calculator generates priority information based on the object tracking information. The frame complexity calculator calculates the frame complexity based on the object tracking information. The mode control unit generates a mode control signal for adjusting the object recognition range and the amount of calculation of the object frame information generating unit based on priority information, frame complexity, and resource occupancy state. According to the present invention, instability of autonomous driving due to overload can be improved.

Description

Object recognition device, autonomous driving system including same, and object recognition method using object recognition device

The present invention relates to signal processing for object recognition, and more particularly, to an object recognition apparatus, an autonomous driving system including the same, and an object recognition method using the object recognition apparatus.

In order to secure the convenience of a user who uses the vehicle, various electronic devices are provided in the vehicle. In particular, autonomous driving technology capable of recognizing objects in the vicinity and preventing collisions with objects to reach a destination is emerging. In order to secure the stability of autonomous driving, it is important to accurately recognize objects such as other vehicles or pedestrians. In addition, it is required to maintain a certain distance from objects and to select an optimized driving route. Object information recognized in various ways is processed in order to select an optimized driving route.

An autonomous vehicle may come into contact with various objects. In particular, driving during commuting time, driving at an intersection, or driving in an alleyway requires precise control of the vehicle based on the number of objects or the complexity of the driving route. In such a complex surrounding situation, the autonomous driving system may face an overload condition.

Autonomous driving systems can cause system freezes or errors under overload conditions. Sudden system stops or errors can cause road congestion, which in turn can lead to traffic accidents. Therefore, a methodology capable of securing the stability of the autonomous driving system in complex surrounding situations is required. In addition, there is a demand for an autonomous driving system capable of quickly responding to surrounding situations by improving the processing speed of recognized object information.

The present invention may provide an object recognition apparatus capable of improving instability of autonomous driving in an overload state, an autonomous driving system including the same, and an object recognition method using the object recognition apparatus.

An object recognition apparatus according to an embodiment of the present invention includes an object frame information generator, a frame analyzer, a driving path analyzer, an object priority calculator, a frame complexity calculator, a resource detector, an operation mode indicator operator, and a mode controller.

The object frame information generating unit generates object frame information by recognizing at least one object based on the mode control signal. The frame analyzer receives object frame information. The frame analyzer generates object tracking information by calculating an expected position or an expected velocity of an object for the next frame. The driving path analyzer generates driving path information for the next frame based on the object tracking information.

The object priority calculator calculates the possibility of collision between the object and the object recognition device based on the object tracking information and the driving route information. The object priority operator generates priority information about objects based on the possibility of collision.

The frame complexity calculator calculates the object complexity based on the object tracking information. The frame complexity calculator calculates the path complexity based on the driving path information. The frame complexity calculator calculates the frame complexity based on the object complexity and the path complexity. The frame complexity calculator calculates the frame complexity based on the number and distribution of objects.

The operation mode indicator operator generates an operation mode indicator based on frame complexity and resource occupancy state. The arithmetic mode indicator operator determines the absolute value of the arithmetic mode indicator based on frame complexity and determines the polarity of the arithmetic mode indicator based on the resource occupancy state.

The mode control unit generates a mode control signal for adjusting the object recognition range and the amount of calculation of the object frame information generating unit based on priority information, frame complexity, and resource occupancy state. The mode controller provides a high-computational mode control signal, a low-computational mode control signal, or a skip mode control signal to the object frame information generator based on the operation mode indicator.

An autonomous driving system according to an embodiment of the present invention includes a sensor unit, an object recognition device, a vehicle control device, a processor, and a memory. The sensor unit generates sensing information by sensing the periphery. The object recognition apparatus may include an object frame information generator, a frame analyzer, a driving path analyzer, an object priority calculator, a frame complexity calculator, a resource detector, an operation mode indicator operator, and a mode controller. The object recognition apparatus adjusts the object recognition range and the amount of computation of the object frame information generator based on the resource occupancy state of the processor or memory, and the frame complexity. The vehicle control device controls the steering angle and speed of the vehicle based on the driving path information.

An object recognition method according to an embodiment of the present invention includes the steps of generating object frame information, generating object tracking information, generating driving route information, generating priority information, calculating frame complexity; detecting a resource occupancy state, and changing the object recognition range by controlling the operation of object frame information.

For example, in the step of changing the object recognition range, if the resource occupancy state is Full, the object frame information having the lowest priority among the objects for which the object frame information was created in the high operation mode in the previous frame is low. It is created in arithmetic mode. Also, the number of object frame information generated in the low-computation mode is adjusted based on the frame complexity.

For example, in the step of changing the object recognition range, if the resource occupancy state is Full and a plurality of object tracking information in the previous frame are all generated in the low-computational mode, the object frame information having the lowest priority information creation is blocked. In addition, the number of blocking object frame information generation is adjusted based on the frame complexity.

For example, in the step of changing the object recognition range, if the resource occupancy state is the Not Full state, the object frame information having the highest priority information among the objects whose object frame information generation is blocked in the skip mode in the previous frame is created in low-computational mode. Also, the number of object frame information generated in the low-computational mode is adjusted based on the frame complexity.

For example, in the step of changing the object recognition range, if the resource occupancy state is in the Not Full state, the object frame information having priority information among the objects for which the object frame information was generated in the low computation mode in the previous frame is converted to the high computation mode is created In addition, the number of object frame information generated in the high computation mode is adjusted based on the frame complexity.

The object recognition apparatus, the autonomous driving system including the same, and the object recognition method using the object recognition apparatus according to an embodiment of the present invention adaptively adjust the operation required for signal processing according to the state of the resource and the state of the peripheral part of the system. Stops or errors can be prevented, and signal processing speed can be improved.

1 is a block diagram of an object recognition apparatus according to an embodiment of the present invention.
2 is a block diagram of an object frame information generator according to an embodiment of the present invention.
3 and 4 are block diagrams of an autonomous driving system according to an embodiment of the present invention.
5 is a block diagram of a sensor unit according to an embodiment of the present invention.
6 to 8 are diagrams for explaining a process of generating priority information of objects according to an embodiment of the present invention.
9 to 12 are diagrams for explaining changes in the object recognition range and operation mode according to an embodiment of the present invention.
13 is a flowchart of an object recognition method according to an embodiment of the present invention.
14 is a flowchart of a method for controlling operation of object frame information according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described clearly and in detail to the extent that those skilled in the art can easily practice the present invention.

1 is a block diagram of an object recognition apparatus according to an embodiment of the present invention. Referring to FIG. 1 , the object recognition apparatus 100 includes an object frame information generator 110 , a frame analyzer 120 , a driving path analyzer 130 , an object priority calculator 140 , and a frame complexity calculator 150 . ), a resource detector 160 , an operation mode indicator operator 170 , and a mode controller 180 . In the object recognition apparatus 100 of FIG. 1 , components are classified according to functions for convenience of description, but the present invention is not limited thereto and components may be implemented in one chip or integrated circuit.

The object frame information generating unit 110 receives sensing information Si from a sensor unit to be described later. As for the sensing information Si, the object frame information generating unit 110 recognizes an object in the vicinity based on the sensing information Si. The object frame information generating unit 110 may recognize the type, shape, distance, or color of the object by extracting the feature vector of the object. The object frame information generating unit 110 generates object frame information Oi based on the recognized object.

The object frame information generator 110 may generate object frame information Oi using a machine learning algorithm. The object frame information generator 110 may extract feature vectors of objects by using parameters learned by a machine learning algorithm. For example, the object frame information generator 110 may extract a feature vector by using a parameter learned to recognize objects such as a vehicle, a pedestrian, and a guard rail. Also, the object frame information generating unit 110 may use the learned parameter so as not to recognize objects such as rainwater.

The object frame information Oi may include various types of information related to characteristics of an object. For example, the object frame information Oi may include at least one of object ID information, object type information, object location information, object size information, and object color information. The object frame information generating unit 110 generates object frame information Oi in units of frames. The object frame information generating unit 110 generates object frame information Oi different from the object frame information for the previous frame according to the movement of the object or the movement of the user's vehicle.

The frame analyzer 120 receives the object frame information Oi. The frame analyzer 120 generates object tracking information Ti based on the object frame information Oi. The frame analyzer 120 generates object tracking information Ti by predicting the position of the object with respect to the next frame. For example, the frame analyzer 120 may receive the object frame information Oi for the current frame, and compare and analyze the object tracking information Ti and the object frame information Oi for the previous frame.

The frame analyzer 120 may predict the expected position or the expected speed of the object based on the difference between the object tracking information Ti and the object frame information Oi for the previous frame. The frame analyzer 120 may generate the object tracking information Ti based on the predicted position or the predicted speed of the object. Alternatively, the frame analyzer 120 may generate the object tracking information Ti by using the object frame information Oi of the previous frame and the object frame information Oi of the current frame. The frame analyzer 120 generates the object tracking information Ti in units of frames.

The object tracking information Ti may include various types of information related to characteristics of an object. For example, the object tracking information Ti may include at least one of object ID information, object type information, object location information, object speed information, object creation information, object destruction information, object size information, or object color information. have. The object location information may be estimated location information of an object for a next frame. The object speed information may be information on the expected speed of the object for the next frame. The object creation information may be provided when a new object is recognized. Alternatively, when the object creation information is changed from a skip mode to a low-computation mode to be described later, the object creation information may be provided to include the object in the tracking target. Object destruction information may be provided when the recognized object is no longer recognized. Alternatively, object destruction information may be provided when an object is excluded from tracking by a skip mode, which will be described later.

The driving path analyzer 130 receives the object tracking information Ti. The driving path analysis unit 130 generates driving path information Di based on the object tracking information Ti. The driving path analysis unit 130 generates driving path information Di by determining the driving path of the vehicle for the next frame. The driving path analyzer 130 may receive driving map information and destination information from an external memory or storage to determine an optimized driving path. The driving path analysis unit 130 may determine an optimized path to the destination based on the driving map information and the destination information.

The driving path analysis unit 130 provides driving path information Di to a vehicle control device to be described later. The vehicle control device may receive the driving path information Di and recognize the driving path to be moved in the next frame. The vehicle control device controls the vehicle based on the driving route information Di. The driving path analysis unit 130 generates driving path information Di in units of frames.

The object priority calculator 140 receives the object tracking information Ti from the frame analyzer 120 . The object priority calculator 140 receives the driving path information Di from the driving path analysis unit 130 . The object priority calculator 140 generates priority information Pri based on the object tracking information Ti and the driving route information Di. When a plurality of objects are recognized, the priority is based on the necessity or importance to be considered preferentially during autonomous driving.

The priority information Pri may include object ID information, object location information, and object priority information. For example, the object priority information may be information related to a priority of a possibility of affecting the driving of a vehicle among recognized objects. The priority information Pri may be generated based on the possibility of collision between the vehicle and the object. That is, the object having a high probability of collision with the vehicle may have priority information Pri corresponding to the priority.

The object priority calculator 140 may calculate the possibility of collision between objects and the vehicle using the object tracking information Ti and the driving path information Di. The object priority calculator 140 may detect the position, speed, and movement direction of the objects by using the object tracking information Ti. The object priority calculator 140 may detect the position, speed, and movement direction of the vehicle by using the driving route information Di. The object priority calculator 140 may detect a distance difference, a speed difference, and a driving direction difference between the objects and the vehicle by using the detected information. The object priority calculator 140 calculates a collision probability for each of the objects based on the relationship between the calculated objects and the vehicle. The object priority calculator 140 gives priority to the object ID having a high probability of collision.

The frame complexity calculator 150 receives the object tracking information Ti from the frame analyzer 120 . The frame complexity calculator 150 receives the driving path information Di from the driving path analyzer 130 . The frame complexity calculator 150 calculates the frame complexity based on the object tracking information Ti and the driving route information Di. The frame complexity calculator 150 generates frame complexity information FCi based on the calculated frame complexity. The frame complexity calculator 150 may receive driving map information from an external memory or storage in order to reflect the driving route to the frame complexity.

Frame complexity can be divided into object complexity and path complexity. The object complexity may be calculated based on the number of objects and the distribution of objects. The frame complexity calculator 150 may calculate the number of objects and a distribution of objects based on the object tracking information Ti. For example, the frame complexity calculator 150 may calculate the number of objects based on the number of object ID information included in the object tracking information Ti. The frame complexity calculator 150 may extract object location information included in the object tracking information Ti to calculate a variance according to the distribution of objects. As the number of objects increases, the object complexity increases, and as the distribution of objects increases, the object complexity decreases. The frame complexity calculator 150 may calculate the object complexity by dividing the number of objects by the variance and then multiplying the weight coefficients.

The path complexity may be calculated based on the state of the driving path. The frame complexity calculator 150 may calculate the route complexity based on the driving route information Di or the driving map information. For example, the frame complexity calculator 150 may calculate the number of lanes of a driving route, a variance according to object distribution, the number of intersections, and the number of traffic signals from the driving route information Di or the driving map information. . The frame complexity calculator 150 may multiply a weight coefficient for each of the number of lanes, the variance according to the object distribution, the number of intersections, and the number of traffic signals. The frame complexity calculator 150 may calculate the route complexity by summing the number of lanes to which weights are reflected, variance according to object distribution, the number of intersections, and the number of traffic signals. The frame complexity may be calculated as the sum of object complexity and path complexity.

The resource detector 160 detects a resource occupancy state. The resource occupancy state corresponds to an occupancy state of a processor or memory, which will be described later. The resource detector 160 detects the occupancy of the processor or memory. The resource detector 160 generates resource information Resi based on the occupancy of the processor or memory.

The object recognition apparatus 100 may generate object frame information Oi, object tracking information Ti, driving route information Di, priority information Pri, and frame complexity information FCi under the control of the processor. can The object recognition apparatus 100 may store object frame information Oi, object tracking information Ti, driving route information Di, priority information Pri, and frame complexity information FCi in a memory. As the number of objects recognized increases, the amount of computation of the processor increases, and overload of the processor is expected. As the number of objects recognized increases, the amount of data stored in the memory increases, and overload of the memory is expected. That is, as the number of objects recognized increases, the occupancy of the processor or memory increases.

The operation mode indicator operator 170 receives the frame complexity information FCi from the frame complexity operator 150 . The operation mode indicator operator 170 receives the resource information Resi from the resource detector 160 . The operation mode indicator operator 170 generates the operation mode indicator Ki based on the frame complexity information FCi and the resource information Resi. The operation mode indicator Ki is generated based on frame complexity and resource occupancy status.

The operation mode indicator Ki may be determined by whether the frame complexity is increased, decreased, or maintained. The operation mode indicator Ki may be determined depending on whether the resource occupancy state is a full state or not a full state. For example, as shown in Table 1, the operation mode indicator Ki may be determined based on frame complexity and resource occupancy state.

Frame Complexity resource occupancy status Operation mode indicator (Ki) increase Full -2 decrease or maintain Full -One increase Not Full One decrease or maintain Not Full 2

Referring to Table 1, the operation mode indicator operator 170 may determine the polarity of the operation mode indicator Ki based on the resource occupancy state. The operation mode indicator operator 170 may determine the absolute value of the operation mode indicator Ki based on the frame complexity. Although it has been described that the generation of the operation mode indicator Ki in Table 1 is divided into four indicators, the present invention is not limited thereto. For example, the number of operation mode indicators Ki may be divided into more than four or less than four by different classification criteria for frame complexity or resource occupancy state.

When the frame complexity increases from the previous frame and the resource occupancy state is the Full state, the operation mode indicator Ki indicates -2. When the frame complexity is reduced or the same as that of the previous frame and the resource occupancy state is the Full state, the operation mode indicator Ki indicates -1. When the frame complexity increases from the previous frame and the resource occupancy state is the Not Full state, the operation mode indicator Ki indicates 1. When the frame complexity is reduced or the same as that of the previous frame and the resource occupancy state is the Not Full state, the operation mode indicator Ki indicates 2. That is, the arithmetic mode indicator operator 170 can generate the arithmetic mode indicator Ki having a high value as the resource occupancy state has a margin and the frame complexity is low.

The mode controller 180 receives the operation mode indicator Ki from the operation mode indicator operator 170 . The mode controller 180 receives the priority information Pri from the object priority operator 140 . The mode controller 180 generates the mode control signal Mi based on the operation mode indicator Ki and the priority information Pri. The mode controller 180 may determine the conversion order of the calculation processing of objects based on the priority information Pri. The mode control unit 180 may determine whether to convert the operation processing of objects based on the operation mode indicator Ki.

The mode control unit 180 may select at least one of a high arithmetic mode control signal, a low arithmetic mode control signal, and a skip mode control signal as the mode control signal Mi. The high-computational mode control signal controls the object recognition apparatus 100 to operate in the high-computational mode. The low-computational mode control signal controls the object recognition apparatus 100 to operate in the low-computational mode. The skip mode control signal controls the object recognition apparatus 100 to operate in the skip mode. When generating the object frame information Oi or the object tracking information Ti in the high computation mode, the object recognition apparatus 100 sets the amount of computation to be high in order to secure accuracy. When generating the object frame information Oi or the object tracking information Ti in the low computation mode, the object recognition apparatus 100 sets the amount of computation to be lower than that in the high computation mode in order to prevent overload of the processor or memory. In the skip mode, the object recognition apparatus 100 blocks generation of object frame information Oi or object tracking information Ti.

The mode controller 180 may generate a high arithmetic mode control signal, a low arithmetic mode control signal, or a skip mode control signal in units of objects. The mode controller 180 may determine a region of objects operated by the high computation mode, a region of objects operated by the low computation mode, and a region of objects whose operation is restricted by the skip mode. The mode control signal Mi may include operation mode information and region information. The operation mode information may be information for determining an operation mode of the object recognition apparatus 100 . The region information may be information about an object recognition range operating in a high-computational mode, a low-computational mode, or a skip mode. The object recognition apparatus 100 may determine operation modes of objects belonging to a corresponding area based on area information.

The mode controller 180 provides the mode control signal Mi to the object frame information generator 110 . The object frame information generator 110 determines the amount of computation for object recognition based on the mode control signal Mi. The object frame information generator 110 determines whether to increase, decrease, or not perform an operation for object recognition according to the mode control signal Mi. The mode controller 180 provides the mode control signal Mi to the frame analyzer 120 . The frame analyzer 120 may determine an object to be excluded from tracking in response to the skip mode control signal among the mode control signals Mi. The frame analyzer 120 does not perform an operation on the object corresponding to the skip mode. Accordingly, it is possible to reduce the amount of computation and computation time for generating the object tracking information Ti for the next frame.

The mode controller 180 may receive the operation mode indicator Ki according to the criteria of Table 1. When the operation mode indicator Ki is a negative number, the object having the lowest priority information Pri among the objects calculated in the high operation mode in the previous frame is calculated in the low operation mode. When the operation mode indicator Ki is a negative number, the object having the lowest priority priority information Pri among the objects calculated in the low operation mode in the previous frame is not calculated. When the operation mode indicator Ki is -2, the number of objects converted from the high operation mode to the low operation mode or from the low operation mode to the skip mode may be set to two. When the operation mode indicator Ki is -1, the number of objects converted from the high operation mode to the low operation mode or from the low operation mode to the skip mode may be set to one.

When the operation mode indicator Ki is a positive number, the object having the highest priority information Pri among objects that have not operated in the skip mode in the previous frame and have not been calculated is calculated in the low operation mode. When the operation mode indicator Ki is a positive number, the object having the highest priority information Pri among the objects calculated in the low operation mode in the previous frame is calculated in the high operation mode. When the operation mode indicator Ki is 2, the number of objects converted from the skip mode to the low operation mode or from the low operation mode to the high operation mode may be set to two. When the operation mode indicator Ki is 1, the number of objects converted from the skip mode to the low operation mode or from the low operation mode to the high operation mode may be set to one.

The object recognition apparatus 100 may adaptively adjust the amount of computation based on frame complexity and resource occupancy state. When the resource occupancy state is the Full state, the object recognition apparatus 100 may gradually decrease the amount of computation and may not recognize lower-order objects. Accordingly, the object recognition apparatus 100 may improve processing speed degradation or errors due to overload of the autonomous driving system. When the frame complexity is complex, the object recognition apparatus 100 may decrease the degree of increase in the amount of calculation or increase the degree of decrease in the amount of calculation. Accordingly, the object recognition apparatus 100 may prevent overload due to computational processing in a complex peripheral situation.

2 is a block diagram of an object frame information generator according to an embodiment of the present invention. The object frame information generation unit 110 corresponds to the object frame information generation unit 110 of FIG. 1 . Referring to FIG. 2 , the object frame information generator 110 includes an object operation controller 111 and an object operator 112 . The object operator 112 includes a plurality of multiplexers (MUX), first to nth feature extraction high operators (H1 to Hn), first to nth feature extraction low operators (L1 to Ln), and a full access network ( Fully Connected Network) operator (FCN) is included.

The object operation controller 111 receives the mode control signal Mi from the mode controller 180 of FIG. 1 . The object operation controller 111 generates a plurality of object operator control signals C0 to Cn+1 based on the mode control signal Mi. The object operation controller 111 controls the object operator 112 using the plurality of object operator control signals C0 to Cn+1. The plurality of object operator control signals C0 to Cn+1 may have different values according to the type of the mode control signal Mi. That is, the plurality of object operator control signals (C0~Cn+1) by the high operation mode control signal, the plurality of object operator control signals (C0~Cn+1) by the low operation mode control signal, and the skip mode control signal The plurality of object operator control signals C0 to Cn+1 may have different values.

The object operator 112 receives a plurality of object operator control signals C0 to Cn+1. The object operator 111 receives the sensing information Si. The plurality of object operator control signals C0 to Cn+1 are applied to the plurality of multiplexers MUX1 to MUXn+1. The plurality of multiplexers MUX1 to MUXn+1 determine an output path of input information based on the plurality of object operator control signals C0 to Cn+1. The object operator 112 includes a plurality of multiplexers (MUX1 to MUXn+1) arranged in a multi-layered structure to extract various features of an object, first to nth feature extraction high operators (H1 to Hn), and first to nth feature extraction low operators (L1 to Ln).

The zeroth multiplexer MUX0 determines a transmission path of the sensing information Si based on the zeroth object operator control signal C0. When the zeroth object operator control signal C0 is generated based on the high computation mode control signal, the zeroth multiplexer MUX0 outputs the sensing information Si to the first feature extraction high operator H1. When the zeroth object operator control signal C0 is generated based on the low operation mode control signal, the zeroth multiplexer MUX0 outputs the sensing information Si to the first feature extraction low operator L1. The first to n-1 th multiplexers MUX1 to MUXn-1 convert input information based on the first to n-1 th object operator control signals C1 to Cn-1 to the second to nth features It outputs to the extraction high operator (H2 to Hn) or the second to nth feature extraction low operator (L2 to Ln).

The first to nth feature extraction high operators H1 to Hn and the first to nth feature extraction low operators L1 to Ln are operators for extracting a specific feature from the sensed sensing information Si. Specific characteristics may include the size, color, type, or location of the object. For example, the first to nth feature extraction high operators (H1 to Hn) and the first to nth feature extraction low operators (L1 to Ln) perform a convolution operation and subsampling (Sub) on the input information. -sampling) can be performed. The convolution operation and subsampling of the object operator 112 may correspond to a convolutional neural network method that limits an area from which a feature is extracted. As the information passes through the first to nth feature extraction high operators (H1 to Hn) and the first to nth feature extraction low operators (L1 to Ln), the area from which features are extracted gradually decreases, and Only information can be extracted.

The first to nth feature extraction high operators H1 to Hn have a greater amount of computation than the first to nth feature extraction low operators L1 to Ln. However, the first to nth feature extraction high operators H1 to Hn can more accurately extract features and recognize objects compared to the first to nth low feature extraction operators L1 to Ln. In the high computation mode, the object frame information generating unit 110 may use the first to nth feature extraction high operators H1 to Hn in order to secure the accuracy of object recognition. In the low-computational mode, the object frame information generator 110 may use the first to nth feature extraction low-computers L1 to Ln to prevent errors or processing time delays due to overload of the autonomous driving system.

A full access network operator (FCN) receives information processed by a number of paths. A full access network calculator (FCN) maps information classified into a plurality of areas. A full access network operator (FCN) generates object frame information Oi by collecting information classified into a plurality of areas. A full access network operator (FCN) may generate object frame information Oi that is a one-dimensional matrix. The object operator 112 may recognize an object using various methods in addition to the deep learning technique by the CNN method described above. For example, the object operator 112 may recognize an object using a haar-like feature vector or a histogram of gradients (HoG) feature vector. Alternatively, the object operator 112 may recognize an object using a classifier such as Adaboost or SVM.

3 is a block diagram of an autonomous driving system. Referring to FIG. 3 , the autonomous driving system 200 includes a sensor unit 210 , an object recognition device 220 , a vehicle control device 230 , a processor 240 , a memory 250 , a storage 260 , and a display ( 270 ), and a bus 280 . The autonomous driving system 200 may further include various components for achieving the purpose of autonomous driving. For example, the autonomous driving system 200 may be a navigation device for receiving route information to a destination, a communication module device for communication with an external electronic device, or a user for engaging in vehicle driving. It may further include an interface and the like.

The sensor unit 210 receives a signal from a peripheral unit. The sensor unit 210 may detect signals provided from surrounding objects. Alternatively, the sensor unit 210 may provide a signal to surrounding objects and detect a signal reflected from the objects. The sensor unit 210 may include a plurality of detection sensors to detect objects in all directions. For example, when the autonomous driving system 200 is provided in a vehicle, the sensor unit 210 may be provided at the front, rear, and sides of the vehicle. The sensor unit 210 may include various types of detection sensors to accurately detect objects.

The object recognition apparatus 220 may be the object recognition apparatus 100 of FIG. 1 . The object recognition apparatus 220 recognizes surrounding objects based on the signal detected by the sensor unit 210 . Objects may include vehicles, pedestrians, animals, guard rails, obstacles, and the like. The object recognition apparatus 220 extracts and classifies features of objects. For example, the object recognizing apparatus 220 may receive image information of a periphery, analyze the image information, and extract feature vectors of objects. The object recognition apparatus 220 may distinguish the background from the object based on the feature vector. The object recognition apparatus 220 may recognize objects in units of frames. Accordingly, the object recognition apparatus 220 may recognize not only the static position of the object, but also the dynamic movement of the object.

The vehicle control device 230 controls the speed and the steering angle of the vehicle. In addition, the vehicle control device 230 may control various devices provided in the vehicle, such as a headlamp. The vehicle control device 230 controls the vehicle depending on the objects recognized by the object recognition device 220 . That is, the vehicle equipped with the autonomous driving system 200 detects the presence, location, and movement of surrounding objects in order to prevent collision with other objects and provide an optimized driving route. The vehicle control apparatus 230 controls the movement of the vehicle based on the object information recognized by the object recognition apparatus 120 .

The processor 240 may perform a function of a central control device of the autonomous driving system 200 . The processor 240 may perform a control operation and a calculation operation required to implement the autonomous driving system 200 . For example, the object recognition apparatus 220 may receive a signal from the sensor unit 210 under the control of the processor 240 and recognize objects. The vehicle control apparatus 230 may receive information on objects from the object recognition apparatus 220 under the control of the processor 240 and control the movement of the vehicle. The processor 240 may operate by utilizing the operation space of the memory 250 . The processor 240 may read files for driving an operating system and executable files of applications from the storage 260 . The processor 240 may execute an operating system and various applications.

The memory 250 may store data and process codes to be processed or to be processed by the processor 240 . For example, the memory 250 stores information detected by the sensor unit 210 , information for recognizing an object by the object recognition apparatus 220 , and information on an object recognized by the object recognition apparatus 220 . can The memory 250 may be used as a main memory of the autonomous driving system 200 . The memory 250 may include a dynamic RAM (DRAM), a static RAM (SRAM), a phase-change RAM (PRAM), a magnetic RAM (MRAM), a ferroelectric RAM (FeRAM), and a resistive RAM (RRAM).

The storage 260 may store data generated for long-term storage by the operating system or applications, a file for driving the operating system, or executable files of applications. For example, the storage 260 may store files for execution of the sensor unit 210 , the object recognition device 220 , and the vehicle control device 230 . The storage 260 may be used as an auxiliary storage device of the autonomous driving system 200 . The storage 260 may include a flash memory, a phase-change RAM (PRAM), a magnetic RAM (MRAM), a ferroelectric RAM (FeRAM), a resistive RAM (RRAM), or the like.

The display 270 may receive and display data generated by the sensor unit 210 , the object recognition device 220 , or the vehicle control device 230 . The display 270 may receive and display data stored in the memory 250 or the storage 260 . The display 270 may include a liquid crystal display (LCD), an organic light emitting diode (OLED), an active matrix OLED (AMOLED), a flexible display, electronic ink, and the like.

The bus 280 may provide a communication path between the components of the autonomous driving system 200 . The sensor unit 210 , the object recognition device 220 , the vehicle control device 230 , the processor 240 , the memory 250 , the storage 260 , and the display 270 transmit data to each other through the bus 280 . can be exchanged Bus 280 may be configured to support various types of communication formats used in autonomous driving system 200 .

4 is a block diagram of an autonomous driving system. Referring to FIG. 4 , the autonomous driving system 300 includes a sensor unit 310 , an object recognition device 320 , and a vehicle control device 330 . The autonomous driving system 300 of FIG. 4 will be understood as a block diagram for specifically explaining the information transmission/reception relationship between each component.

The sensor unit 310 generates sensing information Si by sensing the surrounding area. The sensor unit 310 may convert an analog signal generated by sensing the peripheral portion into sensing information Si, which is a digital signal. The sensor unit 310 may include an analog-to-digital converter for converting an analog signal into a digital signal. The sensor unit 310 may generate the sensing information Si in units of frames by sensing the periphery in units of frames.

The object recognition apparatus 320 may be the object recognition apparatus 100 of FIG. 1 . The object recognition apparatus 320 receives sensing information Si from the sensor unit 310 . The object recognition apparatus 320 analyzes the sensing information Si to recognize objects. The object recognition apparatus 320 receives the sensing information Si in units of frames. Accordingly, the object recognition apparatus 320 may recognize the movement or change of the object over time. The object recognition apparatus 320 may determine the number of recognizable objects in units of frames based on the sensing range of the peripheral portion of the sensor unit 310 . For example, the object recognition apparatus 320 may recognize objects existing in a range of a specific radius.

The object recognition apparatus 320 may add or delete objects to be recognized in units of frames. Alternatively, the object recognition apparatus 320 may adjust the amount of computation for recognizing objects. For example, the object recognition apparatus 320 may not recognize an object that is less likely to affect the stability of autonomous driving based on the position or speed of the object recognized in the previous frame, or may reduce the amount of computation. The object recognition apparatus 320 may recognize a new object not recognized in the previous frame. When a new object is likely to affect the stability of autonomous driving, the object recognition apparatus 320 may recognize a new object or increase the amount of computation for recognizing an existing object. Criteria for adding objects, deleting objects, or changing the amount of computation may be set based on object priority, frame complexity, and resource occupancy state.

The object recognition apparatus 320 generates driving route information Di based on the recognized objects. The driving path information Di includes information on the driving path of the vehicle for the next frame. The object recognition apparatus 320 may generate driving route information Di based on the positions and velocities of the objects. The object recognition apparatus 320 may additionally receive destination information or map information stored in the memory 250 or storage 260 of FIG. 3 . The object recognition apparatus 320 analyzes a relationship between objects and a vehicle based on the sensing information Si, and analyzes an optimized route based on destination information or map information, and then generates driving route information Di. can do.

The vehicle control device 330 receives the driving route information Di from the object recognition device 320 . However, the present invention is not limited thereto, and the vehicle control device 330 may receive information on objects from the object recognition device 320 and directly generate the driving route information Di. In this case, the vehicle control device 330 may receive destination information or map information from the memory 250 or the storage 260 of FIG. 3 .

The vehicle control device 330 generates a vehicle control signal based on the driving path information Di. The vehicle control signal may be a signal for controlling a steering angle or speed for moving the vehicle on a driving path. The vehicle control device 330 provides the vehicle control signal to the vehicle driving device for driving the vehicle. The vehicle driving device moves the vehicle to a driving path without user intervention based on the vehicle control signal. The vehicle drive device may include a device for determining the moving speed and the steering angle of the vehicle.

FIG. 5 is a block diagram of the sensor unit of FIG. 4 . Referring to FIG. 5 , the sensor unit 310 includes a camera sensor 311 , a radar sensor 312 , and a lidar sensor 313 . The sensor unit 310 of FIG. 3 will be understood as an embodiment for detecting a plurality of signals. The sensor unit 310 may further include various sensors for detecting the surrounding area. For example, the sensor unit 310 may further include a microphone for detecting a sound.

The camera sensor 311 detects an image of the surrounding area. The camera sensor 311 generates image frame information Ci based on the detected image. The camera sensor 311 may include an analog-to-digital converter for converting the sensed analog signal into image frame information Ci, which is a digital signal. The camera sensor 311 may include a plurality of cameras. The plurality of cameras may be disposed on the front, side, and rear of the vehicle. The plurality of cameras may capture images in different directions. That is, the camera sensor 311 may include a plurality of cameras for detecting images of the entire periphery in order to secure the stability of the autonomous driving system 300 .

The radar sensor 312 radiates an electromagnetic wave to the periphery and detects the reflected electromagnetic wave. The radar sensor 312 may emit microwaves. The radar sensor 312 generates radar frame information Ri based on the reflected microwave. The radar sensor 312 may include an analog-to-digital converter for converting the sensed analog signal into radar frame information Ri, which is a digital signal. The radar sensor 312 may measure the distance and direction of the object based on the time of the reflected electromagnetic wave. The radar sensor 312 may detect an object existing in all directions within a detection range by radiating electromagnetic waves in all directions of the periphery.

The lidar sensor 313 emits electromagnetic waves to the periphery and detects the reflected electromagnetic waves. The lidar sensor 313 may emit a laser having a wavelength shorter than a microwave. The lidar sensor 313 generates lidar frame information Li based on the reflected laser. The lidar sensor 313 may include an analog-to-digital converter for converting the sensed analog signal into the digital signal, which is the lidar frame information Li. The lidar sensor 313 may measure the distance and direction of the object based on the time of the reflected electromagnetic wave. Since the lidar sensor 313 uses an electromagnetic wave having a shorter wavelength than the radar sensor 312 , precision, resolution, and signal-to-noise ratio of distance measurement may be improved. However, the lidar sensor 313 has a smaller detection range than the radar sensor 312 .

The image frame information Ci, the radar frame information Ri, and the lidar frame information Li are provided to the object recognition apparatus 320 . The sensor unit 310 generates image frame information Ci, radar frame information Ri, and lidar frame information Li in units of frames. The object recognition apparatus 320 recognizes an object in units of frames based on the image frame information Ci, the radar frame information Ri, and the lidar frame information Li. For frame unit object recognition, the sensor unit 310 may further include a sensor synchronizer for synchronizing the image frame information Ci, the radar frame information Ri, and the lidar frame information Li.

The sensor unit 310 may include various types of sensors in order to secure the accuracy, stability, and reliability of object recognition in the surrounding area. For example, the shape and type of the object may be recognized using the image frame information Ci by the camera sensor 311 . By using the radar sensor 312 and the lidar sensor 313 , the position and speed of the object may be recognized in a sophisticated and multifaceted manner.

6 to 8 are diagrams for explaining a process of generating priority information of objects according to an embodiment of the present invention. 6 and 7 are plan views illustrating vehicles traveling on a three-lane road. 8 is a plan view illustrating vehicles traveling on a three-lane road and an intersection. 5 to 7 , a vehicle of a user in which the object recognition apparatuses 100 , 220 , 320 or the autonomous driving systems 200 and 300 of the present invention are mounted is defined as V0. 6 and 7 , the vehicle V0 recognizes the first to fifth objects V1 to V5. In FIG. 8 , the vehicle V0 recognizes the first to sixth objects V1 to V6 .

Priority information Pri for the first to sixth objects V1 to V6 is displayed in FIGS. 6 to 8 . 6 and 7 display first to fifth priority information Pr1 to Pr5, and FIG. 8 displays first to sixth priority information Pr1 to Pr6. The first priority information Pr1 will be understood as the highest priority information, and as the fifth priority information Pr5 or the sixth priority information Pr6 progresses, it will be understood as the lowest priority information. It will be understood that the arrows in FIGS. 6 to 8 indicate the moving speed and direction of each of the vehicles or objects. Such priority information may be generated by the object priority calculator 140 of FIG. 1 .

Referring to FIG. 6 , the first to fifth objects V1 to V5 move in front of the vehicle V0. The first object V1 and the fourth object V4 are traveling in the same lane as the vehicle V0. The third object V3 is driving in the left lane of the vehicle V0. The second object V2 and the fifth object V5 are driving in the right lane. The first object V1 corresponds to the first priority information Pr1. The second object V2 corresponds to the second priority information Pr2. The third object V3 corresponds to the third priority information Pr3. The fourth object V4 corresponds to the fourth priority information Pr4. The fifth object V5 corresponds to the fifth priority information Pr5.

The first object V1 has the closest distance to the vehicle V0 among the first to fifth objects V1 to V5 . Accordingly, the first object V1 has the highest probability of collision with the vehicle V0 , and the first priority information Pr1 is assigned to the first object V1 . The second object V2 and the third object V3 have the closest distance after the first object V1. However, the third object V3 has a higher moving speed than the second object V2 . That is, in the next frame, the third object V3 is farther away from the vehicle V0 than the second object V2. As a result, the collision probability between the second object V2 and the vehicle V0 is higher than the collision probability between the third object V3 and the vehicle V0. Accordingly, the second priority information Pr2 is provided to the second object V2, and the third priority information Pr3 is provided to the third object V3. In view of the possibility of collision, the fourth priority information Pr4 is given to the fourth object V4 , and the fifth priority information Pr5 is given to the fifth object V5 .

Referring to FIG. 7 , the vehicle V0 is entering the left lane from the center lane. The entry into the left lane may be a result of controlling the vehicle V0 by the vehicle control devices 230 and 330 based on the driving path information Di generated by the driving path analyzing unit 130 of FIG. 1 . According to the change of the driving path of the vehicle V0, the probability of collision of the first to fifth objects V1 to V5 is changed. Since the vehicle V0 moves in the lane in which the third object V3 travels, the third object V3 has the highest probability of collision with the vehicle V0. Accordingly, the first priority information Pr1 is given to the third object V3. Then, the second priority information Pr2 is given to the first object V1 having a high collision probability. Sequentially, the third priority information Pr3 is given to the second object V2, the fourth priority information Pr4 is given to the fourth object V4, and the fifth priority information Pr4 is given to the fifth object V5. Priority information Pr5 is given.

Referring to FIG. 8 , the vehicle V0 is entering the right lane from the center lane. The entry into the right lane may be a result of controlling the vehicle V0 by the vehicle control devices 230 and 330 based on the driving path information Di generated by the driving path analyzing unit 130 of FIG. 1 . Since the vehicle V0 moves in the lane in which the second object V2 travels, the second object V2 has the highest probability of collision with the vehicle V0. Accordingly, the first priority information Pr1 is given to the second object V2. Then, the second priority information Pr2 is given to the first object V1 having a high collision probability.

The road in FIG. 8 includes an intersection. The intersection connects with the right lane. The sixth object V6 drives through the intersection and enters the right lane. The vehicle V0 recognizes the sixth object V6 newly entered within the object recognition range. The sensor units 110 and 210 detect the sixth object V6, and the object recognition apparatuses 120 and 220 calculate the object frame information Oi and the object tracking information Ti in consideration of the sixth object V6. create The sixth object V6 and the vehicle V0 are expected to travel in the same lane. The sixth object V6 may have a high probability of colliding with the vehicle V0 after the first object V1 and the second object V2. Accordingly, the sixth object V6 is given the third priority information Pr3. In addition, priority information may be assigned to the third to fifth objects V3 to V5 based on a distance difference and a speed difference between the object and the vehicle V0.

9 to 12 are diagrams for explaining changes in the object recognition range and operation mode according to an embodiment of the present invention. The positions of the objects of FIGS. 9 to 12 and the movement of the vehicle V0 are the same as those of FIG. That is, the same priority information as in FIG. 7 is given to the first to fifth objects V1 to V5. First priority information Pr1, which is the highest priority priority information, is given to the third object V3. The fifth object V5 is given fifth priority information Pr5, which is the lowest priority priority information. The first to fifth priority information Pr1 to Pr5 may be generated by the object priority operator 224 of FIG. 4 .

9 to 12 , the object recognition range is divided into a first area AR1 , a second area AR2 , and a third area AR3 . The first area AR1 is defined as an object recognition range operating in a high computation mode. The second area AR2 is defined as an object recognition range operating in a low-computational mode. The third area AR3 is defined as an object recognition range operating in a skip mode. For convenience of description, objects belonging to the first area AR1 are indicated by solid lines. Objects belonging to the second area AR2 are indicated by dotted lines. The third area AR3 is indicated by hatching.

The object recognition range may be provided in a mixture of the first area AR1 , the second area AR2 , or the third area AR3 . Information corresponding to the first region AR1 may be processed by the first to nth feature extraction high operators H1 to Hn of FIG. 2 . Information corresponding to the second region AR2 may be processed by the first to nth feature extraction low operators L1 to Ln of FIG. 2 . Information corresponding to the third region AR3 may be passed without arithmetic processing by the first to nth feature extraction high operators H1 to Hn or the first to nth feature extraction low operators L1 to Ln. have.

9 to 12 , the change of the object recognition range is based on the operation mode indicator Ki according to the criteria of Table 1. The operation mode indicator Ki may be generated by the operation mode indicator operator 170 of FIG. 1 . The object recognition range of FIGS. 9 to 12 depends on the amount of computation of the object frame information generator 110 adjusted under the control of the mode controller 180 of FIG. 1 . That is, the vehicle V0 recognizes objects belonging to the same object recognition range according to the same operation mode.

Referring to FIG. 9 , the first to fourth objects V1 to V4 belong to the first area AR1 , and the fifth object V5 belongs to the second area AR2 . For example, all of the first to fifth objects V1 to V5 in the previous frame may be recognized as the high computation mode. In this case, all of the first to fifth objects V1 to V5 in the previous frame belong to the first area AR1. 9 may be a case in which the operation mode indicator Ki is -1. That is, the resource occupancy state is Full, and frame complexity can be reduced or maintained compared to the previous frame.

The object recognition apparatuses 100 , 220 , and 320 operate in a low-computational mode on an object having the latest priority information among objects for which the object frame information Oi is generated in the high-computational mode in the previous frame. The object to which the lowest priority priority information is assigned is the fifth object V5 to which the fifth priority information Pr5 is assigned. The fifth object V5 has a smaller collision probability than other recognized objects. The mode control unit 228 changes the mode control signal Mi corresponding to the fifth object V5 into a low-computational mode control signal. The object recognition apparatuses 100 , 220 , and 320 generate a second area AR2 for recognizing an object in a low-computational mode.

Referring to FIG. 10 , first to third objects V1 to V3 belong to a first area AR1 , and fourth and fifth objects V4 to V5 belong to a second area AR2 . . The first to third objects V1 to V3 are recognized by a higher computational amount than the fourth and fifth objects V4 to V5. When the previous frame corresponds to FIG. 9 , in FIG. 10 , the operation mode indicator Ki is -1. That is, the resource occupancy state is Full, and frame complexity can be reduced or maintained compared to the previous frame.

The objects for which the object frame information Oi is generated in the high computation mode in the previous frame are the first to fourth objects V1 to V4. Among them, the object to which the lowest priority information is given is the fourth object V4 to which the fourth priority information Pr4 is assigned. The mode controller 228 changes the mode control signal Mi corresponding to the fourth object V4 into a low-computational mode control signal. The object recognition apparatuses 100 , 220 , and 320 extend the second area AR2 to include the fourth object V4 . Since the resource occupancy state is Full, the object recognition apparatuses 100 , 220 , and 320 extend the range of recognizing the object in the low-computational mode.

Referring to FIG. 11 , the third object V3 to which the first priority information Pr1 is assigned belongs to the first area AR1 . The first object V1 , the second object V2 , the fourth object V4 , and the fifth object V5 belong to the second area AR2 . When the previous frame corresponds to FIG. 10 , FIG. 11 may be a case where the operation mode indicator Ki is -2. That is, the resource occupancy state is Full, and frame complexity may increase compared to the previous frame. Since the resource occupancy state is Full, it is difficult to process quickly. Additionally, since frame complexity increases, the amount of computation may be increased compared to the previous frame. Accordingly, there is a need to significantly lower the calculation amount compared to the case where the operation mode indicator Ki is -1.

The objects for which the object frame information Oi is generated in the high computation mode in the previous frame are the first to third objects V1 to V3. Among them, the object to which the lowest priority information is given is the second object V2 to which the third priority information Pr3 is assigned. The object to which the priority information of the next priority is assigned is the first object V1 to which the second priority information Pr2 is assigned. The mode controller 180 changes the mode control signal Mi corresponding to the first object V1 and the second object V2 into a low-computational mode control signal. The object recognition apparatuses 100 , 220 , and 320 extend the second area AR2 to include the first object V1 and the second object V2 . As frame complexity increases, the number of objects changed to the low-computational mode increases.

Referring to FIG. 12 , an object belonging to the first area AR1 does not exist. The first to fourth objects V1 to V4 belong to the second area AR2 . The fifth object V5 belongs to the third area AR3 . When the previous frame corresponds to FIG. 10 , FIG. 12 may be a case where the operation mode indicator Ki is -2. That is, the resource occupancy state is Full, and frame complexity may increase compared to the previous frame. When the operation mode indicator Ki is -2, the operation mode change for two objects may proceed.

The object in which the object frame information Oi is generated in the high computation mode in the previous frame is unique as the third object V3. The mode controller 180 changes the mode control signal Mi corresponding to the third object V3 into a low-computational mode control signal. The object recognition apparatuses 100 , 220 , and 320 extend the second area AR2 to include the third object V3 . When all recognized objects operate in the low-computational mode and the resource occupancy state is Full, the object to which the lower priority information is assigned operates in the skip mode and no operation for object recognition is performed.

Among the objects for which the object frame information Oi is generated in the low-computational mode, the object to which the lowest priority priority information is given is the fifth object V5. The mode controller 180 changes the mode control signal Mi corresponding to the fifth object V5 into a skip mode control signal. That is, the object frame information generating unit 110 does not generate the object frame information Oi for the fifth object V5. The frame analyzer 120 does not generate object tracking information Ti for the fifth object V5. The object recognition apparatuses 100 , 220 , and 320 generate a third area AR3 for recognizing an object in the skip mode.

13 is a flowchart of an object recognition method according to an embodiment of the present invention. The object recognition method of FIG. 13 is performed by the object recognition apparatus 100 of FIG. 1 , the object recognition apparatus 220 of FIG. 3 , or the object recognition apparatus 320 of FIG. 4 . For convenience of description, the object recognition method of FIG. 13 will be described later as being performed by the object recognition apparatus 100 of FIG. 1 .

In step S100 , the object recognition apparatus 100 generates object frame information Oi. Step S100 may be performed by the object frame information generating unit 110 . The object recognition apparatus 100 receives sensing information Si. The object recognition apparatus 100 may recognize a plurality of objects based on the sensing information Si. The object recognition apparatus 100 may generate object frame information Oi corresponding to each of a plurality of objects. The object recognition apparatus 100 may generate object frame information Oi by recognizing a plurality of objects in a high-computational mode or a low-computational mode.

In step S200 , the object recognition apparatus 100 generates object tracking information Ti. Step S200 may be performed by the frame analyzer 120 . The object recognition apparatus 100 calculates the predicted positions or predicted velocities of a plurality of objects for the next frame based on the object frame information Oi. The object recognition apparatus 100 generates object tracking information Ti based on the calculated estimated positions or estimated speeds of the plurality of objects. In step S300 , the object recognition apparatus 100 generates driving route information Di. Step S300 may be performed by the driving path analysis unit 130 . The object recognition apparatus 100 generates driving path information Di for the next frame based on the object tracking information Ti.

In step S400 , the object recognition apparatus 100 generates priority information Pri on the object. Step S400 may be performed by the object priority operator 140 . The object recognition apparatus 100 may calculate the possibility of collision between the object and the vehicle based on the object tracking information Ti and the driving path information Di. When the object recognition apparatus 100 recognizes a plurality of objects, the object priority calculator 140 may generate priority information Pri based on the possibility of collision between each object and the vehicle. An object having a high probability of collision may be given priority information of a higher priority, and an object having a low probability of collision may be given priority information of a lower priority.

In step S500 , the object recognition apparatus 100 calculates frame complexity. Step S500 may be performed by the frame complexity calculator 150 . The object recognition apparatus 100 may generate frame complexity based on the object tracking information Ti and the driving path information Di. For example, the object recognition apparatus 100 may calculate object complexity based on the object tracking information Ti. The object recognition apparatus 100 may calculate the path complexity based on the driving path information Di. The object recognition apparatus 100 may calculate frame complexity by summing object complexity and path complexity. The object recognition apparatus 100 may generate frame complexity information FCi based on the frame complexity.

In step S600 , the object recognition apparatus 100 detects a resource occupancy state. Step S600 may be performed in the resource detector 160 . The object recognition apparatus 100 detects the occupancy of the processor 240 or the memory 250 of FIG. 3 . The object recognition apparatus 100 may generate resource information Resi based on the occupancy of the processor 240 or the memory 250 .

In step S700 , the object recognition apparatus 100 controls the operation of object frame information and changes the object recognition range. Operational control of object frame information may be performed by the mode controller 180 . The change of the object recognition range may be performed by the object frame information generator 110 . The object recognition apparatus 100 controls the amount of calculation for generating the object frame information Oi based on the priority information Pri, the frame complexity information FCi, and the resource information Resi. In order to control the operation of the object frame information Oi in step S700, the object recognition apparatus 100 may generate an operation mode indicator Ki. The operation mode indicator Ki may be determined based on frame complexity and resource occupancy state. The object recognition apparatus 100 may increase or decrease the amount of computation of the object frame information generator 110 based on the operation mode indicator Ki. The object recognition apparatus 100 may increase or decrease the object recognition range of the object frame information generator 110 based on the operation mode indicator Ki. The object recognition apparatus 100 may increase or decrease the object recognition range recognized in the high computation mode or the object recognition range recognized in the low computation mode.

In step S700, the amount of calculation of the object frame information Oi may be determined differently for each object. For example, the object recognition apparatus 100 may increase the amount of computation of the object having the priority information Pri of the priority. The object recognition apparatus 100 may reduce the amount of computation of an object having lower priority information Pri. The object recognition apparatus 100 may operate in a high-computational mode, a low-computational mode, or a skip mode for each object. When the resource occupancy state is the Full state, the object recognition apparatus 100 may reduce the amount of computation. When the resource occupancy state is the Not Full state, the object recognition apparatus 100 may increase the amount of computation.

In operation S800 , the object recognition apparatus 100 may end object recognition or may perform object recognition for a next frame. The object recognition apparatus 100 may determine whether to terminate object recognition based on the control of the processor 240 of FIG. 3 . When object recognition is terminated, the object recognition apparatus 100 does not generate object frame information Oi for the next frame. The object recognition apparatus 100 may end the operation. If the object recognition is not finished, the object recognition apparatus 100 performs step S100 again. That is, the object recognition apparatus 100 generates object frame information Oi for the next frame. In this case, the object recognition apparatus 100 may repeat steps S100 to S700 for each frame until object recognition is finished.

Step S800 may be viewed as a step of determining the end of autonomous driving of the autonomous driving systems 200 and 300 . For example, when autonomous driving is terminated, the autonomous driving systems 200 and 300 do not generate object frame information Oi for the next frame. If autonomous driving is maintained, the autonomous driving systems 200 and 300 perform step S100 again. That is, the autonomous driving systems 200 and 300 generate object frame information Oi for the next frame. In this case, the object recognition apparatus included in the autonomous driving system may repeat steps S100 to S700 for each frame until the autonomous driving is terminated.

14 is a flowchart of a method for controlling operation of object frame information according to an embodiment of the present invention. 14 is a flowchart illustrating step S700 in FIG. 13 . Referring to FIG. 14 , a method of controlling the operation of object frame information may be performed by the mode controller 180 of FIG. 1 . In addition, in order to control the operation of object frame information, the operation mode indicator operator 170 of FIG. 1 may be used.

In step S710 , the object recognition apparatus 100 determines whether the resource occupancy state is the Full state. When the resource occupancy state is the Full state, since the processor or the memory is overloaded, the object recognition apparatus 100 determines that it is difficult to maintain the amount of computation for generating the object frame information Oi in the previous frame. That is, when the resource occupancy state is the Full state, the object recognition apparatus 100 controls the object frame information generator 110 to reduce the amount of computation. When the resource occupancy state is not the Full state, the object recognition apparatus 100 controls the object frame information generator 110 to increase the amount of computation. The resource occupancy state may be determined based on the operation mode indicator Ki. For example, when the operation mode indicator Ki is a negative number, it may be recognized that the resource occupancy state is the Full state. If the resource occupancy state is the Full state, step S720 proceeds. If the resource occupancy state is not the Full state, step S750 proceeds.

In step S720 , the object recognition apparatus 100 determines whether object frame information generated in the high computation mode in the previous frame exists. When the object frame information generated in the high computation mode exists, step S730 for changing from the high computation mode to the low computation mode is performed. When the object frame information generated in the high computation mode does not exist, step S740 for changing from the low computation mode to the skip mode is performed.

In operation S730, the object recognition apparatus 100 controls to generate object frame information having lower priority information among object frame information generated in the high computation mode in the previous frame in the low computation mode. The object recognition apparatus 100 may determine the number of object frame information generated in the low-computational mode based on the frame complexity. For example, the number of object frame information that is changed to the low-computation mode when the frame complexity increases may be greater than the number of object frame information that is changed to the low-computation mode when the frame complexity decreases.

In operation S740 , the object recognition apparatus 100 blocks generation of object frame information having lower priority information among object frame information generated in the low computation mode in the previous frame. The object recognition apparatus 100 may determine the number of object frame information that is not generated based on frame complexity. For example, the number of object frame information that is changed to the skip mode when the frame complexity increases may be greater than the number of object frame information that is changed to the skip mode when the frame complexity decreases.

In operation S750 , the object recognition apparatus 100 determines whether object frame information that has not been generated by operating in the skip mode in the previous frame exists. If there is object frame information whose generation is blocked, step S760 for changing from the skip mode to the low-computational mode is performed. If there is no object frame information in which generation is blocked, step S770 for changing from the low-computational mode to the high-computational mode is performed.

In operation S760 , the object recognition apparatus 100 controls to generate object frame information having priority information of a priority among object frame information whose generation is blocked in a previous frame in a low-computational mode. The object recognition apparatus 100 may determine the number of object frame information generated in the low-computational mode based on the frame complexity. For example, the number of object frame information that is changed to the low-computation mode when the frame complexity is increased may be less than the number of object frame information that is changed to the low-computation mode when the frame complexity is decreased.

In operation S770 , the object recognition apparatus 100 controls to generate object frame information having priority information of a priority among object frame information generated in the low-computation mode in the previous frame in the high-computational mode. The object recognition apparatus 100 may determine the number of object frame information generated in the high computation mode based on frame complexity. For example, the number of object frame information that is changed to the high computation mode when frame complexity increases may be less than the number of object frame information that is changed to the high computation mode when frame complexity decreases.

The contents described above are specific examples for carrying out the present invention. The present invention will include not only the above-described embodiments, but also simple design changes or easily changeable embodiments. In addition, the present invention will also include techniques that can be easily modified and implemented in the future using the above-described embodiments.

100, 220, 320: object recognition device 110: object frame information generating unit
120: frame analysis unit 130: driving path analysis unit
140: object priority operator 150: frame complexity operator
160: resource detector 170: operation mode indicator operator
180: mode control unit 200, 300: autonomous driving system
210, 310: sensor unit 230, 330: vehicle control device

Claims (20)

an object frame information generator configured to recognize at least one object based on a mode control signal and generate object frame information;
a frame analyzer configured to receive the object frame information, calculate an expected position or an expected velocity of the object for a next frame, and generate object tracking information;
an object priority calculator for receiving the object tracking information and generating priority information on the object based on the distance and speed of the object;
a frame complexity calculator that receives the object tracking information and calculates frame complexity based on the number and distribution of the objects; and
and a mode controller configured to generate the mode control signal for adjusting an object recognition range and an amount of operation of the object frame information generator based on the priority information, the frame complexity, and a resource occupancy state. According to claim 1,
The object recognition apparatus further comprising a driving path analyzer to generate driving path information for the next frame based on the object tracking information. 3. The method of claim 2,
The object priority operator,
An object recognition apparatus for calculating a collision possibility between the object and the object recognition apparatus based on the object tracking information and the driving path information, and generating priority information on the object based on the collision possibility. 3. The method of claim 2,
The frame complexity calculator,
An object recognition apparatus for calculating an object complexity based on the object tracking information, calculating a path complexity based on the driving path information, and calculating the frame complexity based on the object complexity and the path complexity. According to claim 1,
and an operation mode indicator operator generating an operation mode indicator based on the frame complexity and the resource occupancy state and providing the operation mode indicator to the mode control unit. 6. The method of claim 5,
The mode control unit,
The object frame information is generated by generating a high arithmetic mode control signal for operating in a high arithmetic mode, a low arithmetic mode control signal for operating in a low arithmetic mode, or a skip mode control signal for operating in a skip mode, based on the arithmetic mode indicator An object recognition device that provides wealth. 7. The method of claim 6,
The operation mode indicator operator,
determining the absolute value of the operation mode indicator based on the frame complexity, and determining the polarity of the operation mode indicator based on the resource occupancy state;
The object frame information generating unit,
The object recognition apparatus for changing the operation mode of the object based on the polarity of the operation mode indicator, and determining the number of objects for which the operation mode is changed based on the absolute value of the operation mode indicator. 7. The method of claim 6,
The object frame information generating unit,
an object operation controller configured to generate an object operation control signal based on the mode control signal; and
and an object operator configured to generate object frame information based on the object operation control signal and sensing information generated by sensing a periphery. 9. The method of claim 8,
The object operator is
a feature extraction high-calculator for receiving input information generated based on the sensing information in the high-computation mode;
a feature extraction low operator receiving the input information in the low operation mode, and having a lower computational amount than the feature extraction high operator; and
An object comprising a multiplexer that outputs the input information to either a feature extraction high operator or a feature extraction low operator based on the object operation control signal, or blocks output to the feature extraction high operator and the feature extraction low operator recognition device. According to claim 1,
The object recognition apparatus further comprising a resource detector for detecting the resource occupancy state. a sensor unit that detects a periphery and generates sensing information;
an object frame information generator configured to receive the sensing information, recognize at least one object, and generate object frame information;
a vehicle control device for controlling a steering angle and speed of a vehicle based on the recognized object;
a processor for controlling the sensor unit, the object frame information generation unit, and the vehicle control device;
a memory for storing the sensing information or the object frame information;
a frame complexity calculator for calculating frame complexity based on the object frame information; and
and a mode controller for adjusting an object recognition range and an amount of calculation of the object frame information generator based on the resource occupation state of the processor or the memory and the frame complexity. 12. The method of claim 11,
and a frame analyzer configured to receive the object frame information, calculate an expected position or an expected speed of the object for a next frame, and generate object tracking information. 13. The method of claim 12,
and a driving path analyzer configured to generate driving path information of the vehicle for the next frame based on the object tracking information and provide the driving path information to the vehicle control device. 14. The method of claim 13,
The autonomous driving system further comprising: an object priority calculator configured to receive the object tracking information and the driving route information, calculate a collision probability between the object and the vehicle, and generate priority information based on the collision probability. 15. The method of claim 14,
The mode control unit,
generating a mode control signal for determining the object recognition range and the amount of computation of the object frame information generating unit based on the priority information, the frame complexity, and the resource occupancy state;
The object frame information generating unit,
An autonomous driving system operating in any one of a high calculation mode, a low calculation mode having a lower calculation amount than the high calculation mode, and a skip mode based on the mode control signal. generating object frame information by recognizing, by an object recognition apparatus, a plurality of objects in a high-computational mode or a low-computational mode having a lower computational amount than the high-computational mode;
generating, by the object recognition apparatus, priority information on the plurality of objects based on the possibility of collision with the recognized objects;
calculating, by the object recognition apparatus, frame complexity based on the object frame information;
detecting, by the object recognition device, a resource occupancy state of a memory or a processor; and
Changing, by the object recognition device, the object recognition range recognized as the high-computation mode or the object recognition range recognized as the low-computation mode based on the priority information, the resource occupancy state, and the frame complexity How to recognize objects. 17. The method of claim 16,
Changing the object recognition range includes:
When the resource occupancy state is the Full state, the object having the lowest priority priority information among the objects for which the object frame information is generated in the high computation mode in the previous frame is recognized as the low computation mode, and based on the frame complexity to adjust the number of object frame information generated in the low-computational mode. 17. The method of claim 16,
Changing the object recognition range includes:
When the resource occupancy state is Full and the object frame information is generated in the low-computation mode for all of the plurality of objects in the previous frame, the generation of object frame information having the lowest priority information is blocked, and the frame An object recognition method that adjusts the number of blocks to generate object frame information based on complexity. 17. The method of claim 16,
Changing the object recognition range includes:
When the resource occupancy state is the Not Full state, object frame information having the highest priority information among the objects whose object frame information is blocked in the skip mode in the previous frame is generated in the low computation mode, and the frame An object recognition method for adjusting the number of object frame information generated in the low-computational mode based on complexity. 17. The method of claim 16,
Changing the object recognition range includes:
When the resource occupancy state is the Not Full state, object frame information having priority information is generated in the high computation mode among the objects for which the object frame information is generated in the low computation mode in the previous frame, and based on the frame complexity, An object recognition method for adjusting the number of object frame information generated in the high computation mode.

Images