KR20210087496A - Object property detection, neural network training and intelligent driving method, device - Google Patents

Abstract

The present embodiment provides an object attribute detection method, a neural network training method and an intelligent driving method, an apparatus, an electronic device, a computer storage medium, and a computer program. The method for detecting object properties includes: performing semantic segmentation on an image to be processed, and determining a mask image of the image to be processed, wherein the mask image indicates a position of an object in the image to be processed; determining, according to the mask image, an attribute characteristic belonging to the object among attribute characteristic maps of the image to be processed, wherein the attribute characteristic map of the image to be processed indicates attributes of the image to be processed; and determining a property of the object according to the property characteristic of the object.

Description

Object property detection, neural network training and intelligent driving method, device

Cross-referencing of related applications

This application is filed based on a Chinese patent application with an application number of 201911081216.4 and an filing date of November 7, 2019, requesting priority to the Chinese patent application, and the entire contents of the Chinese patent application are incorporated herein by reference. .

The present application relates to computer visual processing technology, and relates to, but is not limited to, object property detection, neural network training and intelligent driving method, apparatus, electronic device, computer storage medium, and computer program.

With the continuous development of computer vision technology, the recognition of object properties in images is gradually becoming a research focus. For example, recognition of lane attributes is beneficial for lane division, route planning and collision warning, etc. In the related art, it has become a technical task to be urgently solved to accurately recognize object properties in an image.

An embodiment of the present application is intended to provide a technical method for object attribute detection.

An embodiment of the present application provides a method for detecting object properties, the method comprising:

performing semantic segmentation on the image to be processed, and determining a mask image of the image to be processed, the mask image indicating a position of an object in the image to be processed;

determining, according to the mask image, an attribute characteristic belonging to the object among attribute characteristic maps of the image to be processed, wherein the attribute characteristic map of the image to be processed indicates attributes of the image to be processed;

and determining a property of the object according to the property characteristic of the object.

An embodiment of the present application further provides a training method of a neural network, the method comprising:

determining, according to the tagged mask image of the sample image, an attribute characteristic belonging to an object in the attribute characteristic map of the sample image, wherein the tagged mask image indicates a location of the object in the sample image, and a property of the sample image The feature map indicates the properties of the sample image;

determining a property of the object according to the property characteristic of the object;

According to a difference between the determined property of the object and the tagged property of the object, and a difference between the tagged mask image and the mask image of the sample image determined by performing semantic segmentation on the sample image, the network parameter value of the neural network including adjusting the

An embodiment of the present application further provides an intelligent driving method, the method comprising

detecting a lane property from a road image acquired by an intelligent driving device using any one of the object property detecting methods described above;

instructing the intelligent driving device to drive on a road corresponding to the road image according to the detected lane attribute.

An embodiment of the present application further provides an object attribute detection apparatus, the apparatus comprising a first processing module, a second processing module and a third processing module,

the first processing module is configured to perform semantic segmentation on the image to be processed, and determine a mask image of the image to be processed, the mask image indicating a position of an object in the image to be processed;

a second processing module is configured to determine, according to the mask image, an attribute characteristic belonging to the object among the attribute characteristic map of the image to be processed, the attribute characteristic map of the image to be processed indicates the attribute of the image to be processed;

The third processing module is configured to determine the property of the object according to the property characteristic of the object.

An embodiment of the present application further provides an apparatus for training a neural network, the apparatus comprising a fourth processing module, a fifth processing module and an adjustment module,

a fourth processing module is configured to determine, according to the tagged mask image of the sample image, an attribute characteristic belonging to an object in the attribute characteristic map of the sample image, wherein the tagged mask image indicates a position of the object in the sample image , the attribute feature map of the sample image indicates the attribute of the sample image;

a fifth processing module is configured to determine, according to the attribute characteristic of the object, a property of the object;

The adjustment module is configured to: according to a difference between the determined property of the object and a tagged property of the object, and a difference between the tagged mask image and a mask image of the sample image determined by performing semantic segmentation on the sample image, of the neural network configured to adjust network parameter values.

An embodiment of the present application further provides an intelligent driving device, the device comprising a detection module and an indication module,

the detecting module is configured to detect the lane property among the road images acquired by the intelligent driving device, using any one of the object property detecting methods described above;

The instructing module is configured to instruct the intelligent driving device to drive on a road corresponding to the road image according to the detected lane attribute.

An embodiment of the present application further submits an electronic device, the electronic device comprising a processor and a memory configured to store a computer program executable in the processor,

When the computer program is executed, the processor is configured to perform any one of the above-described object property detection methods, any of the above-described neural network training methods, or any of the above-described intelligent driving methods.

An embodiment of the present application further submits a computer storage medium in which a computer program is stored, and when the computer program is executed by the processor, any one of the above-described object property detection methods or any of the above-described neural network training methods or any of the above-described methods Implement an intelligent driving method.

An embodiment of the present application further submits a computer program including a computer readable code, and when the computer readable code is executed in an electronic device, the processor of the electronic device may use any one of the above-described object attribute detection methods or any of the above Implement a neural network training method or any of the intelligent driving methods described above.

In the object attribute detection method, neural network training method and intelligent driving method, apparatus, electronic device, computer storage medium and computer program submitted by the embodiments of the present application, semantic segmentation is performed on the image to be processed, determine a mask image of an image, the mask image indicating a position of an object in the image to be processed; determine, according to the mask image, an attribute characteristic belonging to the object among attribute characteristic maps of the image to be processed, wherein the attribute characteristic map of the image to be processed indicates the attribute of the image to be processed; According to the property characteristic of the object, the property of the object is determined. As such, the object attribute detection method provided by the embodiment of the present application divides object attribute detection into two steps, first determines the position of the object in the image to be processed, and then determines the position of the object in the image to be processed, and then determines the position of the object in the image to be processed. After determining the attribute characteristics of an object by combining the attribute characteristic maps of the image, the attributes of the object are determined according to the attribute characteristics of the object. Compared to classifying the object according to the determined feature by determining the feature of the region where the object is located according to the pixel of the position of the object in the image to be processed, feature extraction required for classification is avoided, and provided by the embodiment of the present application The attribute characteristics of the object extracted by the object attribute detection method have more discriminability, and accordingly, the type classification of the object can be more accurately determined.

It should be understood that the foregoing general description and the following detailed description are merely illustrative and interpretative and not limiting of the present application.

BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings herein are incorporated in and constitute a part of this specification, and these accompanying drawings show embodiments consistent with the present application, and together with the specification, describe the technical solutions of the present application.
1 is a flowchart of a method for detecting object properties according to an embodiment of the present application.
2 is a flowchart of lane attribute detection according to an embodiment of the present application.
3 is a flowchart of a neural network training method according to an embodiment of the present application.
4 is a flowchart of an intelligent driving method according to an embodiment of the present application.
5 is a schematic configuration diagram of an apparatus for detecting object properties according to an embodiment of the present application.
6 is a schematic configuration diagram of a neural network training apparatus according to an embodiment of the present application.
7 is a schematic configuration diagram of an intelligent driving apparatus according to an embodiment of the present application.
8 is a schematic configuration diagram of an electronic device according to an embodiment of the present application.

Hereinafter, the present application will be described in more detail in conjunction with the accompanying drawings and examples. It should be understood that the examples provided herein merely interpret the present application and do not limit the present application. In addition, the examples provided below are for implementing some examples of the present application, not all examples for implementing the present application, and in case of no conflict, the technical solutions described in the examples of the present application are arbitrarily combined can be carried out in this way.

It should be noted that, in the embodiments of the present application, the term 'comprising' or any other term similar thereto means non-exclusive inclusion, so that a method or apparatus comprising a series of elements not only includes the elements expressly described Rather, it further includes other elements not expressly disclosed, or further includes elements specific to the practice of a method or apparatus. Without further limitation, an element defined by the phrase "comprising a..." refers to another related element in a method or apparatus comprising the element (e.g., a step of a method or a unit of an apparatus, e.g., A unit may be some circuitry, some processor, some program or software, etc.).

For example, the object attribute detection method, neural network training method and intelligent driving method provided by the embodiments of the present application include a series of steps, but the object attribute detection method, neural network training method and intelligent driving method provided by the embodiments of the present application The method is not limited to the steps described. Similarly, the object property detection apparatus, neural network training apparatus, and intelligent driving apparatus provided by the embodiments of the present application include a series of modules, but the apparatus provided by the embodiments of the present application are not limited to the explicitly described modules, and related information or It may include a module for obtaining the necessary settings when processing based on the information.

In this specification, the term 'and/or' only describes a related relationship of a related object, and indicates that three relationships may exist. For example, A and/or B may represent these three cases in which A exists alone, A and B exist simultaneously, and B exists alone. Also, in the present specification, 'at least one' refers to any one of a plurality or any combination of at least two of the plurality. For example, including at least one of A, B, and C may indicate including any one or a plurality of elements selected from the set consisting of A, B, and C.

Embodiments of the present application may be applied to a computer system consisting of a terminal and a server, and may operate with many other general-purpose or dedicated computing system environments or configurations. Here, the terminal may be a thin client, a thick client, a handheld or laptop device, a microprocessor-based system, a set-top box, a programmable home appliance, a network PC, a small computer system, or the like, and the server may be a server computer system small computer system, large computer system, or the like. system and a distributed cloud computing technology environment comprising any of the systems described above, and the like.

Electronic devices such as terminals and servers may be described in a general language environment of instructions (eg, program modules) executable by the computer system to be executed by the computer system. Generally, program modules may include routines, programs, object programs, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. The computer system/server may be implemented in a distributed cloud computing environment, in which tasks are performed by remote processing devices connected through a communication network. In a distributed cloud computing environment, program modules may be located in a local or remote computing system storage medium including a storage device.

In the related art, attribute detection for an object such as a lane may generally apply an object classification method and a semantic segmentation method. The process of the object classification method includes extracting a region in which an object is located from an image, inputting an image of the region in which the object is located to an object classification network, and acquiring properties of the object through object classification. The main problem that exists in the object classification method is that the pictorial area occupied by the object is small, so the identification probability is low, and it is very difficult for the object classification network to learn useful discriminant characteristics, so the attribute accuracy of the identified object is low. The process of the semantic segmentation method includes predicting the properties of each pixel of the object in the image and then determining the properties of the entire object by taking a mode. That is, the most frequently occurring property among the properties of each pixel of the object is set as the property of the entire object. The main problem of the semantic segmentation method is that the object properties are unified in the entire object, but the semantic segmentation method destroys this integral relationship, and thus the accuracy of the recognized object properties is low.

In consideration of the above-described technical problem, a method for detecting object properties is submitted in some embodiments of the present application, and the embodiments of the present application may be applied to scenarios such as image classification, lane attribute recognition, and autonomous driving.

1 is a flowchart of a method for detecting object properties according to an embodiment of the present application, and as shown in FIG. 1 , the flow may include the following steps.

Step 101: Semantic segmentation is performed on the image to be processed, and a mask image of the image to be processed is determined, wherein the mask image indicates a position of an object in the image to be processed.

Here, the image to be processed is an image for which object attribute identification is to be performed. For example, an object in the image to be processed may be a lane or another object.

Illustratively, an image to be processed may be acquired from a local storage area or a network, and the format of the image to be processed is a Joint Photographic Experts GROUP (JPEG), a bitmap (Bitmap, BMP), and a mobility network graphic ( Portable Network Graphics (PNG) or other formats. What will be described is that the format and visitation of the image to be processed are merely described by way of example, and the embodiment of the present invention is not limited to the format and visitation of the image to be processed.

In the embodiments of the present application, the number of objects among the images to be processed is not limited, and one object or a plurality of objects among the images to be processed may be used. For example, when the object is a lane, a plurality of lanes may exist in the image to be processed.

Obviously, when a plurality of objects exist in the image to be processed, the mask image obtained on the basis of step 101 indicates the position of each object in the image to be processed.

In a practical application, an image to be processed may be input to a trained semantic segmentation network, and a mask image of the image to be processed may be extracted from the image to be processed in the semantic segmentation network.

Step 102: According to the mask image, an attribute characteristic belonging to an object is determined among attribute characteristic maps of the image to be processed, and the attribute characteristic map of the image to be processed indicates attributes of the image to be processed.

In the embodiment of the present application, the properties of the image to be processed may indicate characteristics such as color, pattern, and surface roughness of the image, and the properties of the image to be processed may be obtained from the properties of each pixel of the image to be processed. The property of a pixel of an image to be processed may indicate information such as a color of a pixel of the image to be processed. Similarly, the attribute characteristics of an object may indicate characteristics such as color, pattern, and surface roughness of the object. Exemplarily, the attribute characteristic of the object may be expressed as a characteristic map of the set number of channels. The set number of channels may be set according to the effect of object property recognition. For example, the set number of channels is 5, 6 or 7.

Since the mask image can indicate the position of the object in the image to be processed, it is clear that the attribute characteristic belonging to the object among the attribute characteristic map of the image to be processed according to the mask image can be determined.

Step 103: Determine an object property according to an object property characteristic.

Here, the properties of the object may represent information such as color, size, and shape of the object in the image to be processed. For example, when the object is a lane, the properties of the lane may indicate information such as a color, a line width, and a line type of the lane.

When a plurality of objects exist in the image to be processed, it can be seen that by performing step 103, the properties of each object among the images to be processed can be obtained.

In a practical application, the attribute characteristics of the object may be input into a trained object classification network, and the attribute characteristics of the object may be classified using the object classification network to obtain the attributes of the object in the image to be processed.

In actual application, steps 101 to 103 may be implemented using a processor in an electronic device. The processor is a dedicated integrated circuit (Application Specific Integrated Circuit, ASIC), a digital signal processor (Digital Signal Processor, DSP), a digital signal processing device (Digital Signal Processing Device, DSPD), a programmable logic device (Programmable Logic Device, PLD), It may be at least one of a field programmable gate array (FPGA), a central processing unit (CPU), a controller, a microcontroller, and a microprocessor.

In an embodiment of the present application, first, a mask image of an image to be processed is obtained by applying a semantic segmentation method, and then the property characteristics of the object are determined according to the mask image to determine the properties of the object. As can be seen, the object attribute detection method provided by the embodiment of the present application divides object attribute detection into two steps, first determines the position of the object in the image to be processed, and then processes it based on the position of the object in the image to be processed After determining the attribute characteristics of the object by combining the attribute characteristic maps of the image to be used, the attributes of the object are determined according to the attribute characteristics of the object. Compared to classifying the object according to the determined feature by determining the feature of the region where the object is located according to the pixel of the position of the object in the image to be processed, feature extraction required for classification is avoided, and provided by the embodiment of the present application The attribute characteristics of the object extracted by the object attribute detection method have more discriminability, and accordingly, the type classification of the object can be more accurately determined. In addition, the object property detection method provided by the embodiment of the present application classifies the entire object, and performs detection of the object property in the entire object, compared to the method of only performing object property detection through semantic segmentation, so that the object property is can be obtained precisely.

Illustratively, the implementation method for determining the property of the object according to the property characteristic of the object is to convert the property characteristic of the object to the characteristic of the preset length, and then according to the property characteristic of the converted object of the preset length, the properties can be determined. In an embodiment of the present application, the preset length may be set according to an actual application scenario.

Specifically, one or a plurality of objects may exist in the image to be processed, and an attribute characteristic of the one or plurality of objects may be converted into a characteristic of a preset length. According to a characteristic of a preset length of the one or a plurality of objects, an object classification may be performed on the one or a plurality of objects to obtain a property of the one or a plurality of objects.

In addition, when a plurality of objects exist in the image to be processed and the attribute characteristics of the plurality of objects have the same length, object classification is performed directly on the plurality of objects according to the attribute characteristics of the plurality of objects among the images to be processed, Properties of the plurality of objects may be acquired.

As can be understood, some object classification methods are implemented on the basis of acquiring features of the same length, so in the embodiment of the present application, it is advantageous to implement object classification by converting the attribute characteristics of objects of different lengths into features of preset lengths, and further It is advantageous for obtaining properties of an object.

Illustratively, the implementation method for converting the attribute characteristic of an object into a characteristic of a preset length is to divide the point corresponding to the attribute characteristic of the object into k parts (k is an integer greater than or equal to 1), and Calculate the average value of the attribute characteristics of the object corresponding to the points to obtain k average values, repeat the above step n times (in the process of performing two arbitrary times, the value of k is different, and k corresponds to the attribute characteristic of the object) less than the maximum possible number of points, n is an integer greater than 1), and the obtained average value can be used to construct a feature of a preset length.

, P는 사전 설정된 길이를 나타낸다. Specifically, each of the attribute features of each object is divided n times. By performing the i-th division on the pixel points of the attribute feature of each object, Ki parts may be obtained, where i is 1 to n, and Ki represents the value of k at the time of the i-th division. In an embodiment of the present application, the lengths of the obtained Ki parts may be the same or different from each other. Uniform pooling is performed on each of the Ki parts obtained by the i-th division to obtain an average value of the attribute characteristics of the object corresponding to the points of each part. Next, by connecting the acquired feature of length K1 to the feature of length Kn, it is possible to obtain a feature of length P,

, P denotes a preset length.

In the embodiment of the present application, k may be set according to actual circumstances. Exemplarily, if the maximum possible number of pixel points of an attribute feature of an object is 30, then the value of k is less than or equal to 30.

In one specific embodiment, the object attribute detection method is a lane attribute detection method, the image to be processed is a road image, and the object is a lane. In this way, the feature extraction is performed on the road image to determine the area feature map of the road image and the attribute feature map of the road image, and the mask image of the lane among the road images is determined according to the area feature map of the road image, and the road image According to the mask image of the middle lane, the attribute characteristic belonging to the lane in the attribute characteristic map of the road image is determined, and the attribute of the lane is determined according to the attribute characteristic of the lane.

In the embodiment of the present application, since the area feature map of the road image indicates the location of the lane in the road image, a mask image of the lane in the road image may be obtained according to the area feature map of the road image.

2 is a flowchart of lane attribute detection according to an embodiment of the present application. Referring to FIG. 2 , in the embodiment of the present application, a road image may be input to a trained semantic segmentation network, and a lane segmentation result may be obtained by using the semantic segmentation network in the semantic segmentation network. In FIG. 2 , the lane division result may be expressed as an area feature map of a road image. Also, it is possible to obtain an attribute feature map of a road image using a semantic segmentation network. In this way, the mask image of the lane may be obtained according to the area feature map of the road image. According to the mask image of the lane and the attribute feature map of the road image, attribute features belonging to the lane among the attribute feature maps of the road image may be acquired.

In practical applications, the lengths and angles of lanes are generally different. Therefore, the length of the attribute feature of each lane obtained in the embodiment of the present application is generally different from each other, and when the object classification process is to be implemented on the basis of acquiring the feature of the same length, attribute features of the lanes with different lengths are obtained. It can be converted to features of the same length in advance.

In a specific implementation, referring to FIG. 2 , the attribute feature of each lane may be input to a fixed-length feature extraction module, and the fixed-length feature extraction module may perform the following steps. That is, the points corresponding to the attribute features of each lane are divided into k parts (k is an integer greater than or equal to 1), and the average value of the attribute features of the lane corresponding to the points in each part is calculated to obtain k average values. and repeating the above steps n times (in the process of performing two arbitrary times, the value of k is different, k is less than the maximum possible number of points corresponding to the attribute characteristic of the object, and n is an integer greater than 1), The obtained average value can be used to construct a feature of a preset length.

In a specific embodiment, the first car may directly perform pooling on the attribute feature of one lane to obtain one feature value, and then perform sixth-order segmentation on each pixel of the attribute feature of the lane. The pixel of the attribute feature of the lane is divided into two parts, and the pixel values of each part are averaged to obtain two feature values. The pixel of the suboptimal attribute feature is divided into three parts, and the pixel values of each part are averaged to obtain three feature values. The pixel of the attribute feature of the lane is divided into six parts, and the pixel values of each part are averaged to obtain six feature values. The pixel of the attribute feature of the lane is divided into eight parts, and the pixel values of each part are averaged to obtain eight feature values. The pixel of the attribute feature of the lane is divided into 10 parts, and the pixel values of each part are averaged to obtain 10 feature values. The pixel of the suboptimal attribute feature is divided into 12 parts, and the pixel values of each part are averaged to obtain 12 feature values. By combining the obtained 1, 2, 3, 6, 8, 10, 12 feature values, a feature of length 42 is obtained, and a feature of the preset length of the lane is obtained.

In other words, for the attribute features of each lane, pixel attribute features of the same length (all lengths of 42) may be obtained through the fixed-length feature extraction module.

Referring to FIG. 2 , after converting the attribute features of each lane into features of the same length using the fixed-length feature extraction module, the features of the same length are input to the trained object classification network, and input using the object classification network By performing object classification on the specified features, it is possible to obtain the properties of each lane.

In one specific embodiment, the object classification network may include two fully connected layers, wherein the input of the first fully connected layer is a pixel attribute of the same length (eg, length is 42) corresponding to each object. feature, the number of nodes in the first fully connected layer is 256, the number of nodes in the second fully connected layer is 128, and the second fully connected layer may output properties of each object.

It should be noted that FIG. 2 only exemplifies one specific application scenario of object attribute detection. That is, lane attribute recognition is performed. Embodiments of the present application are not limited to performing attribute detection for a lane. For example, attribute detection may be performed on different types of objects.

In addition, after determining the properties of the lanes, the lanes in the road images may be determined according to the road image, the mask image of the lanes among the determined road images, and the property characteristics of the determined lanes.

By determining the lane of the road image, it can be understood that it is advantageous to provide assistance to driving a vehicle and to improve the safety of driving a vehicle.

In some embodiments of the present application, the method for detecting object properties is performed by a neural network, and the neural network is obtained by training by applying a sample image, a tagged mask image of the sample image, and a tagged property of an object of the sample image.

Here, the sample image is an image including a predetermined object. For example, the object may be a lane or other object.

Illustratively, the sample image may be obtained from a local storage area or a network, and the format of the sample image may be JPEG, BMP, PNG or other format. What will be described is that the format and visitation of the sample image are merely described as examples, and the embodiment of the present invention is not limited to the format and visitation of the sample image.

In an embodiment of the present application, the number of objects in the sample image is not limited, and one or a plurality of objects in the sample image may be used. For example, when the object is a lane, a plurality of lanes may exist in the sample image.

In practical application, the tagged mask image of the sample image can be set in advance. Since the tagged mask image of the sample image indicates the position of the object in the sample image, it is clear that the attribute characteristic belonging to the object in the attribute characteristic map of the sample image can be determined according to the tagged mask image of the sample image. In addition, it is advantageous for the trained neural network to determine the attribute characteristics of the object, and further, it is advantageous for the trained neural network to determine the attributes of the object according to the attribute characteristics of the object.

Hereinafter, the training process of the above-described neural network will be exemplarily described in conjunction with the accompanying drawings.

3 is a flowchart of a neural network training method according to an embodiment of the present application, and as shown in FIG. 3 , the flow may include the following steps.

Step 301: Determine, according to the tagged mask image of the sample image, an attribute characteristic belonging to an object in the attribute characteristic map of the sample image, the tagged mask image indicates the position of the object in the sample image, and the attribute characteristic map of the sample image denotes the property of the sample image.

In an embodiment of the present application, the properties of the sample image may indicate characteristics such as color, pattern, and surface roughness of the image, and the properties of the sample image may be obtained from properties of each pixel of the sample image. The property of the pixel of the sample image may indicate information such as the color of the pixel of the sample image. Similarly, the attribute characteristics of an object may indicate characteristics such as color, pattern, and surface roughness of the object. Exemplarily, the attribute characteristic of the object may be expressed as a characteristic map of the set number of channels. The set number of channels may be set according to the effect of object property recognition. For example, the set number of channels is 5, 6 or 7.

Step 302: According to the property characteristics of the object, determine the property of the object.

The implementation method of this step has already been described in the above description, and will not be further described herein.

Step 303: Adjust the network parameter value of the neural network according to the difference between the determined property of the object and the tagged property of the object, and the difference between the tagged mask image and the mask image of the sample image determined by performing semantic segmentation on the sample image .

Illustratively, the implementation method of this step is, according to the difference between the determined property of the object and the tagged property of the object, and the difference between the tagged mask image and the mask image of the sample image determined by performing semantic segmentation on the sample image, It is possible to calculate the loss of the initial neural network, and adjust the network parameter values of the neural network according to the loss of the neural network.

Step 304: The neural network with the adjusted network parameters determines whether the processing on the sample image meets a preset requirement, if not, repeats steps 301 to 304, and if so, performs step 305.

In some embodiments of the present application, the preset requirement may be that the loss of the neural network for which the network parameter is adjusted is smaller than the set loss value. In the embodiment of the present application, the set loss value may be set in advance according to actual needs.

Step 305: The neural network whose network parameters are adjusted is set as a trained neural network.

In actual application, steps 301 to 305 may be implemented using a processor among electronic devices, and the processor may be at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, controller, microcontroller, and microprocessor. .

In one specific implementation, the neural network is used for detecting lane properties, the sample image is a road sample image, and the object is a lane. As such, first, according to the tagged mask image of the road sample image, an attribute characteristic belonging to the lane among the attribute characteristic map of the road sample image is determined, and the tagged mask image of the road sample image is the road sample image Among them, the position of the lane may be indicated. Next, according to the attribute characteristic of the lane, the attribute of the lane may be determined. Finally, the difference between the determined attribute of the lane and the tagged attribute of the lane, and the mask image of the lane determined according to the tagged mask image of the road sample image and the area feature map of the road sample image (a semantic segmentation network The network parameter value of the neural network may be adjusted according to a difference between the two-way mask images).

As can be seen, in the above-described neural network training process, the properties of the object are first determined according to the tagged mask image to determine the properties of the object, and in the training stage, the split network among the neural networks is not yet trained, so the untrained network Since the classification network among the subsequent neural networks cannot converge when the suboptimal mask image is predicted using , the attribute characteristics of the object are determined using the tagged mask image in the training stage.

In the above-described neural network training process, it is divided into two steps when determining object properties. First, the attribute characteristics of the object are determined according to the tagged mask image, and then the attributes of the object are determined according to the attribute characteristics of the object. By determining the characteristics of the region where the object is located according to the pixel of the position of the object in the image to be processed, more and more distinguishable attribute features can be extracted than classifying the object according to the determined characteristic It learns better, and the trained neural network improves the accuracy of detecting objects. In the training process of the neural network, even when determining the properties of the object, classification is performed on the entire object, so performing the object property detection from the whole object is compared to the method of performing object property detection through only semantic segmentation. It can be obtained more accurately, which can improve the accuracy of detecting objects by a trained neural network as well.

On the basis of the object attribute detection method submitted by the above-described embodiment, the embodiment of the present application further presents an intelligent driving method that can be applied to an intelligent driving device. Here, the intelligent driving device includes, but is not limited to, an autonomous driving vehicle, a vehicle equipped with an Advanced Driving Assistant System (ADAS), a robot equipped with an ADAS, and the like.

4 is a flowchart of an intelligent driving method according to an embodiment of the present application, and as shown in FIG. 4 , the flow may include the following steps.

Step 401: When the object property detecting method is the lane property detecting method and the image to be processed is a road image, the lane property among the road images acquired by the intelligent driving device is detected by using any one of the object property detecting methods described above.

Here, the lane attribute includes, but is not limited to, a type of a line, a color of a line, a line width, and the like. The type of line may be a single line, a double line, a solid line, or a dashed line, and the color of the line may be white, yellow or blue, or a combination of two colors, and the like.

Step 402: Instruct the intelligent driving device to drive on a road corresponding to the road image according to the detected lane attribute.

In practical applications, the intelligent driving device may directly control the driving (automatic driving and robot), or it may transmit a command to the driver to control the driving of the vehicle (for example, a vehicle equipped with ADAS) by the driver. .

As can be seen, it is possible to obtain the lane attribute based on the lane attribute detection method, which is advantageous in providing assistance to driving a vehicle and improving the safety of driving the vehicle.

A person skilled in the art will know that in the above-described method of a specific implementation mode, the description order of each step does not imply a strict execution order, and does not constitute any limitation on the implementation process, and the specific execution order of each step does not imply its function and possible inherent It can be understood that it must be confirmed by logic.

On the basis of the object attribute detecting method submitted by the above-described embodiment, the embodiment of the present application provides an object attribute detecting apparatus.

5 is a schematic configuration diagram of an apparatus for detecting object properties according to an embodiment of the present application. 5 , the apparatus includes a first processing module 501 , a second processing module 502 and a third processing module 503 ,

the first processing module 501 is configured to perform semantic segmentation on the image to be processed, and determine a mask image of the image to be processed, the mask image indicating a position of an object in the image to be processed;

The second processing module 502 is configured to determine, according to the mask image, an attribute characteristic belonging to the object among the attribute characteristic maps of the image to be processed, and the attribute characteristic map of the image to be processed is the attribute of the image to be processed. represents;

The third processing module 503 is configured to determine the property of the object according to the property characteristic of the object.

In some embodiments of the present application, the third processing module 503 converts the attribute characteristic of the object into a characteristic of a preset length, and converts the attribute of the object according to the converted attribute characteristic of the object of the preset length configured to decide.

In some embodiments of the present application, the third processing module 503 converts the attribute characteristic of the object into a characteristic of a preset length, dividing points corresponding to the attribute characteristic of the object into k parts, Calculate the average value of the attribute characteristics of the object corresponding to the points of each part to obtain k average values, repeat the above step n times (the value of k is different in the process of performing two arbitrary times, and k is the value of the object less than the maximum possible number of points corresponding to the attribute feature, n is an integer greater than 1), and use the obtained average value to construct a feature of a preset length.

In some embodiments of the present application, the object property detecting device is a lane property detecting device, and the image to be processed is a road image;

The first processing module 501 performs feature extraction on the road image to determine an area feature map of the road image and a property feature map of the road image, and the road image according to the area feature map of the road image and determine a mask image of a middle lane;

the second processing module 502 is configured to determine, according to the mask image of the lane in the road image, an attribute characteristic belonging to the lane in the attribute characteristic map of the road image;

The third processing module 503 is configured to determine the attribute of the lane according to the attribute characteristic of the lane.

In some embodiments of the present application, the third processing module 503 is further configured to, after determining the attribute of the lane, according to the road image, a mask image of a lane among the determined road images, and the determined attribute of the lane, configured to determine a lane among the road images.

In some embodiments of the present application, the object property detection apparatus is implemented based on a neural network, and the neural network is trained by applying a sample image, a tagged mask image of the sample image, and a tagged property of an object of the sample image. is obtained

In practical applications, the first processing module 501, the second processing module 502, and the third processing module 503 may all be implemented by using a processor in an electronic device, wherein the processor is an ASIC, DSP, DSPD, It may be at least one of a PLD, an FPGA, a CPU, a controller, a microcontroller, and a microprocessor.

6 is a schematic configuration diagram of a neural network training apparatus according to an embodiment of the present application. 6 , the apparatus includes a fourth processing module 601 , a fifth processing module 602 and an adjustment module 603 ,

The fourth processing module 601 is configured to determine, according to the tagged mask image of the sample image, an attribute characteristic belonging to an object in the attribute characteristic map of the sample image, wherein the tagged mask image is the object of the sample image. a location, and the property feature map of the sample image represents the property of the sample image;

the fifth processing module 602 is configured to determine, according to the attribute characteristic of the object, an attribute of the object;

The adjustment module 603 is configured to determine a difference between the determined property of the object and a tagged property of the object, and a difference between the tagged mask image and the mask image of the sample image determined by performing semantic segmentation on the sample image, and adjust a network parameter value of the neural network.

In some embodiments of the present application, the fifth processing module 602 converts the attribute characteristic of the object into a characteristic of a preset length, and converts the attribute of the object according to the converted attribute characteristic of the object of the preset length configured to decide.

In some embodiments of the present application, the fifth processing module 602 converts the attribute characteristic of the object into a characteristic of a preset length, dividing points corresponding to the attribute characteristic of the object into k parts, Calculate the average value of the attribute characteristics of the object corresponding to the points of each part to obtain k average values, repeat the above step n times (the value of k is different in the process of performing two arbitrary times, and k is the value of the object less than the maximum possible number of points corresponding to the attribute feature, n is an integer greater than 1), and use the obtained average value to construct a feature of a preset length.

In some embodiments of the present application, the neural network is used to detect a lane property, the sample image is a road sample image, and the object is a lane.

the fourth processing module 601 is configured to determine, according to the tagged mask image of the road sample image, an attribute characteristic belonging to the lane among the attribute characteristic map of the road sample image, the tagged mask of the road sample image an image indicates the location of the lane in the road sample image;

the fifth processing module 602 is configured to determine, according to the attribute characteristic of the lane, an attribute of the lane;

The adjustment module 603 is configured to determine a difference between the determined attribute of the lane and the tagged attribute of the lane, and between the tagged mask image of the road sample image and the mask image of the lane determined according to an area feature map of the road sample image. According to the difference, the value of the network parameter of the neural network may be adjusted.

In practical application, the fourth processing module 601, the fifth processing module 602, and the adjusting module 603 may all be implemented by using a processor in an electronic device, the processor being ASIC, DSP, DSPD, PLD, It may be at least one of an FPGA, a CPU, a controller, a microcontroller, and a microprocessor.

7 is a schematic configuration diagram of an intelligent driving apparatus according to an embodiment of the present application. 7, the device includes a detection module 701 and an indication module 702,

The detection module 701 is configured to, when the object property detection method is a lane property detection method and the image to be processed is a road image, use any one of the object property detection methods described above to determine a lane among the road images acquired by the intelligent driving device. configured to detect an attribute;

The instructing module 702 is configured to instruct the intelligent driving device to drive on a road corresponding to the road image according to the detected lane attribute.

In practical applications, both the detection module 701 and the instruction module 702 may be implemented using a processor in the intelligent driving device, the processor being ASIC, DSP, DSPD, PLD, FPGA, CPU, controller, microcontroller, It may be at least one of microprocessors.

In addition, in the present embodiment, each functional module may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit. The above-described integrated unit may be implemented in the form of hardware or may be implemented in the form of a software function module.

The integrated unit is implemented in the form of a software function module and, when not sold or used as an independent product, may be stored in one computer-readable storage medium, and based on this understanding, the essential or The part contributing to the prior art, or all or part of the technical solution may be implemented in the form of a software product, the computer software product being stored in a storage medium, and a computer device (personal computer, server or network equipment, etc.) ) or a processor (processor) includes several instructions to execute all or some steps of the method according to the present embodiment. The above-described storage medium includes various media capable of storing a program code, such as a USB memory, a mobile hard disk, a read only memory (ROM), a random access memory (RAM), a diskette or an optical disk.

Specifically, a computer program command corresponding to any one of the object property detection method, the neural network training method, or the intelligent driving method of the present embodiment may be stored in a storage medium such as an optical disk, a hard disk, or a USB memory, and any one object property detection method When a computer program command in a storage medium corresponding to the method, the neural network training method, or the intelligent driving method is read or executed by the electronic device, any one of the object attribute detection method of the above-described embodiment or any of the above-described neural network training method or any of the above Implement an intelligent driving method.

Based on the same technical concept as the above-described embodiment, referring to FIG. 8, an electronic device 80 provided by an embodiment of the present application is shown, and may include a memory 81 and a processor 82,

the memory 81 is configured to store computer programs and data;

The processor 82 is configured to execute the computer program stored in the memory to perform any one of the object attribute detection methods of the above-described embodiments, any one of the above-described neural network training methods, or any of the above-mentioned intelligent driving methods.

In practical applications, the memory 81 is a volatile memory such as RAM, ROM, flash memory, a hard disk drive (HDD) or a solid-state disk (SSD) and The same non-volatile memory (non-volatile memory), or a combination of the above types of memory, provides instructions and data to the processor 82 .

The processor 82 may be at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, controller, microcontroller, and microprocessor. As can be understood, for different devices, an electronic device implementing the above-described processor function may be different, and embodiments of the present application are not specifically limited.

An embodiment of the present application further submits a computer program including a computer readable code, and when the computer readable code is executed in an electronic device, the processor of the electronic device may use any one of the above-described object attribute detection methods or any of the above Implement a neural network training method or any of the intelligent driving methods described above.

In some embodiments, a function or a module included in an apparatus provided by an embodiment of the present application may perform the method described in the method embodiment above, for specific implementation, see the description of the method embodiment above. and, for the sake of brevity, no further explanation is given here.

In the above description of each embodiment, the differences between the embodiments are mainly emphasized, and the same or similar points are not described further here because the same or similar points may be referred to each other.

The methods disclosed in each method embodiment provided by the present application may be arbitrarily combined to obtain a new method embodiment if they do not conflict.

The features disclosed in each product embodiment provided by the present application can be arbitrarily combined to obtain a new product embodiment if they do not conflict.

Features disclosed in each method or device embodiment provided by the present application may be arbitrarily combined to obtain a new method embodiment or device embodiment if they do not conflict.

Through the description of the above embodiment, a person skilled in the art can clearly see that the method of the above-described embodiment can be implemented by the method of a general-purpose hardware platform required for software. Of course, it may be through hardware, but in many cases, the former is more This is a preferred implementation method. Based on this understanding, the technical solution of the present invention can be implemented in the form of a software product essentially or for a part contributing to the prior art, and the computer software product is a storage medium (eg, ROM/RAM, It is stored on a diskette, an optical disk), and includes several instructions to cause a terminal (such as a mobile phone, computer, server, air conditioner or network equipment) to execute the method according to each embodiment of the present invention.

In the above, embodiments of the present invention have been described in conjunction with the accompanying drawings, but the present invention is not limited to the specific embodiments described above, and the specific embodiments described above are merely exemplary and not limiting. Those skilled in the art may make many changes in form without departing from the spirit and scope of the present invention under the teaching of the present invention, all of which fall within the protection scope of the present invention.

Embodiments of the present application provide an object attribute detection method, a neural network training method and an intelligent driving method, an apparatus, an electronic device, a computer storage medium, and a computer program. The method for detecting object properties includes: performing semantic segmentation on an image to be processed, and determining a mask image of the image to be processed, wherein the mask image indicates a position of an object in the image to be processed; determining, according to the mask image, an attribute characteristic belonging to the object among attribute characteristic maps of the image to be processed, wherein the attribute characteristic map of the image to be processed indicates attributes of the image to be processed; and determining a property of the object according to the property characteristic of the object. As such, in the embodiment of the present application, there is no need to extract the region where the object is located from the image, and since the attribute characteristic of the object is determined based on the mask image with further discrimination obtained through semantic segmentation, object attribute detection can improve the accuracy of

Claims (21)

A method for detecting object properties, comprising:
performing semantic segmentation on the image to be processed, and determining a mask image of the image to be processed, the mask image indicating a position of an object in the image to be processed;
determining, according to the mask image, an attribute characteristic belonging to the object among attribute characteristic maps of the image to be processed, wherein the attribute characteristic map of the image to be processed indicates attributes of the image to be processed;
and determining an attribute of the object according to the attribute characteristic of the object. According to claim 1,
According to the property characteristics of the object, the step of determining the property of the object,
converting an attribute characteristic of the object into a characteristic of a preset length;
and determining an attribute of the object according to an attribute characteristic of the object of the converted preset length. 3. The method of claim 2,
The step of converting the attribute characteristic of the object into a characteristic of a preset length comprises:
dividing points corresponding to the attribute characteristics of the object into k parts;
obtaining k average values by calculating an average value of the attribute characteristics of the object corresponding to the points of each part;
repeating the above-described step n times, in which the value of k is different in the course of performing two times, where k is less than the maximum possible number of points corresponding to the attribute characteristic of the object, and n is an integer greater than 1;
and constructing the feature of the preset length by using the obtained average value. 4. The method according to any one of claims 1 to 3,
the object property detection method is a lane property detection method, and the image to be processed is a road image;
performing semantic segmentation on the image to be processed, and determining a mask image of the image to be processed,
determining an area feature map of the road image and an attribute feature map of the road image by performing feature extraction on the road image;
determining a mask image of a lane among the road images according to the area feature map of the road image;
The step of determining, according to the mask image, the attribute characteristic belonging to the object among the attribute characteristic map of the image to be processed,
determining, according to a mask image of a lane in the road image, an attribute characteristic belonging to a lane in an attribute characteristic map of the road image;
According to the property characteristics of the object, the step of determining the property of the object,
and determining an attribute of the lane according to the attribute characteristic of the lane. 5. The method of claim 4,
After determining the properties of the lane, the method
and determining a lane in the road image according to the road image, a mask image of the lane among the determined road images, and the determined attribute of the lane. 6. The method according to any one of claims 1 to 5,
The method for detecting object properties is performed by a neural network, and the neural network is obtained by training by applying a sample image, a tagged mask image of the sample image, and a tagged property of an object of the sample image. A method for training a neural network, comprising:
determining, according to the tagged mask image of the sample image, an attribute characteristic belonging to an object in the attribute characteristic map of the sample image, wherein the tagged mask image indicates a location of the object in the sample image, and a property of the sample image The feature map indicates the properties of the sample image;
determining a property of the object according to the property characteristic of the object;
According to a difference between the determined property of the object and the tagged property of the object, and a difference between the tagged mask image and the mask image of the sample image determined by performing semantic segmentation on the sample image, the network parameter value of the neural network A method of training a neural network, comprising adjusting 8. The method of claim 7,
According to the property characteristics of the object, the step of determining the property of the object,
converting an attribute characteristic of the object into a characteristic of a preset length;
and determining a property of the object according to a property characteristic of the object of the converted preset length. 9. The method of claim 8,
The step of converting the attribute characteristic of the object into a characteristic of a preset length comprises:
dividing points corresponding to the attribute characteristics of the object into k parts;
obtaining k average values by calculating an average value of the attribute characteristics of the object corresponding to the points of each part;
repeating the above-described step n times, in which the value of k is different in the course of performing two times, where k is less than the maximum possible number of points corresponding to the attribute characteristic of the object, and n is an integer greater than 1;
A method of training a neural network comprising the step of constructing a feature of the preset length using the obtained average value. 10. The method according to any one of claims 7 to 9,
the neural network is used for detecting lane properties, the sample image is a road sample image, and the object is a lane;
The step of determining, according to the tagged mask image of the sample image, an attribute characteristic belonging to an object in the attribute characteristic map of the sample image,
determining, according to the tagged mask image of the road sample image, an attribute characteristic belonging to the lane in the attribute characteristic map of the road sample image; indicates a location - including,
According to the property characteristics of the object, the step of determining the property of the object,
determining an attribute of the lane according to the attribute characteristic of the lane;
According to the difference between the determined property of the object and the tagged property of the object, and the difference between the tagged mask image and the mask image of the sample image determined by performing semantic segmentation on the sample image, the network parameter of the neural network The steps to adjust the value are:
According to the difference between the determined attribute of the lane and the tagged attribute of the lane, and the difference between the tagged mask image of the road sample image and the mask image of the lane determined according to the area feature map of the road sample image, A method of training a neural network comprising adjusting network parameter values. An intelligent driving method comprising:
A method comprising: detecting a lane property among road images acquired by an intelligent driving device using the method of any one of claims 4 to 6;
and instructing an intelligent driving device to drive on a road corresponding to the road image according to the detected lane attribute. A device for detecting object properties, comprising:
a first processing module, a second processing module and a third processing module;
the first processing module is configured to perform semantic segmentation on the image to be processed, and determine a mask image of the image to be processed, the mask image indicating a position of an object in the image to be processed;
a second processing module is configured to determine, according to the mask image, an attribute characteristic belonging to the object among the attribute characteristic map of the image to be processed, the attribute characteristic map of the image to be processed indicates the attribute of the image to be processed;
and the third processing module is configured to determine, according to the attribute characteristic of the object, a property of the object. 13. The method of claim 12,
and the third processing module is configured to convert the attribute characteristic of the object into a characteristic of a preset length, and determine the attribute of the object according to the converted attribute characteristic of the object of the preset length. 14. The method of claim 13,
In the third processing module converting the attribute characteristic of the object into a characteristic of a preset length,
The points corresponding to the attribute characteristics of the object are divided into k parts, the average value of the attribute characteristics of the object corresponding to the points of each part is calculated to obtain k average values, and the above-described steps are repeated n times, and - In the process of performing two times, the value of k is different, k is less than the maximum possible number of points corresponding to the attribute feature of the object, and n is an integer greater than 1 - , a feature of a preset length using the obtained average value An object property detection device configured to configure 15. The method according to any one of claims 12 to 14,
the object property detecting device is a lane property detecting device, and the image to be processed is a road image;
The first processing module performs feature extraction on the road image to determine a region feature map of the road image and a property feature map of the road image, and determines a lane of the road image according to the region feature map of the road image. and determine a mask image;
the second processing module is configured to determine, according to the mask image of the lane in the road image, an attribute characteristic belonging to the lane in the attribute characteristic map of the road image;
and the third processing module is configured to determine, according to the attribute characteristic of the lane, an attribute of the lane. 16. The method of claim 15,
The third processing module is further configured to: after determining the attribute of the lane, determine the lane in the road image according to the road image, the determined mask image of the lane among the road image, and the determined attribute of the lane detection device. A neural network training device comprising:
a fourth processing module, a fifth processing module and an adjustment module;
a fourth processing module is configured to determine, according to the tagged mask image of the sample image, an attribute characteristic belonging to an object in the attribute characteristic map of the sample image, wherein the tagged mask image indicates a position of the object in the sample image , the attribute feature map of the sample image indicates the attribute of the sample image;
a fifth processing module is configured to determine, according to the attribute characteristic of the object, a property of the object;
The adjustment module is configured to: according to a difference between the determined property of the object and a tagged property of the object, and a difference between the tagged mask image and a mask image of the sample image determined by performing semantic segmentation on the sample image, of the neural network A neural network training device configured to adjust network parameter values. An intelligent driving device comprising:
a detection module and an indication module;
the detection module is configured to detect, by using the method of any one of claims 4 to 6, a lane property among the road images acquired by the intelligent driving device;
the instructing module is configured to instruct the intelligent driving device to drive on a road corresponding to the road image according to the detected lane attribute. As an electronic device,
a processor and a memory configured to store a computer program executable on the processor;
When the processor executes the computer program, the method for detecting object properties according to any one of claims 1 to 6, the method for training a neural network according to any one of claims 7 to 10, or the method according to claim 11 An electronic device configured to perform an intelligent driving method. As a computer storage medium in which a computer program is stored,
When the computer program is executed by a processor, the method for detecting object properties according to any one of claims 1 to 6 or the method for training a neural network according to any one of claims 7 to 10 or the method according to claim 11 . A computer storage medium that implements an intelligent driving method. A computer program including computer readable code, wherein when the computer readable code is executed in an electronic device, a processor of the electronic device is configured to detect an object attribute according to any one of claims 1 to 6 or claim 7 A computer program for implementing the neural network training method according to any one of claims 10 to 11 or the intelligent driving method according to claim 11 .

Images