CN112784084B

CN112784084B - Image processing method and device and electronic equipment

Info

Publication number: CN112784084B
Application number: CN201911084740.7A
Authority: CN
Inventors: 刘宝龙; 陈永健; 孙凯; 李名杨
Original assignee: Alibaba Group Holding Ltd
Current assignee: Alibaba Group Holding Ltd
Priority date: 2019-11-08
Filing date: 2019-11-08
Publication date: 2024-01-26
Anticipated expiration: 2039-11-08
Also published as: CN112784084A

Abstract

The application discloses an image processing method, comprising the following steps: carrying out road component object target detection on an image to be processed to obtain a target road component object in the image to be processed; obtaining traffic usage information to which the target road component object belongs; obtaining content information in the target road component object; and determining the traffic usage and content information combination category of the target road component object according to the traffic usage information of the target road component object and the content information in the target road component object. According to the image processing method, after the target road component object in the image to be processed is obtained, the traffic purpose and content information combination category of the target road component object is determined according to the traffic purpose information of the target road component object and the content information in the target road component object, so that the accuracy of classifying the target road component object in the image is improved.

Description

Image processing method and device and electronic equipment

Technical Field

The present disclosure relates to the field of computer technologies, and in particular, to an image processing method, an image processing device, and an electronic device.

Background

In the process of completing tasks such as high-precision map making, auxiliary positioning and the like, a great deal of classification and warehousing work of road parts in images can be involved. The road components in the images are accurately classified, and the key effect is achieved on accurately completing tasks such as high-precision map making and auxiliary positioning.

The existing method for classifying road components in an image is generally as follows: firstly, detecting a target road component in an image through a target road component detection algorithm model; then, the image features, such as color features, texture features, and the like, of the road components in the target road components are extracted again, and the target road components are classified according to the image features of the road components, thereby obtaining a classification result of the target road components. However, when classifying the target road components in the image based on the image features of the target road components, there is a problem that the classification result is erroneous due to insufficient degree of discrimination of the image features between the target road components of partially different types.

Disclosure of Invention

The application provides an image processing method, an image processing device, an electronic device and a storage medium, so as to improve the accuracy of classifying target road component objects in an image.

The application provides an image processing method, which comprises the following steps:

carrying out road component object target detection on an image to be processed to obtain a target road component object in the image to be processed;

obtaining traffic usage information to which the target road component object belongs;

obtaining content information in the target road component object;

and determining the traffic usage and content information combination category of the target road component object according to the traffic usage information of the target road component object and the content information in the target road component object.

Optionally, the detecting the object of the road component by the image to be processed to obtain the object of the road component in the image to be processed includes:

obtaining a feature map corresponding to the target road component object according to the feature map corresponding to the image to be processed;

and obtaining the target road component object according to the feature map corresponding to the target road component object.

Optionally, the obtaining the target road component object according to the feature map corresponding to the target road component object includes: and obtaining the position information of the target road component object and the target road component object in the image to be processed according to the feature map corresponding to the target road component object.

Optionally, the obtaining the traffic usage information of the target road component object includes:

obtaining traffic purpose characteristic data of a characteristic diagram corresponding to the target road component object;

and obtaining traffic usage information of the target road component object according to the traffic usage feature data of the feature map corresponding to the target road component object.

Optionally, the obtaining traffic usage information of the target road component object according to the traffic usage feature data of the feature map corresponding to the target road component object includes:

obtaining preset traffic usage feature data, wherein the preset traffic usage feature data is feature data corresponding to traffic usage information to which the target road component object belongs;

and obtaining traffic usage information of the target road component object according to the traffic usage feature data of the feature map corresponding to the target road component object and the preset traffic usage feature data.

Optionally, the obtaining traffic usage information of the target road component object according to the traffic usage feature data of the feature map corresponding to the target road component object and the preset traffic usage feature data includes:

Obtaining the preset traffic usage feature data contained in the traffic usage feature data of the feature map corresponding to the target road component object;

and obtaining traffic usage information of the target road component object according to the preset traffic usage feature data contained in the traffic usage feature data of the feature map corresponding to the target road component object.

Optionally, the obtaining the content information in the target road component object includes:

obtaining content feature data of a feature map corresponding to the target road component object;

and obtaining the content information in the target road component object according to the content feature data of the feature map corresponding to the target road component object.

Optionally, the obtaining the content information in the target road component object according to the content feature data of the feature map corresponding to the target road component object includes:

obtaining a feature map corresponding to the content information according to the content feature data of the feature map corresponding to the target road component object;

and obtaining the content information and the position information of the content information in the target road component object according to the feature map corresponding to the content information.

Optionally, the content information in the target road component object at least includes one of numerical information, letter information, text information and symbol information.

Optionally, the obtaining the feature map corresponding to the target road component object according to the feature map corresponding to the image to be processed includes:

obtaining a feature map corresponding to the candidate target road component object according to the feature map corresponding to the image to be processed;

and obtaining the feature map corresponding to the target road component object according to the feature map corresponding to the candidate target road component object.

Optionally, the obtaining the feature map corresponding to the target road component object according to the feature map corresponding to the candidate target road component object includes:

obtaining image feature data of a feature map corresponding to the candidate target road component object;

and obtaining the feature map corresponding to the target road component object according to the image feature data of the feature map corresponding to the candidate target road component object.

Optionally, the obtaining the feature map corresponding to the target road component object according to the image feature data of the feature map corresponding to the candidate target road component object includes:

Obtaining the preset image feature data, wherein the preset image feature data is the image feature data which needs to be included in the image feature data of the feature map corresponding to the target road component object;

and obtaining the feature map corresponding to the target road component object according to the image feature data of the feature map corresponding to the candidate target road component object and the preset image feature data.

Optionally, the obtaining the feature map corresponding to the target road component object according to the image feature data of the feature map corresponding to the candidate target road component object and the preset image feature data includes:

judging whether the image feature data of the feature map corresponding to the candidate target road component object contains the preset image feature data or not;

and if so, taking the feature map corresponding to the candidate target road component object as the feature map corresponding to the target road component object.

Optionally, if the image feature data of the feature map corresponding to the candidate target road component object does not include the preset image feature data, the feature map corresponding to the candidate target road component object is filtered.

Optionally, the determining the combination category of the traffic usage and the content information of the target road component object according to the traffic usage information of the target road component object and the content information in the target road component object includes:

Obtaining traffic usage categories of the target road component object according to traffic usage information of the target road component object;

and determining the traffic application and content information combination category of the target road component object according to the traffic application category of the target road component object and the content information in the target road component object.

Optionally, the obtaining, according to the traffic usage information to which the target road component object belongs, the traffic usage category to which the target road component object belongs includes:

obtaining a corresponding relation of traffic application categories, wherein the corresponding relation of the traffic application categories is a preset corresponding relation between traffic application information and the traffic application categories;

and obtaining the traffic application category of the target road component object according to the traffic application information of the target road component object and the corresponding relation of the traffic application category.

Optionally, the determining the traffic usage and content information combination category of the target road component object according to the traffic usage category of the target road component object and the content information in the target road component object includes:

Obtaining character data in the content information;

and determining the traffic purpose and content information combination category of the target road component object according to the character data in the content information and the traffic purpose category of the target road component object.

Optionally, the determining the traffic usage and content information combination category of the target road component object according to the character data in the content information and the traffic usage category of the target road component object includes: and determining the target road component objects with different character data in the content information as traffic usage and content information combination categories belonging to the different target road component objects, and obtaining the content information categories of the target road component objects under the traffic usage categories belonging to the target road component objects.

Optionally, the method further comprises: and outputting the traffic purpose and content information combination category to which the target road component object belongs.

Optionally, the image to be processed is a road scene image to be processed, and the target road component object is a traffic signboard.

Optionally, the method further comprises: obtaining an initial image; and carrying out de-distortion treatment on the initial image to obtain the image to be treated.

In another aspect of the present application, there is provided an image processing apparatus including:

a target road component object obtaining unit, configured to perform road component object target detection on an image to be processed, and obtain a target road component object in the image to be processed;

a traffic usage information obtaining unit configured to obtain traffic usage information to which the target road component object belongs;

a content information obtaining unit configured to obtain content information in the target road component object;

and the combination category determining unit is used for determining the combination category of the traffic application and the content information of the target road component object according to the traffic application information of the target road component object and the content information in the target road component object.

In another aspect of the present application, there is provided an electronic device, including:

a processor;

a memory for storing a program of an image processing method, the electronic device being powered on and executing the program of the image processing method by the processor, performing the steps of:

Obtaining content information in the target road component object;

In another aspect of the present application, there is provided a storage medium storing a program of an electronic image processing method, the program being executed by a processor to perform the steps of:

obtaining content information in the target road component object;

In another aspect of the present application, there is provided an image processing method including:

sending an image to be processed to a server;

and obtaining the traffic purpose and content information combination category of the target road component object in the image to be processed output by the server.

In another aspect of the present application, there is provided an image processing apparatus comprising:

the image processing unit is used for processing the image to be processed;

and the combination category obtaining unit is used for obtaining the traffic purpose and content information combination category of the target road component object in the image to be processed output by the server.

a processor;

sending an image to be processed to a server;

In another aspect of the present application, there is provided an image processing system including: a server and a client;

the server side is used for obtaining the image to be processed sent by the client side; carrying out road component object target detection on an image to be processed to obtain a target road component object in the image to be processed; obtaining traffic usage information to which the target road component object belongs; obtaining content information in the target road component object; determining a traffic usage and content information combination category of the target road component object according to traffic usage information of the target road component object and content information in the target road component object; outputting the traffic purpose and content information combination category of the target road component object;

the client is used for sending the image to be processed to the server; and obtaining the traffic purpose and content information combination category of the target road component object output by the server.

Compared with the prior art, the application has the following advantages:

After the target road component object in the image to be processed is obtained, the image processing method further determines the traffic purpose and content information combination category of the target road component object according to the traffic purpose information of the target road component object and the content information in the target road component object. Because the content information of the target road parts is the main distinguishing point between different target road parts with higher similarity, the target road parts in the image to be processed are classified by utilizing the traffic use information of the target road part objects and the content information in the target road parts, and the accuracy of classifying the target road parts in the image can be improved.

Drawings

Fig. 1 is a schematic diagram of a first application scenario embodiment provided in the present application.

Fig. 2 is a schematic diagram of a second application scenario embodiment provided in the present application.

Fig. 3 is a flowchart of an image processing method provided in the first embodiment of the present application.

Fig. 4 is a flowchart of a method for obtaining a feature map corresponding to a target road component object according to the first embodiment of the present application.

Fig. 5 is a flowchart of a method for obtaining traffic usage information provided in the first embodiment of the present application.

Fig. 6 is a flowchart of a method for obtaining content information in a road component object according to the first embodiment of the present application.

Fig. 7 is a flowchart of a method for determining a traffic purpose category to which a target road component object belongs, provided in the first embodiment of the present application.

Fig. 8 is a schematic diagram of an image processing apparatus according to a second embodiment of the present application.

Fig. 9 is a schematic diagram of an electronic device provided in an embodiment of the present application.

Fig. 10 is a flowchart of an image processing method provided in a fifth embodiment of the present application.

Fig. 11 is a schematic diagram of an image processing apparatus provided in a sixth embodiment of the present application.

Detailed Description

In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. The present invention may be embodied in many other forms than those herein described, and those skilled in the art will readily appreciate that the present invention may be similarly embodied without departing from the spirit or essential characteristics thereof, and therefore the present invention is not limited to the specific embodiments disclosed below.

In order to more clearly show the application, an application scenario of the image processing method provided in the embodiment of the application is introduced.

Some embodiments provided herein may be applied to internal processing of a server, as shown in fig. 1, which is a schematic diagram of a first application scenario embodiment provided herein.

After obtaining the image to be processed 102, the server 101 first performs road component object target detection on the image to be processed to obtain a first target road component object 102-1 and a second target road component object 102-2 in the image to be processed 102; secondly, obtaining traffic usage information to which the target road component object 102-1 and the second target road component object 102-2 belong; again, obtaining content information 102-11 in the target road component object; finally, the traffic purpose and content information combination category to which the target road component object 102-1 and the second target road component object 102-2 belong is determined from the traffic purpose information to which the target road component object 102-1 and the second target road component object 102-2 belong and the content information in the target road component object. Wherein the image to be processed 102 is a road scene image and the target road component object 102-1 is a traffic sign.

Some embodiments provided in the present application may also be applied to a scenario where a server interacts with a client, as shown in fig. 2, which is a schematic diagram of a second application scenario embodiment provided in the present application.

Firstly, a connection is established between a server 201 and a client 202; secondly, the client 202 sends an image to be processed to the server 201; thirdly, after obtaining the image to be processed, the server 201 performs object detection on the road component object of the image to be processed, and obtains the object road component object in the image to be processed; obtaining traffic usage information to which a target road component object belongs; obtaining content information in a target road component object; determining the traffic usage and content information combination category of the target road component object according to the traffic usage information of the target road component object and the content information in the target road component object; finally, the server 201 sends the traffic usage and content information combination category to which the target road component object belongs to the client 202.

It should be noted that the above two application scenarios are only two embodiments of the application scenario of the image processing method provided in the present application, and the purpose of these two application scenario embodiments is to facilitate understanding the image processing method provided in the present application, and not to limit the image processing method provided in the present application. The image processing method provided by the application can also be applied to other scenes, such as client internal processing, and the like, and will not be described in detail herein.

First embodiment

A first embodiment of the present application provides a road scene image processing method, which is described below with reference to fig. 3 to 6.

Step S301, detecting a road component object target of the image to be processed, and obtaining a target road component object in the image to be processed.

The image to be processed in the first embodiment of the present application is typically a road scene image including a target road component, and may be other images including a target road component. Common road components include: traffic signs, road signs, etc., the target road component in the first embodiment of the present application is generally referred to as a traffic sign.

Before the road component object target detection is carried out on the image to be processed, the image to be processed needs to be obtained, and specifically, an initial image can be obtained first; and then carrying out de-distortion treatment on the initial image so as to obtain an image to be treated.

The road component object target detection is carried out on the image to be processed, and the process of obtaining the target road component object in the image to be processed is as follows: obtaining a feature map corresponding to the target road component object according to the feature map corresponding to the image to be processed; and obtaining the target road component object according to the feature map corresponding to the target road component object. The method for obtaining the target road component object according to the feature map corresponding to the target road component object comprises the following steps: and obtaining the target road component object and the position information of the target road component object in the image to be processed according to the feature map corresponding to the target road component object.

When obtaining a feature map corresponding to an image to be processed, the feature map is generally: firstly, carrying out convolution feature extraction on an image to be processed through a CNN (Convolutional Neural Networks, convolution neural network) to obtain feature graphs with different scales output by a Backbone network Backbone (neural network model) of the CNN; and then, fusing the feature images with different scales output by a Backbone network Backbone (neural network model) of the CNN through a FPN (Feature Pyramid Structure) algorithm, so as to obtain the feature image corresponding to the image to be processed.

In the specific process of obtaining the target road component object according to the feature map corresponding to the target road component object, please refer to fig. 4, which is a flowchart of a method for obtaining the feature map corresponding to the target road component object provided in the first embodiment of the present application.

Step S401, obtaining a feature map corresponding to the candidate target road component object according to the feature map corresponding to the image to be processed.

In the first embodiment of the present application, when obtaining a feature map corresponding to a candidate target road component object according to a feature map corresponding to an image to be processed, a region of the candidate target road component object is generally extracted from the feature map corresponding to the image to be processed obtained in step S301 by an RPN (regional pro-samalnetwork) algorithm, and then the feature map corresponding to the candidate target road component object is obtained by a region feature aggregation method.

Step S402, according to the feature map corresponding to the candidate target road component object, obtaining the feature map corresponding to the target road component object.

Obtaining the feature map corresponding to the target road component object according to the feature map corresponding to the candidate target road component object, including: obtaining image feature data of a feature map corresponding to the candidate target road component object; and obtaining the feature map corresponding to the target road component object according to the image feature data of the feature map corresponding to the candidate target road component object.

Specifically, according to the image feature data of the feature map corresponding to the candidate target road component object, obtaining the feature map corresponding to the target road component object includes: obtaining preset image feature data, wherein the preset image feature data are image feature data which need to be included in the image feature data of a feature map corresponding to a target road component object; and obtaining the feature map corresponding to the target road component object according to the image feature data of the feature map corresponding to the candidate target road component object and the preset image feature data.

When obtaining the feature map corresponding to the target road component object according to the image feature data of the feature map corresponding to the candidate target road component object and the preset image feature data, whether the image feature data of the feature map corresponding to the candidate target road component object contains the preset image feature data or not needs to be judged first, and then the feature map corresponding to the target road component object is further obtained according to the fact that the image feature data of the feature map corresponding to the candidate target road component object contains the preset image feature data. Specifically, if the image feature data of the feature map corresponding to the candidate target road component object contains preset image feature data, taking the feature map corresponding to the candidate target road component object as the feature map corresponding to the target road component object; and if the image feature data of the feature map corresponding to the candidate target road component object does not contain preset image feature data, filtering the feature map corresponding to the candidate target road component object.

Step S402 in the first embodiment of the present application may be implemented based on a Region-CNN (Region-CNN; CNN-based object detection) algorithm, where an input of the Region-CNN (Region-CNN; CNN-based object detection) algorithm is a feature map corresponding to a candidate target road component object, and output is position information of the target road component object and the target road component object in the image to be processed.

Step S302, obtaining traffic usage information to which the target road component object belongs.

The traffic usage information to which the target road component object belongs is information for explaining an application function of the target road component object in traffic, taking the target road component object as a traffic sign as an example, and the traffic usage information to which the traffic sign belongs is information for explaining the application function of the target road component object in traffic, such as: the traffic usage information of the traffic sign plate with the limit height of 2m is used for explaining that the traffic sign plate with the limit height of 2m is applied to the traffic sign plate with the limit height.

Referring to fig. 5, a flowchart of a method for obtaining traffic usage information provided in the first embodiment of the present application is shown.

Step S501, according to the feature map corresponding to the image to be processed, obtaining the feature map corresponding to the target road component object.

In step S501, the detailed process of obtaining the feature map corresponding to the target road component object according to the feature map corresponding to the image to be processed refers to steps S401-S402.

Step S502, obtaining traffic purpose characteristic data of a characteristic diagram corresponding to the target road component object.

After the feature map corresponding to the target road component object is obtained, the feature map corresponding to the target road component object may be further extracted to obtain traffic usage feature data of the feature map corresponding to the target road component object.

Step S503, obtaining the traffic usage information of the target road component object according to the traffic usage feature data of the feature map corresponding to the target road component object.

The specific process for obtaining the traffic usage information of the target road component object according to the traffic usage feature data of the feature map corresponding to the target road component object comprises the following steps: obtaining preset traffic usage feature data, wherein the preset traffic usage feature data is feature data corresponding to traffic usage information to which a target road component object belongs; and obtaining traffic usage information of the target road component object according to the traffic usage feature data of the feature map corresponding to the target road component object and the preset traffic usage feature data. The method for obtaining the traffic usage information of the target road component object according to the traffic usage feature data of the feature map corresponding to the target road component object and the preset traffic usage feature data comprises the following steps: obtaining preset traffic usage feature data contained in traffic usage feature data of a feature map corresponding to a target road component object; and obtaining traffic usage information of the target road component object according to the preset traffic usage feature data contained in the traffic usage feature data of the feature map corresponding to the target road component object.

Step S303, obtaining content information in the target road component object.

Content information in the target road component object is as follows: "50" in the traffic sign of "speed limit 50",

"2" in the traffic sign of "limit for height 2 m". The process of obtaining the content information in the target road component object in the first embodiment of the present application is a process of performing character recognition on the target road component object, such as obtaining the content information in the target road component object by OCR (Optical Character Recognition ) technology. Content information in the target road component object is obtained through a character recognition technology, and the specific process is as follows:

fig. 6 is a flowchart of a method for obtaining content information in a target road component object according to the first embodiment of the present application.

Step S601, obtaining a feature map corresponding to the target road component object according to the feature map corresponding to the image to be processed.

Step S602, obtaining content feature data of a feature map corresponding to the target road component object.

Step S603, obtaining content information in the target road component object according to the content feature data of the feature map corresponding to the target road component object.

Obtaining content information in the target road component object according to the content feature data of the feature map corresponding to the target road component object, wherein the method comprises the following steps: obtaining a feature map corresponding to the content information according to the content feature data of the feature map corresponding to the target road component object; and obtaining the content information and the position information of the content information in the target road component object according to the feature map corresponding to the content information. Wherein the content information in the target road component object at least comprises one of numerical information, letter information, character information and symbol information. Taking the speed limiting traffic sign board as an example, the speed limiting traffic sign boards are of various types, the content information in the speed limiting traffic sign boards of different types is different, and the content information in some speed limiting traffic sign boards may only contain numerical information, such as: some content information in the speed-limiting traffic signs include not only numerical information but also text information, such as 60km/h, 5km for speed-limiting running, and the like.

Step S304, determining the combination category of the traffic usage and the content information of the target road component object according to the traffic usage information of the target road component object and the content information in the target road component object.

The image processing method provided in the first embodiment of the present application needs to obtain the traffic usage category to which the target road component object belongs according to the traffic usage information to which the target road component object belongs. In order to obtain the traffic usage category to which the target road component object belongs in the image processing method provided in the first embodiment of the present application, since the traffic usage information to which the target road component object belongs has already been obtained in the above step S302, generally after the traffic usage information to which the target road component object belongs, the step S303 can further determine the traffic usage category to which the target road component object belongs according to the traffic usage information to which the target road component object belongs.

The target road component object is divided according to the traffic application category of the target road component object, such as a traffic sign board with the speed limit of 50, a traffic sign board with the speed limit of 60 and a traffic sign board with the speed limit of 80 are all divided into a traffic sign board with the speed limit category, a traffic sign board with the speed limit of 2m, a traffic sign board with the speed limit of 2.5m and a traffic sign board with the speed limit of 3m are all divided into a traffic sign board with the speed limit category. Referring to fig. 7, a specific process of obtaining traffic usage categories to which a target road component object belongs according to traffic usage information to which the target road component object belongs is a flowchart of a method for determining traffic usage categories to which the target road component object belongs according to a first embodiment of the present application.

Step S701, obtaining a corresponding relation of traffic use categories, wherein the corresponding relation of the traffic use categories is a preset corresponding relation between traffic use information and the traffic use categories.

Step S702, according to the corresponding relationship between the traffic usage information and the traffic usage category of the target road component object, the traffic usage category of the target road component object is obtained.

After obtaining the traffic usage category to which the target road component object belongs, it is necessary to further determine the traffic usage and content information combination category to which the target road component object belongs according to the traffic usage category to which the target road component object belongs and the content information in the target road component object. Specifically, determining, according to the traffic purpose category to which the target road component object belongs and the content information in the target road component object, the traffic purpose and content information combination category to which the target road component object belongs includes: obtaining character data in the content information; and determining the traffic purpose and content information combination category of the target road component object according to the character data in the content information and the traffic purpose category of the target road component object.

In determining a traffic usage and content information combination category to which a target road component object belongs, it is necessary to determine a target road component object having different character data in content information as belonging to the traffic usage and content information combination category to which a different target road component object belongs, and obtain the content information category of the target road component object under the traffic usage category to which the target road component object belongs. Taking the content information as the numerical value information as an example, the numerical value "60" is composed of the numerical character "6" and the numerical character "0". The process of obtaining the content information in the first embodiment of the present application is to obtain character data in the content information, and determine numerical information or text information corresponding to the content information according to the character data in the content information.

For example, the traffic sign board of the speed limit 50, the traffic sign board of the speed limit 60 and the traffic sign board of the speed limit 80 are all the traffic sign boards of the speed limit type, but the image processing method provided in the first embodiment of the application can divide the traffic sign board of the speed limit 50, the traffic sign board of the speed limit 60 and the traffic sign board of the speed limit 80 into different traffic sign boards of the speed limit type. That is, detailed numerical information or character information in the "speed limit type" traffic signboard is determined based on the character data, the "speed limit 50" traffic signboard is divided into the "speed limit type" traffic signboard, the "speed limit 60" traffic signboard is divided into the "speed limit type" traffic signboard, the "speed limit 80" traffic signboard is divided into the "speed limit type" traffic signboard, and the "speed limit 80" traffic signboard is divided into the "speed limit value 80" traffic signboard.

The image processing method provided in the first embodiment of the present application further includes: and outputting the traffic purpose and content information combination category to which the target road component object belongs. The image processing method provided in the first implementation of the present application can also obtain the position information of the target road component in the image to be processed while determining the target road component in the image to be processed, so that the position information of the target road component in the image to be processed can also be output while outputting the traffic purpose to which the target road component object belongs and the content information combination purpose.

Second embodiment

The second embodiment of the present application also provides an image processing apparatus corresponding to the image processing method provided in the first embodiment of the present application. Since the apparatus embodiment is substantially similar to the first embodiment of the method, the description is relatively simple, and reference is made to the description of the method embodiment for relevant points. The device embodiments described below are merely illustrative.

The image processing apparatus includes:

a target road component object obtaining unit 801, configured to perform road component object target detection on an image to be processed, and obtain a target road component object in the image to be processed;

A traffic usage information obtaining unit 802 for obtaining traffic usage information to which the target road component object belongs;

a content information obtaining unit 803 for obtaining content information in the target road component object;

a combination category determining unit 804, configured to determine, according to the traffic usage information to which the target road component object belongs and the content information in the target road component object, a traffic usage and content information combination category to which the target road component object belongs.

Optionally, the target road component object obtaining unit 801 is specifically configured to obtain a feature map corresponding to the target road component object according to the feature map corresponding to the image to be processed; and obtaining the target road component object according to the feature map corresponding to the target road component object.

Optionally, the traffic usage information obtaining unit is specifically configured to obtain a feature map corresponding to the target road component object according to the feature map corresponding to the image to be processed; obtaining traffic purpose characteristic data of a characteristic diagram corresponding to the target road component object; and obtaining traffic usage information of the target road component object according to the traffic usage feature data of the feature map corresponding to the target road component object.

Optionally, the content information obtaining unit 803 is specifically configured to obtain a feature map corresponding to the target road component object according to a feature map corresponding to the image to be processed; obtaining content feature data of a feature map corresponding to the target road component object; and obtaining the content information in the target road component object according to the content feature data of the feature map corresponding to the target road component object.

Optionally, the combination category determining unit 804 is specifically configured to obtain, according to the traffic usage information to which the target road component object belongs, a traffic usage category to which the target road component object belongs; and determining the traffic application and content information combination category of the target road component object according to the traffic application category of the target road component object and the content information in the target road component object.

obtaining character data in the content information;

Optionally, the image processing apparatus further includes: and the combination category output unit is used for outputting the traffic application and content information combination category of the target road component object.

Optionally, the image processing apparatus further includes:

an initial image obtaining unit configured to obtain an initial image;

the image to be processed obtaining unit is used for carrying out de-distortion processing on the initial image to obtain the image to be processed.

Third embodiment

Corresponding to the image processing method provided in the first embodiment of the present application, a third embodiment of the present application provides an electronic device.

Fig. 9 is a schematic diagram of an electronic device according to an embodiment of the present application, as shown in fig. 9. The electronic device includes:

a processor 901; and

a memory 902 for storing a program of an image processing method, the apparatus, after powering on and running the program of the image processing method by the processor, performs the steps of:

obtaining content information in the target road component object;

It should be noted that, for the detailed description of the electronic device provided in the third embodiment of the present application, reference may be made to the related description of the first embodiment of the present application, which is not repeated here.

Fourth embodiment

In correspondence with the image processing method provided in the first embodiment of the present application, the fourth embodiment of the present application provides a storage medium storing a program of the image processing method, the program being executed by a processor to perform the steps of:

obtaining content information in the target road component object;

It should be noted that, for the detailed description of the storage medium provided in the fourth embodiment of the present application, reference may be made to the related description of the first embodiment of the present application, which is not repeated here.

Fifth embodiment

In the first embodiment described above, there is provided an image processing method, and in correspondence therewith, a fifth embodiment of the present application provides another image processing method. Since this image processing method embodiment is substantially similar to the method first embodiment, the description is relatively simple, and the relevant points are referred to in the description of the method embodiment. The method embodiments described below are merely illustrative.

Referring to fig. 10, fig. 10 is a flowchart of an image processing method according to a fifth embodiment of the present application.

Step S1001, sending an image to be processed to a server. The image to be processed is a road scene image to be processed.

Step S1002, obtaining a traffic usage and content information combination category of the target road component object in the image to be processed output by the server. Wherein the target road component object is a traffic sign.

Sixth embodiment

In the fifth embodiment described above, there is provided an image processing method, and in correspondence therewith, a sixth embodiment of the present application provides an image processing apparatus. Since the apparatus embodiment is substantially similar to the method of the fifth embodiment, the description is relatively simple, and reference is made to the description of the method embodiment for relevant points. The device embodiments described below are merely illustrative.

Referring to fig. 11, fig. 11 is a schematic diagram of an image processing apparatus according to a sixth embodiment of the present application.

The image processing apparatus includes:

a to-be-processed image sending unit 1101, configured to send an to-be-processed image to a server;

and a combination category obtaining unit 1102, configured to obtain a traffic usage and content information combination category to which the target road component object in the image to be processed output by the server belongs.

Seventh embodiment

The seventh embodiment of the present application provides an electronic device corresponding to the image processing method provided by the fifth embodiment of the present application.

As shown in fig. 9, the electronic device includes:

a processor 901; and

sending an image to be processed to a server;

It should be noted that, for the detailed description of the electronic device provided in the seventh embodiment of the present application, reference may be made to the related description of the fifth embodiment of the present application, which is not repeated herein.

Eighth embodiment

In correspondence with the image processing method provided in the fifth embodiment of the present application, the eighth embodiment of the present application provides a storage medium storing a program of the image processing method, the program being executed by a processor to perform the steps of:

sending an image to be processed to a server;

It should be noted that, for the detailed description of the storage medium provided in the eighth embodiment of the present application, reference may be made to the related description of the fifth embodiment of the present application, which is not repeated here.

Ninth embodiment

In the first embodiment described above, there is provided an image processing method, and in correspondence therewith, a ninth embodiment of the present application provides another image processing system. Since the image processing system in the ninth embodiment is substantially similar to the first embodiment of the method, the description is relatively simple, and reference is made to the partial explanation of the method embodiment for the matters. The device embodiments described below are merely illustrative.

Referring again to fig. 2, the image processing system includes: service end 201 and client end 202

The server 201 is configured to obtain an image to be processed sent by the client 202; carrying out road component object target detection on an image to be processed to obtain a target road component object in the image to be processed; obtaining traffic usage information to which the target road component object belongs; obtaining content information in the target road component object; determining a traffic usage and content information combination category of the target road component object according to traffic usage information of the target road component object and content information in the target road component object; outputting the traffic purpose and content information combination category of the target road component object;

the client 202 is configured to send an image to be processed to the server 201; the traffic usage and content information combination category to which the target road component object output by the server 201 belongs is obtained.

Tenth embodiment

The tenth embodiment of the present application also provides another image processing method corresponding to the one provided in the first embodiment of the present application. Since this method embodiment is substantially similar to the first method embodiment, the description is relatively simple, and reference is made to the description of the method embodiment in part. The method embodiments described below are merely illustrative.

The image processing method provided in the tenth embodiment of the present application may be used for processing a road scene image obtained by a vehicle-mounted image acquisition device during a driving process of a vehicle, for example, when an automatic driving automobile needs to obtain the road scene image by the vehicle-mounted image acquisition device during the driving process, and implements automatic navigation in combination with a vehicle-mounted navigation system. In the tenth embodiment of the present application, the image to be processed is a road scene image obtained by the vehicle-mounted image acquisition device during the driving process of the automatic driving automobile.

After the road scene image is obtained, the image processing system in the vehicle can firstly detect the road component object target of the image to be processed, and obtain the target road component object in the image to be processed. And then obtaining the traffic purpose category of the target road component object, and adjusting the driving mode of the automatic driving automobile according to the obtained traffic purpose category of the target road component object, if the obtained traffic purpose category of the target road component object is 'speed limit category', adjusting the driving mode of the automatic driving automobile to be the speed limit mode, and after the driving mode of the automatic driving automobile is adjusted to be the speed limit mode, starting to adjust the driving speed of the automatic driving automobile.

In order to further determine the travel speed to which the autonomous vehicle needs to be adjusted, it is also necessary to obtain content information in the target road component object and determine the travel speed to which the autonomous vehicle needs to be adjusted based on the content information in the target road component object. Specifically, the process of obtaining the content information in the target road component object is: obtaining a content information obtaining model corresponding to the traffic purpose category to which the target road component object belongs according to the traffic purpose category to which the target road component object belongs, wherein the content information obtaining model is a model for identifying content information in the target road component according to the traffic purpose category to which the target road component object belongs; and obtaining the content information in the target road component object according to the content information obtaining model corresponding to the traffic application category to which the target road component object belongs.

Wherein obtaining the content information in the target road component object according to the content information obtaining model corresponding to the traffic purpose category to which the target road component object belongs comprises: and obtaining the content information corresponding to the traffic application category of the target road component object in the target road component object according to the content information obtaining model corresponding to the traffic application category of the target road component object. The content information in the target road component object includes at least one of numerical information, letter information, character information, and symbol information.

Eleventh embodiment

The eleventh embodiment of the present application also provides another image processing method corresponding to the one provided in the first embodiment of the present application. Since this method embodiment is substantially similar to the first method embodiment, the description is relatively simple, and reference is made to the description of the method embodiment in part. The method embodiments described below are merely illustrative.

The image processing method provided in the eleventh embodiment of the present application may be used for processing a road scene image obtained by a vehicle-mounted image acquisition device during a driving process of a vehicle, for example, when an automatic driving automobile needs to obtain the road scene image by the vehicle-mounted image acquisition device during the driving process, and implements automatic navigation in combination with a vehicle-mounted navigation system. Since the road scene image needs to be quickly recognized after the vehicle acquires the road scene image during traveling, the automatically driven automobile can obtain what operation the automatically driven automobile needs to perform in advance and start executing the operation. However, in order to be able to further determine how the automated driving vehicle should perform the operation, it is necessary to further obtain the content information in the target road component object and determine how the automated driving vehicle should perform the operation from the content information in the target road component object. Since the running speed of the vehicle during running is often fast, the image processing method provided in the eleventh embodiment of the present application needs to be able to quickly obtain the content information in the target road component object. Therefore, the image processing method provided in the eleventh embodiment of the present application obtains the content information in the target road component object in the following manner:

Firstly, obtaining traffic purpose categories of target road component objects of an image to be processed through a first-level neural network model, wherein the first-level neural network model is used for detecting the road component objects of the image to be processed, obtaining the target road component objects in the image to be processed, and obtaining the traffic purpose categories of the target road component objects. Only the model of the traffic purpose class to which the target road component object belongs is obtained, so that the automatic driving automobile can be ensured to be capable of obtaining what operation the automatic driving automobile needs to execute more quickly.

Then, a second neural network model for identifying content information in the target road component is obtained in the first neural network model according to the traffic usage category to which the target road component object belongs.

And finally, obtaining the content information in the target road component object through a second-level neural network model.

While the preferred embodiment has been described, it is not intended to limit the invention thereto, and any person skilled in the art may make variations and modifications without departing from the spirit and scope of the invention, so that the scope of the invention shall be defined by the claims.

In one typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, etc., such as Read Only Memory (ROM) or flash memory (flashRAM). Memory is an example of computer-readable media.

Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage media, or any other non-transmission media, that can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include non-transitory computer-readable media (transshipment) such as modulated data signals and carrier waves.

It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

Claims

1. An image processing method, comprising:

obtaining content information in the target road component object;

determining a traffic usage and content information combination category of the target road component object according to traffic usage information of the target road component object and content information in the target road component object;

wherein the obtaining traffic usage information to which the target road component object belongs includes:

2. The image processing method according to claim 1, wherein the performing road component object target detection on the image to be processed to obtain the target road component object in the image to be processed includes:

3. The image processing method according to claim 2, wherein the obtaining the target road component object according to the feature map corresponding to the target road component object includes: and obtaining the position information of the target road component object and the target road component object in the image to be processed according to the feature map corresponding to the target road component object.

4. The image processing method according to claim 1, wherein the obtaining traffic usage information to which the target road component object belongs from the traffic usage feature data of the feature map corresponding to the target road component object includes:

5. The image processing method according to claim 4, wherein obtaining traffic usage information to which the target road component object belongs from traffic usage feature data of a feature map corresponding to the target road component object and the preset traffic usage feature data includes:

6. The image processing method according to claim 1, wherein the obtaining content information in the target road component object includes:

7. The image processing method according to claim 6, wherein the obtaining content information in the target road component object from content feature data of a feature map corresponding to the target road component object includes:

8. The image processing method according to claim 6, wherein the content information in the target road component object includes at least one of numerical information, letter information, character information, and symbol information.

9. The image processing method according to any one of claims 1, 2, and 6, wherein the obtaining the feature map corresponding to the target road component object according to the feature map corresponding to the image to be processed includes:

10. The image processing method according to claim 9, wherein the obtaining the feature map corresponding to the target road component object from the feature map corresponding to the candidate target road component object includes:

11. The image processing method according to claim 10, wherein the obtaining the feature map corresponding to the target road component object from the image feature data of the feature map corresponding to the candidate target road component object includes:

Obtaining preset image feature data, wherein the preset image feature data are image feature data which need to be included in the image feature data of a feature map corresponding to the target road component object;

12. The image processing method according to claim 11, wherein the obtaining the feature map corresponding to the target road component object from the image feature data of the feature map corresponding to the candidate target road component object and the preset image feature data includes:

13. The image processing method according to claim 12, wherein if the image feature data of the feature map corresponding to the candidate target road component object does not include the preset image feature data, the feature map corresponding to the candidate target road component object is filtered.

14. The image processing method according to claim 1, wherein the determining the traffic usage and content information combination category to which the target road component object belongs based on the traffic usage information to which the target road component object belongs and the content information in the target road component object includes:

15. The image processing method according to claim 14, wherein the obtaining the traffic usage category to which the target road component object belongs from the traffic usage information to which the target road component object belongs includes:

16. The image processing method according to claim 14, wherein the determining the traffic usage and content information combination category to which the target road component object belongs based on the traffic usage category to which the target road component object belongs and the content information in the target road component object includes:

obtaining character data in the content information;

17. The image processing method according to claim 16, wherein the determining the traffic usage and content information combination category to which the target road component object belongs based on the character data in the content information and the traffic usage category to which the target road component object belongs includes: and determining the target road component objects with different character data in the content information as traffic usage and content information combination categories belonging to the different target road component objects, and obtaining the content information categories of the target road component objects under the traffic usage categories belonging to the target road component objects.

18. The image processing method according to claim 1, characterized by further comprising: and outputting the traffic purpose and content information combination category to which the target road component object belongs.

19. The image processing method according to claim 1, wherein the image to be processed is an image of a road scene to be processed, and the target road component object is a traffic signboard.

20. The image processing method according to claim 1, characterized by further comprising:

obtaining an initial image;

and carrying out de-distortion treatment on the initial image to obtain the image to be treated.

21. An image processing apparatus, comprising:

a combination category determining unit configured to determine a combination category of traffic usage and content information to which the target road component object belongs, based on traffic usage information to which the target road component object belongs and content information in the target road component object;

22. An electronic device, comprising:

a processor;

obtaining content information in the target road component object;

23. A storage medium storing a program of an electronic image processing method, the program being executed by a processor to perform the steps of:

obtaining content information in the target road component object;

24. An image processing method, comprising:

sending an image to be processed to a server;

obtaining traffic application and content information combination categories of a target road component object in the image to be processed output by the server;

wherein, the method of claims 1-20 is adopted to obtain the combination of traffic usage and content information to which the target road component object in the image to be processed output by the server belongs.

25. The image processing method according to claim 24, wherein the image to be processed is an image of a road scene to be processed, and the target road component object is a traffic signboard.

26. An image processing apparatus, comprising:

the image processing unit is used for processing the image to be processed;

A combination category obtaining unit, configured to obtain a traffic usage and content information combination category to which a target road component object in the image to be processed output by the server side belongs;

27. An electronic device, comprising:

a processor;

sending an image to be processed to a server;

28. A storage medium storing a program of an electronic image processing method, the program being executed by a processor to perform the steps of:

Sending an image to be processed to a server;

29. An image processing system, comprising: a server and a client;

The client is used for sending the image to be processed to the server; obtaining the traffic purpose and content information combination category of the target road component object output by the server;

30. An image processing method, comprising:

obtaining a traffic purpose category to which the target road component object belongs;

obtaining a content information obtaining model corresponding to the traffic purpose category to which the target road component object belongs according to the traffic purpose category to which the target road component object belongs, wherein the content information obtaining model is a model for identifying content information in the target road component according to the traffic purpose category to which the target road component object belongs;

Obtaining content information in the target road component object according to a content information obtaining model corresponding to the traffic purpose category to which the target road component object belongs;

31. The image processing method according to claim 30, wherein the obtaining the content information in the target road component object from the content information obtaining model corresponding to the traffic usage category to which the target road component object belongs includes:

and obtaining the content information corresponding to the traffic application category of the target road component object in the target road component object according to the content information obtaining model corresponding to the traffic application category of the target road component object.

32. The image processing method according to claim 30, wherein the content information in the target road component object includes at least one of numerical information, letter information, character information, and symbol information.

33. The image processing method according to claim 30, characterized by further comprising: and determining the traffic usage and content information combination category of the target road component object according to the traffic usage information of the target road component object and the content information in the target road component object.

34. An image processing method, comprising:

obtaining a traffic purpose category to which a target road component object of an image to be processed belongs through a first-level neural network model, wherein the first-level neural network model is used for detecting a road component object of the image to be processed, obtaining the target road component object in the image to be processed, and obtaining a model of the traffic purpose category to which the target road component object belongs;

obtaining a secondary neural network model for identifying content information in the target road component in the primary neural network model according to the traffic purpose category to which the target road component object belongs;

Obtaining content information in the target road component object through the secondary neural network model;