KR20210088546A - Image semantic segmentation method and apparatus, storage medium - Google Patents

Abstract

An embodiment of the present invention discloses an image semantic segmentation method and apparatus, and a storage medium, wherein the image semantic segmentation method includes: performing feature extraction on an obtained image to be processed, and obtaining a first feature image; synchronously extracting a plurality of context features having different ranges from the first feature image to obtain a plurality of second feature images; determining a target image according to at least the plurality of second feature images, and synchronously extracting a plurality of context features having different ranges again by using the target image as the new first feature image; and in response to the number of times of synchronously extracting a plurality of context features having different ranges for the first feature image reaches the target number, based on the target image acquired last, corresponding to the image to be processed generating a semantic image.

Description

Image semantic segmentation method and apparatus, storage medium

Cross-referencing of related applications

The present invention was filed based on a Chinese patent application with an application number of 201911397645.2 and an application date of December 30, 2019, and claims the priority of the Chinese patent application, all contents of the Chinese patent application are incorporated herein by reference do.

The present invention relates to the field of deep learning, and more particularly, to an image semantic segmentation method and apparatus, and a storage medium.

As for the movable mechanical device, it is possible to perform semantic segmentation on the image collected by the mounted camera and obtain a semantic understanding of the scenario, thereby implementing functions such as obstacle avoidance and navigation.

At present, on the one hand, in consideration of the cost and maneuverability, the computing resources of movable mechanical devices are often relatively limited. On the other hand, mobile mechanical devices must interact with the real environment in real time. Therefore, with limited computing resources, how to perform real-time semantic segmentation is a challenging technical task.

An embodiment of the present invention provides an image semantic segmentation method and apparatus, and a storage medium.

According to a first aspect of an embodiment of the present invention, there is provided an image semantic segmentation method, the image semantic segmentation method comprising: performing feature extraction on an obtained image to be processed, and obtaining a first feature image; synchronously extracting a plurality of context features having different ranges from the first feature image to obtain a plurality of second feature images; determining a target image according to at least the plurality of second feature images, and synchronously extracting a plurality of context features having different ranges again by using the target image as the new first feature image; and in response to the number of times of synchronously extracting a plurality of context features having different ranges for the first feature image reaches the target number, based on the target image acquired last, corresponding to the image to be processed generating a semantic image.

In some selectable embodiments, the step of synchronously extracting a plurality of context features having different ranges from the first feature image to obtain a plurality of second feature images comprises: obtaining a plurality of third feature images by synchronously performing a dimension reduction process by dividing into channels; and extracting context features having different ranges from at least two third feature images among the plurality of third feature images to obtain a plurality of second feature images.

In some selectable embodiments, the step of obtaining a plurality of second feature images by extracting context features having different ranges for at least two third feature images among the plurality of third feature images includes: Through extension convolution corresponding to possible convolutions and extension coefficients having different convolutional kernels, context features having different ranges are extracted for at least two third feature images among the plurality of third feature images. 2 comprising acquiring a feature image.

In some selectable embodiments, determining the target image according to at least the plurality of second feature images comprises: fusing at least the plurality of second feature images to obtain a fourth feature image; and determining the target image according to at least the fourth feature image.

In some selectable embodiments, obtaining a fourth feature image by fusing at least the plurality of second feature images includes: overlaying the plurality of second feature images to obtain the fourth feature image; or performing an overlay on at least one third feature image among the plurality of second feature images and the plurality of third feature images to obtain the fourth feature image.

In some selectable embodiments, the determining of the target image according to at least the fourth feature image comprises: performing up-sampling on the fourth feature image to obtain the target image; or performing sub-pixel convolution on the fourth feature image to obtain the target image.

In some selectable embodiments, the image semantic segmentation method includes: after performing feature extraction and dimension reduction processing on the image to be processed, obtaining a fifth feature image - a feature corresponding to the fifth feature image the number of layers of extraction is smaller than the number of layers of feature extraction corresponding to the first feature image; In the determining of the target image according to at least the fourth characteristic image, if the number of times is smaller than the target number, over-sampling the fourth characteristic image and the fifth characteristic image and then up-sampling the target image obtaining a; or, when the number is smaller than the target number, overlaying an image obtained after performing sub-pixel convolution on the fourth feature image and the fifth feature image to obtain the target image.

In some selectable embodiments, a dimension corresponding to the last acquired target image is a target dimension, and the target dimension is determined according to a total quantity of object categories included in the preset semantic image.

In some selectable embodiments, after generating a semantic image corresponding to the image to be processed, the image semantic segmentation method further comprises: performing machine device navigation according to the semantic image.

According to a second aspect of an embodiment of the present invention, there is provided an image semantic segmentation apparatus, comprising: a feature extraction module, configured to perform feature extraction on an obtained image to be processed, and obtain a first feature image; a context feature extraction module configured to synchronously extract a plurality of context features having different ranges from the first feature image to obtain a plurality of second feature images; a determining module, configured to determine a target image according to at least the plurality of second feature images, and to use the target image as the new first feature image to again synchronously extract a plurality of ranges different context features; and in response to the number of times of synchronously extracting a plurality of context features having different ranges for the first feature image reaches the target number, based on the target image acquired last, corresponding to the image to be processed and a semantic image generation module configured to generate a semantic image to be

In some selectable embodiments, the context feature extraction module is configured to: divide the first feature image into a plurality of channels and synchronously perform dimension reduction processing to obtain a plurality of third feature images processing sub-module; and a second processing sub-module, configured to extract context features having different ranges for at least two third feature images among the plurality of third feature images to obtain a plurality of second feature images.

In some selectable embodiments, the second processing submodule is configured to perform at least two of the plurality of third feature images through convolution separable by depth and extension convolution corresponding to extension coefficients having different convolution kernels. and extracting context features having different ranges from the third feature image to obtain a plurality of second feature images.

In some selectable embodiments, the determining module comprises: a first determining sub-module, configured to fuse at least the plurality of second feature images to obtain a fourth feature image; and a second determining sub-module, configured to determine the target image according to at least the fourth feature image.

In some selectable embodiments, the first determining sub-module comprises: overlaying the plurality of second feature images to obtain the fourth feature image; or performing overlay on at least one third feature image among the plurality of second feature images and the plurality of third feature images to obtain the fourth feature image.

In some selectable embodiments, the second determining submodule may include: performing up-sampling on the fourth feature image to obtain the target image; or perform sub-pixel convolution on the fourth feature image to obtain the target image.

In some selectable embodiments, the image semantic segmentation device is a processing module, configured to obtain a fifth feature image after performing feature extraction and dimension reduction processing on the image to be processed - corresponding to the fifth feature image The number of layers of feature extraction to be used is smaller than the number of layers of feature extraction corresponding to the first feature image; and when the number is smaller than the target number, overlaying the fourth characteristic image and the fifth characteristic image and then up-sampling to obtain the target image; or a second determining sub, configured to obtain the target image by overlaying an image obtained after performing sub-pixel convolution on the fourth feature image and the fifth feature image when the number is smaller than the target number It further includes a module.

In some selectable embodiments, a dimension corresponding to the last acquired target image is a target dimension, and the target dimension is determined according to a total quantity of object categories included in the preset semantic image.

In some selectable embodiments, the image semantic segmentation device further comprises a navigation module, configured to perform machine device navigation according to the semantic image.

According to a third aspect of an embodiment of the present invention, there is provided a computer-readable storage medium, wherein a computer program is stored in the storage medium, wherein the computer program executes the image semantic segmentation method according to any one of the embodiments of the first aspect. it is to do

According to a fourth aspect of an embodiment of the present invention, there is provided an image semantic segmentation apparatus, the image semantic segmentation apparatus comprising: a processor; and a memory for storing the instructions executable by the processor, wherein the processor is configured to call the executable instructions stored in the memory to implement the image semantic segmentation method according to any one of the first aspects. do.

According to a fifth aspect of an embodiment of the present invention, there is provided a computer program, the computer program causing the computer to execute the image semantic segmentation method according to any one of the first aspects of the embodiment of the present invention.

The technical solutions provided in the embodiments of the present invention may include the following beneficial effects.

In an embodiment of the present invention, feature extraction is performed on the obtained image to be processed to obtain a first feature image, and further, by synchronously extracting a plurality of contextual features having different ranges from the first feature image, , a plurality of second feature images may be acquired. A target image is determined according to at least a plurality of second feature images, and context features having different ranges are synchronously extracted again by using the target image as a new first feature image. When the number of times of synchronously extracting a plurality of context features with different ranges from the first feature image reaches the target number, semantics corresponding to the image to be processed by semantic division based on the last acquired target image You can create an image. An embodiment of the present invention provides sufficient fusion of context information of different scales by synchronously extracting a plurality of context features having different ranges from a feature image corresponding to the image to be processed several times, so that the semantic segmentation improve accuracy.

In an embodiment of the present invention, first, the first feature image is divided into a plurality of channels to perform dimensional reduction processing to obtain a plurality of third feature images, and again for at least two of the plurality of third feature images By extracting context features having different ranges, a plurality of corresponding second feature images may be obtained. By realizing the purpose of synchronously extracting context features having a plurality of ranges from the first feature image, it is advantageous in improving the accuracy of semantic segmentation and reducing the amount of calculation in the semantic segmentation process.

In an embodiment of the present invention, context features having different ranges for at least two of a plurality of third feature images are obtained through convolution separable by depth and extension convolution corresponding to extension coefficients having different convolution kernels. It can be extracted, and by realizing the purpose of synchronously extracting context features having a plurality of ranges different from the first feature image, it is advantageous in improving the accuracy of semantic segmentation.

In an embodiment of the present invention, a fourth feature image may be obtained by directly overlaying a plurality of second feature images, or by overlaying at least one of a plurality of second feature images and a plurality of third feature images Since images can also be obtained, the usability is high, information of more scales can be fused, and the accuracy of performing semantic segmentation is improved.

In an embodiment of the present invention, the target image may be obtained by performing upsampling on the fourth feature image in order to maintain the dimension of the target image. Alternatively, sub-pixel convolution is performed on the fourth feature image to improve the effect of semantic segmentation and make the semantic segmentation result more accurate.

In an embodiment of the present invention, a fifth feature image may be acquired before the target image is determined. Here, the fifth feature image is an image obtained by extracting low-dimensional image features for the image to be processed. The number of layers of feature extraction corresponding to the fifth feature image is smaller than the number of layers of feature extraction corresponding to the first feature image. The fourth feature image and the fifth feature image are overlaid and then up-sampled to obtain the target image, and the number of times of synchronously extracting a plurality of contextual features having different ranges from the first feature image reaches the target number. In this case, by performing upsampling only on the fourth feature image, the target image can be obtained, the possibility of losing some important features in the image to be processed after dimensionality reduction processing is lowered, and the accuracy of semantic segmentation is improved.

In an embodiment of the present invention, the last dimension of the target image obtained is the target dimension, wherein the target dimension is determined according to the total number of object categories included in the preset semantic image. Ensure that the dimension of the finally obtained semantic image matches the dimension of the image to be processed.

In an embodiment of the present invention, the machine device navigation may be performed according to the generated semantic image corresponding to the image to be processed, and the possibility of use is high.

It should be understood that the above general description and the following detailed description are merely illustrative and interpretative, and do not limit the embodiments of the present invention.

Here, the drawings are included in the specification and form a part of the specification, show embodiments consistent with the present invention, and are for interpreting the principles of the present invention together with the specification.
Fig. 1A is a color image of the present invention according to an exemplary embodiment.
Fig. 1B is a semantic image of the present invention according to an exemplary embodiment.
Fig. 2 is a flowchart of an image semantic segmentation method according to an exemplary embodiment of the present invention.
Fig. 3 is a flowchart of another image semantic segmentation method according to an exemplary embodiment of the present invention.
Fig. 4 is an exemplary scenario diagram in which the present invention performs context feature extraction of different ranges according to an exemplary embodiment.
Fig. 5 is a flowchart of another image semantic segmentation method according to an exemplary embodiment of the present invention.
Fig. 6 is a flowchart of another image semantic segmentation method according to an exemplary embodiment of the present invention.
Fig. 7 is an exemplary diagram of a neural network architecture for acquiring a semantic image according to an exemplary embodiment of the present invention.
Fig. 8A is an architectural diagram of a back-end sub-network according to an exemplary embodiment of the present invention.
Fig. 8B is an architectural diagram of another back-end sub-network according to an exemplary embodiment of the present invention.
Fig. 8c is an architectural diagram of another back-end sub-network according to an exemplary embodiment of the present invention.
Fig. 8D is an architectural diagram of another back-end sub-network according to an exemplary embodiment of the present invention.
Fig. 9 is a flowchart of another image semantic segmentation method according to an exemplary embodiment of the present invention.
Fig. 10 is a block diagram of an image semantic segmentation apparatus according to an exemplary embodiment of the present invention.
11 is a structural diagram for an image semantic segmentation apparatus according to an exemplary embodiment of the present invention.

Here, exemplary embodiments will be described in detail, examples of which are shown in the drawings. When the following description refers to drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the present invention. To the contrary, these are merely examples of devices and methods consistent with some aspects of the invention as detailed in the claims.

The terminology used in the present invention is for the purpose of describing specific embodiments only, and is not intended to limit the present invention. The singular forms "a thing" and "the" used in the present invention and claims include the plural forms unless the text clearly indicates another meaning. It should also be understood that the terms "and/or" as used herein or” means and includes any or all possible combinations of one or more related listed items.

It should be understood that various information is described using terms such as “first”, “second”, “third” and the like in the present invention, but such information should not be limited to these terms. These terms are only used to distinguish between the same types of information. For example, first information may be referred to as second information, and likewise second information may also be referred to as first information, without departing from the scope of the present invention. This is determined according to the usage situation of the word, for example, the word “if…” used herein may be interpreted as “when…” or “when…” or “in response to a decision”.

An embodiment of the present invention provides an image semantic segmentation method, and may be for a movable mechanical device, such as a robot, an autonomous vehicle, or a drone. Alternatively, the method provided by the embodiments of the present invention may be implemented through a method in which the processor operates computer-executable code.

Image semantic segmentation predicts the type of an object to which it belongs, for each pixel point among input red, green, and blue (RGB) images, and the type of the object is, for example, grass, people, vehicles, and buildings. , sky, etc., but is not limited thereto, and obtains a semantic map having an object type label belonging to the same size and dimension corresponding to the RGB image. For example, FIG. 1A is an RGB image, and FIG. 1B is a corresponding semantic image.

An embodiment of the present invention is to obtain a first feature image through performing feature extraction on an image to be processed obtained by the mechanical device; Furthermore, a target image is determined according to at least a plurality of second feature images by dividing the first feature image several times and synchronously extracting a plurality of context features having different ranges to obtain a plurality of second feature images, ; Finally, a semantic image may be generated based on the last acquired target image. The embodiment of the present invention can sufficiently fuse context information of different scales through context feature extraction and fusion several times, thereby improving the accuracy of semantic segmentation. The mechanical device avoids obstacles in front of the mechanical device according to the semantic image corresponding to the image to be processed, and rationally plans the driving route, so that the use possibility is high.

The above are only exemplary application scenarios of the present invention, and scenarios of other image semantic segmentation methods in which the present invention can be used are all within the protection scope of the present invention.

Fig. 2 is a flowchart of an image semantic segmentation method according to an exemplary embodiment, and includes the following steps.

In step 101, a first feature image is obtained by performing feature extraction on the obtained image to be processed.

In an embodiment of the present invention, the image to be processed may be a real-time image, and the real-time image may be image collected through a camera preset in the mechanical device. Various objects may be included. The image to be processed may be an image that has already been collected by the mechanical device (eg, an image stored in the mechanical device), or an image that requires performing semantic segmentation, which is transmitted to the mechanical device by another device.

a first feature image can be obtained by converting original image information included in the image to be processed into a group of features having distinct physical or statistical meanings; For example, the first feature image can be obtained by extracting high-dimensional image features from the image to be processed through a convolutional network such as a residual network (Residual Networks, ResNet), a Visual Geometry Group (VGG) network, etc. .

Here, in some embodiments, when feature extraction is performed on an image to be processed, for example, Haar-like features (Haar), a local binary pattern (LBP), Features such as a histogram of oriented gradient (HOG) can be extracted. What the HAR-like feature describes is the pixel value intensity conversion information within the local range of the image, the LBP describes the pattern information that the image corresponds to within the local range, and the HOG describes the image that the image corresponds to within the local range. Shape edge gradient information. Alternatively, in some other embodiments, when feature extraction is performed on the image to be processed, high-dimensional visual features of the image to be processed may be extracted.

In step 102, a plurality of context features having different ranges are synchronously extracted from the first feature image to obtain a plurality of second feature images.

In an embodiment of the present invention, the context feature extraction is statistics performed on the distribution of other pixel points in the area adjacent to the pixel point in the first feature image.

Context feature extraction with different ranges means context feature extraction performed at different pixel count intervals. For example, when context feature extraction is performed on the first feature image, a plurality of elements included in the first feature image are extracted. Context feature extraction is performed synchronously for pixel point intervals (eg, 3, 7, and 12 pixel point intervals) to obtain a plurality of second feature images, respectively.

In step 103, a target image is determined according to at least the plurality of second feature images, and the context features having different ranges are synchronously extracted again by using the target image as the new first feature image. .

In an embodiment of the present invention, the target image is an image obtained according to at least a plurality of second feature images each time. After determining the target image, the target image is used as a new first feature image, and comes back to execute step 102 .

In step 104, in response to the number of times of synchronously extracting a plurality of contextual features having different ranges for the first feature image reaches the target number, based on the target image acquired last, the processing A semantic image corresponding to the image to be created is created.

In an embodiment of the present invention, the target number may be a positive integer greater than or equal to two.

In the above embodiment, by performing feature extraction on the obtained image to be processed to obtain a first feature image, further, by synchronously extracting a plurality of contextual features with different ranges from the first feature image, A plurality of second feature images may be acquired. A target image is determined according to at least a plurality of second feature images, and context features having different ranges are synchronously extracted again by using the target image as a new first feature image. When the number of times of synchronously extracting a plurality of context features with different ranges from the first feature image reaches the target number, semantics corresponding to the image to be processed by semantic division based on the last acquired target image You can create an image. In an embodiment of the present invention, by synchronously extracting a plurality of different contextual features from a feature image corresponding to the image to be processed several times, context information of different scales can be sufficiently fused, Improve the accuracy of semantic segmentation.

In some selectable embodiments, for step 101, the feature extraction network may output a first feature image by using the feature extraction network to input the collected images to be processed into the feature extraction network. Here, the feature extraction network may be a neural network capable of performing feature extraction such as Resnet or VGG.

In some selectable embodiments, for example as shown in FIG. 3 , step 102 may include the following steps.

In step 102-1, a plurality of third feature images are obtained by dividing the first feature image into a plurality of channels and synchronously performing a dimension reduction process.

In an embodiment of the present invention, performing dimensionality reduction processing on the first feature image is for better performing context feature extraction subsequently, which is advantageous in reducing the amount of computation to be subsequently processed. The first feature image may be divided into a plurality of channels to perform dimensional reduction processing, and context features having different ranges may be extracted from the dimensionally reduced image corresponding to the plurality of channels subsequently. It is advantageous in improving the accuracy and reduces the computational amount of the semantic segmentation process.

In an embodiment of the present invention, dimensional reduction processing of the same dimension may be performed synchronously by dividing the first feature image into a plurality of channels, for example, as shown in FIG. 4 , 1×1 convolution The dimensions of the plurality of third feature images obtained after the convolutional layer using the kernel performs multi-channel dimensionality reduction processing may be 1×1×256 dimensions.

In step 102-2, context features having different ranges are extracted with respect to at least two third feature images among the plurality of third feature images to obtain a plurality of second feature images.

In an embodiment of the present invention, through a convolution separable by depth and extension convolution corresponding to extension coefficients having different convolution kernels, ranges of at least two third feature images among a plurality of third feature images are different. By extracting context features, a plurality of second feature images can be obtained, and the purpose of synchronously extracting context features having a plurality of ranges different from the first feature image is realized, thereby improving the accuracy of semantic segmentation. It is advantageous to improve Here, for the extended convolution, a convolution kernel having a size of 3×3 may be selected, and a convolution kernel having a size of 5×5 or 7×7 may be used. In an embodiment of the present invention, the convolution of the extended convolution There is no restriction on the kernel size. Here, the extension coefficient r of the extension convolution may be set to a different value according to the scenario of semantic segmentation, for example, r may be set to 6, 12, 18, 32, etc., and the number of pixel points different according to the value of r Context feature extraction can be performed at intervals.

For example, as shown in FIG. 4 , after dimensionality reduction processing of four channels is performed on the first feature image, four third feature images are obtained, and the third feature image 1 to the third feature image, respectively. denoted as 4; Context feature extraction may not be performed on the third feature image 1, and the values of the extension coefficients r corresponding to the third feature images 2, 3, and 4 are 6, 12, and 18, that is, the third feature image 2 , 3, and 4, at intervals of 6 pixel points, 12 pixel points, and 18 pixel points, respectively, context features are extracted to obtain three second feature images.

In the above embodiment, first, the first feature image is divided into a plurality of channels to perform dimensional reduction processing to obtain a plurality of third feature images, and again at least two third feature images among the plurality of third feature images By extracting context features having different ranges for , at least two corresponding second feature images may be obtained. By realizing the purpose of synchronously extracting context features having a plurality of ranges from the first feature image, it is advantageous in improving the accuracy of semantic segmentation and reducing the amount of calculation in the semantic segmentation process.

In some selectable embodiments, for example, as illustrated in FIG. 5 , the process of determining the target image according to at least the plurality of second feature images in step 103 may include the following steps.

In step 103-1, at least the plurality of second feature images are fused to obtain a fourth feature image.

In an embodiment of the present invention, at least the fourth feature image may be obtained by overlaying the plurality of second feature images obtained in the above step.

For example, a plurality of second feature images are collected in one place, and fusion of multi-scale context features is implemented through a convolution operation to obtain a fourth feature image. A plurality of second feature images may be joined to obtain a fourth feature image.

In step 103-2, the target image is determined according to at least the fourth feature image.

In a possible implementation manner, the fourth feature image may be used directly as the target image. In another possible implementation manner, processing capable of enhancing the semantic segmentation effect may be performed on the fourth feature image, thereby obtaining a target image. In another possible implementation manner, the target image may be determined according to the fourth feature image and the feature image associated with another image to be processed.

In the above embodiment, the target image may be determined according to at least the plurality of second feature images, and the possibility of use is high.

In some selectable embodiments, for step 103-1, in a possible implementation manner, by performing an overlay on a plurality of second feature images, a fourth feature image may be obtained. In order to better preserve the feature information corresponding to the image to be processed and improve the accuracy of semantic segmentation, in another possible implementation manner, a third feature of at least one of a plurality of second feature images and a plurality of third feature images An overlay is performed on the image, and the image obtained by overlaying is used as the fourth feature image.

The plurality of third feature images are images obtained after dimensionally reduction processing by synchronously dividing the first feature image into a plurality of channels, and in an embodiment of the present invention, the plurality of second feature images and the plurality of third features By overlaying at least one third feature image among the images, a fusion of multi-scale context features is implemented through a convolution operation to obtain a fourth feature image. A fourth feature image may be obtained by joining at least one third feature image among the plurality of second feature images and the plurality of third feature images.

In the above embodiment, a fourth feature image may be obtained by directly overlaying a plurality of second feature images, or at least one of a plurality of second feature images and a plurality of third feature images for which context feature extraction is not performed. Since the third feature image may be overlaid to obtain the fourth feature image, the usability is high, information of more scales can be fused, and the accuracy of performing semantic segmentation is improved.

In some selectable embodiments, for step 103-2, the target image may be determined using one of the methods below.

In a possible implementation manner, the determining of the target image according to at least the fourth feature image includes performing up-sampling on the fourth feature image to obtain the target image.

In an embodiment of the present invention, since the target image must also subsequently undergo dimensionality reduction processing or generate a semantic image, up-sampling processing must be performed on the fourth feature image in order to maintain the dimension of the target image. do. After determining the fourth feature image, a target image is obtained by directly performing upsampling processing (eg, linear interpolation) on the fourth feature image. Further, the target image is used as the new first feature image, and the step 102 is returned and executed.

When the up-sampling process is performed on the fourth feature image, the corresponding up-sampling factor t may be 2, 4, 8, etc., and each time the up-sampling process is performed on the fourth feature image, the same or different up-sampling A sampling factor can be used. Here, the upsampling factor is the quantity to insert new pixel points using an appropriate interpolation algorithm between pixel points when zooming in on the original image, for example, when the upsampling factor t is 2, between two adjacent pixel points 2 new pixel points can be inserted using a linear interpolation algorithm in .

In another possible implementation manner, the determining of the target image according to at least the fourth feature image includes performing sub-pixel convolution on the fourth feature image to obtain the target image. .

Sub-pixel convolution improves the spatial resolution of the feature map by tiling the pixels in the output feature map depth direction to decrease the feature map depth and increase the spatial scale of the two-dimensional plane.

By performing sub-pixel convolution on the fourth feature image, the effect of semantic segmentation can be improved, and the semantic segmentation result can be made more accurate. After the sub-pixel convolution processing, up-sampling processing may also be performed, acquiring a target image through up-sampling processing, further using the target image as a new first feature image, and returning to execute step 102 .

In another possible implementation manner, dimensionality reduction processing is performed on the first feature image before, and subsequent images are all obtained based on the plurality of third feature images after dimensionality reduction, but the finally generated semantic image is the image to be processed In order to reduce the possibility of losing some important features in the image to be processed after dimensionality reduction processing and to improve the accuracy of semantic segmentation, a fifth feature image may be obtained before the target image is determined.

Here, the fifth feature image is an image obtained by extracting low-dimensional image features from the image to be processed. The number of layers of feature extraction corresponding to the fifth feature image is smaller than the number of layers of feature extraction corresponding to the first feature image. When 10-layer feature extraction is performed on the image to be processed and a first feature image is obtained, an image obtained after performing feature extraction on the previous four layers may be used as a fifth feature image.

Correspondingly, for example, as shown in FIG. 6 , the image semantic segmentation method may further include the following steps.

In step 105, after feature extraction and dimension reduction processing are performed on the image to be processed, a fifth feature image is obtained.

In an embodiment of the present invention, an up-sampling process may be performed by overlaying the fourth and fifth feature images to obtain a target image.

Here, if the number of times of synchronously extracting a plurality of context features having different ranges from the first feature image is less than the target number, step 103-2 is performed by overlaying the fourth feature image and the fifth feature image and then up-sampling , when the number of times of obtaining the target image and synchronously extracting a plurality of context features having different ranges with respect to the first feature image reaches the target number of times, upsampling is performed only on the fourth feature image, so that the target image can be obtained. Using the target image as the new first feature image, return and execute step 102 . When performing the up-sampling process each time, the corresponding up-sampling factor t may be the same or different.

In another possible implementation manner, the target image may be determined according to the fourth characteristic image and the fifth characteristic image as well.

When the number of times of synchronously extracting a plurality of context features having different ranges from the first feature image is less than the target number, the image obtained after performing sub-pixel convolution on the fourth feature image and the fifth feature image By overlaying on , the target image is obtained. When the number of times of synchronously extracting a plurality of context features having different ranges from the first feature image reaches the target number, sub-pixel convolution is directly performed on the fourth feature image to obtain a target image.

In an embodiment of the present invention, in order to secure the effect of semantic segmentation, similarly, when the number of synchronously extracting context features having a plurality of ranges from the first feature image is smaller than the target number, the fourth feature image is A target image may be obtained by performing sub-pixel convolution on the image and overlaying the obtained image and the fifth feature image. If the number reaches the target number, the target image may be obtained by directly performing sub-pixel convolution on the fourth feature image. Here, after sub-pixel convolution is performed on the fourth feature image, up-sampling may be performed again. Further, the target image is used as the new first feature image, and the step 102 is returned and executed.

It should be explained that after each target image is determined, when the target image is used as a new first feature image to synchronously extract a plurality of context features having different ranges again, a plurality of channels for the new first feature image The dimensions of the new plurality of third feature images obtained when the dimensionality reduction process is performed synchronously by dividing by ? may be the same as or different from the dimensions of the plurality of third feature images obtained after the previous dimension reduction process. For example, after dividing the last first feature image into a plurality of channels and synchronously reducing the dimension, a plurality of third feature images of 1×1×256 dimension are obtained, and the new first feature image is divided into a plurality of channels. After dividing and synchronously reducing the dimension, it is possible to obtain a plurality of new 3rd feature images of 1×1×128 dimensions.

In addition, extension coefficients when performing extension convolution on a plurality of third feature images each time may be the same or different. For example, when extension convolution is performed on at least two of the plurality of third feature images last time, corresponding extension coefficients may be 6, 12, and 18, respectively, and at least two of the new plurality of third feature images When the extension convolution is performed on dogs, the corresponding extension coefficients may be 6 and 12, respectively.

In the above embodiment, by determining one target image according to at least the fourth feature image, the precision and accuracy of semantic segmentation are secured, and the possibility of use is high.

In some selectable embodiments, dimensionality reduction and/or dimensionality expansion processing is performed before the target image is output to ensure that the dimension of the finally obtained semantic image matches the dimension of the image to be processed, whereby the target image It can be ensured that the dimension corresponding to , is the target dimension. Here, the target dimension is determined according to the total number of object categories included in the preset semantic image.

For example, the target dimension may be 1×1×16N, where N is the total number of object categories included in the preset semantic image. If four tangible object categories need to be analyzed in the semantic image, the target dimension can be 1×1×64.

In the above embodiment, the dimension corresponding to the last acquired target image is dimensionally reduced and/or dimensionally expanded before the target image is output (for example, convolution using a convolutional layer with a preset number of channels) operation) to ensure that the dimension of the target image is the target dimension, thereby improving the accuracy and precision of semantic segmentation.

In some selectable embodiments, for step 104, after the last target image is obtained, an interpolation algorithm may be used to generate the semantic image, wherein the interpolation algorithm may include a bilinear interpolation algorithm; However, the present invention is not limited thereto.

Taking the above embodiment as a further example, for example, as shown in FIG. 7 , the collected images to be processed (eg, real-time images shown in the figure) can be input to a fully convolutional neural network. and output the corresponding semantic image in the neural network of the full convolution.

The fully convolutional neural network may include a front-end subnetwork and a back-end subnetwork.

Here, the front-end subnetwork may be a feature extraction network, and a neural network such as Resnet or VGG may be used.

In the process of performing training on the front-end subnetwork, we can use artificially tagged image category sample datasets, such as ImageNet, for example. The ImageNet set contains the image and the corresponding image feature labels, and by adjusting the network parameters of the front-end subnetwork, the output of the front-end subnetwork matches the label content in the ImageNet sample set or is within fault tolerance. do.

A first feature image corresponding to the image to be processed can be obtained through the front-end subnetwork, and further, the first feature image is input to the back-end subnetwork to obtain the semantic image output by the back-end subnetwork do.

When training on the back-end subnetwork, for example, using an artificially tagged image semantic segmentation sample set such as CityScapes, the network parameters of the entire neural network can be trained through a back-propagation algorithm, and the front-end Including the network parameters of the subnetworks and the backend subnetworks, the output results of the backend subnetworks match the label content in the CityScapes sample set or fall within the fault tolerance range.

In order to conveniently introduce the network architecture used by the back-end subnetwork, in the embodiment of the present invention, only the target number is 2 as an example, and it should be noted that the target number is another positive integer value greater than 2 All of them fall within the protection scope of the present invention.

In a possible implementation manner, the network architecture of the back-end subnetwork may be as shown in FIG. 8A .

Through the subnetwork 1, first, the first feature image is divided into a plurality of channels, and dimensionality reduction processing is performed synchronously to obtain a plurality of third feature images. Again, context features with different ranges are extracted for at least two third feature images among the plurality of third feature images, and through convolution separable by depth and extension convolution in which convolution kernels are applied to different extension coefficients. , a plurality of second feature images may be acquired.

Further, by performing an up-sampling process (not shown in FIG. 8A ) by overlaying a plurality of second feature images, a target image can be obtained, and a plurality of first images and context feature extraction are performed A target image may be obtained by overlaying at least one third feature image that has not been processed and then performing an up-sampling process.

The target image is directly used as the new first feature image, and through the subnetwork 2, the new first feature image is first divided into a plurality of channels to perform dimensional reduction processing synchronously, and then the plurality of third feature images are again acquire Again, context features having different ranges are extracted for at least two third feature images among the plurality of third feature images, and for example, a convolution separable by depth and an extension convolution corresponding to a different extension coefficient with a convolution kernel. Through , a plurality of second feature images may be acquired. After overlaying a plurality of second feature images again, the target image can be obtained by up-sampling processing, and after overlaying the plurality of first images and at least one third feature image on which context feature extraction is not performed, up-sampling By sampling, the target image can also be obtained.

The semantic image is generated using a bilinear interpolation algorithm for the target image output by the subnetwork 2 .

In the above embodiment, it is possible to synchronously extract and fuse a plurality of context feature extractions with different ranges for the first feature image by dividing it several times, and by sufficiently fusing context information of different scales, semantic segmentation improve the accuracy of In addition, since an extended convolution that can be separated by depth is used, the amount of computation in the semantic segmentation process is reduced.

In another possible implementation manner, the network architecture of the back-end subnetwork may be as shown in FIG. 8B .

Through the subnetwork 1, first, the first feature image is divided into a plurality of channels, and dimensionality reduction processing is performed synchronously to obtain a plurality of third feature images. Again, a context feature having a different range may be extracted for at least two third feature images among the plurality of third feature images, for example, an extension convolution operation separable by depth may be performed, and extension coefficients are different from each other, , to acquire a plurality of second feature images.

In order to improve the effect of semantic segmentation, a target image may be obtained by overlaying a plurality of second feature images to perform sub-pixel convolution and up-sampling processing. Alternatively, a plurality of second feature images and at least one third feature image on which context feature extraction is not performed are overlaid to perform sub-pixel convolution and up-sampling processing (up-sampling process is not shown in FIG. 8B ), You can get the target image.

The target image is directly used as the new first feature image, and through the subnetwork 2, the new first feature image is first divided into a plurality of channels to perform dimensional reduction processing synchronously, and then the plurality of third feature images are again acquire Again, a context feature having a different range may be extracted for at least two third feature images among the plurality of third feature images, for example, an extension convolution operation separable by depth may be performed, and extension coefficients are different from each other, , to acquire a plurality of second feature images. Again, a plurality of second feature images are overlaid to perform sub-pixel convolution and up-sampling processing to obtain a target image, and the plurality of first images and at least one third feature image without context feature extraction are overlaid to perform sub-pixel convolution and up-sampling processing to obtain a target image.

The semantic image is generated using a bilinear interpolation algorithm for the target image output by the subnetwork 2 .

In the above embodiment, it is possible to synchronously extract and fuse a plurality of context feature extractions with different ranges for the first feature image by dividing it several times, and by sufficiently fusing context information of different scales, semantic segmentation improve the accuracy of In addition, since extended convolution that can be separated by depth is used, the amount of computation in the semantic segmentation process is reduced. In addition, the effect of semantic segmentation can be improved through sub-pixel convolution.

In another possible implementation manner, the network architecture of the back-end sub-network may be as shown in FIG. 8C .

Through the subnetwork 1, first, the first feature image is divided into a plurality of channels, and dimensionality reduction processing is performed synchronously to obtain a plurality of third feature images. Again, it is possible to extract context features with different ranges from at least two of the plurality of third feature images, for example, perform an extension convolution operation that is separable by depth, and the extension coefficients are different from each other, and 2 Acquire a feature image.

Further, by overlaying a plurality of second feature images and at least one third feature image that has not been subjected to context feature extraction, overlaying the fifth feature image again is performed, and up-sampling processing (Fig. By performing an up-sampling process (not shown) in 8c, a target image may be obtained. Here, the number of layers of feature extraction corresponding to the fifth feature image is smaller than the number of layers of feature extraction corresponding to the first feature image.

The target image is directly used as the new first feature image, and through the subnetwork 2, the new first feature image is first divided into a plurality of channels to perform dimensional reduction processing synchronously, and then the plurality of third feature images are again acquire Again, a context feature having a different range may be extracted for at least two third feature images among the plurality of third feature images, for example, an extension convolution operation separable by depth may be performed, and extension coefficients are different from each other, , to acquire a plurality of second feature images. Again, a plurality of second feature images and at least one third feature image on which context feature extraction is not performed are overlaid, and an up-sampling process is performed on the overlaid image to obtain a target image.

The semantic image is generated using a bilinear interpolation algorithm for the target image output by the subnetwork 2 .

In another possible implementation manner, the network architecture of the back-end sub-network may be as shown in FIG. 8D .

Through the subnetwork 1, first, the first feature image is divided into a plurality of channels, and dimensionality reduction processing is performed synchronously to obtain a plurality of third feature images. Again, a context feature having a different range may be extracted for at least two third feature images among the plurality of third feature images, for example, an extension convolution operation separable by depth may be performed, and extension coefficients are different from each other, , to acquire a plurality of second feature images.

Further, sub-pixel convolution and up-sampling processing (up-sampling process not shown in FIG. 8D ) is performed by overlaying a plurality of second feature images and at least one third feature image on which context feature extraction is not performed. and overlaying the fifth feature image again to obtain a target image. Here, the number of layers of feature extraction corresponding to the fifth feature image is smaller than the number of layers of feature extraction corresponding to the first feature image.

The target image is directly used as the new first feature image, and through the subnetwork 2, the new first feature image is first divided into a plurality of channels to perform dimensional reduction processing synchronously, and then the plurality of third feature images are again acquire Again, it is possible to extract context features with different ranges from at least two of the plurality of third feature images, for example, perform an extension convolution operation that is separable by depth, and the extension coefficients are different from each other, and 2 Acquire a feature image. Again, a plurality of second feature images and at least one third feature image on which context feature extraction is not performed are overlaid, and sub-pixel convolution and upsampling processing are performed on the overlaid image to obtain a target image.

In addition, in order to ensure that the dimension of the target image is the target dimension, a plurality of second feature images and at least one third feature image on which context feature extraction is not performed are overlaid to perform dimensionality reduction processing and After dimensional expansion processing is performed, sub-pixel convolution and up-sampling processing may be performed to obtain a target image.

The semantic image is generated using a bilinear interpolation algorithm for the target image output by the subnetwork 2 .

In the above embodiment, it is possible to synchronously extract and fuse a plurality of context feature extractions with different ranges for the first feature image by dividing it several times, and by sufficiently fusing context information of different scales, semantic segmentation improve the accuracy of Since an extended convolution that is separable by depth is used, the amount of computation in the semantic segmentation process is reduced. In addition, it is possible to determine a target image by using the fifth feature image, to ensure that important information in the image to be detected is not lost, and similarly improve the accuracy of semantic segmentation.

In some selectable embodiments, after completing step 104 , for example as shown in FIG. 9 , the image semantic segmentation method further comprises a step 106 of performing machine device navigation according to the semantic image. can

In an embodiment of the present invention, navigation may be performed for a mechanical device according to the generated semantic image. For example, if an obstacle is included in the semantic image, navigation to avoid the obstacle may be performed, and if a fork in the semantic image is included, it may be determined whether going straight or cornering is required according to a designated route.

In the above embodiment, the machine device navigation can be performed according to the generated semantic image corresponding to the image to be processed, and the possibility of use is high.

Corresponding to the method embodiments described above, the present invention further provides an embodiment of the apparatus.

As shown in Fig. 10, Fig. 10 is a block diagram of an image semantic segmentation apparatus according to an exemplary embodiment of the present invention, wherein the image semantic segmentation apparatus performs feature extraction on the obtained image to be processed; a feature extraction module 210, configured to acquire a first feature image; a context feature extraction module 220 configured to synchronously extract a plurality of context features having different ranges from the first feature image to obtain a plurality of second feature images; A determining module 230, configured to determine a target image according to at least the plurality of second feature images, and to synchronously extract a plurality of different ranges of context features by using the target image as the new first feature image again ); and in response to the number of times of synchronously extracting a plurality of context features having different ranges for the first feature image reaches the target number, based on the target image acquired last, corresponding to the image to be processed and a semantic image generation module 240 configured to generate a semantic image that is

In some selectable embodiments, the context feature extraction module 220 is configured to divide the first feature image into a plurality of channels and synchronously perform dimension reduction processing to obtain a plurality of third feature images. a first processing sub-module configured; and a second processing sub-module, configured to extract context features having different ranges for at least two third feature images among the plurality of third feature images to obtain a plurality of second feature images.

In some selectable embodiments, the second processing submodule is configured to perform at least two of the plurality of third feature images through convolution separable by depth and extension convolution corresponding to extension coefficients having different convolution kernels. and extracting context features having different ranges from the third feature image to obtain a plurality of second feature images.

In some selectable embodiments, the determining module comprises: a first determining sub-module, configured to fuse at least the plurality of second feature images to obtain a fourth feature image; and a second determining sub-module, configured to determine the target image according to at least the fourth feature image.

In some selectable embodiments, the first determining submodule is configured to: overlay the plurality of second feature images to obtain the fourth feature image; or performing overlay on at least one third feature image among the plurality of second feature images and the plurality of third feature images to obtain the fourth feature image.

In some selectable embodiments, the second determining submodule is configured to perform up-sampling on the fourth feature image to obtain the target image; or perform sub-pixel convolution on the fourth feature image to obtain the target image.

In some selectable embodiments, the image semantic segmentation device is a processing module, configured to obtain a fifth feature image after performing feature extraction and dimension reduction processing on the image to be processed - corresponding to the fifth feature image The number of layers of feature extraction to be used is smaller than the number of layers of feature extraction corresponding to the first feature image; and when the number is smaller than the target number, overlaying the fourth characteristic image and the fifth characteristic image and then up-sampling to obtain the target image; or a second determining sub, configured to obtain the target image by overlaying an image obtained after performing sub-pixel convolution on the fourth feature image and the fifth feature image when the number is smaller than the target number It further includes a module.

In some selectable embodiments, a dimension corresponding to the last acquired target image is a target dimension, and the target dimension is determined according to a total quantity of object categories included in the preset semantic image.

In some selectable embodiments, the image semantic segmentation device further comprises a navigation module, configured to perform machine device navigation according to the semantic image.

In the apparatus embodiment, since it almost corresponds to the method embodiment, reference may be made to the partial description of the method embodiment for related parts. The device embodiments described above are exemplary only, wherein a unit described as a separating member may or may not be physically separated, and a member shown as a unit may or may not be a physical unit, i.e., it may be located in one place, or It may be distributed in a plurality of network units. According to actual needs, some or all of the modules may be selected to implement the purpose of the present invention. A person skilled in the art can understand and practice even if creative labor is not given.

An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored, the computer program for executing the image semantic segmentation method according to any one of the above.

An embodiment of the present invention further provides a computer program, which causes the computer to execute the image semantic segmentation method according to any one of the above.

In some selectable embodiments, embodiments of the present invention provide a computer program product comprising computer readable code, wherein when the computer readable code is run in a device, the processor in the device is configured in any of the above embodiments. Execute the command to implement the provided image semantic segmentation method.

In some selectable embodiments, the embodiment of the present invention further provides another computer program product for storing computer readable instructions, and when the instruction is executed, causes the computer to cause the image semantic segmentation method provided by any of the above embodiments. to execute the action of

The computer program product may be specifically implemented through hardware, software, or a combination thereof. In an optional embodiment, the computer program product is specifically embodied as a computer storage medium, and in another optional embodiment, the computer program product is a software product such as, for example, a Software Development Kit (SDK) or the like. is specifically embodied as

An embodiment of the present invention further provides an image semantic segmentation apparatus, comprising: a processor; and a memory for storing the instructions executable by the processor, wherein the processor is configured to call the executable instructions stored in the memory to implement the image semantic segmentation method according to any one of the preceding claims.

11 is a diagram illustrating a hardware structure of an image semantic segmentation apparatus provided by an embodiment of the present application. The image semantic segmentation device 310 may include a processor 311 , and may further include an input device 312 , an output device 313 , and a memory 314 . The input device 312 , the output device 313 , the memory 314 and the processor 311 are interconnected via a bus.

Memory can be Random Access Memory (RAM), Read-Only Memory (ROM), Erasable Programmable Read Only Memory (EPROM), or Compact Disk Read Only Memory (Compact). Disc Read-Only Memory (CD-ROM)), wherein the memory is for related instructions and data.

The input device is for inputting data and/or signals, and the output device is for outputting data and/or signals. The output device and the input device may be independent materials or may be a whole material.

A processor may include one or a plurality of processors, for example, one or a plurality of central processing units (CPUs), and when the processor is a single CPU, the CPU may be a single-core CPU It may also be a multi-core CPU.

The memory is for storing program codes and data of the network device.

The processor is configured to call the program code and data in the memory to execute the steps in the image semantic segmentation method embodiment. Specifically, reference may be made to the description in the image semantic segmentation method embodiment, which will not be repeated here any longer.

Understandably, FIG. 11 only shows a simple design of one image semantic segmentation device. In practical applications, the image semantic segmentation device may further include other necessary components, respectively, including, but not limited to, any number of input/output devices, processors, controllers, memories, etc., to implement the embodiments of the present application. All possible image semantic segmentation devices are all within the protection scope of the embodiment of the present invention.

In some embodiments, a function possessed by an apparatus provided in an embodiment of the present invention or a module including the function may be used to execute the method described in the above-described method embodiment, and the specific implementation thereof is described in the above-described method embodiment description. , and for the sake of brevity, the description is not repeated here any further.

Other implementations of embodiments of the present invention can be readily conceived by those skilled in the art after considering the specification and practicing the invention disclosed herein. Embodiments of the present invention are intended to cover any modifications, uses or adaptability changes of the present invention, such modifications, uses or adaptability changes conform to the general principles of the embodiments of the present invention and are not disclosed in the embodiments of the present invention. It includes common knowledge or conventional technical means in the field. The specification and examples are to be regarded as illustrative only, the true scope and spirit of the embodiments of the present invention being indicated by the claims below.

The above descriptions are only selectable embodiments of the present invention, and are not intended to limit the embodiments of the present invention, and any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the embodiment of the present invention is, should be included in the scope of protection.

Claims (21)

An image semantic segmentation method comprising:
performing feature extraction on the obtained image to be processed, and obtaining a first feature image;
synchronously extracting a plurality of context features having different ranges from the first feature image to obtain a plurality of second feature images;
determining a target image according to at least the plurality of second feature images, and synchronously extracting a plurality of context features having different ranges again by using the target image as the new first feature image; and
In response to the number of times of synchronously extracting a plurality of context features having different ranges for the first feature image reaches the target number, based on the target image acquired last, the image corresponding to the image to be processed An image semantic segmentation method comprising the step of generating a semantic image. The method of claim 1,
synchronously extracting a plurality of context features having different ranges with respect to the first feature image to obtain a plurality of second feature images,
obtaining a plurality of third feature images by synchronously performing dimension reduction processing by dividing the first feature image into a plurality of channels; and
and extracting context features having different ranges from at least two third feature images among the plurality of third feature images to obtain a plurality of second feature images. 3. The method of claim 2,
extracting context features having different ranges for at least two third feature images among the plurality of third feature images to obtain a plurality of second feature images,
Extracting context features having different ranges for at least two third feature images among the plurality of third feature images through convolution separable by depth and extension convolution corresponding to extension coefficients having different convolution kernels, Image semantic segmentation method comprising the step of acquiring a plurality of second feature images. 4. The method according to any one of claims 1 to 3,
Determining a target image according to at least the plurality of second feature images includes:
fusing at least the plurality of second feature images to obtain a fourth feature image; and
and determining the target image according to at least the fourth feature image. 5. The method of claim 4,
The step of fusing at least the plurality of second feature images to obtain a fourth feature image comprises:
overlaying the plurality of second feature images to obtain the fourth feature image; or
and performing an overlay on at least one third feature image among the plurality of second feature images and the plurality of third feature images to obtain the fourth feature image. 6. The method according to claim 4 or 5,
Determining the target image according to at least the fourth feature image includes:
performing up-sampling on the fourth feature image to obtain the target image; or
and performing sub-pixel convolution on the fourth feature image to obtain the target image. 6. The method according to claim 4 or 5,
The image semantic segmentation method is
After performing feature extraction and dimension reduction processing on the image to be processed, obtaining a fifth feature image - The number of layers of feature extraction corresponding to the fifth feature image is feature extraction corresponding to the first feature image less than the number of layers of - further comprising;
Determining the target image according to at least the fourth feature image includes:
when the number of times is smaller than the target number, oversampling the fourth characteristic image and the fifth characteristic image and then up-sampling to obtain the target image; or
when the number is smaller than the target number, obtaining the target image by overlaying an image obtained after performing sub-pixel convolution on the fourth feature image and the fifth feature image Image semantic segmentation method with 8. The method according to any one of claims 1 to 7,
and a dimension corresponding to the last acquired target image is a target dimension, and the target dimension is determined according to a total number of object categories included in the preset semantic image. 9. The method according to any one of claims 1 to 8,
After generating a semantic image corresponding to the image to be processed, the image semantic segmentation method comprises:
Image semantic segmentation method according to claim 1, further comprising the step of performing machine device navigation according to the semantic image. An image semantic segmentation device comprising:
a feature extraction module, configured to perform feature extraction on the obtained image to be processed, and obtain a first feature image;
a context feature extraction module configured to synchronously extract a plurality of context features having different ranges from the first feature image to obtain a plurality of second feature images;
a determining module, configured to determine a target image according to at least the plurality of second feature images, and to use the target image as the new first feature image to again synchronously extract a plurality of ranges different context features; and
In response to the number of times of synchronously extracting a plurality of context features having different ranges for the first feature image reaches the target number, based on the target image acquired last, the image corresponding to the image to be processed and a semantic image generation module configured to generate a semantic image. 11. The method of claim 10,
The context feature extraction module,
a first processing sub-module configured to synchronously perform dimension reduction processing by dividing the first characteristic image into a plurality of channels to obtain a plurality of third characteristic images; and
and a second processing sub-module, configured to extract context features having different ranges for at least two third feature images among the plurality of third feature images to obtain a plurality of second feature images. Semantic partitioning device. 12. The method of claim 11,
The second processing sub-module is configured to provide a range for at least two third feature images among the plurality of third feature images through convolution separable by depth and extension convolution corresponding to extension coefficients having different convolution kernels. Image semantic segmentation apparatus, configured to extract different context features to obtain a plurality of second feature images. 13. The method according to any one of claims 10 to 12,
The decision module is
a first determining submodule, configured to fuse at least the plurality of second feature images to obtain a fourth feature image; and
and a second determining sub-module, configured to determine the target image according to at least the fourth feature image. 14. The method of claim 13,
The first determining sub-module may include: overlaying the plurality of second feature images to obtain the fourth feature image; or performing overlay on at least one third feature image among the plurality of second feature images and the plurality of third feature images to obtain the fourth feature image. 15. The method of claim 13 or 14,
The second determining sub-module may include: performing up-sampling on the fourth feature image to obtain the target image; or performing sub-pixel convolution on the fourth feature image to obtain the target image. 15. The method of claim 13 or 14,
The image semantic segmentation apparatus is a processing module, configured to obtain a fifth feature image after performing feature extraction and dimension reduction processing on the image to be processed - the number of layers of feature extraction corresponding to the fifth feature image is the smaller than the number of layers of feature extraction corresponding to the first feature image; and
when the number is smaller than the target number, over-sampling the fourth characteristic image and the fifth characteristic image and then up-sampling to obtain the target image; or a second determining sub, configured to obtain the target image by overlaying an image obtained after performing sub-pixel convolution on the fourth feature image and the fifth feature image when the number is smaller than the target number Image semantic segmentation device, characterized in that it further comprises a module. 17. The method according to any one of claims 10 to 16,
and a dimension corresponding to the last acquired target image is a target dimension, and the target dimension is determined according to a total number of object categories included in the preset semantic image. 18. The method according to any one of claims 10 to 17,
The image semantic segmentation apparatus, characterized in that the image semantic segmentation apparatus further comprises a navigation module, configured to perform machine device navigation according to the semantic image. A computer readable storage medium comprising:
A computer program is stored in the storage medium, and the computer program is for executing the image semantic segmentation method according to any one of claims 1 to 9. An image semantic segmentation device comprising:
processor; and
a memory for storing instructions executable by the processor;
The image semantic segmentation apparatus, characterized in that the processor is configured to implement the image semantic segmentation method according to any one of claims 1 to 9 by calling an executable instruction stored in the memory. A computer program comprising:
The computer program, characterized in that it causes the computer to execute the image semantic segmentation method according to any one of claims 1 to 9.

Images