KR20240073747A - Method of processing image data and Electronic device for performing the Method - Google Patents

Abstract

Embodiments of the present application provide an image processing method, an electronic device, and a storage medium. The image processing method includes obtaining image features of an image including a plurality of image frames, determining a target object representation of the image based on the image features using a neural network, and panoramic segmentation of the image based on the target object representation. Includes steps for determining the outcome. The embodiment of the present application predicts panoramic segmentation results of images through clip-level target object representation, and effectively simplifies the network structure to improve segmentation accuracy and robustness. The image processing method performed in an electronic device may be performed using an artificial intelligence model.

Description

Image processing method and electronic device for performing the method {Method of processing image data and Electronic device for performing the Method}

This application relates to image processing methods, electronic devices, storage media, and program products, and more specifically, to the field of image processing using artificial intelligence technology.

Panoramic segmentation is the process of assigning label information to each pixel of a two-dimensional image. Panoramic segmentation of video is an extension of panoramic segmentation in the time domain, and in addition to panoramic segmentation for each image, it also combines object tracking tasks, that is, assigning the same label to pixels belonging to the same instance in different images.

In existing video panorama segmentation techniques, the accuracy of panorama segmentation is low when determining the representation of a panoramic object for a single frame image. In order to obtain correspondence information between each video frame of a video, an additional tracking module is required, making the network structure complex.

Embodiments of the present application provide an image processing method, electronic device, and storage medium aimed at simplifying the network structure and improving the accuracy and robustness of panoramic segmentation. The technical plan is as follows.

According to one embodiment, an image processing method using a panoramic segmentation model includes obtaining image features of an image including a plurality of image frames, and determining a target object representation of the image based on the image features using a neural network. and determining a panoramic segmentation result of the image based on the target object representation. The step of determining the target object representation of the image based on the image feature using the neural network includes performing a plurality of iterative processing on the image feature using the neural network to determine the target object representation of the image. may include.

Determining the target object representation of the image by performing the plurality of iterative processing on the image feature using the neural network includes using the neural network to determine the image feature and the object by previous iterative processing of the image. It may include performing iterative processing based on the expression to determine the object representation by the current iterative processing of the image.

In the case of the first repetition process among the plurality of repetition processes, the object representation by the previous repetition process may be a pre-configured initial object expression.

The step of performing the iterative processing based on the image features and the object representation by the previous iterative process of the image to determine the object representation by the current iterative process of the image includes the object representation by the previous iterative process of the image. obtaining a mask by performing transformation processing on the image features, processing the object representation from the previous iteration and the mask to obtain a first object representation, and processing the current iteration of the image based on the first object representation. It may include the step of determining the object representation by .

The step of obtaining the first object representation by processing the image feature, the object representation by the previous iteration process, and the mask includes performing attention processing on the image feature, the object representation by the previous iteration process, and the mask. It may include obtaining an object representation related to a mask, and obtaining a first object representation by performing self-attention processing and classification processing based on the object representation related to the mask and the object representation by the previous iteration process.

Obtaining an object representation related to the mask by performing the attention processing on the image feature, the object representation by the previous iterative process, and the mask includes key features corresponding to the image feature, the object by the previous iterative process. Obtaining a second object representation based on the representation and the mask, determining, based on the second object representation, a first probability representing an object category included in the image, and the first probability, the image feature. Obtaining an object representation associated with the mask based on the value feature corresponding to and the image feature.

The step of determining an object representation by current iteration processing of the image based on the first object representation includes selecting an object corresponding to each image frame among at least one image frame based on the image features and the first object representation. It may include determining an expression, and determining an object expression by current iteration processing of the image based on the first object expression and the object expression corresponding to each determined image frame.

Determining an object representation corresponding to each image frame of the at least one image frame based on the image feature and the first object representation comprises determining an object representation corresponding to the key feature corresponding to the image feature and the first object representation. determining a fourth object representation, determining a second probability representing the object category included in the image based on the fourth object representation, and value features corresponding to the second probability and the image feature. It may include determining an object representation corresponding to each image frame among the at least one image frame based on .

Determining an object representation by current iterative processing of the image based on the first object representation and the object representation corresponding to each determined image frame comprises: determining the object representation corresponding to each determined image frame; Obtaining a third object representation corresponding to the image by performing classification processing and self-attention processing, and determining an object representation by current iteration processing of the image based on the first object representation and the third object representation. It may include steps.

Determining a panoramic segmentation result of the image based on the target object representation includes performing linear transformation processing on the target object expression, and based on the linearly transformed target object expression and the image features. It may include determining mask information of the image and determining category information of the image based on the linearly transformed target object representation.

According to one embodiment, a method of training a panoramic segmentation model performed by a processor, wherein the panoramic segmentation model includes a first module and a second module, and obtaining training data, wherein the training data includes a training image, Comprising a first image feature of a training image and a sample panorama segmentation result corresponding to the training image, obtaining a second image feature by changing the frame order of the first image feature, through the first module, determining, through the second module, a first predicted object representation and a second predicted object representation of the training image based on the first image feature and the second image feature, respectively; determining a first prediction result and a second prediction result of the training image based on the predicted object representation, respectively, and the sample panorama segmentation result, the first predicted object representation, the second predicted object representation, and the first prediction and training the panoramic segmentation model using a target loss function based on the result and the second prediction result.

Training the panorama segmentation model using a target loss function based on the sample panorama segmentation result, the first prediction object representation, the second prediction object representation, the first prediction result, and the second prediction result, comprising: determining a first similarity matrix based on the first prediction object representation and the second prediction object representation, determining a second similarity matrix based on the sample panorama segmentation result, the first prediction result, and the second prediction result. It may include determining, and when the target loss function is determined to be minimum based on the first similarity matrix and the second similarity matrix, outputting a trained panorama segmentation model.

The beneficial effects of the technical solutions provided in the embodiments of this application are as follows:

The present application provides image processing methods, electronic devices, storage media and program products, and specifically, the obtained image feature corresponds to an image containing at least two image frames, that is, the image feature corresponds to an image containing at least two image frames. Corresponds to the characteristic information of; Then, using a neural network, the target object representation of the image can be determined based on the image features and the panoramic segmentation result of the image can be predicted based on the target object representation. This application processes clip-level objects (e.g., image features, object representations corresponding to at least two image frames) for each image frame of the image without the need to build an additional object tracking module in the network. Correspondence information between networks can be obtained, and the network structure can be effectively simplified. Additionally, since the clip-level object itself carries time-domain information in the video, the implementation of the proposed method can utilize video information more completely, which helps improve the accuracy and robustness of panoramic segmentation.

In order to provide a clearer description of the technical solution of the embodiments of this application, the following briefly introduces the accompanying drawings required when describing the embodiments of this application.
1 is a flowchart of an image processing method according to an embodiment.
Figure 2 is a schematic diagram of an example panorama segmentation according to one embodiment.
Figure 3 is an example diagram of a decision process for visualizing a clip query according to an embodiment.
Figure 4 is a network architecture diagram of a panoramic segmentation model according to an embodiment.
Figure 5 is a flowchart of a panorama segmentation algorithm according to one embodiment.
Figure 6A is a schematic diagram of the framework of a mask decoder according to one embodiment.
6B is a schematic diagram of the framework of another mask decoder according to one embodiment.
Figure 7A is a schematic diagram of a framework of a hierarchical interaction module according to one embodiment.
7B is a schematic diagram of a framework of another hierarchical interaction module according to one embodiment.
Figure 8 is a schematic diagram of the framework of the clip feature query interaction 401 shown in Figures 7A and 7B, according to one embodiment.
Figure 9 is a schematic diagram of the framework of the clip feature query interaction 402 shown in Figure 7B, according to one embodiment.
FIG. 10 is a schematic diagram of the structure of the mask attention module 501 shown in FIG. 8, according to one embodiment.
FIG. 11 is a schematic diagram of the structure of the frame query generation module 601 shown in FIG. 9, according to one embodiment.
Figure 12 is a schematic diagram of the structure of the mutual attention module 605 shown in Figure 9, according to one embodiment.
Figure 13 is a schematic diagram of the structure of the split head module 303 according to one embodiment.
Figure 14 is a schematic diagram of a training process of a panoramic segmentation model according to an embodiment.
Figure 15 is an effect contrast diagram according to an embodiment.
Figure 16 is another effect contrast diagram according to an embodiment.
Figure 17 is another effect contrast diagram according to an embodiment.
18 is a schematic diagram of the structure of an electronic device according to an embodiment.

Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and may be changed and implemented in various forms. Accordingly, the actual implementation form is not limited to the specific disclosed embodiments, and the scope of the present disclosure includes changes, equivalents, or substitutes included in the technical idea described in the embodiments.

Although terms such as “first” or “second” may be used to describe various components, these terms should be interpreted only for the purpose of distinguishing one component from another component. For example, a “first component” may be named a “second component” and similarly, a “second component” may also be named a “first component”.

When a component is referred to as being “connected” to another component, it should be understood that it may be directly connected or connected to the other component, but that other components may exist in between.

Singular expressions include plural expressions unless the context clearly dictates otherwise. In the present disclosure, terms such as "comprise" or "have" are intended to designate the presence of the described features, numbers, steps, operations, components, parts, or combinations thereof, and are intended to indicate the presence of one or more other features or numbers, It should be understood that this does not exclude in advance the possibility of the presence or addition of steps, operations, components, parts, or combinations thereof.

It is obvious to those skilled in the art that the singular forms "one", "one", and "the" as used herein may also include plural forms, unless specifically stated. As used in the embodiments of this specification, the terms “comprise” and “contains” mean that the corresponding feature can be implemented as a presented feature, information, data, step, operation, element, and/or component, and that the present technology It does not exclude other features, information, data, steps, operations, elements, components, and/or combinations thereof, etc. supported by the field. When an element is said to be "connected" or "coupled" to another element, one element may be directly connected or joined to another element, or a connection may be formed between one element and the other element through intermediate elements. . Additionally, in this specification, “connection” or “coupling” may include wireless connection or wireless coupling. As used herein, the term "and/or" refers to at least one of the items defined by that term, for example, "A and/or B" embodied by "A", embodied by "B", or "A and B". Indicates that it is implemented as .

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related technology, and should not be interpreted in an idealized or overly formal sense unless explicitly defined in the present disclosure. No.

Artificial intelligence (AI) is the theory, method, or method of using digital computers or machines controlled by digital computers to simulate, extend, and extend human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve the best results. It is a technology and application system. In other words, artificial intelligence is a comprehensive technology of computer science that seeks to understand the nature of intelligence and produce new intelligent machines that can respond similarly to human intelligence. Artificial intelligence is the study of design principles and implementation methods of various intelligent machines to enable machines to have perception, reasoning, and decision-making functions. Artificial intelligence technology is a comprehensive study that encompasses a wide range of fields, including both hardware- and software-side technologies. The basic technologies of artificial intelligence generally include technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation/interaction systems, electromechanical integration, etc. Artificial intelligence software technology mainly includes major directions such as computer vision technology, speech processing technology, natural language processing technology and machine learning/deep learning, autonomous driving, and smart transportation.

Specifically, this application covers machine learning (ML) and computer vision technology (Computer Vision, CV), which are cross-disciplinary departments related to various departments such as probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It's related. We specialize in the study of how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and a fundamental method of making computers intelligent, and is applied to various fields of artificial intelligence. Machine learning and deep learning generally include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and formal learning. Computer vision is a science that studies how machines "see". More specifically, it refers to machine vision that identifies and measures objects using cameras and computers instead of human eyes. It further refers to human eye observation and measurement through graphic processing. This refers to performing computer processing on images to make them more suitable for device detection. As a scientific field, computer vision attempts to build artificial intelligence systems that can obtain information from images or multidimensional data by studying related theories and technologies. Computer vision technologies generally include image processing, image recognition, image semantic understanding, image retrieval, OCR, image processing, image semantic understanding, image content/action recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, and synchronization. It includes technologies such as positioning and map building, autonomous driving, and smart transportation, as well as general biometric technologies such as facial recognition and fingerprint recognition.

The present disclosure proposes an image processing method, electronic device, storage medium, and program product, and specifically, a Robust Clip-object-centric Representation Learning for Video Panoptic Segmentation algorithm. It can be implemented, and the implementation of the method does not require an object tracking module, simplifies the algorithm structure, and at the same time improves the segmentation accuracy and robustness to a certain level.

Below, several exemplary embodiments are described. Implementation methods may be cross-referenced, referenced, or combined, and the same terminology, similar functions, similar implementation steps, etc. among different implementation methods will not be repeatedly described.

1 is a flowchart of an image processing method according to an embodiment, and the method may be executed by any electronic device such as a terminal or server. The terminal may be a smartphone, tablet, laptop, desktop computer, smart speaker, smart watch, vehicle-mounted device, etc. A server may be an independent physical server, a server cluster composed of multiple physical servers, or a distributed system, and may include cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, and security. It may be, but is not limited to, a cloud server that provides basic cloud computing services such as services, CDN, big data, and artificial intelligence platforms.

To better explain the embodiment, a panoramic segmented scene is described below by combining FIGS. 2 and 3.

Image panorama segmentation is the process of assigning label information (e.g., semantic label and instance label information) to each pixel of a two-dimensional image. Image content can be divided into two categories, and one type is 'stuff', which refers to content that does not need to distinguish between different objects such as grass, sky, and buildings. As shown in the semantic segmentation result in Figure 2, semantic labels are used. predict. Another type is 'thing', which is content that needs to distinguish between different objects such as people and cars, and predicts the instance label as shown in the instance segmentation result in Figure 2. The panorama segmentation task can be viewed as a composite task of semantic segmentation and instance segmentation, as shown in the panorama segmentation result in Figure 2.

Image panorama segmentation is an extension of image panorama segmentation in the time domain. Specifically, in addition to performing panoramic segmentation for each frame of an image, video panoramic segmentation also combines object tracking operations, that is, assigning the same label to pixels belonging to the same instance in different images.

In one embodiment, for scenes of video panoramic segmentation, a clip-level object representation is proposed to represent panoramic objects in arbitrary video clips. Specifically, two contents, 'stuff' and 'thing', are uniformly expressed as panoramic objects. For 'stuff' content (e.g. sky, grass, etc.), all pixels of the same type in the image form a panorama object (e.g. pixels of all sky categories form a sky panorama object). In the case of 'thing' content (e.g. pedestrians, cars, etc.), each individual constitutes a panoramic object. The panoramic object representation of a single video frame can be called a frame query, that is, an object representation for a single image. When processing a video clip, a panoramic object on a single frame can be processed as a panoramic object within a video clip, that is, an object representation of the video (clip-object representation as shown in Figure 5), and the practice of the present application In the example, it is also called a clip query, and a clip query can be expressed as one vector (the length of the vector is C, and C can be one hyperparameter). Assuming there are L clip queries on a video clip, and L can be an integer greater than or equal to 0, the representation of all clip panorama objects on that video clip forms a matrix of dimension LxC, i.e. a clip object-centric representation (clip- object-centric representation).

For example, a clip query included in a video clip can be expressed as [Equation 1] below.

In [Equation 1], every clip query of a video clip represents L vectors (vector length is C), and each vector represents the target of one clip level; C is the vector dimension and is a hyperparameter that can be used to control the complexity of the object at the clip level.

Optionally, a clip query is a set of learnable parameters that can be randomly initialized during the network training phase and progressively optimized through interaction with spatiotemporal information such as time domain and spatial information.

Specifically, as shown in Figure 3, an example of four clip queries is provided as the video frame progresses over time, and each clip query corresponds one-to-one to the feature map of each video frame.

Hereinafter, dimensions, operators, etc. of some constants representing vectors and tensors that may be included in the accompanying drawings of one embodiment will be described.

T: Indicates the length of the video clip, that is, the number of frames of the video clip.

L: Represents the maximum number of clip queries and is the number of panoramic objects in the clip.

C: Indicates the channel dimension of feature maps and clip queries.

H and W: Indicate the resolution of the image (i.e. video frame), where H is the height and W is the width. It can also indicate the length and width of a two-dimensional image.

nc: Indicates the total number of categories.

: 요소별 가산 연산을 나타낸다.

: Indicates addition operation for each element.

: 행렬 곱셈 연산을 나타낸다.

: Indicates a matrix multiplication operation.

Hereinafter, an image processing method according to an embodiment will be described.

Specifically, as shown in FIG. 1, the image processing method according to one embodiment includes the following steps S101-S103.

Step S101: Obtain image features of the image; The video includes at least two video frames.

Step S102: Determine a target object representation of the image based on the image features using a neural network.

Step S103: Determine a panoramic segmentation result of the image based on the target object representation.

Specifically, the present application can implement the image processing method through a panoramic segmentation model. As shown in Figure 4, the panoramic segmentation model may include a mask decoder and a segmentation head module. Optionally, as shown in Figure 5, the panoramic segmentation model may further include a feature extraction module. In other words, the panoramic segmentation model can process feature maps obtained by extracting images (also called video clips, representing data containing at least two image frames) from another network, and can also extract features from the acquired images themselves. After performing this, the extracted image features can also be processed.

The feature extraction module (Clip feature extractor) may adopt a general-purpose feature extraction network structure such as a backbone network (Res50-backbone) and a pixel decoder for pixels, and the present disclosure is not limited thereto. Optionally, as shown in FIG. 5, the feature extraction module 301 may extract clip-level multi-scale features by performing feature extraction on the input video clip.

For every frame of the input video clip (t-T+1, t-T+2, ??, t; where T represents the length of the video clip, that is, the number of video frames included in the video clip), the features Video features (also called clip features) can be extracted through the extraction module. The feature extraction module can input multiple frames or a single frame. In the case of an image containing multiple image frames, multiple image frames can be input together into the feature extraction module, or each video frame can be input into the feature extraction module frame by frame. Optionally, image features can be extracted through a single frame input method to simplify the feature extraction task and improve feature extraction speed.

A masked decoder may be composed of N hierarchical interaction modules (HIMs), where N is an integer greater than or equal to 1, so the masked decoder may be composed of a plurality of cascaded HIMs. For example, the hierarchical interaction modules at each level have the same structure, but different parameters. Specifically, the mask decoder may determine the target object representation of the image based on the input image features.

The Segmentation Head module can output segmentation results of panoramic objects in video clips based on target object expressions such as category, mask, and object ID. Specifically, the obtained mask is a mask of a clip panorama object defined in several frames, and pixels belonging to the same mask in different frames represent the correspondence between objects in different frames. This means that object IDs can be obtained automatically without matching or tracking between different video frames. A mask can be understood as a template for an image filter. When extracting a target object from a feature map, n*n (the value of n can be considered based on factors such as receptive field and accuracy, and can be set to, for example, 3*3, 5*5, 7*7, etc. After filtering the image through a matrix of ), the target object can be highlighted.

Hereinafter, the interaction process in the embodiments of the present application will be described.

In one feasible embodiment, in step S102, determining a target object representation of the image based on the image features using a neural network includes the following step A1.

Step A1: Perform multiple iterative processing on the image features using a neural network to determine a target object representation of the image.

Each iterative process includes performing the iterative process based on the image features and the object representation by the previous iterative process of the image to determine the object representation by the current iterative process of the image.

Optionally, in the case of the first iteration among a plurality of iterations, the object representation by the previous iteration is a pre-configured initial object representation.

Optionally, the mask decoder may include one or more hierarchical interaction modules. When including two or more hierarchical interaction modules, each hierarchical interaction module is cascaded and aligned, and the output of the previous level module can serve as the input of the next level module. Based on the network structure, the mask decoder can implement multiple repetitions of image features. The number of iterations is related to the number of hierarchical interaction modules included in the mask decoder.

The hierarchical interaction module of the level can process the image features of the corresponding image and the object representation by previous iteration processing (i.e., the output of the hierarchical interaction module of the previous level), and output the object representation by current iteration processing. can do. It is a first-order ordered hierarchical interaction module whose input includes image features of the image and a pre-configured initial object representation. An ordered hierarchical interaction module at the second and subsequent levels, whose inputs include the corresponding image features and the object representation output by the hierarchical interaction module at the previous level (i.e., the object representation by the previous iterative process).

Optionally, the initial object representation is the same when compared to the image requiring panoramic segmentation processing.

As shown in Figures 4 and 5, the input of the first-order hierarchical interaction module in the mask decoder is the image features extracted by the feature extraction module as well as the initial clip query (also called initial object representation, i.e. obtained from network training). parameters), and the clip query input by the hierarchical interaction module of the subsequent level is the clip query output by the hierarchical interaction module of the previous level. Specifically, the mask decoder can perform a relational operation between a clip query and video features to obtain a specific query (i.e., target object representation, i.e., clip query output shown in FIG. 5) of the video clip.

The algorithmic process of the present application embodiment performs relationship operations between clip queries and video features extracted from video clips, and aligns each clip query with a specific clip panorama object on the video clip until finally obtaining the segmentation result of the clip panorama object. .

In one feasible embodiment, as shown in Figure 6A, the mask decoder may include N identical hierarchical interaction modules. Optionally, as shown in FIG. 6B, the hierarchical interaction module may be divided into a first hierarchical interaction module and a second hierarchical interaction module. That is, the mask decoder may include M first hierarchical interaction modules and N-M second hierarchical interaction modules, where M is less than or equal to N.

As shown in Figure 7A, the network structure of the first hierarchical interaction module may include a clip feature query interaction module 401, and as shown in Figure 7B, the second hierarchical interaction module may include It may include a clip feature query interaction module 401 and a clip frame query interaction module 402. That is, the first hierarchical interaction module may be a simplified network of the second hierarchical interaction module.

Optionally, the mask decoder may be comprised of any combination of a first hierarchical interaction module and a second hierarchical interaction module.

Optionally, when the hierarchical interaction module adopted by the mask decoder includes a creep feature query interaction module 401, in step A1, based on the image features and the object representation by previous iterative processing of the image, The step of performing iterative processing to determine the object representation of the image by the current iterative process includes the following steps A11-A13.

Step A11: Obtain a mask by converting the object expression from the previous iteration.

Step A12: Process the image features, the object representation by the previous iterative process, and the mask to obtain a first object representation.

Step A13: Determine the object representation for the current iteration based on the first object representation.

In step A11, the conversion of the object representation by the previous iteration can be processed through mask branching of the network structure as shown in FIG. 13. That is, after performing a plurality of linear transformation processes on the object expression through the previous iterative process, image features and matrix multiplication are performed to obtain the final transformed mask.

In one feasible embodiment, as shown in FIGS. 7A and 7B, in the current iterative process, the data input to the clip feature query interaction module 401 is the image feature (clip feature and object representation by previous iteration processing (if the current iteration is the first, the clip query S can be the initial clip query as shown in Figure 5, and if the current iteration is not the first, the clip query S is output by the hierarchical interaction module of the previous level can be a clip query) and a binary mask (clip mask MA) obtained from the transformation of the object representation by a previous iteration. After processing the clip query and feature map, the clip query output, also called the first object representation, can be obtained. Optionally, the first object representation can be used directly as the object representation by the current iteration.

Processing implemented by the clip feature query interaction module 401 may obtain the location and appearance information of the object from every pixel of the clip feature. Masks can be used to remove the influence of irrelevant regions and accelerate the learning process.

In one feasible embodiment, in step A12, processing the image features, the object representation by the previous iterative processing, and the mask to obtain a first object representation includes the following steps A121-A122.

Step A121: Perform attention processing on the image feature, the object representation by the previous iteration process, and the mask to obtain an object representation related to the mask.

Step A122: Obtain a first object representation by performing self-attention processing and classification processing based on the object representation related to the mask and the object representation by the previous iteration process.

In one feasible embodiment, as shown in Figure 8, input clip features converted), processing can be performed using the mask attention module 501 to obtain a fine-grained clip query. Based on this, the processing results of step A121 can be added element-by-element to the clip query S, and the added sum result can be normalized (implemented through the summing and normalization module 502 shown in FIG. 8). Afterwards, self-attention is processed on the normalized result A through the self-attention module 503, the output result is added to the normalized result A for each element, and the added sum result is normalized (shown in FIG. 8) implemented through the integrated summation and normalization module 504). Then, through the Feed Forward Network (FFN) module 505, a feedforward operation (classification processing) is performed on the normalized result B, the output result is added to the normalized result B for each element, and the addition The sum result is normalized (implemented through the sum and normalization module 506 shown in Figure 8) to obtain a clip query output (first object representation).

As shown in Figure 8, in the self-attention module 503, the input Q, K, and V correspond to the dimensions L, C of the clip query (normalized result A).

Optionally, in the network structure shown in Figure 8, the processing order of the FFN module 505 and other modules including the attention mechanism can be exchanged. If the positions of the self-attention module 503 and the FFN module 505 can be exchanged in the network structure shown in FIG. 8, the output of the summation and normalization module 502 after the exchange will be used as the input of the FFN module 505. You can.

Optionally, in step A121, performing attention processing on the image feature, the object representation by the previous iterative process, and the mask to obtain an object representation related to the mask includes the following steps A121a-A121c.

Step A121a: Obtain a second object representation based on the key feature corresponding to the image feature, the object representation by the previous iterative process, and the mask.

Step A121b: Based on the second object representation, determine a first probability representing an object category included in the image.

Step A121c: Obtain an object representation associated with the mask based on the first probability, a value feature corresponding to the image feature, and the image feature.

In one feasible embodiment, the inputs and outputs of steps A121a to A121c may be tensors, and the dimensions of the tensors may refer to FIG. 10.

First, input Q (clip query, object representation by previous iteration processing, corresponding to dimensions L, C), K (using image features as key features, corresponding to dimensions THW, C), and V (using image features as key features, corresponding to dimensions THW, C) Use, respectively, to perform a linear operation on the dimensions THW, corresponding to C). The linear calculation modules 701, 702, and 703 shown in FIG. 10 may include one or more linear calculation layers. Subsequently, the output of the linear operation module 701 and the output of the linear operation module 702 may perform a matrix multiplication operation through the module 704, and the output result of the module 704 may be clipped through the module 705. By adding each element to the mask MA, the output of module 705 (second object representation) can be converted to a first probability corresponding to the category by performing a softmax operation (can be performed in the THW dimension) through module 706. . The first probability may represent all object categories included in the image, for example, if the category is a car, the probability is x, and if the category is a lamppost, the probability is y. Based on this, the output of the module 706 (transformed first probability) can be linearly transformed through the module 707 and then matrix multiplied by the value feature, and the result output by the module 707 is the module ( 708), the final clip query output (mask-related object representation) can be obtained by overlapping each element with the image features after linear transformation processing.

The location information (P_q) and (P_k) shown in FIG. 10 are location information corresponding to Q and K and can be generated by a general-purpose location information encoding module, and the location information is optional for the mask attention module 501.

Optionally, in step A13, when the hierarchical interaction module adopted by the mask decoder includes a clip feature query interaction module 401 and a clip frame query interaction module 402, based on the first object representation The step of determining the object representation by the current iteration process includes the following steps A131-A132.

Step A131: Determine an object representation corresponding to each image frame among at least one image frame based on the image feature and the first object representation.

Step A132: An object representation by the current iterative process is determined based on the first object representation and the object representation corresponding to the determined image frame.

Specifically, as shown in FIG. 9, for the input clip feature 601) to obtain a frame query on the [1, T] frame (where T represents the number of video frames included in the video). For each C-dimension clip query vector, we can obtain a C-dimension frame query vector for each frame in T frames, so the overall dimension changes from L, C to TL, C.

As shown in Figure 7b, in the current iterative process, after obtaining the clip query output by the clip feature query interaction module 401, the corresponding clip query and frame query (object representation corresponding to each image frame) You can also perform processing between them to obtain the final output (object representation by the current iteration) of the hierarchical interaction module at the current level.

Optionally, the first object representation may be processed as one or more frame queries or frame queries each corresponding 1:1 to all image frames included in the image.

Optionally, in step A131, determining an object representation corresponding to each image frame of the at least one image frame based on the image features and the first object representation includes the following steps A131a-A131c.

Step A131a: Determine a fourth object representation based on the key feature corresponding to the image feature and the first object representation.

Step A131b: Determine a second probability representing an object category included in the image based on the fourth object representation.

Step A131c: Determine an object representation corresponding to each image frame among at least one image frame based on the second probability and the value feature corresponding to the image feature.

Specifically, the input and output of the above-described steps A131a-A131c may be a tensor, and the dimension of the tensor may refer to FIG. 11.

First, input Q (clip query, first object representation, i.e. output of step A12 above, corresponding to dimensions L, C), K (using image features as key features, corresponding to dimensions THW, C) and V (image features As the value feature, we perform a linear operation on each dimension (corresponding to THW, C). The linear calculation modules 801, 802, and 803 shown in FIG. 11 may include one or more linear calculation layers. Next, a matrix multiplication operation is performed on the output of the linear operation module 801 and the output of the linear operation module 802 (executed through the module 804) to produce an output result of the module 804 (fourth object representation). is converted into a second probability corresponding to the category by softmax operation (executed through module 805, the second probability may represent the object category included in the image). Based on this, a matrix multiplication operation is performed on the value features after linear transformation processing through the module 806 and the output (second probability) of the module 805 to output the final frame query (each image among at least one image frame). You can obtain an object representation corresponding to the frame.

Optionally, a reshape operation can be performed on the input of module 806 to change the dimension of the tensor, see Figure 11.

The location information (P_q) and (P_k) shown in FIG. 11 are location information corresponding to Q and K and can be generated by a general-purpose location information encoding module, and the location information is optional for the frame query generation module 601. .

In one embodiment, in step A132, determining the object representation by current iteration process based on the first object representation and the object representation corresponding to the determined image frame includes the following steps B1-B2.

Step B1: Perform classification processing and self-attention processing on the object expression corresponding to the determined image frame to obtain a third object expression corresponding to the image.

Step B2: Determine an object representation by current iteration process based on the first object representation and the third object representation.

Optionally, in the operation of the clip frame query interaction module 402, both its input and output may be tensors, the dimensions of which are shown in Figure 9.

In one feasible embodiment, as shown in Figure 9, a feedforward operation (classification processing) is performed via the FFN module 602 on the results of the module 601 output (object representation corresponding to the determined image frame). And, the result of the output of the module 602 is subjected to self-attention processing through the self-attention module 603. Then, the results after the self-attention operation are superimposed on the results of the output of the module 602 for each element, and normalization processing is performed on the added sum result (implemented through the summation and normalization module 604 shown in FIG. 9). The result C (third object representation) obtained from the normalization process is obtained. Based on this, mutual attention calculation is performed on the result C obtained from normalization processing through the mutual attention module 605 and the input clip query S. At this time, the dimensions of the output of the module 605 are L and C, which are the input It is the same as the clip query S. Finally, the result of the output of module 605 is added to the clip query S for each element, and the added sum result is normalized to obtain a result D obtained through the normalization process. d is the clip query output of the clip frame query interaction module 402 (target object representation by current iteration process).

As shown in Figure 9, in the self-attention module 603, the input Q, K, and V correspond to the dimensions TL, C of the output result of the module 602.

Optionally, in the network structure shown in Figure 9, the processing order of the FFN module 602 and other modules including the attention mechanism can be swapped. If the positions of the self-attention module 603 and the FFN module 602 can be exchanged in the network structure shown in FIG. 9, the output of the frame query generation module 601 after the exchange is input to the self-attention module 603. can be used, and the output of the self-attention module 603 can be used as the input of the FFN module 602.

Optionally, the specific network structure of the mutual attention module 605 shown in FIG. 9 may refer to the structure shown in FIG. 12.

First, using the input Q (a clip query, the first object representation, i.e. the output of the clip feature query interaction module 401, corresponding to dimensions L, C), K (the result of the output of the module 604) as key features; Perform linear operations on dimensions TL, C (corresponding to dimension TL, C) and V (corresponding to dimension TL, C, respectively, using the result of the output of module 604 as the value feature). The linear calculation modules 1201, 1202, and 1203 shown in FIG. 12 may include one or more linear calculation layers. Next, a matrix multiplication operation is performed on the output of the linear operation module 1201 and the result of the output of the linear operation module 1202 (executed through the module 1204), and a softmax operation is performed on the output result of the module 1204. Convert to a third probability (executed through module 1205) corresponding to the category. Based on this, a matrix multiplication operation is performed on the value features of the image features after linear transformation processing through the module 1206 and the output (third probability) of the module 1205, and the output of the module 1206 is linearly transformed. The mutual attention output of the module 605 is obtained by superimposing the processed input Q.

The location information (P_q) and (P_k) shown in FIG. 12 are location information corresponding to Q and K and can be generated by a general-purpose location information encoding module, and the location information is optional for the mutual attention module 605.

Hereinafter, the image panorama segmentation process of the embodiment of the present application will be described.

Specifically, in step S103, determining a panoramic segmentation result of the image based on the target object representation includes the following steps S103a-S103b.

Step S103a: Perform linear transformation processing on the target object representation.

Step S103b: Mask information of the image is determined based on the target object expression after linear transformation and the image characteristics, and category information of the image is determined based on the target object expression after linear transformation.

In one feasible embodiment, as shown in Figure 13, the input of the segmentation head module includes a clip query S (output of the mask decoder) and a clip feature X, the output of which is the predicted mask information (module 903) output) and category information (output of module 905). The split head module can be specifically divided into two branches, one is the mask branch consisting of modules 901, 902 and 903, and the other is the category branch consisting of modules 904 and 905.

In the mask branch, first perform a linear transformation on the clip query S to obtain the output of module 901, then perform a linear transformation on the output of module 901 to obtain the output of module 902 (1, Multiple linear operation modules, such as two or three, can be set in the mask branch (the present application does not limit the number of linear operation modules included in the mask branch). Based on this, a matrix multiplication operation is performed on the output of the module 902 and the clip feature X to obtain a mask output. The mask output (mask information) shown in Figure 13 belongs to the clip level, and object IDs can be automatically obtained without matching or tracking between different video frames. That is, in the embodiment of the present application, mask information output by mask branching may include a mask and an object ID.

In the category branch, first perform a linear transformation on the clip query S to obtain the output of module 904, then perform a linear transformation on the output of module 904 to obtain the output of module 905 (1 , multiple linear operation modules, such as two or three, can be set in a category branch, and the present application does not limit the number of linear operation modules included in a category branch). This is the category output (category information) shown in FIG. 13.

Optionally, if the number of frames of the video clip is greater than 1, the mask information output from the mask branch belongs to the clip level, so the object ID of every frame of the video can be automatically obtained. Therefore, the category information output from the category branch is the semantic information of the object (e.g., car, person, etc.).

Hereinafter, the training process of the panoramic segmentation model in the embodiment of the present application will be described.

Specifically, when training a network, the prediction results (mask and category) should be constrained to be similar to the mask and category of the ground-truth (GT, label), which can be achieved by minimizing the loss function. As shown in Figure 14, the correspondence between the prediction mask and the GT mask can be established using a binary matching algorithm. The loss function used here includes several items such as cross-entropy loss at the pixel level, panoramic quality loss at the panoramic object level, mask ID loss, and mask similarity loss (Dice-based calculation).

In one feasible embodiment, the training phase of the panoramic segmentation model includes the following steps B1-B2.

Step B1: Obtain training data. The training data includes a training image, a first image feature of the training image, and a sample panorama segmentation result corresponding to the training image.

Step B2: Train the panoramic segmentation model based on the training data to obtain a trained panoramic segmentation model. During training, perform the following steps B21-B24.

Step B21: Obtain a second video feature by changing the frame order of the first video feature.

Step B22: Determine a first predicted object representation and a second predicted object representation of the training image based on the first image feature and the second image feature, respectively, through the hierarchical interaction module (i.e., the first module). .

Step B23: Determine a first prediction result and a second prediction result of the training image based on the first prediction object representation and the second prediction object representation, respectively, through the segmentation head module (i.e., the second module).

Step B24: Train the panorama segmentation model using a target loss function based on the sample panorama segmentation result, the first prediction object representation, the second prediction object representation, the first prediction result, and the second prediction result. .

Optionally, in the present application, the frame order of the clip feature map X (the first video feature shown in FIG. 14) can be replaced to obtain a replaced frame feature map . X and In network training, clip query vectors corresponding to the same clip panorama object in out_S and out_S' should be similar, and clip query vectors corresponding to different clip panorama objects should not be similar. out_S (the first prediction object representation) and out_S' (the second prediction object representation) may be processed by the split head module to obtain the first prediction result and the second prediction result, respectively.

Optionally, in step B24, segment the panorama using a target loss function based on the sample panorama segmentation result, the first prediction object representation, the second prediction object representation, the first prediction result and the second prediction result. The step of training the model includes the following steps B242-B243.

Step B241: Determine a first similarity matrix between object representations based on the first predicted object representation and the second predicted object representation.

Step B242: Determine a second similarity matrix between segmentation results based on the sample panorama segmentation result, the first prediction result, and the second prediction result.

Step B243: When the target loss function is determined to be minimum based on the first similarity matrix and the second similarity matrix, the trained panorama segmentation result is output.

Specifically, the target loss function (loss function relative to clip) can be expressed as [Equation 2] below.

where As shown in FIG. 14, if two clip query vectors correspond to clip panorama objects of the same ground-truth, the corresponding position value of Y is 1; otherwise, it is 0). W is the weight matrix, which has the value 1 for positions of the same category and 0 otherwise (in this application there is no need for supervision considering that objects of different categories are obvious). Additionally, in [Equation 2], the multiplication symbol "*" indicates element-wise multiplication, while Ave() is the average of the panoramic objects of all clips.

In the embodiments of this application, the network structure can be implemented using Python, a deep learning framework. Compared with existing technologies, the network can effectively reduce computational complexity while simplifying the network structure, while making full use of image information and effectively improving segmentation accuracy. Here, accuracy can be measured using video panoptic quality (VPQ).

In order to better explain the technical effects that can be achieved in this application, the segmentation results shown in FIGS. 15 to 17 will be described below.

As shown in Figure 15, neither the panorama segmentation results obtained through general techniques nor the sample panorama segmentation results that can be used for network training recognize cars that exist between the cars shown by the two circles (some instances are missing). In the panorama segmentation result obtained using the image processing method according to the present disclosure, a vehicle (front vehicle) located between vehicles (rear vehicle) can be identified.

As shown in Figure 16, in the panorama segmentation result of the general technique, on the one hand, the mask of the bicycle is incomplete, so the ID belonging to the bicycle cannot be assigned; Additionally, the smaller pedestrian in the third video frame cannot be identified. Additionally, a car (front car) existing between cars (rear car) in the first frame image frame cannot be identified. In the sample panorama segmentation results used for network training, cars (indicated by circles) existing between cars in the first frame video frame cannot be recognized. However, the panorama segmentation result obtained using the image processing method provided in the embodiment according to the present disclosure can effectively compensate for the shortcomings that exist between the existing technology and the sample panorama segmentation result.

As shown in Figure 17, the present disclosure can obtain a clearer bicycle outline representation, which means that the present application can achieve better segmentation accuracy. Compared with the sample panoramic segmentation results used for network training, the segmentation results belonging to the bicycle obtained in this application are consistent in each image frame, while the segmentation results for the same bicycle in each image frame of the sample panoramic segmentation results are different; This means that this application can achieve better robustness.

In select embodiments, an electronic device is provided. As shown in FIG. 18, the electronic device 4000 shown in FIG. 18 includes a processor 4001 and a memory 4003. Processor 4001 is connected to memory 4003 via bus 4002, for example. Optionally, the electronic device 4000 may further include a transceiver 4004, which may be used for data interaction between the electronic device and another electronic device, such as transmitting and/or receiving data. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the corresponding electronic device 4000 does not constitute a limitation to the embodiments of the present application.

Processor 4001 may be a CPU, general purpose processor, DSP, application specific integrated circuit (ASIC), field programmable gate array (FPGA) or other programmable logic device, transistor logic device, hardware component, or any combination thereof. there is. It may implement or implement various example logical blocks, modules, and circuits described in this application. The processor 4001 may also be a combination that realizes computing functions, including, for example, a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

Bus 4002 may include a path for transferring information between the components. Bus 4002 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus 4002 can be divided into an address bus, a data bus, a control bus, etc. For convenience of illustration, only one thick line is shown in Figure 18, but there is not one or only one type of bus.

Memory 4003 may be read-only memory (ROM) or another type of static storage device capable of storing static information and instructions, random access memory (RAM), or other type of dynamic storage device capable of storing information and instructions; , EEPROM, CD-ROM or other optical disk storage, optical disk storage (including compressed optical disks, laser disks, optical disks, digital versatile disks, Blu-ray disks, etc.), disk storage media, other magnetic storage devices or computer programs. It may be any other medium that can be used to store or read on a computer, but is not limited herein.

Memory 4003 is used to store computer programs for executing embodiments of the present application and is controlled by processor 4001. The processor 4001 is configured to execute a computer program stored in the memory 4003 to implement the steps shown in the above-described method embodiment.

In the method provided by one embodiment, at least one module among a plurality of modules may be implemented through an AI model. AI-related functions can be performed by non-volatile memory, volatile memory, and processors.

The processor may include one or more processors. At this time, the one or more processors may be a general-purpose processor (e.g., central processing unit (CPU), application processor (AP), etc.) or a pure graphics processing unit (e.g., graphics processing unit (GPU), visual processing unit (VPU)), and /Or it may be an AI-specific processor (e.g., neural processing unit (NPU)).

One or more processors control the processing of input data according to predefined operation rules or artificial intelligence (AI) models stored in non-volatile memory and volatile memory. Provides predefined action rules or artificial intelligence models through training or learning.

Here, provision by learning means applying a learning algorithm to a plurality of learning data to obtain an AI model with predefined operation rules or desired characteristics. This learning may be performed on the device itself where AI according to the embodiment is performed, and/or may be implemented by a separate server/system.

An AI model may consist of multiple neural network layers. Each layer has multiple weight values, and the calculation of one layer is performed based on the calculation results of the previous layer and multiple weights of the current layer. Examples of neural networks include convolutional neural networks (CNN), deep neural networks (DNN), recurrent neural networks (RNN), restricted Boltzmann machines (RBM), deep belief networks (DBN), bidirectional recurrent deep neural networks (BRDNN), and generative correspondence networks ( GAN) and deep Q networks.

A learning algorithm is a method of inducing, allowing, or controlling a target device (e.g., a robot) to determine or predict the target device by training it using a plurality of learning data. Examples of learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

Embodiments of the present application provide a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the steps and corresponding contents of the above-described method embodiments can be implemented.

Embodiments of the present application also provide a computer program product including a computer program capable of implementing the steps and corresponding contents of the above-described method embodiments when executed by a processor.

The terms "first", "second", "third", "fourth", "1", "2", etc. (if any) used in the specification and claims of this application as well as in the attached drawings are It is used to distinguish similar objects without describing a specific order or sequence. It should be understood that the embodiments of the present application described herein may be implemented in an order other than the drawing examples or text descriptions, since data used in this manner may be exchanged where appropriate.

The modules described in the embodiments of this application may be implemented through software or neural networks. In some cases, the module name does not constitute a limitation of the module itself; for example, a self-attention module is also described as "module for self-attention processing", "first module", self-attention network, self-attention neural network, etc. It can be.

The embodiments described above may be implemented with hardware components, software components, and/or a combination of hardware components and software components. For example, the devices, methods, and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, and a field programmable gate (FPGA). It may be implemented using a general-purpose computer or a special-purpose computer, such as an array, programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and software applications running on the operating system. Additionally, a processing device may access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, a single processing device may be described as being used; however, those skilled in the art will understand that a processing device includes multiple processing elements and/or multiple types of processing elements. It can be seen that it may include. For example, a processing device may include multiple processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are possible.

Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing unit to operate as desired, or may be processed independently or collectively. You can command the device. Software and/or data may be used on any type of machine, component, physical device, virtual equipment, computer storage medium or device to be interpreted by or to provide instructions or data to a processing device. , or may be permanently or temporarily embodied in a transmitted signal wave. Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on a computer-readable recording medium.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. A computer-readable medium may store program instructions, data files, data structures, etc., singly or in combination, and the program instructions recorded on the medium may be specially designed and constructed for the embodiment or may be known and available to those skilled in the art of computer software. there is. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Includes optical media (magneto-optical media) and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

The hardware devices described above may be configured to operate as one or multiple software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with limited drawings, those skilled in the art can apply various technical modifications and variations based on this. For example, the described techniques are performed in a different order than the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or other components are used. Alternatively, appropriate results may be achieved even if substituted or substituted by an equivalent.

Therefore, other implementations, other embodiments, and equivalents of the claims also fall within the scope of the claims described below.

Claims (15)

In an image processing method using a panoramic segmentation model performed by a processor,
Obtaining image features of an image including a plurality of image frames;
determining a target object representation of the image based on the image features using a neural network; and
Determining a panoramic segmentation result of the image based on the target object representation
Including, an image processing method.
According to paragraph 1,
The step of determining a target object representation of the image based on the image features using the neural network includes:
Determining a target object representation of the image by performing a plurality of iterative processing on the image features using the neural network,
Image processing method.
According to paragraph 2,
Determining the target object representation of the image by performing the plurality of iterative processing on the image features using the neural network includes:
Using the neural network, performing iterative processing based on the image features and the object representation by a previous iteration of the image to determine an object representation by the current iteration of the image,
Image processing method.
According to paragraph 3,
In the case of the first repetition process among the plurality of repetition processes, the object expression by the previous repetition process is a pre-configured initial object expression,
Image processing method.
According to paragraph 3,
The step of performing the iterative processing based on the image features and the object representation by the previous iterative process of the image to determine the object representation by the current iterative process of the image,
Obtaining a mask by converting the object representation obtained from previous repeated processing of the image;
obtaining a first object representation by processing the image features, the object representation by the previous iterative processing, and the mask; and
Determining an object representation by current iteration processing of the image based on the first object representation
Including,
Image processing method.
According to clause 5,
Obtaining the first object representation by processing the image features, the object representation by the previous iterative processing, and the mask,
Obtaining an object representation related to the mask by performing attention processing on the image features, the object representation by the previous iteration process, and the mask; and
Obtaining a first object representation by performing self-attention processing and classification processing based on the object representation related to the mask and the object representation by the previous iteration process.
Including,
Image processing method.
According to clause 6,
Obtaining an object representation related to the mask by performing the attention processing on the image features, the object representation by the previous iteration process, and the mask,
Obtaining a second object representation based on key features corresponding to the image features, the object representation by the previous iterative process, and the mask;
determining a first probability representing an object category included in the image, based on the second object representation; and
Obtaining an object representation associated with the mask based on the first probability, a value feature corresponding to the image feature, and the image feature.
Including,
Image processing method.
According to clause 5,
The step of determining an object representation by current repetitive processing of the image based on the first object representation,
determining an object representation corresponding to each image frame among at least one image frame based on the image features and the first object representation; and
Determining an object representation by current iterative processing of the image based on the first object representation and the object representation corresponding to each determined image frame.
Including,
Image processing method.
According to clause 8,
Determining an object representation corresponding to each image frame of the at least one image frame based on the image features and the first object representation includes:
determining a fourth object representation based on the first object representation and a key feature corresponding to the image feature;
determining a second probability representing the object category included in the image based on the fourth object representation; and
determining an object representation corresponding to each image frame of the at least one image frame based on the second probability and a value feature corresponding to the image feature.
Including,
Image processing method.
According to clause 8,
The step of determining an object representation by current iterative processing of the image based on the first object representation and the object representation corresponding to each determined image frame,
performing classification processing and self-attention processing on the object expression corresponding to each determined image frame to obtain a third object expression corresponding to the image; and
Determining an object representation by current iteration processing of the image based on the first object representation and the third object representation
Including,
Image processing method.
According to paragraph 1,
The step of determining a panoramic segmentation result of the image based on the target object representation,
performing linear transformation processing on the target object representation; and
Determining mask information of the image based on the linearly transformed target object expression and the image features, and determining category information of the image based on the linearly transformed target object expression.
Including,
Image processing method.
In a method of training a panoramic segmentation model, performed by a processor,
The panoramic segmentation model includes a first module and a second module,
Obtaining training data, wherein the training data includes a training image, a first image feature of the training image, and a sample panorama segmentation result corresponding to the training image;
Obtaining a second image feature by changing the frame order of the first image feature;
determining, through the first module, a first predicted object representation and a second predicted object representation of the training image based on the first image feature and the second image feature, respectively;
determining, through the second module, a first prediction result and a second prediction result of the training image based on the first prediction object representation and the second prediction object representation, respectively; and
Training the panorama segmentation model using a target loss function based on the sample panorama segmentation result, the first prediction object representation, the second prediction object representation, the first prediction result, and the second prediction result.
Including,
Training methods.
According to clause 12,
Training the panorama segmentation model using a target loss function based on the sample panorama segmentation result, the first prediction object representation, the second prediction object representation, the first prediction result, and the second prediction result, comprising:
determining a first similarity matrix based on the first predicted object representation and the second predicted object representation;
determining a second similarity matrix based on the sample panorama segmentation result, the first prediction result, and the second prediction result; and
When the target loss function is determined to be minimum based on the first similarity matrix and the second similarity matrix, outputting a trained panorama segmentation model.
Including,
Training method.
In electronic devices,
Memory;
processor; and
A computer program stored in memory;
Including,
The electronic device, wherein the processor executes the computer program to implement the steps of the method according to any one of claims 1 to 13.
In a computer-readable storage medium storing a computer program,
A computer-readable storage medium, wherein the computer program implements the steps of the method according to any one of claims 1 to 13 when executed by a processor.