JP2023004894A - Image processing device, image processing method, apparatus readable storage medium - Google Patents

Abstract

To provide an image processing device, a method and an apparatus readable storage medium.SOLUTION: A device includes: a first acquisition unit in which a first segmentation model acquires a pseudo label for a subject of a basic class in an image; a second acquisition unit for acquiring a new label for a subject of a new class in the image; a processing unit for, based on the pseudo label and the new label, acquiring a second segmentation model for the subjects of the basic class and the new class; a first calculation unit in which the first segmentation model calculates a first dataset of a first parameter associated with the subject of the basic class in the image; a second calculation unit in which the second segmentation model calculates a second dataset of the first parameter associated with the subject of the basic class in the image; and a comparison unit for comparing the first dataset and the second dataset, and providing the second calculation unit with comparison information so as to correct the second segmentation model.SELECTED DRAWING: Figure 1

Description

TECHNICAL FIELD The present disclosure relates to the technical field of image processing, and more particularly to an image processing apparatus, image processing method, and machine-readable storage medium for incremental learning.

This section provides background information related to the present disclosure, but is not necessarily prior art.

Humans have the capacity to acquire, adjust, and transmit knowledge throughout their lives. The learning of new knowledge has a catastrophic effect on human learned knowledge. This learning ability is referred to as the ability of incremental learning. In the field of machine learning, incremental learning aims to solve a common drawback of catastrophic forgetting in model training. In other words, when general machine learning models (especially backpropagation-based deep learning methods) are trained on new tasks, they often perform significantly worse on old tasks. To overcome catastrophic forgetting, the model exhibits the ability to integrate new knowledge and extract existing knowledge from new data (plasticity), while avoiding apparent interference of new inputs with existing knowledge. It is expected to demonstrate competence (stability). These two conflicting requirements constitute the so-called "Stability-Plasticity Dilemma".

The simplest way to solve catastrophic forgetting is to retrain the network parameters using all known data to adapt to changes in data distribution over time. The method of training the model from scratch can completely solve the problem of catastrophic forgetting, but the efficiency of the method is very low, which greatly prevents the model from learning new data in real time. The main goal of incremental learning is to find the most effective balance point in the stability-plasticity dilemma under the condition of limited computing and storage resources.

At present, incremental learning may be broadly divided into three methods: rule-based learning, rehearsal-based learning, and bias correction-based learning. In the fields of semantic segmentation and instance segmentation, there is little about incremental learning methods.

This section provides a general overview of the disclosure and does not fully disclose its full scope or all of its features.

An object of the present disclosure is to provide an image processing apparatus, an image processing method, and a machine-readable storage medium for incremental learning of object segmentation. The present disclosure solves the incremental learning problem of object segmentation from a new perspective.

In one aspect of the present disclosure, an image processing apparatus, a first obtaining unit for obtaining pseudo labels associated with an object segmentation of a base class object in an image according to a first segmentation model, the first is used for segmentation of objects of a base class, and a second acquisition unit for obtaining new labels associated with the segmentation of objects of a new class different from the base class in the image. and a processing unit for obtaining a second segmentation model for segmentation of objects of the base class and the new class based on the pseudo-label and the new label, wherein the first segmentation model and said second segmentation model is implemented by a neural network, said image processing device generating a first data set of first parameters associated with said base class of objects in said image by said first segmentation model and a second computation unit that computes a second data set of first parameters associated with the base class objects in the image according to the second segmentation model; a comparison unit that compares the first data set and the second data set and provides comparison information to the second calculation unit to modify the second segmentation model. Provide equipment.

In another aspect of the present disclosure, an image processing method, the step of obtaining pseudo-labels associated with an object segmentation of a base class of objects in an image by a first segmentation model, comprising: is used for segmentation of objects of a base class; obtaining new labels associated with the segmentation of objects of a new class of objects in the image different from the base class; obtaining a second segmentation model for segmentation of objects of the base class and the new class based on The image processing method is implemented, comprising the steps of calculating a first data set of first parameters associated with objects of the base class in the image according to the first segmentation model; calculating a second data set of first parameters associated with objects of the base class in the image; comparing the first data set and the second data set; modifying the segmentation model of .

In another aspect of the present disclosure, a machine-readable storage medium recording a program product storing machine-readable instruction code, wherein when the instruction code is read and executed by a computer, , a storage medium capable of causing the computer to execute the image processing method of the present disclosure.

The image processing apparatus, image processing method, and machine-readable storage medium according to the present disclosure can suitably implement incremental learning of instance segmentation and effectively prevent catastrophic forgetting of a training model.

With the description provided herein, the scope of applicability of the present disclosure will become clearer. The descriptions and specific examples in this section are for illustrative purposes only and are not intended to limit the scope of the disclosure.

The drawings described herein are intended to illustrate preferred embodiments, are not all possible embodiments, and are not intended to limit the scope of the disclosure.
1 is a block diagram showing the configuration of an image processing device according to an embodiment of the present disclosure; FIG. 1 is a block diagram illustrating the basic structure of a first segmentation model according to an embodiment of the present disclosure; FIG. FIG. 4 is a block diagram illustrating the basic structure of a second segmentation model according to an embodiment of the present disclosure; FIG. 2 is a block diagram showing detailed operations of a partial structure of an image processing apparatus according to an embodiment of the present disclosure; FIG. 4 is a schematic diagram illustrating calculated classification confidence according to an embodiment of the present disclosure; 2 is a schematic diagram showing internal computing details of an image processing device according to an embodiment of the present disclosure; FIG. 4 is a flowchart illustrating an image processing method according to an embodiment of the present disclosure; 1 is a block diagram showing an exemplary configuration of a general-purpose personal computer capable of implementing an image processing apparatus and method according to embodiments of the present disclosure; FIG. While various modifications and alternatives may be made to the disclosure, specific examples thereof will be described in detail with reference to the drawings. The description of specific embodiments does not limit the present disclosure to specific aspects of the disclosure, and various changes, equivalent modifications, and substitutions may be made within the spirit and scope of the present disclosure. good. In the drawings, the same components are denoted by the same reference numerals.

Exemplary embodiments of the present disclosure are described in detail below with reference to the drawings. The following description is merely exemplary and is not intended to limit the disclosure, applications and uses.

The following describes the present disclosure in detail and provides illustrative examples so that those skilled in the art can fully appreciate the scope of the present disclosure. Numerous specific details, such as examples of specific means, devices and methods, are set forth in order to provide a thorough understanding of the embodiments of the present disclosure. However, as will be appreciated by those skilled in the art, the exemplary embodiments may be implemented using different schemes without the need for specific details, and these embodiments do not limit the scope of the present disclosure. . In some example embodiments, well-known processes, well-known configurations, and well-known techniques have not been described in detail.

In this disclosure, it is assumed that the scenes (image backgrounds) of the base and new classes are similar. A good instance segmentation model (hereinafter also referred to as the first segmentation model) was trained on the original image data containing the base class of objects. When a new class appears in the image data, the new data contains instances of the original base class and the new class. The original instance segmentation model (first segmentation model) only has the ability to detect and segment base classes, whereas the present disclosure detects base classes and new classes with minimal cost. For the segmentation, we aim to train an instance segmentation model (hereinafter also referred to as the second segmentation model).

This disclosure proposes an innovative method that does not require source data and only needs to label partial data in new samples containing new classes. We obtain the pseudo-labels of the base class and the new labels of the new class from the original instance segmentation model, and combine these labels as teacher labels. Also, to prevent catastrophic forgetting of the trained model, we add another loss function, eg two distillation losses in the detection and classification branches. The method has a good effect on incremental learning of instance segmentation. A configuration of an image processing apparatus according to an embodiment of the present disclosure for achieving the above object will be described below with reference to FIG.

FIG. 1 is a block diagram showing the configuration of an image processing device according to an embodiment of the present disclosure. As shown in FIG. 1, the image processing apparatus 100 according to the embodiment of the present disclosure includes a first acquisition unit 110, a second acquisition unit 120, a processing unit 130, a first calculation unit 140, a second calculation unit 150 and a comparison unit 160 may be included.

The first obtaining unit 110 may obtain the pseudo-labels associated with the object segmentation of the base class objects in the image according to the first segmentation model. A first segmentation model is implemented by a neural network and is used for segmentation of a base class of objects. For example, the first segmentation model may be a yolact++ instance segmentation model. Here, a pseudo-label may be understood as a labeling or label information of objects of a base class.

The second obtaining unit 120 may also obtain a new label associated with the target segmentation of the target of a new class different from the base class in the image. Here, a new label may be understood as an annotation or labeling information of a new class of objects different from the base class in the image. The new labels may be obtained by manual labeling, for example by manually labeling the classification identifiers, bounding boxes and masks of new classes of objects in the image. The class identifier represents the class to which the new class object belongs, the bounding box may represent the location of the object, and the mask represents the mask of the object. Accordingly, the base class object labeling information (pseudo-labels) obtained by the first obtaining unit 110 is, for example, classification confidence, anchor box offset and mask. Here, the offset of the anchor box represents the position of the bounding box, ie the position of the object.

Note that in this disclosure, an image is an image of a particular class of objects (human, cat, dog, bottle, etc.), and objects of this class may include objects of different subclasses. Thus, a base class is a generic term for the subclasses of objects of this class that can be segmented with an existing first segmentation model, and a new class of objects means objects of this class that are different from the base class. For example, the first existing segmentation model relates to bottles, where the base class may include plastic bottles, brown glass bottles, blue or green glass bottles, and the new class is clear glass bottles. may include

The processing unit 130 may also obtain a second segmentation model for segmentation of objects of the base class and the new class based on the pseudo-label and the new label. Note that the second segmentation model is implemented by a neural network and used for instance segmentation of images containing objects of base and novel classes. For example, the second segmentation model may be a yolact++ instance segmentation model. Also, the operation of the processing unit 130 corresponds to the training process of incremental learning of the second segmentation model.

By using the pseudo-labels of the base class and the novel labels of the newly labeled novel class obtained by the first segmentation model, and combining these labels as teacher labels, the target A second segmentation model can be obtained that can segment .

The first computation unit 140 may also compute a first data set of first parameters associated with the base class objects in the image according to the first segmentation model. The second computing unit 150 may also compute a second data set of first parameters associated with the base class objects in the image according to the second segmentation model obtained by the processing unit 130 . A first parameter associated with the base class object in the image may be, for example, the offset of the anchor box.

The comparison unit 160 may also compare the first data set and the second data set and provide comparison information to the second calculation unit 150 to modify the second segmentation model. The second computing unit 150 may continuously optimize the second segmentation model based on the comparison information. Preferably, the second computation unit 150 modifies the second segmentation model to minimize the difference between the first data set and the second data set.

Thereby, by further feeding back the comparison information of the parameters related to the objects of the base class and optimizing the second segmentation model based on the comparison information, the image processing apparatus 100 according to the embodiment of the present disclosure can: Incremental learning of instance segmentation can be favorably realized, and catastrophic forgetting of the training model can be effectively prevented.

The following further describes the first segmentation model and the second segmentation model with reference to FIGS. Since the functions of the first acquisition unit 110 and the first calculation unit 140 are realized by the first segmentation model, the configuration of the first segmentation model in FIG. It corresponds to the configuration of the unit 140 .

FIG. 2 is a block diagram illustrating the basic structure of a first segmentation model according to an embodiment of the disclosure. As shown in FIG. 2 , the first segmentation model 200 may include a feature extractor 210 , a predictor 220 , a mask template acquirer 230 and a post-processor 240 .

The feature extractor 210 may acquire feature information about the input image. For example, the feature information is a feature map. Also, the feature extraction unit 210 may provide feature information to the prediction unit 220 and the mask template acquisition unit 230 .

Based on the feature information, the prediction unit 220 may obtain prediction information about the input image, such as mask coefficients, bounding box or anchor box offsets, classification reliability, and the like. Classification confidence may represent the probability that an object in an input image belongs to a particular class. For example, for an input image containing multiple bottles, the probability that an object belongs to a particular class may refer to the probability that an object belongs to a particular class among multiple classes of bottles.

The mask template acquisition unit 230 may acquire the mask template based on the feature information. A mask template may be a mask base vector. The original mask of the input image may be calculated from the mask base vector and the mask coefficients obtained by the predictor 220 . For example, the original mask of the input image may be computed by multiplying the mask base vector by the mask coefficients.

The post-processing unit 240 may obtain the mask of the base class objects in the image and obtain the pseudo-labels of the base class objects based on the prediction information and the mask template.

FIG. 3 is a block diagram illustrating the basic structure of a second segmentation model according to an embodiment of the disclosure. Note that FIG. 3 shows the overall structure of the second segmentation model. As shown in FIG. 3 , the second segmentation model 300 may include a feature extractor 310 , a predictor 320 , a mask template acquirer 330 , a post-processor 340 and a restrictor 350 . The operations of the feature extraction unit 310, the prediction unit 320, the mask template acquisition unit 330, and the post-processing unit 340 are the same as the operations of the feature extraction unit 210, the prediction unit 220, the mask template acquisition unit 230, and the post-processing unit 240 in FIG. be.

Note that the post-processing unit 340 is not required during the training phase of the second segmentation model. In order for the processing unit 130 to obtain the second segmentation model for segmentation of the target of the base class and the new class based on the first segmentation model, the configuration of the processing unit 130 is similar to that of the second segmentation model in FIG. This corresponds to a configuration in which the post-processing section 340 is not included.

Also, after the second segmentation model is trained, the second segmentation model may not include restriction 350 . In this case, the configuration of the second segmentation model is similar to that of the first segmentation model. The configuration of the second calculation unit 150 corresponds to the configuration of the second segmentation model in this case.

The basic operation of each part of the second segmentation model 300 is described below. Here, the input of the second segmentation model includes both pseudo-labels of the base class and novel labels of the novel class.

The feature extractor 310 may acquire feature information about the input image. The prediction section 320 may acquire prediction information about the input image based on the feature information, and the mask template acquisition section 330 may acquire the mask template based on the feature information.

The restriction unit 350 may also perform incremental training according to target constraints in the neural network based on the feature information and the mask template to obtain a second segmentation model.

After the second segmentation model is trained, the post-processing unit 340 computes base class objects and At the same time that a mask of objects for a new class is obtained, labeling information for the corresponding objects may be obtained.

For a better understanding of the techniques of the present disclosure, the operations of the first calculation unit 140, the second calculation unit 150 and the comparison unit 160 in FIG. 1 are described in detail below with reference to FIG.

FIG. 4 is a block diagram illustrating detailed operation of a partial structure of an image processing apparatus according to an embodiment of the present disclosure. In the image processing apparatus 400 of FIG. 4, the first acquisition unit, second acquisition unit, and processing unit described in detail with reference to FIGS. 1 to 3 above are omitted. The image processing device 400 further includes a first calculator 440 , a second calculator 450 and a comparator 460 .

(1) Example of adding distillation loss to the detection branch of the segmentation model The description of the unit 150 and the comparison unit 160 is the same. Here, the first parameter may be the offset of the anchor box, ie the position of the bounding box. The first parameter data set is the result of calculating the offset of the anchor box. In this example, the MSE (Mean Squared Error) loss is added to the bounding box regression layer of the detection branch.

Specifically, the first computation unit 440 may compute the offset [x, y, x_offset, y_offset] of the anchor box associated with the base class object in the image according to the first segmentation model.

The second computation unit 450 may compute offsets [x, y, x_offset, y_offset] of anchor boxes associated with base class objects in the image according to a second segmentation model provided by the processing unit.

Also, the comparison unit 460 may compare the anchor box offsets calculated by the first calculation unit 440 and the second calculation unit 450, respectively, and calculate, for example, the MSE loss of both.

The second computation unit 450 also modifies the second segmentation model based on the MSE loss of the anchor box offsets. Preferably, the second computation unit 450 modifies the second segmentation model by minimizing the difference between the first data set and the second data set.

(2) Example of adding distillation loss to the classification branch of the segmentation model Preferably, the first computation unit 440 and the second computation unit 450 further compute other parameters related to the base class objects in the image. Alternatively, the comparison unit 460 may further modify the second segmentation model by comparing other parameters. For example, another parameter (second parameter) may be the confidence of the classification. The second parameter data set is the calculated result of the classification confidence. In this example, the KLDiv (Kullback-Leibler divergence: KL divergence) loss is added to the logs layer of the classification branch.

Specifically, the first computation unit 440 may compute the confidence of the classification associated with the base class object in the image according to the first segmentation model.

The second computing unit 450 may also compute the confidence of the classification associated with the base class object in the image according to the second segmentation model provided by the processing unit.

Further, the comparison unit 460 compares the reliability of the base class classification calculated by the first calculation unit 440 and the second calculation unit 450, for example, calculates the KLDiv loss of both, and calculates the second calculation unit At 450, classification confidence comparison information may be provided for further refinement of the second segmentation model. Similarly, the second segmentation model is modified by minimizing the difference between the base class classification confidences computed by the first computational unit 440 and the second computational unit 450 .

であり、ここで、ｎは、基本クラスの数を表す。図５の下部は、第２のセグメンテーションモデルにより計算された分類の信頼度の例［ｏ_１，ｏ_２，…，ｏ_ｎ，ｏ_ｎ＋１，…，ｏ_ｎ＋ｍ］を示し、ここで、前のｎ個は、ｎ個の基本クラスの分類の信頼度であり、後のｍ個は、ｍ個の新規クラスの分類の信頼度である。比較部４６０は、
（外２）

と［ｏ_１，ｏ_２，…，ｏ_ｎ］とを比較し、例えば両者の間のＫＬＤｉｖ損失を計算する。 FIG. 5 is a schematic diagram illustrating calculated classification confidences according to an embodiment of the present disclosure, showing examples of confidences calculated by a first segmentation model and a second segmentation model, respectively. As shown in FIG. 5, the top is an example of the classification confidence calculated by the first segmentation model (Extra 1)

where n represents the number of base classes. The lower part of FIG. 5 shows an example of classification confidences [o ₁ , o ₂ , . . . , o _n , o _{n +1} , . is the confidence of the classification of the n base classes, and the latter m is the confidence of the classification of the m new classes. The comparison unit 460
(outside 2)

and [ _{o 1} , o ₂ , .

The second computation unit 450 also further modifies the second segmentation model based on the KLDiv loss of classification confidence.

FIG. 6 is a schematic diagram illustrating internal computing details of an image processing apparatus according to an embodiment of the present disclosure. In the above process of calculating anchor box offsets and classification confidence losses, anchor box offsets and classification confidences corresponding to a plurality of anchor boxes calculated by the first segmentation model and the second segmentation model are calculated. to obtain a dataset of anchor box offsets and a dataset of classification confidences for comparison. Details are provided below.

Specifically, the second calculation unit 450 calculates the overlap between each of the plurality of anchor boxes in the second segmentation model and the object in the image, such as the Jaccard coefficient, using the second segmentation model. In addition, the second calculation unit 450 selects anchor boxes whose overlap degree is greater than a predetermined threshold and whose overlap target class is the base class.

Also, the second calculation unit 450, based on the index of the selected anchor, from the data set of the first parameter or the second parameter corresponding to the plurality of anchor boxes obtained by the second segmentation model, Select the data set that matches the index of the anchor box, eg, the data set of the matched anchor box offset and the matched classification confidence at the bottom of FIG.

The second calculator 450 also provides the index of the selected anchor box to the first calculator 440 .

Next, the first computation unit 440 computes a data set of first parameters or second parameters corresponding to the plurality of anchor boxes obtained by the first segmentation model, based on the indices of the selected anchor boxes. , select the dataset that matches the index of the anchor box, eg, the dataset of matched anchor box offsets at the top of FIG. 6 and the dataset of matched classification confidence.

The comparator 460 calculates the matched anchor box offsets selected by the first calculator 440 and the second calculator 450 and the matched classifications selected by the first calculator 440 and the second calculator 450 . You can compare the reliability of

As an example, the second segmentation model has 57744 anchor boxes. 600 to 700 anchor boxes can be selected by making the above multiplicity greater than 0.5. Only 400-500 anchor boxes among them belong to the base class. The indices of these 400-500 anchor boxes are recorded and used to filter the dataset of parameters corresponding to the anchor boxes calculated by the first segmentation model and the second segmentation model. A dataset of selected parameters is used to calculate the loss function and is used to modify the second segmentation model.

Accordingly, the image processing apparatus 100 according to the embodiment of the present disclosure can prevent catastrophic forgetting of the training model by adding two distillation losses (MSE loss, KLDiv loss).

An image processing method according to an embodiment of the present disclosure will be described below with reference to FIG.

As shown in FIG. 7, the image processing method according to embodiments of the present disclosure begins at step S110. In step S110, the pseudo-labels associated with the object segmentation of the base class objects in the image are obtained by the first segmentation model. A first segmentation model is used for segmentation of base class objects.

Next, in step S120, the new label associated with the object segmentation of the object of the new class different from the base class in the image is obtained.

Then, in step S130, obtain a second segmentation model for segmentation of the object of the base class and the new class based on the pseudo-label and the new label.

Next, in step S140, the first segmentation model computes a first data set of first parameters associated with the base class objects in the image.

Next, in step S150, a second data set of first parameters associated with the base class objects in the image is calculated by a second segmentation model.

Next, in step S160, the first data set and the second data set are compared to modify the second segmentation model. After this the process ends.

Embodiments of the present disclosure modify the second segmentation model to minimize the difference between the first data set and the second data set.

In an embodiment of the present disclosure, the method further includes calculating, with the second segmentation model, the degree of overlap between each of the plurality of anchor boxes in the second segmentation model and the object in the image. Calculating the first data set comprises selecting the first data set from the first parameter data sets corresponding to the plurality of anchor boxes obtained by the first segmentation model based on the degree of multiplicity. including. Calculating the second data set comprises selecting the second data set from the first parameter data set corresponding to the plurality of anchor boxes obtained by the second segmentation model based on the degree of multiplicity. including.

In an embodiment of the present disclosure, the second segmentation model selects anchor boxes whose degree of overlap is greater than a predetermined threshold and whose overlapping target class is the base class. The first segmentation model and the second segmentation model respectively select the first dataset and the second dataset based on the index of the selected anchor box.

In an embodiment of the present disclosure, the method comprises computing a third data set of second parameters associated with base class objects in the input image according to the first segmentation model; , computing a fourth dataset of second parameters associated with objects of the base class in the input image; comparing the third dataset with the fourth dataset to further generate a second segmentation model; and modifying.

Embodiments of the present disclosure modify the second segmentation model to minimize the difference between the third data set and the fourth data set.

In an embodiment of the present disclosure, the first parameter is the anchor box offset and the second parameter is the classification confidence. In this case, the MSE loss function is used to compare the first data set to the second data set, and the KLDiv loss function is used to compare the third data set to the fourth data set.

In embodiments of the present disclosure, the pseudo-labels and novel labels include the classification identifier, bounding box and mask of the corresponding object in the image. Also, the first segmentation model and the second segmentation model are yolact++ instance segmentation models.

Accordingly, the image processing method according to the embodiment of the present disclosure can realize class increment learning for instance segmentation only with the label of the new class, and solve the problem of catastrophic forgetting.

Each aspect of the above steps of the image processing method according to the embodiments of the present disclosure has already been described in detail, and the description thereof is omitted here.

Each process of the image processing method according to the present disclosure may be realized by a computer-executable program stored in a storage medium readable by various devices.

Another object of the present disclosure is to provide, directly or indirectly, a storage medium storing the above executable program code to a system or device, so that a computer or central processing unit (CPU) in the system or device executes the program code. may be implemented by reading and executing In this case, the system or device only needs to have a function capable of executing a program, and the embodiments of the present disclosure are not limited to programs. Also, the program may be in any form, such as a target program, a program executed by an interpreter, or a script program provided to the operating system.

The above machine-readable storage media include various memories, storage units, semiconductor devices, disks such as optical disks, magnetic disks and magneto-optical disks, and other media capable of storing information. Not limited.

Also, the embodiments of the present disclosure can be realized by connecting the computer to a corresponding website on the Internet and downloading, installing, and executing the computer program code of the present disclosure on the computer.

FIG. 8 is a block diagram showing an exemplary configuration of a general-purpose personal computer that can implement the image processing apparatus and method according to the embodiments of the present disclosure.

As shown in FIG. 8, the CPU 801 executes various processes using programs stored in a read only memory (ROM) 802 or programs loaded from a storage unit 808 to a random access memory (RAM) 803 . The RAM 803 stores data necessary for the CPU 801 to execute various processes as needed. The CPU 801 , ROM 802 and RAM 803 are interconnected via a bus 804 . Input/output interface 805 is also connected to bus 804 .

Input unit 806 (including keyboard, mouse, etc.), output unit 807 (display, such as cathode ray tube (CRT), liquid crystal display (LCD), etc., speaker, etc.), storage unit 808 (including hard disk, etc.), communication Unit 809 (eg, including network interface cards, such as LAN cards, modems, etc.) is connected to input/output interface 805 . A communication unit 809 executes communication processing via a network such as the Internet. A driver 810 may be connected to the input/output interface 805 as desired. The removable medium 811 is, for example, a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory. 808 installed.

When the above processing is performed by software, a program that constitutes the software is installed via a network such as the Internet or a storage medium such as the removable medium 811 .

Note that these storage media are not limited to the removable media 811 shown in FIG. 8 that stores the program and provides the program to the user separately from the device. Removable media 811 may be, for example, magnetic disks (including floppy disks), optical disks (including optical disks—read only memory (CD-ROM) and digital versatile disks (DVD)), magneto-optical disks (mini disk (MD) (registered trademark)) and semiconductor memory. Alternatively, the storage medium may be the ROM 802, the hard disk included in the storage unit 808, or the like, which stores the program and is provided to the user together with the device including them.

It should be noted that in the system and method of the present disclosure, each unit or each step may be disassembled and/or recombined. These disassembly and/or recombination are considered equivalents of this disclosure. Also, the methods of the present disclosure are not limited to performing in the chronological order set forth herein, and may be performed sequentially, in parallel, or independently in other chronological orders. As such, the order in which the methods described herein are performed should not limit the scope of this disclosure.

Although the embodiments of the present disclosure are described in detail above with reference to the drawings, the above-described embodiments and examples are merely illustrative and are not intended to limit the present disclosure. Those skilled in the art may make various modifications and changes to this disclosure within the spirit and scope of the claims. These modifications and changes fall within the protection scope of this disclosure.

In addition, the following additional remarks will be disclosed regarding the embodiments including the above-described examples.
(Appendix 1)
An image processing device,
A first acquisition unit for acquiring pseudo-labels associated with object segmentation of base class objects in an image by a first segmentation model, wherein the first segmentation model is used for segmentation of base class objects. , a first obtaining unit;
a second obtaining unit for obtaining a new label associated with an object segmentation of an object of a new class different from the base class in the image;
a processing unit for obtaining a second segmentation model for segmentation of objects of the base class and the new class based on the pseudo-label and the new label;
The first segmentation model and the second segmentation model are realized by a neural network,
The image processing device is
a first computation unit for computing a first data set of first parameters associated with the base class of objects in the image according to the first segmentation model;
a second computing unit that computes a second data set of first parameters associated with the base class objects in the image according to the second segmentation model;
a comparison unit that compares the first data set and the second data set and provides comparison information to the second computation unit to modify the second segmentation model. processing equipment.
(Appendix 2)
2. The image processing apparatus according to appendix 1, wherein the second calculation unit modifies the second segmentation model to minimize a difference between the first data set and the second data set.
(Appendix 3)
The second calculation unit calculates a degree of overlap between each of the plurality of anchor boxes in the second segmentation model and the target in the image, using the second segmentation model;
The first calculation unit selects the first data set from first parameter data sets corresponding to the plurality of anchor boxes obtained by the first segmentation model, based on the multiplicity. ,
The second calculation unit selects the second data set from first parameter data sets corresponding to the plurality of anchor boxes obtained by the second segmentation model, based on the degree of redundancy. , the image processing apparatus according to appendix 1.
(Appendix 4)
The second calculation unit selects an anchor box whose degree of overlap is greater than a predetermined threshold and whose overlapping target class is a base class;
3. The image processing according to attachment 3, wherein the first calculation unit and the second calculation unit select the first data set and the second data set, respectively, based on the index of the selected anchor box. Device.
(Appendix 5)
the first computation unit computes a third data set of second parameters associated with the base class objects in the image according to the first segmentation model;
the second computation unit computes a fourth data set of second parameters associated with the base class objects in the image according to the second segmentation model;
the comparison unit compares the third data set and the fourth data set and provides other comparison information to the second computation unit to further refine the second segmentation model; 5. The image processing apparatus according to any one of Appendices 1 to 4.
(Appendix 6)
6. The image processing device according to appendix 5, wherein the second calculation unit modifies the second segmentation model to minimize a difference between the third data set and the fourth data set.
(Appendix 7)
the first parameter is the offset of the anchor box;
6. The image processing device according to appendix 5, wherein the second parameter is the reliability of classification.
(Appendix 8)
comparing the first data set and the second data set using an MSE loss function;
8. The image processing apparatus of claim 7, wherein the third data set and the fourth data set are compared using a KLDiv loss function.
(Appendix 9)
2. The image processing apparatus of claim 1, wherein the pseudo-labels and the new labels include classification identifiers, bounding boxes and masks of corresponding objects in the images.
(Appendix 10)
An image processing method comprising:
obtaining pseudo-labels associated with object segmentation of base class objects in an image by a first segmentation model, wherein the first segmentation model is used for segmentation of base class objects;
obtaining new labels associated with object segmentations of objects of a new class different from the base class in the image;
obtaining a second segmentation model for segmentation of objects of the base class and the new class based on the pseudo-label and the new label;
The first segmentation model and the second segmentation model are realized by a neural network,
The image processing method includes
calculating a first data set of first parameters associated with the base class of objects in the image with the first segmentation model;
calculating a second data set of first parameters associated with the base class of objects in the image according to the second segmentation model;
comparing the first data set and the second data set and modifying the second segmentation model.
(Appendix 11)
11. The image processing method of clause 10, wherein the second segmentation model is modified to minimize differences between the first data set and the second data set.
(Appendix 12)
calculating, with the second segmentation model, the degree of overlap between each of a plurality of anchor boxes in the second segmentation model and the object in the image;
The step of calculating the first data set includes calculating the first data from a first parameter data set corresponding to the plurality of anchor boxes obtained by the first segmentation model, based on the multiplicity. selecting a set;
The step of calculating the second data set includes calculating the second data from a first parameter data set corresponding to the plurality of anchor boxes obtained by the second segmentation model, based on the multiplicity. 11. The image processing method of clause 10, comprising selecting a set.
(Appendix 13)
The second segmentation model selects anchor boxes whose degree of overlap is greater than a predetermined threshold and whose overlapping target class is a base class;
13. The image processing of clause 12, wherein the first segmentation model and the second segmentation model select the first dataset and the second dataset, respectively, based on indices of selected anchor boxes. Method.
(Appendix 14)
calculating a third data set of second parameters associated with the base class of objects in the input image with the first segmentation model;
calculating a fourth data set of second parameters associated with the base class of objects in the input image with the second segmentation model;
14. The image processing method according to any one of Appendices 10 to 13, further comprising comparing the third data set and the fourth data set and further modifying the second segmentation model.
(Appendix 15)
15. The image processing method of clause 14, wherein the second segmentation model is modified to minimize differences between the third data set and the fourth data set.
(Appendix 16)
the first parameter is the offset of the anchor box;
15. The image processing method according to appendix 14, wherein the second parameter is a classification confidence.
(Appendix 17)
comparing the first data set and the second data set using an MSE loss function;
17. The image processing method of clause 16, wherein a KLDiv loss function is used to compare the third data set and the fourth data set.
(Appendix 18)
11. The image processing method of clause 10, wherein the pseudo-labels and the new labels comprise classification identifiers, bounding boxes and masks of corresponding objects in the images.
(Appendix 19)
11. The image processing method of claim 10, wherein the first segmentation model and the second segmentation model are yolact++ instance segmentation models.
(Appendix 20)
A machine-readable storage medium in which a program product storing machine-readable instruction code is recorded, wherein when the instruction code is read and executed by the computer, the computer can read the instructions 10 to 19. A storage medium capable of executing any of the image processing methods described above.

Claims (10)

An image processing device,
A first acquisition unit for acquiring pseudo-labels associated with object segmentation of base class objects in an image by a first segmentation model, wherein the first segmentation model is used for segmentation of base class objects. , a first obtaining unit;
a second obtaining unit for obtaining a new label associated with an object segmentation of an object of a new class different from the base class in the image;
a processing unit for obtaining a second segmentation model for segmentation of objects of the base class and the new class based on the pseudo-label and the new label;
The first segmentation model and the second segmentation model are realized by a neural network,
The image processing device is
a first computation unit for computing a first data set of first parameters associated with the base class of objects in the image according to the first segmentation model;
a second computing unit that computes a second data set of first parameters associated with the base class objects in the image according to the second segmentation model;
a comparison unit that compares the first data set and the second data set and provides comparison information to the second computation unit to modify the second segmentation model. processing equipment. 2. The image processing apparatus according to claim 1, wherein said second calculation unit modifies said second segmentation model so as to minimize a difference between said first data set and said second data set. The second calculation unit calculates a degree of overlap between each of the plurality of anchor boxes in the second segmentation model and the target in the image, using the second segmentation model;
The first calculation unit selects the first data set from first parameter data sets corresponding to the plurality of anchor boxes obtained by the first segmentation model, based on the multiplicity. ,
The second calculation unit selects the second data set from first parameter data sets corresponding to the plurality of anchor boxes obtained by the second segmentation model, based on the degree of redundancy. 2. The image processing apparatus according to claim 1. The second calculation unit selects an anchor box whose degree of overlap is greater than a predetermined threshold and whose overlapping target class is a base class;
4. The image of claim 3, wherein the first computation unit and the second computation unit select the first data set and the second data set, respectively, based on the index of the selected anchor box. processing equipment. the first computation unit computes a third data set of second parameters associated with the base class objects in the image according to the first segmentation model;
the second computation unit computes a fourth data set of second parameters associated with the base class objects in the image according to the second segmentation model;
the comparison unit compares the third data set and the fourth data set and provides other comparison information to the second computation unit to further refine the second segmentation model; 5. The image processing apparatus according to claim 1. 6. The image processing apparatus according to claim 5, wherein said second calculation unit modifies said second segmentation model so as to minimize a difference between said third data set and said fourth data set. the first parameter is the offset of the anchor box;
6. The image processing apparatus according to claim 5, wherein said second parameter is a reliability of classification. comparing the first data set and the second data set using an MSE loss function;
8. The image processing apparatus of claim 7, wherein a KLDiv loss function is used to compare the third data set and the fourth data set. An image processing method comprising:
obtaining pseudo-labels associated with object segmentation of base class objects in an image by a first segmentation model, wherein the first segmentation model is used for segmentation of base class objects;
obtaining new labels associated with object segmentations of objects of a new class different from the base class in the image;
obtaining a second segmentation model for segmentation of objects of the base class and the new class based on the pseudo-label and the new label;
The first segmentation model and the second segmentation model are realized by a neural network,
The image processing method includes
calculating a first data set of first parameters associated with the base class of objects in the image with the first segmentation model;
calculating a second data set of first parameters associated with the base class of objects in the image according to the second segmentation model;
comparing the first data set and the second data set and modifying the second segmentation model. 10. A machine-readable storage medium in which a program product storing machine-readable instruction code is recorded, wherein when the instruction code is read and executed by the computer, the computer reads and executes the machine-readable instruction code. A storage medium capable of executing the image processing method of

Images