CN113515971A - Data processing method and system, network system and training method and device thereof - Google Patents

Abstract

The embodiment of the application provides a data processing method and system, a network system and a training method and device thereof. Wherein the method comprises the following steps: acquiring a first image and a second image; performing difference analysis on the first image and the second image to obtain an output result containing difference information; classifying the image content in the first image to obtain a first classification result; classifying the image content in the second image to obtain a second classification result; and determining the difference between the first image and the second image according to the output result, the first classification result and the second classification result. The difference determined by the technical scheme provided by the embodiment of the application not only reflects the place of the difference, but also reflects which types of image contents are changed, so that the output result information is richer, and the detection effect of change detection is improved.

Description

Data processing method and system, network system and training method and device thereof

Technical Field

The application relates to the cross field of computer technology and remote sensing technology, in particular to a data processing method and system, a network system and a training method and device thereof.

Background

In recent years, the difference comparison of two images to obtain information desired by a user from the difference has been performed in many fields, for example, the comparison field of two remote sensing images. The two-stage remote sensing image is as follows: aerial or satellite photographs acquired of the same area at two different times. In the task of comparing the two-stage remote sensing images, the change detection of the target area is the most common one, and the change detection of the remote sensing images not only comprises the detection of newly added buildings, but also comprises the detection of building demolition, construction of agricultural land, forest land and soil movement, construction of water area into land, construction of land into water area, pavement hardening of existing construction roads and the like.

However, the existing two-stage remote sensing image change detection technology can only know which block of the target area is changed and which block is not changed, and the output result is single.

Disclosure of Invention

In view of the above, the present application is proposed to provide a data processing method and system, a network system, and a training method and device thereof, which solve the above problems or at least partially solve the above problems.

Thus, in one embodiment of the present application, a data processing method is provided. The method comprises the following steps:

acquiring a first image and a second image;

performing difference analysis on the first image and the second image to obtain an output result containing difference information;

classifying the image content in the first image to obtain a first classification result;

classifying the image content in the second image to obtain a second classification result;

and determining the difference between the first image and the second image according to the output result, the first classification result and the second classification result.

In another embodiment of the present application, a data processing method is provided. The method comprises the following steps:

acquiring a first image and a second image;

performing difference analysis on the first image and the second image to obtain an output result containing difference information;

and classifying the output result to obtain the difference between the first image and the second image.

In yet another embodiment of the present application, a data processing method is provided. The method comprises the following steps:

acquiring a first image and a second image;

performing feature extraction on the first image to obtain at least one piece of first feature information;

performing feature extraction on the second image to obtain at least one piece of second feature information;

and determining the difference reflecting the first image and the second image and the category of the content of the difference image according to the at least one first characteristic information and the at least one second characteristic information.

In yet another embodiment of the present application, a data processing method is provided. The method comprises the following steps:

responding to an image comparison event triggered by a user through an interactive interface, and acquiring a first image and a second image designated by the user in the image comparison event;

identifying the difference between the first image and the second image and the category of the content of the difference image to obtain an identification result;

generating a third image which reflects the difference between the first image and the second image and shows the content of different types of images in the difference in a distinguishing way based on the identification result;

and displaying the third image.

In another embodiment of the present application, a network system is provided. The network system comprises

The first network model comprises at least one first network layer and is used for extracting the characteristics of the first image and outputting at least one piece of first characteristic information;

the second network model comprises at least one second network layer and is used for extracting the characteristics of the second image and outputting at least one piece of second characteristic information;

the third network model is connected with the output ends of the first network model and the second network model and used for performing difference analysis on the at least one first characteristic information and the at least one second characteristic information to obtain an output result containing difference information;

the fourth network model is connected with the first network model and used for classifying the at least one first characteristic message to obtain a first classification result;

the fifth network model is connected with the second network model and used for classifying the at least one second characteristic message to obtain a second classification result;

and the output model is connected with the third network model, the fourth network model and the fifth network model and is used for outputting the difference between the first image and the second image according to the output result, the first classification result and the second classification result.

In another embodiment of the present application, a network training method is provided. The network training method comprises the following steps:

performing feature extraction on the first sample image by using a first network model to obtain at least one piece of first sample feature information;

performing feature extraction on the second sample image by using a second network model to obtain at least one piece of second sample feature information;

performing difference analysis on the at least one first sample characteristic information and the at least one second sample characteristic information by using a third network model to obtain an output result;

optimizing network parameters of at least one of the first network model, the second network model and the third network model according to the output result and the first sample label;

analyzing the at least one first sample characteristic information by using a fourth network model to obtain a first classification result;

optimizing network parameters of at least one of the first network model and the fourth network model according to the first classification result and a second sample label;

wherein the first sample label, second sample label, first sample image and second sample image are associated.

In another embodiment of the present application, a network system is provided. The network system includes:

the first network model comprises at least one first network layer and is used for extracting the characteristics of the first image and outputting at least one piece of first characteristic information;

the second network model comprises at least one second network layer and is used for carrying out feature extraction on the second image to obtain at least one second feature information;

the third network model is connected with the output ends of the first network model and the second network model and used for carrying out difference analysis on at least one piece of first characteristic information and at least one piece of second characteristic information to obtain an output result;

and the multi-classification model is connected with the third network model and used for carrying out multi-class classification on the output result to obtain the difference between the first image and the second image.

In another embodiment of the present application, a network training method is provided. The network training method comprises the following steps:

performing feature extraction on the first sample image by using a first network model to obtain at least one piece of first sample feature information;

performing feature extraction on the second sample image by using a second network model to obtain at least one piece of second sample feature information;

analyzing the at least one first sample characteristic information and the at least one second sample characteristic information by using a third network model to obtain an output result;

performing multi-class classification on the output result by using a multi-classification model to obtain multi-class classification results;

optimizing network parameters of at least one of the first network model, the second network model, the third network model and the multi-classification model according to the multi-class classification result and the multi-class sample label;

wherein the multi-class sample label, the first sample image, and the second sample image are associated.

In another embodiment of the present application, a data processing system is provided. The system comprises:

the image acquisition equipment is used for acquiring a first image and a second image of a target area corresponding to two different moments;

the processor is used for acquiring the first image and the second image, performing difference analysis on the first image and the second image and obtaining an output result containing difference information; classifying the image content in the first image to obtain a first classification result; classifying the image content in the second image to obtain a second classification result; and determining the difference between the first image and the second image according to the output result, the first classification result and the second classification result.

In another embodiment of the present application, a data processing system is provided. The system comprises:

the image acquisition equipment is used for acquiring a first image and a second image of a target area corresponding to two different moments;

the processor is used for acquiring the first image and the second image, performing difference analysis on the first image and the second image and obtaining an output result containing difference information; and classifying the output result to obtain the difference between the first image and the second image.

In another embodiment of the present application, a data processing system is provided. The system comprises:

the image acquisition equipment is used for acquiring a first image and a second image of a target area corresponding to two different moments;

the processor is used for acquiring the first image and the second image, and performing feature extraction on the first image to obtain at least one piece of first feature information; performing feature extraction on the second image to obtain at least one piece of second feature information; and determining the difference between the first image and the second image and the category of the content of the difference image according to the at least one piece of first characteristic information and the at least one piece of second characteristic information.

In another embodiment of the present application, a remote sensing device is provided. The remote sensing device includes:

the remote sensing device is used for acquiring a first remote sensing image and a second remote sensing image of a target area at two different moments;

the processor is used for acquiring the first remote sensing image and the second remote sensing image, and performing difference analysis on the first remote sensing image and the second remote sensing image to obtain an output result containing difference information; classifying the image content in the first remote sensing image to obtain a first classification result; classifying the image content in the second remote sensing image to obtain a second classification result; and determining the difference between the first remote sensing image and the second remote sensing image according to the output result, the first classification result and the second classification result.

In another embodiment of the present application, a remote sensing device is provided. The remote sensing device includes:

the remote sensing device is used for acquiring a first remote sensing image and a second remote sensing image of a target area at two different moments;

the processor is used for acquiring the first remote sensing image and the second remote sensing image, and performing difference analysis on the first remote sensing image and the second remote sensing image to obtain an output result containing difference information; and classifying the output result to obtain the difference between the first remote sensing image and the second remote sensing image.

In another embodiment of the present application, a remote sensing device is provided. The remote sensing device includes:

the remote sensing device is used for acquiring a first remote sensing image and a second remote sensing image of a target area at two different moments;

the processor is used for acquiring the first remote sensing image and the second remote sensing image, and extracting the characteristics of the first image to obtain at least one piece of first characteristic information; performing feature extraction on the second image to obtain at least one piece of second feature information; and determining the difference between the first image and the second image and the category of the content of the difference image according to the at least one piece of first characteristic information and the at least one piece of second characteristic information.

In another embodiment of the present application, an electronic device is provided. The electronic device includes: a memory and a processor, wherein,

the memory is used for storing programs;

the processor, coupled to the memory, configured to execute the program stored in the memory to:

acquiring a first image and a second image;

performing difference analysis on the first image and the second image to obtain an output result containing difference information;

classifying the image content in the first image to obtain a first classification result;

classifying the image content in the second image to obtain a second classification result;

and determining the difference between the first image and the second image according to the output result, the first classification result and the second classification result.

In another embodiment of the present application, an electronic device is provided. The electronic device includes: a memory and a processor, wherein,

the memory is used for storing programs;

the processor, coupled to the memory, configured to execute the program stored in the memory to:

acquiring a first image and a second image;

performing difference analysis on the first image and the second image to obtain an output result containing difference information;

and classifying the output result to obtain the difference between the first image and the second image.

In another embodiment of the present application, an electronic device is provided. The electronic device includes: a memory and a processor, wherein,

the memory is used for storing programs;

the processor, coupled to the memory, configured to execute the program stored in the memory to:

acquiring a first image and a second image;

performing feature extraction on the first image to obtain at least one piece of first feature information;

performing feature extraction on the second image to obtain at least one piece of second feature information;

and determining the difference between the first image and the second image and the category of the content of the difference image according to the at least one piece of first characteristic information and the at least one piece of second characteristic information.

In yet another embodiment of the present application, a display device is provided. The display device includes: a memory, a processor, and a display, wherein,

the memory is used for storing programs;

the processor, coupled to the memory, configured to execute the program stored in the memory to:

responding to an image comparison event triggered by a user through an interactive interface, and acquiring a first image and a second image designated by the user in the image comparison event;

identifying the difference between the first image and the second image and the category of the content of the difference image to obtain an identification result;

generating a third image which reflects the difference between the first image and the second image and shows the content of different types of images in the difference in a distinguishing way based on the identification result;

controlling the display to display the third image.

According to the technical scheme provided by the embodiment of the application, the difference analysis is carried out on the first image and the second image, and the image contents of different types in the difference between the first image and the second image are also identified, so that the determined difference between the first image and the second image not only reflects the difference place, but also shows the change of the image contents of the different types; compared with the prior art, the technical scheme provided by the embodiment of the application has richer output result information and improves the detection effect of change detection.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be utilized in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative efforts.

Fig. 1 is a schematic flow chart of a data processing method according to an embodiment of the present application;

fig. 2 is a schematic flowchart of a data processing method according to another embodiment of the present application;

fig. 3 is a schematic flowchart of a data processing method according to another embodiment of the present application;

fig. 4 is a block diagram of a network structure according to an embodiment of the present application;

fig. 5 is a block diagram of a network architecture according to another embodiment of the present application;

fig. 6 is a block diagram of a network architecture according to another embodiment of the present application;

fig. 7a is a schematic flowchart of a data processing method according to another embodiment of the present application;

fig. 7b is a schematic diagram corresponding to a scene detected by combining a specific remote sensing image change in the technical scheme provided by the embodiment of the present application;

FIG. 7c is a diagram illustrating an implementation of an interactive interface according to an embodiment of the present application;

fig. 8 is a schematic diagram of a network system structure according to an embodiment of the present application;

fig. 9 is a flowchart illustrating a network system training method according to an embodiment of the present application;

fig. 10 is a schematic diagram of a network system structure according to another embodiment of the present application;

fig. 11 is a schematic flowchart of a network system training method according to another embodiment of the present application;

FIG. 12 is a block diagram of a data processing system according to an embodiment of the present application;

FIG. 13 is a block diagram of a data processing system according to another embodiment of the present application;

FIG. 14 is a schematic diagram of a remote sensing device according to an embodiment of the present application;

fig. 15 is a block diagram of a data processing apparatus according to an embodiment of the present application;

fig. 16 is a block diagram of a data processing apparatus according to another embodiment of the present application;

fig. 17 is a block diagram of a data processing apparatus according to another embodiment of the present application;

fig. 18 is a block diagram illustrating a network system training apparatus according to an embodiment of the present application;

fig. 19 is a block diagram illustrating a network system training apparatus according to another embodiment of the present application;

fig. 20 is a block diagram of an electronic device according to an embodiment of the present application.

Detailed Description

In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. It is to be understood that the embodiments described are only a few embodiments of the present application and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

Further, in some flows described in the specification, claims, and above-described figures of the present application, a number of operations are included that occur in a particular order, which operations may be performed out of order or in parallel as they occur herein. The sequence numbers of the operations, e.g., 101, 102, etc., are used merely to distinguish between the various operations, and do not represent any order of execution per se. Additionally, the flows may include more or fewer operations, and the operations may be performed sequentially or in parallel. It should be noted that, the descriptions of "first", "second", etc. in this document are used for distinguishing different messages, devices, modules, etc., and do not represent a sequential order, nor limit the types of "first" and "second" to be different. In addition, the embodiments described below are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

The technical scheme provided by each embodiment of the application is a scheme for detecting the difference of two images and the content of different types of images in the difference; the two images to be detected may be collected by a specific hardware device in real time, such as two remote sensing images collected by a remote sensing device of an aircraft in different periods, or two images corresponding to the same area at different times collected by a camera. In addition, the two images to be detected may be generated maps, such as maps generated by collecting data; and detecting city planning changes of the last year and the present year of a certain city or a certain area of the city, or detecting shop planning changes of the last year and the present year of a certain off-line mall, and the like.

Fig. 1 shows a schematic flow chart of a data processing method according to an embodiment of the present application. The execution subject of the method can be an electronic device with a logical operation function, and the electronic device can be a client or a server. The client can be any terminal equipment such as a mobile phone, a tablet personal computer and intelligent wearable equipment; the server may be a common server, a cloud, a virtual server, or the like, which is not specifically limited in this embodiment of the application. As shown in fig. 1, the method includes:

101. a first image and a second image are acquired.

102. And carrying out difference analysis on the first image and the second image to obtain an output result containing difference information.

103. And classifying the image content in the first image to obtain a first classification result.

104. And classifying the image content in the second image to obtain a second classification result.

105. And determining the difference between the first image and the second image according to the output result, the first classification result and the second classification result.

In the foregoing 101, when the execution subject of the method provided in this embodiment is a server, the first image and the second image may be sent to the server by a user through a client, or the server automatically acquires the first image and the second image from a database, a satellite, or an aircraft according to a task preset by the system.

The above "acquiring the first image and the second image" may be acquired as follows:

responding to an image input operation triggered by a user through an interactive interface, and acquiring the first image and the second image input by the user; or

And acquiring the first image and the second image which are acquired aiming at the target area at two different moments in response to target area input operation triggered by a user through an interactive interface.

One possible implementation of the above-mentioned method 102 "performing difference analysis on the first image and the second image to obtain an output result containing difference information" may include the following steps:

1021. performing feature extraction on the first image to obtain at least one piece of first feature information;

1022. performing feature extraction on the second image to obtain at least one piece of second feature information;

1023. and determining an output result containing difference information according to the at least one first characteristic information and the at least one second characteristic information.

The step 1021 can utilize the first network model to realize feature extraction of the first image. Specifically, the first network model includes at least one first network layer, and any one of the first network layers outputs a first feature information correspondingly. Accordingly, the step 1021 may specifically be: and performing feature extraction on the first image by using a first network model to obtain at least one piece of first feature information.

Step 1022 described above may utilize a second network model to perform feature extraction on the second image. Specifically, the second network model includes at least one second network layer, and any one of the second network layers correspondingly outputs a second feature information; the first network model and the second network model are twin network models with the same network structure and network parameters. Correspondingly, the step 1022 may specifically be: and utilizing a second network model to perform feature extraction on the second image to obtain at least one piece of second feature information.

The model structures of the first network model and the second network model and related more detailed information will be described below.

On this basis, step 103 in this embodiment may specifically be: and classifying the at least one piece of first characteristic information by using a fourth network model to obtain a first classification result. In a specific embodiment, the fourth network model comprises at least one fourth network layer; the number of the fourth network layers contained in the fourth network model is the same as the number of the first network layers contained in the first network model, and an association relationship exists. Correspondingly, the "classifying the at least one first feature information by using the fourth network model to obtain the first classification result" may specifically include the following steps:

according to the association relationship between the fourth network layer in the fourth network model and the first network layer in the first network model, inputting the first characteristic information corresponding to the first network layer with the association relationship into the corresponding fourth network layer, and executing the fourth network model to obtain the first classification result.

Similarly, the step 104 "classify the image content in the second image to obtain a second classification result" may specifically be: classifying the at least one second characteristic message by using a fifth network model to obtain a second classification result; and the network structures and the network parameters of the fifth network model and the fourth network model are the same.

The model structures of the fourth network model and the fifth network model and related more detailed information will be described below.

According to the technical scheme provided by the embodiment, the difference analysis is carried out on the first image and the second image, and the image contents of different types in the difference between the first image and the second image are also identified, so that the determined difference between the first image and the second image not only reflects the difference place, but also reflects the change of the image contents of the different types; compared with the prior art, the technical scheme provided by the embodiment has richer output result information and improves the detection effect of change detection.

Fig. 2 is a schematic flowchart illustrating a data processing method according to an embodiment of the present application. Also, the execution subject of the method may be a client or a server. As shown in fig. 2, the method includes:

201. acquiring a first image and a second image;

202. performing difference analysis on the first image and the second image to obtain an output result containing difference information;

203. and classifying the output result to obtain the difference between the first image and the second image.

For details of the above 201 and 202, reference may be made to corresponding parts in other embodiments, which are not described herein again.

In 203, classifying the output result to obtain a difference between the first image and the second image includes:

taking the output result as an input of a multi-classification model, and executing the multi-classification model to output the difference of the first image and the second image; the multi-classification model comprises convolution layers, and the number of input channels of each convolution layer corresponds to the number of output results; the number of output channels of the convolutional layer is determined by the number of image content categories to be identified.

Further, the method provided by this embodiment may further include the following steps:

204. and generating a third image which reflects the difference between the first image and the second image and shows the content of different types of images in the difference in a distinguishing way based on the difference between the first image and the second image which are output.

According to the technical scheme provided by the embodiment, the output result is classified by using the multi-classification model, so that a third image which can reflect the difference between the first image and the second image and can also distinguish and display the contents of different types of images in the difference can be obtained; through the third image, the user can know the difference and can also obtain which types of image contents are changed; compared with the prior art, the technical scheme provided by the embodiment has richer output result information and improves the detection effect of change detection.

Fig. 3 is a schematic flowchart illustrating a data processing method according to an embodiment of the present application. The execution subject of the method can be an electronic device with a logical operation function, and the electronic device can be a client or a server. The client can be any terminal equipment such as a mobile phone, a tablet personal computer and intelligent wearable equipment; the server may be a common server, a cloud, a virtual server, or the like, which is not specifically limited in this embodiment of the application. As shown in fig. 3, the method includes:

301. acquiring a first image and a second image;

302. performing feature extraction on the first image to obtain at least one piece of first feature information;

303. performing feature extraction on the second image to obtain at least one piece of second feature information;

304. and determining the difference between the first image and the second image and the category of the content of the difference image according to the at least one first characteristic information and the at least one second characteristic information.

In 301, the first image and the second image may be acquired by an image sensor or acquired from an image data set. The image sensor may be a camera, a remote sensing device disposed on an aircraft or a satellite, or the like, and this embodiment is not particularly limited thereto. When the execution main body of the method provided by the embodiment is the client, the client can receive the first image and the second image input by the user on the client interface by responding to the image input event triggered by the user; when the execution subject of the method provided by the embodiment is the server, the first image and the second image input by the user and transmitted by the client can be received.

One implementation of the above 302 "performing feature extraction on the first image to obtain at least one first feature information" is: performing feature extraction on the first image by using a first network model to obtain at least one piece of first feature information; the first network model comprises at least one first network layer, and any first network layer correspondingly outputs first characteristic information. In a specific implementation, the number of the first network layers included in the first network model may be one or more, and may be specifically set according to an actual need. The feature extraction of the first image by using the first network model can be realized by adopting convolution operation, and when any first network layer in the first network model extracts the feature of the first image, first feature information is correspondingly output.

One implementation scheme of "extracting the features of the second image to obtain at least one piece of second feature information" in 303 above is as follows: performing feature extraction on the second image by using a second network model to obtain at least one piece of second feature information; the second network model comprises at least one second network layer, and any second network layer correspondingly outputs second characteristic information. In specific implementation, the first network model and the second network model are twin network models with the same network structure and network parameters. The feature extraction of the second image by using the second network model can also be realized by adopting an image feature extraction technology (such as convolution operation), and when any second network layer in the second network model performs the feature extraction of the first image, second feature information is correspondingly output.

The twin network model is a multi-branch and weight-sharing network structure and is characterized in that two images can be received as input at the same time, and weights of two branch networks are shared. In a specific embodiment, as shown in fig. 4, the twin network model 200 is composed of a first network model 210 and a second network model 220 with the same network structure and completely shared weight; wherein, the first network model is used for processing a first image and extracting the characteristics of the first image; the second network model is used for processing a second image and extracting the characteristics of the second image. In the process of extracting the features, any network layer in each network model correspondingly outputs a piece of feature information, that is, a piece of first feature information is correspondingly output for any first network layer of the first network model, and a piece of second feature information is correspondingly output for any second network layer of the second network model. Here, any one of the twin network models (i.e., the first network model and the second network model) includes at least one network layer, the feature maps (i.e., feature information) output by the same network layer have the same size, and the feature maps output by different network layers have the same or different sizes, and the former network layer is directly connected to the latter network layer.

In an implementation manner, the aforementioned step 304 of determining the difference between the first image and the second image and the category of the content of the difference image according to the at least one first feature information and the at least one second feature information may specifically include:

3041. analyzing the at least one first characteristic information and the at least one second characteristic information by using a third network model to obtain an output result;

3042. and carrying out multi-class classification on the output result by using a multi-classification model to obtain the difference between the first image and the second image and the class of the content of the difference image.

3041, the third network model includes at least one third network layer, and each network layer is activated by an activation function; the number of third network layers contained in the third network model is the same as that of the first network layers contained in the first network model, and an association relationship exists; the number of third network layers contained in the third network model is the same as the number of second network layers contained in the second network model, and an association relationship exists; correspondingly, step 3041 "analyze the at least one first feature information and the at least one second feature information by using the third network model to obtain an output result", which may be specifically implemented by the following steps:

and inputting the first characteristic information corresponding to the first network layer and the second characteristic information corresponding to the second network layer which have the association relationship to the corresponding third network layer according to the association relationship between the third network layer in the third network model and the first network layer in the first network model and the second network layer in the second network model respectively, and executing the third network model to obtain the output result.

In specific implementation, the overall structures of the third network model are respectively symmetrical to the overall structures corresponding to the first network model and the second network model. Correspondingly, the association relationship between the third network layer in the third network model and each of the first network layer in the first network model and the second network layer in the second network model is: first feature information output by a first network layer of the first network model and second feature information output by a second network layer of the second network model can be transmitted to a third network layer of the third network model, which is symmetrical to the first network layer and the second network layer, as inputs of the third network layer, so that the third network model can be executed to obtain corresponding output results. Namely, the association relationship can be understood as a data flow relationship corresponding to different network layers between different network models.

For a specific embodiment, reference is continued to FIG. 4. The direction of the arrows in fig. 4 is the data flow direction. The number of first network layers included in the first network model 210, the number of second network layers included in the second network model 220, and the number of third network layers included in the third network model 300 are all 4. According to the association relationship between the third network layer in the third network model and the first network layer in the first network model and the second network layer in the second network model, the output result N4 of the 4 th first network layer 14 in the first network model can be obtained₁And the output result N4 of the 4 th second network layer 24 in the second network model₂Input into the 1 st third network layer 31 in the third network model, the 1 st third network layer in the third network model receives the output result N4₁And said output result N4₂Then pair N4₁And N4₂Merging according to the channel dimension to obtain an output result d4₁₂And outputs the output result d4₁₂Into the 2 nd third network layer 32 in the third network model; then, the 2 nd third network layer in the third network model outputs the output result d4₁₂Output result N3 with the 3 rd first network layer 13 in the first network model₁And the output result N3 of the 3 rd second network layer 23 in the second network model₂Merging results by channel dimension d3₁₂Performing fusion to obtain an output result D1, and integrally inputting the output result D1 into the 3 rd third network layer 33 of the third network model; then, the 3 rd third network layer of the third network model further combines the output result D1 with the output result N2 of the 2 nd first network layer 12 in the first network model₁And 2 nd second network layer 22 in the second network modelOutput result N2₂The result of the combination of d2₁₂Performing fusion to obtain an output result D2, and inputting the output result D2 into the 4 th third network layer 34 of the third network model as a whole; finally, the 4 th third network layer of the third network model further combines the output result D2 with the output N1 of the 1 st first network layer 11 in the first network model₁And the output N1 of the 1 st second network layer 21 in the second network model₂Combined result d1 of₁₂And fusing to obtain a final output result D3 of the third network model, wherein the final output result D3 is to be input into the multi-classification model 410 and/or the two-classification model 420 so that the multi-classification model or the two-classification model can classify the output result. If the number of first network layers included in the first network model, the number of second network layers included in the second network model, and the number of third network layers included in the third network model are n, the final output result of the third network model may be obtained with reference to the above-described iterative stacking calculation method. The data stream transfer connection between the first network model and the second network model and the third network model may be referred to as symmetric tight connection. Through the symmetrical tight connection, the feature information output by the multilevel correspondence can be effectively fused, so that not only is the high-level feature with abstract semantics utilized, but also the low-level feature of the multi-representation color texture shape is utilized, and the feature of each intermediate layer between the high-level feature and the low-level feature can be utilized, and the accuracy of the output result can be improved.

It should be added that: the specific 4 third network layers included in the third network model in fig. 4 may be: the 1 st third network layer 31 is a convolutional layer with a convolutional kernel of 1 × 1, the 2 nd third network layer 32 is an deconvolution layer with a convolutional kernel of 3 × 3, the 3 rd third network layer 33 is a convolutional layer with a convolutional kernel of 1 × 1, and the number of output channels of the last third network layer 34 (i.e., the 4 th third network layer 34) is 128. And an activation layer is arranged between any adjacent third network layers, and the activation function corresponding to the activation layer is a RELU activation function.

3042, the multi-class model includes convolutional layers, the number of input channels of the convolutional layers corresponds to the number of output results; the number of output channels of the convolutional layer is determined by the number of image content categories to be identified. The multi-classification model performs classification processing on the received output result of the third network model, so that a third image which reflects the difference between the first image and the second image and shows different types of image contents in a distinguishing manner can be obtained. In addition, the obtained third image can be sent to a display device to be displayed on a display interface of the display device.

Further, the method further includes:

305. and classifying the output result by using a binary model to obtain a fourth image reflecting the difference between the first image and the second image. In specific implementation, the second classification model performs classification processing on the received output result of the third model to obtain a fourth image reflecting the difference between the first image and the second image.

What needs to be supplemented is: the two-classification model and the multi-classification model are connected with the last layer of the third network model in parallel, and other parameters of the multi-classification model and the two-classification model are the same except that the number of channels of the multi-classification model is the number of classes for multi-class learning.

In the technical scheme provided by this embodiment, feature extraction is performed on a first image and a second image respectively to obtain at least one piece of first feature information corresponding to the first image and at least one piece of second feature information corresponding to the second image; and determining a third image reflecting the difference between the first image and the second image and showing different types of image contents by utilizing a multi-classification model in combination with the at least one first characteristic information and the at least one second characteristic information. The change of the image is detected by utilizing the multi-level feature fusion information of the image, so that the accuracy of image change detection can be effectively improved, and meanwhile, the false reduction rate and the omission ratio are reduced.

In another implementation, the aforementioned step 304 of determining the difference between the first image and the second image and the type of the content of the difference image according to the at least one first feature information and the at least one second feature information may specifically include:

3041', analyzing the at least one first characteristic information and the at least one second characteristic information by using a third network model to obtain an output result;

3042', using a fourth network model to classify the at least one first feature information to obtain a first classification result;

3043', classifying the at least one second feature information by using a fifth network model to obtain a second classification result;

3044', obtaining the difference between the first image and the second image and the category of the content of the difference image according to the output result, the first classification result and the second classification result.

The above 3041' can refer to the corresponding contents of the above embodiments, and will not be described herein.

3042' above, the fourth network model includes at least one fourth network layer; the number of fourth network layers contained in the fourth network model is the same as the number of first network layers contained in the first network model, and an association relationship exists; correspondingly, in step 3042' ″ classifying the at least one first feature information by using the fourth network model to obtain a first classification result, the following steps may be specifically adopted to implement:

according to the incidence relation between a fourth network layer in a fourth network model and a first network layer in the first network model, inputting first characteristic information corresponding to the first network layer with the incidence relation to the corresponding fourth network layer, and executing the fourth network model to obtain the classification results of different types of image contents in the first image.

Specifically, the fourth network model and the first network model are symmetrical in structure and are tightly connected with each other symmetrically; correspondingly, the relationship between the fourth network layer in the fourth network model and the first network layer in the first network model is as follows: the first feature information correspondingly output by the first network layer of the first network model is transmitted to the fourth network layer as the input of the fourth network layer in the fourth network model symmetrical to the first network layer, so that the fourth network model is utilized to classify the at least one first feature information to obtain a first classification result;

as an embodiment, referring to fig. 5, the arrow direction in fig. 5 is a data flow direction, the fourth network model and the first network model have symmetrical structures, and the number of network layers included in both the fourth network model and the first network model is 4. According to the association relationship between the fourth network layer in the fourth network model 510 and the first network layer in the first network model 210, the output result N4 of the 4 th first network layer 14 in the first network model can be obtained₁Input into the 1 st fourth network layer 41 in the fourth network model, the 1 st fourth network layer receiving the output result N4₁Performing classification processing to obtain an output result d4₁And outputs the output result d4₁Into the 2 nd fourth network layer 42 in the fourth network model; then, the 2 nd fourth network layer 42 outputs the output result d4₁Output result N3 with the 3 rd first network layer 13 in the first network model₁Performing fusion and classification processing to obtain an output result d3₁And outputs the result d3₁Into the 3 rd fourth network layer 43 of the fourth network model; then, the 3 rd fourth network layer of the fourth network model further outputs the output result d3₁Output result N2 with the 2 nd first network layer 12 in the first network model₁Performing fusion and classification processing to obtain an output result d2₁The output result d2 is used₁The whole is input into the 4 th fourth network layer 44 of the fourth network model; finally, the 4 th fourth network layer further outputs the output result d2₁Output N1 from the 1 st first network layer 11 in the first network model₁Performing fusion and classification processing to obtain a final output result d1 of the fourth network model₁The final output result d1₁The first classification result is obtained by classifying the at least one piece of first feature information by using the fourth network model.

3043', the fifth network model 520 and the fourth network model have the same structure and network parameters, that is, the number of the fifth network layers included in the fifth network model is the same as the number of the second network layers included in the second network model, and there is an association relationship therebetween; based on this, the detailed implementation of step 3043 'can be referred to the corresponding content in 3042' and will not be described herein again.

3044', the output result of the third network model, the first classification result of the fourth network model, and the second classification result of the fifth model are combined to obtain the difference between the first image and the second image and the type of the content of the difference image, so as to effectively improve the reliability of the output result and reduce the false detection rate.

Further, the method provided by this embodiment may further include the following steps:

306. acquiring the first image and the second image input by a user in response to an image input event triggered by the user;

307. and displaying a third image reflecting the difference between the first image and the second image and differentially displaying different types of image contents in the difference on a display interface.

According to the technical scheme provided by the embodiment of the application, the third image which reflects the difference between the first image and the second image and shows different types of image contents in a distinguishing manner is determined by integrating the output result obtained by analyzing the at least one first characteristic information and the at least one second characteristic information by the third network model, the first classification result obtained by classifying the at least one first characteristic information by the fourth network model and the second classification result obtained by classifying the at least one second characteristic information by the fifth model, so that the reliability of the output result can be effectively improved, and the false detection rate is reduced.

In order to more clearly illustrate the practical effect of the change detection in the technical solution of the present application, a specific application scenario will be combined below. If the first image and the second image are two-stage remote sensing images in different periods, with reference to fig. 4, a first network model 210 may be used to perform feature extraction on an input previous-stage remote sensing image (i.e., the first image 100) to obtain at least one piece of first feature information corresponding to the previous-stage remote sensing image; performing feature extraction on the input later-stage remote sensing image (namely, the second image 110) by using the second network model 220 to obtain at least one piece of second feature information corresponding to the later-stage remote sensing image; inputting the at least one first feature information and the at least one second feature information into the third network model in a data flow direction indicated by arrows in fig. 4, where the third network model obtains a corresponding output result by performing fusion analysis on the received at least one first feature information and the received at least one second feature information; and respectively inputting the output result into the two-classification model and the multi-classification model, classifying the output result through the two-classification model to obtain a fourth image reflecting the difference between the first image and the second image, and performing multi-class classification processing on the output result through the multi-classification model to obtain a third image reflecting the difference between the first image and the second image and displaying different types of image contents in a distinguishing manner, so that the class result of change detection of the two phases of remote sensing images can be obtained. Wherein, the change detection refers to a technology of obtaining change information of an object or a phenomenon by observing the state of the object or the phenomenon at different time; the change detection category of the two-stage remote sensing image can be as follows: newly adding a building, newly adding push filling soil, newly adding a greenhouse, removing the building, changing the land into the water area, changing the water area into the push filling soil, paving the existing movable soil and the like.

In addition, in order to realize more accurate classification of the change difference corresponding to the two-stage remote sensing images, the detection of the ground object class in the two-stage remote sensing images is required. Referring to fig. 5, a fourth network model 510 and a fifth network model 520, which are symmetrically and tightly connected to the first network model and the second network model, may be added to an original base of a basic network model including the first network model 210, the second network model 220 and the third network model, respectively, where the fourth network model 510 and the fifth network model 520 have the same structure and the same network parameters as the third network model, but the number of channels corresponding to the fourth network model 510 and the fifth network model 520 is half of that of the third network model, and the number of channels corresponding to the last layer of each of the fourth network model 510 and the fifth network model 520 is the number of classified ground object types. A first classification result can be obtained by performing a feature classification process on at least one first feature information corresponding to a previous remote sensing image output by the first network model 210 through the fourth network model 510, the fifth network model 520 performs a ground object classification process on at least one second feature information corresponding to a later-stage remote sensing image output by the second network model 220 to obtain a second classification result, and combines a change detection result corresponding to a second-stage remote sensing image output by the third network model, a ground object classification result corresponding to a previous-stage remote sensing image output by the fourth network model 510, and a ground object classification result corresponding to a later-stage remote sensing image output by the fifth network model 520 to determine a third image which reflects a difference between the previous-stage remote sensing image and the later-stage remote sensing image and differentially displays different types of image contents. The ground object types in the two-stage remote sensing image can comprise types of buildings, water bodies and the like. In the above, the network models (such as the first network model, the second network model, the third network model, the binary model and the multi-classification model) related to the change detection, except for the multi-classification model, parameters in the rest of the network models are migrated from the base network model (i.e. the first network model, the second network model and the third network model).

It should be noted that: in the whole network structure training process in fig. 4, the two-class model and the multi-class model for change detection are synchronously trained; in the whole process of training the network structure of fig. 5, the feature classification (i.e. the fourth network model and the fifth classification model) and the change detection (i.e. the third network model) are trained alternately.

Table 1 shows the evaluation results of the change detection performed on the two-stage remote sensing images corresponding to 9 regions of the province by using the data processing method, where the corresponding evaluation index is the IOU cross-over ratio. The IOU intersection ratio refers to that: the larger the value of the intersection and union of the prediction box and the labeling box is, the better the performance of the network structure is, and generally, the network structure is considered to have good detection performance if IoU is greater than 0.5. And the lifting points in table 1 are: the percentage of the difference between the evaluation result corresponding to the second network structure and the evaluation result corresponding to the first network structure, that is, the lifting point is (the evaluation result corresponding to the second network structure — the evaluation result corresponding to the first network structure) × 100%.

Evaluation results corresponding to Table 1

Model (model)	Basic network model	First network structure	Second network structure
				IOU	0.5841	0.5969	0.6033
Lifting point	0	1.28 points	Point 1.92

As can be seen from table 1, in the present solution, when the network system shown in fig. 4 is used to perform change detection on the two-stage remote sensing image and the network system shown in fig. 5 is used to perform change detection on the two-stage remote sensing image, the obtained change detection results are respectively improved by 1.28 points and 1.92 points compared with the scheme shown in fig. 6 that uses the basic network model, which can effectively ensure the accuracy of change detection of the two-stage remote sensing image and reduce the false rate.

Table 2 shows the evaluation result of the classification result of the change detection category of the two-stage remote sensing image based on the data processing method corresponding to each of the basic model structure and the first network structure. Wherein the types of the change detection are respectively from 1 to 8: the method is characterized by comprising the following steps of adding a new building, adding construction site push-filling soil, increasing a shed, dismantling a greenhouse, dismantling the building, changing land into a water area, changing a water area into a construction site push-filling soil and paving the existing movable soil.

TABLE 2 evaluation results corresponding to multiple classification results

	1	2	3	4	5	6	7	8
									Single task	0.484	0.4881	0.119	0.6098	0.3198	0.4635	0.4614	0.187
Multitasking	0.492	0.5038	0.1013	0.699	0.3429	0.4725	0.4582	0.227
									Lifting point	0.8 point	1.57 point	Point-1.0887	8.92 point	2.31 point	0.9 point	-0.53 point	4 points

As can be seen from table 2, compared with the prior art that the remote sensing image change detection is realized by using the basic network model shown in fig. 6, 1.8757 points are improved integrally by using the technical scheme provided by the present application. Therefore, the technical scheme can accurately classify the types of the change detection of the two-stage remote sensing image.

Fig. 7a and 7b are schematic flow charts illustrating a data processing method according to another embodiment of the present application. As shown in fig. 7a, the method comprises:

401. responding to an image comparison event triggered by a user through an interactive interface, and acquiring a first image and a second image designated by the user in the image comparison event;

402. identifying the difference between the first image and the second image and the category of the content of the difference image to obtain an identification result;

403. generating a third image which reflects the difference between the first image and the second image and shows the content of different types of images in the difference in a distinguishing way based on the identification result;

404. and displaying the third image.

In 402, the "recognizing the difference between the first image and the second image and the category of the content of the difference image to obtain the recognition result" includes:

performing difference analysis on the first image and the second image to obtain an output result containing difference information; classifying the output result to obtain an identification result containing difference information and different types of image contents in the difference; or

Performing difference analysis on the first image and the second image to obtain an output result containing difference information; classifying the image content in the first image to obtain a first classification result; classifying the image content in the second image to obtain a second classification result; and obtaining the identification results containing the difference information and the image contents of different types in the difference according to the output result, the first classification result and the second classification result.

It should be noted that, for more detailed contents of the above steps, reference may be made to corresponding contents in the above embodiments, which are not described herein again.

Further, the method provided by this embodiment further includes:

405. responding to the operation that a user inputs a first image and a second image through the interactive interface, and triggering the image comparison event; or

406. Responding to an operation that a user inputs a target area through an interactive interface, and triggering an image comparison event aiming at the target area; and the first image and the second image designated by the user in the image comparison event are images corresponding to the target area at different moments.

Referring to the example shown in fig. 7b, the first image and the second image may be remote sensing images acquired by a remote sensing device mounted on the aircraft 03, and the first image and the second image are remote sensing images corresponding to the same area 05 at different times. For example, the first image is obtained by the remote sensing device performing remote sensing image acquisition on the area 05 at a certain day in the last year; the second image is acquired by the remote sensing device from the area 05 at a certain day of the year. The remote sensing device may transmit the acquired image to the electronic device 04. The electronic device 04 may be a desktop computer as shown in fig. 7 b. The image sent by the remote sensing device may be stored in a local storage medium of the electronic device 04, so that the user may call the corresponding image in an interactive manner provided by the electronic device and trigger an image comparison event. For example, the user may retrieve the first image and the second image stored locally through the interactive interface 01 as shown in fig. 7 b. Then, by clicking on the "start detection" control as in fig. 7b, an image alignment event is triggered. After the image comparison event is triggered, the electronic device 04 performs an operation of identifying the difference between the two images and the content of the images in different categories in the difference, so as to generate a third image (i.e. a detection result); subsequently, the third image may be displayed in the display interface 02 as shown in fig. 7 b. The user can visually observe the change condition of the area through the third image, and the specific content of the change is that the land of the area originally becomes an artificial lake, or the land of the area originally becomes bare land, and the like. Further, since only different types of image contents are distinctively displayed in the third image, information on the image contents of the different types may be displayed around the third image on the display interface 02 shown in fig. 7 b.

Alternatively, the user may also input the target region, specifically, the name or code of the target region, etc., through the interactive interface 011 shown in fig. 7 c. After the input, the user can not input the time information, and the system defaults to acquire the images corresponding to the two moments closest to the current time. Of course, the user may also input a first time or a time period and a second time or a time period through the interactive interface, so that the electronic device may acquire a first image acquired in the first time or the time period and perform change detection on a second image acquired in the second time or the time period.

A network system will be described in detail below. Referring to fig. 8, a schematic structural diagram of a network system provided in an embodiment of the present application is shown. The network system specifically includes: a first network model 210, a second network model 220, a third network model 300, and a multi-classification model 410; wherein the content of the first and second substances,

the first network model 210 includes at least one first network layer, and is configured to perform feature extraction on a first image and output at least one first feature information;

the second network model 220 includes at least one second network layer, and is configured to perform feature extraction on the second image and output at least one second feature information;

the third network model 300 is connected to the output ends of the first network model 210 and the second network model 220, and configured to perform difference analysis on the at least one first feature information and the at least one second feature information to obtain an output result containing difference information;

the multi-classification model 410 is connected to the third network model 300, and is configured to perform multi-class classification on the output result to obtain a difference between the first image and the second image.

In the above, the third network model 300 comprises at least one third network layer; the number of third network layers included in the third network model is the same as the number of first network layers included in the first network model 210, and an association relationship exists; the number of third network layers included in the third network model 300 is the same as the number of second network layers included in the second network model 220, and there is an association relationship; accordingly, the third network model 300 is specifically configured to: and inputting first characteristic information corresponding to the first network layer and second characteristic information corresponding to the second network layer which have the association relationship to the corresponding third network layer according to the association relationship between the third network layer in the third network model and the first network layer in the first network model and the second network layer in the second network model respectively, and executing the third network model to obtain the output result.

More specifically, referring to fig. 4, the first network model 210 includes four first network layers, respectively: a first network layer 11, a first network layer 12, a first network layer 13, and a first network layer 14. The second network model 220 includes four second network layers, respectively: a second network layer 21, a second network layer 22, a second network layer 23, and a second network layer 24. The third network model 300 may also include four third network layers, namely a third network layer 31, a third network layer 32, a third network layer 33, and a third network layer 34. The embodiment does not specifically limit the specific implementation of each first network layer, each second network layer and each third network layer, and can be implemented by directly adopting a binary single-task learning basic network with twin coding and multi-level intensive decoding in the prior art.

For example, the third network layer 31 in the third network model shown in fig. 4 may be a convolutional layer with a convolution kernel of 1 × 1, and the number of input channels is the sum of the number of output channels of the first network layer 14 and the number of output channels of the second network layer 24; the output channels of the third network layer 31 are one quarter of the input channels. The third network layer 32 is a deconvolution layer with the kernel size of 3 × 3, and the input channel number is the sum of the output channel number of the first network layer 13, the output channel number of the second network layer 23 and the output channel number of the third network layer 31; the number of output channels is the same for the third network layer 31. The third network layer 33 is a convolutional layer with a convolutional kernel of 1 x 1, the number of input channels is the sum of the number of output channels of the first network layer 12, the number of output channels of the second network layer 22 and the number of output channels of the third network layer 32, and the number of output channels is the sum of the number of output channels of the first network layer 13 and the number of output channels of the second network layer 23. And activating functions between each layer in the third network model, such as sigmoid, ReLU and ELU activating functions.

Further, the multi-class model 410 connected to the third network model 300 includes convolution layers, the number of input channels of the convolution layers corresponds to the number of output results; the number of output channels of the convolutional layer is determined by the number of image content categories to be identified.

Further, the network system may further include a binary model 420. The classification model 420 is connected to the third network model 300, and is configured to classify the output result to obtain a fourth image reflecting a difference between the first image and the second image.

In the technical scheme provided by this embodiment, feature extraction is performed on a first image and a second image respectively to obtain at least one piece of first feature information corresponding to the first image and at least one piece of second feature information corresponding to the second image; and determining a third image reflecting the difference between the first image and the second image and showing different types of image contents by utilizing a multi-classification model in combination with the at least one first characteristic information and the at least one second characteristic information. The accuracy of change detection of the first image and the second image can be effectively improved through the multi-level characteristic information fusion mode.

Here, it should be noted that: for the content of each network model in the network system provided in the embodiment of the present application, which is not described in detail in the foregoing embodiments, reference may be made to the corresponding content in each embodiment, and details are not described here again.

A method for training the network system will be described with reference to fig. 9. As shown in fig. 9, the training method includes:

501. performing feature extraction on the first sample image by using a first network model to obtain at least one piece of first sample feature information;

502. performing feature extraction on the second sample image by using a second network model to obtain at least one second sample feature information;

503. performing difference analysis on the at least one first sample characteristic information and the at least one second sample characteristic information by using a third network model to obtain an output result;

504. performing multi-class classification on the output result by using a multi-class model to obtain a multi-class classification result;

505. optimizing network parameters of at least one of the first network model, the second network model, the third network model and the multi-classification model according to the multi-class classification result and the multi-class sample label;

wherein the multi-class sample label, the first sample image, and the second sample image are associated.

For specific implementation of the steps 501 to 505, reference may be made to corresponding contents in the above embodiments, which are not described herein again.

According to the technical scheme provided by the embodiment of the application, the first image and the second image are respectively subjected to feature extraction to obtain at least one piece of first feature information corresponding to the first image and at least one piece of second feature information corresponding to the second image; then, analyzing the at least one first characteristic information and the at least one second characteristic information by using the third network model to obtain a corresponding output result; classifying the output result by utilizing the multi-classification model to obtain a multi-class classification result; parameters in each network model can be optimized based on the multi-class classification result and the multi-class sample label. The multi-level feature fusion information has strong expression capability and higher value, and the classification accuracy of the image change detection category can be effectively improved by utilizing the multi-level feature fusion information to classify the image change detection.

Further, the training method provided in this embodiment may further include:

classifying the output result by using a two-classification model to obtain a two-classification result; accordingly, the step 505 "optimizing the network parameters of at least one of the first network model, the second network model, the third network model and the multi-class model according to the multi-class classification result and the multi-class sample label" may be specifically implemented by:

5051. determining a first loss function according to the double-class classification result and the double-class sample label;

5052. determining a second loss function according to the multi-class classification result and the multi-class sample label;

5053. optimizing network parameters of at least one of the first network model, the second network model, the third network model, and the second classification model based on the first loss function and the second loss function;

wherein the dual-class sample label is associated with the first sample image and the second sample image.

In 5051, the first loss function determined according to the dual-class classification result and the dual-class sample label may be:

Loss1＝BCE+DICE (1)

wherein, DICE 1-2TP/(2TP + FN + FP)

BCE(X)＝-[ylogf(x)+(1-y)log(1-f(x))]

Wherein, TP, FP and FN are the number of true positive, false positive and false negative respectively; y is the label, f (x) is the network output.

In 5052, the second loss function determined according to the multi-class classification result and the multi-class sample label may be:

Loss2＝CE+DICE (2)

wherein the content of the first and second substances,

wherein the DICE definition is the same as the first loss function; c is a category of change detection multi-classification.

In 5053, based on the first loss function and the second loss function, it can be determined that the total loss function corresponding to the network system is formula (3):

Loss_G＝w1*Loss1+w2*Loss2 (3)

w1 is a weight parameter corresponding to the first Loss function Loss 1; w2 is the weight parameter corresponding to the second Loss function Loss 2. And determining at least one network model needing to be optimized in the first network model, the second network model, the third network model and the two classification models based on the total Loss function Loss _ G, and further optimizing network parameters in the network model.

Another network system will be described with reference to fig. 10 and 5. As shown in fig. 10 and 5, the network system specifically includes: a first network model 210, a second network model 220, a third network model 300, a fourth network model 510, a fifth network model 520, and an output model 23; wherein the content of the first and second substances,

the first network model 210 includes at least one first network layer, and is configured to perform feature extraction on a first image and output at least one first feature information;

the second network model 220 includes at least one second network layer, and is configured to perform feature extraction on the second image and output at least one second feature information;

the third network model 300 is connected to the output ends of the first network model and the second network model, and is configured to analyze the at least one first feature information and the at least one second feature information to obtain an output result;

the fourth network model 510 is connected to the first network model, and configured to perform classification processing on the at least one first feature information to obtain a first classification result;

the fifth network model 520 is connected to the second network model, and is configured to perform classification processing on the at least one piece of second feature information to obtain a second classification result;

the output model 23 is connected to the third network model 300, the fourth network model 510, and the fifth network model 520, and is configured to output a difference between the first image and the second image according to the output result, the first classification result, and the second classification result.

In the above, the fourth network model 510 and the fifth network model 520 have the same structure and network parameters; the fourth network model 510 comprises at least one fourth network layer; the number of the fourth network layers included in the fourth network model is the same as the number of the first network layers included in the first network model 210, and an association relationship exists; correspondingly, the fourth network model 510 is specifically configured to input the first feature information corresponding to the first network layer in which the association exists to the corresponding fourth network layer according to the association relationship between the fourth network layer in the fourth network model and the first network layer in the first network model, and execute the fourth network model to obtain the classification result of the different types of image contents in the first image. Referring to fig. 5, a fifth network layer in the fifth network model and a second network layer in the second network model have an association relationship, and similarly, second feature information corresponding to the second network layer having the association relationship may be input to the corresponding fifth network layer, and the fifth network model is executed to obtain classification results of different types of image contents in the second image.

In specific implementation, for different application scenarios, the fourth network model and the fifth network model in the network system provided by this embodiment have different image contents and different number of categories to be classified, so that the specific implementation schemes are different. For example, in a remote sensing image detection scenario, the purpose of the fourth network model and the fifth network model is a surface feature classification, wherein the surface feature classification may include, but is not limited to: buildings, bodies of water, etc. Therefore, according to the requirements in the actual application scene, a network model can be constructed or a corresponding model can be selected from the existing network models to obtain the fourth network model and the fifth network model.

According to the technical scheme provided by the embodiment of the application, the third image which reflects the difference between the first image and the second image and shows different types of image contents in a distinguishing manner is determined by integrating the output result obtained by analyzing the at least one first characteristic information and the at least one second characteristic information by the third network model, the first classification result obtained by classifying the at least one first characteristic information by the fourth network model and the second classification result obtained by classifying the at least one second characteristic information by the fifth model, so that the reliability of the output result can be effectively improved, and the false detection rate is reduced.

Further, the output model 23 is further configured to output a third image reflecting the difference between the first image and the second image and displaying different types of image content in the difference in a distinguishing manner.

Here, it should be noted that: for the content of each network model in the network system provided in the embodiment of the present application, which is not described in detail in the foregoing embodiments, reference may be made to the corresponding content in each embodiment, and details are not described here again.

A method for training the network system will be described with reference to fig. 11. As shown in fig. 11, the training method includes:

601. performing feature extraction on the first sample image by using a first network model to obtain at least one piece of first sample feature information;

602. performing feature extraction on the second sample image by using a second network model to obtain at least one second sample feature information;

603. analyzing the at least one first sample characteristic information and the at least one second sample characteristic information by using a third network model to obtain an output result;

604. optimizing network parameters of at least one of the first network model, the second network model and the third network model according to the output result and the first sample label;

605. analyzing the at least one first sample characteristic information by using a fourth network model to obtain a first classification result;

606. optimizing network parameters of at least one of the first network model and the fourth network model according to the first classification result and a second sample label;

wherein the first and second swatch labels are each associated with the first and second swatch images.

The specific implementation of the steps 601 to 605 can refer to the corresponding content in the above embodiments, and will not be described herein again.

What needs to be supplemented is: a fourth loss function corresponding to the fourth network model is:

Loss_4＝CE+DICE (4)

where the definitions of CE and DICE are consistent with the second loss function, C may be a category of the ground feature classification ground feature.

Further, the training method provided in this embodiment may further include:

607. and obtaining a fifth network model based on the optimized fourth network model.

In the technical solution provided in the embodiment of the present application, the third network model is integrated to analyze the at least one first sample feature information and the at least one second sample feature information to obtain an output result and a first sample tag, so as to optimize the network parameters of at least one of the first network model, the second network model, and the third network model, thereby effectively improving the reliability of the output result and reducing the false detection rate. In addition, the network parameters of at least one of the first network model and the fourth network model are optimized based on the first classification result and the second sample label which are correspondingly output by the fourth network model, so that the classification accuracy of the fourth network model is improved.

The technical solutions provided by the embodiments of the present application can be implemented based on the following hardware system architecture or specific hardware devices.

FIG. 12 shows a block diagram of a data processing system. Specifically, the data processing system includes: image acquisition equipment and processing apparatus. The image acquisition equipment can be a camera or a remote sensing device. The remote sensing device is used for acquiring remote sensing images of a target area, and the remote sensing images are films or photos for recording the size of electromagnetic waves of various ground objects and are mainly divided into aerial photos and satellite photos. The processing device may be a device with data processing and computing capabilities, such as a desktop computer, a notebook computer, a tablet computer, a smart phone, a smart wearable device, etc. on the user side; or a server on the server side, a virtual server deployed on a server cluster, a cloud, etc. The data processing system may specifically include several architectures:

first, referring to the example shown in fig. 7a, an aircraft 03, such as an unmanned aerial vehicle, a manned aircraft, etc., is deployed with remote sensing devices; and a processing device (i.e., electronic device 04 in fig. 7 a) communicatively coupled to the aircraft. The processing device may be a portable device, such as a laptop, a tablet, a smart phone, a smart wearable device, etc.; and management equipment deployed in a monitoring management department and the like. In the scene, the change condition of the area and the specific content of the change can be obtained by detecting the remote sensing images of the same area in different periods. Alternatively, as shown in fig. 12, the processing device is a server 05 on the server side, and the server 05 transmits a third image obtained by processing the first image and the second image to the client device 06. The user may use the client device 06 to request an image comparison service from the server 05, for example, the user may input a target area and time of two images to be compared, etc. through an interactive interface, and then send a request to the server 05 based on data input by the user; after receiving the request, the server 05 calls a first image and a second image of the target area, which are acquired by the aircraft at two corresponding times or time periods, from a storage medium; then, the first image and the second image are processed according to the scheme provided by the embodiments to obtain a third image; the server 05 feeds back the third image to the client device 06 for viewing by the user. Of course, the first image and the second image may also be input by the user through the interactive interface. The image information sent to the server 05 after the remote sensing device acquires the image may include but is not limited to: images, acquisition time, region identification, etc. And after receiving the image information, the server stores the image information into a storage medium. When necessary, the server 05 may retrieve the image to be processed according to the area identifier and the acquisition time associated with the image.

Second, as shown in fig. 13, an image pickup device 07 provided at a fixed position periodically picks up an image of a fixed area; and a processing device 08, such as a desktop computer, a tablet computer, a smart phone, a smart wearable device, and the like, in communication with the image capture device 07. Under the system architecture, the processing device 08 may obtain the change condition of the fixed area and the specific content of the change by detecting images of the fixed area at different periods. For example, images of a certain floor of an online shopping mall at different periods can be obtained according to the technical scheme provided by the embodiments of the application, such as shop change, people flow distribution change and the like of the floor; thereby helping the operation of off-line shopping malls.

The image acquisition equipment is used for acquiring a first image and a second image of a target area corresponding to two different moments; the processing device may analyze the difference between the two images and may also identify different types of image content in the difference.

In an implementable technical solution, the processing device is configured to obtain the first image and the second image, perform difference analysis on the first image and the second image, and obtain an output result containing difference information; classifying the image content in the first image to obtain a first classification result; classifying the image content in the second image to obtain a second classification result; and determining the difference between the first image and the second image according to the output result, the first classification result and the second classification result.

In another implementable technical solution, the processing device is configured to obtain the first image and the second image, perform difference analysis on the first image and the second image, and obtain an output result containing difference information; and classifying the output result to obtain the difference between the first image and the second image.

In another implementable technical solution, the processing device is configured to acquire the first image and the second image, and perform feature extraction on the first image to obtain at least one piece of first feature information; performing feature extraction on the second image to obtain at least one piece of second feature information; and determining the difference between the first image and the second image and the category of the content of the difference image according to the at least one piece of first characteristic information and the at least one piece of second characteristic information.

It should be noted here that the processing device in this embodiment may also implement other functions besides the above functions, and specifically, refer to the description of the foregoing embodiments.

FIG. 14 illustrates a remote sensing device provided by an embodiment of the present application. The specific hardware form of the remote sensing device may be the aircraft shown in fig. 14, an unmanned aircraft, or a manned aircraft. Specifically, the remote sensing device includes: remote sensing device 031 and processor 032. The remote sensing device 031 is used for gathering the first remote sensing image and the second remote sensing image that the target area corresponds at two different moments. The processor can adopt the following specific technical schemes to realize the difference analysis of the two remote sensing images and the identification of different types of image contents in the difference.

In an achievable technical solution, the processor 032 is configured to obtain the first remote sensing image and the second remote sensing image, perform difference analysis on the first remote sensing image and the second remote sensing image, and obtain an output result containing difference information; classifying the image content in the first remote sensing image to obtain a first classification result; classifying the image content in the second remote sensing image to obtain a second classification result; and determining the difference between the first remote sensing image and the second remote sensing image according to the output result, the first classification result and the second classification result.

In another implementable technical scheme, the processor 032 is configured to obtain the first remote sensing image and the second remote sensing image, perform difference analysis on the first remote sensing image and the second remote sensing image, and obtain an output result containing difference information; and classifying the output result to obtain the difference between the first remote sensing image and the second remote sensing image.

In another implementable technical solution, the processor 032 is configured to obtain the first remote sensing image and the second remote sensing image, and perform feature extraction on the first image to obtain at least one first feature information; performing feature extraction on the second image to obtain at least one piece of second feature information; and determining the difference between the first image and the second image and the category of the content of the difference image according to the at least one piece of first characteristic information and the at least one piece of second characteristic information.

It should be noted here that the processing device in this embodiment may also implement other functions besides the above functions, and specifically, refer to the description of the foregoing embodiments.

Fig. 15 shows a block diagram of a data processing apparatus according to an embodiment of the present application. As shown in fig. 15, the data processing apparatus includes: an acquisition module 51, an analysis module 52, a classification module 53 and a determination module 54. The acquiring module 51 is configured to acquire a first image and a second image. The analysis module 52 is configured to perform difference analysis on the first image and the second image to obtain an output result containing difference information. The classification module 53 is configured to perform classification processing on image content in the first image to obtain a first classification result; and classifying the image content in the second image to obtain a second classification result. The determining module 54 is configured to determine a difference between the first image and the second image according to the output result, the first classification result, and the second classification result.

Further, when the acquiring module 51 acquires the first image and the second image, it is specifically configured to:

responding to an image input operation triggered by a user through an interactive interface, and acquiring the first image and the second image input by the user; or

And acquiring the first image and the second image which are acquired aiming at the target area at two different moments in response to target area input operation triggered by a user through an interactive interface.

Further, when the analysis module 52 performs difference analysis on the first image and the second image to obtain an output result containing difference information, it is specifically configured to: extracting characteristics of the first image to obtain at least one piece of first characteristic information; performing feature extraction on the second image to obtain at least one piece of second feature information; and determining an output result containing difference information according to the at least one first characteristic information and the at least one second characteristic information.

Further, when the analysis module 52 performs feature extraction on the first image to obtain at least one piece of first feature information, it is specifically configured to: performing feature extraction on the first image by using a first network model to obtain at least one piece of first feature information; the first network model comprises at least one first network layer, and any first network layer correspondingly outputs first characteristic information.

Further, when the analysis module 52 performs feature extraction on the second image to obtain at least one piece of second feature information, it is specifically configured to: performing feature extraction on the second image by using a second network model to obtain at least one piece of second feature information; the second network model comprises at least one second network layer, and any second network layer correspondingly outputs second characteristic information; the first network model and the second network model are twin network models with the same network structure and network parameters.

Further, the classification module 53 is specifically configured to, when performing classification processing on the image content in the first image to obtain a first classification result: and classifying the at least one piece of first characteristic information by using a fourth network model to obtain a first classification result.

Further, the fourth network model comprises at least one fourth network layer; the number of the fourth network layers contained in the fourth network model is the same as the number of the first network layers contained in the first network model, and an association relationship exists. Correspondingly, when the classification module 53 uses the fourth network model to classify the at least one piece of first feature information to obtain a first classification result, the classification module is specifically configured to:

according to the association relationship between the fourth network layer in the fourth network model and the first network layer in the first network model, inputting the first characteristic information corresponding to the first network layer with the association relationship into the corresponding fourth network layer, and executing the fourth network model to obtain the first classification result.

Further, when the classification module 53 performs classification processing on the image content in the second image to obtain a second classification result, the classification module is specifically configured to: classifying the at least one second characteristic information by using a fifth network model to obtain a second classification result; and the network structures and the network parameters of the fifth network model and the fourth network model are the same.

Fig. 16 shows a block diagram of a data processing apparatus according to an embodiment of the present application. As shown in fig. 16, the data processing apparatus includes: an acquisition module 61, an analysis module 62 and a classification module 63. The acquiring module 61 is configured to acquire a first image and a second image. The analysis module 62 is configured to perform difference analysis on the first image and the second image to obtain an output result containing difference information. The classification module 63 is configured to perform classification processing on the output result to obtain a difference between the first image and the second image.

Further, when the analysis module 62 performs difference analysis on the first image and the second image to obtain an output result containing difference information, it is specifically configured to: extracting characteristics of the first image to obtain at least one piece of first characteristic information; performing feature extraction on the second image to obtain at least one piece of second feature information; and determining the output result containing the difference information according to the at least one first characteristic information and the at least one second characteristic information.

Further, when performing feature extraction on the first image to obtain at least one piece of first feature information, the analysis module 62 is specifically configured to perform feature extraction on the first image by using a first network model to obtain at least one piece of first feature information; the first network model comprises at least one first network layer, and any first network layer correspondingly outputs first characteristic information.

Further, when the analysis module 62 performs feature extraction on the second image to obtain at least one piece of second feature information, the analysis module is specifically configured to: performing feature extraction on the second image by using a second network model to obtain at least one piece of second feature information; the second network model comprises at least one second network layer, and any second network layer correspondingly outputs second characteristic information; the first network model and the second network model are twin network models with the same network structure and network parameters.

Further, the third network model comprises at least one third network layer; the number of third network layers contained in the third network model is the same as the number of first network layers contained in the first network model, and an association relationship exists; the number of third network layers contained in the third network model is the same as the number of second network layers contained in the second network model, and an association relationship exists. Correspondingly, when determining the output result containing the difference information according to the at least one first feature information and the at least one second feature information, the analysis module 62 is specifically configured to:

and inputting first characteristic information corresponding to the first network layer and second characteristic information corresponding to the second network layer which have the association relationship into the corresponding third network layer according to the association relationship between the third network layer in the third network model and the first network layer in the first network model and the second network layer in the second network model respectively, and executing the third network model to obtain the difference comparison result.

Further, the classifying module 63 is specifically configured to, when classifying the output result to obtain a difference between the first image and the second image: taking the output result as an input of a multi-classification model, and executing the multi-classification model to output the difference of the first image and the second image; the multi-classification model comprises convolution layers, and the number of input channels of each convolution layer corresponds to the number of output results; the number of output channels of the convolutional layer is determined by the number of image content categories to be identified.

Fig. 17 shows a block diagram of a data processing apparatus according to an embodiment of the present application. As shown in fig. 17, the data processing apparatus includes: an acquisition module 71, a first feature extraction module 72, a second feature extraction module 73, and a determination module 74; wherein the content of the first and second substances,

the acquiring module 71 is configured to acquire a first image and a second image of the target area at different times;

the first feature extraction module 72 is configured to perform feature extraction on the first image to obtain at least one piece of first feature information;

the first feature extraction module 73 is further configured to perform feature extraction on the second image to obtain at least one piece of second feature information;

the determining module 74 is configured to determine a category reflecting a difference between the first image and the second image and a content of the difference image according to the at least one first feature information and the at least one second feature information.

In the technical scheme provided by this embodiment, feature extraction is performed on a first image and a second image respectively to obtain at least one piece of first feature information corresponding to the first image and at least one piece of second feature information corresponding to the second image; and determining a difference between the first image and the second image using a multi-classification model in combination with the at least one first feature information and the at least one second feature information. The multi-level feature fusion information has strong expression capability and higher value, and the classification accuracy of the image change detection categories can be effectively improved by utilizing the multi-level feature fusion information to classify the image change detection.

Further, the data processing apparatus provided in this embodiment further includes: a response module 75 and a display module 76, wherein the response module 75 is configured to obtain the first image and the second image input by the user in response to an image input event triggered by the user; the display module 76 is configured to display, on a display interface, a third image reflecting a difference between the first image and the second image and displaying different types of image contents in the difference in a differentiated manner.

Further, the first feature extraction module 72 is specifically configured to: performing feature extraction on the first image by using a first network model to obtain at least one piece of first feature information; the first network model comprises at least one first network layer, and any first network layer correspondingly outputs first characteristic information.

Further, the second feature extraction module 73 is specifically configured to: performing feature extraction on the second image by using a second network model to obtain at least one piece of second feature information; the second network model comprises at least one second network layer, and any second network layer correspondingly outputs second characteristic information; the first network model and the second network model are twin network models with the same network structure and network parameters.

Further, the determining module 74 includes: a feature analysis unit 741 and a multi-classification unit 742; wherein the content of the first and second substances,

the feature analysis unit 741 is configured to analyze the at least one first feature information and the at least one second feature information by using a third network model to obtain an output result;

the multi-classification unit 742 is configured to perform multi-class classification on the output result by using a multi-classification model to obtain a difference between the first image and the second image.

In the above, the multi-classification model includes convolution layers, and the number of input channels of the convolution layers corresponds to the number of output results; the number of output channels of the convolutional layer is determined by the number of image content categories to be identified. The third network model comprises at least one third network layer; the number of third network layers contained in the third network model is the same as that of the first network layers contained in the first network model, and an association relationship exists; the number of third network layers contained in the third network model is the same as the number of second network layers contained in the second network model, and an association relationship exists; accordingly, the feature analysis unit 741 is specifically configured to: and inputting the first characteristic information corresponding to the first network layer and the second characteristic information corresponding to the second network layer which have the association relationship to the corresponding third network layer according to the association relationship between the third network layer in the third network model and the first network layer in the first network model and the second network layer in the second network model respectively, and executing the third network model to obtain the output result.

Further, the determining module 74 further includes: a classification unit 743, where the classification unit 743 is configured to classify the output result by using a classification model to obtain a fourth image reflecting a difference between the first image and the second image.

Further, the determining module 74 is further specifically configured to: analyzing the at least one first characteristic information and the at least one second characteristic information by using a third network model to obtain an output result; classifying the at least one piece of first characteristic information by using a fourth network model to obtain a first classification result; classifying the at least one second characteristic information by using a fifth network model to obtain a second classification result; and obtaining a third image which reflects the difference between the first image and the second image and shows different image contents in a distinguishing manner according to the output result, the first classification result and the second classification result.

Further, the fourth network model and the fifth network model have the same structure and network parameters; the fourth network model comprises at least one fourth network layer; the number of fourth network layers contained in the fourth network model is the same as the number of first network layers contained in the first network model, and an association relationship exists; accordingly, when the determining module 74 uses the fourth network model to classify the at least one first feature information to obtain the first classification result, the following method may be specifically adopted: according to the incidence relation between a fourth network layer in a fourth network model and a first network layer in the first network model, inputting first characteristic information corresponding to the first network layer with the incidence relation to the corresponding fourth network layer, and executing the fourth network model to obtain classification results of different types of image contents in the first image.

Here, it should be noted that: the data processing apparatus provided in this embodiment may implement the technical solutions described in the data processing method embodiments, and the specific implementation principles of the modules or units may refer to the corresponding contents in the data processing method embodiments, which are not described herein again.

Fig. 18 is a block diagram illustrating a network system training apparatus according to another embodiment of the present application. As shown in fig. 18, the training device of the network system includes: a feature extraction module 81, an analysis module 82, a classification module 83, and an optimization module 84; wherein the content of the first and second substances,

the feature extraction module 81 is configured to perform feature extraction on the first sample image by using a first network model to obtain at least one piece of first sample feature information; the feature extraction module 501 is further configured to perform feature extraction on the second sample image by using a second network model to obtain at least one second sample feature information;

the analysis module 82 is configured to analyze the at least one first sample feature information and the at least one second sample feature information by using a third network model to obtain an output result;

the classification module 83 is configured to perform multi-class classification on the output result by using a multi-class model to obtain a multi-class classification result;

the optimizing module 84 is configured to optimize network parameters of at least one of the first network model, the second network model, the third network model, and the multi-class model according to the multi-class classification result and the multi-class sample label;

wherein the multi-class sample label, the first sample image, and the second sample image are associated.

According to the technical scheme provided by the embodiment of the application, the first image and the second image are respectively subjected to feature extraction to obtain at least one piece of first feature information corresponding to the first image and at least one piece of second feature information corresponding to the second image; then, analyzing the at least one first characteristic information and the at least one second characteristic information by using the third network model to obtain a corresponding output result; classifying the output result by utilizing the multi-classification model to obtain a multi-class classification result; parameters in each network model can be optimized based on the multi-class classification result and the multi-class sample label. The multi-level feature fusion information has strong expression capability and higher value, and the classification accuracy of the image change detection category can be effectively improved by utilizing the multi-level feature fusion information to classify the image change detection.

Further, the classification module 83 is further configured to classify the output result by using a two-classification model to obtain a two-class classification result; accordingly, the optimization module 84 is specifically configured to: determining a first loss function according to the double-class classification result and the double-class sample label; determining a second loss function according to the multi-class classification result and the multi-class sample label; optimizing network parameters of at least one of the first network model, the second network model, the third network model, and the second classification model based on the first loss function and the second loss function; wherein the dual-class sample label is associated with the first sample image and the second sample image.

Here, it should be noted that: the data processing apparatus provided in this embodiment may implement the technical solutions described in the data processing method embodiments, and the specific implementation principles of the modules or units may refer to the corresponding contents in the data processing method embodiments, which are not described herein again.

Fig. 19 is a block diagram illustrating a network system training apparatus according to another embodiment of the present application. As shown in fig. 19, the training device of the network system includes: a feature extraction module 91, an analysis module 92 and an optimization module 91; wherein the content of the first and second substances,

the feature extraction module 91 is configured to perform feature extraction on the first sample image by using the first network model to obtain at least one piece of first sample feature information; the feature extraction module 91 is further configured to perform feature extraction on the second sample image by using a second network model to obtain at least one second sample feature information;

the analysis module 92 is configured to analyze the at least one first sample feature information and the at least one second sample feature information by using a third network model to obtain an output result;

the optimizing module 91 is configured to optimize a network parameter of at least one of the first network model, the second network model, and the third network model according to the output result and the first sample label;

the analysis module 92 is further configured to analyze the at least one first sample feature information by using a fourth network model to obtain a first classification result;

the optimizing module 91 is further configured to optimize a network parameter of at least one of the first network model and the fourth network model according to the first classification result and the second sample label;

wherein the first sample label, second sample label, first sample image and second sample image are associated.

Further, the optimization module 91 is further configured to obtain a fifth network model based on the optimized fourth network model.

In the technical solution provided in the embodiment of the present application, the third network model is integrated to analyze the at least one first sample feature information and the at least one second sample feature information to obtain an output result and a first sample tag, so as to optimize the network parameters of at least one of the first network model, the second network model, and the third network model, thereby effectively improving the reliability of the output result and reducing the false detection rate. In addition, the network parameters of at least one of the first network model and the fourth network model are optimized based on the first classification result and the second sample label which are correspondingly output by the fourth network model, so that the classification accuracy of the fourth network model is improved.

Here, it should be noted that: the data processing apparatus provided in this embodiment may implement the technical solutions described in the data processing method embodiments, and the specific implementation principles of the modules or units may refer to the corresponding contents in the data processing method embodiments, which are not described herein again.

Fig. 20 shows a schematic structural diagram of an electronic device according to an embodiment of the present application. As shown in fig. 20, the electronic apparatus includes: a memory 1001 and a processor 1002. The memory 1001 may be configured to store other various data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device. The memory 1001 may be implemented by any type of volatile or non-volatile memory device or combination thereof, such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disk.

The processor 1002, coupled to the memory 1001, is configured to execute the program stored in the memory 1001, so as to:

acquiring a first image and a second image;

performing difference analysis on the first image and the second image to obtain an output result containing difference information;

classifying the image content in the first image to obtain a first classification result;

classifying the image content in the second image to obtain a second classification result;

and determining the difference between the first image and the second image according to the output result, the first classification result and the second classification result.

When the processor 1002 executes the program in the memory 1001, the processor 1002 may implement other functions in addition to the above functions, which may be specifically referred to the description of the foregoing embodiments.

Further, as shown in fig. 20, the electronic apparatus further includes: communication component 1003, display 1004, power component 1005, and other components. Only some of the components are schematically shown in fig. 20, and it is not meant that the electronic device includes only the components shown in fig. 20.

Another embodiment of the present application provides an electronic device, which has the same structure as fig. 20. Specifically, the electronic device includes: a memory and a processor. The memory is used for storing programs. The processor, coupled with the memory, to execute the program stored in the memory to: acquiring a first image and a second image; performing difference analysis on the first image and the second image to obtain an output result containing difference information; and classifying the output result to obtain the difference between the first image and the second image.

When the processor executes the program in the memory, the processor may implement other functions in addition to the above functions, which may be specifically referred to the description of the foregoing embodiments.

Another embodiment of the present application provides an electronic device, which has the same structure as fig. 20. Specifically, the electronic device includes: a memory and a processor. The memory is used for storing programs. The processor, coupled with the memory, to execute the program stored in the memory to:

acquiring a first image and a second image;

performing feature extraction on the first image to obtain at least one piece of first feature information;

performing feature extraction on the second image to obtain at least one piece of second feature information;

and determining the difference between the first image and the second image and the category of the content of the difference image according to the at least one piece of first characteristic information and the at least one piece of second characteristic information.

When the processor executes the program in the memory, the processor may implement other functions in addition to the above functions, which may be specifically referred to the description of the foregoing embodiments.

Another embodiment of the present application provides an electronic device, which has the same structure as fig. 20. Specifically, the electronic device includes: a memory and a processor. The memory is used for storing programs. The processor, coupled with the memory, to execute the program stored in the memory to:

responding to an image comparison event triggered by a user through an interactive interface, and acquiring a first image and a second image designated by the user in the image comparison event;

identifying the difference between the first image and the second image and the category of the content of the difference image to obtain an identification result;

generating a third image which reflects the difference between the first image and the second image and shows the content of different types of images in the difference in a distinguishing way based on the identification result;

controlling the display to display the third image.

When the processor executes the program in the memory, the processor may implement other functions in addition to the above functions, which may be specifically referred to the description of the foregoing embodiments.

Accordingly, embodiments of the present application also provide a computer readable storage medium storing a computer program, which when executed by a computer, can implement the steps or functions of the data processing method provided in the foregoing embodiments.

Another embodiment of the present application provides an electronic device, which has the same structure as fig. 20. Specifically, the electronic device includes: a memory and a processor. The memory is used for storing programs. A processor, coupled with the memory, for executing the program stored in the memory to: performing feature extraction on the first sample image by using a first network model to obtain at least one piece of first sample feature information; performing feature extraction on the second sample image by using a second network model to obtain at least one second sample feature information; analyzing the at least one first sample characteristic information and the at least one second sample characteristic information by using a third network model to obtain an output result; performing multi-class classification on the output result by using a multi-classification model to obtain a multi-class classification result; optimizing network parameters of at least one of the first network model, the second network model, the third network model and the multi-classification model according to the multi-class classification result and the multi-class sample label; wherein the multi-class sample label, the first sample image, and the second sample image are associated.

When the processor executes the program in the memory, the processor may implement other functions in addition to the above functions, which may be specifically referred to the description of the foregoing embodiments.

Another embodiment of the present application provides an electronic device, which has the same structure as fig. 20. Specifically, the electronic device includes: a memory and a processor. The memory is used for storing programs. A processor, coupled with the memory, for executing the program stored in the memory for: performing feature extraction on the first sample image by using a first network model to obtain at least one piece of first sample feature information; performing feature extraction on the second sample image by using a second network model to obtain at least one second sample feature information; performing difference analysis on the at least one first sample characteristic information and the at least one second sample characteristic information by using a third network model to obtain an output result; optimizing network parameters of at least one of the first network model, the second network model and the third network model according to the output result and a first sample label; analyzing the at least one first sample characteristic information by using a fourth network model to obtain a first classification result; optimizing network parameters of at least one of the first network model and the fourth network model according to the first classification result and a second sample label; wherein the first sample label, second sample label, first sample image and second sample image are associated.

When the processor executes the program in the memory, the processor may implement other functions in addition to the above functions, which may be specifically referred to the description of the foregoing embodiments.

Accordingly, embodiments of the present application also provide a computer-readable storage medium storing a computer program, where the computer program can implement the steps or functions of the network system training method provided in the foregoing embodiments when executed by a computer.

The technical scheme provided by each embodiment of the application can be applied to the comparison task of remote sensing images in different periods in the same region, for example, newly added building detection in a certain region is the most common one, and the method is mainly used in homeland management law enforcement. Besides, the method can also be used for ecological environment management (such as agricultural land and forest land change supervision), disaster degree evaluation, marine offshore area management, urban planning and other scenes. The technical solution provided by the present application will be further described below with reference to city planning as an example.

Firstly, acquiring city planning drawings of two different planning schemes aiming at a new city area; then, the scheme provided by each embodiment of the application can be adopted to perform difference detection on the city planning drawings of two different planning schemes, so as to obtain the difference between the two city planning drawings. And planning workers can discuss the difference part only through the difference of the two city planning drawings so as to further optimize the city planning scheme and help city planning.

Or, acquiring an urban area planning map for a certain old urban area of the city; the scheme provided by the embodiments of the application is utilized to carry out difference detection on the urban planning map and the urban map of the old urban area in the current situation, so as to obtain the difference. Planners may focus on the differences to further optimize the old urban planning scheme.

In another application scenario, as exemplified below, the technical solutions provided by the embodiments of the present application can also be used. That is, an image of a certain object (such as a region, an object or a person) is acquired periodically, and the acquired image can be stored in a database for archiving; in the archiving process, the technical scheme provided by each embodiment of the application can be utilized to compare the image acquired this time with the image acquired by the object in the database at the previous time, and if the comparison results are the same, the image acquired this time can not be archived; if the comparison result is different, the image acquired this time is archived while outputting the difference (such as the place of the difference and which image content or some types of images are changed).

Certainly, in some cases, when the image acquired this time is archived, the image acquired last time can be deleted; namely, the image acquired this time is used for replacing the image acquired last time.

The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.

Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method according to the embodiments or some parts of the embodiments.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and are not limited thereto; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions in the embodiments of the present application.

Claims (42)

1. A data processing method, comprising:

acquiring a first image and a second image;

performing difference analysis on the first image and the second image to obtain an output result containing difference information;

classifying the image content in the first image to obtain a first classification result;

classifying the image content in the second image to obtain a second classification result;

and determining the difference between the first image and the second image according to the output result, the first classification result and the second classification result.

2. The method of claim 1, wherein acquiring the first image and the second image comprises:

responding to an image input operation triggered by a user through an interactive interface, and acquiring the first image and the second image input by the user; or

And acquiring the first image and the second image which are acquired aiming at the target area at two different moments in response to target area input operation triggered by a user through an interactive interface.

3. The method of claim 1 or 2, wherein performing a difference analysis on the first image and the second image to obtain an output result containing difference information comprises:

performing feature extraction on the first image to obtain at least one piece of first feature information;

performing feature extraction on the second image to obtain at least one piece of second feature information;

and determining an output result containing difference information according to the at least one first characteristic information and the at least one second characteristic information.

4. The method of claim 3, wherein extracting features from the first image to obtain at least one first feature information comprises:

performing feature extraction on the first image by using a first network model to obtain at least one piece of first feature information;

the first network model comprises at least one first network layer, and any first network layer correspondingly outputs first characteristic information.

5. The method of claim 4, wherein performing feature extraction on the second image to obtain at least one second feature information comprises:

performing feature extraction on the second image by using a second network model to obtain at least one piece of second feature information;

the second network model comprises at least one second network layer, and any second network layer correspondingly outputs second characteristic information; the first network model and the second network model are twin network models with the same network structure and network parameters.

6. The method of claim 5, wherein classifying the image content in the first image to obtain a first classification result comprises:

and classifying the at least one piece of first characteristic information by using a fourth network model to obtain a first classification result.

7. The method of claim 6, wherein the fourth network model comprises at least one fourth network layer; the number of fourth network layers contained in the fourth network model is the same as the number of first network layers contained in the first network model, and an association relationship exists; and

classifying the at least one piece of first feature information by using a fourth network model to obtain a first classification result, including:

according to the incidence relation between the fourth network layer in the fourth network model and the first network layer in the first network model, inputting the first characteristic information corresponding to the first network layer with the incidence relation to the corresponding fourth network layer, and executing the fourth network model to obtain the first classification result.

8. The method of claim 7, wherein classifying the image content in the second image to obtain a second classification result comprises:

classifying the at least one second characteristic information by using a fifth network model to obtain a second classification result;

and the network structures and the network parameters of the fifth network model and the fourth network model are the same.

9. A data processing method, comprising:

acquiring a first image and a second image;

performing difference analysis on the first image and the second image to obtain an output result containing difference information;

and classifying the output result to obtain the difference between the first image and the second image.

10. The method of claim 9, wherein performing a difference analysis on the first image and the second image to obtain an output result containing difference information comprises:

performing feature extraction on the first image to obtain at least one piece of first feature information;

performing feature extraction on the second image to obtain at least one piece of second feature information;

and determining the output result containing the difference information according to the at least one first characteristic information and the at least one second characteristic information.

11. The method of claim 10, wherein extracting features from the first image to obtain at least one first feature information comprises:

performing feature extraction on the first image by using a first network model to obtain at least one piece of first feature information;

the first network model comprises at least one first network layer, and any first network layer correspondingly outputs first characteristic information.

12. The method of claim 11, wherein performing feature extraction on the second image to obtain at least one second feature information comprises:

performing feature extraction on the second image by using a second network model to obtain at least one piece of second feature information;

the second network model comprises at least one second network layer, and any second network layer correspondingly outputs second characteristic information; the first network model and the second network model are twin network models with the same network structure and network parameters.

13. The method of claim 12, wherein the third network model comprises at least one third network layer; the number of third network layers contained in the third network model is the same as that of the first network layers contained in the first network model, and an association relationship exists; the number of third network layers contained in the third network model is the same as the number of second network layers contained in the second network model, and an association relationship exists; and

determining the output result containing the difference information according to the at least one first feature information and the at least one second feature information, including:

and inputting first characteristic information corresponding to the first network layer and second characteristic information corresponding to the second network layer which have the association relationship into the corresponding third network layer according to the association relationship between the third network layer in the third network model and the first network layer in the first network model and the second network layer in the second network model respectively, and executing the third network model to obtain the difference comparison result.

14. The method according to any one of claims 9 to 13, wherein classifying the output result to obtain a difference between the first image and the second image comprises:

taking the output result as an input of a multi-classification model, and executing the multi-classification model to output the difference of the first image and the second image;

generating a third image which reflects the difference between the first image and the second image and shows the content of different types of images in the difference in a distinguishing way based on the difference between the first image and the second image;

the multi-classification model comprises convolution layers, and the number of input channels of each convolution layer corresponds to the number of output results; the number of output channels of the convolutional layer is determined by the number of image content categories to be identified.

15. A data processing method, comprising:

acquiring a first image and a second image;

performing feature extraction on the first image to obtain at least one piece of first feature information;

performing feature extraction on the second image to obtain at least one piece of second feature information;

and determining the difference between the first image and the second image and the category of the content of the difference image according to the at least one piece of first characteristic information and the at least one piece of second characteristic information.

16. The method of claim 15, further comprising:

acquiring the first image and the second image input by a user in response to an image input event triggered by the user;

and displaying a third image reflecting the difference between the first image and the second image and differentially displaying different types of image contents in the difference on a display interface.

17. The method according to claim 15 or 16, wherein determining the difference between the first image and the second image and the category of the content of the difference image according to the at least one first feature information and the at least one second feature information comprises:

performing difference analysis on the at least one first characteristic information and the at least one second characteristic information by using a third network model to obtain an output result containing difference information;

and carrying out multi-class classification on the output result by using a multi-classification model to obtain the class of the difference image content.

18. The method of claim 17, further comprising:

and classifying the output result by using a binary model to obtain a fourth image reflecting the difference between the first image and the second image.

19. The method according to claim 15 or 16, wherein determining the difference between the first image and the second image and the category of the content of the difference image according to the at least one first feature information and the at least one second feature information comprises:

performing difference analysis on the at least one first characteristic information and the at least one second characteristic information by using a third network model to obtain an output result containing difference information;

classifying the at least one piece of first characteristic information by using a fourth network model to obtain a first classification result;

classifying the at least one second characteristic information by using a fifth network model to obtain a second classification result;

and obtaining the difference between the first image and the second image and the category of the content of the difference image according to the output result, the first classification result and the second classification result.

20. A data processing method, characterized by further comprising:

responding to an image comparison event triggered by a user through an interactive interface, and acquiring a first image and a second image designated by the user in the image comparison event;

identifying the difference between the first image and the second image and the category of the content of the difference image to obtain an identification result;

generating a third image which reflects the difference between the first image and the second image and shows the content of different types of images in the difference in a distinguishing way based on the identification result;

and displaying the third image.

21. The method of claim 20, further comprising:

responding to the operation that a user inputs a first image and a second image through the interactive interface, and triggering the image comparison event; or

Responding to an operation that a user inputs a target area through an interactive interface, and triggering an image comparison event aiming at the target area; and the first image and the second image designated by the user in the image comparison event are images corresponding to the target area at different moments.

22. The method according to claim 20 or 21, wherein identifying the difference between the first image and the second image and the category of the content of the difference image, and obtaining the identification result comprises:

performing difference analysis on the first image and the second image to obtain an output result containing difference information; classifying the output result to obtain an identification result containing difference information and a difference image content category; or

Performing difference analysis on the first image and the second image to obtain an output result containing difference information; classifying the image content in the first image to obtain a first classification result; classifying the image content in the second image to obtain a second classification result; and obtaining an identification result containing difference information and a difference image content category according to the output result, the first classification result and the second classification result.

23. A network system, comprising:

the first network model comprises at least one first network layer and is used for extracting the characteristics of the first image and outputting at least one piece of first characteristic information;

the second network model comprises at least one second network layer and is used for extracting the characteristics of the second image and outputting at least one piece of second characteristic information;

the third network model is connected with the output ends of the first network model and the second network model and used for carrying out difference analysis on the at least one first characteristic information and the at least one second characteristic information to obtain an output result containing difference information;

the fourth network model is connected with the first network model and used for classifying the at least one piece of first characteristic information to obtain a first classification result;

the fifth network model is connected with the second network model and used for classifying the at least one piece of second characteristic information to obtain a second classification result;

and the output model is connected with the third network model, the fourth network model and the fifth network model and is used for outputting the difference between the first image and the second image according to the output result, the first classification result and the second classification result.

24. The network system of claim 23, wherein the fourth network model comprises at least one fourth network layer; the number of fourth network layers contained in the fourth network model is the same as the number of first network layers contained in the first network model, and an association relationship exists; and

the fourth network model is specifically configured to input, according to an association relationship between a fourth network layer in the fourth network model and a first network layer in the first network model, first feature information corresponding to the first network layer in the association relationship to the corresponding fourth network layer, and execute the fourth network model to obtain classification results of different types of image contents in the first image.

25. The network system according to claim 23 or 24, wherein the fourth network model and the fifth network model have the same structure and network parameters.

26. A method for training a network system, comprising:

performing feature extraction on the first sample image by using a first network model to obtain at least one piece of first sample feature information;

performing feature extraction on the second sample image by using a second network model to obtain at least one second sample feature information;

performing difference analysis on the at least one first sample characteristic information and the at least one second sample characteristic information by using a third network model to obtain an output result;

optimizing network parameters of at least one of the first network model, the second network model and the third network model according to the output result and the first sample label;

analyzing the at least one first sample characteristic information by using a fourth network model to obtain a first classification result;

optimizing network parameters of at least one of the first network model and the fourth network model according to the first classification result and a second sample label;

wherein the first sample label, second sample label, first sample image and second sample image are associated.

27. The method of claim 26, further comprising:

and obtaining a fifth network model based on the optimized fourth network model.

28. A network system, comprising:

the first network model comprises at least one first network layer and is used for extracting the characteristics of the first image and outputting at least one piece of first characteristic information;

the second network model comprises at least one second network layer and is used for carrying out feature extraction on the second image to obtain at least one second feature information;

the third network model is connected with the output ends of the first network model and the second network model and used for carrying out difference analysis on at least one piece of first characteristic information and at least one piece of second characteristic information to obtain an output result;

and the multi-classification model is connected with the third network model and used for carrying out multi-class classification on the output result to obtain the difference between the first image and the second image.

29. The network system of claim 28, wherein the multi-class model comprises convolutional layers, the number of input channels of the convolutional layers corresponding to the number of output results; the number of output channels of the convolutional layer is determined by the number of image content categories to be identified.

30. The network system according to claim 28 or 29, further comprising:

and the classification model is connected with the third network model and used for classifying the output result to obtain the difference between the first image and the second image.

31. A method for training a network system, comprising:

performing feature extraction on the first sample image by using a first network model to obtain at least one piece of first sample feature information;

performing feature extraction on the second sample image by using a second network model to obtain at least one second sample feature information;

analyzing the at least one first sample characteristic information and the at least one second sample characteristic information by using a third network model to obtain an output result;

performing multi-class classification on the output result by using a multi-classification model to obtain multi-class classification results;

optimizing network parameters of at least one of the first network model, the second network model, the third network model and the multi-classification model according to the multi-class classification result and the multi-class sample label;

wherein the multi-class sample label, the first sample image, and the second sample image are associated.

32. The method of claim 31, further comprising:

classifying the output result by using a two-classification model to obtain a two-classification result; and

optimizing network parameters of at least one of the first network model, the second network model, the third network model and the multi-classification model according to the multi-classification result and the multi-classification sample label, including:

determining a first loss function according to the double-class classification result and the double-class sample label;

determining a second loss function according to the multi-class classification result and the multi-class sample label;

optimizing network parameters of at least one of the first network model, the second network model, the third network model, and the second classification model based on the first loss function and the second loss function;

wherein the dual-class sample label is associated with the first sample image and the second sample image.

33. A data processing system, comprising:

the image acquisition equipment is used for acquiring a first image and a second image of a target area corresponding to two different moments;

the processing device is used for acquiring the first image and the second image, performing difference analysis on the first image and the second image and obtaining an output result containing difference information; classifying the image content in the first image to obtain a first classification result; classifying the image content in the second image to obtain a second classification result; and determining the difference between the first image and the second image according to the output result, the first classification result and the second classification result.

34. A data processing system, comprising:

the image acquisition equipment is used for acquiring a first image and a second image of a target area corresponding to two different moments;

the processing device is used for acquiring the first image and the second image, performing difference analysis on the first image and the second image and obtaining an output result containing difference information; and classifying the output result to obtain the difference between the first image and the second image.

35. A data processing system, comprising:

the image acquisition equipment is used for acquiring a first image and a second image of a target area corresponding to two different moments;

the processing device is used for acquiring the first image and the second image, and performing feature extraction on the first image to obtain at least one piece of first feature information; performing feature extraction on the second image to obtain at least one piece of second feature information; and determining the difference between the first image and the second image and the category of the content of the difference image according to the at least one piece of first characteristic information and the at least one piece of second characteristic information.

36. A remote sensing device, comprising:

the remote sensing device is used for acquiring a first remote sensing image and a second remote sensing image of a target area at two different moments;

the processor is used for acquiring the first remote sensing image and the second remote sensing image, and performing difference analysis on the first remote sensing image and the second remote sensing image to obtain an output result containing difference information; classifying the image content in the first remote sensing image to obtain a first classification result; classifying the image content in the second remote sensing image to obtain a second classification result; and determining the difference between the first remote sensing image and the second remote sensing image according to the output result, the first classification result and the second classification result.

37. A remote sensing device, comprising:

the remote sensing device is used for acquiring a first remote sensing image and a second remote sensing image of a target area at two different moments;

the processor is used for acquiring the first remote sensing image and the second remote sensing image, and performing difference analysis on the first remote sensing image and the second remote sensing image to obtain an output result containing difference information; and classifying the output result to obtain the difference between the first remote sensing image and the second remote sensing image.

38. A remote sensing device, comprising:

the remote sensing device is used for acquiring a first remote sensing image and a second remote sensing image of a target area at two different moments;

the processor is used for acquiring the first remote sensing image and the second remote sensing image, and extracting the characteristics of the first image to obtain at least one piece of first characteristic information; performing feature extraction on the second image to obtain at least one piece of second feature information; and determining a third image reflecting the difference between the first image and the second image and displaying different types of image contents in the difference according to the at least one piece of first characteristic information and the at least one piece of second characteristic information.

39. An electronic device, comprising: a memory and a processor, wherein,

the memory is used for storing programs;

the processor, coupled with the memory, to execute the program stored in the memory to:

acquiring a first image and a second image;

performing difference analysis on the first image and the second image to obtain an output result containing difference information;

classifying the image content in the first image to obtain a first classification result;

classifying the image content in the second image to obtain a second classification result;

and determining the difference between the first image and the second image according to the output result, the first classification result and the second classification result.

40. An electronic device, comprising: a memory and a processor, wherein,

the memory is used for storing programs;

the processor, coupled with the memory, to execute the program stored in the memory to:

acquiring a first image and a second image;

performing difference analysis on the first image and the second image to obtain an output result containing difference information;

and classifying the output result to obtain the difference between the first image and the second image.

41. An electronic device, comprising: a memory and a processor, wherein,

the memory is used for storing programs;

the processor, coupled with the memory, to execute the program stored in the memory to:

acquiring a first image and a second image;

performing feature extraction on the first image to obtain at least one piece of first feature information;

performing feature extraction on the second image to obtain at least one piece of second feature information;

and determining the difference between the first image and the second image and the category of the content of the difference image according to the at least one piece of first characteristic information and the at least one piece of second characteristic information.

42. A display device, comprising: a memory, a processor, and a display, wherein,

the memory is used for storing programs;

the processor, coupled with the memory, to execute the program stored in the memory to:

responding to an image comparison event triggered by a user through an interactive interface, and acquiring a first image and a second image designated by the user in the image comparison event;

identifying the difference between the first image and the second image and the image content of different types in the difference to obtain an identification result;

generating a third image which reflects the difference between the first image and the second image and shows the content of different types of images in the difference in a distinguishing way based on the identification result;

controlling the display to display the third image.

Images