CN117788302A - Mapping graphic processing system - Google Patents

Abstract

The application discloses survey and drawing graphic processing system relates to the survey and drawing image processing field, include: preprocessing the mapping image, including time synchronization and spatial calibration; carrying out Kalman filtering treatment on the preprocessed mapping image, and fusing mapping images of different data sources; multitask learning is carried out on the mapping images after the fusion processing, and an image analysis model is generated; and adopting asynchronous communication and message queues to transmit and process the mapping image. Aiming at the problem of low mapping image processing precision in the prior art, the method and the device adopt preprocessing of time synchronization and space calibration to ensure consistency of image data. Secondly, images of different data sources are fused through Kalman filtering processing, and the methods of wide area coordinate conversion, lattice coordinate conversion, maika projection and the like are adopted in the coordinate conversion and projection conversion stages through time axis calibration and time stamp calculation, so that the consistency and accuracy of the images in a space coordinate system are enhanced, and the processing precision of mapping images is improved.

Description

Mapping graphic processing system

Technical Field

The present application relates to the field of mapping image processing, and in particular, to a mapping graphics processing system.

Background

With the continued development of geographic information technology, acquisition and processing of mapping images is becoming increasingly important in many areas. However, in the prior art, mapping image processing has problems of low precision, inconsistent data, discontinuous time and the like, and limits the accuracy and reliability of geographic information application. In particular, where multiple data sources are involved, how to efficiently integrate and process these heterogeneous data, improving the accuracy of the mapping image, is an urgent issue to be addressed.

Traditional mapping image processing methods often have bottlenecks in time synchronization, spatial calibration, data fusion, analysis, and the like. The problems of inconsistent time axis, coordinate conversion error, asynchronous data sources and the like affect the comprehensive performance of the geographic information system.

In the related art, for example, chinese patent document CN116303600a provides a basic mapping data integration method and system, which relate to the technical field of mapping data integration. In the invention, aiming at each mapping data acquisition terminal device, the to-be-processed mapping data corresponding to the mapping data acquisition terminal device is obtained, and the feature extraction processing is carried out on the to-be-processed mapping data to obtain a feature tag set corresponding to the to-be-processed mapping data; for every two pieces of mapping data to be processed, calculating the similarity between feature tag sets corresponding to the two pieces of mapping data to be processed to obtain the set similarity corresponding to the two pieces of mapping data to be processed; and classifying the plurality of pieces of mapping data to be processed based on the set similarity corresponding to every two pieces of mapping data to be processed in the plurality of pieces of mapping data to be processed, so as to obtain at least one mapping data classification set corresponding to the plurality of pieces of mapping data to be processed. However, the scheme only uses the feature labels to carry out simple feature expression, which can result in serious semantic information loss of the mapping image, and the extracted features are not strong in representativeness.

Disclosure of Invention

1. Problems to be solved

Aiming at the problem of low mapping image processing precision in the prior art, the application provides a mapping graph processing system, which improves the processing precision of mapping images through Kalman filtering, multi-task learning and the like.

2. Technical proposal

The aim of the application is achieved by the following technical scheme.

Embodiments of the present description provide a mapping graphics processing system, comprising: preprocessing the mapping image, including time synchronization and spatial calibration; carrying out Kalman filtering treatment on the preprocessed mapping image, and fusing mapping images of different data sources; multitask learning is carried out on the mapping images after the fusion processing, and an image analysis model is generated; and adopting asynchronous communication and message queues to transmit and process the mapping image.

The time synchronization refers to aligning time axes of mapping images of different sources to a unified time coordinate system, so that shooting time points of different mapping images are in one-to-one correspondence, and the problem of inconsistent time axes caused by different time stamp references of different platforms is solved. In the application, the time stamp information in the image metadata is extracted and unified into a unified time system such as GPS time or coordinated universal time; determining the time stamp of the image without the time stamp by carrying out spline interpolation on the time relationship between the image without the time stamp and other images with the time stamp; the multi-source heterogeneous images are reordered according to the time stamps so that their time axes are consistent.

The spatial calibration refers to converting coordinate systems and projection modes of mapping images from different sources, so that the mapping images are projected under a uniform geographic coordinate system, and the problem of spatial misalignment caused by different coordinate references of different platforms is solved. In the application, wide-area coordinate conversion is carried out, and each image coordinate is converted into a uniform geographic coordinate system; performing lattice coordinate conversion to eliminate the influence of different coordinate lattices; and carrying out projection conversion such as Michator and the like to realize coordination of different projection modes.

The Kalman filtering is a recursive filtering algorithm, and the system state is recursively estimated and updated by constructing a process model and an observation model. In the present application, a process model is set upDescribing the evolution of the system state; setting an observation model->Describing the relationship of the measurement to the state; setting covariance matrix->、/>Representing process noise and observation noise; recursively calculating the state estimator using the Kalman filtering algorithm>The method comprises the steps of carrying out a first treatment on the surface of the Estimating +.>And carrying out adaptive fusion of image contents. Compared with simple pixel-level operation, the content self-adaptive fusion based on Kalman filtering state estimation can deeply mine the inherent correlation of different source images, and achieve a better fusion effect.

Wherein, multitasking is a machine learning method that allows a plurality of related tasks to promote each other. In the application, various characteristics such as text, spectrum, space and the like are extracted from the fused images. A multi-tasking convolutional neural network is constructed that includes a shared layer and a tasking layer. Multiple image parsing tasks are jointly trained to facilitate each other in back propagation. Finally, a multi-angle image analysis model of texts, spectrums, spaces and the like is obtained. Kafka: high throughput, low latency, high scalability distributed message queues; rabbit MQ: developed by Erlang language, has excellent performance and is robust and stable; active MQ: pure Java implementation, cross-language and cross-platform open source message middleware. Preferably, a combination of Web socket+Kafka is adopted to carry out asynchronous communication and distributed message queue processing, and the transmission efficiency and the expandability of the combination are strong.

The asynchronous communication is a communication mode, and allows the sender and the receiver of the information to avoid the need of synchronously waiting for each other, thereby improving the communication efficiency. Message queues are a mechanism by which messages are stored and transmitted through a queue. Specifically, web Socket may be used: the system is used for connecting a client and a server and providing full duplex asynchronous communication; MQTT: lightweight asynchronous communication protocol based on publish-subscribe mode; the AMQP, advanced message queuing protocol, provides asynchronous message transmission.

Specifically, the mapping image is preprocessed, so that the consistency of the data in time and space is ensured. This involves time synchronizing the images so that they are aligned on the time axis and spatially aligning to ensure consistency of the images in different coordinate systems. And introducing Kalman filtering processing to the preprocessed mapping image, and fusing images of different data sources by utilizing the superior performance of the Kalman filtering processing in time series data fusion. The Kalman filtering improves the accuracy of the fused image by effectively considering the past observation and the dynamic evolution of the model. And performing multitask learning on the fused mapping images. And constructing the multi-task convolutional neural network by extracting text features, spectrum features and space features of the image. The network comprises a shared coding layer, a task coding layer and a task decoding layer, and depth characteristics of images are obtained through training to generate an image analysis model. Asynchronous communication and message queues are adopted to improve the transmission efficiency and processing speed of the mapping image. This includes Web Socket and Kafka based communication mechanisms, ensuring that the system is able to efficiently transmit and process large amounts of image data.

Further, according to the time stamps of the data sources of different mapping images, performing time axis calibration; extracting metadata containing time information according to a data source of the mapping image to serve as a time stamp of the mapping image; comparing the time stamps of the mapping images, judging whether the time intervals are continuous and consistent, and if not, recording discontinuous points of the time intervals; calculating an insertion time stamp at the discontinuous points of the time interval by using a linear interpolation method, and filling the discontinuous points of the time interval; rearranging the sequence of mapping images according to the time stamp information calculated after filling so as to synchronize the images of different data sources on a time axis;

The linear interpolation method is a numerical interpolation method, and a smooth interpolation curve is fitted by adding new points between given two points by using a linear function. In the present application, a time stamp discontinuity of the interval points is detected on the time axis; taking two nearest effective timestamp points before and after the interval as control points; calculating new timestamp points to be inserted in the interval in a linear proportion mode according to the two control points; the calculated time stamps are inserted so that the interval time axes are continuous. Specifically, it is possible to employ: lagrange polynomial Lagrange interpolation is carried out by utilizing a plurality of points to carry out interpolation, a higher-order Lagrange interpolation polynomial is fitted, and higher precision can be obtained. The Newton interpolation method utilizes a plurality of points to fit Newton interpolation polynomials, and utilizes a recurrence relation to perform rapid calculation, thereby having high precision and rapid calculation speed. And the three linear interpolation is carried out on three points in total on the line segments formed by two adjacent known points by taking one point, so that the smoothness is higher.

Specifically, first, metadata including time information is extracted as a time stamp of a survey image from data sources of different survey images. This ensures that the image of each data source has its corresponding time stamp. And comparing the extracted time stamp with the time stamp of the adjacent image to judge whether the time intervals are consistent continuously. If there are discontinuities, the time intervals between the time stamps are not consistent, the locations of the discontinuities are recorded. On the identified discontinuity points, the insertion time stamp is calculated by adopting a linear interpolation method, and the discontinuity points of the time interval are filled. By linear interpolation, missing time stamps can be estimated from the information of known time stamps, so that the time intervals are continuously consistent. The order of mapping images is rearranged based on the time stamp information calculated after filling to ensure that the images of the different data sources are synchronized on the time axis. In this way, even images from different data sources can be effectively compared and analyzed in the time dimension. The time axis calibration of the mapping image is realized, and the consistency of images of different data sources in the time dimension is ensured. The time stamp information is extracted, compared, filled and rearranged, so that the condition of discontinuous time intervals is effectively processed, and the time continuity and accuracy in the image processing process are ensured.

Further, taking the mapping image without time metadata as the mapping image which cannot directly correspond to the time stamp; respectively acquiring N mapping images with time stamps before and after the mapping images which cannot directly correspond to the time stamps, and taking the N mapping images with the time stamps as control points; and calculating the time stamp of the mapping image which cannot directly correspond to the time stamp through a cubic spline interpolation function according to the acquired control point.

The cubic spline interpolation function is a smooth interpolation method, and a cubic polynomial is constructed by using the idea that a polynomial approaches in sequence according to a certain sequence so as to realize smooth function interpolation. In the application, N images with time stamps are taken as control points before and after the image with the missing time stamp; constructing a cubic spline interpolation function according to the N control points; and carrying the position corresponding to the image with the missing time stamp into spline interpolation function calculation to obtain the missing time stamp. Compared with linear interpolation, the cubic spline interpolation function can construct a smoother interpolation curve by utilizing a plurality of control points, so that more accurate and more continuous time stamp supplementation is realized, and the effect of time axis synchronization is improved.

In particular, for those mapping images that do not contain time metadata, they are identified as images that cannot directly correspond to a timestamp because of the lack of direct time information. For each mapping image which cannot directly correspond to the time stamp, N mapping images with the time stamp are acquired before and after the mapping image as control points. These control points provide information of known time stamps for subsequent interpolation calculations. And performing interpolation calculation by using the obtained control points and adopting a cubic spline interpolation function. Cubic spline interpolation maintains a certain smoothness and continuity at the interpolation points by using a piecewise low order polynomial. In this way, for a mapping image that cannot directly correspond to a time stamp, an estimated time stamp is obtained by interpolation calculation. Those map images that do not contain time metadata are processed so that they can also obtain an approximate timestamp. And performing time stamp estimation on the images without the time stamp by using the previous and subsequent mapping images with the time stamp as control points and using a cubic spline interpolation function.

Further, coordinate conversion and projection conversion are carried out on the mapping image after the time axis calibration; the coordinate conversion adopts wide area coordinate conversion; the projection conversion adopts a Michaet projection.

The wide-area coordinate conversion is to carry out mathematical conversion on coordinates of different coordinate systems so as to realize unification of the coordinates. In the application, extracting longitude and latitude coordinate values of an image as source coordinates; converting the source coordinates to a target coordinate system through a Moloden sky model; the conversion between coordinate systems of WGS84, CGCS2000 and the like is realized. The Maka bracket projection is a map projection method with a guaranteed angle and equal product. In the application, determining a Maka bracket projection parameter according to a coordinate range; calculating plane coordinates by using a Maka support projection formula; resampling the image to realize projection conversion; and unifying the projection modes of different images.

Specifically, extracting image time metadata, converting to a unified time system, giving time stamps to the non-time metadata images by using an interpolation method, and rearranging the images according to the time stamp order to realize time axis alignment. And extracting longitude and latitude coordinates in the image as source coordinates, and converting the source coordinates into a target coordinate system by utilizing a Moloden sky model to realize coordinate system conversion from WGS84 to CGCS2000 and the like. And determining a Maillard projection area and parameters according to the latitude and longitude range, calculating plane coordinates for each image point by using a Maillard projection formula, resampling the images according to the plane coordinates, and realizing unified projection conversion of different data sources. The time axis calibration solves the problem of inconsistent time of the multi-source heterogeneous images, the coordinate conversion eliminates the influence of different coordinate systems, the projection conversion coordinates different map projection differences, and finally the time and space alignment of the multi-source mapping images is realized.

Further, the coordinate transformation adopts wide-area coordinate transformation, including: analyzing a data source of the mapping image, and extracting longitude and latitude coordinates of the mapping image as source coordinates; converting the acquired source coordinates into coordinate expressions of Gaussian plane coordinates or geodetic coordinates; establishing a Moloden sky three-parameter or seven-parameter conversion relation model according to the coordinate expression; and calculating coordinate values corresponding to the source coordinates under the target coordinate system according to the coordinate expression of the source coordinates by using the constructed conversion relation model so as to perform wide-area coordinate conversion.

The Gaussian plane coordinate is a plane rectangular coordinate system, the determined central meridian is taken as the origin of coordinates, and the determined central meridian is mapped to the plane coordinate according to a certain proportion. In the present application, longitude and latitude coordinates are converted into a gaussian plane coordinate system to be expressed as intermediate expression coordinates of wide-area coordinate conversion. The geodetic coordinates are ellipsoidal coordinate systems using the earth as ellipsoids, and in the application, longitude and latitude coordinates are converted into the geodetic coordinate system to be expressed as a target coordinate system, so that the conversion with Gaussian plane coordinates is realized. Coordinate system conversion is adopted to unify coordinate expression of source images of different platforms, so that a foundation is laid for subsequent processing.

The Moloden sky three-parameter model is a wide-area coordinate system conversion model, and the conversion relation of two coordinate systems is described through three parameters. Comprising the following steps: and establishing a three-parameter conversion model between the source coordinate system and the target coordinate system, wherein the conversion parameters comprise origin coordinate differences and geographic coordinate deviations, and realizing simple and efficient coordinate system conversion. The Moloden sky seven parameter model is a more complex coordinate transformation model that uses seven parameters to describe coordinate system transformation relationships, including: and establishing a seven-parameter conversion model between the source coordinate system and the target coordinate system, wherein conversion parameters comprise three-dimensional origin coordinate differences, three-dimensional angle deviations and scale factors, and realizing more accurate and fine coordinate conversion. And according to the precision requirement, selecting and using a Moloden sky three-parameter or seven-parameter model to realize conversion between complex coordinate systems, and laying a foundation for subsequent processing.

Specifically, first, a data source of a mapping image is parsed to obtain relevant information of the image. This may include metadata, tags, or other additional information for the image file. And extracting longitude and latitude coordinates of the mapping image from the acquired data source as source coordinates. This ensures that each image has its corresponding geographical coordinate information. And converting the acquired longitude and latitude coordinates into Gaussian plane coordinates or geodetic coordinates by using a corresponding coordinate conversion tool or algorithm, and expressing the Gaussian plane coordinates or geodetic coordinates as source coordinates. And establishing a Moloden sky three-parameter or seven-parameter conversion relation model according to the coordinate expression. These parametric models describe the conversion relationship between the source coordinate system to the target coordinate system, including translation, rotation, and scale transformation information. And calculating coordinate values corresponding to the source coordinates under the target coordinate system according to the coordinate expression of the source coordinates by using the constructed conversion relation model. This step enables wide area coordinate conversion, mapping the mapping image from the source coordinate system to the target coordinate system.

Further, the projection conversion adopts a micentroton projection, and includes: determining a geographic coordinate range of the mapping image according to the coordinate value obtained by wide-area coordinate conversion; according to the determined geographic coordinate range, determining the number k of the wheat middling straps and the central meridian lambda 0: each image point obtained according to wide-area coordinate conversionCalculating the coordinate of the point P on the Maika plane>，/>Wherein, the method comprises the steps of, wherein,longitude of image point P +.>The latitude of the image point P; according to the calculated plane coordinates of the Maka bracket +.>Resampling the mapping image to perform the Maka-tray projection conversion.

Wherein, the Maacket number k is a key parameter of the Maacket projection, which controls the scaling relationship when the spherical coordinates are mapped to the planar coordinates. In the application, the proper Maka support band number k is determined according to the longitude and latitude coordinate range of the image, and the larger the k value is, the closer to the equatorial region is, the larger the coordinate scaling is. The central meridian λ0 is the meridian origin for constructing the macadam coordinate system, and in this application, the central meridian λ0 is selected according to the overall longitude distribution of the image, and the median of the longitude distribution is generally selected as the central meridian, so that the coordinate system origin is at the center position of the image.

Specifically, the geographic coordinate range of the mapping image is determined by using the coordinate values obtained by the wide-area coordinate conversion. This includes the geographic coordinate points on the four borders of the image, up, down, left, and right. Determining a Mylabris belt throwing belt where an image is located through a geographic coordinate range, and further determining the Mylabris belt number k and a central meridian lambda ₀ . And resampling the original mapping image by using the calculated plane coordinates of the Michaar support so as to perform Michaar support projection conversion. This step maps the pixel coordinates of the image to coordinates on the maka support plane.

Further, the Kalman filtering process is adopted, and the Kalman filtering process comprises the following steps: extracting pixel values in mapping images of different preprocessed data sources as observation dataThe method comprises the steps of carrying out a first treatment on the surface of the According to observation data->Constructing an improved Kalman filtering model: process model: />Wherein->Is a state vector +.>Is a state transition matrix; observation model:wherein->For observing vector, +.>Is an observation matrix; process noise covariance matrix:wherein->Is an adjustable parameter->A priori estimated covariance matrix representing process noise; observing a noise covariance matrix: />Wherein->Is an adjustable parameter- >A priori estimated covariance matrix representing observed noise; iterative calculation is carried out by adopting a Kalman filtering algorithm to obtain an estimated state variable; and fusing mapping images of different data sources according to the estimated state variables.

Wherein the data is observedMeans that in a Kalman filtering model, the measurement result of the system state is used for mapping images of different sources after pretreatment, and a pixel value matrix of the images is extracted as observation data +.>。/>Reflecting the observed value of each image at the current point in time. Status vector->Representing the state of the system at time k is a key variable in the Kalman filtering process model. />Representing the pixel state of the image at time k, may contain various statistical features of the image. State transition matrix->Description of the System Current State->And the last state->Is a relationship of (3). In this scheme, < >>Describing the correlation and evolution laws between image states, modeling is typically required based on image content characteristics.

Wherein the observation vector: representing the status +.>I.e. the pixel values of the image. Observation matrix->: the state vector is mapped to an observation vector modeling the relationship between the image pixel values and the state variables. Process noise covariance matrix->: noise distribution in the process model is described reflecting the uncertainty of the state transition process. Prior estimated covariance matrix of process noise +. >: and estimating covariance distribution of the process noise according to the historical data. Observation noiseCovariance matrix->: noise distribution in the observation model is described, reflecting uncertainty of the observation process. A priori estimated covariance matrix of observed noise>: and estimating covariance distribution of the observed noise according to the historical data. Application: the parameters are reasonably set, so that uncertainty of modeling process and observation can be better improved, and estimation accuracy and fusion effect of Kalman filtering are improved. The a priori statistics may provide a more reliable estimate of the noise distribution.

Specifically, define state variablesRepresenting the pixel state of the image at time k, creating a state transition matrix +.>Describing the association between image states, taking the image pixel value as the observation variable +.>Establishing an observation matrix->The state is mapped to an observed value. And predicting prior state estimation according to the process model, calculating residual errors according to the observation model, updating the state estimation, and repeating the steps to realize recursive estimation. Optimized state variable using the filtering result>And calculating weight coefficients corresponding to different images, and weighting and fusing different source images to obtain a fusion result. The Kalman filtering makes full use of the time correlation of the images to carry out recursion optimization, so as to realize the seamless fusion of the multi-source heterogeneous images with self-adaptive contents.

Further, fusing mapping images of different data sources according to the estimated state variables, including: adopting a structural similarity algorithm SSIM to calculate nothingState vector of mapping image of same data sourceSimilarity between; for mapping images with similarity greater than threshold value, the state variables of different data sources are superimposed according to weight>Generating a fusion image x:wherein->Is the weight coefficient of the i-th source.

Wherein superposition refers to the weighted combination of two or more images according to a certain weight to generate a new fused image. In this application, the similarity between the mapped images of the different data sources is calculated. And selecting images with similarity larger than a set threshold value for fusion. For selected images, the state variables xk of the images after Kalman filtering are acquired. And giving different weights to the state variables xk according to the time similarity of the images. And superposing the weighted state variables xk, namely, carrying out weighted summation. And generating a new fused state variable, namely a fused image. The process is repeated to realize the fusion of a plurality of groups of images.

Specifically, the structural similarity algorithm is a method for measuring the similarity of two images, and comprises the comparison of three aspects of brightness, contrast and structure. For each pair of images of different data sources, their SSIM indices are computed, resulting in similarity between their state vectors. To determine which images may be considered similar, a similarity threshold is set. Images with a similarity greater than the threshold will be considered sufficiently similar for fusion processing to take place. For images with similarity greater than the threshold, their state vectors (x_k) are weighted and superimposed according to their weight coefficients. The weight coefficient reflects the contribution degree of different data sources in fusion. Calculating the similarity of mapping images of different data sources through an SSIM algorithm, screening out images with the similarity larger than a threshold according to a set similarity threshold, and then carrying out weighted superposition on state vectors of the images according to a weight coefficient to generate a fusion image. The fusion method can fully utilize the information of different data sources and improve the image quality.

Further, performing multi-task learning on the mapped image after the fusion processing to generate an image analysis model, including: extracting the characteristics of the mapping image after fusion processing, wherein the characteristics comprise text characteristics, spectrum characteristics and space characteristics; constructing a multi-task convolutional neural network, wherein the multi-task convolutional neural network comprises a shared coding layer, a task coding layer and a task decoding layer, and the shared coding layer adopts a convolutional layer and a pooling layer to extract characteristics; the task coding layer adopts a convolution network to code different tasks by adopting an independent convolution network; the task decoding layer adopts independent decoding network for each task; training the constructed multi-task convolutional neural network by utilizing the extracted features, and transmitting gradient information between decoding layers of different tasks during back propagation so as to acquire internal correlations among the features of different tasks; in the training process, the loss function of each task decoding layer is minimized, and a multi-task image analysis model of text characteristics, spectrum characteristics and space characteristics is obtained.

Specifically, the text region in the fused mapping image is located and extracted using image processing techniques, such as OCR (Optical Character Recognition) or text detection algorithms. And (3) performing OCR or other text recognition technology on the detected text region, and extracting text information in the text region. In this way text features in the image can be acquired. And selecting a proper wave band for analysis according to the requirements of the spectral characteristics. This may involve the use of multispectral or hyperspectral data. Spectral features in the image are extracted using a spectral analysis method, such as Principal Component Analysis (PCA) or spectral angular decomposition, using the selected bands. This helps to understand the spectral response of different features in the image. Different spatial objects in the mapping image are identified and extracted using computer vision techniques, such as object detection and image segmentation. And carrying out space statistical analysis, and extracting the space characteristics of the object, such as shape, size, distribution and the like. Various spatial analysis tools may be used, such as Geographic Information System (GIS) technology. Feature information extracted from three aspects of text, spectrum and space is fused. This may involve a combination of feature vectors or other fusion techniques. The fused feature information is expressed in a proper data structure, such as feature vectors, feature graphs and the like.

Specifically, a convolution layer and a pooling layer are adopted for feature extraction, and a shared coding layer is responsible for extracting shared features in the image. The parameters of this layer are shared by multiple tasks to improve the model's general feature learning ability for the image. The different tasks are encoded using separate convolutional networks. Each task has its own convolutional encoding layer to capture specific information related to the task. A separate decoding network is employed for each task for restoring task-specific output. The output of this layer will be used to train and evaluate each task. And training the constructed multitasking convolutional neural network by using the extracted features. Gradient information is transferred between decoding layers of different tasks during the back propagation. This helps the model learn the inherent correlations between the different tasks, thereby improving the overall performance of the model. A corresponding loss function is defined for each task, reflecting the task-specific learning objective. The overall loss function is a weighted sum of the loss functions of the tasks, and the weight can be adjusted according to the importance of the tasks. The network parameters are updated using an optimization algorithm such as Adam, which is gradient descent or a variant thereof. The shared coding layer is helpful to extract the general features of the image, so that not only can the knowledge of task sharing be learned, but also the information specific to each task can be learned in the task coding layer. Gradient information is transferred between decoding layers of different tasks during back propagation, so that the different tasks can mutually influence and learn. Through common training, the model can comprehensively learn the relevance among a plurality of tasks, and the generalization and effect of the model are improved.

Specifically, for the text feature, the spectral feature and the spatial feature, corresponding loss functions are respectively designed. These loss functions may be cross entropy loss, mean square error loss, etc., depending on the nature of the task. The loss functions of the tasks are weighted and summed to construct a total loss function of the multiple tasks. The weight setting can be adjusted according to the importance of each task. An optimization algorithm such as Adam, employing gradient descent or variations thereof, minimizes the overall loss function of the multitasking. In the training process, the total loss is transmitted in the network through a back propagation algorithm, and the weights of the shared coding layer, the task coding layer and the task decoding layer are updated. The weights of the shared coding layer are shared by a plurality of tasks, and the weights of the task coding layer and the task decoding layer are independent. During back propagation, the weights are updated through gradients, so that the task sharing and independent learning are realized. And after training, obtaining a prediction result of the text characteristic, the spectrum characteristic and the space characteristic through a task decoding layer of the model. By minimizing the loss function of each task decoding layer, the model can learn text, spectrum and spatial characteristics simultaneously in the training process, and a comprehensive multi-task image analysis model is obtained. When the model meets the task demands of diversity, the rich information of the image can be better extracted, and the robustness and generalization of the model are improved.

Further, asynchronous communication and message queues based on Web Socket and Kafka are adopted to transmit and process mapping images. Real-time two-way communication is performed using the Web Socket protocol. The Web Socket allows the establishment of persistent connection between the client and the server, and realizes low-delay and high-efficiency real-time communication. In the image transmission and processing process, the Web Socket can be used for transmitting real-time state information, progress update and the like. Apache Kafka was used as the message queue system. Kafka is a high throughput distributed messaging system capable of handling large-scale data streams. Asynchronous messaging can be achieved by publishing relevant information of the mapping image to the Kafka theme, ensuring loose coupling and scalability of the system. And the client uploads the mapping image to the server through the Web Socket. The Web Socket connection remains open to support subsequent real-time communications. After receiving the image, the server packages relevant information (such as image file, task type, etc.) into a message, and asynchronously sends the message to a preset Kafka theme. In the Kafka message queue, messages are stored and may be delivered asynchronously to the corresponding consumers. These consumers may be task modules, distributed processing units, etc. that process images. The image processing tasks are assigned to an appropriate processing unit, which may be a single server or a cluster of multiple servers. These processing units consume messages from the Kafka theme, process the mapping images and generate results. And the processing unit feeds back the processing result to the Web Socket connection to inform the client in real time. This includes information on the progress of the process, the status of the results, the characteristics generated, etc. By adopting the technical scheme of asynchronous communication and message queue based on Web Socket and Kafka, the high-efficiency transmission and asynchronous processing of mapping images are realized, the instantaneity, the scalability and the loose coupling of the system are improved, and the method is suitable for complex application scenes for processing large-scale image data.

3. Advantageous effects

Compared with the prior art, the advantage of this application lies in:

(1) Firstly, preprocessing of time synchronization and space calibration is adopted to ensure consistency of image data. Secondly, images of different data sources are fused by Kalman filtering, so that errors are effectively reduced, and data precision is improved;

(2) The problem of inconsistent time intervals of different data sources is solved through time axis calibration and time stamp calculation, and accurate alignment of images in a time dimension is ensured;

(3) The coordinate conversion and projection conversion stage adopts methods such as wide area coordinate conversion, lattice coordinate conversion and Maka support projection, so that the consistency and accuracy of the image in a space coordinate system are enhanced;

(4) Finally, an image analysis model is generated through multi-task learning, text, spectrum and spatial characteristics are extracted, and analysis and recognition accuracy of the mapping image is further improved.

Drawings

The present specification will be further described by way of exemplary embodiments, which will be described in detail by way of the accompanying drawings. The embodiments are not limiting, in which like numerals represent like structures, wherein:

FIG. 1 is an exemplary flow diagram of a mapping graphics processing system, shown in accordance with some embodiments of the present description;

FIG. 2 is an exemplary flow chart of time synchronization shown in accordance with some embodiments of the present description;

FIG. 3 is an exemplary flow chart of spatial synchronization shown in accordance with some embodiments of the present description;

FIG. 4 is an exemplary flow chart for constructing a multi-tasking convolutional neural network according to some embodiments of the present description.

Detailed Description

The method and system provided in the embodiments of the present specification are described in detail below with reference to the accompanying drawings.

Example 1

Fig. 1 is an exemplary flow chart of a mapping graphics processing system according to some embodiments of the present disclosure, a data acquisition end, acquiring multi-source heterogeneous mapping images, recording metadata (including time stamps), transmitting the images to a message queue (Kafka) through a Web Socket, receiving downstream multi-spectral remote sensing images from satellites, parsing satellite platform, sensor parameters as metadata, and image sensing time as time stamps. The unmanned aerial vehicle-mounted camera shoots mapping images frame by frame, gesture parameters and flight tracks of the unmanned aerial vehicle are recorded as metadata, and photo time of the camera is used as a time stamp. The vehicle-mounted camera shoots a road view video image, records vehicle-mounted GPS positioning parameters as metadata, and takes video frame time as a time stamp. The engineering measurement uses a digital camera to shoot a target area, records parameters such as a camera model, shooting positions and the like as metadata, and takes shooting time of a photo as a time stamp. The SDK/API is used for collecting images and metadata, and the images and the metadata are packaged into a data format, such as JPEG/TIFF plus EXIF, and are sent to a message queue through a network interface.

The image acquisition module encapsulates images and metadata into a JSON format, initializes a Web Socket client, connects to a specified Web Socket server, sends the image data in the JSON format through the Web Socket connection, initializes a Web Socket server, listens to a specified port, receives a connection request from the client, establishes the Web Socket connection, receives the JSON data sent by the client, parses the data, issues a message to a specified topic through Kafka Producer API, creates a theme in advance, configures the partition number and the copy number, issues the data by a message producer (the Web Socket server), writes the data into a relevant partition for use by consumers, initializes a Kafka consumer, subscribes to the specified theme, reads the data from a Kafka message queue, parses the message, converts the message into the image data for processing, provides continuous and bidirectional communication by using the Web Socket, provides a high throughput and lasting buffering function, and decouples the data sending and consuming processes.

A message queue (Kafka) receives the mapped images from the different sources and buffers the images for subsequent consumption. The Kafka theme, such as imagetopic, is created with a partition number of 3, multiple copies are set per partition, fault tolerance is improved, and data retention time is set, such as 7 days. The image acquisition systems of different sources serve as producers to issue images and metadata to imagetopic through Kafka Producer API, and partitionkey is designated to realize partition storage. The plurality of subsequent processing nodes serve as consumers to read and consume data from the partition for processing by subscribing to imagetopic through kafkaConsumerAPI. Monitoring the data generation rate, dynamically adjusting the partition number, monitoring the consumption rate, balancing partition Leader allocation, and periodically checking out-of-date data to realize storage space recovery. Kafka provides a persistent, reliable buffer to enable production cost decoupling, supporting streaming data processing.

FIG. 2 is an exemplary flow chart of time synchronization, a data preprocessing node consuming images from Kafka and verifying metadata, performing spatiotemporal unified correction on the images, and returning the processed images to Kafka, according to some embodiments of the present description. Initializing Kafka consumers, subscribing imagetopic, designating consumption groups, enabling automatic submission of offset, acquiring data by batch, and preprocessing efficiency in batch. And analyzing the consumed information, extracting images and metadata, checking whether the necessary metadata is complete, and filtering out the images which do not meet the conditions. Initializing Kafka consumers, subscribing imagetopic, designating consumption groups, enabling automatic submission of offset, setting batch consumption, and acquiring 100 messages each time. And analyzing the message, and extracting the image and the metadata thereof.

Checking whether the time stamp of the image, the coordinate information and the like are complete, filtering out the images which do not meet the conditions, reorganizing the image sequence according to the time stamp sequence, checking whether the time stamp intervals are continuous, inserting the time stamp at the intervals, and completing the time line. The Kafka message is parsed, image data and metadata are extracted, whether the metadata has a time stamp and a coordinate information field or not is checked, and if the time stamp or the coordinate is empty, the piece of image data is filtered. And extracting time stamps from the FilteridList, constructing an ordered list, and recombining the FilteridList into a SortedList according to the time stamp value sequence. Traversing the timestamp differences in the sortdlist, if the interval is found to be greater than a threshold (e.g., 5 seconds), the interval is recorded. And performing equal interval interpolation on the recorded time stamp interval segments, calculating the inserted time stamp value, and inserting the time stamp value into a corresponding position of the SortedList. And reorganizing the image sequence according to the SortedList after the time stamp is inserted, and continuously changing the time stamp interval. The recombined sequences were published to Kafka different topics.

And extracting coordinate information from the image, uniformly projecting and converting the image into a designated coordinate system, geometrically correcting the image, and issuing the image to a new topic according to the original sequence after processing, so that the throughput is improved by batch processing. The method comprises the steps of analyzing longitude and latitude information from image metadata, loading the image data from a message consumed by Kafka, analyzing metadata json character strings attached to the message, searching a key value pair of longitude and latitude from the metadata, returning empty coordinates if the key value pair of longitude and latitude does not exist, checking a longitude range [180, 180], checking a latitude range [90, 90], creating a coordinate object chord if the check passes, setting attributes chord. Longitude and chord. Latitudinal, adding the chord object to a coordinate list chord_list, and returning an extracted coordinate list chord_list.

The method comprises the steps of recording a coordinate list, designating a coordinate conversion target projection system, such as WGS84, applying a projection conversion formula to the coordinate list, converting the coordinate list into a coordinate list new_coordinates of a target coordinate system, extracting the coordinate list coordinates from image metadata according to new_coordinates, wherein the coordinate list coordinates_list contains a plurality of chord objects, each object has longitude and latitude attributes, designating the target as a WGS84 coordinate system, and looking up a table to obtain WGS84 ellipsoid parameters. Traversing each chord in the chord_list, converting the longitude and latitude into XYZ rectangular coordinates, applying a projection conversion formula, converting into WGS84 coordinates, adding the WGS84 coordinates into the new_chord_list, defining a function of a cover (chord, target), inputting a source chord and a target coordinate system target, and returning a new converted coordinate new_chord. Traversing the new_coordinate_list, calling the cover, entering the chord object and the WGS84, adding new coordinates to the new_coordinate_list, and returning a converted new coordinate list new_coordinate_list. And carrying out geometric correction on the image by utilizing a resampling algorithm, and reissuing the image subjected to geometric correction to Kafka according to the original sequence, and releasing 100 images in batches to improve the throughput. The alignment of the time axis of the image and the unification of the coordinates are realized, the time correlation requirement of the follow-up algorithm is met, and Kafka provides an intermediate data stream with good expansibility.

FIG. 3 is an exemplary flow chart of spatial synchronization shown in some embodiments of the present description, initializing a Kafka consumer, reading an image from image_topic, parsing a message to obtain a picture and metadata from the message, loading the image and metadata from the message, checking if there are time stamps and coordinate fields in the text, and if not, adding to a filtered list for filtering. And extracting a time stamp timelist from the filtered images, and sequencing the images according to the size of the time stamp to obtain a dissolved_list. And loading the image from the dissolved_list, extracting coordinates from the metadata, performing projection conversion, and updating the coordinate information of the image. And applying a resampling algorithm to the image to perform geometric correction according to the converted coordinate information. And returning the processed images to a new image sequence, thereby completing the standardization in time and space. Initializing an empty list processed_list, traversing the preprocessed image sequence processed_images, constructing a Kafka message object for each image, and setting values as bytes data of the image. Adding key value pairs in the metadata of the message, the key value pairs are: "original topic": "image_topic", "processing parameters": "[ coordinate conversion mode, interpolation method ]". The message object is added to the processed_list and the encapsulated message list processed_list is returned. The image byte data and the key metadata are packaged into the message, the source and the processing parameters are recorded, traceability is ensured, and the format requirement of issuing to the downstream is met.

The KafkaProducer object is created, connecting the preprocessed_topic topics the function preprocess_images are defined, input: original image list original_list, output: the pre-processed image list, processed_list, is traversed, each message is published using the send method of the producer, and pre-processed_topic is designated as the subject. An original image list original_list is acquired, preprocess_images are called to preprocess images, processed_list is obtained, and processed_list is traversed and issued to Kafka. The encapsulation ensures that the processing flow is clear and reusable, the batch transmission improves the message release efficiency, and the data flow is transmitted to downstream processing through Kafka.

And the Kalman filtering node acquires the preprocessed image from the Kafka, constructs an improved Kalman filtering model, performs iterative calculation to realize image fusion, and sends the fusion result to the Kafka. Initializing the Kafka consumer, subscribing preprocessed topic, obtaining the preprocessed images from the partition batch, and organizing the images into a sequence in time order. Defining state variables, including image multispectral features, constructing a state transition matrix, modeling feature changes, setting a process noise covariance matrix, generating an observation matrix, connecting states and observation, and setting an observation noise covariance matrix. And predicting the state and updating the estimation step by step in sequence, recursively optimizing the state variable, and realizing sequence fusion. And re-publishing the fused and optimized state sequence to Kafka, wherein the image theme is fusedtopic. Kalman filter fusion improves time correlation, and Kafka provides a flexible intermediate data pipeline that provides an enhanced data set for subsequent algorithms.

FIG. 4 is an exemplary flow chart of constructing a multi-tasking convolutional neural network, a multi-tasking network node acquiring fused images from Kafka, network training to obtain an image analysis model, storing the model to a knowledge base, initializing Kafka consumers, subscribing to fusedtopic, and acquiring fused enhanced images in batches, according to some embodiments of the present description. And constructing a convolutional neural network, wherein the convolutional neural network comprises a shared coding layer and a task related decoding layer, and multitasks such as image classification, target detection and the like are defined. Extracting image features, tag data, performing iterative training, optimizing a multi-task loss function, and recording indexes such as loss, precision and the like. After training is terminated, the model is saved to a file, the model format is Tensor Flow Saved Model, and the model format is uploaded to an object store as a model knowledge base. A unified model is obtained that can support multiple image analysis tasks, kafka provides an extensible training data pipeline, and a knowledge base enables persistent storage and management of the model.

The model application node loads a trained model from the knowledge base, analyzes, classifies, detects and the like on the new image, outputs a result to return to a user, downloads a stored model file from the object storage knowledge base, loads the model file to a prediction program, such as a TensorFlow/PyTorch environment, and initializes a model structure and weight parameters. And receiving an image uploaded or newly acquired by a user, carrying out standardization processing, and converting the image into a model input format. And inputting an image, acquiring and outputting forward propagation, classifying a model, acquiring the class with the highest probability, analyzing a coordinate frame for a detection model, and filtering. And (3) formatting information organizations such as image classification labels, detection frames and the like, returning the information organizations to the user terminal through an API interface, and supporting the visualized marked images as a result. The knowledge base storage enables the multiplexing model to be simple and efficient, the multiplexing model serves multiple image analysis requirements, and output accords with a user use scene.

The user terminal defines task requirements of analyzing images, subscribes and receives analysis results, displays, interacts and feeds back, and a user selects analysis task types, such as classification and detection, sets related parameters, such as classification types and detection targets, at the client side, and uploads the images to be analyzed. The client subscribes the result theme of the corresponding task to the server, supports real-time pushing or pulling of the result as required, and adopts different data formats for different tasks. And obtaining an analysis result, visually presenting in an interface, classifying and displaying labels and confidence, and detecting and displaying image frame selection and target names. Supporting result editing, such as modifying a detection frame, adding user feedback, such as classifying error marks, and submitting to improve algorithm performance. The user task definition drives personalized image analysis, interaction feedback improves a model, analysis quality is improved, user experience is good, and Closethoop is achieved.

Example 2

This example uses the same processing scheme as example 1, wherein the coordinate transformation uses the Moloden sky seven parameters: and acquiring longitude and latitude coordinates (B, L and H) of the image, and acquiring a source coordinate system parameter and a target coordinate system parameter. Calculating two coordinate systems (f, f '), calculating semimajoranxes (a, a') of the two coordinate systems, and setting seven parameters: Δx, Δy, Δz, Δf, Δa, Δb, Δl. Converting the latitude phi and the longitude lambda into rectangular coordinates (X, Y, Z), and converting by applying a Moloden sky formula: x ' =Δx+ (1+Δf) (x+Δa Δbsin Φcos λ), Y ' =Δy+ (1+Δf) (y+Δa Δlsin Φsin λ), Z ' =Δz+ (1+Δf) (z+Δa Δhcos Φ). And converting the rectangular coordinates (X ', Y', Z ') back to longitude and latitude (B', L ', H'). Image resampling is performed using the transformed coordinates (B ', L ', H '). The accurate conversion from the source coordinates to the target coordinates is realized, the seven-parameter difference is considered by the Moloden sky model, and an accurate positioning reference is provided for the subsequent processing of the image.

Example 3

This example uses the same processing scheme as example 1, wherein the coordinate transformation uses the Moloden sky three parameters: longitude and latitude coordinates (B, L, H) of an image point are acquired, and reference ellipsoid parameters of a source coordinate system and a target coordinate system are acquired. Calculating the oblate f and f' of two coordinate system reference ellipsoids, and setting three parameters: Δx, Δy, Δz. The latitude phi and the longitude lambda are converted into rectangular coordinates (X, Y, Z), and a Moloden sky three-parameter formula is applied for conversion: x ' =Δx+ (1 f ') Xf ' Δz, Y ' =Δyf ' Δx+ (1 f ') Y, Z ' =Δzf ' Δy+ (1 f ') Z), the rectangular coordinates (X ', Y ', Z ') are converted back to latitude and longitude coordinates (B ', L ', H '). Resampling the image using the transformed coordinates (B ', L ', H '). The conversion from the source to the target coordinate system is realized, and the Moloden sky three-parameter model simplifies the calculation and provides accurate coordinate reference for the subsequent image processing.

Example 4

The present embodiment adopts the same processing scheme as embodiment 1, wherein the message queue adopts a Rabbit MQ: the image acquisition system initializes the producer client of the Rabbit MQ, connects to the Rabbit MQ server, opens a channel, declares a switch, such as "image_exchange". The data processing system initializes the consumer client of the Rabbit MQ, connects to the Rabbit MQ server, opens a channel, declares a queue, such as "image_queue", and binds to the switch. The producer client encapsulates the image and metadata into a message, which is posted to a queue through the switch. The consumer client acquires the message from the queue, analyzes the message, extracts the image data and processes the image data. The Rabbit MQ provides reliable message queue service, realizes decoupled asynchronous data transmission, and ensures the consistency of data processing.

The foregoing has been described schematically the invention and embodiments thereof, which are not limiting, but can be embodied in other specific forms without departing from the spirit or essential characteristics thereof. The drawings are also intended to depict only one of the embodiments of the invention, and therefore the actual construction is not intended to be limiting, as any reference number in the claims should not be limiting to the claims that issue. Therefore, if one of ordinary skill in the art is informed by this disclosure, a structural manner and an embodiment similar to the technical scheme are not creatively designed without departing from the gist of the present invention, and all the structural manners and the embodiments belong to the protection scope of the present patent. In addition, the word "comprising" does not exclude other elements or steps, and the word "a" or "an" preceding an element does not exclude the inclusion of a plurality of such elements. The various elements recited in the product claims may also be embodied in software or hardware. The terms first, second, etc. are used to denote a name, but not any particular order.

Claims (10)

1. A mapping graphics processing system, comprising:

preprocessing the mapping image, including time synchronization and spatial calibration;

Carrying out Kalman filtering treatment on the preprocessed mapping image, and fusing mapping images of different data sources;

multitask learning is carried out on the mapping images after the fusion processing, and an image analysis model is generated;

and adopting asynchronous communication and message queues to transmit and process the mapping image.

2. The mapping graphics processing system of claim 1, wherein:

performing time axis calibration according to the time stamps of the data sources of different mapping images;

extracting metadata containing time information according to a data source of the mapping image to serve as a time stamp of the mapping image;

comparing the time stamps of the mapping images, judging whether the time intervals are continuous and consistent, and if not, recording discontinuous points of the time intervals;

calculating an insertion time stamp at the discontinuous points of the time interval by using a linear interpolation method, and filling the discontinuous points of the time interval;

the order of mapping the images is rearranged based on the time stamp information calculated after filling so that the images of the different data sources are synchronized on the time axis.

3. The mapping graphics processing system of claim 2, wherein:

taking the mapping image without time metadata as the mapping image which cannot directly correspond to the time stamp;

Respectively acquiring N mapping images with time stamps before and after the mapping images which cannot directly correspond to the time stamps, and taking the N mapping images with the time stamps as control points;

and calculating the time stamp of the mapping image which cannot directly correspond to the time stamp through a cubic spline interpolation function according to the acquired control point.

4. The mapping graphics processing system of claim 2, wherein:

coordinate conversion and projection conversion are carried out on the mapping image after the time axis calibration;

the coordinate conversion adopts wide area coordinate conversion;

the projection conversion adopts a Michaet projection.

5. The mapping graphics processing system of claim 4, wherein:

the coordinate conversion adopts wide-area coordinate conversion, and comprises the following steps:

analyzing a data source of the mapping image, and extracting longitude and latitude coordinates of the mapping image as source coordinates;

converting the acquired source coordinates into coordinate expressions of Gaussian plane coordinates or geodetic coordinates;

establishing a Moloden sky three-parameter or seven-parameter conversion relation model according to the coordinate expression;

and calculating coordinate values corresponding to the source coordinates under the target coordinate system according to the coordinate expression of the source coordinates by using the constructed conversion relation model so as to perform wide-area coordinate conversion.

6. The mapping graphics processing system of claim 5, wherein:

The projection conversion adopts the Maka support projection, includes:

determining a geographic coordinate range of the mapping image according to the coordinate value obtained by wide-area coordinate conversion;

determining the number k of the wheat card carrier bands and the central meridian according to the determined geographic coordinate range：

Each image point obtained according to wide-area coordinate conversionCalculating the coordinate of the image point P on the Michaelis plane>：

；

；

Wherein,longitude of image point P +.>The latitude of the image point P;

according to the calculated plane coordinates of the MickeyResampling the mapping image to perform the Maka-tray projection conversion.

7. The mapping graphics processing system of any of claims 2-6, wherein:

a kalman filter process is employed comprising:

extracting pixel values in the mapping images as observation data zk from the preprocessed mapping images of different data sources;

based on observed dataConstructing an improved Kalman filtering model:

process model:；

wherein,is a state vector +.>Is a state transition matrix;

observation model:；

wherein,for observing vector, +.>Is an observation matrix;

process noise covariance matrix:；

wherein,is an adjustable parameter->A priori estimated covariance matrix representing process noise;

Observing a noise covariance matrix:；

wherein,is an adjustable parameter->A priori estimated covariance matrix representing observed noise;

iterative calculation is carried out by adopting a Kalman filtering algorithm to obtain an estimated state variable;

and fusing mapping images of different data sources according to the estimated state variables.

8. The mapping graphics processing system of claim 7, wherein:

fusing mapping images of different data sources according to estimated state variables, including:

calculating state vectors of mapping images of different data sources by adopting structural similarity algorithm SSIMSimilarity between;

similarity to is greater thanMapping images of threshold values, state variables of different data sources being superimposed according to weightsGenerating a fusion image x:

；

wherein the method comprises the steps ofIs the weight coefficient of the i-th source.

9. The mapping graphics processing system of claim 8, wherein:

multitask learning is carried out on the mapping images after the fusion processing, and an image analysis model is generated, which comprises the following steps:

extracting the characteristics of the mapping image after fusion processing, wherein the characteristics comprise text characteristics, spectrum characteristics and space characteristics;

constructing a multi-task convolutional neural network, wherein the multi-task convolutional neural network comprises a shared coding layer, a task coding layer and a task decoding layer, and the shared coding layer adopts a convolutional layer and a pooling layer to extract characteristics; the task coding layer adopts a convolution network to code different tasks by adopting an independent convolution network; the task decoding layer adopts independent decoding network for each task;

Training the constructed multi-task convolutional neural network by utilizing the extracted features, and transmitting gradient information between decoding layers of different tasks during back propagation so as to acquire internal correlations among the features of different tasks;

in the training process, the loss function of each task decoding layer is minimized, and a multi-task image analysis model of text characteristics, spectrum characteristics and space characteristics is obtained.

10. The mapping graphics processing system of claim 9, wherein:

and adopting asynchronous communication and message queues based on Web Socket and Kafka to transmit and process the mapping image.