CN117710419A - Mapping processing method, training method, computer equipment and storage medium - Google Patents
Mapping processing method, training method, computer equipment and storage medium Download PDFInfo
- Publication number
- CN117710419A CN117710419A CN202311483523.1A CN202311483523A CN117710419A CN 117710419 A CN117710419 A CN 117710419A CN 202311483523 A CN202311483523 A CN 202311483523A CN 117710419 A CN117710419 A CN 117710419A
- Authority
- CN
- China
- Prior art keywords
- mapping
- target
- area
- original image
- preset
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000013507 mapping Methods 0.000 title claims abstract description 318
- 238000000034 method Methods 0.000 title claims abstract description 65
- 238000012549 training Methods 0.000 title claims abstract description 26
- 238000003672 processing method Methods 0.000 title abstract description 14
- 238000012545 processing Methods 0.000 claims abstract description 67
- 230000008569 process Effects 0.000 claims description 22
- 230000009466 transformation Effects 0.000 claims description 22
- 230000008447 perception Effects 0.000 claims description 6
- 230000001131 transforming effect Effects 0.000 claims description 5
- 238000010586 diagram Methods 0.000 description 13
- 230000004927 fusion Effects 0.000 description 12
- 230000000694 effects Effects 0.000 description 5
- 238000013528 artificial neural network Methods 0.000 description 4
- 241001465754 Metazoa Species 0.000 description 3
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000001514 detection method Methods 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 239000011159 matrix material Substances 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 230000003796 beauty Effects 0.000 description 1
- 239000002131 composite material Substances 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000011176 pooling Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Image Analysis (AREA)
Abstract
The application discloses a mapping processing method, a training method, computer equipment and a storage medium. The method comprises the following steps: acquiring an original image and a mapping target; identifying a map area in the original image; registering the mapping area and the mapping target to obtain a registered mapping target; and fusing the registered mapping target and the mapping area to obtain a target image subjected to mapping processing. By means of the scheme, the accuracy of mapping can be improved.
Description
Technical Field
The present disclosure relates to the field of image processing technologies, and in particular, to a mapping processing method, a training method of a mapping processing model, a computer device, and a storage medium.
Background
In the process of processing an image, an image processing model is generally used to process the image so as to obtain a corresponding image processing result. Image processing such as object detection and gesture detection is widely used in various fields.
It is generally necessary to train an image processing model by using a large number of sample images, and the sample images are usually acquired by a manual acquisition mode, and the sample images acquired by the mode are single in form and are difficult to acquire a large number of sample images.
At present, in order to overcome the defect of a sample image with a target, the sample image with the target can be expanded by adopting a target mapping mode, but the existing mapping method has the problem of inaccurate mapping.
Disclosure of Invention
The technical problem that this application mainly solves is to provide a mapping processing method, training method, computer equipment and storage medium, can improve the degree of accuracy of mapping.
In order to solve the above problem, a first aspect of the present application provides a mapping processing method, which includes: acquiring an original image and a mapping target; identifying a map area in the original image; registering the mapping area and the mapping target to obtain a registered mapping target; and fusing the registered mapping target and the mapping area to obtain a target image subjected to mapping processing.
To solve the above problem, a second aspect of the present application provides a training method of a mapping processing model, including: obtaining an image sample to be mapped and a mapping target sample; taking the image sample to be attached and the mapping target sample as an original image and a mapping target respectively, and processing the image sample to be attached and the mapping target sample by utilizing any step of a mapping processing method realized by using a mapping processing model to obtain mapping processing loss; based on the mapping processing loss, training the mapping processing model to obtain a trained mapping processing model.
To solve the above-mentioned problem, a third aspect of the present application provides a computer device, where the computer device includes a memory and a processor, where the memory stores program data, and the processor is configured to execute the program data to implement any step of any one of the above-mentioned mapping processing method and the training method of the mapping processing model.
In order to solve the above-mentioned problems, a fourth aspect of the present application provides a computer-readable storage medium storing program data executable by a processor for implementing any one of the steps of the above-mentioned mapping processing method, training method of a mapping processing model.
According to the scheme, unlike the prior art, the mapping area in the original image is identified by acquiring the original image and the mapping target, the position of the area required to be mapped in the original image can be automatically identified, the mapping area and the mapping target are registered to obtain the registered mapping target, and the direction and the size of the mapping target and the mapping area can be registered to enable the mapping target to be more consistent with the mapping area, so that the registered mapping target and the mapping area are fused to obtain the target image subjected to mapping processing, the mapping accuracy can be increased, and the mapping processing efficiency and the authenticity of the mapping effect are improved.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application.
Drawings
For a clearer description of the technical solutions in the present application, the drawings required in the description of the embodiments will be briefly described below, it being obvious that the drawings described below are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art. Wherein:
FIG. 1 is a flow chart of an embodiment of a mapping method of the present application;
FIG. 2 is an example schematic diagram of an embodiment of an original image of the present application;
FIG. 3 is an example schematic diagram of an embodiment of a mapping object of the present application;
FIG. 4 is a flowchart illustrating an embodiment of step S12 in FIG. 1 of the present application;
FIG. 5 is an example schematic diagram of an embodiment of a mapping area of the present application;
FIG. 6 is a flowchart illustrating another embodiment of step S12 in FIG. 1 of the present application;
FIG. 7 is an example schematic diagram of an embodiment of a preset map template of the present application;
FIG. 8 is a flow chart illustrating another embodiment of step S12 in FIG. 1 of the present application;
FIG. 9 is a flowchart illustrating an embodiment of step S13 in FIG. 1 of the present application;
FIG. 10 is a schematic diagram illustrating the structure of an embodiment of a direction-aware network of the present application;
FIG. 11 is an example schematic diagram of an embodiment of a registered map object of the present application;
FIG. 12 is an example schematic diagram of an embodiment of a target image of the present application;
FIG. 13 is a flow chart of one embodiment of a training method of the mapping process model of the present application;
FIG. 14 is a schematic view of an embodiment of a mapping apparatus according to the present application;
FIG. 15 is a schematic view of an embodiment of a training device of the mapping process model of the present application;
FIG. 16 is a schematic diagram of an embodiment of a computer device of the present application;
FIG. 17 is a schematic diagram of an embodiment of a computer-readable storage medium of the present application.
Detailed Description
The following description of the technical solutions in the embodiments of the present application will be made clearly and completely with reference to the accompanying drawings in the embodiments of the present application, and it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present disclosure.
The terms "first," "second," and the like in this application are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include at least one such feature. In the description of the present application, the meaning of "plurality" means at least two, for example, two, three, etc., unless specifically defined otherwise. Furthermore, the terms "comprise" and "have," as well as any variations thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those listed steps or elements but may include other steps or elements not listed or inherent to such process, method, article, or apparatus.
Reference in the specification to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearances of such phrases in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those of skill in the art will explicitly and implicitly appreciate that the embodiments described herein may be combined with other embodiments.
The term "and/or" is herein merely an association relationship describing an associated object, meaning that there may be three relationships, e.g., a and/or B, may represent: a exists alone, A and B exist together, and B exists alone. In addition, the character "/" herein generally indicates that the front and rear associated objects are an "or" relationship. Further, "a plurality" herein means two or more than two. In addition, the term "at least one" herein means any one of a plurality or any combination of at least two of a plurality, for example, including at least one of A, B, C, and may mean including any one or more elements selected from the group consisting of A, B and C.
The inventor of the application has long-term research discovers that the existing mapping method generally cannot accurately identify the position of the area needing mapping, and the mapping mismatch condition exists in the mapping process, so that the attached target is not matched with the background of the image, the mapping accuracy is not high, and the mapping effect is hard and has no attractive appearance and authenticity.
In order to solve the above-described distinguishing problem, the present application provides the following examples, and the following details of each example are described.
It is understood that the mapping method and the training method of the mapping model in the present application may be performed by a computer device, which may be any device having processing capability, for example, a computer, a mobile phone, a tablet computer, etc. The present application is not limited in this regard.
Referring to fig. 1, fig. 1 is a flowchart illustrating an embodiment of a mapping method of the present application. The method may comprise the steps of:
s11: and obtaining an original image and a mapping target.
The original image and the mapped object may be acquired using a camera device or otherwise, such as the internet.
The original image is a sample image to be expanded, or the original image is an image to be subjected to mapping, i.e. a background image of a mapping target. The original image may be acquired by using an image capturing apparatus for a target scene, a target object, a target area, or the like. The target object may be a human body, a car, an animal, an article, or the like. Alternatively, the original image is obtained through other modes, such as the internet, an image library, and the like, and the method for obtaining the original image is not limited in the application.
The mapping target may be an image acquired by using an image capturing apparatus, such as an image acquired for a target object, which may be a human body, a car, an animal, an article, or the like.
In some embodiments, the target may be obtained by capturing a preset image, and if the preset image is an image including the target, the target is detected in the preset image, and a region belonging to the target is captured to obtain the target. If the image comprises the vehicle, the area where the vehicle is located is intercepted, and a mapping target corresponding to the vehicle is obtained. It can be appreciated that the application can select the corresponding mapping target and the original image according to specific needs. And the number of each of the mapping targets and the original images, the method for acquiring the original images and the mapping targets is not limited.
As an example, referring to fig. 2 to 3, the original image may be an image acquired for a target scene. The map image may be an image corresponding to an object or target object.
S12: the mapped regions in the original image are identified.
The template matching network can be utilized to perform region identification on the original image so as to identify the region needing mapping in the original image, namely, the mapping region.
In some embodiments, the original image may be matched to a preset map area to identify the map area in the original image.
In some embodiments, the original image may be matched with a preset mapping template, and a preset mapping module matched with the original image in the preset mapping module may be obtained, so as to identify a mapping area in the original image.
In some embodiments, target recognition, target region recognition, or the like may be performed on the original image to obtain a mapping region in the original image that needs mapping.
In some embodiments, referring to fig. 4, step S12 of the above embodiments may be further extended. Identifying a map area in an original image, the present embodiment may include the steps of:
s1211: the original image is input into a template matching network.
The original image may be input into a template matching network, where the template matching network may be an algorithm with matching and similarity detection functions, and may match the original image.
In some implementations, the template matching network may be at least one of a feature point matching algorithm, a template matching algorithm, a phase correlation matching algorithm, a neural network matching algorithm, and the like.
The feature point matching algorithm can achieve image matching by extracting key points and descriptors in the image. The feature points include corner points, edges and regions, which can be extracted and described by various algorithms. Once the keypoints and descriptors are extracted, various matching algorithms (e.g., nearest neighbor matching or RANSAC, etc.) can be used to find the correspondence between the two images.
The template matching algorithm may find a matching target image by comparing a small template image with a large target image.
The phase correlation matching algorithm is a frequency domain algorithm that can find a phase difference between two images. The algorithm can be used to measure deformation, rotation and translation of objects, as well as to perform three-dimensional shape reconstruction and measurement.
The neural network matching algorithm is a machine learning method that can identify and match images by training the neural network.
S1212: and matching the original image with a preset mapping area by using a template matching network so as to identify a sub-area matched with the original image.
The preset map area may be a small image, i.e., a small template image, which is smaller than the original image, and may be an image containing an area to be mapped.
And matching each sub-region of the original image with a preset mapping region by using a template matching network, wherein the algorithm can be used for matching by comparing the preset mapping region with the pixel value of the original image so as to identify the matched sub-region of the original image.
In some embodiments, the number of the preset mapping areas is one or more, and each sub-area of the original image and the preset mapping area can be respectively matched by using a template matching network, so as to respectively determine the matched sub-areas. Alternatively, a plurality of preset map areas may be respectively matched in the original image to identify the most matched map areas.
In some embodiments, the original image has a similar spatial distribution as the pre-mapped region. At least a part of the subareas in the original image are similar to the preset map area. Wherein, the similarity is understood as the similarity is larger than a preset similarity threshold.
S1213: and taking the sub-region matched with the original image as a mapping region.
Thus, a sub-region that matches the preset map region may be taken as a map region.
Referring to fig. 5, the mapping region may be a sub-region in the original image that matches a preset mapping region, and in this way, the mapping region that needs mapping processing may be automatically identified in the original image through the template matching network.
In some embodiments, referring to fig. 6, step S12 of the above embodiments may be further extended. Identifying a map area in an original image, the present embodiment may include the steps of:
s1221: the original image is input into a template matching network.
In some implementations, the template matching network may be at least one of a feature point matching algorithm, a template matching algorithm, a phase correlation matching algorithm, a neural network matching algorithm, and the like. The present application is not limited in this regard.
S1222: matching the mapping area of the original image with a preset mapping template by using a template matching network so as to identify a matched preset mapping template; the preset mapping template is provided with at least one preset mapping area.
Referring to fig. 7, the preset mapping template may be a template image provided with at least one preset mapping region, and each preset mapping region may be represented as a region requiring mapping processing.
In some embodiments, where multiple preset map areas are provided, the multiple preset map areas may be respectively matched in the original image to identify the most matched map area or areas. This approach can identify areas of the original image that are relatively more suitable for mapping.
In some embodiments, the difference in size between the pre-set mapping template and the original image is less than the pre-set difference to facilitate determining the mapping region in the original image.
In some embodiments, a template matching network is utilized to match a map region of an original image with a preset map template to identify a matched preset map template. It can be appreciated that the template matching network of the present application may use an image matching algorithm to match the mapping area of the original image with the preset mapping template, and the matching algorithm of the mapping area of the original image with the preset mapping template is not limited in the present application.
S1223: and acquiring the position coordinates of the preset mapping area of the matched preset mapping template.
The position coordinates of the preset mapping area in the matched preset mapping template are obtained, the position coordinates can be coordinates obtained by identifying the preset mapping area, and can be coordinates configured for the preset mapping area of the preset mapping template in advance, and the method is not limited.
S1224: and taking the area of the original image at the position coordinates as a mapping area.
Thus, an area located at the position coordinates of the preset map area is acquired in the original image as a map area.
In this embodiment, a preset mapping template that is most matched with the original image may be selected, and then a mapping area of the original image may be determined according to a preset mapping area set by the preset mapping template.
In some embodiments, after the step S1222, that is, after the template matching network is used to match the mapping area of the original image with the preset mapping template to identify the matched preset mapping template, the preset mapping area of the preset mapping template may be obtained, and then the preset mapping area is matched with the original image to identify the matched sub-area in the original image as the mapping area.
In some embodiments, referring to fig. 8, step S12 of the above embodiments may be further extended. Identifying a map area in an original image, the present embodiment may include the steps of:
s1231: and carrying out target recognition on the original image, and determining an object area of the target object.
The original image can be subjected to target recognition by adopting a target recognition algorithm to determine the area where the target object is in the original image, and the area is taken as the object area. The target recognition algorithm may be set for a target object contained in the original image, such as a human body, an animal, a vehicle, a house, an article, etc., and may be determined according to an application scenario, which is not limited in this application.
S1232: and selecting a preset area of the original image as a mapping area based on the object area, wherein the preset area is at least a partial area of the object area, or at least a partial overlapping area exists between the preset area and the object area.
Based on the object region, a preset region related to the object region can be selected from the original image as a map region. The preset area may be at least a partial area of the object area, for example, when mapping (or shielding) is required to be performed on a certain portion of the target object, an area where the certain portion of the target object is located may be used as the preset area. Or, if the preset area and the object area have at least a partial overlapping area, and if the target object is a license plate and the upper edge of the license plate needs to be mapped, the area overlapping with the upper edge of the area where the license plate is located can be used as the preset area.
In some embodiments, the size of the preset area may be set according to a specific application scenario, for example, a size difference between the size of the preset area and the size of the mapping target is within a preset size difference, or a contour difference between a contour of the preset area and a contour of the mapping target is within a preset contour difference, which is not limited in this application.
In this embodiment, by selecting the preset area of the original image as the mapping area based on the object area identified by the target, the mapping area can be determined for the target object, that is, the mapping area can be determined to be suitable for the target object of the original image, and mapping processing can be performed.
By the method, the position of the proper and ideal mapping area in the original image can be automatically positioned and identified, and the area needing mapping does not need to be manually determined.
S13: registering the mapping area and the mapping target to obtain a registered mapping target.
The mapping region and the mapping target can be registered, the mapping target is registered to be consistent with the direction of the mapping region, the same size or the same size difference meets the preset size requirement, and the mapping target can be registered to be consistent with the direction of the mapping region and the same size.
Because the size and the direction of the attached mapping target are not consistent with the background of the original image, the mapping accuracy is not high, the mapping effect is vivid and has no beauty and reality.
In some embodiments, referring to fig. 9, step S13 of the above embodiments may be further extended. Registering the map area and the map target to obtain a registered map target, the embodiment may include the following steps:
s131: and respectively acquiring the direction perception characteristics of the mapping area and the mapping target.
The direction-aware characteristics of the mapping region and the mapping target may be obtained by using a direction-aware network, respectively, wherein the direction-aware characteristics may include direction angle information and size ratio information.
In some embodiments, the mapping area and the mapping target may be used as input images, and the input images may be processed by using a direction sensing network to obtain direction sensing features corresponding to the input images.
In some embodiments, the direction-aware network may include multiple perceptual convolution kernels, with different ones of the perceptual convolution kernels corresponding to different directions. Such as at least one of 4 different directions, 8 different directions, 10 different directions, 16 different directions, etc., for each different perceptual convolution kernel. The corresponding directions of different perception convolution kernels can be obtained according to the corresponding different direction and angle average. It can be appreciated that the direction and size corresponding to the perceptual convolution kernel may be determined according to a specific application scenario. The present application is not limited in this regard.
The convolution of the sensing convolution kernels in different directions and the input image is carried out to obtain the phase consistency in each direction, the direction of the phase consistency represents the drastic change of the image structural characteristics along the direction, the direction sensing characteristics represent the energy change of the input image in a certain direction similar to the gradient direction, the phase consistency in each direction can reflect the edge strength information in different directions, thus the respective directions can be sensed, and the direction sensing characteristics F of the target of the mapping can be obtained are output 1 Directional awareness feature F of sum map area 2 。
As an example, the direction-aware network described above may contain 8 different direction-aware convolution kernels, each of 5×5 in size. The 8 perceptual convolution kernels may correspond to directions of 0 °, 22.5 °, 45 °, 67.5 °, 90 °, 112.5 °, 135 ° and 157.5 °, respectively, with the perceptual convolution kernels for each direction being different.
As an example, the 8 perceptual convolution kernels may each be as follows:
(1) The perceptual convolution kernel corresponding to 0 ° is as follows:
(2) 22.5 ° corresponding perceptual convolution kernel as follows:
(3) The 45 ° corresponding perceptual convolution kernel is as follows:
(4) 67.5 ° corresponding perceptual convolution kernel as follows:
(5) The corresponding perceptual convolution kernel of 90 ° is as follows:
(6) 112.5 ° corresponding perceptual convolution kernel as follows:
(7) The 135 ° corresponding perceptual convolution kernel is as follows:
(8) 157.5 ° corresponding perceptual convolution kernel as follows:
in some embodiments, referring to fig. 10, the direction-aware network may sequentially comprise a convolutional layer, a plurality of different direction-aware convolutional kernels, a max-pooling layer, and the like. The convolution layer is used for extracting features of an input image to obtain image features, the image features are respectively subjected to convolution processing by using perception convolution kernels (also called as convolution modules) in different directions to obtain the convolution features of the perception convolution kernels, and the maximization layer is used for carrying out maximization processing on the convolution features to obtain the direction perception features of the input image.
S132: and registering the direction sensing characteristic of the mapping area and the direction sensing characteristic of the mapping target to obtain the spatial transformation parameter.
And inputting the direction sensing characteristics of the mapping region and the direction sensing characteristics of the mapping target into a registration network, and registering the direction sensing characteristics of the mapping region and the direction sensing characteristics of the mapping target by using the registration network to obtain the space transformation parameters. The spatial transformation parameters may be characterized as a spatial transformation relationship between the mapping target and the mapping region, such as a size scaling relationship, a direction transformation relationship, and the like. Registration networks such as ResNet networks or other registration networks that may obtain spatial transformation parameters, as not limited in this application.
In some embodiments, the directional-sensing features of the mapped region and the directional-sensing features of the mapped object are registered using a registration network to obtain deformation parameters (a) 1 、a 2 、b 1 、b 2 ) Such as deformation of direction angle, deformation of dimension ratio, etc. Based on deformation parameters (a) 1 、a 2 、b 1 、b 2 ) Constructing and obtaining space transformation parameters; the deformation parameters comprise a preset relation between the direction angle deformation and the size proportion deformation of the direction sensing characteristics.
In some embodiments, the spatial transformation parameters constructed as described above may be represented in the form of a matrix. For example, by rotating the map object about the z-axis, the matrix of spatial transformation parameters may be represented as follows:
wherein θ represents a directional angular deformation, and s represents a dimensional proportional deformation. s is(s) x Scaling scale s representing scaling of corresponding dimension scale in X-axis direction y The scale ratio indicating the scale ratio of the size ratio corresponding to the Y-axis direction.
S133: and transforming the mapping target by using the spatial transformation parameters to obtain a registered mapping target.
And transforming the direction and the size of the mapping target by using the spatial transformation parameters to obtain a transformed mapping target which is used as a registered mapping target. That is, it means that the mapping target is rotated clockwise around a specified axis (the direction of the axis is outward), and scaling is performed to obtain a transformed mapping target. The spatial transformation network can utilize spatial transformation parameters to transform the direction and the size of the mapping target, and the spatial transformation network adopts the existing plug-and-play network module and has no learning parameters.
In some embodiments, the registered map targets are oriented in the same direction and size as the map areas. Or, the size difference meets the preset size requirement. The preset size requirement includes that the size difference is smaller than a preset value, and the preset value can be 0, 0.1, 0.5, etc., which is not limited to the present application.
As an example, referring to fig. 11, by registering the mapping area and the mapping target, the direction of the registered mapping target is consistent with the direction of the mapping area, and the size of the registered mapping target is the same, so that the mapping effect is reasonable and real.
S14: and fusing the registered mapping target and the mapping area to obtain a target image subjected to mapping processing.
The registered mapping target and the mapping area of the original image can be fused by utilizing a fusion network, namely, the registered mapping target is mapped in the mapping area of the original image, and a target image subjected to mapping processing is obtained, wherein the target image has a target object of the mapping target. And the target image can be expressed as a target image obtained by mapping and expanding the original image.
Referring to fig. 12, as an example, after fusing the registered mapping target with the mapping region of the original image, a new image with the mapping target is synthesized, and intelligent mapping is completed, so as to obtain a target image after mapping processing.
In this embodiment, by acquiring the original image and the mapping target, the mapping region in the original image is identified, so that the position of the region required to be mapped in the original image can be automatically identified, the mapping region and the mapping target are registered to obtain a registered mapping target, and the direction and the size of the mapping target and the mapping region can be registered, so that the mapping target and the mapping region are more consistent, and therefore, the registered mapping target and the mapping region are fused to obtain a target image subjected to mapping processing, the mapping accuracy can be increased, and the mapping processing efficiency and the authenticity of the mapping effect can be improved.
Referring to fig. 13, fig. 13 is a flowchart illustrating an embodiment of a training method of the mapping process model of the present application. The method may comprise the steps of:
s21: and obtaining the image sample to be mapped and the mapping target sample.
The method for obtaining the to-be-attached image sample set and the attached image target sample may refer to a specific obtaining method for obtaining the original image and the attached image target, which is not described herein in detail.
S22: and respectively taking the image sample to be attached and the mapping target sample as an original image and a mapping target, and processing the image sample to be attached and the mapping target sample by utilizing a mapping processing model to realize the steps of any one of the mapping processing methods, so as to obtain mapping processing loss.
The mapping processing model comprises a template matching network, a direction sensing network, a registration network and a fusion network.
The template matching network is used to identify the mapped regions in the original image.
The direction sensing network is used for respectively acquiring the direction sensing characteristics of the mapping area sample and the mapping target sample of the image sample to be mapped.
The registration network is used for registering the mapping region sample and the mapping target sample to obtain a registered mapping target sample. Specifically, the direction sensing feature of the mapping region sample and the direction sensing feature of the mapping target sample are registered, the spatial transformation parameters are obtained, and the mapping target sample is transformed by using the spatial transformation parameters, so that the registered mapping target sample is obtained.
The fusion network is used for fusing the registered mapping target sample and the mapping region sample to obtain a target image subjected to mapping processing.
And respectively taking the image sample to be attached and the mapping target sample as an original image and a mapping target, and processing the image sample to be attached and the mapping target sample by utilizing the mapping processing model to obtain mapping processing loss.
In some implementations, the mapping processing penalty includes a sum of the region penalty, the registration penalty, and the fusion penalty.
In some embodiments, the area loss may be expressed as:
wherein, loss 1 Representing the area loss, (x, y) is the position coordinates of the real map area sample,the location coordinates of the sample of the map area predicted for the template matching network.
Parameters of the template matching network can be trained through region loss corresponding to the position coordinates of the mapped region samples.
In some embodiments, the registration loss may be expressed as:
wherein, loss 2 Representing the registration loss, θ is the true direction angle information,direction angle information predicted for the direction-aware network; s is the true size proportion information, +.>Is predicted size ratio information.
The parameters of the direction sensing network and the registration network can be trained together through registration loss obtained through the direction angle information and the size proportion information.
In some embodiments, the fusion loss may be expressed as:
wherein, loss 3 Representing the fusion loss, f is the true image,is a composite image sample.
Parameters of the fusion network can be trained through fusion loss, and the process can utilize difference supervision of a real mapping result (real image) and an image sample of a fusion mapping target (synthesized image sample) to generate a result.
For the implementation of this embodiment, reference may be made to the implementation process of the foregoing embodiment, which is not described herein.
S23: based on the mapping processing loss, training the mapping processing model to obtain a trained mapping processing model.
Based on the above-mentioned mapping processing loss including the sum of the region loss, registration loss and fusion loss, training each network of the mapping processing model, and adjusting parameters of the mapping processing model to obtain a trained mapping processing model.
For the above embodiments, the present application provides a mapping processing apparatus for implementing any step of the above mapping processing method.
Referring to fig. 14, fig. 14 is a schematic structural diagram of an embodiment of a mapping device of the present application.
The map processing device 30 includes an image module 31, an identification module 32, a registration module 33, and a fusion module 34.
The image module 31 is used for acquiring an original image and a mapping target.
The identification module 32 is used to identify the map area in the original image.
The registration module 33 is configured to register the mapping region and the mapping target to obtain a registered mapping target.
The fusion module 34 is configured to fuse the registered mapping target with the mapping region, so as to obtain a target image after mapping processing.
For the implementation of this embodiment, reference may be made to the implementation process of the foregoing embodiment, which is not described herein.
For the above embodiments, the present application provides a training device for a mapping process model, which is used to implement any step of training the mapping process model.
Referring to fig. 15, fig. 15 is a schematic structural diagram of an embodiment of a training device for mapping process model of the present application. The training device 40 of the mapping process model comprises a sample module 41, a processing module 42 and a training module 43.
The sample module 41 is used for obtaining an image sample to be mapped and a mapping target sample.
The processing module 42 is configured to take the image sample to be mapped and the target sample to be mapped as the original image and the target to be mapped respectively, and process the image sample to be mapped and the target sample to be mapped by using the mapping processing model to implement any one of the mapping processing methods, so as to obtain a mapping processing loss.
The training module 43 is configured to train the mapping process model based on the mapping process loss, to obtain a trained mapping process model.
For the implementation of this embodiment, reference may be made to the implementation process of the foregoing embodiment, which is not described herein.
For the foregoing embodiments, the present application provides a computer device, please refer to fig. 16, fig. 16 is a schematic structural diagram of an embodiment of the computer device of the present application. The computer device 50 comprises a memory 51 and a processor 52, wherein the memory 51 and the processor 52 are coupled to each other, and the memory 51 stores program data, and the processor 52 is configured to execute the program data to implement the steps of any embodiment of any one of the above-mentioned mapping processing method and the training method of the mapping processing model.
In the present embodiment, the processor 52 may also be referred to as a CPU (Central Processing Unit ). The processor 52 may be an integrated circuit chip having signal processing capabilities. Processor 52 may also be a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components. The general purpose processor may be a microprocessor or the processor 52 may be any conventional processor or the like.
For the method of the above embodiment, which may be implemented in the form of a computer program, the present application proposes a computer readable storage medium, please refer to fig. 17, fig. 17 is a schematic structural diagram of an embodiment of the computer readable storage medium of the present application. The computer readable storage medium 60 stores therein program data 61 that can be executed by a processor, and the program data 61 can be executed by the processor to implement the steps of any embodiment of any one of the above-mentioned mapping processing method and the training method of the mapping processing model.
The computer readable storage medium 60 of the present embodiment may be a medium such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, which may store the program data 61, or may be a server storing the program data 61, which may send the stored program data 61 to another device for operation, or may also run the stored program data 61 by itself.
In the several embodiments provided in the present application, it should be understood that the disclosed methods and apparatus may be implemented in other manners. For example, the apparatus embodiments described above are merely illustrative, e.g., the division of modules or units is merely a logical functional division, and there may be additional divisions when actually implemented, e.g., multiple units or components may be combined or integrated into another system, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical, or other forms.
The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed over a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiment.
In addition, each functional unit in each embodiment of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.
The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer-readable storage medium, which is a computer-readable storage medium. Based on such understanding, the technical solution of the present application may be embodied essentially or in part or all or part of the technical solution contributing to the prior art or in the form of a software product stored in a storage medium, including several instructions to cause an electronic device (which may be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of the methods of the embodiments of the present application.
It will be apparent to those skilled in the art that the modules or steps of the application described above may be implemented in a general purpose computing device, they may be centralized on a single computing device, or distributed over a network of computing devices, or they may alternatively be implemented in program code executable by computing devices, such that they may be stored in a computer readable storage medium for execution by computing devices, or they may be separately fabricated into individual integrated circuit modules, or multiple modules or steps within them may be fabricated into a single integrated circuit module. Thus, the present application is not limited to any specific combination of hardware and software.
The foregoing description is only exemplary embodiments of the present application and is not intended to limit the scope of the present application, and all equivalent structures or equivalent processes using the descriptions and the drawings of the present application, or direct or indirect application in other related technical fields are included in the scope of the present application.
Claims (10)
1. A method of mapping, comprising:
acquiring an original image and a mapping target;
identifying a map area in the original image;
registering the mapping area and the mapping target to obtain a registered mapping target;
and fusing the registered mapping target with the mapping region to obtain a target image subjected to mapping processing.
2. The method of claim 1, wherein registering the map region and the map target results in a registered map target, comprising:
respectively acquiring direction perception characteristics of the mapping area and the mapping target;
registering the direction sensing characteristics of the mapping area and the direction sensing characteristics of the mapping target to obtain space transformation parameters;
and transforming the mapping target by using the spatial transformation parameters to obtain a registered mapping target.
3. The method of claim 2, wherein the separately obtaining directional-aware features of the mapped region and the mapped object comprises:
respectively taking the mapping area and the mapping target as input images;
respectively processing the input images by using a direction sensing network to obtain direction sensing characteristics corresponding to the input images; the direction sensing network comprises a plurality of sensing convolution kernels, and the directions corresponding to different sensing convolution kernels are different.
4. The method according to claim 2, wherein registering the direction-aware features of the map region and the direction-aware features of the map object to obtain the spatial transformation parameters comprises:
registering the direction sensing characteristics of the mapping area and the direction sensing characteristics of the mapping target by using a registration network to obtain deformation parameters between the mapping area and the mapping target;
constructing and obtaining the space transformation parameters based on the deformation parameters;
the deformation parameters comprise a preset relation between the direction angle deformation and the size proportion deformation of the direction sensing feature.
5. The method of claim 2, wherein transforming the map object using the spatial transformation parameters results in a registered map object, comprising:
transforming the direction and the size of the mapping target by utilizing the space transformation parameters to obtain a transformed mapping target which is used as a registered mapping target; wherein the registered map target is identical in direction and size to the map region.
6. The method of claim 1, wherein the identifying a map region in the original image comprises:
inputting the original image into a template matching network;
matching the original image with a preset mapping area by using the template matching network so as to identify a sub-area matched with the original image;
taking the sub-region matched with the original image as the mapping region;
alternatively, the identifying the map area in the original image includes:
inputting the original image into a template matching network;
matching the mapping area of the original image with a preset mapping template by using the template matching network so as to identify a matched preset mapping template; wherein, the preset mapping template is provided with at least one preset mapping area;
acquiring position coordinates of a preset mapping area of the matched preset mapping template;
and taking the area of the original image at the position coordinates as the mapping area.
7. The method of claim 1, wherein the identifying a map region in the original image comprises:
performing target recognition on the original image, and determining an object area of a target object;
and selecting a preset area of the original image as the mapping area based on the object area, wherein the preset area is at least a partial area of the object area, or at least a partial overlapping area exists between the preset area and the object area.
8. A method of training a mapping process model, comprising:
obtaining an image sample to be mapped and a mapping target sample;
taking the image sample to be attached and the mapping target sample as an original image and a mapping target respectively, and processing the image sample to be attached and the mapping target sample by utilizing a mapping processing model to realize the steps of the method of any one of claims 1 to 7, so as to obtain mapping processing loss;
and training the mapping processing model based on the mapping processing loss to obtain a trained mapping processing model.
9. A computer device comprising a memory and a processor coupled to each other, the memory having stored therein program data, the processor being adapted to execute the program data to implement the steps of the method of any of claims 1 to 8.
10. A computer readable storage medium, characterized in that program data executable by a processor are stored, said program data being for implementing the steps of the method according to any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311483523.1A CN117710419A (en) | 2023-11-08 | 2023-11-08 | Mapping processing method, training method, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311483523.1A CN117710419A (en) | 2023-11-08 | 2023-11-08 | Mapping processing method, training method, computer equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117710419A true CN117710419A (en) | 2024-03-15 |
Family
ID=90155965
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311483523.1A Pending CN117710419A (en) | 2023-11-08 | 2023-11-08 | Mapping processing method, training method, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117710419A (en) |
-
2023
- 2023-11-08 CN CN202311483523.1A patent/CN117710419A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Rong et al. | Radial lens distortion correction using convolutional neural networks trained with synthesized images | |
| CN110909611B (en) | Method and device for detecting attention area, readable storage medium and terminal equipment | |
| Zhao et al. | SPHORB: A fast and robust binary feature on the sphere | |
| Goshtasby | 2-D and 3-D image registration: for medical, remote sensing, and industrial applications | |
| CN110675487B (en) | Three-dimensional face modeling and recognition method and device based on multi-angle two-dimensional face | |
| Xiang-yang et al. | Invariant quaternion radial harmonic Fourier moments for color image retrieval | |
| Xiaohui et al. | Automatic human body feature extraction and personal size measurement | |
| Tafti et al. | A comparative study on the application of SIFT, SURF, BRIEF and ORB for 3D surface reconstruction of electron microscopy images | |
| CN105488541A (en) | Natural feature point identification method based on machine learning in augmented reality system | |
| CN107644227A (en) | A kind of affine invariant descriptor of fusion various visual angles for commodity image search | |
| CN110704652A (en) | Vehicle image fine-grained retrieval method and device based on multiple attention mechanism | |
| Mekhalfi et al. | Toward an assisted indoor scene perception for blind people with image multilabeling strategies | |
| Liang et al. | Image registration using two-layer cascade reciprocal pipeline and context-aware dissimilarity measure | |
| Li et al. | Medical image stitching using parallel sift detection and transformation fitting by particle swarm optimization | |
| Paradowski et al. | Local keypoints and global affine geometry: Triangles and ellipses for image fragment matching | |
| CN107633506B (en) | Image symmetry detection method and device and terminal equipment | |
| CN112767457A (en) | Principal component analysis-based plane point cloud matching method and device | |
| CN111126508A (en) | Hopc-based improved heterogeneous image matching method | |
| CN117710419A (en) | Mapping processing method, training method, computer equipment and storage medium | |
| Liu et al. | Recognition and location of typical automotive parts based on the RGB-D camera | |
| Lavin-Delgado et al. | Fractional speeded up robust features detector with the Caputo-Fabrizio derivative | |
| CN114511631A (en) | Method and device for measuring height of visual object of camera and computer readable storage medium | |
| Hou et al. | Robust point correspondence with gabor scale-invariant feature transform for optical satellite image registration | |
| Zhang et al. | Effective image registration model using optimized KAZE algorithm | |
| Liu et al. | Image Mosaic Technology Based on Harris Corner Feature |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |