CN116228763B - Image processing method and system for eyeglass printing - Google Patents
Image processing method and system for eyeglass printing Download PDFInfo
- Publication number
- CN116228763B CN116228763B CN202310505728.9A CN202310505728A CN116228763B CN 116228763 B CN116228763 B CN 116228763B CN 202310505728 A CN202310505728 A CN 202310505728A CN 116228763 B CN116228763 B CN 116228763B
- Authority
- CN
- China
- Prior art keywords
- face
- feature map
- image
- decoding
- encoder
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000007639 printing Methods 0.000 title claims abstract description 92
- 238000003672 processing method Methods 0.000 title claims abstract description 31
- 239000011521 glass Substances 0.000 claims abstract description 85
- 238000001514 detection method Methods 0.000 claims abstract description 25
- 230000001131 transforming effect Effects 0.000 claims abstract description 10
- 239000010410 layer Substances 0.000 claims description 72
- 230000004927 fusion Effects 0.000 claims description 48
- 238000012545 processing Methods 0.000 claims description 47
- 238000005457 optimization Methods 0.000 claims description 29
- 238000010586 diagram Methods 0.000 claims description 18
- 238000009826 distribution Methods 0.000 claims description 17
- 230000001815 facial effect Effects 0.000 claims description 13
- 230000004913 activation Effects 0.000 claims description 10
- 238000004364 calculation method Methods 0.000 claims description 10
- 238000011176 pooling Methods 0.000 claims description 10
- 230000009466 transformation Effects 0.000 claims description 7
- 239000002356 single layer Substances 0.000 claims description 6
- 239000011159 matrix material Substances 0.000 claims description 4
- 238000004519 manufacturing process Methods 0.000 abstract description 25
- 230000002829 reductive effect Effects 0.000 abstract description 7
- 238000010146 3D printing Methods 0.000 description 24
- 238000005516 engineering process Methods 0.000 description 22
- 238000000034 method Methods 0.000 description 19
- 239000004973 liquid crystal related substance Substances 0.000 description 12
- 230000014509 gene expression Effects 0.000 description 11
- 238000004422 calculation algorithm Methods 0.000 description 7
- 230000004048 modification Effects 0.000 description 6
- 238000012986 modification Methods 0.000 description 6
- 230000008569 process Effects 0.000 description 6
- 238000003860 storage Methods 0.000 description 6
- 230000000694 effects Effects 0.000 description 5
- 230000007246 mechanism Effects 0.000 description 5
- 230000003287 optical effect Effects 0.000 description 4
- 230000000670 limiting effect Effects 0.000 description 3
- 230000036544 posture Effects 0.000 description 3
- 238000009877 rendering Methods 0.000 description 3
- 206010063385 Intellectualisation Diseases 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 2
- 238000013459 approach Methods 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000018109 developmental process Effects 0.000 description 2
- 230000015654 memory Effects 0.000 description 2
- 210000000887 face Anatomy 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 238000013178 mathematical model Methods 0.000 description 1
- 230000036961 partial effect Effects 0.000 description 1
- 230000002085 persistent effect Effects 0.000 description 1
- 230000001902 propagating effect Effects 0.000 description 1
- 230000003252 repetitive effect Effects 0.000 description 1
- 230000002441 reversible effect Effects 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000007619 statistical method Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B29—WORKING OF PLASTICS; WORKING OF SUBSTANCES IN A PLASTIC STATE IN GENERAL
- B29C—SHAPING OR JOINING OF PLASTICS; SHAPING OF MATERIAL IN A PLASTIC STATE, NOT OTHERWISE PROVIDED FOR; AFTER-TREATMENT OF THE SHAPED PRODUCTS, e.g. REPAIRING
- B29C64/00—Additive manufacturing, i.e. manufacturing of three-dimensional [3D] objects by additive deposition, additive agglomeration or additive layering, e.g. by 3D printing, stereolithography or selective laser sintering
- B29C64/30—Auxiliary operations or equipment
- B29C64/386—Data acquisition or data processing for additive manufacturing
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B33—ADDITIVE MANUFACTURING TECHNOLOGY
- B33Y—ADDITIVE MANUFACTURING, i.e. MANUFACTURING OF THREE-DIMENSIONAL [3-D] OBJECTS BY ADDITIVE DEPOSITION, ADDITIVE AGGLOMERATION OR ADDITIVE LAYERING, e.g. BY 3-D PRINTING, STEREOLITHOGRAPHY OR SELECTIVE LASER SINTERING
- B33Y50/00—Data acquisition or data processing for additive manufacturing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/60—Rotation of whole images or parts thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
- G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20092—Interactive image processing based on input by user
- G06T2207/20104—Interactive definition of region of interest [ROI]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30196—Human being; Person
- G06T2207/30201—Face
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Materials Engineering (AREA)
- Chemical & Material Sciences (AREA)
- Software Systems (AREA)
- Manufacturing & Machinery (AREA)
- Health & Medical Sciences (AREA)
- Geometry (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Databases & Information Systems (AREA)
- Multimedia (AREA)
- Computer Graphics (AREA)
- Evolutionary Computation (AREA)
- Computing Systems (AREA)
- Artificial Intelligence (AREA)
- Quality & Reliability (AREA)
- Mechanical Engineering (AREA)
- Optics & Photonics (AREA)
- Image Processing (AREA)
- Processing Or Creating Images (AREA)
Abstract
An image processing method and system for eyeglass printing are disclosed. Firstly, acquiring a face image acquired by a camera, then, passing the face image through a face region of interest generator to obtain a face region of interest image, then, performing key point detection on the face region of interest image to obtain a plurality of face key feature points, then, calculating the rotation angle and the scaling of the face region of interest image based on the face key feature points, transforming the face region of interest image based on the rotation angle and the scaling to obtain a transformed face region of interest image, then, passing the transformed face region of interest image through a 3DMM model to obtain a face 3D model, and finally, calculating parameters of glasses based on the face 3D model and performing glasses 3D model printing based on the parameters of the glasses. Thus, the production efficiency can be improved, and the production cost can be reduced.
Description
Technical Field
The present application relates to the field of intelligent printing, and more particularly, to an image processing method and system for eyeglass printing.
Background
With the development of computer technology for processing, analyzing and understanding images and the increasing maturity of 3D printing technology, image recognition technology and 3D printing technology have been perfectly combined gradually and widely applied to various production and manufacturing industries.
For example, chinese patent application No. 202011404717.4 discloses an image processing method for 3D printing, comprising the steps of: projecting the layer to be printed to generate an initial image; moving the initial image along a first direction, so that the initial image moves a first distance from an initial position to a first limiting position to obtain a first image, wherein the first distance is not greater than the length of one pixel in the initial image along the first direction; fusing the initial image and the first image to obtain a fused image; and printing the fused image. The image processing system for 3D printing comprises a projection module, a mobile module, a fusion module and a printing module. By the method and the system, the gray value of the adjacent pixels in the image can be changed more gradually, the printing precision is improved, and a 3D printing model with a smoother surface can be obtained under the condition that the resolution of the existing optical machine is not changed.
Furthermore, as disclosed in the chinese patent application No. 202210621905.5, an image processing method and apparatus, a storage medium, and a terminal in 3D printing relate to the technical field of 3D printing, and mainly aim at solving the problem of poor slice layer image rendering performance in the existing 3D printing. Comprising the following steps: acquiring contour line information of a target printing model slice layer and calculating contour line slope of each contour line according to the contour line information; determining edge pixel blocks covered by all the contour lines according to the slope of the contour lines, and calculating gray values of the edge pixel blocks; and rendering the corresponding edge pixel block based on the gray value of the edge pixel block, so that image rendering in 3D printing is improved.
With the development of market demand, image processing and 3D printing technologies are also beginning to be used gradually in the production of customized glasses. Eyeglass printing of 3D printing technology is an emerging eyeglass manufacturing approach based on the recognition and processing of user facial images, and then using a 3D printer to manufacture eyeglass frames and eyeglass accessories. This technique allows manufacturers to produce fully customized eyewear according to the individual needs of the customer.
The key technology of glasses printing based on 3D printing technology is: the face is accurately 3D modeled, and customized eyeglass parameters are calculated based on the obtained face 3D model, so that 3D printing can be performed based on the eyeglass parameters.
Some schemes for carrying out face 3D modeling based on face two-dimensional images exist, but the modeling precision of the modeling schemes also has a large optimization space, so that full intellectualization and full automation cannot be truly realized due to eyeglass printing based on a 3D printing technology. In the existing glasses printing scheme based on the 3D printing technology, related parameters are also required to be manually measured to carry out auxiliary judgment so as to ensure that the finally formed printing glasses can meet the requirement of customization.
Thus, an optimized image processing scheme for eyeglass printing is desired.
Disclosure of Invention
The present application has been made in order to solve the above technical problems. The embodiment of the application provides an image processing method and system for eyeglass printing. Firstly, acquiring a face image acquired by a camera, then, passing the face image through a face region of interest generator to obtain a face region of interest image, then, performing key point detection on the face region of interest image to obtain a plurality of face key feature points, then, calculating the rotation angle and the scaling of the face region of interest image based on the face key feature points, transforming the face region of interest image based on the rotation angle and the scaling to obtain a transformed face region of interest image, then, passing the transformed face region of interest image through a 3DMM model to obtain a face 3D model, and finally, calculating parameters of glasses based on the face 3D model and performing glasses 3D model printing based on the parameters of the glasses. Thus, the production efficiency can be improved, and the production cost can be reduced.
According to an aspect of the present application, there is provided an image processing method for eyeglass printing, including:
acquiring a face image acquired by a camera;
passing the face image through a face region of interest generator based on an encoder-decoder structure to obtain a face region of interest image, wherein the encoder and the decoder have symmetrical network structures;
performing key point detection on the image of the region of interest of the human face to obtain a plurality of key characteristic points of the human face;
calculating a rotation angle and a scaling of the face region-of-interest image based on the plurality of face key feature points, and transforming the face region-of-interest image based on the rotation angle and the scaling to obtain a transformed face region-of-interest image;
the transformed face region of interest image is passed through a 3DMM model to obtain a face 3D model;
based on the face 3D model, calculating parameters of the glasses; and
and printing the 3D model of the glasses based on the parameters of the glasses.
In the above image processing method for eyeglass printing, the face image is passed through a face region of interest generator based on an encoder-decoder structure to obtain a face region of interest image, wherein the encoder and the decoder have a symmetrical network structure, and the method comprises:
Inputting the face image into the encoder to obtain first to fifth face feature maps, wherein the encoder comprises five convolution layers;
inputting the fifth facial feature map into a first decoding module of the decoder to obtain a first decoding feature map;
fusing the first decoding feature map and the fifth face feature map to obtain a first fused decoding feature map; and
and performing feature distribution optimization on the first fusion decoding feature map to obtain an optimized first fusion decoding feature map as input of a second decoding module of the decoder.
In the above image processing method for eyeglass printing, the face image is input to the encoder to obtain first to fifth face feature maps, wherein the encoder includes five convolution layers, including:
inputting the face image into a first convolution layer of the encoder to obtain the first face feature map;
inputting the first face feature map into a second convolution layer of the encoder to obtain the second face feature map;
inputting the second face feature map into a third convolution layer of the encoder to obtain the third face feature map;
Inputting the third face feature map into a fourth convolution layer of the encoder to obtain the fourth face feature map; and
and inputting the fourth face feature map into a fifth convolution layer of the encoder to obtain the fifth face feature map.
In the above image processing method for eyeglass printing, inputting the face image into a first convolution layer of the encoder to obtain the first face feature map includes:
and respectively carrying out two-dimensional convolution processing, feature matrix-based mean value pooling processing and nonlinear activation processing on input data by using a first convolution layer of the encoder to output the first face feature map by the first convolution layer of the encoder, wherein the input of the first convolution layer of the encoder is the face image.
In the above image processing method for eyeglass printing, inputting the fifth face feature map into a first decoding module of the decoder to obtain a first decoded feature map, including:
and performing deconvolution, pooling and nonlinear activation on the fifth face feature map by using a first decoding module of the decoder to obtain the first decoding feature map.
In the above image processing method for eyeglass printing, fusing the first decoding feature map and the fifth face feature map to obtain a first fused decoding feature map includes:
fusing the first decoding feature map and the fifth face feature map with the following cascade formula to obtain the first fused decoding feature map;
wherein, the cascade formula is:
；
wherein, the liquid crystal display device comprises a liquid crystal display device,representing said first decoding profile, < >>Representing the fifth facial feature map, < >>Representing a cascade function->And representing the first fusion decoding characteristic diagram.
In the above image processing method for eyeglass printing, performing feature distribution optimization on the first fused decoding feature map to obtain an optimized first fused decoding feature map as an input of a second decoding module of the decoder, including:
performing feature distribution optimization on the first fusion decoding feature map by using the following optimization formula to obtain the optimized first fusion decoding feature map;
wherein, the optimization formula is:
wherein, the liquid crystal display device comprises a liquid crystal display device,representing said first fused decoding profile, < >>Representing the optimized first fusion decoding characteristic diagram,representing a single layer convolution operation,/->Position-by-position addition of the representation feature map, +. >Position-wise subtraction of the representation characteristic map, +.>Position-wise multiplication of the characteristic map, and +.>And->Is a bias profile.
According to another aspect of the present application, there is provided an image processing system for eyeglass printing, comprising:
the image acquisition module is used for acquiring the face image acquired by the camera;
a coding and decoding module, configured to pass the face image through a face region of interest generator based on an encoder-decoder structure to obtain a face region of interest image, where the encoder and the decoder have a symmetrical network structure;
the key point detection module is used for carrying out key point detection on the image of the region of interest of the human face so as to obtain a plurality of key feature points of the human face;
the image transformation module is used for calculating the rotation angle and the scaling of the face region-of-interest image based on the plurality of face key feature points, and transforming the face region-of-interest image based on the rotation angle and the scaling to obtain a transformed face region-of-interest image;
the 3DMM module is used for enabling the transformed face region-of-interest image to pass through a 3DMM model to obtain a face 3D model;
The parameter calculation module is used for calculating parameters of the glasses based on the face 3D model; and
and the printing module is used for printing the 3D model of the glasses based on the parameters of the glasses.
In the above image processing system for eyeglass printing, the codec module includes:
the encoding unit is used for inputting the face image into the encoder to obtain first to fifth face feature images, wherein the encoder comprises five convolution layers;
the first decoding unit is used for inputting the fifth face feature map into a first decoding module of the decoder to obtain a first decoding feature map;
the fusion unit is used for fusing the first decoding feature map and the fifth face feature map to obtain a first fusion decoding feature map; and
and the second decoding unit is used for optimizing the characteristic distribution of the first fusion decoding characteristic diagram to obtain an optimized first fusion decoding characteristic diagram which is used as the input of a second decoding module of the decoder.
In the above image processing system for eyeglass printing, the encoding unit is configured to:
inputting the face image into a first convolution layer of the encoder to obtain the first face feature map;
Inputting the first face feature map into a second convolution layer of the encoder to obtain the second face feature map;
inputting the second face feature map into a third convolution layer of the encoder to obtain the third face feature map;
inputting the third face feature map into a fourth convolution layer of the encoder to obtain the fourth face feature map; and
and inputting the fourth face feature map into a fifth convolution layer of the encoder to obtain the fifth face feature map.
Compared with the prior art, the image processing method and the system for printing the glasses are characterized in that firstly, a face image acquired by a camera is acquired, then the face image is processed through a face region of interest generator to obtain a face region of interest image, then key point detection is conducted on the face region of interest image to obtain a plurality of face key feature points, then the rotation angle and the scaling of the face region of interest image are calculated and obtained based on the face key feature points, the face region of interest image is transformed based on the rotation angle and the scaling to obtain a transformed face region of interest image, then the transformed face region of interest image is processed through a 3DMM model to obtain a face 3D model, finally, parameters of the glasses are calculated and the glasses 3D model is printed based on the parameters of the glasses. Thus, the production efficiency can be improved, and the production cost can be reduced.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the description of the embodiments will be briefly introduced below, it being obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art. The following drawings are not intended to be drawn to scale, with emphasis instead being placed upon illustrating the principles of the present application.
Fig. 1 is an application scenario diagram of an image processing method for eyeglass printing according to an embodiment of the present application.
Fig. 2 is a flowchart of an image processing method for eyeglass printing according to an embodiment of the present application.
Fig. 3 is a schematic architecture diagram of an image processing method for eyeglass printing according to an embodiment of the present application.
Fig. 4 is a flowchart of substep S120 of the image processing method for eyeglass printing according to the embodiment of the present application.
Fig. 5 is a flowchart of sub-step S121 of the image processing method for eyeglass printing according to the embodiment of the present application.
Fig. 6 is a block diagram of an image processing system for eyeglass printing according to an embodiment of the present application.
Fig. 7 is a block diagram of the codec module in the image processing system for glasses printing according to the embodiment of the present application.
Detailed Description
The following description of the embodiments of the present application will be made clearly and fully with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some, but not all embodiments of the present application. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments of the present application without making any inventive effort, are also within the scope of the present application.
As used in this application and in the claims, the terms "a," "an," "the," and/or "the" are not specific to the singular, but may include the plural, unless the context clearly dictates otherwise. In general, the terms "comprises" and "comprising" merely indicate that the steps and elements are explicitly identified, and they do not constitute an exclusive list, as other steps or elements may be included in a method or apparatus.
Although the present application makes various references to certain modules in a system according to embodiments of the present application, any number of different modules may be used and run on a user terminal and/or server. The modules are merely illustrative, and different aspects of the systems and methods may use different modules.
Flowcharts are used in this application to describe the operations performed by systems according to embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed in order precisely. Rather, the various steps may be processed in reverse order or simultaneously, as desired. Also, other operations may be added to or removed from these processes.
Hereinafter, example embodiments according to the present application will be described in detail with reference to the accompanying drawings. It should be apparent that the described embodiments are only some of the embodiments of the present application and not all of the embodiments of the present application, and it should be understood that the present application is not limited by the example embodiments described herein.
Aiming at the technical problems, the technical conception of the application is as follows: the method comprises the steps of detecting a human face region of interest through a human face region of interest generator based on an encoder-decoder, improving consistency between the human face region of interest and a real human face through key point detection, rotation angle calculation, scaling and other operations, obtaining a high-quality human face 3D model through a 3DMM model, and further achieving calculation of eyeglass parameters and printing of the eyeglass 3D model. Therefore, the accuracy of face modeling is greatly improved through the data processing flow, meanwhile, the requirements of manual intervention are eliminated through operations such as key point detection and rotation angle calculation, full automation and full intellectualization are realized, and completely customized glasses can be produced according to individual requirements of customers, so that better use experience is provided. Meanwhile, the requirements of eyeglass manufacturers on efficiency and cost can be met, the production efficiency is improved, and the production cost is reduced.
Specifically, in the technical scheme of the application, a face image acquired by a camera is firstly acquired. It should be understood that in the technical solution of the present application, the technical route is to perform face 3D modeling based on a face two-dimensional image, so that a face two-dimensional image of a customized object needs to be acquired first. It should be understood that after the face image is acquired, a computer graphics technology may be used to convert the image into a three-dimensional face model, and further calculate parameters of the glasses according to the obtained three-dimensional face model, so as to print the 3D model of the glasses.
It is worth mentioning that, in the technical scheme of this application, through the camera gathers the production flow that the people's face image can realize the full automatization. Specifically, the camera can be connected with the 3D printer, so that a series of automatic processes from the collection of face images to the printing of glasses can be realized. The method can greatly improve the production efficiency, reduce the manufacturing cost and meet the personalized customization requirement of customers.
The face image is then passed through a face region of interest generator based on an encoder-decoder structure to obtain a face region of interest image, wherein the encoder and the decoder have a symmetrical network structure. Here, the technical purpose of passing the face image through a face region of interest generator based on an encoder-decoder structure to obtain a face region of interest image is to improve the accuracy and precision of the 3D modeling of the face.
Those of ordinary skill in the art will appreciate that in conventional face modeling methods, it is often necessary to manually mark or select a region of interest of a face and then perform the related operations. This approach, while having a certain degree of accuracy, requires significant labor and time costs. The human face region of interest generator based on the encoder-decoder structure can automatically locate and extract the human face region of interest, so that full-automatic modeling is realized.
Specifically, in the technical scheme of the application, a symmetrical encoder-decoder network structure is adopted to firstly encode the face image to obtain a face encoding feature map, and the face encoder feature map is restored to a high-resolution face image by using the decoder. Further, image semantic segmentation is carried out on the face image with high resolution to obtain the face region-of-interest image.
In a specific example of the present application, the encoder includes five connected convolution layers, which are defined as first to fifth convolution layers, and after the face image is input into the encoder, the encoder can perform stepwise deep convolution encoding on the face image through the five connected convolution layers to obtain first to fifth face feature maps. Then, based on the layer jump connection between the encoder and the decoder, the fifth face feature map is input into the decoder for decoding to obtain the high-resolution face image.
In a specific example of the present application, the decoder includes five deconvolution layers for deconvolution encoding input data to obtain a decoding profile. The first decoding module of the decoder is described here as an example: a layer jump connection between the encoder and the decoder, and a decoding process of the first decoding module.
Firstly, inputting the fifth face feature map into a first decoding module of the decoder to obtain a first decoding feature map, wherein the first decoding module performs convolution coding, pooling processing and nonlinear activation processing on the fifth face feature map to obtain the first decoding feature map; next, the first decoding feature map and the fifth face feature map are fused to obtain a first fused decoding feature map, that is, the output of the fifth convolution layer of the encoder is transferred to the first decoding module of the decoder and is subjected to feature fusion with the output of the first decoding module (that is, the first decoding feature map).
Here, considering that the first fused decoding feature map is obtained by fusing the first decoding feature map and the fifth face feature map, it contains both the image semantic decoding feature of the first decoding feature map and the image semantic encoding feature of the fifth face feature map, and the fifth face feature map has a higher image semantic encoding abstraction degree, whereas the first decoding feature map has a lower image decoding abstraction degree with respect to the fifth face feature map, and therefore has feature redundancy about the face image on different feature encoding and decoding abstraction scales, which affects the expression effect of the first fused decoding feature map, thereby reducing the accuracy of the decoding result.
Thus, the applicant of the present application decodes the first fused decoding profile, e.g. denoted asFeature redundancy optimization based on low-cost bottleneck-mechanism stacking is performed to obtain an optimized first fusion decoding feature graph, for example, the feature redundancy optimization is recorded as
The method is specifically expressed as follows:
representing a single layer convolution operation,/->、/>And->Respectively representing the position-by-position addition, subtraction and multiplication of the feature maps, and +.>And->For biasing the feature map, for example, a global mean feature map or a unit feature map of the first fused decoding feature map may be initially set, wherein the initial biasing feature map ∈ ->And->Different.
Here, the feature redundancy optimization based on the low-cost bottleneck mechanism stack may perform feature expansion using the low-cost bottleneck mechanism of the multiply-add stack of two low-cost transform features, and match the residual paths by biasing the stack channels with uniform values, so as to reveal hidden distribution information under intrinsic features in the redundancy features through low-cost operation transformation similar to the basic residual modules, so as to obtain more intrinsic expression of features through simple and effective convolution operation architecture, thereby optimizing redundant feature expression of the first fusion decoding feature map, improving expression effect of the first fusion decoding feature map, and improving accuracy of decoding result of the second decoding module.
That is, in the technical solution of the present application, after the first fused decoding feature map is obtained, feature distribution optimization is performed on the first fused decoding feature map to obtain an optimized first fused decoding feature map as an input of a second decoding module of the decoder.
It should be understood that although only the first decoding module of the decoder is illustrated as an example: the layer jump connection between the encoder and the decoder, and the decoding process of the first decoding module should be easily understood based on the same principle, and the decoding processes of the second to fifth decoding modules are not expanded again in order to avoid redundancy.
And after the human face region-of-interest image is obtained, carrying out key point detection on the human face region-of-interest image to obtain a plurality of human face key feature points. It should be understood that the key point detection of the image of the region of interest of the face to obtain a plurality of key feature points of the face is one of important steps of modeling the face in the eyeglass printing based on the 3D printing technology, which can determine the shape and position of the face more accurately, thereby improving the precision and accuracy of the subsequent calculation.
By means of a key point detection algorithm, key points (such as eyes, nose, mouth and the like) at specific positions in the image of the region of interest of the face can be automatically identified. These key points may provide detailed information about the face, such as facial contours, expressions, etc. It should be appreciated that in eyeglass printing based on 3D printing technology, the key point detection algorithm can help us obtain the exact position and angle of the eyeglass on the face, so as to better calculate eyeglass parameters. By acquiring the key feature points, the shape and the position of the face can be more accurately represented and described, and a foundation is laid for subsequent transformation and 3D modeling.
And then, based on the plurality of face key feature points, calculating a rotation angle and a scaling of the face region-of-interest image, and based on the rotation angle and the scaling, transforming the face region-of-interest image to obtain a transformed face region-of-interest image. In eyeglass printing based on 3D printing technology, since the face sizes and shapes of different people are different, the acquired face images need to be appropriately transformed for better analysis and modeling. Specifically, the rotation angle and the scaling of the face region-of-interest image are obtained through calculation, and the face region-of-interest image can be converted into a standard size and direction, so that better support is provided for subsequent 3D modeling and other operations. Then, by transforming the image of the region of interest of the face, the influence caused by deformation and posture can be effectively eliminated, so that the sizes and the shapes of all faces can be uniformly represented and processed. Therefore, more accurate and fine data can be provided for subsequent face 3D modeling, and better eyeglass customization and individuation effects are achieved.
And then, the transformed face region-of-interest image is passed through a 3DMM model to obtain a face 3D model. In eyeglass printing based on the 3D printing technology, the eyeglass to be customized needs to accurately establish a 3D model of a human face. By passing the transformed face region of interest image through a 3DMM (3D Morphable Model) model, the 2D face image can be converted into a 3D face model, and the shape, texture, etc. of the model can be finely controlled. The 3DMM model is a mathematical model obtained by carrying out statistical analysis by utilizing a large number of known face sample data, a large number of face 3D models with different postures, expressions and ages can be quickly and efficiently generated by using the 3DMM model, and the 3D models can be correspondingly adjusted and modified according to the sizes and the shapes of the glasses so as to adapt to the demands of different clients.
And then, calculating parameters of the glasses based on the face 3D model. In eyeglass printing based on 3D printing technology, eyeglass to be customized needs to calculate parameters of the eyeglass, such as contour, size, position, etc., of the eyeglass according to individual characteristics of customers. By calculating the parameters of the glasses based on the face 3D model, the design and manufacture of the glasses can be matched with the facial features of the customer, and the personalized requirements of the customer on the glasses can be met. Specifically, parameters of the glasses, including width, height, frame shape, etc., of the glasses are calculated by adopting a method of combining a face 3D model and a glasses 3D model. Through the calculated parameters, the 3D model of the glasses suitable for different customers can be generated, and further modification and adjustment can be carried out according to the requirements of the customers so as to meet the personalized requirements of the customers.
Specifically, based on the face 3D model, the process of calculating parameters of the glasses includes the following steps: step 1: positioning the eyeglass region: the position and the gesture of the glasses in the face 3D model can be positioned by matching the glasses model based on the face 3D model; step 2: calculating the size of the glasses: the size of the glasses can be calculated by measuring the width, the height and other data of the frames of the glasses; step 3: calculating the positions of the glasses: by determining the position and the posture of the glasses in the face 3D model, the distance, the inclination angle and other data of the glasses can be calculated, so that the position of the glasses can be accurately determined, and the following steps are carried out: calculating the shape of the glasses: by modeling the eyeglass rim, the shape and contour of the eyeglass can be calculated and matched to the facial features of the customer. Through the steps, the parameters of the glasses can be obtained based on the face 3D model, and the 3D models of the glasses suitable for different clients can be generated, so that the personalized requirements of the clients are met.
Finally, eyeglass 3D model printing is performed based on the eyeglass parameters. In this way, fully customized eyewear may be produced according to the customer's personalized needs, providing a better use experience. Meanwhile, the requirements of eyeglass manufacturers on efficiency and cost can be met, the production efficiency is improved, and the production cost is reduced.
Fig. 1 is an application scenario diagram of an image processing method for eyeglass printing according to an embodiment of the present application. As shown in fig. 1, in this application scenario, first, a face image (e.g., D illustrated in fig. 1) acquired by a camera (e.g., C illustrated in fig. 1) is acquired, then, the face image is input to a server (e.g., S illustrated in fig. 1) in which an image processing algorithm for glasses printing is deployed, wherein the server is capable of processing the face image using the image processing algorithm for glasses printing to obtain a face 3D model, then, based on the face 3D model, parameters of glasses are calculated, and glasses 3D model printing is performed based on the parameters of the glasses.
Having described the basic principles of the present application, various non-limiting embodiments of the present application will now be described in detail with reference to the accompanying drawings.
Fig. 2 is a flowchart of an image processing method for eyeglass printing according to an embodiment of the present application. As shown in fig. 2, the image processing method for eyeglass printing according to the embodiment of the present application includes the steps of: s110, acquiring a face image acquired by a camera; s120, the face image is passed through a face region of interest generator based on an encoder-decoder structure to obtain a face region of interest image, wherein the encoder and the decoder have symmetrical network structures; s130, detecting key points of the image of the region of interest of the human face to obtain a plurality of key feature points of the human face; s140, calculating a rotation angle and a scaling ratio of the face region-of-interest image based on the plurality of face key feature points, and transforming the face region-of-interest image based on the rotation angle and the scaling ratio to obtain a transformed face region-of-interest image; s150, the transformed face region-of-interest image is passed through a 3DMM model to obtain a face 3D model; s160, calculating parameters of the glasses based on the face 3D model; and S170, printing the 3D model of the glasses based on the parameters of the glasses.
Fig. 3 is a schematic architecture diagram of an image processing method for eyeglass printing according to an embodiment of the present application. As shown in fig. 3, in the network architecture, first, a face image acquired by a camera is acquired; then, the face image is passed through a face region of interest generator based on an encoder-decoder structure to obtain a face region of interest image, wherein the encoder and the decoder have a symmetrical network structure; then, carrying out key point detection on the image of the region of interest of the human face to obtain a plurality of key characteristic points of the human face; then, based on the plurality of face key feature points, calculating to obtain a rotation angle and a scaling of the face region-of-interest image, and based on the rotation angle and the scaling, transforming the face region-of-interest image to obtain a transformed face region-of-interest image; then, the transformed face region-of-interest image is passed through a 3DMM model to obtain a face 3D model; then, based on the face 3D model, calculating parameters of the glasses; finally, eyeglass 3D model printing is performed based on the eyeglass parameters.
More specifically, in step S110, a face image acquired by a camera is acquired. It should be understood that the technical solution of the present application performs face 3D modeling based on a face two-dimensional image, and therefore, first, a face two-dimensional image of a customized object needs to be acquired. It should be understood that after the face image is acquired, a computer graphics technology may be used to convert the image into a three-dimensional face model, and further calculate parameters of the glasses according to the obtained three-dimensional face model, so as to print the 3D model of the glasses.
More specifically, in step S120, the face image is passed through a face region of interest generator based on an encoder-decoder structure to obtain a face region of interest image, wherein the encoder and the decoder have a symmetrical network structure. Here, the technical purpose of passing the face image through a face region of interest generator based on an encoder-decoder structure to obtain a face region of interest image is to improve the accuracy and precision of the 3D modeling of the face. The human face region of interest generator based on the encoder-decoder structure can automatically locate and extract the human face region of interest, thereby realizing full-automatic modeling.
Here, the encoder includes five connected convolution layers, defined as first to fifth convolution layers, and after the face image is input to the encoder, the encoder can progressively and deeply convolutionally encode the face image through the five connected convolution layers to obtain first to fifth face feature maps. Then, based on the layer jump connection between the encoder and the decoder, the fifth face feature map is input into the decoder for decoding to obtain the high-resolution face image. The decoder comprises five deconvolution layers for deconvolution encoding the input data to obtain a decoding profile, respectively. The first decoder module of the decoder performs partial convolution encoding, pooling and nonlinear activation processing on the fifth face feature map to obtain the first decoding feature map; next, the first decoding feature map and the fifth face feature map are fused to obtain a first fused decoding feature map, that is, the output of the fifth convolution layer of the encoder is transferred to the first decoding module of the decoder and is subjected to feature fusion with the output of the first decoding module (that is, the first decoding feature map).
Accordingly, in one specific example, as shown in fig. 4, the face image is passed through a face region of interest generator based on an encoder-decoder structure to obtain a face region of interest image, wherein the encoder and the decoder have a symmetrical network structure, comprising: s121, inputting the face image into the encoder to obtain first to fifth face feature images, wherein the encoder comprises five convolution layers; s122, inputting the fifth facial feature map into a first decoding module of the decoder to obtain a first decoding feature map; s123, fusing the first decoding feature map and the fifth face feature map to obtain a first fused decoding feature map; and S124, performing feature distribution optimization on the first fusion decoding feature map to obtain an optimized first fusion decoding feature map as an input of a second decoding module of the decoder.
Accordingly, in one specific example, as shown in fig. 5, the face image is input to the encoder to obtain first to fifth face feature maps, wherein the encoder includes five convolution layers, including: s1211, inputting the face image into a first convolution layer of the encoder to obtain the first face feature map; s1212, inputting the first face feature map into a second convolution layer of the encoder to obtain the second face feature map; s1213, inputting the second face feature map into a third convolution layer of the encoder to obtain the third face feature map; s1214, inputting the third face feature map into a fourth convolution layer of the encoder to obtain the fourth face feature map; and S1215, inputting the fourth face feature map into a fifth convolution layer of the encoder to obtain the fifth face feature map.
Accordingly, in one specific example, inputting the face image into a first convolution layer of the encoder to obtain the first face feature map includes: and respectively carrying out two-dimensional convolution processing, feature matrix-based mean value pooling processing and nonlinear activation processing on input data by using a first convolution layer of the encoder to output the first face feature map by the first convolution layer of the encoder, wherein the input of the first convolution layer of the encoder is the face image.
Accordingly, in one specific example, inputting the fifth face feature map into a first decoding module of the decoder to obtain a first decoded feature map includes: and performing deconvolution, pooling and nonlinear activation on the fifth face feature map by using a first decoding module of the decoder to obtain the first decoding feature map.
Accordingly, in a specific example, fusing the first decoding feature map and the fifth face feature map to obtain a first fused decoding feature map includes: fusing the first decoding feature map and the fifth face feature map with the following cascade formula to obtain the first fused decoding feature map; wherein, the cascade formula is:
Wherein, the liquid crystal display device comprises a liquid crystal display device,representing said first decoding profile, < >>Representing the fifth facial feature map, < >>Representing a cascade function->And representing the first fusion decoding characteristic diagram.
Here, considering that the first fused decoding feature map is obtained by fusing the first decoding feature map and the fifth face feature map, it contains both the image semantic decoding feature of the first decoding feature map and the image semantic encoding feature of the fifth face feature map, and the fifth face feature map has a higher image semantic encoding abstraction degree, whereas the first decoding feature map has a lower image decoding abstraction degree with respect to the fifth face feature map, and therefore has feature redundancy about the face image on different feature encoding and decoding abstraction scales, which affects the expression effect of the first fused decoding feature map, thereby reducing the accuracy of the decoding result. Therefore, the applicant of the present application performs feature redundancy optimization on the first fusion decoding feature map based on the low-cost bottleneck mechanism stack, so as to obtain an optimized first fusion decoding feature map.
Accordingly, in a specific example, performing feature distribution optimization on the first fused decoding feature map to obtain an optimized first fused decoding feature map as an input of a second decoding module of the decoder, including: performing feature distribution optimization on the first fusion decoding feature map by using the following optimization formula to obtain the optimized first fusion decoding feature map; wherein, the optimization formula is:
Wherein, the liquid crystal display device comprises a liquid crystal display device,representing said first fused decoding profile, < >>Representing the optimized first fusion decoding characteristic diagram,representing a single layer convolution operation,/->Position-by-position addition of the representation feature map, +.>Position-wise subtraction of the representation characteristic map, +.>Position-wise multiplication of the characteristic map, and +.>And->Is a bias profile.
Here, the feature redundancy optimization based on the low-cost bottleneck mechanism stack may perform feature expansion using the low-cost bottleneck mechanism of the multiply-add stack of two low-cost transform features, and match the residual paths by biasing the stack channels with uniform values, so as to reveal hidden distribution information under intrinsic features in the redundancy features through low-cost operation transformation similar to the basic residual modules, so as to obtain more intrinsic expression of features through simple and effective convolution operation architecture, thereby optimizing redundant feature expression of the first fusion decoding feature map, improving expression effect of the first fusion decoding feature map, and improving accuracy of decoding result of the second decoding module.
More specifically, in step S130, the key point detection is performed on the image of the region of interest of the face to obtain a plurality of key feature points of the face. It should be understood that the key point detection of the image of the region of interest of the face to obtain a plurality of key feature points of the face is one of important steps of modeling the face in the eyeglass printing based on the 3D printing technology, which can determine the shape and position of the face more accurately, thereby improving the precision and accuracy of the subsequent calculation. By means of a key point detection algorithm, key points (such as eyes, nose, mouth and the like) at specific positions in the image of the region of interest of the face can be automatically identified. These key points may provide detailed information about the face, such as facial contours, expressions, etc. It should be appreciated that in eyeglass printing based on 3D printing technology, the key point detection algorithm can help us obtain the exact position and angle of the eyeglass on the face, so as to better calculate eyeglass parameters. By acquiring the key feature points, the shape and the position of the face can be more accurately represented and described, and a foundation is laid for subsequent transformation and 3D modeling.
More specifically, in step S140, based on the plurality of face key feature points, a rotation angle and a scaling of the face region-of-interest image are calculated, and based on the rotation angle and the scaling, the face region-of-interest image is transformed to obtain a transformed face region-of-interest image. In eyeglass printing based on 3D printing technology, since the face sizes and shapes of different people are different, the acquired face images need to be appropriately transformed for better analysis and modeling. Specifically, the rotation angle and the scaling of the face region-of-interest image are obtained through calculation, and the face region-of-interest image can be converted into a standard size and direction, so that better support is provided for subsequent 3D modeling and other operations.
More specifically, in step S150, the transformed face region of interest image is passed through a 3DMM model to obtain a face 3D model. By passing the transformed face region of interest image through a 3DMM (3D Morphable Model) model, the 2D face image can be converted into a 3D face model, and the shape, texture, etc. of the model can be finely controlled.
More specifically, in step S160, parameters of the glasses are calculated based on the face 3D model. In eyeglass printing based on 3D printing technology, eyeglass to be customized needs to calculate parameters of the eyeglass, such as contour, size, position, etc., of the eyeglass according to individual characteristics of customers. By calculating the parameters of the glasses based on the face 3D model, the design and manufacture of the glasses can be matched with the facial features of the customer, and the personalized requirements of the customer on the glasses can be met. Specifically, parameters of the glasses, including width, height, frame shape, etc., of the glasses are calculated by adopting a method of combining a face 3D model and a glasses 3D model. Through the calculated parameters, the 3D model of the glasses suitable for different customers can be generated, and further modification and adjustment can be carried out according to the requirements of the customers so as to meet the personalized requirements of the customers.
More specifically, in step S170, eyeglass 3D model printing is performed based on the parameters of the eyeglass. In this way, fully customized eyewear may be produced according to the customer's personalized needs, providing a better use experience. Meanwhile, the requirements of eyeglass manufacturers on efficiency and cost can be met, the production efficiency is improved, and the production cost is reduced.
In summary, according to the image processing method for printing with glasses according to the embodiment of the application, firstly, a face image acquired by a camera is acquired, then the face image is passed through a face region of interest generator to obtain a face region of interest image, then key point detection is performed on the face region of interest image to obtain a plurality of face key feature points, then based on the plurality of face key feature points, a rotation angle and a scaling ratio of the face region of interest image are calculated, and based on the rotation angle and the scaling ratio, the face region of interest image is transformed to obtain a transformed face region of interest image, then the transformed face region of interest image is passed through a 3DMM model to obtain a face 3D model, finally, based on the face 3D model, parameters of the glasses are calculated, and glasses 3D model printing is performed based on the parameters of the glasses. Thus, the production efficiency can be improved, and the production cost can be reduced.
Fig. 6 is a block diagram of an image processing system 100 for eyeglass printing according to an embodiment of the present application. As shown in fig. 6, an image processing system 100 for eyeglass printing according to an embodiment of the present application includes: an image acquisition module 110, configured to acquire a face image acquired by a camera; a codec module 120, configured to pass the face image through a face region of interest generator based on an encoder-decoder structure to obtain a face region of interest image, where the encoder and the decoder have a symmetrical network structure; the key point detection module 130 is configured to perform key point detection on the image of the region of interest of the face to obtain a plurality of key feature points of the face; the image transformation module 140 is configured to calculate, based on the plurality of face key feature points, a rotation angle and a scaling of the face region-of-interest image, and transform the face region-of-interest image based on the rotation angle and the scaling to obtain a transformed face region-of-interest image; the 3DMM module 150 is configured to pass the transformed face region of interest image through a 3DMM model to obtain a face 3D model; the parameter calculation module 160 is configured to calculate parameters of the glasses based on the face 3D model; and a printing module 170, configured to perform eyeglass 3D model printing based on parameters of the eyeglasses.
In one example, as shown in fig. 7, in the image processing system 100 for glasses printing described above, the codec module 120 includes: an encoding unit 121, configured to input the face image into the encoder to obtain first to fifth face feature maps, where the encoder includes five convolution layers; a first decoding unit 122, configured to input the fifth facial feature map into a first decoding module of the decoder to obtain a first decoded feature map; a fusion unit 123, configured to fuse the first decoding feature map and the fifth face feature map to obtain a first fused decoding feature map; and a second decoding unit 124, configured to perform feature distribution optimization on the first fused decoding feature map to obtain an optimized first fused decoding feature map as an input of a second decoding module of the decoder.
In one example, in the image processing system 100 for eyeglass printing described above, the encoding unit 121 is configured to: inputting the face image into a first convolution layer of the encoder to obtain the first face feature map; inputting the first face feature map into a second convolution layer of the encoder to obtain the second face feature map; inputting the second face feature map into a third convolution layer of the encoder to obtain the third face feature map; inputting the third face feature map into a fourth convolution layer of the encoder to obtain the fourth face feature map; and inputting the fourth face feature map into a fifth convolution layer of the encoder to obtain the fifth face feature map.
In one example, in the above image processing system 100 for glasses printing, inputting the face image into the first convolution layer of the encoder to obtain the first face feature map includes: and respectively carrying out two-dimensional convolution processing, feature matrix-based mean value pooling processing and nonlinear activation processing on input data by using a first convolution layer of the encoder to output the first face feature map by the first convolution layer of the encoder, wherein the input of the first convolution layer of the encoder is the face image.
In one example, in the image processing system 100 for eyeglass printing described above, the first decoding unit 122 is configured to: and performing deconvolution, pooling and nonlinear activation on the fifth face feature map by using a first decoding module of the decoder to obtain the first decoding feature map.
In one example, in the image processing system 100 for eyeglass printing described above, the fusing unit 123 is configured to: fusing the first decoding feature map and the fifth face feature map with the following cascade formula to obtain the first fused decoding feature map; wherein, the cascade formula is:
Wherein, the liquid crystal display device comprises a liquid crystal display device,representing said first decoding profile, < >>Representing the fifth facial feature map, < >>Representing a cascade function->And representing the first fusion decoding characteristic diagram.
In one example, in the image processing system 100 for eyeglass printing described above, the second decoding unit 124 is configured to: performing feature distribution optimization on the first fusion decoding feature map by using the following optimization formula to obtain the optimized first fusion decoding feature map; wherein, the optimization formula is:
wherein, the liquid crystal display device comprises a liquid crystal display device,representing said first fused decoding profile, < >>Representing the optimized first fusion decoding characteristic diagram,representing a single layer convolution operation,/->Position-by-position addition of the representation feature map, +.>Position-wise subtraction of the representation characteristic map, +.>Position-wise multiplication of the characteristic map, and +.>And->Is a bias profile.
Here, it will be understood by those skilled in the art that the specific functions and operations of the respective modules in the above-described image processing system 100 for glasses printing have been described in detail in the above description of the image processing method for glasses printing with reference to fig. 1 to 5, and thus, repetitive descriptions thereof will be omitted.
As described above, the image processing system 100 for glasses printing according to the embodiment of the present application may be implemented in various wireless terminals, for example, a server or the like having an image processing algorithm for glasses printing. In one example, the image processing system 100 for eyeglass printing according to embodiments of the present application may be integrated into a wireless terminal as one software module and/or hardware module. For example, the image processing system 100 for eyeglass printing may be a software module in the operating system of the wireless terminal, or may be an application developed for the wireless terminal; of course, the image processing system 100 for eyeglass printing may also be one of a number of hardware modules of the wireless terminal.
Alternatively, in another example, the image processing system 100 for glasses printing and the wireless terminal may be separate devices, and the image processing system 100 for glasses printing may be connected to the wireless terminal through a wired and/or wireless network and transmit interactive information in a contracted data format.
According to another aspect of the present application, there is also provided a non-volatile computer-readable storage medium having stored thereon computer-readable instructions which, when executed by a computer, can perform a method as described above.
Program portions of the technology may be considered to be "products" or "articles of manufacture" in the form of executable code and/or associated data, embodied or carried out by a computer readable medium. A tangible, persistent storage medium may include any memory or storage used by a computer, processor, or similar device or related module. Such as various semiconductor memories, tape drives, disk drives, or the like, capable of providing storage functionality for software.
All or a portion of the software may sometimes communicate over a network, such as the internet or other communication network. Such communication may load software from one computer device or processor to another. For example: a hardware platform loaded from a server or host computer of the video object detection device to a computer environment, or other computer environment implementing the system, or similar functioning system related to providing information needed for object detection. Thus, another medium capable of carrying software elements may also be used as a physical connection between local devices, such as optical, electrical, electromagnetic, etc., propagating through cable, optical cable, air, etc. Physical media used for carrier waves, such as electrical, wireless, or optical, may also be considered to be software-bearing media. Unless limited to a tangible "storage" medium, other terms used herein to refer to a computer or machine "readable medium" mean any medium that participates in the execution of any instructions by a processor.
This application uses specific words to describe embodiments of the application. Reference to "a first/second embodiment," "an embodiment," and/or "some embodiments" means that a particular feature, structure, or characteristic is associated with at least one embodiment of the present application. Thus, it should be emphasized and should be appreciated that two or more references to "an embodiment" or "one embodiment" or "an alternative embodiment" in various positions in this specification are not necessarily referring to the same embodiment. Furthermore, certain features, structures, or characteristics of one or more embodiments of the present application may be combined as suitable.
Furthermore, those skilled in the art will appreciate that the various aspects of the invention are illustrated and described in the context of a number of patentable categories or circumstances, including any novel and useful procedures, machines, products, or materials, or any novel and useful modifications thereof. Accordingly, aspects of the present application may be performed entirely by hardware, entirely by software (including firmware, resident software, micro-code, etc.) or by a combination of hardware and software. The above hardware or software may be referred to as a "data block," module, "" engine, "" unit, "" component, "or" system. Furthermore, aspects of the present application may take the form of a computer product, comprising computer-readable program code, embodied in one or more computer-readable media.
Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.
The foregoing is illustrative of the present invention and is not to be construed as limiting thereof. Although a few exemplary embodiments of this invention have been described, those skilled in the art will readily appreciate that many modifications are possible in the exemplary embodiments without materially departing from the novel teachings and advantages of this invention. Accordingly, all such modifications are intended to be included within the scope of this invention as defined in the following claims. It is to be understood that the foregoing is illustrative of the present invention and is not to be construed as limited to the specific embodiments disclosed, and that modifications to the disclosed embodiments, as well as other embodiments, are intended to be included within the scope of the appended claims. The invention is defined by the claims and their equivalents.
Claims (7)
1. An image processing method for eyeglass printing, comprising:
acquiring a face image acquired by a camera;
passing the face image through a face region of interest generator based on an encoder-decoder structure to obtain a face region of interest image, wherein the encoder and the decoder have symmetrical network structures;
performing key point detection on the image of the region of interest of the human face to obtain a plurality of key characteristic points of the human face;
calculating a rotation angle and a scaling of the face region-of-interest image based on the plurality of face key feature points, and transforming the face region-of-interest image based on the rotation angle and the scaling to obtain a transformed face region-of-interest image;
the transformed face region of interest image is passed through a 3DMM model to obtain a face 3D model;
based on the face 3D model, calculating parameters of the glasses; and
printing a 3D model of the glasses based on parameters of the glasses;
passing the face image through a face region of interest generator based on an encoder-decoder structure to obtain a face region of interest image, wherein the encoder and the decoder have symmetrical network structures, comprising:
Inputting the face image into the encoder to obtain first to fifth face feature maps, wherein the encoder comprises five convolution layers;
inputting the fifth facial feature map into a first decoding module of the decoder to obtain a first decoding feature map;
fusing the first decoding feature map and the fifth face feature map to obtain a first fused decoding feature map; and
performing feature distribution optimization on the first fusion decoding feature map to obtain an optimized first fusion decoding feature map as input of a second decoding module of the decoder;
performing feature distribution optimization on the first fusion decoding feature map to obtain an optimized first fusion decoding feature map as an input of a second decoding module of the decoder, including:
performing feature distribution optimization on the first fusion decoding feature map by using the following optimization formula to obtain the optimized first fusion decoding feature map;
wherein, the optimization formula is:
F b ＝Cov(F a )；
wherein F represents the first fusion decoding feature map, F' represents the optimized first fusion decoding feature map, cov represents a single-layer convolution operation,position-by-position addition of the representation feature map, +.>Represents a position-wise subtraction of the feature map, +., And B is 1 And B 2 Is a bias profile.
2. The image processing method for eyeglass printing according to claim 1, wherein the face image is input to the encoder to obtain first to fifth face feature maps, wherein the encoder includes five convolution layers, comprising:
inputting the face image into a first convolution layer of the encoder to obtain the first face feature map;
inputting the first face feature map into a second convolution layer of the encoder to obtain the second face feature map;
inputting the second face feature map into a third convolution layer of the encoder to obtain the third face feature map;
inputting the third face feature map into a fourth convolution layer of the encoder to obtain the fourth face feature map; and
and inputting the fourth face feature map into a fifth convolution layer of the encoder to obtain the fifth face feature map.
3. The image processing method for eyeglass printing according to claim 2, wherein inputting the face image into a first convolution layer of the encoder to obtain the first face feature map comprises:
and respectively carrying out two-dimensional convolution processing, feature matrix-based mean value pooling processing and nonlinear activation processing on input data by using a first convolution layer of the encoder to output the first face feature map by the first convolution layer of the encoder, wherein the input of the first convolution layer of the encoder is the face image.
4. The image processing method for eyeglass printing according to claim 3, wherein inputting the fifth face feature map into a first decoding module of the decoder to obtain a first decoded feature map comprises:
and performing deconvolution, pooling and nonlinear activation on the fifth face feature map by using a first decoding module of the decoder to obtain the first decoding feature map.
5. The image processing method for eyeglass printing according to claim 4, wherein fusing the first decoding feature map and the fifth face feature map to obtain a first fused decoding feature map, comprising:
fusing the first decoding feature map and the fifth face feature map with the following cascade formula to obtain the first fused decoding feature map;
wherein, the cascade formula is:
F＝Concat[F 1 ，F 2 ]；
wherein F is 1 Representing the first decoding feature map, F 2 Representing the fifth face feature map, concat [ · ]]And representing a cascading function, wherein F represents the first fusion decoding characteristic diagram.
6. An image processing system for eyeglass printing, comprising:
the image acquisition module is used for acquiring the face image acquired by the camera;
A coding and decoding module, configured to pass the face image through a face region of interest generator based on an encoder-decoder structure to obtain a face region of interest image, where the encoder and the decoder have a symmetrical network structure;
the key point detection module is used for carrying out key point detection on the image of the region of interest of the human face so as to obtain a plurality of key feature points of the human face;
the image transformation module is used for calculating the rotation angle and the scaling of the face region-of-interest image based on the plurality of face key feature points, and transforming the face region-of-interest image based on the rotation angle and the scaling to obtain a transformed face region-of-interest image;
the 3DMM module is used for enabling the transformed face region-of-interest image to pass through a 3DMM model to obtain a face 3D model;
the parameter calculation module is used for calculating parameters of the glasses based on the face 3D model; and
the printing module is used for printing the 3D model of the glasses based on the parameters of the glasses;
the codec module includes: the encoding unit is used for inputting the face image into the encoder to obtain first to fifth face feature images, wherein the encoder comprises five convolution layers;
The first decoding unit is used for inputting the fifth face feature map into a first decoding module of the decoder to obtain a first decoding feature map;
the fusion unit is used for fusing the first decoding feature map and the fifth face feature map to obtain a first fusion decoding feature map; and
the second decoding unit is used for optimizing the characteristic distribution of the first fusion decoding characteristic diagram to obtain an optimized first fusion decoding characteristic diagram which is used as the input of a second decoding module of the decoder;
performing feature distribution optimization on the first fusion decoding feature map by using the following optimization formula to obtain the optimized first fusion decoding feature map;
wherein, the optimization formula is:
F b ＝Cov(F a )；
wherein F represents the first fusion decoding feature map, F' represents the optimized first fusion decoding feature map, cov represents a single-layer convolution operation,position-by-position addition of the representation feature map, +.>Represents a position-wise subtraction of the feature map, +. 1 And B 2 Is a bias profile.
7. The image processing system for eyeglass printing according to claim 6, wherein the encoding unit is configured to:
inputting the face image into a first convolution layer of the encoder to obtain the first face feature map;
Inputting the first face feature map into a second convolution layer of the encoder to obtain the second face feature map;
inputting the second face feature map into a third convolution layer of the encoder to obtain the third face feature map;
inputting the third face feature map into a fourth convolution layer of the encoder to obtain the fourth face feature map; and
and inputting the fourth face feature map into a fifth convolution layer of the encoder to obtain the fifth face feature map.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310505728.9A CN116228763B (en) | 2023-05-08 | 2023-05-08 | Image processing method and system for eyeglass printing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310505728.9A CN116228763B (en) | 2023-05-08 | 2023-05-08 | Image processing method and system for eyeglass printing |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116228763A CN116228763A (en) | 2023-06-06 |
| CN116228763B true CN116228763B (en) | 2023-07-21 |
Family
ID=86584582
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310505728.9A Active CN116228763B (en) | 2023-05-08 | 2023-05-08 | Image processing method and system for eyeglass printing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116228763B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112614213A (en) * | 2020-12-14 | 2021-04-06 | 杭州网易云音乐科技有限公司 | Facial expression determination method, expression parameter determination model, medium and device |
| WO2021164429A1 (en) * | 2020-02-21 | 2021-08-26 | 京东方科技集团股份有限公司 | Image processing method, image processing apparatus, and device |
Family Cites Families (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108154386A (en) * | 2017-12-08 | 2018-06-12 | 上海翰达眼镜销售有限公司 | A kind of method for choosing and customizing glasses |
| CN108490641B (en) * | 2018-02-14 | 2019-04-26 | 天目爱视（北京）科技有限公司 | Glasses automated design system based on 3D header data |
| CN108537126B (en) * | 2018-03-13 | 2021-03-23 | 东北大学 | Face image processing method |
| CN108876793A (en) * | 2018-04-13 | 2018-11-23 | 北京迈格威科技有限公司 | Semantic segmentation methods, devices and systems and storage medium |
| US11308611B2 (en) * | 2019-10-09 | 2022-04-19 | Siemens Healthcare Gmbh | Reducing false positive detections of malignant lesions using multi-parametric magnetic resonance imaging |
| CN111062328B (en) * | 2019-12-18 | 2023-10-03 | 中新智擎科技有限公司 | Image processing method and device and intelligent robot |
| US11574390B2 (en) * | 2020-01-20 | 2023-02-07 | Canon Kabushiki Kaisha | Apparatus and method for image processing |
| CN112883843B (en) * | 2021-02-02 | 2022-06-03 | 清华大学 | Driver visual salient region detection method and device and computer equipment |
| CN113223154A (en) * | 2021-05-24 | 2021-08-06 | 深圳市摩客眼镜智能科技有限公司 | Personalized spectacle frame three-dimensional model generation method and personalized spectacle frame manufacturing method |
| CN113724302B (en) * | 2021-09-02 | 2024-03-29 | 扛造数字科技(成都)有限公司 | Personalized glasses customizing method and customizing system |
| CN115375835A (en) * | 2022-07-13 | 2022-11-22 | 平安科技（深圳）有限公司 | Three-dimensional model establishing method based on two-dimensional key points, computer and storage medium |
| CN115294301B (en) * | 2022-08-11 | 2023-05-16 | 广州沃佳科技有限公司 | Head model construction method, device, equipment and medium based on face image |
| CN115330947A (en) * | 2022-08-12 | 2022-11-11 | 百果园技术(新加坡)有限公司 | Three-dimensional face reconstruction method and device, equipment, medium and product thereof |
-
2023
- 2023-05-08 CN CN202310505728.9A patent/CN116228763B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2021164429A1 (en) * | 2020-02-21 | 2021-08-26 | 京东方科技集团股份有限公司 | Image processing method, image processing apparatus, and device |
| CN112614213A (en) * | 2020-12-14 | 2021-04-06 | 杭州网易云音乐科技有限公司 | Facial expression determination method, expression parameter determination model, medium and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116228763A (en) | 2023-06-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108510535B (en) | High-quality depth estimation method based on depth prediction and enhancer network | |
| CN110782490B (en) | Video depth map estimation method and device with space-time consistency | |
| CN110298798B (en) | Image restoration method based on low-rank tensor completion and discrete total variation | |
| CN110738697A (en) | Monocular depth estimation method based on deep learning | |
| Passieux et al. | High resolution digital image correlation using proper generalized decomposition: PGD‐DIC | |
| CN106447601B (en) | Unmanned aerial vehicle remote sensing image splicing method based on projection-similarity transformation | |
| CN111462120A (en) | Defect detection method, device, medium and equipment based on semantic segmentation model | |
| CN103994755B (en) | A kind of space non-cooperative object pose measuring method based on model | |
| CN110910437A (en) | Depth prediction method for complex indoor scene | |
| CN115908311B (en) | Lens forming detection equipment and method based on machine vision | |
| CN114429555A (en) | Image density matching method, system, equipment and storage medium from coarse to fine | |
| CN110689558A (en) | Multi-sensor image enhancement method and device | |
| CN115546273A (en) | Scene structure depth estimation method for indoor fisheye image | |
| CN114066959A (en) | Single-stripe image depth estimation method based on Transformer | |
| CN117132461B (en) | Method and system for whole-body optimization of character based on character deformation target body | |
| CN116228763B (en) | Image processing method and system for eyeglass printing | |
| CN109741389B (en) | Local stereo matching method based on region base matching | |
| CN110533663B (en) | Image parallax determining method, device, equipment and system | |
| CN113962904B (en) | Method for filtering and denoising hyperspectral image | |
| CN113628261B (en) | Infrared and visible light image registration method in electric power inspection scene | |
| CN115423927A (en) | ViT-based multi-view 3D reconstruction method and system | |
| CN115035164A (en) | Moving target identification method and device | |
| US11954867B2 (en) | Motion vector generation apparatus, projection image generation apparatus, motion vector generation method, and program | |
| CN112199815A (en) | Method for reducing influence of temperature on camera internal parameters | |
| CN114638748A (en) | Image processing method, image restoration method, computer device, and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |