CN116228763A

CN116228763A - Image processing method and system for eyeglass printing

Info

Publication number: CN116228763A
Application number: CN202310505728.9A
Authority: CN
Inventors: 邓清凤; 伍强; 黄渠洪; 黄剑
Original assignee: Chengdu Ruitong Technology Co ltd
Current assignee: Chengdu Ruitong Technology Co ltd
Priority date: 2023-05-08
Filing date: 2023-05-08
Publication date: 2023-06-06
Anticipated expiration: 2043-05-08
Also published as: CN116228763B

Abstract

An image processing method and system for eyeglass printing are disclosed. Firstly, acquiring a face image acquired by a camera, then, passing the face image through a face region of interest generator to obtain a face region of interest image, then, performing key point detection on the face region of interest image to obtain a plurality of face key feature points, then, calculating the rotation angle and the scaling of the face region of interest image based on the face key feature points, transforming the face region of interest image based on the rotation angle and the scaling to obtain a transformed face region of interest image, then, passing the transformed face region of interest image through a 3DMM model to obtain a face 3D model, and finally, calculating parameters of glasses based on the face 3D model and performing glasses 3D model printing based on the parameters of the glasses. Thus, the production efficiency can be improved, and the production cost can be reduced.

Description

Image processing method and system for eyeglass printing

Technical Field

The present application relates to the field of intelligent printing, and more particularly, to an image processing method and system for eyeglass printing.

Background

With the development of computer technology for processing, analyzing and understanding images and the increasing maturity of 3D printing technology, image recognition technology and 3D printing technology have been perfectly combined gradually and widely applied to various production and manufacturing industries.

For example, chinese patent application No. 202011404717.4 discloses an image processing method for 3D printing, comprising the steps of: projecting the layer to be printed to generate an initial image; moving the initial image along a first direction, so that the initial image moves a first distance from an initial position to a first limiting position to obtain a first image, wherein the first distance is not greater than the length of one pixel in the initial image along the first direction; fusing the initial image and the first image to obtain a fused image; and printing the fused image. The image processing system for 3D printing comprises a projection module, a mobile module, a fusion module and a printing module. By the method and the system, the gray value of the adjacent pixels in the image can be changed more gradually, the printing precision is improved, and a 3D printing model with a smoother surface can be obtained under the condition that the resolution of the existing optical machine is not changed.

Furthermore, as disclosed in the chinese patent application No. 202210621905.5, an image processing method and apparatus, a storage medium, and a terminal in 3D printing relate to the technical field of 3D printing, and mainly aim at solving the problem of poor slice layer image rendering performance in the existing 3D printing. Comprising the following steps: acquiring contour line information of a target printing model slice layer and calculating contour line slope of each contour line according to the contour line information; determining edge pixel blocks covered by all the contour lines according to the slope of the contour lines, and calculating gray values of the edge pixel blocks; and rendering the corresponding edge pixel block based on the gray value of the edge pixel block, so that image rendering in 3D printing is improved.

With the development of market demand, image processing and 3D printing technologies are also beginning to be used gradually in the production of customized glasses. Eyeglass printing of 3D printing technology is an emerging eyeglass manufacturing approach based on the recognition and processing of user facial images, and then using a 3D printer to manufacture eyeglass frames and eyeglass accessories. This technique allows manufacturers to produce fully customized eyewear according to the individual needs of the customer.

The key technology of glasses printing based on 3D printing technology is: the face is accurately 3D modeled, and customized eyeglass parameters are calculated based on the obtained face 3D model, so that 3D printing can be performed based on the eyeglass parameters.

Some schemes for carrying out face 3D modeling based on face two-dimensional images exist, but the modeling precision of the modeling schemes also has a large optimization space, so that full intellectualization and full automation cannot be truly realized due to eyeglass printing based on a 3D printing technology. In the existing glasses printing scheme based on the 3D printing technology, related parameters are also required to be manually measured to carry out auxiliary judgment so as to ensure that the finally formed printing glasses can meet the requirement of customization.

Thus, an optimized image processing scheme for eyeglass printing is desired.

Disclosure of Invention

The present application has been made in order to solve the above technical problems. The embodiment of the application provides an image processing method and system for eyeglass printing. Firstly, acquiring a face image acquired by a camera, then, passing the face image through a face region of interest generator to obtain a face region of interest image, then, performing key point detection on the face region of interest image to obtain a plurality of face key feature points, then, calculating the rotation angle and the scaling of the face region of interest image based on the face key feature points, transforming the face region of interest image based on the rotation angle and the scaling to obtain a transformed face region of interest image, then, passing the transformed face region of interest image through a 3DMM model to obtain a face 3D model, and finally, calculating parameters of glasses based on the face 3D model and performing glasses 3D model printing based on the parameters of the glasses. Thus, the production efficiency can be improved, and the production cost can be reduced.

According to an aspect of the present application, there is provided an image processing method for eyeglass printing, including:

acquiring a face image acquired by a camera;

passing the face image through a face region of interest generator based on an encoder-decoder structure to obtain a face region of interest image, wherein the encoder and the decoder have symmetrical network structures;

performing key point detection on the image of the region of interest of the human face to obtain a plurality of key characteristic points of the human face;

calculating a rotation angle and a scaling of the face region-of-interest image based on the plurality of face key feature points, and transforming the face region-of-interest image based on the rotation angle and the scaling to obtain a transformed face region-of-interest image;

the transformed face region of interest image is passed through a 3DMM model to obtain a face 3D model;

based on the face 3D model, calculating parameters of the glasses; and

and printing the 3D model of the glasses based on the parameters of the glasses.

In the above image processing method for eyeglass printing, the face image is passed through a face region of interest generator based on an encoder-decoder structure to obtain a face region of interest image, wherein the encoder and the decoder have a symmetrical network structure, and the method comprises:

Inputting the face image into the encoder to obtain first to fifth face feature maps, wherein the encoder comprises five convolution layers;

inputting the fifth facial feature map into a first decoding module of the decoder to obtain a first decoding feature map;

fusing the first decoding feature map and the fifth face feature map to obtain a first fused decoding feature map; and

and performing feature distribution optimization on the first fusion decoding feature map to obtain an optimized first fusion decoding feature map as input of a second decoding module of the decoder.

In the above image processing method for eyeglass printing, the face image is input to the encoder to obtain first to fifth face feature maps, wherein the encoder includes five convolution layers, including:

inputting the face image into a first convolution layer of the encoder to obtain the first face feature map;

inputting the first face feature map into a second convolution layer of the encoder to obtain the second face feature map;

inputting the second face feature map into a third convolution layer of the encoder to obtain the third face feature map;

Inputting the third face feature map into a fourth convolution layer of the encoder to obtain the fourth face feature map; and

and inputting the fourth face feature map into a fifth convolution layer of the encoder to obtain the fifth face feature map.

In the above image processing method for eyeglass printing, inputting the face image into a first convolution layer of the encoder to obtain the first face feature map includes:

and respectively carrying out two-dimensional convolution processing, feature matrix-based mean value pooling processing and nonlinear activation processing on input data by using a first convolution layer of the encoder to output the first face feature map by the first convolution layer of the encoder, wherein the input of the first convolution layer of the encoder is the face image.

In the above image processing method for eyeglass printing, inputting the fifth face feature map into a first decoding module of the decoder to obtain a first decoded feature map, including:

and performing deconvolution, pooling and nonlinear activation on the fifth face feature map by using a first decoding module of the decoder to obtain the first decoding feature map.

In the above image processing method for eyeglass printing, fusing the first decoding feature map and the fifth face feature map to obtain a first fused decoding feature map includes:

fusing the first decoding feature map and the fifth face feature map with the following cascade formula to obtain the first fused decoding feature map;

wherein, the cascade formula is:

；

wherein ,

representing said first decoding profile, < >>

Representing the fifth facial feature map, < >>

Representing a cascade function->

And representing the first fusion decoding characteristic diagram.

In the above image processing method for eyeglass printing, performing feature distribution optimization on the first fused decoding feature map to obtain an optimized first fused decoding feature map as an input of a second decoding module of the decoder, including:

performing feature distribution optimization on the first fusion decoding feature map by using the following optimization formula to obtain the optimized first fusion decoding feature map;

wherein, the optimization formula is:

wherein ,

representing said first fused decoding profile, < >>

Representing the optimized first fusion decoding characteristic diagram,

representing a single layer convolution operation,/->

Position-by-position addition of the representation feature map, +. >

Position-wise subtraction of the representation characteristic map, +.>

Position-wise multiplication of the characteristic map, and +.>

and />

Is a bias profile.

According to another aspect of the present application, there is provided an image processing system for eyeglass printing, comprising:

the image acquisition module is used for acquiring the face image acquired by the camera;

a coding and decoding module, configured to pass the face image through a face region of interest generator based on an encoder-decoder structure to obtain a face region of interest image, where the encoder and the decoder have a symmetrical network structure;

the key point detection module is used for carrying out key point detection on the image of the region of interest of the human face so as to obtain a plurality of key feature points of the human face;

the image transformation module is used for calculating the rotation angle and the scaling of the face region-of-interest image based on the plurality of face key feature points, and transforming the face region-of-interest image based on the rotation angle and the scaling to obtain a transformed face region-of-interest image;

the 3DMM module is used for enabling the transformed face region-of-interest image to pass through a 3DMM model to obtain a face 3D model;

The parameter calculation module is used for calculating parameters of the glasses based on the face 3D model; and

and the printing module is used for printing the 3D model of the glasses based on the parameters of the glasses.

In the above image processing system for eyeglass printing, the codec module includes:

the encoding unit is used for inputting the face image into the encoder to obtain first to fifth face feature images, wherein the encoder comprises five convolution layers;

the first decoding unit is used for inputting the fifth face feature map into a first decoding module of the decoder to obtain a first decoding feature map;

the fusion unit is used for fusing the first decoding feature map and the fifth face feature map to obtain a first fusion decoding feature map; and

and the second decoding unit is used for optimizing the characteristic distribution of the first fusion decoding characteristic diagram to obtain an optimized first fusion decoding characteristic diagram which is used as the input of a second decoding module of the decoder.

In the above image processing system for eyeglass printing, the encoding unit is configured to:

Compared with the prior art, the image processing method and the system for printing the glasses are characterized in that firstly, a face image acquired by a camera is acquired, then the face image is processed through a face region of interest generator to obtain a face region of interest image, then key point detection is conducted on the face region of interest image to obtain a plurality of face key feature points, then the rotation angle and the scaling of the face region of interest image are calculated and obtained based on the face key feature points, the face region of interest image is transformed based on the rotation angle and the scaling to obtain a transformed face region of interest image, then the transformed face region of interest image is processed through a 3DMM model to obtain a face 3D model, finally, parameters of the glasses are calculated and the glasses 3D model is printed based on the parameters of the glasses. Thus, the production efficiency can be improved, and the production cost can be reduced.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the description of the embodiments will be briefly introduced below, it being obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art. The following drawings are not intended to be drawn to scale, with emphasis instead being placed upon illustrating the principles of the present application.

Fig. 1 is an application scenario diagram of an image processing method for eyeglass printing according to an embodiment of the present application.

Fig. 2 is a flowchart of an image processing method for eyeglass printing according to an embodiment of the present application.

Fig. 3 is a schematic architecture diagram of an image processing method for eyeglass printing according to an embodiment of the present application.

Fig. 4 is a flowchart of substep S120 of the image processing method for eyeglass printing according to the embodiment of the present application.

Fig. 5 is a flowchart of sub-step S121 of the image processing method for eyeglass printing according to the embodiment of the present application.

Fig. 6 is a block diagram of an image processing system for eyeglass printing according to an embodiment of the present application.

Fig. 7 is a block diagram of the codec module in the image processing system for glasses printing according to the embodiment of the present application.

Detailed Description

The following description of the embodiments of the present application will be made clearly and fully with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some, but not all embodiments of the present application. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments of the present application without making any inventive effort, are also within the scope of the present application.

As used in this application and in the claims, the terms "a," "an," "the," and/or "the" are not specific to the singular, but may include the plural, unless the context clearly dictates otherwise. In general, the terms "comprises" and "comprising" merely indicate that the steps and elements are explicitly identified, and they do not constitute an exclusive list, as other steps or elements may be included in a method or apparatus.

Although the present application makes various references to certain modules in a system according to embodiments of the present application, any number of different modules may be used and run on a user terminal and/or server. The modules are merely illustrative, and different aspects of the systems and methods may use different modules.

Flowcharts are used in this application to describe the operations performed by systems according to embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed in order precisely. Rather, the various steps may be processed in reverse order or simultaneously, as desired. Also, other operations may be added to or removed from these processes.

Hereinafter, example embodiments according to the present application will be described in detail with reference to the accompanying drawings. It should be apparent that the described embodiments are only some of the embodiments of the present application and not all of the embodiments of the present application, and it should be understood that the present application is not limited by the example embodiments described herein.

Aiming at the technical problems, the technical conception of the application is as follows: the method comprises the steps of detecting a human face region of interest through a human face region of interest generator based on an encoder-decoder, improving consistency between the human face region of interest and a real human face through key point detection, rotation angle calculation, scaling and other operations, obtaining a high-quality human face 3D model through a 3DMM model, and further achieving calculation of eyeglass parameters and printing of the eyeglass 3D model. Therefore, the accuracy of face modeling is greatly improved through the data processing flow, meanwhile, the requirements of manual intervention are eliminated through operations such as key point detection and rotation angle calculation, full automation and full intellectualization are realized, and completely customized glasses can be produced according to individual requirements of customers, so that better use experience is provided. Meanwhile, the requirements of eyeglass manufacturers on efficiency and cost can be met, the production efficiency is improved, and the production cost is reduced.

Specifically, in the technical scheme of the application, a face image acquired by a camera is firstly acquired. It should be understood that in the technical solution of the present application, the technical route is to perform face 3D modeling based on a face two-dimensional image, so that a face two-dimensional image of a customized object needs to be acquired first. It should be understood that after the face image is acquired, a computer graphics technology may be used to convert the image into a three-dimensional face model, and further calculate parameters of the glasses according to the obtained three-dimensional face model, so as to print the 3D model of the glasses.

It is worth mentioning that, in the technical scheme of this application, through the camera gathers the production flow that the people's face image can realize the full automatization. Specifically, the camera can be connected with the 3D printer, so that a series of automatic processes from the collection of face images to the printing of glasses can be realized. The method can greatly improve the production efficiency, reduce the manufacturing cost and meet the personalized customization requirement of customers.

The face image is then passed through a face region of interest generator based on an encoder-decoder structure to obtain a face region of interest image, wherein the encoder and the decoder have a symmetrical network structure. Here, the technical purpose of passing the face image through a face region of interest generator based on an encoder-decoder structure to obtain a face region of interest image is to improve the accuracy and precision of the 3D modeling of the face.

Those of ordinary skill in the art will appreciate that in conventional face modeling methods, it is often necessary to manually mark or select a region of interest of a face and then perform the related operations. This approach, while having a certain degree of accuracy, requires significant labor and time costs. The human face region of interest generator based on the encoder-decoder structure can automatically locate and extract the human face region of interest, so that full-automatic modeling is realized.

Specifically, in the technical scheme of the application, a symmetrical encoder-decoder network structure is adopted to firstly encode the face image to obtain a face encoding feature map, and the face encoder feature map is restored to a high-resolution face image by using the decoder. Further, image semantic segmentation is carried out on the face image with high resolution to obtain the face region-of-interest image.

In a specific example of the present application, the encoder includes five connected convolution layers, which are defined as first to fifth convolution layers, and after the face image is input into the encoder, the encoder can perform stepwise deep convolution encoding on the face image through the five connected convolution layers to obtain first to fifth face feature maps. Then, based on the layer jump connection between the encoder and the decoder, the fifth face feature map is input into the decoder for decoding to obtain the high-resolution face image.

In a specific example of the present application, the decoder includes five deconvolution layers for deconvolution encoding input data to obtain a decoding profile. The first decoding module of the decoder is described here as an example: a layer jump connection between the encoder and the decoder, and a decoding process of the first decoding module.

Firstly, inputting the fifth face feature map into a first decoding module of the decoder to obtain a first decoding feature map, wherein the first decoding module performs convolution coding, pooling processing and nonlinear activation processing on the fifth face feature map to obtain the first decoding feature map; next, the first decoding feature map and the fifth face feature map are fused to obtain a first fused decoding feature map, that is, the output of the fifth convolution layer of the encoder is transferred to the first decoding module of the decoder and is subjected to feature fusion with the output of the first decoding module (that is, the first decoding feature map).

Here, considering that the first fused decoding feature map is obtained by fusing the first decoding feature map and the fifth face feature map, it contains both the image semantic decoding feature of the first decoding feature map and the image semantic encoding feature of the fifth face feature map, and the fifth face feature map has a higher image semantic encoding abstraction degree, whereas the first decoding feature map has a lower image decoding abstraction degree with respect to the fifth face feature map, and therefore has feature redundancy about the face image on different feature encoding and decoding abstraction scales, which affects the expression effect of the first fused decoding feature map, thereby reducing the accuracy of the decoding result.

Thus, the applicant of the present application decodes the first fused decoding profile, e.g. denoted as

Feature redundancy optimization based on low-cost bottleneck-mechanism stacking is performed to obtain an optimized first fusion decoding feature graph, for example, the feature redundancy optimization is recorded as

The method is specifically expressed as follows:

representing a single layer convolution operation,/->

、/>

and />

Respectively representing the position-by-position addition, subtraction and multiplication of the feature maps, and +.>

and />

For biasing the feature map, for example, a global mean feature map or a unit feature map of the first fused decoding feature map may be initially set, wherein the initial biasing feature map ∈ ->

and />

Different.

Here, the feature redundancy optimization based on the low-cost bottleneck mechanism stack may perform feature expansion using the low-cost bottleneck mechanism of the multiply-add stack of two low-cost transform features, and match the residual paths by biasing the stack channels with uniform values, so as to reveal hidden distribution information under intrinsic features in the redundancy features through low-cost operation transformation similar to the basic residual modules, so as to obtain more intrinsic expression of features through simple and effective convolution operation architecture, thereby optimizing redundant feature expression of the first fusion decoding feature map, improving expression effect of the first fusion decoding feature map, and improving accuracy of decoding result of the second decoding module.

That is, in the technical solution of the present application, after the first fused decoding feature map is obtained, feature distribution optimization is performed on the first fused decoding feature map to obtain an optimized first fused decoding feature map as an input of a second decoding module of the decoder.

It should be understood that although only the first decoding module of the decoder is illustrated as an example: the layer jump connection between the encoder and the decoder, and the decoding process of the first decoding module should be easily understood based on the same principle, and the decoding processes of the second to fifth decoding modules are not expanded again in order to avoid redundancy.

And after the human face region-of-interest image is obtained, carrying out key point detection on the human face region-of-interest image to obtain a plurality of human face key feature points. It should be understood that the key point detection of the image of the region of interest of the face to obtain a plurality of key feature points of the face is one of important steps of modeling the face in the eyeglass printing based on the 3D printing technology, which can determine the shape and position of the face more accurately, thereby improving the precision and accuracy of the subsequent calculation.

By means of a key point detection algorithm, key points (such as eyes, nose, mouth and the like) at specific positions in the image of the region of interest of the face can be automatically identified. These key points may provide detailed information about the face, such as facial contours, expressions, etc. It should be appreciated that in eyeglass printing based on 3D printing technology, the key point detection algorithm can help us obtain the exact position and angle of the eyeglass on the face, so as to better calculate eyeglass parameters. By acquiring the key feature points, the shape and the position of the face can be more accurately represented and described, and a foundation is laid for subsequent transformation and 3D modeling.

And then, based on the plurality of face key feature points, calculating a rotation angle and a scaling of the face region-of-interest image, and based on the rotation angle and the scaling, transforming the face region-of-interest image to obtain a transformed face region-of-interest image. In eyeglass printing based on 3D printing technology, since the face sizes and shapes of different people are different, the acquired face images need to be appropriately transformed for better analysis and modeling. Specifically, the rotation angle and the scaling of the face region-of-interest image are obtained through calculation, and the face region-of-interest image can be converted into a standard size and direction, so that better support is provided for subsequent 3D modeling and other operations. Then, by transforming the image of the region of interest of the face, the influence caused by deformation and posture can be effectively eliminated, so that the sizes and the shapes of all faces can be uniformly represented and processed. Therefore, more accurate and fine data can be provided for subsequent face 3D modeling, and better eyeglass customization and individuation effects are achieved.

And then, the transformed face region-of-interest image is passed through a 3DMM model to obtain a face 3D model. In eyeglass printing based on the 3D printing technology, the eyeglass to be customized needs to accurately establish a 3D model of a human face. By passing the transformed face region of interest image through a 3DMM (3D Morphable Model) model, the 2D face image can be converted into a 3D face model, and the shape, texture, etc. of the model can be finely controlled. The 3DMM model is a mathematical model obtained by carrying out statistical analysis by utilizing a large number of known face sample data, a large number of face 3D models with different postures, expressions and ages can be quickly and efficiently generated by using the 3DMM model, and the 3D models can be correspondingly adjusted and modified according to the sizes and the shapes of the glasses so as to adapt to the demands of different clients.

And then, calculating parameters of the glasses based on the face 3D model. In eyeglass printing based on 3D printing technology, eyeglass to be customized needs to calculate parameters of the eyeglass, such as contour, size, position, etc., of the eyeglass according to individual characteristics of customers. By calculating the parameters of the glasses based on the face 3D model, the design and manufacture of the glasses can be matched with the facial features of the customer, and the personalized requirements of the customer on the glasses can be met. Specifically, parameters of the glasses, including width, height, frame shape, etc., of the glasses are calculated by adopting a method of combining a face 3D model and a glasses 3D model. Through the calculated parameters, the 3D model of the glasses suitable for different customers can be generated, and further modification and adjustment can be carried out according to the requirements of the customers so as to meet the personalized requirements of the customers.

Specifically, based on the face 3D model, the process of calculating parameters of the glasses includes the following steps: step 1: positioning the eyeglass region: the position and the gesture of the glasses in the face 3D model can be positioned by matching the glasses model based on the face 3D model; step 2: calculating the size of the glasses: the size of the glasses can be calculated by measuring the width, the height and other data of the frames of the glasses; step 3: calculating the positions of the glasses: by determining the position and the posture of the glasses in the face 3D model, the distance, the inclination angle and other data of the glasses can be calculated, so that the position of the glasses can be accurately determined, and the following steps are carried out: calculating the shape of the glasses: by modeling the eyeglass rim, the shape and contour of the eyeglass can be calculated and matched to the facial features of the customer. Through the steps, the parameters of the glasses can be obtained based on the face 3D model, and the 3D models of the glasses suitable for different clients can be generated, so that the personalized requirements of the clients are met.

Finally, eyeglass 3D model printing is performed based on the eyeglass parameters. In this way, fully customized eyewear may be produced according to the customer's personalized needs, providing a better use experience. Meanwhile, the requirements of eyeglass manufacturers on efficiency and cost can be met, the production efficiency is improved, and the production cost is reduced.

Fig. 1 is an application scenario diagram of an image processing method for eyeglass printing according to an embodiment of the present application. As shown in fig. 1, in this application scenario, first, a face image (e.g., D illustrated in fig. 1) acquired by a camera (e.g., C illustrated in fig. 1) is acquired, then, the face image is input to a server (e.g., S illustrated in fig. 1) in which an image processing algorithm for glasses printing is deployed, wherein the server is capable of processing the face image using the image processing algorithm for glasses printing to obtain a face 3D model, then, based on the face 3D model, parameters of glasses are calculated, and glasses 3D model printing is performed based on the parameters of the glasses.

Having described the basic principles of the present application, various non-limiting embodiments of the present application will now be described in detail with reference to the accompanying drawings.

Fig. 2 is a flowchart of an image processing method for eyeglass printing according to an embodiment of the present application. As shown in fig. 2, the image processing method for eyeglass printing according to the embodiment of the present application includes the steps of: s110, acquiring a face image acquired by a camera; s120, the face image is passed through a face region of interest generator based on an encoder-decoder structure to obtain a face region of interest image, wherein the encoder and the decoder have symmetrical network structures; s130, detecting key points of the image of the region of interest of the human face to obtain a plurality of key feature points of the human face; s140, calculating a rotation angle and a scaling ratio of the face region-of-interest image based on the plurality of face key feature points, and transforming the face region-of-interest image based on the rotation angle and the scaling ratio to obtain a transformed face region-of-interest image; s150, the transformed face region-of-interest image is passed through a 3DMM model to obtain a face 3D model; s160, calculating parameters of the glasses based on the face 3D model; and S170, printing the 3D model of the glasses based on the parameters of the glasses.

Fig. 3 is a schematic architecture diagram of an image processing method for eyeglass printing according to an embodiment of the present application. As shown in fig. 3, in the network architecture, first, a face image acquired by a camera is acquired; then, the face image is passed through a face region of interest generator based on an encoder-decoder structure to obtain a face region of interest image, wherein the encoder and the decoder have a symmetrical network structure; then, carrying out key point detection on the image of the region of interest of the human face to obtain a plurality of key characteristic points of the human face; then, based on the plurality of face key feature points, calculating to obtain a rotation angle and a scaling of the face region-of-interest image, and based on the rotation angle and the scaling, transforming the face region-of-interest image to obtain a transformed face region-of-interest image; then, the transformed face region-of-interest image is passed through a 3DMM model to obtain a face 3D model; then, based on the face 3D model, calculating parameters of the glasses; finally, eyeglass 3D model printing is performed based on the eyeglass parameters.

More specifically, in step S110, a face image acquired by a camera is acquired. It should be understood that the technical solution of the present application performs face 3D modeling based on a face two-dimensional image, and therefore, first, a face two-dimensional image of a customized object needs to be acquired. It should be understood that after the face image is acquired, a computer graphics technology may be used to convert the image into a three-dimensional face model, and further calculate parameters of the glasses according to the obtained three-dimensional face model, so as to print the 3D model of the glasses.

More specifically, in step S120, the face image is passed through a face region of interest generator based on an encoder-decoder structure to obtain a face region of interest image, wherein the encoder and the decoder have a symmetrical network structure. Here, the technical purpose of passing the face image through a face region of interest generator based on an encoder-decoder structure to obtain a face region of interest image is to improve the accuracy and precision of the 3D modeling of the face. The human face region of interest generator based on the encoder-decoder structure can automatically locate and extract the human face region of interest, thereby realizing full-automatic modeling.

Here, the encoder includes five connected convolution layers, defined as first to fifth convolution layers, and after the face image is input to the encoder, the encoder can progressively and deeply convolutionally encode the face image through the five connected convolution layers to obtain first to fifth face feature maps. Then, based on the layer jump connection between the encoder and the decoder, the fifth face feature map is input into the decoder for decoding to obtain the high-resolution face image. The decoder comprises five deconvolution layers for deconvolution encoding the input data to obtain a decoding profile, respectively. The first decoder module of the decoder performs partial convolution encoding, pooling and nonlinear activation processing on the fifth face feature map to obtain the first decoding feature map; next, the first decoding feature map and the fifth face feature map are fused to obtain a first fused decoding feature map, that is, the output of the fifth convolution layer of the encoder is transferred to the first decoding module of the decoder and is subjected to feature fusion with the output of the first decoding module (that is, the first decoding feature map).

Accordingly, in one specific example, as shown in fig. 4, the face image is passed through a face region of interest generator based on an encoder-decoder structure to obtain a face region of interest image, wherein the encoder and the decoder have a symmetrical network structure, comprising: s121, inputting the face image into the encoder to obtain first to fifth face feature images, wherein the encoder comprises five convolution layers; s122, inputting the fifth facial feature map into a first decoding module of the decoder to obtain a first decoding feature map; s123, fusing the first decoding feature map and the fifth face feature map to obtain a first fused decoding feature map; and S124, performing feature distribution optimization on the first fusion decoding feature map to obtain an optimized first fusion decoding feature map as an input of a second decoding module of the decoder.

Accordingly, in one specific example, as shown in fig. 5, the face image is input to the encoder to obtain first to fifth face feature maps, wherein the encoder includes five convolution layers, including: s1211, inputting the face image into a first convolution layer of the encoder to obtain the first face feature map; s1212, inputting the first face feature map into a second convolution layer of the encoder to obtain the second face feature map; s1213, inputting the second face feature map into a third convolution layer of the encoder to obtain the third face feature map; s1214, inputting the third face feature map into a fourth convolution layer of the encoder to obtain the fourth face feature map; and S1215, inputting the fourth face feature map into a fifth convolution layer of the encoder to obtain the fifth face feature map.

Accordingly, in one specific example, inputting the face image into a first convolution layer of the encoder to obtain the first face feature map includes: and respectively carrying out two-dimensional convolution processing, feature matrix-based mean value pooling processing and nonlinear activation processing on input data by using a first convolution layer of the encoder to output the first face feature map by the first convolution layer of the encoder, wherein the input of the first convolution layer of the encoder is the face image.

Accordingly, in one specific example, inputting the fifth face feature map into a first decoding module of the decoder to obtain a first decoded feature map includes: and performing deconvolution, pooling and nonlinear activation on the fifth face feature map by using a first decoding module of the decoder to obtain the first decoding feature map.

Accordingly, in a specific example, fusing the first decoding feature map and the fifth face feature map to obtain a first fused decoding feature map includes: fusing the first decoding feature map and the fifth face feature map with the following cascade formula to obtain the first fused decoding feature map; wherein, the cascade formula is:

wherein ,

representing said first decoding profile, < >>

Representing the fifth facial feature map, < >>

Representing a cascade function->

And representing the first fusion decoding characteristic diagram.

Here, considering that the first fused decoding feature map is obtained by fusing the first decoding feature map and the fifth face feature map, it contains both the image semantic decoding feature of the first decoding feature map and the image semantic encoding feature of the fifth face feature map, and the fifth face feature map has a higher image semantic encoding abstraction degree, whereas the first decoding feature map has a lower image decoding abstraction degree with respect to the fifth face feature map, and therefore has feature redundancy about the face image on different feature encoding and decoding abstraction scales, which affects the expression effect of the first fused decoding feature map, thereby reducing the accuracy of the decoding result. Therefore, the applicant of the present application performs feature redundancy optimization on the first fusion decoding feature map based on the low-cost bottleneck mechanism stack, so as to obtain an optimized first fusion decoding feature map.

Accordingly, in a specific example, performing feature distribution optimization on the first fused decoding feature map to obtain an optimized first fused decoding feature map as an input of a second decoding module of the decoder, including: performing feature distribution optimization on the first fusion decoding feature map by using the following optimization formula to obtain the optimized first fusion decoding feature map; wherein, the optimization formula is:

wherein ,

representing said first fused decoding profile, < >>

Representing the optimized first fusion decoding characteristic diagram,

representing a single layer convolution operation,/->

Position-by-position addition of the representation feature map, +.>

Position-wise subtraction of the representation characteristic map, +.>

Position-wise multiplication of the characteristic map, and +.>

and />

Is a bias profile.

More specifically, in step S130, the key point detection is performed on the image of the region of interest of the face to obtain a plurality of key feature points of the face. It should be understood that the key point detection of the image of the region of interest of the face to obtain a plurality of key feature points of the face is one of important steps of modeling the face in the eyeglass printing based on the 3D printing technology, which can determine the shape and position of the face more accurately, thereby improving the precision and accuracy of the subsequent calculation. By means of a key point detection algorithm, key points (such as eyes, nose, mouth and the like) at specific positions in the image of the region of interest of the face can be automatically identified. These key points may provide detailed information about the face, such as facial contours, expressions, etc. It should be appreciated that in eyeglass printing based on 3D printing technology, the key point detection algorithm can help us obtain the exact position and angle of the eyeglass on the face, so as to better calculate eyeglass parameters. By acquiring the key feature points, the shape and the position of the face can be more accurately represented and described, and a foundation is laid for subsequent transformation and 3D modeling.

More specifically, in step S140, based on the plurality of face key feature points, a rotation angle and a scaling of the face region-of-interest image are calculated, and based on the rotation angle and the scaling, the face region-of-interest image is transformed to obtain a transformed face region-of-interest image. In eyeglass printing based on 3D printing technology, since the face sizes and shapes of different people are different, the acquired face images need to be appropriately transformed for better analysis and modeling. Specifically, the rotation angle and the scaling of the face region-of-interest image are obtained through calculation, and the face region-of-interest image can be converted into a standard size and direction, so that better support is provided for subsequent 3D modeling and other operations.

More specifically, in step S150, the transformed face region of interest image is passed through a 3DMM model to obtain a face 3D model. By passing the transformed face region of interest image through a 3DMM (3D Morphable Model) model, the 2D face image can be converted into a 3D face model, and the shape, texture, etc. of the model can be finely controlled.

More specifically, in step S160, parameters of the glasses are calculated based on the face 3D model. In eyeglass printing based on 3D printing technology, eyeglass to be customized needs to calculate parameters of the eyeglass, such as contour, size, position, etc., of the eyeglass according to individual characteristics of customers. By calculating the parameters of the glasses based on the face 3D model, the design and manufacture of the glasses can be matched with the facial features of the customer, and the personalized requirements of the customer on the glasses can be met. Specifically, parameters of the glasses, including width, height, frame shape, etc., of the glasses are calculated by adopting a method of combining a face 3D model and a glasses 3D model. Through the calculated parameters, the 3D model of the glasses suitable for different customers can be generated, and further modification and adjustment can be carried out according to the requirements of the customers so as to meet the personalized requirements of the customers.

More specifically, in step S170, eyeglass 3D model printing is performed based on the parameters of the eyeglass. In this way, fully customized eyewear may be produced according to the customer's personalized needs, providing a better use experience. Meanwhile, the requirements of eyeglass manufacturers on efficiency and cost can be met, the production efficiency is improved, and the production cost is reduced.

In summary, according to the image processing method for printing with glasses according to the embodiment of the application, firstly, a face image acquired by a camera is acquired, then the face image is passed through a face region of interest generator to obtain a face region of interest image, then key point detection is performed on the face region of interest image to obtain a plurality of face key feature points, then based on the plurality of face key feature points, a rotation angle and a scaling ratio of the face region of interest image are calculated, and based on the rotation angle and the scaling ratio, the face region of interest image is transformed to obtain a transformed face region of interest image, then the transformed face region of interest image is passed through a 3DMM model to obtain a face 3D model, finally, based on the face 3D model, parameters of the glasses are calculated, and glasses 3D model printing is performed based on the parameters of the glasses. Thus, the production efficiency can be improved, and the production cost can be reduced.

Fig. 6 is a block diagram of an image processing system 100 for eyeglass printing according to an embodiment of the present application. As shown in fig. 6, an image processing system 100 for eyeglass printing according to an embodiment of the present application includes: an image acquisition module 110, configured to acquire a face image acquired by a camera; a codec module 120, configured to pass the face image through a face region of interest generator based on an encoder-decoder structure to obtain a face region of interest image, where the encoder and the decoder have a symmetrical network structure; the key point detection module 130 is configured to perform key point detection on the image of the region of interest of the face to obtain a plurality of key feature points of the face; the image transformation module 140 is configured to calculate, based on the plurality of face key feature points, a rotation angle and a scaling of the face region-of-interest image, and transform the face region-of-interest image based on the rotation angle and the scaling to obtain a transformed face region-of-interest image; the 3DMM module 150 is configured to pass the transformed face region of interest image through a 3DMM model to obtain a face 3D model; the parameter calculation module 160 is configured to calculate parameters of the glasses based on the face 3D model; and a printing module 170, configured to perform eyeglass 3D model printing based on parameters of the eyeglasses.

In one example, as shown in fig. 7, in the image processing system 100 for glasses printing described above, the codec module 120 includes: an encoding unit 121, configured to input the face image into the encoder to obtain first to fifth face feature maps, where the encoder includes five convolution layers; a first decoding unit 122, configured to input the fifth facial feature map into a first decoding module of the decoder to obtain a first decoded feature map; a fusion unit 123, configured to fuse the first decoding feature map and the fifth face feature map to obtain a first fused decoding feature map; and a second decoding unit 124, configured to perform feature distribution optimization on the first fused decoding feature map to obtain an optimized first fused decoding feature map as an input of a second decoding module of the decoder.

In one example, in the image processing system 100 for eyeglass printing described above, the encoding unit 121 is configured to: inputting the face image into a first convolution layer of the encoder to obtain the first face feature map; inputting the first face feature map into a second convolution layer of the encoder to obtain the second face feature map; inputting the second face feature map into a third convolution layer of the encoder to obtain the third face feature map; inputting the third face feature map into a fourth convolution layer of the encoder to obtain the fourth face feature map; and inputting the fourth face feature map into a fifth convolution layer of the encoder to obtain the fifth face feature map.

In one example, in the above image processing system 100 for glasses printing, inputting the face image into the first convolution layer of the encoder to obtain the first face feature map includes: and respectively carrying out two-dimensional convolution processing, feature matrix-based mean value pooling processing and nonlinear activation processing on input data by using a first convolution layer of the encoder to output the first face feature map by the first convolution layer of the encoder, wherein the input of the first convolution layer of the encoder is the face image.

In one example, in the image processing system 100 for eyeglass printing described above, the first decoding unit 122 is configured to: and performing deconvolution, pooling and nonlinear activation on the fifth face feature map by using a first decoding module of the decoder to obtain the first decoding feature map.

In one example, in the image processing system 100 for eyeglass printing described above, the fusing unit 123 is configured to: fusing the first decoding feature map and the fifth face feature map with the following cascade formula to obtain the first fused decoding feature map; wherein, the cascade formula is:

wherein ,

representing said first decoding profile, < >>

Representing the fifth facial feature map, < >>

Representing a cascade function->

And representing the first fusion decoding characteristic diagram.

In one example, in the image processing system 100 for eyeglass printing described above, the second decoding unit 124 is configured to: performing feature distribution optimization on the first fusion decoding feature map by using the following optimization formula to obtain the optimized first fusion decoding feature map; wherein, the optimization formula is:

wherein ,

representing said first fused decoding profile, < >>

Representing the optimized first fusion decoding characteristic diagram,

representing a single layer convolution operation,/->

Position-by-position addition of the representation feature map, +.>

Position-wise subtraction of the representation characteristic map, +.>

Position-wise multiplication of the characteristic map, and +.>

and />

Is a bias profile.

Here, it will be understood by those skilled in the art that the specific functions and operations of the respective modules in the above-described image processing system 100 for glasses printing have been described in detail in the above description of the image processing method for glasses printing with reference to fig. 1 to 5, and thus, repetitive descriptions thereof will be omitted.

As described above, the image processing system 100 for glasses printing according to the embodiment of the present application may be implemented in various wireless terminals, for example, a server or the like having an image processing algorithm for glasses printing. In one example, the image processing system 100 for eyeglass printing according to embodiments of the present application may be integrated into a wireless terminal as one software module and/or hardware module. For example, the image processing system 100 for eyeglass printing may be a software module in the operating system of the wireless terminal, or may be an application developed for the wireless terminal; of course, the image processing system 100 for eyeglass printing may also be one of a number of hardware modules of the wireless terminal.

Alternatively, in another example, the image processing system 100 for glasses printing and the wireless terminal may be separate devices, and the image processing system 100 for glasses printing may be connected to the wireless terminal through a wired and/or wireless network and transmit interactive information in a contracted data format.

According to another aspect of the present application, there is also provided a non-volatile computer-readable storage medium having stored thereon computer-readable instructions which, when executed by a computer, can perform a method as described above.

Program portions of the technology may be considered to be "products" or "articles of manufacture" in the form of executable code and/or associated data, embodied or carried out by a computer readable medium. A tangible, persistent storage medium may include any memory or storage used by a computer, processor, or similar device or related module. Such as various semiconductor memories, tape drives, disk drives, or the like, capable of providing storage functionality for software.

All or a portion of the software may sometimes communicate over a network, such as the internet or other communication network. Such communication may load software from one computer device or processor to another. For example: a hardware platform loaded from a server or host computer of the video object detection device to a computer environment, or other computer environment implementing the system, or similar functioning system related to providing information needed for object detection. Thus, another medium capable of carrying software elements may also be used as a physical connection between local devices, such as optical, electrical, electromagnetic, etc., propagating through cable, optical cable, air, etc. Physical media used for carrier waves, such as electrical, wireless, or optical, may also be considered to be software-bearing media. Unless limited to a tangible "storage" medium, other terms used herein to refer to a computer or machine "readable medium" mean any medium that participates in the execution of any instructions by a processor.

This application uses specific words to describe embodiments of the application. Reference to "a first/second embodiment," "an embodiment," and/or "some embodiments" means that a particular feature, structure, or characteristic is associated with at least one embodiment of the present application. Thus, it should be emphasized and should be appreciated that two or more references to "an embodiment" or "one embodiment" or "an alternative embodiment" in various positions in this specification are not necessarily referring to the same embodiment. Furthermore, certain features, structures, or characteristics of one or more embodiments of the present application may be combined as suitable.

Furthermore, those skilled in the art will appreciate that the various aspects of the invention are illustrated and described in the context of a number of patentable categories or circumstances, including any novel and useful procedures, machines, products, or materials, or any novel and useful modifications thereof. Accordingly, aspects of the present application may be performed entirely by hardware, entirely by software (including firmware, resident software, micro-code, etc.) or by a combination of hardware and software. The above hardware or software may be referred to as a "data block," module, "" engine, "" unit, "" component, "or" system. Furthermore, aspects of the present application may take the form of a computer product, comprising computer-readable program code, embodied in one or more computer-readable media.

Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

The foregoing is illustrative of the present invention and is not to be construed as limiting thereof. Although a few exemplary embodiments of this invention have been described, those skilled in the art will readily appreciate that many modifications are possible in the exemplary embodiments without materially departing from the novel teachings and advantages of this invention. Accordingly, all such modifications are intended to be included within the scope of this invention as defined in the following claims. It is to be understood that the foregoing is illustrative of the present invention and is not to be construed as limited to the specific embodiments disclosed, and that modifications to the disclosed embodiments, as well as other embodiments, are intended to be included within the scope of the appended claims. The invention is defined by the claims and their equivalents.

Claims

1. An image processing method for eyeglass printing, comprising:

acquiring a face image acquired by a camera;

based on the face 3D model, calculating parameters of the glasses; and

2. The image processing method for eyeglass printing according to claim 1, wherein the face image is passed through a face region of interest generator based on an encoder-decoder structure to obtain a face region of interest image, wherein the encoder and the decoder have a symmetrical network structure, comprising:

3. The image processing method for eyeglass printing according to claim 2, wherein the face image is input to the encoder to obtain first to fifth face feature maps, wherein the encoder includes five convolution layers, comprising:

4. The image processing method for eyeglass printing according to claim 3, wherein inputting the face image into a first convolution layer of the encoder to obtain the first face feature map comprises:

5. The image processing method for eyeglass printing according to claim 4, wherein inputting the fifth face feature map into a first decoding module of the decoder to obtain a first decoded feature map comprises:

6. The image processing method for eyeglass printing according to claim 5, wherein fusing the first decoding feature map and the fifth face feature map to obtain a first fused decoding feature map, comprising:

wherein, the cascade formula is:

；

wherein ,

representing said first decoding profile, < >>

Representing the fifth facial feature map, < >>

Representing a cascade function->

And representing the first fusion decoding characteristic diagram.

7. The image processing method for eyeglass printing according to claim 6, wherein performing feature distribution optimization on the first fusion decoding feature map to obtain an optimized first fusion decoding feature map as an input of a second decoding module of the decoder, comprises:

wherein, the optimization formula is: