US20200105007A1 - Apparatus and method for model reconstruction using photogrammetry - Google Patents

Apparatus and method for model reconstruction using photogrammetry Download PDF

Info

Publication number
US20200105007A1
US20200105007A1 US16/145,721 US201816145721A US2020105007A1 US 20200105007 A1 US20200105007 A1 US 20200105007A1 US 201816145721 A US201816145721 A US 201816145721A US 2020105007 A1 US2020105007 A1 US 2020105007A1
Authority
US
United States
Prior art keywords
nail
image
model
shape
target
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US16/145,721
Other languages
English (en)
Inventor
Jooyeon Song
David Miro Llopis
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
ManiMe Inc
Original Assignee
ManiMe Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by ManiMe Inc filed Critical ManiMe Inc
Priority to US16/145,721 priority Critical patent/US20200105007A1/en
Assigned to Mani.Me, Inc. reassignment Mani.Me, Inc. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: MIRO LLOPSIS, DAVID, SONG, Jooyeon
Priority to PCT/KR2019/012475 priority patent/WO2020067725A1/fr
Priority to EP19867687.6A priority patent/EP3859686A4/fr
Priority to CN201980063981.5A priority patent/CN113678170A/zh
Priority to JP2021543106A priority patent/JP2022502230A/ja
Priority to KR1020217008322A priority patent/KR20210042160A/ko
Priority to CA3112538A priority patent/CA3112538A1/fr
Publication of US20200105007A1 publication Critical patent/US20200105007A1/en
Priority to US17/076,684 priority patent/US11147353B2/en
Assigned to Mani.Me, Inc. reassignment Mani.Me, Inc. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: MIRO LLOPIS, David, SONG, Jooyeon
Abandoned legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G06T17/20—Finite element generation, e.g. wire-frame surface description, tesselation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/55—Depth or shape recovery from multiple images
    • G06T7/593—Depth or shape recovery from multiple images from stereo images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • A—HUMAN NECESSITIES
    • A45—HAND OR TRAVELLING ARTICLES
    • A45D—HAIRDRESSING OR SHAVING EQUIPMENT; EQUIPMENT FOR COSMETICS OR COSMETIC TREATMENTS, e.g. FOR MANICURING OR PEDICURING
    • A45D31/00—Artificial nails
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/00—2D [Two Dimensional] image generation
    • G06T11/20—Drawing from basic elements, e.g. lines or circles
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G06T17/10—Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/55—Depth or shape recovery from multiple images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/97—Determining parameters from multiple pictures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/255—Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/20—Scenes; Scene-specific elements in augmented reality scenes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/107—Static hand or arm
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2211/00—Image generation
    • G06T2211/40—Computed tomography
    • G06T2211/416—Exact reconstruction

Definitions

  • the present invention relates generally to image recognition technology and three-dimensional (3D) modeling technology, and more particularly, to model reconstruction technology using photogrammetry.
  • U.S. Pat. No. 6,196,234 entitled “Precision fit fingernails” and U.S. Pat. No. 6,382,217 entitled “Process for fabricating custom fit removable and reusable metal fingernails” disclose a method for fabricating custom-fit artificial nails using molds made from a user's nails.
  • Korean Patent Application Publication No. 10-2011-0101106 is limited in user accessibility in that a 3D scanning device dedicated to scanning nails must be used.
  • an object of the present invention is to conveniently and simply provide a user-customized artificial nail model suitable for a user.
  • Another object of the present invention is to generate a user-customized artificial nail model from a reference artificial nail model that complies with existing standard specifications without requiring a complicated process or special tool.
  • a model reconstruction apparatus using photogrammetry including an image-receiving unit for receiving image data in which images of a nail of a target, for which an artificial nail model is to be generated, and a standard specification object are taken together; an image-analyzing unit for recognizing shapes of a nail plate and a nail tip of the nail of the target from the image data, and determining parameters for the shape of the nail plate and the shape of the nail tip based on the standard specification object; and a model-generating unit for generating an artificial nail model based on the parameters.
  • the standard specification object may be an object manufactured in conformity with international standards.
  • the image data may include a first image for recognizing the shape of the nail plate and a second image for recognizing the shape of the nail tip.
  • the first image and the second image may be taken in a state in which the standard specification object is in close contact with a bottom of a finger of the target.
  • the first image may be taken of the nail plate of the target together with the standard specification object from a viewpoint in which a back of a hand of the target is viewed.
  • the second image may be taken of the nail tip together with the standard specification object from a viewpoint in which a fingertip of the target is viewed.
  • the second image may be taken in a state in which a bottom of a thumb of the target is in close contact with a first surface of the standard specification object and in which all of bottoms of remaining four fingers of the target are in close contact with a second surface of the standard specification object.
  • the image-analyzing unit may recognize the shape of the nail plate from the first image from a viewpoint in which the back of the hand is viewed, and determines parameters for the shape of the nail plate based on the standard specification object.
  • the image-analyzing unit may set at least four points on a nail wall of the nail plate in the first image from a viewpoint in which the back of the hand of the target is viewed, and may then determine the parameters using the at least four points as coordinate values for the shape of the nail plate.
  • the image-analyzing unit may determine a type of a nail shape of at least one selectable artificial nail model from the first image, based on the shape of the nail plate recognized from a viewpoint in which the back of the hand of the target is viewed.
  • the image-analyzing unit may recognize the shape of the nail tip from the second image from a viewpoint in which the fingertip of the target is viewed, and may determine parameters for the shape of the nail tip based on the standard specification object.
  • the image-analyzing unit may set two points at positions corresponding to both sides of the nail in the second image in such a way as to set one point per position from a viewpoint in which the fingertip of the target is viewed, may set an additional point at at least one position on the nail tip, and may then determine the parameters using the two points and the additional point as coordinate values for the shape of the nail tip.
  • the image-analyzing unit may generate a straight line connecting the two points, and may set a position on the nail tip, which is perpendicular to that straight line and meets a perpendicular line passing through the nail plate of the target, as the additional point.
  • the model-generating unit may adjust shapes of a nail plate and a nail tip of a reference artificial nail model based on the coordinate values for the shape of the nail plate and the shape of the nail tip, and may generate the artificial nail model from the reference artificial nail model, the shapes of the nail plate and the nail tip of which have been adjusted.
  • the model-generating unit may determine a scale value of the artificial nail model corresponding to the standard specification object based on coordinate values of the artificial nail model and coordinate values of the standard specification object.
  • a model reconstruction method using photogrammetry performed by a model reconstruction apparatus using photogrammetry, including receiving image data in which images of a nail of a target, for which an artificial nail model is to be generated, and a standard specification object are taken together; recognizing shapes of a nail plate and a nail tip of the nail of the target from the image data, and determining parameters for the shape of the nail plate and the shape of the nail tip based on the standard specification object; and generating an artificial nail model based on the parameters.
  • FIG. 1 is a block diagram illustrating a model reconstruction apparatus using photogrammetry according to an embodiment of the present invention
  • FIG. 2 is a diagram illustrating an image taken of four fingers, which are in close contact with a standard specification object, from a viewpoint in which the back of a target's hand is viewed according to an embodiment of the present invention
  • FIG. 3 is a diagram illustrating an image taken of a thumb, which is in close contact with a standard specification object, from a viewpoint in which the back of a target's hand is viewed according to an embodiment of the present invention
  • FIG. 4 is a diagram illustrating an image taken of fingers, which are in close contact with a standard specification object, from a viewpoint in which the fingertips of a target are viewed according to an embodiment of the present invention
  • FIG. 5 is a diagram illustrating point settings for recognition of the shapes of nail bodies (nail plates) and determination of coordinate values of the nail plates in an image according to an embodiment of the present invention
  • FIG. 6 is a diagram illustrating the types of nail shapes of selectable artificial nail models according to an embodiment of the present invention.
  • FIG. 7 is a diagram illustrating point settings for recognition of the shapes of nail tips (free edges of nails) and determination of coordinate values of the nail tips in an image according to an embodiment of the present invention
  • FIG. 8 is a diagram illustrating detailed adjustment of an artificial nail model according to an embodiment of the present invention.
  • FIG. 9 is an operation flowchart illustrating a model reconstruction method using photogrammetry according to an embodiment of the present invention.
  • FIG. 10 is an operation flowchart illustrating in detail an example of the image data-receiving step of FIG. 9 ;
  • FIG. 11 is an operation flowchart illustrating in detail an example of the image data-analyzing step of FIG. 9 ;
  • FIG. 12 is an operation flowchart illustrating in detail an example of the artificial nail model-generating step of FIG. 9 ;
  • FIG. 13 is a diagram illustrating a computer system according to an embodiment of the present invention.
  • FIG. 1 is a block diagram illustrating a model reconstruction apparatus using photogrammetry according to an embodiment of the present invention.
  • FIG. 2 is a diagram illustrating an image taken of four fingers, which are in close contact with a standard specification object, from a viewpoint in which the back of a target's hand is viewed according to an embodiment of the present invention.
  • FIG. 3 is a diagram illustrating an image taken of a thumb, which is in close contact with a standard specification object, from a viewpoint in which the back of a target's hand is viewed according to an embodiment of the present invention.
  • FIG. 4 is a diagram illustrating an image taken of fingers, which are in close contact with a standard specification object, from a viewpoint in which the fingertips of a target are viewed according to an embodiment of the present invention.
  • FIG. 1 is a block diagram illustrating a model reconstruction apparatus using photogrammetry according to an embodiment of the present invention.
  • FIG. 2 is a diagram illustrating an image taken of four fingers, which are in close contact with a standard specification
  • FIG. 5 is a diagram illustrating point settings for recognition of the shapes of nail bodies (nail plates) and determination of coordinate values of the nail plates in an image according to an embodiment of the present invention.
  • FIG. 6 is a diagram illustrating the types of nail shapes of selectable artificial nail models according to an embodiment of the present invention.
  • FIG. 7 is a diagram illustrating point settings for recognition of the shapes of nail tips (free edges of nails) and determination of coordinate values of the nail tips in an image according to an embodiment of the present invention.
  • FIG. 8 is a diagram illustrating detailed adjustment of an artificial nail model according to an embodiment of the present invention.
  • the model reconstruction apparatus using photogrammetry includes an image-receiving unit 110 , an image-analyzing unit 120 , a model-generating unit 130 , and an image-learning unit 140 .
  • the image-receiving unit 110 may receive image data in which images of the nails of a target, for which a model of artificial nails is to be generated, and a standard specification object are taken together.
  • a user terminal device may generate the image data by taking an image of the nails of the target together with the standard specification object.
  • the image data may be generated using various types of user terminal devices, such as a smartphone, a smart tablet, or a digital camera, which can take a two-dimensional (2D) digital image and then generate image data.
  • a smartphone such as a smartphone, a smart tablet, or a digital camera, which can take a two-dimensional (2D) digital image and then generate image data.
  • 2D two-dimensional
  • the user terminal device may request the user to generate image data needed to generate the model of artificial nails (hereinafter also referred to as the “artificial nail model”) under the guidance of an application, software or a website for providing an artificial nail model generation service.
  • artificial nail model the model of artificial nails
  • the application, software, or website for providing the artificial nail model generation service may output information needed to generate the artificial nail model as a message through the user terminal device.
  • the information needed to generate the artificial nail model may include a message for explaining a method for taking an image of the target's hand in order to generate the artificial nail model, and the like.
  • the application, software, or website for providing the artificial nail model generation service may control the user terminal device so that image data needed to generate the artificial nail model is taken using a built-in camera installed in the user terminal device.
  • the user terminal device may transmit the image data to the model reconstruction apparatus using photogrammetry over a network.
  • the user terminal device may also upload the image data onto the application, software, or website for providing the artificial nail model generation service.
  • the image-receiving unit 110 may receive the image data from the user terminal device over the network, or may download the image data, uploaded by the user to request the generation of artificial nails, from the application, software, or website for providing the artificial nail model generation service.
  • the image data may include a first image for recognizing the shape of each nail body (i.e. each nail plate) and a second image for recognizing the shape of each nail tip (i.e. the free edge of each nail).
  • FIGS. 2 and 3 illustrate examples of the first image
  • FIG. 4 illustrates an example of the second image
  • the first image and the second image may be taken in the state in which a standard specification object is in close contact with the bottoms of the fingers of the target.
  • the first image may be obtained by taking an image of the nail plates of the target together with the standard specification object from a viewpoint in which the back of the target's hand is viewed.
  • the first image may include an image taken in the state in which the bottom of the thumb of a left hand is in close contact with the standard specification object from a viewpoint in which the back of the target's hand is viewed, an image taken in the state in which the bottoms of the remaining four fingers of the left hand are in close contact with the standard specification object from a viewpoint in which the back of the target's hand is viewed, an image taken in the state in which the bottom of the thumb of a right hand is in close contact with the standard specification object from a viewpoint in which the back of the target's hand is viewed, and an image taken in the state in which the bottoms of the remaining four fingers of the right hand are in close contact with the standard specification object from a viewpoint in which the back of the target's hand is viewed.
  • the second image may be obtained by taking an image of nail tips (free edges of nails) together with the standard specification object from a viewpoint in which the fingertips of the target are viewed.
  • the second image may include a left hand image and a right hand image taken in the state in which the bottom of the thumb of the target's corresponding hand is in close contact with one surface of the standard specification object and in which all of the bottoms of the remaining four fingers are in close contact with the other surface of the standard specification object.
  • the image of the fingertips of the target must be taken from a viewpoint opposite the direction in which the fingers are extended in order to recognize the extents to which the nail tips are curved (i.e. the extents of curvature of the nail tips).
  • the image data may further include a third image obtained by taking an image of the sides of the fingers in order to recognize the extents to which nail plates are curved.
  • the standard specification object may be an object made based on international standards.
  • the standard specification object may be an object manufactured based on international standards developed by the International Organization for Standardization (ISO), and the object manufactured based on the international standards may be any of various objects such as credit cards, bills, coins, and paper.
  • ISO International Organization for Standardization
  • a credit card will be described below as an example of the standard specification object according to the embodiment of the present invention.
  • the image-analyzing unit 120 may recognize the shapes of nail plates and nail tips of the nails of the target from the image data, and may determine parameters for the shapes of nail plates and the shapes of nail tips based on the standard specification object.
  • the image-analyzing unit 120 may recognize, from the first image, the shapes of the nail plates from a viewpoint in which the back of the hand is viewed, and may determine parameters for the shapes of the nail plates based on the standard specification object.
  • the image-analyzing unit 120 may set at least four points on the nail wall of each nail plate, from a viewpoint in which the back of the target's hand is viewed, in the first image, and may then determine the parameters using the at least four points as coordinate values for the shape of the corresponding nail plate.
  • the image-analyzing unit 120 may determine the type of nail shape of at least one selectable artificial nail model based on the shape of the nail plate recognized from the first image from a viewpoint in which the back of the target's hand is viewed.
  • FIG. 6 it can be seen that the types of nail shapes of the artificial nail models, which are selectable by the model reconstruction apparatus using photogrammetry according to the embodiment of the present invention, are depicted.
  • the types of nail shapes according to the embodiment of the present invention may include a square shape, a square oval, squoval or semi-square shape, a rounded shape, an oval shape, an almond shape, a ballerina or coffin shape, and a stiletto shape, and may further include additional nail shapes.
  • the image-analyzing unit 120 may select one or more of the nail shapes of the artificial nail model in order to recommend the nail shape of an artificial nail model to the user by comparing the recognized shape of the nail plate of the target with the nail plate of the reference artificial nail model, and may recommend the selected nail shapes of the artificial nail model to the user through the user terminal device.
  • the image-analyzing unit 120 may select a square, square oval, squoval or semi-square shape.
  • the image-analyzing unit 120 may recommend a rounded shape.
  • the image-analyzing unit 120 may select an oval or almond shape.
  • the image-analyzing unit 120 may recognize the shape of each nail tip from the second image from a viewpoint in which the fingertips of the target are viewed, and may determine parameters for the shape of the nail tip based on the standard specification object.
  • the image-analyzing unit 120 may set two points at positions corresponding to both sides of each nail from the second image in such a way as to set one point per position from a viewpoint in which the fingertips of the target are viewed, may set an additional point at at least one position on the nail tip, and may then determine the parameters using the two points and the additional point as coordinate values for the shape of the nail tip.
  • the image-analyzing unit 120 may generate a straight line connecting the two points, and may set a point on the nail tip, which is perpendicular to that straight line and meets a perpendicular line passing through the nail plate of the target, as the additional point.
  • the image-analyzing unit 120 may set two points on the standard specification object, may generate a straight line connecting the two points, and may adjust the locations of coordinate values for the shape of the nail tip using the slope of the straight line generated on the standard specification object.
  • the image-analyzing unit 120 may recognize, from the third image, the shape of each nail side, taken from a viewpoint in which the sides of the fingers of the target are viewed, and may determine parameters for the shape of the nail side.
  • the image-analyzing unit 120 may set two points at the nail tip (free edge) and the nail root of the nail plate in the third image in such a way as to set one point at the nail tip and the other point at the nail root from a viewpoint in which the side of each finger is viewed, may set an additional point at at least one position on the nail plate, and may then determine the parameters using the two points and the additional point as coordinate values for the shape of the nail side.
  • the image-analyzing unit 120 may generate a straight line connecting the two points, and may set a position on the nail plate, which is perpendicular to that straight line and meets a perpendicular line passing through the nail plate of the target, as the additional point.
  • the image-analyzing unit 120 may set two points on the standard specification object, may generate a straight line connecting the two points, and may adjust the locations of coordinate values for the shape of the nail side using the slope of the straight line generated in the standard specification object.
  • the model-generating unit 130 may generate an artificial nail model based on the parameters.
  • the model-generating unit 130 may adjust the shape of each nail plate for the reference artificial nail model based on the coordinate values for the shape of each nail plate, and may generate the artificial nail model from the reference artificial nail model, the shape of the nail plate of which has been adjusted.
  • the model-generating unit 130 may adjust the shape of each nail tip of the reference artificial nail model based on the coordinate values for the shape of each nail tip, and may generate the artificial nail model from the reference artificial nail model, the shape of the nail tip of which has been adjusted.
  • the model-generating unit 130 may adjust the shape of the side of each nail plate of the reference artificial nail model based on the coordinate values for the shape of the side of each nail plate, and may generate the artificial nail model from the reference artificial nail model, the shape of the side of the nail plate of which has been adjusted.
  • the model-generating unit 130 may determine the scale value of the artificial nail model corresponding to the standard specification object based on the coordinate values of the artificial nail model and the coordinate values of the standard specification object.
  • model-generating unit 130 may determine the scale value of the artificial nail model by comparing the coordinate values of each nail and the standard specification object, which are taken together in the first image, the second image, and the third image, and may then calculate the actual size of the artificial nail model.
  • the model-generating unit 130 may designate that the nail and the standard specification object, taken in the first image, are located in an X-Y plane of a 3D coordinate system, may calculate the extent of curvature of the nail tip and the extent of curvature of the side of the nail plate based on the coordinate values recognized from the shape of the nail tip in the second image and the coordinate values recognized from the shape of the side of the nail plate in the third image, and may calculate a scalar space for which the extent of curvature of the nail tip and the extent of curvature of the side of the nail plate are calculated as coordinates on a Z axis with respect to the X-Y plane, thus enabling the artificial nail model in the 3D coordinate system to be generated.
  • FIG. 8 it can be seen that detailed adjustment of the artificial nail model according to an embodiment of the present invention is depicted.
  • model-generating unit 130 may finely adjust the thickness, horizontal curvature height, vertical curvature height, and nail length of the generated artificial nail model in millimeters (mm). Also, additional parameters may be further included and adjusted, and the units of detailed adjustment are not limited thereto.
  • the model-generating unit 130 may deliver data about the finally generated artificial nail model to the user through a download function (DOWNLOAD), and may store the artificial nail model through a project-saving function (SAVE PROJECT).
  • DOWNLOAD download function
  • SAVE PROJECT project-saving function
  • the model-generating unit 130 may generate the artificial nail model in the form of a 3D model file (e.g. a stereolithography (STL) file or the like) that can be 3D-printed.
  • a 3D model file e.g. a stereolithography (STL) file or the like
  • model-generating unit 130 may transfer the image data used in a procedure for generating the artificial nail model, the coordinate values, and the parameters adjusted from the reference artificial nail model to the image-learning unit 140 in order to generate learning data from the image data, the coordinate values, and the adjusted parameters.
  • the image-learning unit 140 may generate learning data by learning the image data and the coordinate values, which are used to generate the artificial nail model, and the parameters, which are adjusted from the reference artificial nail model, and may provide the learning data to the model-generating unit 130 .
  • model-generating unit 130 may more effectively determine the shape and scale value of the artificial nail model in a 3D coordinate system using the learning data.
  • FIG. 9 is an operation flowchart illustrating a model reconstruction method using photogrammetry according to an embodiment of the present invention.
  • FIG. 10 is an operation flowchart illustrating in detail an example of the image data-receiving step of FIG. 9 .
  • FIG. 11 is an operation flowchart illustrating in detail an example of the image data-analyzing step of FIG. 9 .
  • FIG. 12 is an operation flowchart illustrating in detail an example of the artificial nail model-generating step of FIG. 9 .
  • the model reconstruction method using photogrammetry may receive image data at step S 210 .
  • step S 210 image data in which images of the nails of a target, for which an artificial nail model is to be generated, and a standard specification object are taken together may be received.
  • the image data may be generated at step S 211 .
  • a user terminal device may generate the image data by taking an image of the nails of the target together with the standard specification object.
  • the image data may be generated using various types of user terminal devices, such as a smartphone, a smart tablet, or a digital camera, which can take a two-dimensional (2D) digital image and then generate image data.
  • a smartphone such as a smartphone, a smart tablet, or a digital camera, which can take a two-dimensional (2D) digital image and then generate image data.
  • 2D two-dimensional
  • the user terminal device may request the user to generate image data needed to generate the artificial nail model under the guidance of an application, software or a website for providing an artificial nail model generation service.
  • the application, software, or website for providing the artificial nail model generation service may output information needed to generate the artificial nail model as a message through the user terminal device.
  • the information needed to generate the artificial nail model may include a message for explaining a method for taking an image of the target's hand in order to generate the artificial nail model, and the like.
  • the application, software, or website for providing the artificial nail model generation service may control the user terminal device so that image data needed to generate the artificial nail model is taken using a built-in camera installed in the user terminal device.
  • the image data may be transmitted at step S 212 .
  • the user terminal device may transmit the image data to the model reconstruction apparatus using photogrammetry over a network.
  • the user terminal device may also upload the image data onto the application, software, or website for providing the artificial nail model generation service.
  • the image data may be received at step S 213 .
  • the image data may be received from the user terminal device over the network, or alternatively, the image data, uploaded by the user to request the generation of artificial nails, may be downloaded from the application, software, or website for providing the artificial nail model generation service.
  • the image data may include a first image for recognizing the shape of each nail body (i.e. each nail plate) and a second image for recognizing the shape of each nail tip (i.e. the free edge of each nail).
  • FIGS. 2 and 3 illustrate examples of the first image
  • FIG. 4 illustrates an example of the second image
  • the first image and the second image may be taken in the state in which a standard specification object is in close contact with the bottoms of the fingers of the target.
  • the first image may be obtained by taking an image of the nail plates of the target together with the standard specification object from a viewpoint in which the back of the target's hand is viewed.
  • the first image may include an image taken in the state in which the bottom of the thumb of a left hand is in close contact with the standard specification object from a viewpoint in which the back of the target's hand is viewed, an image taken in the state in which the bottoms of the remaining four fingers of the left hand are in close contact with the standard specification object from a viewpoint in which the back of the target's hand is viewed, an image taken in the state in which the bottom of the thumb of a right hand is in close contact with the standard specification object from a viewpoint in which the back of the target's hand is viewed, and an image taken in the state in which the bottoms of the remaining four fingers of the right hand are in close contact with the standard specification object from a viewpoint in which the back of the target's hand is viewed.
  • the second image may be obtained by taking an image of nail tips (free edges of nails) together with the standard specification object from a viewpoint in which the fingertips of the target are viewed.
  • the second image may include a left hand image and a right hand image taken in the state in which the bottom of the thumb of the target's corresponding hand is in close contact with one surface of the standard specification object and in which all of the bottoms of the remaining four fingers are in close contact with the other surface of the standard specification object.
  • the image of the fingertips of the target must be taken from a viewpoint opposite the direction in which the fingers are extended in order to recognize the extents to which the nail tips are curved (i.e. the extents of curvature of the nail tips).
  • the image data may further include a third image obtained by taking an image of the sides of the fingers in order to recognize the extents to which nail plates are curved.
  • the standard specification object may be an object made based on international standards.
  • the standard specification object may be an object manufactured based on international standards developed by the International Organization for Standardization (ISO), and the object manufactured based on the international standards may be any of various objects such as credit cards, bills, coins, and paper.
  • ISO International Organization for Standardization
  • a credit card will be described below as an example of the standard specification object according to the embodiment of the present invention.
  • the model reconstruction method using photogrammetry may analyze the image data at step S 220 .
  • the shapes of nail plates and nail tips of the nails of the target may be recognized from the image data, and parameters for the shapes of nail plates and the shapes of nail tips may be determined based on the standard specification object.
  • the shapes of the nail plates may be recognized from the first image from a viewpoint in which the back of the hand is viewed, and parameters for the shapes of the nail plates may be determined based on the standard specification object.
  • parameters for the shape of each nail plate may be determined first at step S 221 . That is, at step S 221 , at least four points may be set on the nail wall of each nail plate in the first image, from a viewpoint in which the back of the target's hand is viewed, and then the parameters may be determined using the at least four points as coordinate values for the shape of the corresponding nail plate.
  • the type of nail shape of at least one selectable artificial nail model may be determined based on the shape of the nail plate recognized from the first image from a viewpoint in which the back of the target's hand is viewed.
  • FIG. 6 it can be seen that the types of nail shapes of the artificial nail models, which are selectable by the model reconstruction apparatus using photogrammetry according to the embodiment of the present invention, are depicted.
  • the types of nail shapes according to the embodiment of the present invention may include a square shape, a square oval, squoval or semi-square shape, a rounded shape, an oval shape, an almond shape, a ballerina or coffin shape, and a stiletto shape, and may further include additional nail shapes.
  • one or more of the nail shapes of the artificial nail model may be selected in order to recommend the nail shape of an artificial nail model to the user by comparing the recognized shape of the nail plate of the target with the nail plate of the reference artificial nail model, and the selected nail shapes of the artificial nail model may be recommended to the user through the user terminal device.
  • a square, square oval, squoval or semi-square shape may be selected.
  • a rounded shape may be selected.
  • an oval or almond shape may be selected.
  • parameters for the shape of each nail tip may be determined at step S 222 .
  • the shape of each nail tip may be recognized from the second image from a viewpoint in which the fingertips of the target are viewed, and parameters for the shape of the nail tip may be determined based on the standard specification object.
  • two points may be set at positions corresponding to both sides of each nail from the second image in such a way as to set one point per position from a viewpoint in which the fingertips of the target are viewed, an additional point may be set at at least one position on the nail tip, and then the parameters may be determined using the two points and the additional point as coordinate values for the shape of the nail tip.
  • a straight line connecting the two points may be generated, and a point on the nail tip, which is perpendicular to that straight line and meets a perpendicular line passing through the nail plate of the target, may be set as the additional point.
  • two points may be set on the standard specification object, a straight line connecting the two points may be generated, and the locations of coordinate values for the shape of the nail tip may be adjusted using the slope of the straight line generated on the standard specification object.
  • parameters for the shape of the side of each nail may be determined at step S 223 .
  • each nail side taken from a viewpoint in which the sides of the fingers of the target are viewed, may be recognized from the third image, and parameters for the shape of the nail side may be determined.
  • two points may be set at the nail tip (free edge) and the nail root of the nail plate in the third image in such a way as to set one point at the nail tip and the other point at the nail root from a viewpoint in which the side of each finger is viewed, an additional point may be set at at least one position on the nail plate, and then the parameters may be determined using the two points and the additional point as coordinate values for the shape of the nail side.
  • a straight line connecting the two points may be generated, and a position on the nail plate, which is perpendicular to that straight line and meets a perpendicular line passing through the nail plate of the target, may be set as the additional point.
  • two points may be set on the standard specification object, a straight line connecting the two points may be generated, and the locations of coordinate values for the shape of the nail side may be adjusted using the slope of the straight line generated in the standard specification object.
  • step S 220 the execution sequence of steps S 221 , S 222 , and S 223 included in step S 220 may also be changed.
  • the model reconstruction method using photogrammetry may generate an artificial nail model at step S 230 .
  • the artificial nail model may be generated based on the parameters.
  • an artificial nail model may be generated by adjusting the shape of a reference nail model at step S 231 .
  • the shape of each nail plate for the reference artificial nail model may be adjusted based on the coordinate values for the shape of each nail plate, and the artificial nail model may be generated from the reference artificial nail model, the shape of the nail plate of which has been adjusted.
  • the shape of each nail tip of the reference artificial nail model may be adjusted based on the coordinate values for the shape of each nail tip, and the artificial nail model may be generated from the reference artificial nail model, the shape of the nail tip of which has been adjusted.
  • the shape of the side of each nail plate of the reference artificial nail model may be adjusted based on the coordinate values for the shape of the side of each nail plate, and the artificial nail model may be generated from the reference artificial nail model, the shape of the side of the nail plate of which has been adjusted.
  • the scale value of the artificial nail model may be determined at step S 232 .
  • the scale value of the artificial nail model corresponding to the standard specification object may be determined based on the coordinate values of the artificial nail model and the coordinate values of the standard specification object.
  • the scale value of the artificial nail model may be determined by comparing the coordinate values of each nail and the standard specification object, which are taken together in the first image, the second image, and the third image, and the actual size of the artificial nail model may then be calculated.
  • the nail and the standard specification object, taken in the first image may be designated to be located in an X-Y plane of a 3D coordinate system
  • the extent of curvature of the nail tip and the extent of curvature of the side of the nail plate may be calculated based on the coordinate values recognized from the shape of the nail tip in the second image and the coordinate values recognized from the shape of the side of the nail plate in the third image
  • a scalar space for which the extent of curvature of the nail tip and the extent of curvature of the side of the nail plate are calculated as coordinates on a Z axis with respect to the X-Y plane may be calculated, thus enabling the artificial nail model in the 3D coordinate system to be generated.
  • FIG. 8 it can be seen that detailed adjustment of the artificial nail model according to an embodiment of the present invention is depicted.
  • the thickness, horizontal curvature height, vertical curvature height, and nail length of the generated artificial nail model may be finely adjusted in millimeters (mm). Also, additional parameters may be further included and adjusted, and the units of detailed adjustment are not limited thereto.
  • step S 232 data about the finally generated artificial nail model may be delivered to the user through a download function (DOWNLOAD), and the artificial nail model may be stored through a project-saving function (SAVE PROJECT).
  • DOWNLOAD download function
  • SAVE PROJECT project-saving function
  • the artificial nail model may be generated in the form of a 3D model file (e.g. a stereolithography (STL) file or the like) that can be 3D-printed.
  • a 3D model file e.g. a stereolithography (STL) file or the like
  • learning data may be generated at step S 233 .
  • the learning data may be generated from the image data and the coordinate values, which are used in a procedure for generating the artificial nail model, and the parameters, which are adjusted from the reference artificial nail model.
  • the learning data may be generated by learning the image data and the coordinate values, which are used to generate the artificial nail model, and the parameters, which are adjusted from the reference artificial nail model.
  • the learning data may be used at steps S 231 and S 232 , thus enabling the shape and the scale value of the artificial nail model in a 3D coordinate system to be more effectively determined.
  • FIG. 13 is a diagram illustrating a computer system according to an embodiment of the present invention.
  • a model reconstruction apparatus using photogrammetry may be implemented in a computer system 1100 such as a computer-readable storage medium.
  • the computer system 1100 may include one or more processors 1110 , memory 1130 , a user interface input device 1140 , a user interface output device 1150 , and storage 1160 , which communicate with each other through a bus 1120 .
  • the computer system 1100 may further include a network interface 1170 connected to a network 1180 .
  • Each of the processors 1110 may be either a Central Processing Unit (CPU) or a semiconductor device which executes processing instructions stored in the memory 1130 or the storage 1160 .
  • Each of the memory 1130 and the storage 1160 may be any of various types of volatile or nonvolatile storage media.
  • the memory may include Read-Only Memory (ROM) 1131 or Random Access Memory 1132 .
  • the present invention may conveniently and simply provide a user-customized artificial nail model suitable for a user.
  • the present invention may generate a user-customized artificial nail model from a reference artificial nail model that complies with existing standard specifications without requiring a complicated process or special tool.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Geometry (AREA)
  • Computer Graphics (AREA)
  • Software Systems (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Human Computer Interaction (AREA)
  • Image Processing (AREA)
  • Image Analysis (AREA)
US16/145,721 2018-09-28 2018-09-28 Apparatus and method for model reconstruction using photogrammetry Abandoned US20200105007A1 (en)

Priority Applications (8)

Application Number Priority Date Filing Date Title
US16/145,721 US20200105007A1 (en) 2018-09-28 2018-09-28 Apparatus and method for model reconstruction using photogrammetry
CA3112538A CA3112538A1 (fr) 2018-09-28 2019-09-25 Appareil et procede de reconstruction de modele a l'aide d'une photogrammetrie
JP2021543106A JP2022502230A (ja) 2018-09-28 2019-09-25 フォトグラメトリを使用するモデル再構成装置と方法
EP19867687.6A EP3859686A4 (fr) 2018-09-28 2019-09-25 Appareil et procédé de reconstruction de modèle à l'aide d'une photogrammétrie
CN201980063981.5A CN113678170A (zh) 2018-09-28 2019-09-25 一种利用摄影测量法的模型重建的设备和方法
PCT/KR2019/012475 WO2020067725A1 (fr) 2018-09-28 2019-09-25 Appareil et procédé de reconstruction de modèle à l'aide d'une photogrammétrie
KR1020217008322A KR20210042160A (ko) 2018-09-28 2019-09-25 사진 측량을 이용한 모델 재구성 장치 및 방법
US17/076,684 US11147353B2 (en) 2018-09-28 2020-10-21 Apparatus and method for model reconstruction using photogrammetry

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
US16/145,721 US20200105007A1 (en) 2018-09-28 2018-09-28 Apparatus and method for model reconstruction using photogrammetry

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/076,684 Continuation US11147353B2 (en) 2018-09-28 2020-10-21 Apparatus and method for model reconstruction using photogrammetry

Publications (1)

Publication Number Publication Date
US20200105007A1 true US20200105007A1 (en) 2020-04-02

Family

ID=69947732

Family Applications (2)

Application Number Title Priority Date Filing Date
US16/145,721 Abandoned US20200105007A1 (en) 2018-09-28 2018-09-28 Apparatus and method for model reconstruction using photogrammetry
US17/076,684 Active US11147353B2 (en) 2018-09-28 2020-10-21 Apparatus and method for model reconstruction using photogrammetry

Family Applications After (1)

Application Number Title Priority Date Filing Date
US17/076,684 Active US11147353B2 (en) 2018-09-28 2020-10-21 Apparatus and method for model reconstruction using photogrammetry

Country Status (7)

Country Link
US (2) US20200105007A1 (fr)
EP (1) EP3859686A4 (fr)
JP (1) JP2022502230A (fr)
KR (1) KR20210042160A (fr)
CN (1) CN113678170A (fr)
CA (1) CA3112538A1 (fr)
WO (1) WO2020067725A1 (fr)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2020176560A1 (fr) * 2019-02-28 2020-09-03 Cochran Brianna Storm Ongles artificiels personnalisés imprimés en 3d solubles conçus au moyen d'une application mobile
WO2022235709A1 (fr) * 2021-05-03 2022-11-10 Ditto Technologies, Inc. Génération d'un modèle 3d d'un objet de référence pour effectuer une mise à l'échelle d'un modèle de la tête d'un utilisateur

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR102509296B1 (ko) * 2022-02-03 2023-03-10 이경은 네일팁제품 매칭 방법

Family Cites Families (25)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5196234A (en) 1986-08-29 1993-03-23 Gte Products Corporation Method for preparing zinc orthosilicate phosphor particle
DE4232956C2 (de) 1992-10-01 2002-02-21 Jagenberg Diana Gmbh Verfahren und Vorrichtung zum Nachbrechen der Längsrillinien bei der Herstellung von quaderförmigen Faltschachteln
US5968302A (en) 1997-05-16 1999-10-19 Ova Nail Products, Inc. Methods for manufacturing precision fit fingernails
US6328949B1 (en) * 1998-10-14 2001-12-11 Dino Tessarolo Nail covering system
US6382217B2 (en) 2000-04-06 2002-05-07 Wade Coker Process for fabricating custom fit removable and reusable metal fingernails
ATE554675T1 (de) * 2002-11-13 2012-05-15 Miic America Inc System und verfahren zur herstellung eines anpassbaren künstlichen fingernagels mit einer kontaktlosen optischen messvorrichtung
US20090092310A1 (en) * 2004-02-06 2009-04-09 Gifford Craig P System and method for precision fit artificial fingernails
WO2005077132A2 (fr) * 2004-02-06 2005-08-25 Nielson Scott L Ongles artificiels personnalises et systemes, procedes et logiciel associes
US20110087351A1 (en) * 2009-10-09 2011-04-14 Rohit Sachdeva Customized artificial nail
JP5141742B2 (ja) * 2010-10-28 2013-02-13 カシオ計算機株式会社 ネイルプリント装置及び印刷制御方法
JP5338848B2 (ja) * 2011-05-09 2013-11-13 カシオ計算機株式会社 ネイルプリント装置及びネイルプリント装置の印刷制御方法
JP2012232042A (ja) * 2011-05-09 2012-11-29 Casio Computer Co Ltd 付け爪サービス提供装置および端末装置
KR20110101106A (ko) 2011-07-08 2011-09-15 조광식 개인별 맞춤형 인조손톱 제작공급 서비스 시스템
JP5382078B2 (ja) * 2011-08-12 2014-01-08 カシオ計算機株式会社 ネイルプリント装置及び印刷制御方法
JP5831441B2 (ja) * 2012-12-17 2015-12-09 カシオ計算機株式会社 ネイルプリント装置及びネイルプリント装置の印刷方法
JP2014121344A (ja) * 2012-12-20 2014-07-03 Casio Comput Co Ltd ネイルプリント装置及びネイルプリント装置の印刷制御方法
US9227359B2 (en) * 2012-12-31 2016-01-05 Yong Li System and method for manufacturing custom-fit three-dimensional artificial nails
CA2933633A1 (fr) * 2013-12-20 2015-06-25 Avon Products, Inc. Autocollants pour ongle imprimables
WO2015126533A1 (fr) * 2014-02-19 2015-08-27 Jobani Itamar Système implémenté par ordinateur et procédé de production d'ongles artificiels personnalisés et individualisés
WO2015132734A1 (fr) * 2014-03-05 2015-09-11 Daniel Moyal Ongles imprimés en 3d
JP6699176B2 (ja) * 2016-01-06 2020-05-27 カシオ計算機株式会社 描画装置、その描画制御方法及び描画制御プログラム
CN106127181B (zh) * 2016-07-02 2020-03-20 乐活无限（北京）科技有限公司 一种虚拟试戴美甲的方法、系统
US10777018B2 (en) * 2017-05-17 2020-09-15 Bespoke, Inc. Systems and methods for determining the scale of human anatomy from images
US11515019B2 (en) * 2017-06-21 2022-11-29 H3Alth Technologies Inc. System and method for manufacturing custom nails
KR101844347B1 (ko) * 2017-08-31 2018-05-14 김지환 증강된 네일아트 서비스 제공 방법

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2020176560A1 (fr) * 2019-02-28 2020-09-03 Cochran Brianna Storm Ongles artificiels personnalisés imprimés en 3d solubles conçus au moyen d'une application mobile
WO2022235709A1 (fr) * 2021-05-03 2022-11-10 Ditto Technologies, Inc. Génération d'un modèle 3d d'un objet de référence pour effectuer une mise à l'échelle d'un modèle de la tête d'un utilisateur
US12014462B2 (en) 2021-05-03 2024-06-18 Ditto Technologies, Inc. Generation of a 3D model of a reference object to perform scaling of a model of a user's head

Also Published As

Publication number Publication date
US20210030133A1 (en) 2021-02-04
US11147353B2 (en) 2021-10-19
KR20210042160A (ko) 2021-04-16
WO2020067725A1 (fr) 2020-04-02
CA3112538A1 (fr) 2020-04-02
CN113678170A (zh) 2021-11-19
EP3859686A4 (fr) 2022-08-10
EP3859686A1 (fr) 2021-08-04
JP2022502230A (ja) 2022-01-11

Similar Documents

Publication Publication Date Title
US11147353B2 (en) Apparatus and method for model reconstruction using photogrammetry
US10650224B2 (en) Facial structural shaping
JP7191213B2 (ja) 顔モデルの生成方法、装置、電子機器及びコンピュータ可読記憶媒体
US20200226811A1 (en) Electronic device for generating avatar and method thereof
US9204702B2 (en) Golden ratio eyebrow shaping method
JP2021008126A (ja) ３ｄプリントされたカスタム着用物の生成
KR101556992B1 (ko) 얼굴 성형 시뮬레이션을 이용한 3차원 스캔 시스템
CN107609490B (zh) 控制方法、控制装置、智能镜子和计算机可读存储介质
KR102102944B1 (ko) 눈썹 스타일을 추천하는 디스플레이 방법 및 전자 장치
CN105956997A (zh) 图像形变处理的方法和装置
CN105229673A (zh) 一种装置和相关联的方法
Katina et al. The definitions of three‐dimensional landmarks on the human face: an interdisciplinary view
JP6202456B1 (ja) 眉描き補助マスク
TWI780919B (zh) 人臉影像的處理方法、裝置、電子設備及儲存媒體
CN108921148A (zh) 确定正脸倾斜角度的方法及装置
CN103955962A (zh) 一种虚拟人头发生成的装置及方法
KR20140006138A (ko) 가상성형 단말기 및 그를 이용한 가상성형 시스템
KR20190010767A (ko) 발 모양 측정 시스템 및 방법
CN217718730U (zh) 一种确定患者佩戴面罩型号的装置
US20230124480A1 (en) Customized artificial fingernail generation systems
JP2013164850A (ja) 類似および所定の特徴を有する実在モデルから三次元描写を作成するシステム
EP3568835A1 (fr) Système de fabrication de produits personnalisés au moyen d'une fabrication additive réalisant une reconnaissance à base d'image à l'aide de dispositifs électroniques à caméra unique
US20240268928A1 (en) System, method, program and computer-readable recording medium for manufacturing 3-dimentional dental laminate
JP2024106458A (ja) 爪曲面の曲率測定が可能な爪測定システム
JP7094753B2 (ja) レンズモデルの作成方法、及び、眼鏡レンズの作成方法

Legal Events

Date Code Title Description
AS Assignment

Owner name: MANI.ME, INC., CONNECTICUT

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:SONG, JOOYEON;MIRO LLOPSIS, DAVID;REEL/FRAME:047005/0628

Effective date: 20180920

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED

AS Assignment

Owner name: MANI.ME, INC., CALIFORNIA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:SONG, JOOYEON;MIRO LLOPIS, DAVID;REEL/FRAME:055402/0839

Effective date: 20210222

STCB Information on status: application discontinuation

Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION