WO2023130067A1 - Method for nipple replication - Google Patents
Method for nipple replication Download PDFInfo
- Publication number
- WO2023130067A1 WO2023130067A1 PCT/US2022/082618 US2022082618W WO2023130067A1 WO 2023130067 A1 WO2023130067 A1 WO 2023130067A1 US 2022082618 W US2022082618 W US 2022082618W WO 2023130067 A1 WO2023130067 A1 WO 2023130067A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- nipple
- user
- processor
- scanned images
- scan
- Prior art date
Links
- 210000002445 nipple Anatomy 0.000 title claims abstract description 220
- 238000000034 method Methods 0.000 title claims abstract description 104
- 230000010076 replication Effects 0.000 title claims description 6
- 230000002068 genetic effect Effects 0.000 claims abstract description 62
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 50
- 238000010801 machine learning Methods 0.000 claims abstract description 36
- 238000001514 detection method Methods 0.000 claims abstract description 12
- 239000013598 vector Substances 0.000 claims abstract description 10
- 238000010146 3D printing Methods 0.000 claims abstract description 6
- 230000008569 process Effects 0.000 claims description 34
- 238000012549 training Methods 0.000 claims description 8
- 238000005259 measurement Methods 0.000 claims description 5
- 238000000926 separation method Methods 0.000 claims description 4
- 210000000481 breast Anatomy 0.000 description 19
- 238000004891 communication Methods 0.000 description 11
- 238000013528 artificial neural network Methods 0.000 description 10
- 238000005457 optimization Methods 0.000 description 9
- 238000012545 processing Methods 0.000 description 9
- 238000010586 diagram Methods 0.000 description 8
- 238000012805 post-processing Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 6
- 230000009471 action Effects 0.000 description 4
- 238000004458 analytical method Methods 0.000 description 4
- 238000013527 convolutional neural network Methods 0.000 description 4
- 230000003362 replicative effect Effects 0.000 description 4
- 210000004883 areola Anatomy 0.000 description 3
- 238000003860 storage Methods 0.000 description 3
- 230000000007 visual effect Effects 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000006073 displacement reaction Methods 0.000 description 2
- 238000002986 genetic algorithm method Methods 0.000 description 2
- 238000010191 image analysis Methods 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 230000003278 mimic effect Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000000306 recurrent effect Effects 0.000 description 2
- 230000036548 skin texture Effects 0.000 description 2
- 239000007787 solid Substances 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 239000003990 capacitor Substances 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000001815 facial effect Effects 0.000 description 1
- 238000009432 framing Methods 0.000 description 1
- 230000036541 health Effects 0.000 description 1
- 238000002955 isolation Methods 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 229940127554 medical product Drugs 0.000 description 1
- 239000008267 milk Substances 0.000 description 1
- 210000004080 milk Anatomy 0.000 description 1
- 235000013336 milk Nutrition 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003062 neural network model Methods 0.000 description 1
- 210000002569 neuron Anatomy 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000000644 propagated effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 208000011580 syndromic disease Diseases 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000001052 transient effect Effects 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
- 238000012800 visualization Methods 0.000 description 1
- 230000003442 weekly effect Effects 0.000 description 1
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61J—CONTAINERS SPECIALLY ADAPTED FOR MEDICAL OR PHARMACEUTICAL PURPOSES; DEVICES OR METHODS SPECIALLY ADAPTED FOR BRINGING PHARMACEUTICAL PRODUCTS INTO PARTICULAR PHYSICAL OR ADMINISTERING FORMS; DEVICES FOR ADMINISTERING FOOD OR MEDICINES ORALLY; BABY COMFORTERS; DEVICES FOR RECEIVING SPITTLE
- A61J11/00—Teats
- A61J11/0035—Teats having particular shape or structure
- A61J11/0045—Aesthetic properties
- A61J11/005—Aesthetic properties for imitating a nipple
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B33—ADDITIVE MANUFACTURING TECHNOLOGY
- B33Y—ADDITIVE MANUFACTURING, i.e. MANUFACTURING OF THREE-DIMENSIONAL [3-D] OBJECTS BY ADDITIVE DEPOSITION, ADDITIVE AGGLOMERATION OR ADDITIVE LAYERING, e.g. BY 3-D PRINTING, STEREOLITHOGRAPHY OR SELECTIVE LASER SINTERING
- B33Y50/00—Data acquisition or data processing for additive manufacturing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
- G06T7/55—Depth or shape recovery from multiple images
- G06T7/579—Depth or shape recovery from multiple images from motion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30068—Mammography; Breast
Definitions
- the present disclosure relates generally to feeding devices for infants. Specifically, the present disclosure relates to methods of customizing feeding device nipples.
- Feeding devices such as baby bottles, are often used to feed babies from newborns to toddlers for various reasons.
- Reasons for using a feeding device include, but are not limited to: latching difficulties by the baby, inability for the mother to produce enough milk, feeding by a caregiver or physician other than the mother, inability for the mother to breastfeed for health reasons, weaning of the baby, etc.
- Embodiments of the present disclosure relate to a method including receiving, by a processor, from a camera of a mobile device, a plurality of scanned images. The method also includes applying, by the processor, an object detection model of a machine learning engine to the plurality of scanned images to identify a user’s nipple within the plurality of scanned images based on a plurality of object feature vectors; generating, by the processor, an scan image output comprising a plurality of object feature vectors identifying the user’s nipple.
- the method also includes applying, by the processor, a genetic algorithm to extract a plurality of geometric features from the scan image output, where the plurality of geometric features identifies: nipple-related height, nipple-related width, nipple-related texture, and nipple-related color.
- the method also includes generating, by the processor, a 3D model of the user’s nipple based on the plurality of geometric features.
- the method also includes determining, by the processor, based on the 3D model of the user’s nipple, a baby bottle nipple profile corresponding to the user’s nipple to allow 3D printing of a custom baby bottle nipple, where the custom baby bottle nipple is a 3D replication of the user’s nipple.
- the method further includes training, by the processor, the machine learning engine to identify the nipple within the plurality of scanned images based at least in part on the plurality of scanned images comprising at least a portion of a nipple of a human.
- the plurality of scanned images is a video comprising at least two image frames.
- the machine learning engine is trained to identify the nipple within each frame of the plurality of scanned images.
- the method further includes prompting, by the processor, the user to rescan the nipple if the machine learning engine does not identify a nipple within each frame of the plurality of scanned images.
- the method further includes gathering and creating a point cloud, by the processor, by stitching each image frame of the plurality of scanned images together.
- a first genetic process of the genetic algorithm includes orienting, by the processor, the point cloud from the scan image with a teat of the user’s nipple in a predetermined direction.
- the predetermined direction is along a positive z axis.
- a second genetic process of the genetic algorithm comprises setting, by the processor, an average normal at a top portion of the point cloud as close to a positive z axis as possible.
- a third genetic process of the genetic algorithm includes maximizing, by the processor, a height at which the teat exceeds a predetermined cross-sectional diameter.
- the predetermined cross-sectional diameter is 30 mm.
- the genetic algorithm is configured to filter out a normal of each point in the point cloud further and further away from the positive z axis, until there is a clear separation between the teat and a base of the user’s nipple.
- the method further includes rebuilding, by the processor, the user’s nipple without any gaps or holes by extracting key contours and measurements from the image scan.
- the plurality of geometric features comprises a bounding box.
- FIG 1 is a block diagram illustrating an operating computer architecture for predictive visualization of a medical procedure of a user, according to one or more embodiments of the present disclosure.
- FIG 3 is a schematic diagram depicting an exemplary path for scanning a nipple of a user, according to one or more embodiments of the present disclosure.
- FIG . 4 is a schematic diagram of a plurality of frames of a scan image, according to one or more embodiments of the present disclosure.
- FIG 5 is an exemplary output scan image with a bounding box, according to one or more embodiments of the present disclosure.
- FIG 6 is a schematic diagram of a nipple depicting a base diameter and a teat diameter, according to one or more embodiments of the present disclosure.
- FIG 7A is an isolated nipple point cloud, according to one or more embodiments of the present disclosure.
- FIG 7B is a nipple point cloud oriented using point cloud normal, according to one or more embodiments of the present disclosure.
- FIG 7C is a nipple point cloud with point cloud normals filtered to create a gap in the scan that isolates the teat from the breast, according to one or more embodiments of the present disclosure.
- FIG 7D is a nipple point cloud with predetermined contours combined to rebuild the user’s nipple without any gaps or holes, according to one or more embodiments of the present disclosure.
- FIG 7E is a marching cube watertight mesh, according to one or more embodiments of the present disclosure.
- FIG. 8 is a flow diagram illustrating a genetic algorithm method, according to one or more embodiments of the present disclosure.
- FIG 9 is a three-dimensional point cloud after a first genetic process of a genetic algorithm, according to one or more embodiments of the present disclosure.
- FIG 10 is a three-dimensional point cloud after a second genetic process of a genetic algorithm, according to one or more embodiments of the present disclosure.
- FIG 11 is a three-dimensional point cloud after a third genetic process of a genetic algorithm, according to one or more embodiments of the present disclosure.
- FIG 12 is a three-dimensional point cloud after the genetic algorithm filters out the normal, according to one or more embodiments of the present disclosure.
- FIG 13 is a three-dimensional point cloud after the genetic algorithm filters out the normal, according to one or more embodiments of the present disclosure.
- FIG 14 is a set of graphs depicting a roughness algorithm output, according to one or more embodiments of the present disclosure.
- FIG 15 is a fine scale analysis and a rough scale analysis, according to one or more embodiments of the present disclosure.
- FIG 16 is smooth intensity map and a rough intensity map, according to one or more embodiments of the present disclosure.
- FIG 17 is a custom displacement map that estimates the skin texture roughness across the user’s nipple, according to one or more embodiments of the present disclosure.
- Nipple confusion is a common syndrome in which newborn babies have trouble latching to their mother’s breast after being fed with a baby bottle.
- one potential solution to this problem may be to make the baby bottle’s nozzle and mechanisms resemble those of the mother’s nipple.
- Described herein are methods for scanning and replicating the shape of a mother’s nipple such that the replicated nipple may be integrated with a baby bottle.
- an automated mass is created using a customization technique for replicating the shape of a mother’s nipple and integrating the replicated nipple with a larger baby bottle.
- platforms for providing enhanced scans of the mother’s breast are described herein.
- artificial intelligence is used to optimize the scans provided by the platform.
- this methods described herein use a combination of 3D scanning, Al object detection, a novel mesh post-processing and a novel texturizing procedure to create custom nipples for users.
- Al is used to produce a scan image of the mother’s breast using object detection.
- the Al includes at least one machine learning model, such as a neural network.
- the neural network is a convolutional neural network.
- the neural network is a deep learning network, a generative adversarial network, a recurrent neural network, a fully connected network, or combinations thereof.
- the present disclosure provides a method of scanning for users to use in their homes using a structural light sensor often found in the front-facing camera in modern mobile phones.
- the scanning method utilizes a mobile phone’s built-in facial recognition system to take multiple scans of a mother’ s breast by stitching individual scans of the mother’s breast, taken by the mobile phone, to form a sparse 3D point cloud.
- the scanning method also includes recognizing the mother’s breast in 3D space and detecting a frontal image of the mother’s nipple to use for color matching.
- the scanning method includes extracting key measurements and contours of the mother’ s nipple scan to direct the mother to the most appropriate nipple product for her unique body.
- the term “or” is an inclusive “or” operator, and is equivalent to the term “and/or,” unless the context clearly dictates otherwise.
- the term “based on” is not exclusive and allows for being based on additional factors not described, unless the context clearly dictates otherwise.
- the meaning of “a,” “an,” and “the” include plural references.
- the meaning of “in” includes “in” and “on.”
- the term “real-time” is directed to an event/action that can occur instantaneously or almost instantaneously in time when another event/action has occurred.
- the “real-time processing,” “real-time computation,” and “real-time execution” all pertain to the performance of a computation during the actual time that the related physical process (e.g., a user interacting with an application on a mobile device) occurs, in order that results of the computation can be used in guiding the physical process.
- events and/or actions in accordance with the present invention can be in real-time and/or based on a predetermined periodicity of at least one of: nanosecond, several nanoseconds, millisecond, several milliseconds, second, several seconds, minute, several minutes, hourly, several hours, daily, several days, weekly, monthly, etc.
- the inventive specially programmed computing systems with associated devices are configured to operate in the distributed network environment, communicating over a suitable data communication network (e.g., the Internet, etc.) and utilizing at least one suitable data communication protocol (e.g., IPX/SPX, X.25, AX.25, AppleTalkTM, TCP/IP (e.g., HTTP), etc.).
- a suitable data communication network e.g., the Internet, etc.
- at least one suitable data communication protocol e.g., IPX/SPX, X.25, AX.25, AppleTalkTM, TCP/IP (e.g., HTTP), etc.
- image(s) and “image data” are used interchangeably to identify data representative of visual content which includes, but not limited to, images encoded in various computer formats (e.g., “.jpg”, “.bmp,” etc.), streaming video based on various protocols (e.g., Real-time Streaming Protocol (RTSP), Real-time Transport Protocol (RTP), Real-time Transport Control Protocol (RTCP), etc.), recorded/generated non-streaming video of various formats (e.g., “,mov,” “.mpg,” “.wmv,” “.avi,” “.flv,” ect.), and real-time visual imagery acquired through a camera application on a mobile device.
- RTSP Real-time Streaming Protocol
- RTP Real-time Transport Protocol
- RTCP Real-time Transport Control Protocol
- recorded/generated non-streaming video of various formats e.g., “,mov,” “.mpg,” “.wmv,” “.avi,” “.flv,” ect.
- real-time visual imagery acquired
- server should be understood to refer to a service point which provides processing, database, and communication facilities.
- server can refer to a single, physical processor with associated communications and data storage and database facilities, or it can refer to a networked or clustered complex of processors and associated network and storage devices, as well as operating software and one or more database systems and application software that support the services provided by the server. Cloud servers are examples.
- a machine-readable medium may include any medium and/or mechanism for storing or transmitting information in a form readable by a machine (e.g., a computing device).
- a machine-readable medium may include read only memory (ROM); random access memory (RAM); magnetic disk storage media; optical storage media; flash memory devices; electrical, optical, acoustical or other forms of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.), and others.
- a non-transitory article such as a non-transitory computer readable medium, may be used with any of the examples mentioned above or other examples except that it does not include a transitory signal per se. It does include those elements other than a signal per se that may hold data temporarily in a “transitory” fashion such as RAM and so forth.
- Examples of hardware elements may include processors, microprocessors, circuits, circuit elements (e.g., transistors, resistors, capacitors, inductors, and so forth), integrated circuits, application specific integrated circuits (ASIC), programmable logic devices (PLD), digital signal processors (DSP), field programmable gate array (FPGA), logic gates, registers, semiconductor device, chips, microchips, chip sets, and so forth.
- the one or more processors may be implemented as a Complex Instruction Set Computer (CISC) or Reduced Instruction Set Computer (RISC) processors; x86 instruction set compatible processors, multicore, or any other microprocessor or central processing unit (CPU).
- the one or more processors may be dual-core processor(s), dual-core mobile processor(s), and so forth.
- Determining whether an embodiment is implemented using hardware elements and/or software elements may vary in accordance with any number of factors, such as desired computational rate, power levels, heat tolerances, processing cycle budget, input data rates, output data rates, memory resources, data bus speeds and other design or performance constraints.
- One or more aspects of at least one embodiment may be implemented by representative instructions stored on a machine-readable medium which represents various logic within the processor, which when read by a machine causes the machine to fabricate logic to perform the techniques described herein.
- Such representations known as “IP cores” may be stored on a tangible, machine readable medium and supplied to various customers or manufacturing facilities to load into the fabrication machines that actually make the logic or processor.
- FIGS. 1 through 17 illustrate exemplary systems and methods for scanning and replicating the shape of a mother’s nipple such that the replicated nipple may be integrated with a baby bottle.
- At least one technical difficulty is the efficiency of computing system in extracting from images, e.g., pixels, useful visual data that can be utilized to replicate a mother’s nipple.
- the present disclosure provides a technically advantageous computer architecture that improves scan images of a mother’s breast, based at least in part on scan image data of other users (i.e., other breastfeeding mothers), to create a more realistic and lifelike baby bottle nipple that mimics the nipple of the breastfeeding mother, thereby reducing nipple confusion by the baby.
- the systems and methods are technologically improved by being programmed with machine-learning modeling to create a 3D scan image.
- Some embodiments leverage the wide-spread use of mobile personal communication devices (e.g., smart phones with integrated cameras) to facilitate the inputting of user-generated data to enhance the 3D scan image.
- FIG 1 illustrates a block diagram illustration of an exemplary nipple scanning system 100 consistent with some embodiments of the present disclosure.
- the nipple scanning system 100 may include a server 106 in communication with a first user computing device 104 of a first user 102 and a second user computing device 105 of a second user 103 via a network 108.
- Network 108 may be of any suitable type, including individual connections via the internet such as cellular or Wi-Fi networks.
- network 108 may connect participating devices using direct connections such as radio-frequency identification (RFID), near-field communication (NFC), BluetoothTM, low-energy BluetoothTM (BLE), Wi-FiTM, ZigBeeTM ambient backscatter communications (ABC) protocols, USB, WAN or LAN.
- RFID radio-frequency identification
- NFC near-field communication
- BLE low-energy BluetoothTM
- Wi-FiTM ZigBeeTM ambient backscatter communications
- USB WAN or LAN
- Server 106 may be associated with a medical practice or other type of practice or entity.
- server 106 may manage user information.
- server 106 may include one or more logically or physically distinct systems.
- the server 106 may include hardware components such as a processor (not shown), which may execute instructions that may reside in local memory and/or transmitted remotely.
- the processor may include any type of data processing capacity, such as a hardware logic circuit, for example, an application specific integrated circuit (ASIC) and a programmable logic, or such as a computing device, for example a microcomputer or microcontroller that includes a programmable microprocessor.
- ASIC application specific integrated circuit
- the first user computing device 104 may be associated with the first user 102 who is a breastfeeding mother.
- the second user computing device 105 may be associated with the second user 103 who is an entity, such as a medical practice or medical products company.
- the server 106 may prompt the first user 102 to input user information and a scan image via the first user computing device 104.
- the first user computing device 104 and/or the second user computing device 105 may be a mobile computing device.
- the first user computing device 104 and/or the second user computing device 105, or mobile user devices may generally include at least computer-readable non-transient medium, a processing component, an Input/Output (I/O) subsystem and wireless circuitry. These components may be coupled by one or more communication buses or signal lines.
- I/O Input/Output
- the first user computing device 104 and/or the second user computing device 105 may be any portable electronic devices, including a mobile phone, a handheld computer, a tablet computer, a laptop computer, a tablet device, a multifunction device, a portable gaming device, a vehicle display device, or the like, including a combination of two or more of these items.
- the mobile user device may be any appropriate device capable of taking still images or video with an equipped front camera.
- the first user computing device 104 and/or the second user computing device 105 may be a desktop computer.
- the first user computing device 104 includes a user camera 110.
- At least one user image may be captured by the user camera 110 and transmitted via network 108.
- the at least one scan image capture may be performed by a nipple scanning application 130 available to all users of the first user computing device 104.
- the at least one scan image capture may be performed by a camera application that comes with a mobile first user computing device 104, and the resulting at least one scan image may be uploaded to the nipple scanning application 130.
- wireless circuitry is used to send and receive information over a wireless link or network to one or more other devices' suitable circuitry such as an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a CODEC chipset, memory, etc.
- suitable circuitry such as an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a CODEC chipset, memory, etc.
- the wireless circuitry can use various protocols, e.g., as described herein.
- first user computing device 104 and/or the second user computing device 105 can have more or fewer components than shown, or a different configuration of components.
- the various components described above can be implemented in hardware, software, or a combination of both hardware and software, including one or more signal processing and/or application specific integrated circuits.
- the first user computing device 104 may include an application such as the nipple scanning application 130 (or application software) which may include program code (or a set of instructions) that performs various operations (or methods, functions, processes, etc.), as further described herein.
- an application such as the nipple scanning application 130 (or application software) which may include program code (or a set of instructions) that performs various operations (or methods, functions, processes, etc.), as further described herein.
- Examples of software may include software components, programs, applications, computer programs, application programs, system programs, machine programs, operating system software, middleware, firmware, software modules, routines, subroutines, functions, methods, procedures, software interfaces, application program interfaces (API), instruction sets, computing code, computer code, code segments, computer code segments, words, values, symbols, or any combination thereof. Determining whether an embodiment is implemented using hardware elements and/or software elements may vary in accordance with any number of factors, such as desired computational rate, power levels, heat tolerances, processing cycle budget, input data rates, output data rates, memory resources, data bus speeds and other design or performance constraints.
- the nipple scanning application 130 enables the first user 102 to upload a scan image to the server 106.
- the nipple scanning application 130 may be an application provided by a medical entity or other entity.
- the nipple scanning application 130 may be automatically installed onto the first user computing device 104 after being downloaded.
- the nipple scanning application 130 or a component thereof may reside (at least partially) on a remote system (e.g., server 106) with the various components (e.g., front-end components of the nipple scanning application 130) residing on the first user computing device 104.
- the nipple scanning application 130 and the server 106 may perform operations (or methods, functions, processes, etc.) that may require access to one or more peripherals and/or modules.
- the server 106 includes a scan optimization module 120 and an algorithm module 140, as will be described in further detail below.
- the scan image 112 may be processed by the scan optimization module 120, which is specifically programmed in accordance with the principles of the present invention with one or more specialized inventive computer algorithms. Further, in some embodiments, the scan optimization module 120 can be in operational communication (e.g., wireless/wired communication) with the server 106 which can be configured to support one or more functionalities of the scan optimization module 120.
- the scan optimization module 120 can be in operational communication (e.g., wireless/wired communication) with the server 106 which can be configured to support one or more functionalities of the scan optimization module 120.
- FIG 2 illustrates a flow diagram of an exemplary method 200 of creating a 3D nipple from a scan image, according to some embodiments of the present disclosure.
- the first user 102 places the first user computing device 104 approximately 1 to 2 feet underneath the user’s breast 117 with a front camera 110 of the first user computing device 104 framing the nipple 119 she wants to replicate.
- the first user 102 may also enter user information.
- the nipple scanning application 130 gathers and creates a full point cloud.
- the scan image 112 may be a video stream including a plurality of frames 114.
- an exemplary video stream captured by the user camera 110 e.g., a camera of a mobile phone
- each frame 114 may contain an image data with any known color model, including but not limited to: YCrCb, RGB, LAB, etc.
- the nipple scanning application 130 creates an RGB image sequence.
- the nipple scanning application 130 takes each frame 114 of the scan image 112 and stitches the frames 114 together to create the point cloud.
- the scan image 112 (e.g., input video stream) may include any appropriate type of source for video contents.
- the contents from the scan image 112 e.g., the scanning video of FIG. 4
- the scan image 112 is a scan of the user’s breast, captured by the front camera of the first user computing device 104.
- the scan optimization module 120 may recognize the nipple complex in the scan image 112.
- the scan optimization module 120 may be implemented as an application (or set of instructions) or software/hardware combination configured to perform operations (or methods, functions, processes, etc.) for receiving and processing image data inputs (e.g., without limitation, image(s), video(s), etc.), via the network 108, from the first user computing device 104.
- the scan optimization module 120 may receive the scan image 112 from the first user 102 and employ a machine learning engine 144 to identify the user’s nipple within the scan image 112.
- the machine learning engine 144 may include, e.g., software, hardware and/or a combination thereof.
- the machine learning engine 144 may include a processor and a memory, the memory having instructions stored thereon that cause the processor to generate, without limitation, at least one 3D image.
- the machine learning engine 144 may be configured to utilize a machine learning technique.
- the machine learning engine 144 may include one or more of a neural network, such as a feedforward neural network, radial basis function network, an image classifier, recurrent neural network, convolutional network, generative adversarial network, a fully connected neural network, or some combination thereof, for example.
- the machine learning engine 144 may be composed of a single level of linear or non-linear operations or may include multiple levels of non-linear operations.
- the machine learning engine 144 may include numerous layers and/or hidden layers that perform calculations (e.g., dot products) using various neurons.
- an exemplary implementation of Neural Network may be executed as follows: i) Define Neural Network architecture/model, ii) Transfer the input data to the exemplary neural network model, iii) Train the exemplary model incrementally, iv) determine the accuracy for a specific number of timesteps, v) apply the exemplary trained model to process the newly-received input data, vi) optionally and in parallel, continue to train the exemplary trained model with a predetermined periodicity.
- the scan optimization module 120 may employ object recognition techniques to identify a nipple within a scan image 112.
- the scan optimization module 120 may employ an object detection model.
- the object detection model may employ the machine learning engine 144 to recognize a nipple in the scan image 112.
- the machine learning engine 144 is a convolutional neural network that performs a convolution operation to recognize objects in images.
- a deep convolutional neural network (CNN) may be run to retrieve a feature vector, known as the encoder part.
- scan image data may be connected with the feature vector and nonlinear convolutional layers are run to identify an object in a scan image.
- the object detection model of the present disclosure includes a base architecture or series of “layers” it uses to process scan image data and return information about what is in the scan image 112.
- the object detection model is trained on a unique dataset to recognize information in an image.
- the dataset is gathered from multiple users taking images of their breasts, mostly along a specific path 115, as depicted in FIG. 3, moving from below the breast to just above the nipple.
- the machine learning engine 144 of the object detection model is trained on a set of scan images of previous users used in a wide variety of applications that contain a nipple to detect the nipple location in a two-dimensional image.
- the machine learning engine 144 is trained on hundreds of training scan images. In other embodiments, the machine learning engine 144 is trained on thousands of training scan images. In other embodiments, the machine learning engine 144 is trained on tens of thousands of training scan images. In other embodiments, the machine learning engine 144 is trained on hundreds of thousands of training scan images. [0079] In some embodiments, the machine learning engine 144 may identify a single key frame of the nipple for color, help ensure that the user’s scan image 112 is correctly seeing the nipple and isolate the nipple part of the point cloud for a genetic algorithm of the algorithm module 140.
- the machine learning engine 144 may output an output scan image 116 with one or more bounding boxes 118 (e.g. defined by a point, width, and height), and a class label for each bounding box 118, as depicted in FIG. 5.
- the trained machine learning engine 144 may detect at least one group of classes.
- the groups of classes may be: 1) a nipple complex including the areola and the teat; 2) only a teat; or 3) a frontal view of the entire nipple (for collecting information on color).
- step 215 in some embodiments, if the machine learning engine 144 may detect during scanning that too many of the scan image frames 114 do not contain a nipple, the application provides an error message to the user and asks the user to re-scan her nipple, starting the method again at step 210.
- the nipple scanning application 130 may transmit the output scan image 116 with the 3D bounding box 118 around the nipple, the colored .ply point cloud, and the series of color (RGB) images to the server 106 for a novel post-processing and texturizing procedure for nipple geometries.
- the process for collecting the output scan image is designed to be easy, the resulting output scan image may contain holes and noise due to the lower resolution of the scanner and the natural fluctuations of the scan image.
- a series of post-processing techniques may be used to fill in the holes within the output scan image 116, while crucially collecting important dimensional information about the nipple itself.
- the post-processing procedure shows an error message, the user may be given information as to why an error occurred and is asked to re-scan her nipple.
- Every nipple contains two important dimensions as it pertains to nipple confusion: base diameter and teat diameter.
- the base diameter is defined as the diameter around which the nipple meets the breast
- the teat diameter is defined as the diameter around which the nipple starts to curve downward or its inflection point, as depicted in FIG. 6.
- the postprocessing techniques described below replicate nipple geometry and extract base and teat diameter information for both tailored and customized baby bottle nipple solutions.
- FIGS. 7A-7E depicts each step of the post-processing method applied to an isolated nipple point cloud, depicted in FIG. 7A, according to embodiments of the present disclosure.
- the algorithm module 140 may use the genetic algorithm to perform various tasks, as will be described in further detail below.
- the genetic algorithm may automatically orient the isolated nipple point cloud upwards along the z-axis, as depicted in FIG. 7B.
- the algorithm module may also calculate the normals at each point.
- the algorithm module may gradually filter out those normals that are too far from the z-axis to create a gap in the scan that isolates the teat from the breast, as depicted in FIG. 7C.
- the genetic algorithm in some embodiments, may work in three steps, described below with reference to FIGS.
- nipple height 1) teat width; 3) nipple base width; 4) large scale textural roughness; 5) small scale textural roughness; or 6) nipple color.
- FIG 8 is a flow chart depicting the genetic algorithm method 300.
- the genetic algorithm may orient the isolated point cloud from the scan image 112 teat up (+z axis) using a series of genetic processes.
- the series of genetic processes are meant to minimize the cross-sectional area of the point cloud at the top, minimize the height of the overall scan and minimize the average normal at the heights part of the scan.
- the series of genetic processes may randomly mutate parameters through multiple iterations until a local minima is achieved.
- these genetic processes are different than machine learning in that they do not require training.
- three genetic processes may be used in the order described below.
- the first genetic process of the genetic algorithm may minimize the overall height of the scan by rotating the point cloud in the x and y axis and the ratio of the x and y dimensions (Xo and Yo) at the top quarter of the scan to the x and y dimension of the scan as a whole, as depicted in FIG. 9.
- the second genetic process of the genetic algorithm may set the average normal at the top portion of the scan as close to the +Z axis as possible, as depicted in FIG. 10.
- the third genetic process of the genetic algorithm may maximize a height at which the teat exceeds a predetermined cross-sectional diameter (Z c ), as depicted in FIG. 11.
- the predetermined cross-sectional diameter is 30 mm.
- the predetermined cross-sectional diameter is 25 mm to 35 mm; or 27 mm to 35 mm; or 29 mm to 35 mm; or 31 mm to 35 mm; or 33 mm to 35 mm; or 25 mm to 33 mm; or 25 mm to 31 mm; or 25 mm to 29 mm; or 25 mm to 27 mm; or 26 mm to 30 mm; or 29 mm to 30 mm; or 30 mm to 31 mm.
- each generation in the genetic process may score itself based on rotations along the x and y axes. In some embodiments, once the smallest possible score is achieved, the genetic algorithm may move on to the next genetic process until the scan is properly oriented. In some embodiments, if the genetic algorithm sees that there are too many holes around the edge of the teat, the genetic algorithm will ask the user to redo the scan image, making sure that the user moves slowly from underneath the breast to above the nipple.
- the algorithm module 140 may use the genetic algorithm to filter out the normal of each point in the point cloud further and further away from the +Z axis, until there is a clear separation between the teat and the nipple base, as depicted in FIG. 11.
- the normal of each point in the point cloud may be estimated based on the positions of the point’s nearest neighbors.
- the dot product of each normal is taken with the +Z axis ([0,0,1]).
- the genetic algorithm may then measure the width at the bottom of the teat and the top of the nipple base to get the teat width and the nipple base width, respectively.
- the genetic algorithm may then measure the distance between the +Z axis at the base and the +Z axis at the top of the nipple scan to get the nipple height, as depicted in FIG. 12. In some embodiments, if the genetic algorithm is unable to separate the nipple base and the teat, the genetic algorithm may indicate that the user should try to stimulate the nipple more. In some embodiments, if the genetic algorithm sees that the top of the scan is above a quarter of the width of the bottom of the scan, the genetic algorithm may provide an error and asks the user if she moved during the scan.
- the genetic algorithm may analyze the keyframe image extracted from the scan image 112 at the center of the nipple’s detected bounding box and the surround area of the teat’s bounding box to get information on texture and color, as depicted in FIG. 13. Specifically, in some embodiments, once the nipple scanning application 130 returns images of the breast during scanning and a front “key frame” image is identified, the trained machine learning engine 144 may identify the teat. In some embodiments, once the teat’s location is identified in a 2D frame, the average color of the pixels around the teat area may be used as the color value for that user.
- predetermined contours may be combined to rebuild the user’s nipple without any gaps or holes, as depicted in FIG. 7D.
- nipple scanning application 130 may extract key features and measurements from the scan images that are used to replicate the shape and size of the user’s nipple.
- the predetermined contours may include: a predetermined contour along the breast, a predetermined contour at the base of the nipple, a predetermined contour at the teat and a predetermined contour around the nipple apex.
- a marching cubes function may be used to create a solid watertight mesh, as depicted in FIGS. 7E.
- the RGB frames provided by the nipple scanning application 130 through the scanning process may be used to generate a texture of the scanned nipple.
- the object detection model may determine which frame 114 of the scan image 112 is aligned with the majority of the nipple and areola (frontal view).
- the frame 114 may be isolated and used to measure “roughness” across the areola’s and nipple’s surfaces utilizing at least one gray scale image analysis tool.
- a sample output graphs of a roughness algorithm are depicted in FIG. 14.
- At least one gray scale image analysis tool may be used at two different resolutions to determine large scale roughness values and fine scale roughness values. Examples of fine scale analysis and rough scale analysis are depicted in FIG. 15, according to embodiments of the present disclosure. In some embodiments these roughness values may be used to generate intensity maps, indicating where roughness/bumpiness is strongest, and where roughness/bumpiness is weakest, based on white and black values, respectively.
- FIG. 16 depicts an exemplary smooth intensity map and a rough intensity map, according to embodiments of the present disclosure.
- the intensity maps may be plugged into a novel procedural texture based on multiple voronoi algorithms built in Blender® to create a custom displacement map that estimates the skin texture roughness across the user’s nipple, as depicted in FIG. 17.
- the nipple’s 3D custom shape may be translated into a 3D baby bottle nipple profile that fits with a standard baby bottle.
- the 3D baby bottle nipple while having the shape of the user’s nipple, may be size and formatted to fit to a standard commercial baby bottle.
- the baby bottle nipple profile may include a base portion that may connect with a standard baby bottle via a baby bottle collar.
- the baby bottle nipple profile may include a base portion that connects directly to a standard baby bottle.
- the custom 3D baby bottle nipple profile is sent to the server 106.
- the standard 3D nipple profile is accessible on the server 106 by the second user computing device 105 for 3D printing.
- a custom baby bottle nipple based on the custom 3D baby bottle nipple profile, is 3D printed.
- the custom baby bottle nipple may be integrated into a standard manufactured baby bottle part.
- the result is a baby bottle capped with a 3D printed nipple designed to mimic the geometry and feel of the user’s nipple.
- the entire process sits in three different “locations.”
- the initial process of scanning the nipple and identifying its location in 3D space exists on the user’s phone - the first location.
- the phone application produces a colored scan, eight points defining the nipple’s location and a series of RGB images that are sent to a virtual machine for post-processing and texturing using the Genysis® API - the second location.
- the Genysis® API then sends the resulting key dimensions, solid watertight printable mesh .stl file, a single color value and any error information to a server for 3D printing and archiving - the third location.
- a method including: receiving, by a processor, from a camera of a mobile device, a plurality of scanned images; applying, by the processor, an object detection model of a machine learning engine to the plurality of scanned images to identify a user’s nipple within the plurality of scanned images based on a plurality of object feature vectors; generating, by the processor, an scan image output comprising a plurality of object feature vectors identifying the user’s nipple; applying, by the processor, a genetic algorithm to extract a plurality of geometric features from the scan image output; where the plurality of geometric features identifies: nipple-related height, nipple-related width, nipple-related texture, and nipple-related color; generating, by the processor, a 3D model of the user’s nipple based on the plurality of geometric features; and determining, by the processor, based on the 3D model of the user’s nipppp
- a third genetic process of the genetic algorithm comprises maximizing, by the processor, a height at which the teat exceeds a predetermined cross- sectional diameter.
- 11 The method of clause 10, where the predetermined cross-sectional diameter is 30 mm.
- 12 The method of clause 8, where the genetic algorithm is configured to filter out a normal of each point in the point cloud further and further away from the positive z axis, until there is a clear separation between the teat and a base of the user’s nipple.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Chemical & Material Sciences (AREA)
- Health & Medical Sciences (AREA)
- Geometry (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Computer Graphics (AREA)
- Manufacturing & Machinery (AREA)
- Materials Engineering (AREA)
- Software Systems (AREA)
- Life Sciences & Earth Sciences (AREA)
- Animal Behavior & Ethology (AREA)
- General Health & Medical Sciences (AREA)
- Public Health (AREA)
- Veterinary Medicine (AREA)
- Image Processing (AREA)
Abstract
Description
Claims
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
AU2022424120A AU2022424120A1 (en) | 2022-01-03 | 2022-12-30 | Method for nipple replication |
IL314116A IL314116A (en) | 2022-01-03 | 2022-12-30 | Method for nipple replication |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US202263296008P | 2022-01-03 | 2022-01-03 | |
US63/296,008 | 2022-01-03 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2023130067A1 true WO2023130067A1 (en) | 2023-07-06 |
Family
ID=87000355
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/US2022/082618 WO2023130067A1 (en) | 2022-01-03 | 2022-12-30 | Method for nipple replication |
Country Status (3)
Country | Link |
---|---|
AU (1) | AU2022424120A1 (en) |
IL (1) | IL314116A (en) |
WO (1) | WO2023130067A1 (en) |
Citations (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20170312185A1 (en) * | 2016-05-02 | 2017-11-02 | Denslojac, Llc | Custom-made artificial nipple |
US20180276841A1 (en) * | 2017-03-23 | 2018-09-27 | Intel Corporation | Method and system of determining object positions for image processing using wireless network angle of transmission |
US20190042871A1 (en) * | 2018-03-05 | 2019-02-07 | Intel Corporation | Method and system of reflection suppression for image processing |
WO2020200087A1 (en) * | 2019-03-29 | 2020-10-08 | Ai Technologies Inc. | Image-based detection of ophthalmic and systemic diseases |
WO2020219856A1 (en) * | 2019-04-26 | 2020-10-29 | University Of Houston System | Systems and methods for modeling the breast using spherical harmonics |
US20210342836A1 (en) * | 2018-05-06 | 2021-11-04 | Strong Force TX Portfolio 2018, LLC | Systems and methods for controlling rights related to digital knowledge |
US20220062111A1 (en) * | 2020-08-31 | 2022-03-03 | Proxamama LLC | Method and product for a customized 3d printed nipple for breastfeeding mothers |
US20230069584A1 (en) * | 2021-08-30 | 2023-03-02 | Proxamama LLC | Method for nipple replication |
-
2022
- 2022-12-30 IL IL314116A patent/IL314116A/en unknown
- 2022-12-30 WO PCT/US2022/082618 patent/WO2023130067A1/en active Application Filing
- 2022-12-30 AU AU2022424120A patent/AU2022424120A1/en active Pending
Patent Citations (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20170312185A1 (en) * | 2016-05-02 | 2017-11-02 | Denslojac, Llc | Custom-made artificial nipple |
US20180276841A1 (en) * | 2017-03-23 | 2018-09-27 | Intel Corporation | Method and system of determining object positions for image processing using wireless network angle of transmission |
US20190042871A1 (en) * | 2018-03-05 | 2019-02-07 | Intel Corporation | Method and system of reflection suppression for image processing |
US20210342836A1 (en) * | 2018-05-06 | 2021-11-04 | Strong Force TX Portfolio 2018, LLC | Systems and methods for controlling rights related to digital knowledge |
WO2020200087A1 (en) * | 2019-03-29 | 2020-10-08 | Ai Technologies Inc. | Image-based detection of ophthalmic and systemic diseases |
WO2020219856A1 (en) * | 2019-04-26 | 2020-10-29 | University Of Houston System | Systems and methods for modeling the breast using spherical harmonics |
US20220062111A1 (en) * | 2020-08-31 | 2022-03-03 | Proxamama LLC | Method and product for a customized 3d printed nipple for breastfeeding mothers |
US20230069584A1 (en) * | 2021-08-30 | 2023-03-02 | Proxamama LLC | Method for nipple replication |
Non-Patent Citations (2)
Title |
---|
MATKOWSKI WOJCIECH MICHAL; MATKOWSKI KRZYSZTOF; KONG ADAMS WAI-KIN; LLOYD HALL CORY: "The Nipple-Areola Complex for Criminal Identification", 2019 INTERNATIONAL CONFERENCE ON BIOMETRICS (ICB), IEEE, 4 June 2019 (2019-06-04), pages 1 - 6, XP033709055, DOI: 10.1109/ICB45273.2019.8987341 * |
XU ET AL.: "3D Joints Estimation of the Human Body in Single-Frame Point Cloud", IEEE ACCESS, 30 September 2020 (2020-09-30), pages 178900 - 178908, XP011813042, Retrieved from the Internet <URL:https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9210042> [retrieved on 20230315], DOI: 10.1109/ACCESS.2020.3027892 * |
Also Published As
Publication number | Publication date |
---|---|
AU2022424120A1 (en) | 2024-07-25 |
IL314116A (en) | 2024-09-01 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN114514562B (en) | Predictive personalized three-dimensional body model | |
US11756291B2 (en) | Scene and user-input context aided visual search | |
Lo et al. | Image-based food classification and volume estimation for dietary assessment: A review | |
US11436437B2 (en) | Three-dimension (3D) assisted personalized home object detection | |
US20210092286A1 (en) | Augmented Reality for Three-Dimensional Model Reconstruction | |
US11514642B2 (en) | Method and apparatus for generating two-dimensional image data describing a three-dimensional image | |
US20240161399A1 (en) | Systems and methods for baby bottle manufacturing | |
US20180260669A1 (en) | Image processing apparatus, image processing method, template generation apparatus, object recognition processor, and object recognition processing program | |
WO2019040123A1 (en) | Object separation in three-dimensional camera images | |
CN107430776A (en) | Template construct device and template construct method | |
US11861860B2 (en) | Body dimensions from two-dimensional body images | |
CN114511931A (en) | Action recognition method, device and equipment based on video image and storage medium | |
US20220414375A1 (en) | Image cropping using depth information | |
US20220414900A1 (en) | Item identification using multiple cameras | |
Nesteruk et al. | PseudoAugment: Enabling Smart Checkout Adoption for New Classes Without Human Annotation | |
WO2023130067A1 (en) | Method for nipple replication | |
Parashar et al. | A robust covariate‐invariant gait recognition based on pose features | |
Akhund et al. | Iot based low-cost posture and bluetooth controlled robot for disabled and virus affected people | |
Cruz-Silva et al. | 3-D Human Body Posture Reconstruction by Computer Vision | |
US20240320392A1 (en) | Systems and methods for three-dimensional body part modelling | |
Arnarson et al. | Towards automatic generation of image recognition models for industrial robot arms | |
Figueiredo et al. | Shape-based attention for identification and localization of cylindrical objects | |
Burić et al. | An overview of action recognition in videos | |
Dubey et al. | Survey on food intake methods using visual technologies | |
Petkovic et al. | Machine vision in the 1990s: applications and how to get there |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 22917601 Country of ref document: EP Kind code of ref document: A1 |
|
WWE | Wipo information: entry into national phase |
Ref document number: 314116 Country of ref document: IL |
|
WWE | Wipo information: entry into national phase |
Ref document number: AU2022424120 Country of ref document: AU |
|
ENP | Entry into the national phase |
Ref document number: 2022424120 Country of ref document: AU Date of ref document: 20221230 Kind code of ref document: A |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2022917601 Country of ref document: EP |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
ENP | Entry into the national phase |
Ref document number: 2022917601 Country of ref document: EP Effective date: 20240805 |