EP4689980A1

EP4689980A1 - Automatic design of custom ear devices

Info

Publication number: EP4689980A1
Application number: EP24715151.7A
Authority: EP
Inventors: Søren Hegelund MAAGAARD; Rasmus KONGSKOV; Jakob MØLLER-ANDERSEN
Original assignee: 3Shape AS
Current assignee: 3Shape AS
Priority date: 2023-03-31
Filing date: 2024-03-26
Publication date: 2026-02-11
Also published as: CN121079687A; WO2024200441A1

Abstract

Described herein is a computer-implemented method for constructing in the form of designing in a virtual environment a custom ear device in an automated process utilizing a trained neural network. The computer-implemented method described is configured to allow for changing the output form the neural network to create change to an automatic generated custom ear device design. Furthermore, described herein is a system utilizing a processor running the method described herein to automatically generate virtually constructed custom ear devices that may be modified to adjust for any incorrect outputs generated by a neural network.

Description

AUTOMATIC DESIGN OF CUSTOM EAR DEVICES

FIELD

The disclosure relates to a method for automatically designing custom ear devices and a system comprising a design application configured to perform the method.

BACKGROUND

To digitally design custom ear devices, such as hearing aids, in-ear monitors, noise protection devices, custom headphones etc. a digital representation of a user's ear is obtained to ensure a design of the custom ear device that fits exactly to the shape of the ear canal of the user wearing the custom ear device. To ensure a substantially correct anatomical fit of the custom ear device to the user’s ear (including the ear canal, and the inner and outer parts of the ear, an impression is normally taken of the user’s ear. From the physical impression a digital representation of the user’s ear shape can be generated, and the shape of the final custom device be decided upon. The impression is configured as a physical, e.g., silicone, impression which is used for producing the custom ear device. The standard procedure for producing custom ear devices comprises a plurality of steps including, acquiring an impression of the users ear, selecting the device properties (device configuration requirements) desired by the user and suitable for covering e.g. a hearing loss, shipping the physical impression to a design service site being e.g. at a manufacturer and/or a design modeler, and subsequently digitally manufacturing the device at a manufacturing site using the custom ear device as designed in a design service software application.

During or after acquiring of the physical impression, the user is to select the desired device properties (also denoted throughout as device configuration requirements or merely configuration requirements), which in case of the custom ear device being a hearing aid includes a hearing test to ensure correct hearing aid properties of the custom ear device. The correct property for the desired device is often decided upon together with an audiologist and/or at a retail store, where the hearing test is performed. The custom ear device may be designed as a none-hearing aid device, where a hearing test may not be part of the configuration requirements. In any case, whether the custom ear device being configured as a hearing aid or a none-hearing aid, other device properties, such as noise cancellation requirements, fit etc. may form part of the desired device properties. The device properties may include in a non-limiting order size, telecoil properties, faceplate system, integration or non-integration of faceplate, wax guard, transducer, vent, shape, sound bore etc. The chosen device properties by the user may be stored in a device property file in a database which may be transferred to a design service software application and/or to a manufacture of the custom ear device.

At the manufacturing site and/or at a clinical site and/or at a retail site, the physical impression is converted into e.g., a digital file by obtaining a digital scan of the physical impression. Such digital scans are often obtained using a 3D scanner. Other methods may skip the impression step and directly scan the ear using a specialized scanning device or even using depth sensors and/or camera on modern smartphones may be used for obtaining such a scan. In any case, when the digital scan of the ear of a patient have been obtained, the desired device properties and the digital scan is used to manufacture a custom ear device which fits accurately to the anatomy of the user’ s ear and complies with the device properties as desired by the user.

The general procedure of manufacturing custom ear devices based on the digital scan and the device properties relies on trained professionals (also denoted modelers) using specialized design service software to design the final custom device. Accordingly, the manufacturing of the custom ear device requires a design process which is supported by specialized software allowing the trained professionals to design the outer shapes, inner structures, component placement and arrangement and capabilities of the custom ear device in a manner complying with the desired device properties of the user and the anatomy of the ear as provided by the digital impression scan. The trained professionals generally use their experience and in-depth knowledge of the ears anatomy to learn how to effectively design custom ear device as the ones described herein. After the trained professionals have designed the custom ear device using the digital impression scan, the custom ear device is 3D printed, assembled, and shipped back to the user.

These known manual methods of designing custom ear devices require extensive experience and knowledge from the trained professional in order to ensure an optimal design of the custom ear device that fits the user accurately. Some of the critical aspects in the design process that are taken care of purely by the extensive knowledge from the trained professionals is among other things to reduce fitting errors (e.g. that the device fall out of the patients ear because of missing retention, or the device hurts because of too much pressure against certain parts of the ear geometry, etc.), and to increase success in the following production steps (printing, assembly, QA sound check, etc.). Many of the steps needed to be performed by a trained professional in the design process to fulfill the requirements of fitting and production success are not written down, or the written instructions are vague, are up for interpretation or wrong, and so the actual design process is implicit know-how built up by the modelers over time. Generally, all the skill and tacit knowledge used during modelling that affects the final output lies with the trained professionals and no standard rules are normally applied. Such lack of knowledge storage and rule scheme used in the design process challenges the reproducibility of the design process if/when needed to design a replica of a previously designed custom ear device. Today reproducibility is ensured for by saving one or more of the design steps (also denoted operations throughout) that a trained professional manually performs during the design process.

Solutions providing an automatic end-to-end design process have been suggested, where a design service software application is configured to design automatically a final custom ear device on the basis of given device properties for specific ear impression scans. However, these end-to-end solutions does not allow for any consistent reproducible changes to be made to the final custom ear device as the automatic design applications performs a method directly generating a final custom ear device. This has the drawback that the final design cannot be changed, why there still exists a need for automating the manually design process, while allowing highly trained professionals and/or other specialists to adjust any automatically designed custom ear device, in case the final output does not comply in full with the device properties given. The automatic design process providing the option of modifying the output, should preferably maintain and utilize the extensive knowledge gained from professional experience and extensive knowledge about how to effectively and accurately design a custom ear device in a plurality of manual design steps including one or more operations which in standard approaches may only be performed by the highly skilled professionals to ensure reproducibility of the device.

SUMMARY

The present disclosure addresses the above-mentioned challenges by providing a computer- implemented method configured for constructing a custom ear device by loading into a processor an ear impression data. Further loading into the processor one or more configuration requirements (also denoted device properties throughout the description) for designing a custom ear device matching the ear impression data and receiving a user instruction causing the processor to load the ear impression data and the one or more configuration requirements into a neural network (NN). The neural network may be configured to output at least one partially constructed representation of a custom ear device comprising at least a set of operations as applied by the neural network and complying with at least a part of the one or more configuration requirements. The at least one partially constructed representation may in addition to the set of operations comprise at least one partially constructed 3D model of a custom ear device complying with at least a part of the one or more configuration requirements. By utilizing such a neural network that is configured to output at least one partially constructed representation (in an example a 3D model) of a custom ear device represented by a set of operations, it is possible to gain access to the automatic modelling steps (i.e. the set of operations) as performed by the neural network instead of only getting a final custom device which may not be possible to change. Accordingly, with the presented method the possibility of receiving a user instruction causing the processor to perform changes to the partially constructed 3D model in accordance with the user instruction, and to output a finally constructed ear custom device and a final operations datafile, is provided for. This allows a user applying e.g., a design application configured to run the method described herein, to gain access to one or more partial designs as generated during the automatic design process (as performed by the neural network) of the custom ear device and to choose a partial design which the user might not be satisfied with as it might not fulfill the given configuration requirements. The user may then perform changes to the partial design and thereby create a new partial and/or final custom ear device, which fulfill the given configuration requirements. The methods described herein is generally considered to result in a final operations datafile comprising production instructions configured with 3D printer instructions allowing a 3D printer to manufacture the custom ear device as described by the final operations datafile. Especially the 3D printing of the shell of the custom ear device is described in the operations datafile, which can be transmitted to a 3D printer for actual physical 3D printing of the custom ear device. With the method described herein, the custom shell to be 3D printed is automatically designed using the output from the neural network and is thus considered to be designed in an operator independent fashion, which optimizes time spent on designing custom hearing aids by the manual process used today. With the method described herein, the expert designing the custom hearing aid, especially the shell with tip cuts, bottom cuts, splines, CAD component placement etc. need not to spent time on manually placing these things, as the neural network is trained to predict these placements, thereby generating an automatic process allowing optimization of time on a day-to-day basis when producing custom hearing aids.

The wording “constructing” used herein should be understood as to virtually design in e.g. a design software running on a computer, the custom ear device. The “constructing” merely refers to the operations used to virtually design, as will be apparent throughout the description.

It is noted that the output from the neural network comprises at least a partially constructed representation of the custom ear device, where the representation may be a set of operations as predicted by the neural network. The set of operations as will be more specifically explained throughout is considered as geometrical transformations for a point, pixel etc. forming the input data, that is output from the neural network. The geometrical transformation predictions, may be one of a plane, spline, transformation matrix etc. that can be used in a subsequent post-processing step, to apply the output from the neural network to the input ear impression data and thereby create a partial 3D representation of the input ear impression data. It is noted that the neural network may be considered as a neural network structure comprises a plurality of different target operations, wherein the method comprises a step of training each of the plurality of neural networks on the basis of training data sets comprising: a plurality of training ear impression data wherein the training ear impression data is acquired by an optical scanner; a plurality of target partial constructed representations of custom ear devices each complying with a plurality of target configuration requirements; and one or more target operations specific to the plurality of target partially constructed representations and complying with at least a part of the one or more target configuration requirements. Using a neural network structure that is trained on different target operations, it is especially possible to output each partial prediction independently of each other, thereby allowing an independent assessment of each target operation, which supports the possibility of changing a single target operation, in case a user of the system do not agree with the neural network predictions. It should be noted that when talking about a neural network in the following, it may be considered to include one or more different neural networks each trained on different target operations.

Therefore, to provide a neural network (NN) that is able to output at least one partial representation of a custom ear device, the neural network (NN) may be trained on the basis of a training data set comprising: a plurality of training ear impression data and a plurality of target partial and/or fully constructed representations (such as 3D models) of custom ear devices each complying with a plurality of target configuration requirements. Further, the neural network may be trained on one or more target operations specific to the plurality of target partially constructed representations (such as 3D models) and complying with at least a part of the one or more target configuration requirements. It may also be that an end-to- end neural network is applied, where one single network is trained on all target operations, whereby the neural network is configured to predict all target operations at once. However, it is preferred to provide a plurality of neural networks each ensuring at least a single target operation prediction, to allow visualization thereof and the possibility of changing a single operation at a time in accordance with a user input. In this way it is ensured that the neural network is trained on a training data set which comprises the ground truth data (i.e., the ground truth data being denoted “target” herein and used for the training of the neural network). Using the ground truth data, the training of the neural network is provided with the optimal settings to learn modeler operations (i.e., the one or more target operations) sufficiently good to apply such knowledge to an unknown ear impression data and an unknown configurations requirement. In this way, the manually acquired modeler experience and extensive knowledge as explained in the background is with this method instead learned by a neural network, which in the training process of the neural network acquires the skill sets used by trained professional modelers to utilize the intermediate design steps (i.e. target operations) to generate partially and/or fully constructed representations (such as 3D models) of custom ear devices when given a configuration requirement and an impression data.

Training the neural network on the intermediate steps (i.e. the set of target operations) manually performed by a trained professional to obtain a partially constructed representation (such as a 3D model) of a custom ear device, allows the neural network to output a set of operations that may be automatically applied to the ear impression data when designing a partially constructed representation, such as a 3D model, as well as the fully constructed 3D model. The set of operations may be output from the neural network so as to mimic to a modeler (the previous mentioned trained professional) the steps needed to reach a certain partially constructed 3D model. The set of operations may be reflected as metadata (described later) and/or as design representations visualized on a 3D model of the partially generated 3D model of the custom ear device. This in the end allows a trained professional or other user of the system executing the method (such as a design software installed on a computer) to evaluate the operations applied by the neural network in constructing a partially 3D model of the custom ear device and adjust the operations if the set of operations performed by the neural network in one or more operations steps of the constructed partially constructed 3D models was wrong, inaccurate and/or unsatisfactory.

In an embodiment the set of operations as output from the neural network, may be as set of operations that should be applied to e.g., the ear impression data. That is, the neural network may automatically apply the set of operations to the ear impression data which the neural network takes as input. “Applying” is considered to mean that the neural network predicts the set of operations that as previously mentioned should be applied to the ear impression data, potentially at a sub-sequent post-processing step. Alternatively, the neural network may output an operation file, which contains information coding for the operations that should be applied to the ear impression data in e.g., a subsequent automatic step performed by e.g., an algorithmic procedure. That is, the operation file as output from the neural network may in a subsequent step be input to a further step of the method providing a modification procedure to the ear impression data in accordance with the instructions provided by the operation file. With this, the method may comprise outputting from the neural network an operation file containing information coding for the set of operations; and modifying the ear impression data in a modification procedure using the operation file. With this, the output from the NN may be an operation file comprising e.g. sets of points and normals defining e.g. planes, splines, CAD components positions etc. of the input ear impression data. In other words, the operation file may comprise the point or voxel predictions used to classify if a point or voxel for example belongs to a cutting plane, a CAD component position, etc. Several examples exist which will be elaborated on in relation to the Figures.

It should be appreciated throughout that the output from the neural network may be a representation of the partially and/or fully constructed custom ear device, where the representation at least comprises the one or more set of operations as generated by the neural network. The representation may further comprise a 3D model to which the set of operations is applied or may be subsequently applied. It will be apparent that the application describes this representation mainly by using the term 3D model and set of operations for the mere purpose of improving the understanding of the embodiments described herein. It is equally suitable and possible that the representation may not necessarily include the 3D model, but only comprises the set of operations. The set of operations is preferably considered to comprise geometrical transformations, wherein the method comprises applying the geometrical transformations to points, areas, facets or vertices of the ear impression data to form at least the partial constructed representation of the custom ear device. This preferably being done in a post-processing step using the set of operations (i.e. geometrical transformation data) as output from the neural network.

In the context of the application, the ear impression data may comprise a digital scan of an ear impression taken from a user of a custom ear device. The ear impression data may be obtained from a scanning process acquiring a scan of a physical ear impression of a user’s ear. That is the physical ear impression may be scanned by e.g., an optical scanner and subsequently transformed into a 3D digital representation to be used in the method of a design application as described herein. The ear impression data may comprise an impression mesh. The typical standard files for the ear impression data are one or more of STL, PLY file formats and/or HPS, DCM or TMD file formats. The file formats for which the ear impression data could form part of, could include any one of voxel models, point cloud models, neural radiance field representations and similar representations suitable for representing a 3D geometry. In accordance herewith the method may comprise converting the ear impression data into a file format configured as a voxel model or a point cloud representation.

In embodiments described herein, the method may further comprise generating by the processor a 3D model of the ear impression data. This allows easy visualization of the ear impression data and the partially and/or fully constructed 3D models based on the ear impression data. Furthermore, the generated 3D model of the ear impression data may be used as input to the neural network, and accordingly the training process may require to also generate 3D models of the target ear impression data.

The configuration requirements may comprise product desires such as device properties as defined by e.g., an audiologist together with a user of the custom ear device and/or the manufacturing site. The audiologist and user configuration requirements may contain the product desire for a chosen product type, whereas the manufacturing site configuration requirements, may comprise information concerning producibility of chosen product types. This means that one manufacture may require certain properties for the chosen product type to be producible, whereas other manufactures may require certain other properties for a chosen product type to be producible. In the context of this application, the configuration requirements may preferable be considered as the device properties defined by e.g., an audiologist together with a user of the chosen custom ear device type. Accordingly, the one or more configuration requirements comprise a specified product type and design requirements for the specified product type as required by an audiologist and/or by a manufacture of custom ear devices through the configuration data. Examples of product types may comprise in a none-limiting manner completely-in-the-canal (CIC) devices, invisible-in-the-canal (IIC) devices, in-the-canal (ITC) devices, half-shells (HS) devices, receiver-in-the-canal (RIC) devices, receiver-in-the-ear (RITE) devices, behind-the-ear (BTE) devices, in-ear monitors etc. Further examples of product types may be comprises of one or more of custom hearing aid devices as the ones previously mentioned, custom in-ear monitors, custom earphones, custom earmolds, custom ear shells, noise protection (for e.g., concerts, hunting, shooting, motor sports), custom sleep plugs, custom swim plugs, custom communication ear device (for example for use within the police, law enforcement and military), custom radio earpiece (for e.g. news casters and security guards), and other similar types of custom ear devices having similar properties as the mentioned which are to be designed based on a custom ear impression of a patient/user using such as device for different purposes. The product types, as also apparent from the later described figures, is different and comprises different design constrains and different component and shell construction. Therefore, to allow automatic design of each product type, a neural network structure as described herein is considered to be generated for each of the different product types. With this, the method and accordingly, the system described herein, comprises a further step of evaluating from the configuration requirements as input to a processor running the neural network, which specific neural network structure needs to be activated. In an example, if the product type is e.g. a receiver in the ear, the processor of the system is configured to read from the configuration file, that the product type corresponds to a first type product and load only the neural network structure which have been trained to automatically predict the set of operations associated with the first product type, e.g. the RITE-type. In this way, a plurality of different neural network structure, each trained on a plurality of different target set of operations associated with specific product types, are stored on the processor of the system.

The set of operations or one or more operations described herein may comprise the required steps to design the partially and/or fully constructed custom ear device. In other words, the set of one or more operations output from the neural network is considered to mimic the manually performed designs steps that a trained professional would have performed manually to design a partial custom ear device and/or the fully constructed custom ear device, but in the context of the automatic procedure described herein instead being performed by the neural network. It is noted that the “mimic” wording is not considered to be construed as to embodiments, where the neural network only applies a set of operations corresponding one-to-one with the step that a modeler manually would have done. The neural network is considered to be able to learn - on the basis of the described training herein - any operation (i.e. the set of operations) required to reach one or more partially constructed 3D models and/or the finally constructed 3D model in any none-ordered manner independent of how and in comparison to how a manual process by a modeler would have reached the partially and/or fully constructed representations (such as 3D models and set of operations). Accordingly, the one or more (set) of operations may as previously described form part of the output by the trained neural network.

In more detail, the set of operations may in an example be configured as e.g., a datafile (for example a metadata file) comprising one or more 3D data types for example a mesh, a point cloud, a spline or a plane and 3D interacting operations on one or more of these 3D data types, which could be moving, adding, removing, combining, cutting, connecting, annotating etc. This moving, adding, removing, combining, cutting, connecting, annotating, etc may be applied to the input ear impression data at a subsequent processing step to e.g. perform a moving, adding, removing, combining, cutting, connecting, annotating of points or voxels of the ear impression data. A subsequent visualization of the set of operations may be generated to allow a user of the system to accept or correct the operations predicted by the neural network.

In accordance with the above, the set of operations may comprise a datafile comprising one or more 3D data types and 3D interacting operations for one or more of the 3D data types. In other words, the datafile may for each 3D data type contained in the datafile provide geometrical transformations (i.e., 3D interacting operations) encoding a set of operations applied to the 3D data type.

Different types of 3D datatypes could also include group of vertices or facets on a mesh (such as the ear impression data) which is identified by the neural network to be modified in a set of operations as described herein. That is, in an example, a group of vertices or facets on a mesh (e.g., the ear impression data) could for example mark a region to be moved or removed. The neural network described herein would, based on the training, be configured to identify such movement operation, and encode that into the datafile of the set of operations. The moving operation could be performed by translating a series of points with point-specific translation values, and such movement operation may form part of the datafile of the set of operations. Similarly, the remove operation could be by removing points and edges from the surface and re-connecting new boundary edges. A spline could mark some part of one 3D object of the ear impression data, for example the center axis, that is about to be joined with, or cut out of, another 3D object of the ear impression data. A spline could mark the cut of a surface into two or more surfaces. The surface cut could be by splitting all surface elements along the spline lines projected to the surface elements. Two splines could mark surface boundaries to be connected by a new surface. A plane or a spline could mark a surface region to be removed or deleted. The surface region removal could for example be by removing all surface elements on a specified side of a plane. A spline could mark a surface region where the surface should be deformed, for example by smoothing the surface. A plane could mark a flat surface where parts of another surface could be moved toward. A mesh could be placed according to another mesh with or without an overlapping region. A point cloud could mark a specific surface shape. All of the active operations described, such as moving, removing, annotating, deleting, marking etc., may be considered as a geometrical transformation identified by the neural network as an operation to be applied to points, areas, facets, vertices etc. of the ear impression data to form the partial and/or fully constructed representation of the custom ear device.

All of the above-mentioned set of operations are considered to possibly form part of the datafile and accordingly considered to form part of the output of the neural network described herein. Accordingly, the set of operations may be considered to comprise a set of 3D interaction operations for a given point, group of points, facets and/or vertices on the ear impression data identified by the neural network as to be modified, wherein the 3D interaction operations comprise one or more geometrical transformations for each identified point, group of points, facets and/or vertices on the ear impression data. In one example covering the embodiments described herein the partially constructed 3D model may be a voxel model, in this case the set of operations may be the previous mentioned operations applied to one or more voxels of the 3D model.

The set of operations may comprise one of one or more set of decision rules and/or one or more production data, wherein each of the one or more decision rules and/or production data provides instructions for production of the partially and/or fully constructed 3D model of a custom ear device as output from the neural network.

In an embodiment, the one or more set of decision rules comprises data representing the set of operations applied by the neural network to construct the partially and/or finally constructed custom ear design to fit the anatomy of an ear canal of a user.

The decision rules may be configured to comprise at least one or more set of operations representing: shaping a tip of the ear impression data wherein the tip is configured to face the inside of an ear canal when inserted therein; cutting a part of the ear impression data configured to face the outside on an ear canal when inserted therein; shaping the outer shape of the ear impression data; placing of the electronic components; casting for a soft silicone production of the digital ear impression; placing of a sound bore; and/or placing of a vent canal in the partially and/or fully constructed custom ear device.

In an embodiment, the one or more production data comprises a plurality of production settings applied by the neural network to construct the partially and/or fully configured custom ear device.

The production settings may be configured to comprise one or more of a 3D representation of the electronic components to be arranged in the custom ear device, placement rules for the electronic components; and/or production settings defining device materials and sizes.

In accordance herewith, the training of the neural network is based on similar data as the previous described, ear impression data, configuration requirements, set of (or one or more) operations and partially and/or fully constructed 3D models of custom ear device. Again, as previously mentioned, the 3D model described herein, may simply be a data representation and not necessarily a generated model. The previous mentioned data is generally input or output to the trained neural network, which is able to provide the described output because it has been trained on similar data as will be described in the following. The data representation may be a point cloud or a voxel model representation of the ear impression data.

The process of training the neural network to be able to output the desired one or more partially constructed 3D models and/or the fully constructed 3D model together with the applied operations to reach the outputs, the training process utilizes one or more target ear impression data which is to be understood as a plurality of ear impression data that may comprise a digital scans of an ear impressions taken from users of a custom ear device, as previously described in relation to the input data being an ear impression data.

Each of the plurality of target ear impression data used for the training of the neural network comprises interlinked information concerning the target configuration requirements, the target set of operations and also a target partial and/or fully constructed 3D model of the custom ear device corresponding to each of the plurality of target ear impression and complying with each of target configuration requirements, target set of operations interlinked with the target ear impression data. In accordance herewith the target operations may comprise target product desires for a specific one of the plurality of target ear impression data. The target set of operations may comprise a set of intermediate designs steps manually performed by a trained professional when designing the custom ear device corresponding to a specific one of the plurality of target ear impression data and complying with a target configuration requirement for that ear impression data. The training of the neural network will be elaborated on in more detail in the detailed description of the Figures. The target set of operations may be configured as previously described in relation to the datafile and the geometrical transformations applied to one or more points, areas, facets and/or vertices. In the training data these geometrical transformations is contained in a training data set gathered from trained professionals design of custom ear devices. To efficiently use the target set of operations for training of the neural network, the data containing the target set of operations may be pre-processed by e.g., generating a feature tensor arranging the data in a structured manner suitable as input to one or more neural networks described herein. The data may comprise the following types, but not limited to, and methods for preprocessing and transforming the target set of operations into a feature tensor. In one example the preprocessing and transforming of the target set of operations may comprise to generate one or more sets of 3D vectors from the ear impression data along with zero, one or more numerical feature descriptors for each point of the 3D vector. These points and their respective feature descriptors could be concatenated together in one or more dimensions, and in batches to form a feature tensor. The vectors could be the 3Dcoordinates of all or a subset of vertices and normals. The normals could be estimated from a triangle mesh via locally weighted facet normals, or weighted by the density of points, etc. The feature descriptors could be estimated from the representation of the impression data, say the estimated mean curvature at a point or some other value representing the operation step as described in some of the following examples. A subset of these points may be formed by randomly selecting vertex points themselves, or sampling points on the surface according to some distribution. The distribution could be uniform, or depend on other extracted features of the data, these features being geometric, say by regions of high curvature, or prioritizing more samples near user-defined features like splines and planes also from the training data. The target set of operations data maybe comprise a selection of representation of curves in 3D space in relation to the ear impression data; these curve representations may be piece-wise weighted polynomials (colloquially referred to as splines), or other linear or non-linear weighted combinations of families of simpler parametrized functions. This curve data can be transformed into a feature tensor, say by concatenating the control points of the spline or the samples of the spline by some distribution (again uniform or dependent on some features of the curve). The curves can also be considered as lying on the representation of the ear impression data or any partial 3D model, whereby the transformation into a feature tensor could be by projecting the previously mentioned control points or sampled points onto the appropriate surface by some choice of projection method. The projection method could be a via closest point, closest vertex, along normal line, etc. The projection onto a surface could also be represented by a feature tensor of vertex, facet or voxel references of the 3d representation of the model with respective numerical feature labels indicating whether the reference in question should be considered as part of the spline, not part of the spline or some soft assignment in between. The data may be representing a plane in relation to the representation of the impression data, say by point and normal. The point and normal may be concatenated together to form a feature tensor, or a subset of points on the representation of the surface may be selected to represent the plane data and then transformed according to the previously mentioned point transformation method. The representative points maybe selected by a subset of vertices or sampled points by closeness to the plane, say by selecting all points within some distance of the plane and computing a numerical label representing the closeness to the plane, after which the label and point data can be concatenated into a feature tensor. The data may contain CAD component placements, in the form of a 3x3 rotation matrices and vectors. The transformations may be concatenated to form a feature tensor. The CAD placement could also be represented by a set of feature descriptors on a subset of points on the representation of the impression data, for example the distance of the point to the closest point on the placed CAD component. The data may also contain sculpt data, as an example represented as a point-to-point deformation field, where each point has a feature descriptor composed of the 3d coordinates of a desired destination for that point. The data may comprise a set of numerically annotated set of vertices, facets or voxels where the annotation value may be concatenated together with the points to form a feature vector. Any of the previously mentioned feature descriptors may be used as part of an input feature tensor, or as an output feature tensor of the target training data to evaluate the training performance of the neural network. Any of the previously mentioned data may be preprocessed according to any standard normalization methods, say re-scaling to unit size, centering, before concatenation into the feature tensor. The final feature tensor may be formed by concatenating any number of the previously mentioned feature tensors into one, either by combining via appropriate dimension sizes and/or some standard padding method.

In an example according to the methods described herein, the training of the neural network may be configured to be trained without the use of the target operations data previously described. In this case the neural network may learn the operations by comparing a target partially constructed model with the output of the neural network during training instead of comparing the output from the neural network during training with a target operation and a partially constructed 3D model.

The final operations datafile may comprise the production instructions configured to be utilized for manufacturing the finally constructed custom ear device. That is the final operations datafile may be configured to describe data representing the operations steps applied by the neural network to reach the fully constructed 3D model.

Furthermore, as e.g. mentioned in the introductory section, the designed custom ear device is generally 3D printed, assembled and shipped back to the user. In accordance herewith it should be understood that the production instructions of the final operations data file comprise 3D printer instructions configured to be applied by a 3D printer for manufacturing the custom ear device. That is, at least the production instructions comprise 3D printer instructions for printing at least the shell of the hearing aid or a cast thereof.

The final operations datafile may comprise one or more partial set of operations as applied by the neural network, wherein each of the partial set of operations represents intermediate design steps applied by the neural network. As some of the operations predicted by the neural network includes e.g. positioning of CAD components, the final operations datafile may also comprise positioning information, which may be used by a manufacture to physically position the CAD component in the 3D printed shell. At least it is considered that the final operations datafile can be used to visualize the designed custom ear device, with its placement of CAD component in a system at a manufacturing site, allowing the manufactures to check the position of the CAD components in the 3D printed shell during assembly of the custom hearing aid. Accordingly, with the method herein, the automatic process does not only optimize time spend on designing the hearing aid shell and internal CAD component placement, but also provides an effective manufacturing tool at a manufacturing site. With the system and method provided herein, the operations datafile may at a manufacturing site be used to visualize the finally constructed custom hearing aid, with all the internal parts in there respective positions. Using the software and the final operations datafile to visualize the fully designed custom hearing aid, allows an easy and fast assertion of the placement of e.g. the CAD components, which ensures that these are placed inside the custom hearing aid in accordance with the intended design.

In case the user of the system described herein, in accordance with the method, has interacted with the design application to change and/or modify at least one partially constructed 3D model as output from the neural, the final operations datafile may comprise both the set of operations as applied by the neural network in designing the partially constructed ear device, and a set of operations performed by the processor in accordance with the user instructions. In other words, the final operations datafile may comprise data generating directly by the neural network and data resembling manual design changes of a partial constructed 3D model of a custom ear device. Again, in this way, it is possible to provide a design application which is capable of outputting a fully constructed 3D model but also the partially constructed 3D models and corresponding operations applied by a neural network, to allow a user of the design application to performed changes to the automatically generated custom ear device operations.

When the user of the application has made changes to e.g., a partially constructed 3D model, the method described is preferably configured to use the changes made by the user and create a finally constructed custom ear device.

Accordingly, in an embodiment, the method may be configured to receive a user instruction causing the processor to feed the changed partially constructed 3D model into at least one other neural network, wherein the at least one other neural network is configured to output an updated partially constructed 3D model of a custom ear device based on at least another set of operations, wherein the one other neural network is trained on the basis of a second training data set comprising at least a set of target operations corresponding to a specific partially and/or fully constructed 3D model. In other words, the at least one other neural network is considered a second neural network which is trained on different training data than a first neural network. Accordingly, the neural networks described herein may be a plurality of neural networks, each trained on different training dataset, wherein the different training dataset comprises different target operations. The different target operations are configured to mimic the manual design steps normally performed by a trained professional when designing a custom ear device.

By providing a neural networks structure comprising a plurality of neural networks trained especially on different target operations it is possible for the neural network to output a plurality of intermediate operations corresponding to a partially constructed 3D model of a custom ear device.

In an embodiment, changes performed by a user of the application to e.g., a partially constructed 3D model, may be input to another neural network as previously described and/or into an algorithm setup to perform a set of operations different from the ones performed by the neural network. That is, in an embodiment, the method is configured to receiving a user instruction causing the processor to feed the changed partially constructed 3D model into at least one algorithmic operation, wherein the algorithmic operation is configured to output an updated partially constructed 3D model and/or the finally constructed 3D model.

By providing a design application, such as the system described herein, utilizing one or more neural networks and/or algorithmic operations to design partially and fully constructed 3D models of a custom ear device, it is possible to output one or more partially designed models to a user to allow the user to change any intermediate operation, which the user finds unsatisfactory in view of the configuration requirements for a specified custom ear device. With the method described herein it is possible to provide a design application forming part of a system executing the method and to allow for a fast forward procedure, which will be described in more detail in relation to the detailed description of the figures. In short, the design application provided by the method described herein, allows a user to e.g. evaluate a finally constructed 3D model of a custom ear device (as output from one or more neural networks and/or one or more algorithmic operations) by assessing a visual representation of the fully constructed 3D model in a user interface of the design application, such as an CAD/CAM software application, and to digitally interact with the design application to mark an area and/or point of the fully constructed 3D model, which the user might not be satisfied with. Such interaction may cause the design application to jump to the set of operations (as output from the neural network and/or the algorithmic operation) which represents the set of operations applied for the marked area and/or point.

Accordingly, in an embodiment, the system described herein may comprise a graphical user interface configured to operatively communicate with the editing engine described herein. The user interface comprises one or more interactive elements, wherein upon activating an interactive element of the one or more elements, the editing engine is configured to cause the processor to displaying, in a current view of the user interface, the finally constructed custom ear device with a representation of the final operations datafile; and receiving, while in the current view of the user interface, a user input requesting that the at least one partially constructed representation of the custom ear device with a representation of the corresponding set of operations, is to be displayed; and modifying, in response to the user input, the current view of the user interface to display the at least one partially constructed representation of the custom ear device with the representation of the corresponding set of operations. In this way, the provided method ensures that the design application may receive a user input causing the processor to change the current view of the finally constructed custom ear device to a partially constructed custom ear device. The set of operations applied to the partially constructed ear device may be represented visually in the user interface such as e.g., where a receiver is placed, where a tip cut is placed and so forth, which will be explained in more detail in connection with the drawings. In this way, a fast-forward method may be considered applied by the method described herein, as the operations steps preceding the chosen partially constructed custom ear device displayed in the user interface have already been applied automatically.

The user input received by the editing engine may be configured as marking an area and/or pointing to an area on the fully constructed custom ear device, causing the processor of the editing engine to modify the current view of the user interface to display a partially constructed representation of the custom ear device corresponding to the set of operations applied to the marked and/or pointed at area.

Furthermore, when the user of the application has modified the chosen area/and or point in the partially constructed representation (such as a 3D model) of the custom ear device, the processor is configured to perform changes to the partially constructed representation of the custom ear device, such as a 3D model, in accordance with the user instruction, and to output an updated finally constructed ear custom device and an updated final operations datafile.

In the embodiments described herein, the user input may be configured as a click of a mouse communicatively connected with the user interface and/or a touch on the display of the user interface. In this way, a user may directly interact with the user interface of the system and thereby create instructions of for example changing an area of the custom ear device representation.

Accordingly, a system is provided which is configured to perform the method steps described herein. The system comprising at least one application (such as a CAD/CAM software) configured to execute the described method is also disclosed. The system comprising a processor configured to receive one or more user input causing the processor to control the execution of a neural network engine comprising at least one neural network configured to output at least one partially and/or fully constructed representations (such as e.g., a 3D model) of a custom ear device and trained in accordance with embodiments described herein. The system may further comprise a fast forward engine configured to receive the output from the neural network engine and to output at least one partially constructed representation, such as a 3D model, of the custom ear device, and comprising at least a set of operations applied by the neural network and complying with at least a part of one or more configuration requirements. Furthermore, the system comprises an editing engine configured to receive the partially constructed representation, such as a 3D model, and the set of operations from the fast forward engine. In order to allow a user to edit (i.e., also discussed throughout as to change or modification) at least one set of operations as applied by the neural network in generating the partially and/or fully constructed representation, such as a 3D model, of a custom ear device, the editing engine is configured to receive a user input representing changes applied to the partially constructed 3D model by the user input.

In one embodiment, the editing engine may be configured to receive a user input causing the processor to perform changes to the partially constructed representation (such as a 3D model) in accordance with the user input, and to output a finally constructed custom ear device and a final operations datafile. In this case, the user may perform changes via the editing engine to a partially constructed 3D model which results in a final constructed 3D model of the custom ear device.

In one embodiment, the editing engine may be configured to receive a user input causing the processor to perform changes to the partially constructed representation, such as a 3D model, of the custom ear device in accordance with the user instruction, outputting an updated partially constructed representation of the custom ear device with an updated set of operations; and feeding the updated partially constructed custom ear device into another neural network and/or an algorithmic operation. In such an embodiment the changes made to the partially constructed representation, such as in the form of a 3D model, by the user may be run through a second neural network forming part of the neural network engine, wherein the second neural network applies one or more set of operations, not already performed by either the user via the editing engine or by another neural network, and output a new partially constructed 3D model and/or a finally constructed 3D model.

To allow the system and method described herein to access efficiently the data inputs to the neural network (i.e., the ear impression data and the corresponding configure requirements), the processor may be configured to be operatively connected with a storage engine, wherein the storage engine is configured to store thereon one or more ear impression data and one or more corresponding configuration requirements.

The storage engine may be configured as a cloud server configured to receive one or more ear impression data and corresponding one or more configuration requirements from one or more remote servers. This allows the application of the system described herein to effectively communicate with for example a manufacturing site, an audiologist and/or retail site, where at least the configuration requirements may be collected from the audiologist site and/or retail site, whereas the described production data is collected from the manufacturing site. To allow the system of design application described herein to transfer the finally constructed custom ear device and the final operations datafile to a manufacturer, the storage engine may be configured to store thereon at least the finally constructed custom ear device and the final operations datafile.

In further examples according to the systems described herein, the system may be configured such that the editing engine is configured to generate a toggle menu together with the final constructed custom ear device and to display the generated toggle menu in the user interface, wherein the toggle menu comprises a representative active field area for each of the one or more set of operations as output from the neural network. The mentioned active field may also be considered as e.g., a string, a text string, a box, or object or similar.

The editing engine may be configured to receive a user input instructing a deactivation and/or an activation of one or more of the representative strings for each of the set of operations, wherein a deactivation or activation causes the processor to update the current view of the custom ear device with a removal of the set of operation in case of a deactivation or a displaying of the set of operation in case of activation of the respective set of operation. The word representative string used herein may correspond to the previously mentioned active field. That is the active field may be e.g., a text string, a box, an object or similar field, which may be activated or deactivated upon receiving a user instruction to the field.

The toggle menu may in embodiments described herein, comprise an active field area representing a left and right ear impression data, wherein the editing engine upon activation of at least one of the active field area representing the left and right ear impression data is configured to activate or deactivate in the current view of the user interface, an editing mode enabling the editing engine to receive user inputs representing modifications to an activated left or right view independently.

The user interface according to embodiments described herein may comprise several view depending on the user input given. In one view, also mentioned as a current view, the user interface may be configured as a split view, where in a first side of the split view a left ear impression data is configured to be displayed, and in a second side of the split view a right ear impression data is configured to be displayed.

In a further example, the editing engine may be configured to automatically mirror changes made to one ear impression data in a first side of the current view to do similar changes automatically in the other ear impression data in a second side of the current view. In this way, any changes performed to e.g., a left ear impression data may be similarly applied to the right ear impression data.

Generally, the finally designed custom ear device described herein, is considered a result of applying the different target operations as output from the different neural networks of the neural network structure of the ear impression data. Applying is considered to mean utilizing the set of operations as output from the neural networks to modify the ear impression data in accordance with the set of operations. As an example, if a set of operations output from a neural network is a prediction of points associated with a tip cut, these set of operations when used to modify the neural network ensures that the data points of the ear impression data associated with the tip cut is marked and used to e.g. remove points on one side of the marking and allow points on another side of the marking to stay, thereby creating the tip cut. Further examples will be described in relation to the Figures.

Generation of the training data

As explained, the neural network structure, as described herein and illustrated in the appended Figures may be configured as a plurality of neural networks trained to output a plurality of different sets of operations. In the following, examples of how data sets can be constructed for training a neural network will be explained.

First it is noted that for each product type (e.g. BTE, RITE, CIC, etc.) a neural network structure comprising a plurality of neural networks each being trained on a plurality of target set of operations may be generated.

For each of the plurality of neural networks a series of pre-processing steps may be applied to generate training data sets suitable for predicting operations used to design a specific product type. The target set of operations used for training may be taken from manual sets of operations, also denoted as modelling tress, which are available from existing software solutions used to manually design custom hearing aids.

A target set of operations, taken from the modelling tree, may described a geometrical transformation, that an expert has manually used to modify the ear impression data. The geometrical transformation can be any of a positioning a plane identifying a tip or bottom cut, positioning a spline, positioning a CAD component and so forth. Any of these geometrical transformations is provided as input data to a pre-processing step.

The training data in more detail may be generated by loading a target ear impression data into a processor, generating a point cloud or voxel model from the ear impression data; loading a target operation (i.e. data) from a modelling tree (e.g. a cut plane comprising a point and a normal described the cut plane associated with the target operation); labelling the points of the point cloud or voxels of the voxel model associated with the target operation and generating a neural network with the point cloud or voxel model as input and the labelled points or voxels as targets. With this training data construction the neural network it is ensured that the neural network is able to learn how the input data is associated (i.e. interlinked) with the target data, where it is apparent that the target data is the set of operations as acquired from a known modelling tree. This general method may be used for all examples given herein, where more examples will be elaborated on in more detail in relation to the Figures. With this it should be understood that the target set of operations in an example can be used to label the points of the ear impression data with target information, allowing the neural network to learn what to look for.

In another example, the neural network training may utilize a none-labeling approach, where the steps of generating a training dataset generally comprises: Loading a target ear impression data into the processor; generating a point cloud or voxel model of the ear impression data; loading a target operation from the modelling tree (e.g. the previous mentioned cut plane) and generating a neural network with eh point cloud or voxel model as input and the target operations as the training target for the neural network. With this example, the labelling step is left out and the learning of the neural network is not given any direct indication as to which points of the ear impression data are associated with the target operation data. This may result in a less accurate prediction than the first mentioned example.

The generation of the training data set may comprise generation of the previously described feature tensors suitable as input to the neural network(s) described herein.

As a further step considered to be applied together with any of the above given examples, is that the method described herein may comprise segmenting the ear impression data into ear canal geometries, wherein ear canal geometries comprises the inner ear canal part, the middle ear canal part and the outer ear canal part. By segmenting the ear impression data in a pre-processing step the amount of data to be processed in the further labelling or nonlabelling examples and accordingly input to the neural network for training is minized allowing data processing time optimization.

When a segmentation step has been performed in the pre-processing, only the target operations associated with e.g. the inner ear canal part, the middle ear canal part and the outer canal part are used to e.g. label the data with. In an example, as a bottom cut for example is mainly associated with the outer ear canal part, the target operation data from the modelling tree is only processed together with the part of the ear impression data forming the outer ear canal part according to the segmentation step.

In an example, the segmentation step may also utilize a trained neural network, but a manual segmentation to generate the training dataset may also be considered. Further explanations on the generating the training dataset will be given in the detailed description with relation to the Figures.

All of the embodiment described above in relation to both the system and method may equally combined as the system is configured to run the method described herein. Details and a further understanding of the disclosed method and system will be apparent from the following detailed description. BRIEF DESCRIPTION OF THE FIGURES

Examples as described herein may be best understood from the following detailed description taken in conjunction with the accompanying figures. The figures are schematic and simplified for clarity, and they just show details to improve the understanding of the claims, while other details are left out. Throughout, the same reference numerals are used for identical or corresponding parts. The individual features of each example described may each be combined with any or all features of the other examples unless stated otherwise. These and other examples, features and/or technical effect will be apparent from and elucidated with reference to the illustrations described hereinafter in which:

Figure 1 illustrates a processor configured to perform a computer implemented method according to examples of the disclosure;

Figure 2 illustrates the method steps performed by the computer-implemented method according to examples of the disclosure;

Figure 3 illustrates a machine learning model structure for training a neural network according to examples of the disclosure;

Figure 4 illustrates a training dataset according to examples of the disclosure;

Figure 5 illustrates a training dataset according to examples of the disclosure;

Figure 6 illustrates a design service application according to example methods described herein;

Figure 7 illustrates a processor configured to perform steps of the computer implemented method according to examples described herein;

Figure 8 illustrates a machine learning model structure for training a neural network according to examples of the disclosure;

Figure 9 illustrates a communication setup between a design service site configured with the design service application described herein and an audiology/retail site and manufacturing site for sharing of data between the different sites;

Figure 10 illustrates a machine learning model structure for training a neural network according to examples of the disclosure;

Figure 11 illustrates a machine learning model structure for training a neural network according to examples of the disclosure; Figure 12 illustrates a machine learning model structure for training a neural network according to examples of the disclosure;

Figure 13 illustrates a machine learning model structure for training a neural network according to examples of the disclosure;

Figure 14 illustrates a division of the training data set into a training data and validation data in accordance with the training processes described herein;

Figure 15 illustrates an example overview of a data transformation for an example neural network architecture as described herein;

Figure 16 illustrates a none-exhaustive list of example configuration requirements described herein;

Figure 17 illustrates an example of a digital representation of an ear impression data as described herein and used for both the training of the neural network and in the design application software using the trained neural network;

Figure 18 illustrates an example operation of shaping a tip of the ear impression data, that the neural network is trained upon;

Figure 18A illustrates an example training data generation for a tip cut target operation;

Figure 19 illustrates an example operation of tapering the ear impression data, that the neural network is trained upon;

Figure 20a illustrates an example operation of cutting areas of the ear impression data, that the neural network is trained upon;

Figure 20b illustrates an example operation of extending areas of the ear impression data, that the neural network is trained upon;

Figure 21a illustrates one example of shape operations performed on the ear impression data and that the neural network may be trained upon;

Figure 21b illustrates a result of the shape operation achieved in accordance with Figure 21a which the neural network may be trained upon;

Figure 21c illustrates a second example of shape operations performed on the ear impression data and that the neural network may be trained upon;

Figure 21d illustrates a result of the shape operation achieved in accordance with Figure 21c which the neural network may be trained upon;

Figure 22a illustrates one example of a casting operation performed on the ear impression data and that the neural network may be trained upon; Figure 22b illustrates a second example of a casting operation performed on the ear impression data and that the neural network may be trained upon;

Figure 22c illustrates a third example of a casting operation performed on the ear impression data and that the neural network may be trained upon;

Figure 23a illustrates an example operation for placing a sound bore and sound entrance in the custom ear device that the neural network may be trained upon;

Figure 23b illustrates an example operation for placing a vent in the custom ear device that the neural network may be trained upon;

Figure 23c illustrates an example operation of placing a receiver in the custom ear device that the neural network may be trained upon;

Figure 24 illustrates an example of an ordering sheet providing the configuration requirements that a custom ear device may be designed based upon and which is feed into the neural network.

Figure 25 illustrates in an exemplified manner, the input and output to and from the neural network described herein;

Figure 26 illustrates a user interface according to the system described herein together with the underlying fast forward and editing engine processes running; and

Figure 27 illustrates an example user interface of the system and method described herein.

DETAILED DESCRIPTION

The detailed description set forth below in connection with the appended drawings is intended as a description of various examples according to the disclosure. The detailed description includes specific details for the purpose of providing a thorough understanding of various concepts and examples covered throughout the disclosure. However, it will be apparent to those skilled in the art that these concepts and examples may be practiced without the specific details mentioned or in combination with one or more examples described herein. Several examples of the devices, systems, media or mediums, programs and methods are described by various modules, components, steps, processes, algorithms, etc. Depending upon particular application, design constraints or other reasons, these elements may be implemented using electronic hardware, computer program, or any combination thereof. In the following several examples of the methods and system described herein will be disclosed in more detail. The method and systems described herein will be described in the following examples with reference to the appended figures. To perform the method steps described herein, a system comprising at least one application (such as a CAD/CAM software) configured to execute the described method is also disclosed. In short, the system comprises a processor configured to receive one or more user instructions causing the processor to control the execution of a neural network engine comprising at least one neural network configured to output at least one partially and/or fully constructed 3D model of a custom ear device and trained in accordance with embodiments described herein. Preferably the neural network engine comprises a plurality of neural network structures each configured to predict a set of operations coding for modification that should be applied to the input ear impression data to generate a partially constructed 3D model.

An example of such a system 1 comprising a processor 2 configured to perform the method steps described herein is illustrated in Figure 1. The processor 2 of Figure 1 is configured to perform a computer-implemented method according to the examples provided in the following description. The processor 2 is configured to run on a computer and/or in a cloud service and is configured to form part of a design software application (such as e.g., a CAD and/or CAM software application). The computer and/or the cloud service comprises stored thereon an application configured with code containing instructions for performing the method steps described herein. The application module may be considered as a design service software for designing custom in ear devices at least partly automatically.

In more detail, and as illustrated in Figure 1, the computer-implemented method run on the processor 2 is configured to construct a custom ear device of a user based on data 3 input to the processor 2 upon receiving a user instruction 4 for loading the data 3 into the processor 2. The user instruction 4 may be any instruction given to the processor 2 through e.g., a user interface of the design application software. That is, the user instruction may e.g., be the click of a mouse, the touch on a screen and/or similar interaction with a graphical user interface of the design software application executing a method on the processor 2. In accordance herewith and illustrated in Figure 2, the computer-implemented method comprises loading into a processor an ear impression data 100. The ear impression data 5 can be obtained from a local storage unit (i.e., a local server storage database) and/or from a cloud storage as illustrated in Figure 1, where the cloud database is denoted with no. 3, previously also described as data 3.

To ensure that the loading ear impression data can be processed efficiently by the neural network(s) described herein, the processor is configured to generating a 3D digital representation of the ear impression data, wherein the ear impression data has been acquired by an optical scanner. The 3D digital representation can be a point cloud or a voxel model representation of the ear impression data.

Furthermore, the computer-implemented method is configured to loading 200 (see Figure 2) into the processor 2 one or more configuration requirements 6 for designing a custom ear device matching the ear impression data 5, as illustrated in Figures 1 and 2. The configuration requirements 6 may in correspondence with the ear impression data 5 be configured to be stored on a local storage unit (i.e. a local server storage database) and/or on a cloud storage as illustrated in Figure 1, where the cloud database is denoted with no. 3, previously also described as data 3. Both the ear impression data 5 and the configuration requirements 6 may be configured to be loaded into the processor 2 via a network, such as an internet 7, as illustrated in Figure 1. The configuration requirements is considered as a datafile comprising information about e.g. a product type, such as a BTE, RITE, CIC , ITC or any other product type as mentioned in relation to Figure 16. The neural network(s) activated by the processor upon a user activation comprises a neural network structure for each of the different product types. This also means that each of the neural network for each product type main contain sub-neural networks each predicting the set of operations associated with a specific product type.

The loading of the ear impression data 5 and the configuration requirements 6 may be a result of a user instruction 4 applied to a user interface of design service application as described herein. Accordingly, the computer-implemented method is configured to receive a user instruction 4 in Figure 1, and step 300 in Figure 2, causing the processor to load the ear impression data 5 and the configuration data 6 into a neural network (NN) 8. The neural network 8 is configured to output 9 at least one partially constructed 3D model 10 of a custom ear device complying with at least a part of the one or more configuration requirements 6, and a set of operations 11 as applied by the neural network and complying with at least the part of the configuration requirements 6. It should be noted that Figure 1 illustrates that the neural network output comprises a set of operations and a 3D model of a partially constructed custom ear device. However, as already elaborated on, the neural network may be configured to only output the set of operations 11 and not necessarily to output the partial constructed 3D model, as such. That is, the set of operations coding for the operations to be applied to form a partial constructed model may be the main output form the neural network. That the neural network 8 described herein is configured to output at least a set of operations 11 coding for a partially constructed 3D model 10 allows a user (such as a professional modeler) to assess the partially constructed set of operations 11 and/or 3D model 10 as output from the neural network. That is, the neural network 8 is configured to output not only the finally constructed 3D model, which is also a possibility, but at least one or more partially constructed representations comprising the set of operations 11 coding for a partially constructed 3D model 10 which a user (such as the professional modeler) may then adjust and/or modify to create a more accurate and/or improved design of the custom ear device. As mentioned, the partially constructed 3D model may be considered as an output as such, but it is more preferred that the neural network output is merely the set of operations coding for the modifications that have been applied to the input ear impression data to generate the partial constructed custom ear device.

In accordance herewith, the computer-implemented method is therefore also configured to receiving 400 of Figure 2, a user instruction 12 illustrated in Figure 1, causing the processor 2 to perform changes 13 to the partially constructed representation of the 3D model 10 in accordance with the user instruction 12. This allows the computer-implemented method to output a finally constructed ear custom device 14 and a final operations datafile 15, which contain information of the operations applied to design the finally constructed custom ear device 14 being by a neural network automatically and/or using further manually changes to e.g., adjust the partial designed custom ear devices output from the neural network 8.

In one example, the neural network is configured to output the set of operations 11 coding for operations that should be applied to the ear impression data 5 to create a partially constructed 3D model 10 of the ear impression data. The set of operations 11 may therefore be output from the neural network, such that the user may change in part one or more of the set of operations as output from the neural network. Accordingly, the user instruction 12 may be a change to the set of operations 11 provided in a datafile as described herein.

By providing a computer-implemented method described above it is possible to provide an automatically design of custom ear devices based on ear impression data and configuration requirements given by a user, while allowing an evaluation of the output from a neural network of any partial design steps that are needed in designing a custom ear device. The evaluation possibility that is provided for by the solution described herein, allows a user, such as a skilled professional modeler to adjust the design given by the neural network to more accurately reflect and comply with the requirements given, in case the neural network did not predict a partial design step optimally.

To achieve such a design application solution running a method described herein, a training process (process 500 in Figure 2) of the neural network to learn how to design partial custom ear devices is needed. Therefore, the computer-implemented method described herein provides for a neural network 8 which has been trained during a training process 500 illustrated in Figure 2. In accordance herewith, reference is made to Figure 3, which illustrates a machine learning model structure for training a neural network 8 according to examples of the disclosure. The neural network 8 illustrated in Figure 3 is configured with a neural network 8, which may be trained on the basis of a training data set comprising: a plurality of training ear impression data 85; a plurality of target partial and/or fully constructed 3D models 84 of custom ear devices each complying with a plurality of target configuration requirements 82; one or more target operations 81 specific to the plurality of target partially and/or finally constructed 3D models 84 and complying with at least a part of the one or more target configuration requirements 82. By training the neural network 8 on a plurality of target ear impressions 85 and corresponding partial constructed 3D models 84 with its corresponding target operations 81 and target configuration requirements 82 it is possible to construct a trained neural network 8 which is able to output at least one partial constructed 3D model 86 and/or a set of operations 87 complying with at least a part of the requirements 82 given and the ear impression data 85 given. This partial constructed 3D model, such as for example a representation thereof comprising the set of operations 87, which is output from the neural network (potentially together with a finally constructed 3D model) may be used to ensure that a skilled professional user can modify a partial target set of operations to modify and/or change an output from the neural network.

The neural network 8 illustrated in Figure 3 can be considered as a machine learning model with a neural network 8 having a neural network architecture 83 chosen for a specific purpose, such as e.g. predicting set of operations associated with a specific product type. The neural network architecture may be chosen from different convolutional neural networks, where the convolutional network is configured as a type of deep learning algorithms that are used to process data a type of deep learning algorithm that is used to process data that has a spatial or temporal relationship, and which comprises a series of convolutional layers that performs a set of operations. Generally, a convolutional network architecture comprises a convolutional layer, a pooling layer and one or more fully connected layers. Each of the layers is configured to perform a set of different tasks, where the main task of a convolutional layer is to filter the input using one or more filters to create feature map. The architecture of the neural network may use one or more convolutional layers stacked together so as to create complex model structure. The feature map output from the convolutional layer may be input to a pooling layer, which is configured to reduce the spatial size of the input, making it easier to process and requiring less memory. Furthermore, applying pooling layers also helps to reduce the number of parameters and makes training faster. Generally, there are two main types of pooling: max pooling and average pooling. Max pooling takes the maximum value from each feature map, while average pooling takes the average value. Pooling layers are typically used after convolutional layers in order to reduce the size of the input before it is fed into a fully connected layer. The fully-connected layers is configured to connected the different layers of the network together. That is, each layer comprises one or more neurons, which in the fully connected layer are connected for all the layers. The fully connected layer is configured to calculate predictions based on the neuron connections between the different layers in the neural network architecture. An example of a convolutional network is illustrated in Figure 15, where an example network architecture is provided with the label 2000. The CNN described herein may be considered to be used in connection with image data, point cloud data and/or voxel model representations of the ear impression data. In accordance herewith the ear impression data may be pre-processed to generate e.g. a point cloud or a voxel model representation of the data, which is further pre-processed to generate the target labels for each point or voxel in the data associated with the target operations. Other neural networks considered useful in the setup described herein includes PointNet type networks.

As the ear impression data 5, 85 described herein (that is both the plurality of target ear impression data 85, and the ear impression data 5) may be configured as a virtual 3D representation of the ear impression and that the ear impression data 5, 85 may be processed in a method step so as to generate by the processor a 3D model of the ear impression data. The ear impression data may be considered as image data, which is suitable for input to for example the described convolutional network architecture 83. Accordingly, the ear impression data 5, 85 may be processed to generate image data, such as a virtual representation of the ear impression data for both training of the neural network 8, but also when using the trained neural network 8 in a design service application as described herein. The ear impression data may also as previously explained be processed to generate a point cloud or voxel model representation.

A general machine learning model that may use one or more neural network architectures for training a neural network 8 according to the method described herein, is illustrated in a simplified manner in Figure 3. As previously touched upon, here it is illustrated that the network architecture 83 in the training process takes as input a plurality of ear impression data 85 and their corresponding target configuration requirements 82. The network architecture may be configured to output a predicted partial constructed 3D model 86 and a predicted set of operations 87 found by the neural network which matches the predicted partial constructed 3D model 86. To evaluate if the neural network has performed a good prediction (i.e. to allow the neural network to learn), the predictions 86, 87 is feed to a loss function 88 which also takes as input the target operations 81 that were expected for a target partial constructed 3D model 84. The loss function performs an error estimation which is feed back into the neural network architecture to adjust the performance thereof. With this, it should be understood that the neural network architecture (also denoted structure earlier) may comprises a plurality of neural networks each trained on a different set of target operations. Accordingly, as illustrated in Figure 3, an output of the neural network architecture may be at least a predicted set of operations 87 associated with a partial constructed 3D model 86. The predicted set of operations 87 may e.g. code for a tip cut modification as will be explained in relation to Figures 17 to 23c for different partial output predictions.

The error estimates could comprise one of a set of weights which the neural network architecture applies to its predictions to allow the network to better emphasize certain predictions and features above others.

The neural network 8 described in relation to Figure 3 is considered a first example. This neural network could output not only a or one or more partial constructed representations of e.g. the 3D model but also the finally constructed 3D model or at least a representation thereof. The output of the neural network all depends on what data the neural network has been trained upon as will be apparent through this description. In view of outputting a final constructed 3D model, it is considered that the neural network architecture preferably comprises sub-steps of post-processing each of the partial predicted target operations by modifying the ear impression data as input to the neural network, with the predicted target operations, and based on those modifications of the ear impression data generate a finally constructed 3D custom hearing aid.

Turning now to Figure 4 and Figure 5 different examples of training data 108a, 108b is illustrated. Figure 4 illustrates a training dataset 108a comprising a plurality of ear impression data 85(a-e), a plurality of target configurations data 82(a-e), a plurality of target operations 81(a-e) and a plurality of target partial and/or finally constructed 3D models 84(a-e). Each of the plurality of ear impression data 85a, 85b, 85c, 85d is interlinked with a plurality of target configuration requirements 82a, 82b, 82c, 82d, 82e. In other words, for each of the ear impression data 85a, 85b, 85c, 85d, 85e one or more target configuration requirements 82a, 82b, 82c, 82d, 82e has been specified. That is, the one or more configuration requirements comprises for examples a specified product type and design requirements for the specified product type as required by an audiologist and/or by a manufacture of custom ear devices through the configuration data. The “interlinking” is considered to mean that if a configuration requirement comprises e.g. a first product type, the ear impression data corresponds to that type of product type. This means for example, that if the product type is a RITE style product type, then the ear impression data represents an ear canal impression originally used to design the RITE style product.

An example of the target configuration requirements 82a, 82b, 82c, 82d, 82e that could be used for training the neural network is seen in Figure 24. Here, a list of different characteristics that can be chosen for different products in e.g., a design software application. Such list of specification (i.e., configuration requirements) may be used as input to the neural network for training purposes to allow the neural network to extract information needed to design a specified type of e.g., hearing aid based on an ear impression data and the configuration requirements given. As can be seen from the order form 600 illustrated in Figure 24 some of the requirements forming part of the configuration requirements may be identification of the ear impression data being for the right 601 or left ear 602. The device category 603, device model 604, device size 605, if the device should contain a telecoil 606, if a faceplate if needed and what type of faceplate 607, if a transducer it to be included 608 is a wax guard 609 is needed, vent 610, shape 611, sound bore 612, color 613 etc. The list given in Figure 24 is a none-exhaustive list of example and it should be understood that other requirements and/or less or more requirements could form part of the configuration requirements described herein.

Some of the configuration requirements also illustrated in Figure 24, may also be more visually illustrated in Figure 16 which illustrates a none-exhaustive list of example configuration requirements described herein. Figure 16 especially illustrates examples of device categories 603, device fit styles 614, device shapes 611, devices material 615, list of device components 616 and list of canal shaping’s 617 all of which may be chosen by a user when ordering a custom ear device. All of these device configurations (i.e., configuration requirements) may be used as input for training the neural network as described herein. Turning back to Figure 4 another part of the training dataset 108a comprises a plurality of target operations 81a, 81b, 81c, 8 Id, 81e interlinked with especially a plurality of target partially and/or fully constructed 3D models 84a, 84b, 84c, 84d, 84e. The target operations 81a, 81b, 81c, 8 Id, 81e are especially interlinked with the constructed 3D models 84a, 84b, 84c, 84d, 84e as the target operations forming part of the training data set are representing the design steps that a skilled professional would have applied in a manual design setup to design the respective target partially and/or fully constructed 3D models forming part of the training data 108a. Accordingly the target operations 81a, 81b, 81c, 8 Id, 81e should be understood as a set of data representing a plurality of intermediate design steps each representing the design of a corresponding plurality of the target plurality of partial constructed 3D models. It is especially relevant for the method described herein, that the target operations represent intermediate design steps as such steps is used for the neural network to learn how to design a partially constructed 3D model.

The target operations described in relation to Figure 4 (and later figures) may be collected from a modelling tree of a prior art design service for designing custom hearing aids. Such modellings trees may comprise any of the operations that a manual modeler (i.e. also denoted modeler actions) has applied to an input ear impression data to generate a certain type of product, e.g. the above mentioned RITE style product type. Accordingly, for training purposes, the method comprises generating a training dataset by loading from the modelling tree the target set of operations associated with specific target configuration requirements, loading the associated ear impression data and for each target set of operation of the modelling tree, generate a neural network model configured to output a prediction of each of the set of target operations. With this, the neural network architecture used for a specific product type may be configured to predict each partial set of operations corresponding to the target set of operations.

The target operations 81a, 81b, 81c, 8 Id, 8 le described herein, may comprise one of one or more set of decision rules, one or more production data, wherein each of the one or more decision rules and/or production data provides instructions for production of the partially and/or fully constructed 3D model of a custom ear device as output from the neural network. In an example, the one or more set of decision rules is configured as modeler actions that a skilled professional would have applied to the ear impression data in the design of the custom ear device. Accordingly, the neural network may be trained on such manually applied decision rules to learn how to correctly design at least one or more partially constructed 3D models. In an example, the one or more set of decision rules comprises data representing operations applied by the neural network described herein to construct the partially and/or finally constructed custom ear design to fit the anatomy of an ear canal of a user. The decision rules may form part of the target set of operations, and is configured to be applied in the training of the neural network architectures for different product types to allow predictions of partial set of target operations used to automatically design the custom hearing aid.

Examples of the decision rules forming part of the set of operations as described herein is now to be described in relation to Figures 17 to 23. In general the decision rules described here may comprise one or more of operations representing shaping a tip of the ear impression data, wherein the tip is configured to face the inside of an ear canal when inserted therein; cutting a part of the ear impression data configured to face the outside of an ear canal when inserted therein; shaping the outer shape of the ear impression data; placing of the electronic components; casting for a soft silicone production of the digital ear impression placing of a sound bore; placing of a vent canal.

Turning first to Figure 17 an example of a digital representation 51 of an ear impression data as described herein and used for both the training of the neural network and in the design application software using the trained neural network is illustrated. The ear impression data may be configured as an impression mesh 51 and represents preferably a digital scan of a physical impression acquired from a patients ear. For training of the neural network a plurality of such ear impression data is used as input to the neural network together with the configuration requirements described herein. The neural network described herein is configured to learn how to adapt the ear impression data so as to create a custom ear device that supports the configuration requirements given. The ear impression data may be a mesh comprising voxels, facets and points representing the data structure, and used to generate a digital 3D model of the ear impression data. The set of operations described herein, and as previously elaborated on, is configured as instructions applied to the voxels, facets and/or points in the ear impression data to modify the ear impression data to generate the partial 3D custom hearing aid design.

Turning to Figure 18, an example operation 501 of shaping a tip of the ear impression data, that the neural network is trained upon, is illustrated. An operation causing a shaping of the tip data is configured to ensure that the tip is of the custom ear device comprises a clear surface to allow sound to exit without interference and providing a direct path towards the eardrum. In a manual operation, on which the neural network is trained upon, a modeler may achieve a preferred tip shaping by cutting the ear impression data as seen in Figure 18. The tip shaping in Figure 18 is illustrated by a plane 510 overlaying the ear impression data 51 in such a manner that the plane 510 defines the cut surface to be applied to the ear impression data. The edges around the cut surface may be smoothed to avoid causing pain when inserting the device in the ear. The direction and placement of the cut surface can have a direct impact on the sound quality. It is preferred that the plane 510 is orthogonal to the direction towards the eardrum. How to exactly place the cut surface (i.e. the plane 510) in the manual setup may be difficult and relies a lot on the modelers experience and qualifications. Accordingly, by using such operations data as input to a neural network as described herein, it is possible to allow the neural network to learn how to place the cut surface 510 automatically and likely perform a better prediction of the cut surface than what a manual modeler could have done as the neural network is configured to learn on an extensive amount of data gathered from preferably different modelers, different ear impressions and different target configurations etc. With reference to Figure 18, the training data set is generated using such plane 510 information to generate the target operation used for training the neural network. That is in more detailed example of generating a training data set configured to predict a target operation of a tip cut, the method comprises:

Loading the ear impression data 51 into a processor; generating a point cloud of the ear impression data 51 and loading a target operation from e.g. a modelling tree, wherein in the example given in Figure 18, the target operation is configured as the cut plane 510 comprising points and normals describing the cut plane. Using the target operation in the form of the mentioned points and normals to labelling the points of the point cloud associated with the cut plane as plane cut points. From this, a dataset is generated which is suitable for training a neural network to predict a target operation coding for the tip cut plane shown in Figure 18. Therefore, the training part of the method further comprises generating a neural network, where the input comprising the generated point cloud with the labelled points. In this way, the neural network is able to predict the target operations associated with the tip cut plane. A sub-sequent step may be to post-process the ear impression data which was input to the neural network, by modifying the point cloud to cut away the points above or below the points predicted to belong to the tip cut plane. In this way, for example the top part of the ear impression in Figure 18 may be cut away from the ear impression data, as these points according to the target operations, which the NN is trained upon should not form part of the finally constructed custom hearing aid.

In a bit more detail, the labelling may comprise labelling with T the points that belong to a band where the plane is placed, wherein the points of the band are those that are a defined distance to the plane within a margin, which depends on a set threshold. Generating the training data for e.g. a tip cut plane is best illustrated in Figure 18 A, where it is seen that the labelling part of the pre-processing of the ear impression data and the target operation may comprise labelling points belonging to a band 510a representing the tip cut operation with 1 and the rest of the points as 0. In this way the target points representing the possible tip cut target in the training data is generated. The band 510a is marked as a dotted line in Figure 18a, and all the points lying within the band may be considered as belonging to the tip-cut. With this it should be understood that the input to the neural network for training purposes in generally the data represented in Figure 18 A. The training data describing the tip cut planes 510 and 520 illustrated in Figure 19 is constructed in the same manner as just described in relation to Figure 18 A, however not illustrated in detail, but described in the following.

Another example of an operation, which the neural network may be trained upon is illustrated in Figure 19. Here an example operation 502 of tapering the ear impression data, that the neural network is trained upon is illustrated. As the auditory canal of the ear is surrounded by both cartilage, tissue, and bone, some of these areas are sensitive to touch, and can trigger neuro-reflexes, such as coughing and gaging, but can also be a cause of pain and discomfort. Accordingly, it is preferred if the neural network can learn how to avoid such issues when designing the custom ear device, and thus to apply operations causing a tapering of the ear impression data to ensure minimal contact with the ear canal of the partially and/or finally constructed custom ear device. Such an operation is illustrated in Figure 19, where it is seen that a plane 520 can be positioned on the ear impression data to allow tapering of the ear impression data to ensure optimal fit with the ear canal. The plane 520 may in the manual process be adjusted to fit the desired tapering by a modeler, and such plane positioning data may be used to form part of the set of target operations as described herein. It is noted that Figure 19 also illustrated the plane 510 used for cutting the tip in accordance with the example described in relation to Figure 18. However, the operation of cutting the tip and the operation of tapering the ear impression data 51 may not need to be interlinked and could be performed separately from each other. Thus, the neural network described herein is considered to receive independent set of operations for e.g. forming the tip and/or tapering the ear impression data. It may also be that the two operations are interlinked and that such interlinked information forms part of the target operation data to allow the neural network to learn how one or more different set of operations are linked together.

Figure 20a illustrates yet another example of a target operation on which the neural network is trained upon. In this example, the operation concerns a bottom cut operation 503, which is configured to cut away the bottom of the ear impression data (pointing out of the ear when the custom ear device is inserted in the ear). This is to ensure that the part of the physical impression acquired that represents an area of the silicone not touching the ear is cut away. This is especially relevant for certain types of devices, such as in ear monitors, where electronics used in the device might not fit in the ear and the device must be extended outwards. This extension must ensure to not collide with any of the ear geometry. In this operation, illustrated in Figure 20a, the bottom of the ear impression data 51 may be cut away using e.g. a plane 530 positioned in relation to the ear impression data 51. The plane is as in the previously described examples visualized as an adjustable circular plane segment surrounding the ear impression data 51. For explanatory purposes, the plane 530 defines an area above the plane and an area below the plane, wherein in this bottom cut operation, the area below the plane represents an area which may be cut away from the ear impression data 51. Instead of cutting away the bottom as described in relation to Figure 20a, an operation may also include to extend the bottom outwards of the ear if more room inside the finally designed custom ear device is needed. An operation 504 of extending the bottom outwards is illustrated in Figure 20b. Also the operations described here in relation to Figures 20a and 20b may be processed to generate the training data set in a similar manner as described in relation to Figure 18 A.

With this, it should be understood that the target operations illustrated in Figures 18, 18 A, and 19, 20a and 20b may be used to generate a different neural network each predicted each one of these target operations. In this way, each of the target operations may be output from a neural network architecture comprising a plurality of trained neural networks described herein allowing a user of the design software to modify each of the set of operations as predicted by the neural network.

Figures 21a to 21d illustrates different target operations for shaping 505 the ear impression to the desired shape in accordance with the configuration requirements given. Figure 21a to 21d illustrates in more detail different shape splines and shape plane examples that may be used for shaping the ear impression data 51 to the desired shape in accordance with a given configuration requirement. In Figure 21a and Figure 21b it is illustrated how a plane 550, preferably configured as a ring plane may be used to shape the ear impression data 51, and further how a spline 551 overlayed the ear impression data 51 may be used to shape the ear impression data 51. As can be seen that spline comprises one or more points which may be suitable to define the placement of the spline 551 on the ear impression data. Accordingly the points of the spline 551 may define where the ear impression data should be cut to shape the ear impression data 51 into the desired shape according to the configuration requirements corresponding to such a type of custom ear device. The result of the operation following the example given in Figure 21a may be seen in Figure 21b, where it is clearly seen that the areas not contained within the spline 551 is removed or at least greyed out. In this case, the splines 551 and the points associated with the spline and not associated with the spline 551 may be used to generate the training data used for training the neural network. In another example, shown in Figure 21c and Figure 21d for another type of custom ear device than shown in Figure 21a and Figure 21b, the spline and plane may be positioned differently. Accordingly, the operations 505, 507 needed to design the two different devices may be different as is apparent. In accordance herewith, it should be understood and appreciated that the target operations used for training the neural network comprises a plurality of different operations for different design steps and are interrelated with the type of device and/or other configuration requirements used for specified device types and wishes. In more detail, in Figure 21c an operation shaping 507 the ear impression data to form a specified product type is illustrated. Here it is seen that a spline 570 and a plane 571 is arranged to form the ear impression data shape shown in Figure 2 Id. In general, the shape is an important operation, since it gives the designed custom device proper retention, and ensures it fits tightly in the ear without discomfort, and do not fall out. In the manual process the modeler may decide on the shape with optimal retention, but also take the patient’s wishes into account. If the modeler in the manual process evaluates the ear canal to e.g., have enough curvature or a shape with sufficient retention, the device might not need additional retention in the outer ear. But it could also be necessary to extend select a shape that extends into the outer part of the ear. The shape of the custom device can have many forms, depending on where the modeler finds it necessary to add retention. The target operations given as examples in Figures 21a to 21d to shape the ear impression data may be based on manual modeler decision which the neural network uses as input in the training process to learn how to effectively and accurately shape the ear impression data to comply with the configuration requirements given. In the example operations given in Figures 21a to 2 Id, the target operations may provide data representing actions of drawing and adjusting splines and points on the model to set the correct shape, and to ensure that the thickness of the shape is sufficient to be printable. Other possible target operations which may be represented may include retention restriction on defined areas, volume restrictions etc.

In all of the above given examples, the modeler actions shown as splines, tip cut planes, bottom cut planed etc is used as input in a pre-processing step to generate a labelling or otherwise ground truth definition to the ear impression data used for training purposes. Each of the modeler actions illustrated in the previous described figures is considered to be trained in single independent neural networks, which is configured to output predictions of each of these modeler actions separately. The combination of all predictions as output from the plurality of different neural networks may be used to generally the finally designed custom hearing aid that is described as an output from the neural networks architecture.

To be able to produce the final custom device, target operations used for training the neural network may also comprise data representing the production method for the custom ear device. Such information on production may form part of the configuration requirements. Accordingly, instead of printing in hard material, using a 3D printer, the custom ear device can also be created in soft silicone. This often achieved by first, designing the desired custom ear device, and then creating a cast around it. This cast is then printed and filled with soft silicone. The cast can be made digitally in a design application and may as with the other target operations described herein be part of an input to the neural network. Such target operations configured to create a cast for soft silicone may comprise placement of an injection port on the cast. The injection port allows for a syringe with soft silicone to be inserted, so the silicone can be injected. Placement of drain holes on the cast, to allow otherwise trapped air to escape. Especially the placement of drain holes and the injection port may be done with care, so as to allow the silicone to flow easily through the cast and to make sure no air bubbles are trapped inside the cast. Failure to do so can result in the device having to be re-designed and re-created. As with the previously described example, the target operations used for input to the neural network may comprise the decision rules required to place e.g., the drain holes and the injection hole to allow the neural network to learn how and where such holes should be placed for specific device types according to given configuration requirements. Examples of the operations for creating a cast may be seen in Figures 22a to 22c, where Figure 22a illustrates a designed custom ear device 700, Figure 22b illustrates a cast 701 positioned around the designed custom ear device and Figure 22c illustrates the positioning of the injection hole 702 and drain holes 703 on the cast 701.

A further target operation that may be used for training the neural network is illustrated in Figure 23a, where an example operation for placing a sound bore 801 and sound entrance in the ear impression data 800 and subsequently is the produced custom ear device is illustrated. The soundbore is configured to carry sound from a speaker into the ear, through it and exits at the tip of the custom device, where the sound will travel towards the eardrum. The geometry of the soundbore can greatly affect the sound quality if there e.g., is too many twists and bends. The soundbore end in the tip of the canal, can also distort the sound if not places correctly so the sound can travel freely towards the eardrum. In solid devices the soundbore typically also have a component attached to the entrance. This is where the sound tube coming from the electronics is placed. The orientation of this component is interlinked since the electronics is placed behind the ear and the component must be orientated so it allows for the shortest sound tube between the electronics and the ear device. All of these considerations may be learned by trained neural network when using the target operation data as described in examples herein. Also in these cases, the target operations as loaded from the modelling tree to generate the training data may comprise point information associating points of the target operation to points of the ear impression data. This point associating is used in a pre-processing step to generate the input data for training a neural network configured to output e.g. the sound bore position prediction. In the case of the sound bore, the ear impression data may be configured in a pre-processing step as a voxel model, where voxels of the ear impression data associated with the soundbore is labelled with a first value and voxels not associated with the soundbore position is labelled with a second value. In this way, the training data set codes for the ground truth positioning of the sound bore and thereby the neural network can be trained to predict the sound bore positioning.

In a further example of target operations used to train the neural network reference is made to Figure 23b. Here an example target operation for placing a vent 901 in the ear impression data 900 is illustrated. As in the previously given examples, splines, planes or other operations may be used to define the optimal position of the vent. A vent is often positioned in a custom ear device to prevent over/under-pressure between the device and the eardrum. The vent may be carefully chosen and placed to not create a sound feedback loop, and care must be taken to ensure it does not collide with the sound bore. In this case, the preferred method is as described in relation to Figure 23a also using a voxel model to label the data in light of the target operation associated with the vent placing.

As is apparent from the description of the plurality of target operation example used for training the neural network, one or more target operations may be interlinked, as a position of e.g., a vent or soundbore may conflict and/or a given size may conflict with other target operations needed for a specified target configuration requirement. Accordingly, the target operation data may also comprise decision rules representing an interlink between one or more other target operations having different decision rules.

Furthermore, for all of the operations described above, may be understood that the target operations for training of the neural network may take the form of a data structure comprising one or more groups of vertices and/or edges on e.g., a mesh (being the ear impression data) which is collectively transformed to create a certain shape, cutting or other construction of the mesh. The operations may e.g., include to move a part of the mesh by e.g., a spline operation, twist a part of the mesh by an operation of twisting, rotating the mesh in a rotation operation etc. as a collective group of set of operations.

The set of operations described in relation to any of Figure 17 to 23c may form part of the target set of operations used to train the neural network. Accordingly, as previously mentioned the data contained in such operations, such as the mentioned splines, points, geometrical translations etc. may be used in generating a tensor to be used in a neural network as previously elaborated on. In addition, it should be understood that the set of operations just described is also the form of set of operations which the neural network, based on the training described herein, may be able to predict and output as a representation of a partially constructed custom ear device.

Furthermore, as previously mentioned, the set of target operations may comprise one or more production data comprises a plurality of production settings applied by the neural network (NN) to construct the partially and/or fully configured custom ear device. The production data may be configured as one or more of a 3D representation of for example the electronic components to be arranged in the custom ear device; placement rules for the electronic components; and production settings defining device materials and sizes. All these production data may be considered static as they rarely change but will vary a lot from device to device. Accordingly, the type of component may be device specific and accordingly may be interlinked with the configuration requirements. An example of production data may be configuration requirements that express the type of device is an in-the-ear shell. The production data may then comprise the minimal wall thickness, a CAD model and settings for a battery and/or faceplate component, a CAD model and settings for an amplifier to be placed inside the shell, a CAD model and settings for an earwax filter to be placed at the end of the sound tube exit and similar. Note this is not an exhaustive list, but a few examples.

The production data forming a part of the target operation data may thus represent production settings that the neural network needs to comply with to ensure for example correct minimal wall thickness, sufficient room for battery and amplifier inside the device, correct placement of the earwax filter etc. An example of production settings is provided in Figure 23c, where it is illustrated that a receiver may be placed in the custom ear device in accordance with such production settings. The production settings may include CAD component meshes for different types of devices and which may be used as input to the neural network for training thereof. For generating a training data set using e.g. CAD components, such as a receiver, the preferred method is to generate a voxel model representation of the ear impression data, and a loading the CAD component placement from the modelling tree to allow labelling of the voxels in the voxel model representation of the ear impression data with a first label if belonging to the CAD component placement. If the CAD component itself comprises several sides, the labelling method may be a multiclass labelling, where a first side of the CAD component is represented with a first value., a second side is represented with a second value, a third side is represented with a third value, and so forth in the voxel model representation of the ear impression data. In this each of the sides of a CAD model can be represented in the training data set and be associated with specific target voxels in the ear impression data.

As previously explained, different target operation data (i.e. target set of operations) may be used for training one or more different neural networks as illustrated in Figures 11 to 13. The target operations may e.g. be the operations explained about in relation to Figures 17 to 23c. In Figure 11 it is seen that a set of target operations Y is used for training one neural network. In another example illustrated in Figure 12, a set of target operations X may be used, where the set of target operations X is different from the set of target operations Y. Similarly, a further set of target operations Z is illustrated in Figure 13, where the set of target operations Z differs from the set of target operations X and Y. Each of the set of target operations represents design action and /or operations as previously described which the neural network may perform to reach the target partially and/or fully constructed 3D model. Each of the set of target operations X, Y, Z should be understood to be output from the final trained neural network as a representation of the operations that the trained neural network has predicted to design the partially 3D constructed custom ear device.

Turning now back to Figure 4, a further part of the training data set 108a may comprise the target partially and/or fully constructed 3D models 84a, 84b, 84c, 84d substantially interlinked with the previously described plurality of target operations. For one or more of the previously described target operations a corresponding target partially constructed 3D model is present in the training data set. That is, one of the target operations just describe may represent a target partially constructed 3D model in the training dataset. Alternatively, if one or more target operations are so interlinked that one cannot be performed without the other, such target operation, representing e.g., two operations described above, may be interlinked with a target partial constructed 3D model of the training data set. With this understanding it should also be understood that a target finally constructed 3D model may be a result of a plurality of target operations combined. In this way, the neural network may be trained on target operations representing partially constructed 3D models, thereby allowing the neural network to output not only the finally constructed custom ear device with all of the operations needed to reach such a design, but also to output the partial steps (i.e., the partial designed models) and the respective operations needed to reach such a partially constructed 3D model.

Figure 5 illustrates an example of a training dataset 108b used to allow the neural network to learn on its own target operations needed to reach either a partially and/or a fully constructed 3D model of a custom ear device. That is, in this example, the training data set does not comprise the target operations, and the neural network is configured to be trained only on the previously described ear impression data, target configuration requirements and the target partial and/or fully constructed 3D models of custom ear devices. The training data may, as illustrated in Figure 14 be stored in a storage database 1000. The full training data set 1000 containing target configuration requirements, target production rules, target decision rules, ear impression data, target partially and/or fully constructed 3D models may be split into a training data set 1001 and a validation data set 1002. The training data set may be configured to be used for training the neural network to predict partially and/or fully constructed 3D models and the respective predicted operations. The validation data may be configured to be used for validating the training neural network performance. That is the validation training data is not used to train the neural network but is only presented to the neural network after the training process has finished. This ensures that the neural network can be tested on known data to evaluate its performance.

From the previous described embodiment, it is apparent that the neural network described herein may be comprised of at least one neural network being trained on the different target data provided. It may also be that the neural network is configured as a plurality of neural networks which is trained on different target data, where the different target data represents especially different target partially constructed custom ear devices.

Accordingly, with reference to Figure 3, the network may be trained on target ear impression data 85 as the input. In the example illustrated in Figure 3, the neural network is further trained on the target configuration requirements and a final and/or a partially constructed 3D model 84. This allows the neural network to predict at least a partially constructed 3D model 86 as described herein and its corresponding set of operations 87. In the example shown in Figure 10, the input to the neural network may be so that the network is capable of predicting a fully constructed 3D model from an ear impression data, as the network may also be trained on such targets.

Turning to Figure 8 an example of a neural network structure trained on only partially constructed 3D models 185 is illustrated. In this case, the neural network takes as input a partially constructed 3D model and not as in the case of Figure 3 a target ear impression data. This type of network training allows the neural network to predict e.g., a second partially constructed 3D model 186 on the basis of a partially constructed 3D model 185 given as an input. This provides the possibility of a method applied by the design service application which allows a partially constructed 3D model as an input to the neural network of the design service application. In such case an ear impression data may not be needed and/or the neural network as trained in this manner may be used as a suitable network for outputting the partial set of target operations performed. The application of several neural networks used in a design application described herein may be better understood in the later description of such design application.

In an example illustrated in Figure 10, the neural network may also be trained on a final constructed 3D model 900 as input to allow the network to learn exactly how to design a finally constructed 3D model 901.

Turning now to Figure 15, an example of a neural network architecture 2000 and the data input 2001 to the neural network architecture is illustrated. The figure illustrates one neural network architecture 2000 according to convolutional neural networks mentioned herein, but other architectures could be chosen. The architecture is merely provided to give an example of the data structures 2001 which the neural network may take as an example use as input. To ensure that the neural network may effectively process the data used for input to the neural network, one or more data transformations may be performed to the input data as illustrated in box 2001 of Figure 15. In Figure 15 the input data is exemplified by different data structures, such as the order sheet 2002 corresponding to the illustration of Figure 24, component, and production settings 2003, component CAD meshes 2004, meshes 2005, feature strings 2007 and other possibilities not illustrated. Each of the input data may have different format and may be transformed into substantially the same format for processing by the neural network. Accordingly, in the example illustrated in Figure 15, order sheet 2002 may be transformed into a feature vector representation 2002a and subsequently into a matrix representation 2002b to match the matrix input of the ear impression data 2006. Correspondingly, the component and production settings 2003 may be transformed into a feature vector representation 2003a and subsequently a matrix transformation 2003b. The component CAD meshes 2004 is considered to be or at least contain a matrix transformation as the original data and no transformation is necessarily needed, as long as the data representation matches that of the ear impression data. If having feature strings 2007, such as text or other strings these may also be transformed into a feature vector representation 2007a and subsequently to a matrix representation 2007b. All of the transformed data may then be input to the neural network architecture 2000, where convolutional and pooling operations may be performed as previously described in relation to different neural networks. The data transformations illustrated herein may be considered to correspond to the previously described tensor generation, where the data is structured in a manner to allowing a neural network to efficiently handle the data.

In the previous sections, the neural network(s) according to the method and systems described herein and the training thereof has been described in detail. When the neural network has been trained (as described) it may be used in the method of automatically designing a custom ear device as previously touched upon. To recap, the method described herein uses the neural network to output at least one partially constructed 3D model together with a set of operations considered by the neural network as previously elaborated on in connection with Figure 1. In a step of the method as previously described a user instruction may cause the processor to perform changes to the partially constructed 3D model output from the neural network and produce based on those changes a finally constructed 3D model and final operations datafile representing the operations performed to create the finally constructed 3D model. That is, in an example, the final operations datafile comprises production instructions configured to be utilized for manufacturing the finally constructed custom ear device.

In an example, the final operations datafile comprises the set of operations as applied by the neural network in designing the partially constructed 3D model of the custom ear device, and a set of operations performed by the processor in accordance with the user instructions. In this way both the partial design obtained by the neural network and the changes made by e.g., a skilled professional is reflected in the final operations datafile.

In an example of the method and system described herein, the method may further comprise receiving a user instruction causing the processor to feed the changed partially constructed 3D model or the representation thereof in the form of set of operations into at least one other neural network as illustrated in Figure 7. Figure 7 represents the method and system described in relation to Figure 1, why the numbering is considered substantially the same for at least corresponding features common to Figure 1 and Figure 7. Figure 7 illustrates the further method steps of receiving a second user instruction 16 causing the processor to feed the changed partially constructed 3D model or the representation thereof in the form of set of operations into at least one other neural network 17. The one other neural network 17 is configured to output an updated partially constructed 3D model of a custom ear device based on at least another set of operations than the ones applied for the first neural network, wherein the one other neural network 17 is trained on the basis of a second training data set comprising at least a set of target operations corresponding to a specific partially and/or fully constructed 3D model. That is the other neural network 17 described in relation to Figure 7 may e.g., be trained in accordance with any of the neural networks described herein, as long as the one other neural network 17 preferable differs from the trained neural network 8 in how these networks 8, 17 was trained. In an example, the neural network 8 may be trained in accordance with e.g., the neural network illustrated in Figure 8 and 12 as these take as input the ear impression data and is configured to output at least one partially constructed 3D model. The at least one other neural network may be configured to be trained in accordance with e.g., Figures 10 and 13, which described the training of a neural network, where the neural network is configured to output the finally constructed 3D model based on e.g., a partial input. As seen in Figure 7, the one other neural network may also be configured to output a finally constructed custom ear device 18 and its corresponding updated operations datafile 19 corresponding to the changes made in the first user instruction 13 and the outcome of the second neural network 17.

In another example (not illustrated in more detail), the changed partially constructed 3D model may be feed into at least one algorithmic operation different from the set operations applied by the neural network, wherein the algorithmic operation is configured to output an updated partially constructed 3D model and/or the finally constructed 3D model.

In any of the described examples, the neural network may at least output the set of operations required to generate a partial constructed 3D model. That is, in one example, the neural network may feed the set of operations into a further step of the method, wherein the further step allows a processor to utilize the set of operations to apply the modifications to the ear impression data as coded by the set of operations. Turning now to Figure 6, a fast forward application forming part of the method and design service system described herein will be explained in more detail. That is Figure 6 illustrates in an exemplified manner a design service application corresponding to the system 1 described herein and configured to execute by a processor the method described. The design service application as illustrated in Figure 6 is configured to perform the method by reading into the processor the ear impression data 5 and the configuration requirements 6 as previously elaborated on based on a user instruction. The ear impression data 5 and configuration requirements is feed into a neural network 8 which may be configured to output at least one partially constructed 3D model 20, preferably at least two partially constructed 3D models 21 and/or even more partially constructed 3D models 20, 21, 22, 23, 24, 25 as illustrated in Figure 6. As illustrated in Figure 6, each of the partially constructed 3D models or representations thereof in for example in the form of the set of operations coding for the changes done to reach to partially constructed 3D models 20, 21, 22, 23,24 as output from the neural network 6 also comprises a set of operations different from each other, as the partial models 20, 21, 22, 23,24 represents different operations applied by the neural network 8. The different operations for each of the plurality of partially constructed 3D models 20, 21, 22, 23,24 are denoted set of operations 1, 2, 3, 4 and 5 as seen in Figure 6. As also illustrated in Figure 6, the neural network 8 may also output a fully constructed 3D model 25 with is final set of operations.

Turning to Figure 25, the system running the method described herein is illustrated in a more exemplified manner to illustrate examples of the set of operations which may be output from the neural network and used in a fast forward situation where a user is allowed to modify a partial or fully constructed model by changing one or more of the set of operations as output from the neural network. That is, Figure 25 illustrates, similarly, to Figure 6, that an ear impression data 5 together with the corresponding configuration requirements 6 may be input to a trained neural network 8. The neural network 8 is configured to output a plurality of set of operations as illustrated as a first set of operations 20a, second set of operations 21a, third set of operations 22a and a fourth set of operations 23a. As seen in Figure 25, the first to fourth set of operations may be represented as a spline 20a, 21a, 23a or e.g., a placement of e.g., a receiver 22a. All other possibilities described herein applies equally with this understanding and the set of operations shown in Figure 25 is merely for illustrative example. Many other types of operations may be output from the neural network as in apparent throughout the description. Further, it should be noted that Figure 25 illustrates the set of operations 20a, 21a, 22a, 23a as applied to a 3D model of the ear impression data. However, the neural network need not to output the 3D model but may only output the datafile containing the set of operations coding for the geometrical transformations that should be applied to the ear impression data to generate the final custom ear device. In other words, Figure 25 generally illustrates the neural network engine 8a described herein comprising at least one neural network 8 configured to output at least one partially and/or fully constructed representation 20a, 21a, 22a, 23a of a custom ear device.

The outputs 20a, 21a, 22a, 23a may be stored in a processor so as to be used in a fast forward engine if needed. The system and method described herein is configured to support a user flow, loading into the processor the ear impression data and the configurations requirements wherein the neural network 8 is configured to output a fully constructed custom ear device 25, as illustrated in Figure 26. The output (i.e. the fully constructed custom ear device 25) may be displayed in a graphical user interface 3000 comprising one or more interactive elements 3001 (here exemplified as a hand), wherein upon receiving a user input may activate an interactive element 3001, the system is configured to cause the processor (previously described as no. 2 in other figures), to display the fully constructed custom ear device 25 with the representation of the final operations datafile. The final operations datafile preferably comprises the set of operations exemplified in e.g., Figure 25 and Figure 6. The processor 2 is configured to respond to a user input, to one of the interactive elements 3001, by executing instructions causing at least one partially constructed representation with its corresponding set of operations to be displayed. Such activation may cause the processor to modify the user interface (illustrated with arrow 3002 in Figure 26) to change display from the current view in one of the displays 3000 to an updated view 3003, wherein the partially constructed represented of the custom ear device may be displayed. In other words, as seen in Figure 26, when the user activates e.g. an area or zone on the fully constructed 3D model 25, such as the end of the receiver 23b, the processor may be instructed to load and render on the display of the user interface the set of operations 23a output from the neural network 8, for that operation as predicted by the neural network 8. Furthermore, as illustrated in Figure 26, when the user activates via a user interaction element 3001, e.g., an area on the fully constructed representation of the custom ear device, the processor may in the background (without direct display) execute a fast forward engine 3004, which is configured to applying all other set of operations to the ear impression data except for the operations associated with the activated area. Accordingly, all operations, which acceptable output from the neural network may stay unchanged, while the user may apply manual corrections to unsatisfactory output form the neural network. It is an advantage, that an operator may inspect the automatically generated models and are able to provide swift modifications to the output without the need to re-run the method from the start.

The process of editing the area, such as the end of the receiver 23b, exemplified in Figure 26, as the operation 23a, may be performed by an editing engine illustrated as a dotted box 3005 in Figure 26. The editing engine is configured to receive the partially constructed representation of the custom ear device with the set of operations 23 a from the fast forward engine 3004 and to receive a user instruction representing changes applied to the partially constructed representation of the custom ear device.

The editing engine 3005 may, as illustrated in Figure 26, perform changes to the partially constructed representation model in accordance with the user instruction, and output an updated finally constructed custom ear device and a final operations datafile.

In other embodiments (not illustrated in detail) the editing engine may also perform the changes according to the user instructions and feed the changes into a new neural network and/or other algorithms which then applies the changes made by the user to update the design to create an updated fully constructed 3D model of a custom ear device.

In another related example, illustrated in Figure 27, the processor of the system may be configured to generate a toggle menu 4000 together with the final constructed custom ear device 25 and to display the generated toggle menu in the user interface, wherein the toggle menu, upon activation, comprises a representative string 20b, 21b, 22b, 23c, 24b for each of the one or more set of operations generated by the neural network. This is illustrated in one example in Figure 27, where the toggle menu 4000 comprises five set of operations 20b, 21b, 22b, 23c, 24b. Each of the set of operations may be activated or deactivated by a user, wherein a deactivation or activation causes the processor to update the current view of the custom ear device with a removal of the set of operation in case of a deactivation or a displaying of the set of operation in case of activation of the respective set of operation. The word representative string used herein may correspond to the previously mentioned active field. That is the active field may be e.g., a text string, a box, an object or similar field, which may be activated or deactivated upon receiving a user instruction to the field.

The toggle menu may also in an example illustrated in Figure 27 comprise a string 25a, 25b, representing the left 5a, 25a or right ear 5b, 25b, wherein the processor of the system upon activation of one or more of the left 25a, and right 25b string is configured to activate or deactivate in the current view an editing mode enabling the editing engine to receive user inputs representing modifications to the activated left or right view.

In accordance herewith a current view of the user interface may be configured as a split view, where in a first side of the split view a left ear impression data is configured to be displayed, and in a second side of the split a right ear impression data is configured to be displayed.

In one example, the editing engine may be configured to perform simultaneous changes to both split views in the current display. Furthermore, the editing engine may be configured to mirror the set of operations applied to the left ear impression data as instructed to the processor from a user input to the right ear impression data. Accordingly, the method and system may be configured to automatically mirror changes made to one ear impression data in a first side of the current view to do similar change automatically in the other ear impression data in a second side of the current view. This may speed up the process of applying manual adjustment the models, as an unsatisfactory output form the neural network may be modified on both models simultaneously. Accordingly, in the system and method described herein the method may comprise displaying, in a current view of a user interface (not illustrated in further detail), the finally constructed custom ear device 25 with a representation of the final operations datafile belonging to the finally constructed custom ear device 25. This allows the user of a system performing the method described herein to evaluate the finally constructed custom ear device as it was output from the neural network and to judge if changes need to be made. Accordingly, the method described in examples herein is further configured to receiving, while in the current view of the user interface, a user input requesting that the at least one partially constructed custom ear device (that is at least one of the partially constructed custom ear devices 20, 21, 22, 23, 24, 25) with a representation of the corresponding set of operations, is to be displayed. This allows the user to jump to partially constructed custom ear device which the user then may modify by providing a user input. Accordingly, the method and system running the method may be configured to modifying, in response to the user input, the current view of the user interface to display the at least one partially constructed custom ear device with the representation of the corresponding set of operations. In other words, the method and system described herein is configured to jump from a current view e.g., being the fully constructed 3D model to a second view being e.g., a partially constructed 3D model when a user instruction is received by the processor. The set of operations applied to the partially constructed ear device may be represented visually in the user interface such as e.g., where a receiver is placed, where a tip cut is placed and so forth, which will be explained in more detail in connection with the drawings. In this way, a fast- forward process 30 may be considered applied by the method described herein, as the operations steps (e.g., steps 20, 21 in Figure 6) preceding the chosen partially constructed custom ear device (e.g., 22 in Figure 6) displayed in the user interface have already been applied automatically.

In an example, the received user instruction for changing the current view into an updated view as previously described may be configured as marking an area and/or pointing to an area on the fully constructed custom ear device, causing the processor to modify the current view of the user interface to display a partially constructed custom ear device corresponding to the set of operations applied to the marked and/or pointed at area. As illustrated in Figure 6, and also previously elaborated on, the receiving of a user instruction 13 causing the processor to perform changes to the partially constructed 3D model in accordance with the user instruction may trigger to feed the changed partially constructed 3D model into one other neural network or algorithmic operations and/or to perform manually changes 31 needed to reach an updated finally constructed ear custom device and an updated final configuration data.

As already described, the method described herein may be configured as an application that is run in connection with a system, such as the design service application described herein. Accordingly, a system for data processing, comprising means for executing the method is disclosed herein. The system may comprise a processor configured to receive one or more user instructions causing the processor to control the execution of a neural network engine comprising at least one neural network configured to output at least one partially and/or fully constructed 3D model of a custom ear device and trained as previously described. The system may comprise a fast forward engine, as described in an example relating to Figure 6, wherein the fast forward engine is configured to receive the output from the neural network engine and to output at least one partially constructed 3D model and a set of operations applied by the neural network. The fast forward engine may be in communicatively contact with an editing engine configured to receive the partially constructed 3D model and the set of operations from the fast forward engine and to receive a user instruction representing changes applied to the partially constructed 3D model. Details of the fast forward engine and its relation to the neural network is considered elaborated on in relation to for example Figure 6 and in an exemplified manner in relation to Figures 25 to 27. The editing engine may be configured to receive a user instruction causing the processor to perform changes to the partially constructed 3D model in accordance with the user instruction, and to output a finally constructed custom ear device and a final operations datafile.

The system 1, such as the design service application described herein and illustrated in Figures 1, 7 and 9 may comprise in one example a processor 2 that is configured to be operatively connected with a storage engine 71, wherein the storage engine 71 is configured to store thereon one or more ear impression data 5 and one or more corresponding configuration requirements 6. The storage engine may also be configured to store thereon all the target data used for training the neural network.

The storage engine 71 may be configured as a cloud server configured to receive one or more ear impression data 5 and corresponding one or more configuration requirements 6 from one or more remote servers 72, 73. The remote servers 72, 73 may represent e.g., a clinical site, such as an audiologist site and/or a retail site, wherein the ear impression data 5may be gathered together with the configuration requirements 6. Furthermore, the remote server may represent a manufacturing site, where ear impression data 5 may also be stored together with the configuration requirements 6 for specific ear impression data. In any case, the design service application (i.e., the system 1) may be configured to communicate directly with the server sites either directly or via a cloud storage engine to allow efficient sharing of data between an audiology/retail site and/or a manufacturing site.

In accordance herewith, the storage engine 71 may be configured to store thereon at least the finally constructed custom ear device and the final operations datafile to allow especially a manufacturing site to gather the information needed to manufacture the finally constructed custom ear device that has been automatically or at least partly automatically designed by the design service application.

Disclosed herein is also a computer program product embodied in a non-transitory computer readable medium comprising computer readable program code configured to be executed by a hardware processor to cause the hardware data processor to perform the methods described herein when the computer readable program code is executed by the hardware data processor.

According to examples described herein, the electronic hardware may include microprocessors, microcontrollers, digital signal processors (DSPs), field programmable gate arrays (FPGAs), programmable logic devices (PLDs), gated logic, discrete hardware circuits, and other suitable hardware configured to perform the various functionality described throughout this disclosure. Computer program shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executables, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.

Although some embodiments have been described and shown in detail, the disclosure is not restricted to such details, but may also be embodied in other ways within the scope of the subject matter defined in the following claims. In particular, it is to be understood that other embodiments may be utilized, and structural and functional modifications may be made without departing from the scope of the present invention.

Benefits, other advantages, and solutions to problems have been described herein with regards to specific embodiments. However, the benefits, advantages, solutions to problems, and any component(s)/ unit(s) that may cause any benefit, advantage, or solution to occur or become more pronounced are not to be construed as critical, required, or essential features or components/ elements of any or all the claims or the invention. The scope of the invention is accordingly to be limited by nothing other than the appended claims, in which reference to a component/ unit/ element in the singular is not intended to mean “one and only one” unless explicitly so stated, but rather “one or more.” A claim may refer to any of the preceding claims, and “any” is understood to mean “any one or more” of the preceding claims.

It is intended that the structural features of the devices described above, either in the detailed description and/or in the claims, may be combined with steps of the method, when appropriately substituted by a corresponding process.

As used, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well (i.e., to have the meaning “at least one”), unless expressly stated otherwise. It will be further understood that the terms “includes,” “comprises,” “including,” and/or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. It will also be understood that when an element is referred to as being “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, but an intervening element may also be present, unless expressly stated otherwise. Furthermore, “connected” or “coupled” as used herein may include wirelessly connected or coupled. As used herein, the term “and/or" includes any and all combinations of one or more of the associated listed items. The step of any disclosed method is not limited to the exact order stated herein, unless expressly stated otherwise.

It should be appreciated that reference throughout this specification to "one embodiment" or "an embodiment" or “an aspect” or features included as “may” means that a particular feature, structure or characteristic described in connection with the embodiment is included in at least one embodiment of the disclosure. Furthermore, the particular features, structures or characteristics may be combined as suitable in one or more embodiments of the disclosure. The previous description is provided to enable any person skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects.

The claims are not intended to be limited to the aspects shown herein, but is to be accorded the full scope consistent with the language of the claims, wherein reference to an element in the singular is not intended to mean “one and only one” unless specifically so stated, but rather “one or more.” Unless specifically stated otherwise, the term “some” refers to one or more.

Claims

1. A computer-implemented method for designing a custom ear device, comprising

- loading into a processor an ear impression data, wherein the ear impression data is acquired by an optical scanner;

- generating a 3D digital representation of the ear impression data;

- loading into the processor one or more configuration requirements for designing a custom ear device matching the ear impression data;

- receiving a user instruction causing the processor to load the ear impression data and the one or more configuration requirements into a neural network (NN) structure, wherein the NN structure comprises a plurality of neural networks trained on a plurality of different target operations, wherein the method comprises a step of training each of the plurality of neural networks on the basis of training data sets comprising: a plurality of training ear impression data wherein the training ear impression data is acquired by an optical scanner; a plurality of target partial constructed representations of custom ear devices each complying with a plurality of target configuration requirements; and one or more target operations specific to the plurality of target partially constructed representations and complying with at least a part of the one or more target configuration requirements;

- receiving an output from each of the plurality of neural networks, wherein eachoutput comprises at least one partially constructed representation of the custom ear device comprising at least a set of operations as predicted by the neural network, and complying with at least a part of the one or more configuration requirements, wherein the set of operations forms part of a datafile comprising 3D interaction operations configured to be applied to the ear impression data;

- receiving a user instruction causing the processor to perform changes to at least one of the set of operations of the partially constructed representation in accordance with the user instruction, and - outputting a finally constructed ear custom device, and a final operations datafile wherein the final operations datafile comprises production instructions configured to be utilized for manufacturing of the finally constructed custom ear device.

2. The computer-implemented method according to claim 1, further comprising converting the ear impression data into a file format configured as a voxel model or a point cloud representation.

3. The computer-implemented method according to claim 1, wherein the production instructions comprise 3D printer instructions configured to be applied by a 3D printer for manufacturing the custom ear device.

4. The computer-implemented method according to claim 1 further comprising outputting from the neural network an operation file containing information coding for the set of operations; and

- modifying the ear impression data in a modification procedure using the operation file.

5. The computer-implemented method according to claim 1, wherein the set of operations comprises geometrical transformations, wherein the method comprises applying the geometrical transformations to points, areas, facets or vertices of the ear impression data to form at least the partial constructed representation of the custom ear device.

6. The computer-implemented method according to claim 1, further comprising performing a movement operation by translating a series of points of the ear impression data with pointspecific values, wherein the movement operation forms part of the datafile.

7. The computer-implemented method according to claim 1, wherein the target set of operations comprises geometrical transformations configured to be applied to the one or more points, areas, facet and/or vertices of the training ear impression data.

8. The computer-implemented method according to claim 1, wherein the training comprises - generating one or more sets of 3D vectors from the ear impression data along with zero, one or more numerical feature descriptors for each point of the 3D vector, and

- generating a feature tensor from the one or more sets of 3D vectors.

9. The computer-implemented method according to claim 8, further comprising

- estimating the feature descriptors by identifying a value of the ear impression data representing an operation step described by the estimated feature descriptor.

10. The computer-implemented method according to claim 9, wherein the target set of operations comprises CAD component placements, wherein the method comprises loading a rotation matrix associated with an operation and generating a feature tensor with the rotation matrix.

11. Computer-implemented method according to any of the previous claims, wherein the final operations datafile comprises the set of operations as applied by the neural network structure in designing the partially constructed representation of the custom ear device, and a set of operations performed by the processor in accordance with the user instructions.

12. Computer-implemented method according to any of the previous clams, the method comprising:

- receiving a user instruction causing the processor to feed the changed partially constructed representation into at least one other neural network, wherein the at least one other neural network is configured to output an updated partially constructed representation of a custom ear device based on at least another set of operations, wherein the one other neural network is trained on the basis of a second training data set comprising at least a set of target operations corresponding to a specific partially and/or fully constructed representation.

13. The computer-implemented method according to claim 1, comprising

- receiving a user instruction causing the processor to feed the changed partially constructed representation into at least one algorithmic operation different from the set operations applied by the neural network structure, wherein the algorithmic operation is configured to output an updated partially constructed 3D model and/or the finally constructed representation.

14. The computer-implemented method according to any of the previous claims, wherein the one or more configuration requirements comprises a specified product type and design requirements for the specified product type as required by an audiologist and/or by a manufacture of custom ear devices through the configuration data.

15. Computer-implemented method according to any of the previous claims, wherein the set of operations comprises:

- one or one or more set of decision rules,

- one or more production data, wherein each of the one or more decision rules and/or production data provides instructions for production of the partially and/or fully constructed representation of a custom ear device as output from the neural network.

16. Computer-implemented method according to claim 15, wherein the one or more set of decision rules comprises data representing operations applied by the neural network (NN) to construct the partially and/or finally constructed representation of the custom ear device to fit the anatomy of an ear canal of a user.

17. Computer-implemented method according to claim 7, wherein the production data comprise one or more of

- a 3D representation of the electronic components to be arranged in the custom ear device;

- placement rules for the electronic components; and

- production settings defining device materials and sizes.

18. Computer-implemented method according to claim 15, wherein the one or more set of decision rules comprises at least one or more set of operations representing:

- shaping a tip of the ear impression data, wherein the tip is configured to face the inside of an ear canal when inserted therein; - cutting a part of the ear impression data configured to face the outside of an ear canal when inserted therein;

- shaping the outer shape of the ear impression data;

- placing of the electronic components;

- casting for a soft silicone production of the digital ear impression

- placing of a sound bore;

- placing of a vent canal.

19. The computer-implemented method according to any of the previous claims, further comprising:

- displaying, in a current view of a user interface, the finally constructed custom ear device with a representation of the final operations datafile;

- receiving, while in the current view of the user interface, a user input requesting that the at least one partially constructed representation of the custom ear device with a representation of the corresponding set of operations, is to be displayed; and

- modifying, in response to the user input, the current view of the user interface to display the at least one partially constructed representation of the custom ear device with the representation of the corresponding set of operations.

20. The computer-implemented method according to claim 19, wherein the received user instruction is configured as marking an area and/or pointing to an area on the fully constructed custom ear device, causing the processor to modify the current view of the user interface to display a partially constructed representation of the custom ear device corresponding to the set of operations applied to the marked and/or pointed at area.

21. The computer-implemented method according any of claim 19 and 20, comprising:

- receiving a user instruction causing the processor to perform changes to the partially constructed representation of the custom ear device in accordance with the user instruction, and to output an updated finally constructed ear custom device and an updated final operations datafile.

22. A system configured for executing a method for constructing a custom ear device , the system comprising: a processor configured to receive one or more user inputs causing the processor to:

- loading an ear impression data, wherein the ear impression data is acquired by an optical scanner and generating a 3D digital representation of the ear impression data, and

- controlling based on the user input, the execution of:

- a neural network engine comprising at least one neural network structure comprising a plurality of neural networks trained on a plurality of different target operations and configured to receive as input, the ear impression data and provide as output at least one partially constructed representation of a custom ear device;

- a fast forward engine configured to receive the output from the neural network engine and to output at least one partially constructed representation of the custom ear device comprising at least a set of operations output by the neural network and complying with at least a part of one or more configuration requirements; and

- an editing engine configured to receive the partially constructed representation of the custom ear device with the set of operations from the fast forward engine and to receive a user input representing changes applied to the partially constructed representation of the custom ear device.

23. The system according to claim 22, wherein the training of the neural network forming part of the system comprises training each of the plurality of neural networks on the basis of a training data sets comprising: a plurality of training ear impression data wherein the training ear impression data is acquired by an optical scanner; a plurality of target partial constructed representations of custom ear devices each complying with a plurality of target configuration requirements; and one or more target operations specific to the plurality of target partially constructed representations and complying with at least a part of the one or more target configuration requirements.

24. The system according to claim 1 wherein the editing engine is configured to receive a user input causing the processor to perform changes to the partially constructed representation model in accordance with the user input, and to output a finally constructed custom ear device and a final operations datafile.

25. The system according to claim 22, wherein the editing engine is configured to

- receive a user input causing the processor to perform changes to the partially constructed representation in accordance with the user instruction;

- output an updated partially constructed representation with an updated set of operations;

- feeding the updated partially constructed representation into another neural network and/or an algorithmic operation.

26. The system according to claim 22, wherein the processor is configured to be operatively connected with a storage engine, wherein the storage engine is configured to store thereon one or more ear impression data and one or more corresponding configuration requirements.

27. The system according to claim 26, wherein the storage engine is configured as a cloud server configured to receive one or more ear impression data and corresponding one or more configuration requirements from one or more remote servers.

28. The system according to claim 27, wherein the storage engine is configured to store thereon at least the finally constructed custom ear device and the final operations datafile.

29. The system according to any of the previous claims, wherein the system comprises a graphical user interface configured to operatively communicate with the editing engine, wherein the user interface comprises one or more interactive elements, wherein upon activating an interactive element of the one or more interactive elements, the editing engine is configured to cause the processor to:

- displaying, in a current view of the user interface, the finally constructed representation of the custom ear device with a representation of the final operations datafile;

- receiving, while in the current view of the user interface, a user input requesting that the at least one partially constructed representation of the custom ear device with a representation of the corresponding set of operations, is to be displayed; and - modifying, in response to the user input, the current view of the user interface to display the at least one partially constructed representation of the custom ear device with the representation of the corresponding set of operations.

30. The system according to claim 29, wherein the received user input is configured as marking an area and/or pointing to an area on the fully constructed representation of the custom ear device, causing the editing engine to modify the current view of the user interface to display a partially constructed representation of the custom ear device corresponding to the set of operations applied to the marked and/or pointed at area.

31. The system according to any of claims 29 and 30, comprising:

- receiving by the editing engine, the user input causing the editing engine to perform changes to the partially constructed representation of the custom ear device in accordance with the user input, and; to output an updated finally constructed ear custom device and an updated final operations datafile.

32. System according to any of the previous claims, wherein the user input is configured as a click of a mouse communicatively connected with the user interface and/or a touch on display of the user interface.

33. System according to any of the previous claims, wherein the editing engine is configured to generate a toggle menu together with the final constructed custom ear device and to display the generated toggle menu in the user interface, wherein the toggle menu comprises a representative active field area for each of the one or more set of operations as output from the neural network.

34. System according to claim 33, wherein the editing engine is configured to receive a user input instructing a deactivation and/or an activation of one or more of the representative strings for each of the set of operations, wherein a deactivation or activation causes the processor to update the current view of the custom ear device with a removal of the set of operation in case of a deactivation or a displaying of the set of operation in case of activation of the respective set of operation.

35. System according to any of the previous claims, wherein the toggle menu comprises an active field area representing a left and right ear impression data, wherein the editing engine upon activation of at least one of the active field area representing the left and right ear impression data is configured to activate or deactivate in the current view of the user interface, an editing mode enabling the editing engine to receive user inputs representing modifications to an activated left or right view independently.

36. System according to any of the previous claims, wherein the current view of the user interface may be configured as a split view, where in a first side of the split view a left ear impression data is configured to be displayed, and in a second side of the split view a right ear impression data is configured to be displayed.

37. System according to any of the previous claims, wherein the editing engine is configured to automatically mirror changes made to one ear impression data in a first side of the current view to do similar changes automatically in the other ear impression data in a second side of the current view.