KR102672239B1 - Method, device and recording medium for displaying orthodontic diagnosis images using landmarks - Google Patents

Abstract

According to one embodiment, the positions of a plurality of landmarks are determined using a 3D image, and the positions of reference points for bilateral landmarks among the plurality of landmarks are determined and provided in the 2D image, so that the land determined in the 3D image is determined. A method is disclosed that facilitates real-time confirmation of where a mark is located in a 2D image and enables more intuitive analysis of the locations of left and right landmarks of bilateral structures.

Description

Method for displaying orthodontic diagnostic images using landmarks, device and recording medium thereof {METHOD, DEVICE AND RECORDING MEDIUM FOR DISPLAYING ORTHODONTIC DIAGNOSIS IMAGES USING LANDMARKS}

The present disclosure relates to a method, device, and recording medium for displaying images used for orthodontic diagnosis using landmarks.

In general, 2D analysis methods are used to diagnose patient problems when undergoing orthodontic treatment. Recently, various equipment technologies, such as CBCT (cone beam computed tomography), have been developed to precisely photograph 3D data of the patient's treatment area. However, 3D analysis methods that use 3D data to diagnose patient problems still lack research data compared to 2D analysis methods.

Accordingly, there is a need for technology to solve the above-mentioned problems and utilize 3D data with high data accuracy to diagnose orthodontic patients, while providing efficient and reliable analysis results by applying 2D analysis methods accumulated over a long period of time.

Korean Patent Publication No. 10-2017-0007309 (2017.01.18) Method for 3-D head radiography analysis

An embodiment of the present disclosure is intended to solve the problems of the prior art described above, and provides a method, device, and record that can efficiently provide analysis results by performing analysis according to a 2D analysis method based on 3D data during correction diagnosis. We want to provide media.

The technical challenges to be solved are not limited to the technical challenges described above, and may further include various technical challenges within the scope apparent to those skilled in the art.

A method of displaying an image according to a first aspect of the present disclosure includes acquiring a 3D image of an object; determining the positions of a plurality of landmarks included in the 3D image; Specifying the location of a bilateral landmark, which is a landmark for a bilateral structure, among the determined plurality of landmarks; determining a location of a reference point for the bilateral landmark based on the location of the designated bilateral landmark; and displaying a 2D image of the object in which reference points for the plurality of landmarks and the bilateral landmarks are displayed.

In addition, determining the position of the reference point with respect to the bilateral landmark may include determining a first bilateral landmark and a second bilateral landmark constituting the bilateral landmark; determining an average position for the positions of the first bilateral landmarks and the positions of the second bilateral landmarks; and determining the position of the reference point to correspond to the average position.

Additionally, the positions of reference points for the plurality of landmarks and the bilateral landmarks displayed in the 2D image may be determined using at least one of a perspective projection method and an orthographic projection method.

In addition, determining the positions of the plurality of landmarks may include determining the plurality of landmarks that are subject to position determination among all landmarks for the object; Providing a message confirming the location of a target landmark that is the target of location determination at the current stage among the plurality of landmarks; and determining the location of the target landmark based on the response to the message.

In addition, determining the location of the target landmark may include determining a location on the one side of the 3D image based on a user input for the message obtained for one side of the 3D image; determining a location on the other side of the target landmark based on attributes of the target landmark; and determining the location of the target landmark based on the location on the one side and the location on the other side.

In addition, the step of determining the location of the target landmark involves displaying the positions of each of the first bilateral landmark and the second bilateral landmark constituting the bilateral landmark on the 3D image to display different locations within one screen. simultaneously displaying in windows; and determining the location of the first bilateral landmark or the second bilateral landmark based on user input received through the different windows.

Additionally, determining the positions of the plurality of landmarks may include determining the positions of the plurality of landmarks included in the 3D image using previously acquired training data.

In addition, determining the positions of the plurality of landmarks may include displaying the positions of the plurality of landmarks determined through the learning data on the 3D image; and updating the positions of the plurality of landmarks based on user input received through the 3D image.

In addition, the 3D image and the 2D image are displayed simultaneously in distinct areas, and the plurality of landmarks displayed in the 2D image according to the positions of the plurality of landmarks determined based on user input for the 3D image. The location can be updated in real time.

Additionally, the 2D image includes at least one of a lateral image, a front image, a panoramic image, and an SMV image for the object, and the reference point may be displayed in a different manner from the plurality of landmarks.

A device for displaying an image according to a second aspect of the present disclosure acquires a 3D image of an object, determines the positions of a plurality of landmarks included in the 3D image, and locates a bilateral structure among the determined plurality of landmarks. a processor that specifies the location of a bilateral landmark, which is a landmark for the bilateral landmark, and determines a location of a reference point for the bilateral landmark based on the location of the designated bilateral landmark; and a display that displays a 2D image of the object on which reference points for the plurality of landmarks and the bilateral landmarks are displayed.

In addition, the processor determines a first bilateral landmark and a second bilateral landmark constituting the bilateral landmark, and determines the location of the first bilateral landmark and the location of the second bilateral landmark. The average position can be determined, and the position of the reference point can be determined to correspond to the average position.

Additionally, the positions of reference points for the plurality of landmarks and the bilateral landmarks displayed in the 2D image may be determined using at least one of a perspective projection method and an orthographic projection method.

In addition, the processor determines the plurality of landmarks that are the target of location determination among all landmarks for the object, and determines the location of the target landmark that is the target of location determination at the current stage among the plurality of landmarks. A message may be provided, and the location of the target landmark may be determined based on the response to the message.

In addition, the display displays the positions of each of the first and second bilateral landmarks constituting the bilateral landmarks on the 3D image and simultaneously provides them in different windows within one screen, and the processor may determine the location of the first bilateral landmark or the second bilateral landmark based on user input received through the different windows.

Additionally, the processor may determine the positions of the plurality of landmarks included in the 3D image using previously acquired learning data.

A third aspect of the present invention can provide a computer-readable recording medium recording a program for executing the method according to the first aspect on a computer. Alternatively, the fourth aspect of the present invention may provide a computer program stored in a recording medium to implement the method according to the first aspect.

According to one embodiment, a 3D image used to input a landmark location and a 2D image extracted from the 3D image can be provided at the same time. Accordingly, the user can check in real time where the landmark determined in the 3D image is located in the 2D image.

In addition, by providing reference points for bilateral landmarks in 3D images and 2D images, it is possible to more intuitively analyze the locations of the left and right landmarks of bilateral structures.

Additionally, even if a landmark is entered only once in a 3D image, not only can a reference point be automatically derived, but multiple 2D images can also be automatically provided. Additionally, because the location of the landmark is input through a 3D image, the location of the landmark can be determined more accurately.

Additionally, user convenience can be improved by displaying the landmark positions of the 3D image and the landmark positions of the 2D image in real time.

The effects of the present invention are not limited to the effects described above, and should be understood to include all effects that can be inferred from the configuration of the invention described in the detailed description or claims of the present invention.

1 is a block diagram showing an example of the configuration of a device according to an embodiment.
FIG. 2 is a diagram illustrating an operation in which a device receives a user input for a landmark, according to an embodiment.
FIG. 3 is a diagram illustrating an operation in which a device acquires a 2D image from a 3D image displaying a plurality of landmarks and reference points according to an embodiment.
FIG. 4 is a diagram illustrating an operation in which a device provides analysis results for a plurality of landmarks displayed in a 2D image, according to an embodiment.
Figure 5 is a flowchart showing a method by which a device displays an image, according to an embodiment.

The terms used in the embodiments are general terms that are currently widely used as much as possible while considering the function in the present invention, but this may vary depending on the intention or precedent of a person working in the art, the emergence of new technology, etc. In addition, in certain cases, there are terms arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the relevant invention. Therefore, the terms used in the present invention should be defined based on the meaning of the term and the overall content of the present invention, rather than simply the name of the term.

When it is said that a part “includes” a certain element throughout the specification, this means that it does not exclude other elements, but may further include other elements, unless specifically stated to the contrary. In addition, “…” stated in the specification. Terms such as “unit” refer to a unit that processes at least one function or operation, which may be implemented as hardware or software, or as a combination of hardware and software.

Below, with reference to the attached drawings, embodiments of the present invention will be described in detail so that those skilled in the art can easily implement it. However, the present invention may be implemented in many different forms and is not limited to the embodiments described herein.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

FIG. 1 is a block diagram illustrating an example of the configuration of a device 100 according to an embodiment.

Referring to FIG. 1, device 100 may include a processor 110 and a display 120.

The processor 110 may acquire a 3D image of the object. Here, the object may include a body part (eg, head) of the patient. In one embodiment, the 3D image may be a computed tomography (CT) image obtained based on a tomography method, but is not limited thereto. For example, the 3D image may be a 3D cone beam CT (CBCT) image that includes information about the internal space, density, and multiple cross-sections of a portion of the patient's teeth. As another example, the 3D image may be a DICOM (digital imaging and communications in medicine) digital image or various other types of medical images. In one embodiment, the 3D image may include a 3D CT image consisting of voxels for the head for diagnosis of teeth, a 3D CT image in which only the area of the tooth is captured, or a 3D CT image in which the entire body is captured. You can.

In one embodiment, the processor 110 may receive a 3D image captured about an object from another device (eg, an image capturing device). In another embodiment, the processor 110 may control an image capture device (not shown) included in the device 100 to capture a 3D image of an object. In another embodiment, the processor 110 may load a 3D image stored in memory.

The processor 110 may determine the positions of the plurality of landmarks 10 with respect to the 3D image. Here, the landmark 10 represents an anatomical point on a 3D image required for orthodontic diagnosis, and may be defined based on a two-dimensional (x, y) or three-dimensional (x, y, z) direction. In one embodiment, the landmarks 10 include the sella, nasion, pterygoid, orbitale, basis, and porion, which are used in orthodontic diagnosis. , corpus, pogonion, anterior nasal spine (ANS), posterior nasal spine (PNS), maxilla 1 root (e.g. maxilla 1 root), mandible 1 crown (e.g. mandible 1 crown) ), etc., but is not limited thereto, and various anatomical points may be used. Details regarding this will be described with further reference to FIG. 2.

FIG. 2 is a diagram illustrating an operation of the device 100 receiving a user input for the landmark 10 according to an embodiment.

Referring to FIG. 2, the processor 110 may load a 3D image of an object and transmit it to the display 120, and the display 120 may display the received 3D image. Additionally, the processor 110 may determine the location of the landmark 10 based on a user input for the landmark 10 received through a 3D image. For example, the display 120 displays a 3D image corresponding to the patient information determined according to the user input in the first window 210, and also displays a message including the name of the landmark 10 or confirming the location. can do. For example, according to the set landmark order, the name corresponding to the landmark 10 for which location input is required (e.g., 'Nage On' or 'N', the abbreviation for Naige On) is displayed, and the mouse cursor is positioned on the name. In this case, the definition of the corresponding landmark 10 may be displayed as a tool tip message. In addition, when the processor 110 receives a user input requesting movement to the next landmark 10 (e.g., a selection input for an arrow button, an input for a space bar button, etc.), the processor 110 moves according to the set landmark order. A name or message can be output to confirm the location of the next landmark 10.

Subsequently, when a user input (e.g., selection input, location coordinate value input, etc.) for a specific location on the 3D image is received as a response to the message, the processor 110 determines the location of the landmark 10 according to the selected location. You can decide. Additionally, the processor 110 may update the 3D image so that the determined location of the landmark 10 is displayed. Throughout the specification, user input can broadly refer to user selection input, text input, etc. authorized using a mouse, keyboard, joystick, etc. Additionally, 'display' may mean marking numbers or symbols on an image or showing a value.

The processor 110 may determine a plurality of landmarks 10 that are the target of location determination among all landmarks. In one embodiment, the display 120 displays a complete landmark list including all landmarks (e.g., 50 types), and the processor 110 positions the entire landmark list according to the user's selection input. A plurality of landmarks (10) (e.g., 10) that are the target of decision may be determined. In another embodiment, the processor 110 may load setting information (eg, landmark type) for a plurality of landmarks 10 that are the target of location determination from memory.

The processor 110 provides a message to inquire about the location of the target landmark that is the target of location determination at the current stage among the plurality of landmarks 10, and determines the location of the target landmark based on the user input to the message. You can. For example, the processor 110 may query the location of a target landmark (e.g., right orbit) in the current step according to a specified landmark order (e.g., right orbit, left orbit, pogonion, etc.), receive user input, and determine the location. The first step, such as decision, can be performed. Subsequently, the processor 110 may perform a second step, such as inquiring about the location of the updated target landmark (e.g., left eye socket), receiving user input, and determining the location, in the next step according to the designated landmark order. In this way, the processor 110 may sequentially determine the positions of the N landmarks 10 by repeating the above-described operations according to the first to Nth steps.

In one embodiment, the processor 110 determines the location of a plurality of landmarks 10 that are the target of location determination among all landmarks according to user input, and determines the location of the remaining landmarks from the 3D image according to a set method. Each location can be determined. For example, the processor 110 may detect the location of each remaining landmark in the 3D image using training data obtained based on the image.

The processor 110 may determine the positions of a plurality of landmarks included in the 3D image using previously acquired AI (Artificial Intelligence)-based learning data. In one embodiment, the processor 110 may apply a convolutional neural network (CNN) to a training data set included in the training data to trace the position coordinates of the landmark 10 in the 3D image. For example, the training data may include a CNN model that traces the position of the landmark 10 from the input image through learning on a set of training images showing the positions of the various anatomical points described above. Additionally, the learning image set can be created by augmenting images from sample images based on the GAN (Generative Adversarial Net) algorithm. The processor 110 may input a 3D image into the CNN model and detect the positions of each of the plurality of landmarks 10 included in the 3D image. In one embodiment, the processor 110 may receive learning data from an external device (e.g., a server) that performs the above-described learning, and location information of the landmark 10 determined through user input for the 3D image. You can continuously update learning data using .

The processor 110 displays the positions of a plurality of landmarks 10 determined using learning data on a 3D image, and determines the positions of the plurality of landmarks 10 based on user input received through the 3D image. It can be renewed. For example, the processor 110 first displays the location of the landmark 10 detected through the CNN model on the 3D image to preempt the landmark location, and then receives a user input requesting location correction. As a result, the position of the landmark 10 can be modified.

The processor 110 may specify the location of the bilateral landmark 11, which is a landmark for the bilateral structure, among the determined plurality of landmarks 10. In one embodiment, the processor 110 divides the landmark 10, the location of which is determined according to the user input, into the remaining landmarks 11, which are landmarks for bilateral structures (e.g., eye bones) or the bilateral structures. They can be classified as non-bilateral landmarks (12), which are landmarks for structures (e.g. pogonions).

Here, a bilateral structure (e.g., eye bone) has two substructures (e.g., right eye) on either side of a reference plane (e.g., coronal, sagittal, or horizontal plane) for a body part of the patient (e.g., head). Bone, left eye bone) represents a structure located separately. In one embodiment, the bilateral landmarks 11 for bilateral structures (e.g., eye bones) may include the orbits (e.g., lowest part of the eye bones), porions, etc. In Figure 2, 'orbital' is shown as an example of a bilateral landmark 11, and 'pogonion' is shown as an example of a non-bilateral landmark 12, but the present invention is not limited thereto. For example, various landmarks (10) such as the sella and the porion may correspond to the bilateral landmarks (11), and various landmarks (10) such as the nasion, the root of the maxillary incisor No. 1, and the crown of the mandibular incisor No. 1. These may correspond to non-bilateral landmarks 12.

The processor 110 may specify the positions of the first bilateral landmark 11a and the second bilateral landmark 11b constituting the bilateral landmark 11. In one embodiment, the processor 110, with respect to the determined bilateral landmark 11, creates a first bilateral landmark (e.g., right side) located on one side (e.g., right) with respect to the reference plane of the bilateral landmark 11. It can be divided into 11a) and a second bilateral landmark 11b located on the other side (eg, left) with respect to the reference plane.

In one embodiment, the processor 110 selects a landmark 10 for the right orbitale and left orbitale corresponding to the bilateral structure among the plurality of landmarks 10 positioned according to the user input. By detecting, the location of the bilateral landmarks 11 can be specified. In addition, the processor 110 specifies the position of the first bilateral landmark 11a according to the position of the landmark 10 with respect to the right orbit, and the second bilateral landmark 11a according to the position of the landmark 10 with respect to the left orbit. The location of the bilateral landmarks 11b can be specified.

In addition to the above-described embodiments, the processor 110 may specify the location of the bilateral landmarks 11 according to the following embodiments.

In the first embodiment, the processor 110 receives user inputs from both sides of the bilateral landmark 11 through a 3D image, and creates a first bilateral landmark (11) according to the received user inputs from both sides. The location of 11a) and the location of the second bilateral landmark 11b can be designated, respectively. For example, the display 120 displays a 3D image at a specific view point (e.g., the front), and the processor 110 determines the position of the first bilateral landmark 11a on the 3D image displayed at the view point. and user input regarding the positions of the second bilateral landmarks 11b may be sequentially received.

In the second embodiment, the processor 110 receives only a user input from one side of the bilateral landmark 11 through a 3D image, and uses the received user input from one side to create a first bilateral landmark ( Both the location of 11a) and the location of the second bilateral landmark 11b can be specified.

Specifically, the processor 110 may determine a location on one side of the target landmark based on a user input for a message obtained for one side of the 3D image. Subsequently, the processor 110 may determine a location on the other side of the target landmark based on the properties of the target landmark. Additionally, the processor 110 may determine the location of the target landmark based on the location on one side and the location on the other side. For example, the processor 110 outputs a message inquiring about the location of the 'right orbit', and as a user input for the location on the right side based on the sagittal plane is received, the location of the first bilateral landmark 11a can be decided. Subsequently, the processor 110 determines a reference plane (e.g., sagittal plane) corresponding to the attribute (e.g., type) of the target landmark called 'orbital', and determines the first bilateral landmark 11a based on the reference plane. The position of the second bilateral landmark 11b can be determined by detecting the position opposite the position.

In this way, the processor 110 utilizes the first bilateral landmark 11a, whose position is determined according to the user input, to estimate the location of the second bilateral landmark 11b, thereby determining the position of the bilateral landmark 11. The location can be determined. Accordingly, even if only the location of one side is specified from the user's perspective, the location of the other side is automatically predicted based on landmark properties, thereby improving user convenience.

In the third embodiment, the processor 110 receives user input on one side of the bilateral landmark 11 through a 3D image, and then receives user input on the other side of the 3D image provided according to the rotation request. By receiving the information, the location of the bilateral landmarks 11 can be determined.

Specifically, the processor 110 may determine a location on one side of the target landmark based on a user input for a message obtained for one side of the 3D image. Processor 110 may then obtain a rotation request requesting to provide another side of the 3D image. Additionally, the processor 110 may determine the location of the target landmark on the other side based on a user's additional input for the other side. For example, the display 120 may display a 3D image at a specific view point (e.g., front view) and output a message inquiring about the location of the 'right orbit'. In addition, as the processor 110 receives a user input for the location of one side (e.g., right side) based on the sagittal plane on the 3D image displayed two-dimensionally at the corresponding viewpoint, the first bilateral landmark 11a ) can be determined. Subsequently, the processor 110 receives a user input (e.g., front view) requesting to rotate the 3D image of the corresponding view point (e.g., front view) by a specific axis (e.g., x, y, z) and a specific angle (e.g., 90 degrees). drag and drop input, axis input, angle input, etc.), and the display 120 can update and display the 3D image with a different view point (e.g., side, top, etc.) according to the user input. In addition, the processor 110 determines the location of the second bilateral landmark 12b as a user input for the location of the other side (e.g., left) is received based on the sagittal plane on the 3D image displayed as a different view point. You can decide.

The display 120 can display the positions of each of the first bilateral landmark 11a and the second bilateral landmark 11b on a 3D image simultaneously in different windows within one screen. In one embodiment, the display 120 displays a 3D image indicating the position of the first bilateral landmark 11a in the 1-1 window (e.g., one end of the first window 210), and displays the 3D image in the second bilateral landmark 11a. A 3D image showing the location of the castle landmark 11b may be displayed in the first and second windows (eg, the other end of the first window 210). For example, the processor 110 may display the positions of each of the first bilateral landmarks 11a and the second bilateral landmarks 11b on the 3D image in separate windows. Accordingly, user convenience can be improved by simultaneously providing the left and right positions of the bilateral landmarks 11 as 3D images in different areas.

The processor 110 may update the position of the first bilateral landmark 11a or the second bilateral landmark 11b based on user input received through different windows. In one embodiment, the processor 110 may update the location of the first bilateral landmark 11a based on the user input received through the 1-1 window, and the location of the first bilateral landmark 11a received through the 1-2 window. The location of the second bilateral landmark 11b may be updated based on user input. For example, when a rotation request for a 3D image is received through the 1-1 window and a user input for a point on the rotated 3D image is received, the processor 110 generates the first bilateral landmark 11a ) can be updated to that point. For another example, when a rotation request for a 3D image is received through the 1-2 window and a user input for a point on the rotated 3D image is received, the processor 110 generates a second bilateral landmark ( The location of 11b) can be updated to the corresponding point.

Accordingly, the processor 110 preempts the location of the landmark 10 based on AI, and places the positions of the first bilateral landmark 11a and the second bilateral landmark 11b on the left side, respectively. The 3D image and the 3D image on the right can be divided into separate windows and presented on one screen. Additionally, each 3D image can be rotated according to user input and used for position correction by the user, thereby improving user convenience.

In this way, the processor 110 may provide an interface that allows rotation of the view point of the 3D image. Accordingly, user convenience can be improved because the 3D position of the bilateral landmarks 11 can be set in detail on the 3D image from the user's perspective.

Generally, when a user inputs a landmark position in a 2D image, the landmark position is specified for that 2D image, but the inconvenience of having to re-enter the landmark position for other 2D images is caused.

However, according to the above-described embodiment, when a user inputs a landmark location in a 3D image, more accurate 3D location information is input than when the user inputs the landmark location in a 2D image. In addition, even if the landmark location is entered only once from the user's perspective, the landmark location is specified in three dimensions, so a 2D image displaying the landmark 10 can be obtained without limitation at various angles.

In the fourth embodiment, the processor 110 determines the position of the first bilateral landmark 11a according to a user input of one side received through a 3D image, and determines the position of the first bilateral landmark 11a Accordingly, the initial position of the second bilateral landmark 11b can be determined. Subsequently, the processor 110 may update the position of the second bilateral landmark 11b by receiving a user input regarding the other side of the 3D image provided according to the rotation request. For example, the processor 110 may provide the predicted position of one side only by specifying the position of one side, and may finely adjust the predicted position of the other side according to user input through a rotated 3D image.

In one embodiment, if a rotation request is not received within a set time from the time the position on one side of the target landmark is determined, the processor 110 indicates the other side using the current viewpoint and the properties of the target landmark. The rotation axis and rotation angle suitable for the machine can be determined. Next, the display 120 may display a 3D image rotated according to the determined rotation axis and rotation angle, and output a message inquiring about positioning on the other side on the rotated 3D image. For example, if there is no request for rotation within a set time (e.g., 30 seconds) after the position of the first bilateral landmark 11a, which is the right corpus, is determined according to the user input in the frontal 3D image, the processor 110 moves the position to the left corpus. A 3D image may be displayed on the display 120 with a view point (e.g., axial view) rotated 90 degrees about an axis (e.g., x-axis) perpendicular to the corresponding sagittal plane.

The processor 110 may determine the position of the reference point 20 with respect to the bilateral landmarks 11 based on the determined positions of the bilateral landmarks 11 . Here, the reference point 20 represents a point that can be used to provide information about whether the location of the bilateral landmarks 11 is appropriate. For example, the position of the reference point 20 can be used as a reference position for whether the first bilateral landmark 11a or the second bilateral landmark 11b is not biased in one direction with respect to the sagittal plane. there is.

The processor 110 may determine the average position of the position of the first bilateral landmark 11a and the second bilateral landmark 11b, and determine the position of the reference point 20 to correspond to the average position. there is. For example, the processor 110 uses the position coordinates (x ₁ , y ₁ , z ₁ ) of the first bilateral landmark 11a and the position coordinates (x) of the second bilateral landmark 11b within the 3D image. ₂ , y ₂ , z ₂ ) for each axis, the average position ((x ₁ +x ₂ )/2, (y ₁ +y ₂ )/2, (z ₁ +z ₂ )/2) It can be calculated. Subsequently, the processor 110 determines the reference point 20 according to the position coordinates corresponding to the average position ((x ₁ +x ₂ )/2, (y ₁ +y ₂ )/2, (z ₁ +z ₂ )/2). ) can be determined. Here, the fact that the location corresponds may include a case where the location matches the target point or is located within a set range from the target point.

In one embodiment, the processor 110 acquires a line (e.g., a straight line) connecting the position of the first bilateral landmark 11a and the position of the second bilateral landmark 11b, and The position of the reference point 20 can be determined based on one point. For example, the processor 110 calculates the midpoint coordinates of a straight line connecting the position coordinates of the first bilateral landmark 11a and the position coordinates of the second bilateral landmark 11b to determine the position of the reference point 20. can be decided. For another example, the processor 110 calculates the position coordinates located as close as a set ratio toward the first bilateral landmark 11a or the second bilateral landmark 11b on the corresponding straight line to determine the reference point 20. You can also determine the location of . In one embodiment, the processor 110 uses the bilateral landmarks 11 expressed in 3D to use algorithms that can define a straight line or curve in 3D (e.g., Bezier curve, spline interpolation, polynomial regression ( The position information of the line can be determined using polynomial regression, etc.).

The processor 110 performs tracing to determine the positions of the plurality of landmarks 10 and the reference point 20 in the 3D image, and then displays the plurality of landmarks 10 and the reference point 20 from the 3D image. 2D images can be acquired. Here, the 2D image is a cross-sectional image extracted from a 3D image, and in one embodiment, may include at least one of a lateral image, a front image, a panoramic image, and a SubMentoVertex (SMV) image for the object. You can. In one embodiment, the lateral image may be an image showing the object from the side (e.g., an image viewed from behind while looking to the right). The front image may be an image showing the object from the front. The SMV image represents an image of the view point seen from underneath the chin with the chin lifted 90 degrees, and can be used to quantify the degree of left and right asymmetry of the temporomandibular joint.

FIG. 3 is a diagram illustrating an operation in which the device 100 acquires a 2D image from a 3D image displaying a plurality of landmarks 10 and reference points 20 according to an embodiment.

Referring to FIG. 3 , the processor 110 may obtain a 2D image displaying a plurality of landmarks 10 and reference points 20 from a 3D image of an object using a perspective projection method. In one embodiment, the processor 110 may determine the positions of a plurality of landmarks 10 and reference points 20 displayed in a 2D image using a perspective projection method. For example, the processor 110 may project a plurality of landmarks 10 and reference points 20 in the 3D image onto the perspective projection point 30, which is the standard for perspective projection. Processor 110 may then obtain a first clip plane (far clip plane) 310 located far from perspective projection point 30 and a second clip plane (near clip plane) 320 located near. In addition, the processor 110 determines a proportional expression according to the perspective distance relationship between the first and second clip planes, and is extracted from the 3D projection image (e.g., 3D CBCT image) according to the proportional expression to produce a 2D projection image (e.g., real 2D The positions of a plurality of landmarks 10 and reference points 20 displayed on an X-ray can be determined.

In another embodiment, the processor 110 may obtain a 2D image displaying a plurality of landmarks 10 and reference points 20 from a 3D image of the object using an orthographic projection method.

In one embodiment, the processor 110 may obtain a 2D image from a 3D image through axis settings after performing the above-described tracing. In one embodiment, the axis setting initially receives the positions of a certain number (e.g., 5) of landmarks (e.g., naigeon, right/left orbit, right/left porion) from the user, and sets the corresponding landmark locations. This means setting the axis using . For example, the processor 110 may set the coordinate axis with the position of Navigation at (0, 0, 0). The processor 110 may generate a 2D image using a perspective projection method from a 3D image using a plurality of landmarks 10 derived through tracing in an axis setting state.

The display 120 may display a 2D image of an object on which a plurality of landmarks 10 and reference points 20 are displayed. For example, as shown in FIG. 2, the display 120 may display a 2D image extracted from a 3D image of an object on the second window 220.

In one embodiment, a 3D image and a 2D image of an object may be displayed simultaneously in distinct areas. The display 120 displays a 3D image showing a plurality of landmarks 10 and reference points 20 on a first window 210, and displays a 2D image extracted from the 3D image on a second window 210 adjacent to the first window 210. It can be displayed simultaneously on the window 220.

In one embodiment, the positions of the plurality of landmarks 10 displayed in the 2D image may be updated in real time according to the positions of the plurality of landmarks 10 determined based on user input for the 3D image. The processor 110 may receive a position update input for the landmark 10 whose position has already been determined or a position determination input for the landmark 10 whose position has not yet been determined through the first window 210. The processor 110 updates or determines the location of the landmark 10 according to the user input to update the 3D image, and the display 120 displays the 2D image extracted from the updated 3D image in the second window 220. It can be updated and displayed in real time.

In one embodiment, the processor 110 may receive a user input requesting to move the location of the landmark 10 or the reference point 20 through the first window 210. For example, the processor 110 may receive a user input (e.g., Drag and drop input, location coordinate value input, etc.) can be received. In this case, the processor 110 updates the position of the landmark 10 or the reference point 20 to the second point according to the received user input, and sequentially updates the 3D image and the 2D image according to the updated position. there is.

In one embodiment, the display 120 may display the reference point 20 in a different manner from the plurality of landmarks 10. For example, the display 120 may display the color, size, display method, etc. of the point displayed as the reference point 20 differently from the plurality of landmarks 10 including the bilateral landmarks 11. .

In one embodiment, the display 120 may display the bilateral landmark 11 in a different manner from the landmarks other than the bilateral landmark 11 among the plurality of landmarks 10. For example, the display 120 displays the bilateral landmarks 11 as dots of a first color (e.g., red) and a first size (e.g., 1) and the remaining landmarks 10 as dots of a second color (e.g., 1). : black) and can be displayed as dots of the first size. Additionally, the display 120 may display the reference point 20 as a dot of a first color (eg, red) and a second size (eg, 2).

In one embodiment, the display 120 may display lines connecting the bilateral landmarks 11 on a 3D image. For example, when displaying a 3D image, the display 120 displays a straight line connecting the first bilateral landmark 11a, which is the right orbit, and the second bilateral landmark 11b, which is the left orbit, in a third color (e.g. : blue) can be displayed together.

FIG. 4 is a diagram illustrating an operation in which the device 100 provides analysis results for a plurality of landmarks 10 displayed in a 2D image according to an embodiment.

Referring to FIG. 4 , the processor 110 may determine a plurality of measurement values for a plurality of analysis items based on a plurality of landmarks 10 and/or reference points 20 displayed in the 2D image. Here, the analysis item represents a target item (e.g., Facial Convexity) that requires analysis according to a set analysis method, for example, the location of a specific landmark 10, the location between one landmark 10 and another landmark 10. May include differences, slope differences, etc. Additionally, the measured value represents the value (e.g., 10.52) measured at the corresponding landmark location for the analysis item. In one embodiment, the processor 110 may obtain a user input for selecting one of a plurality of analysis methods and determine a plurality of measurement values for a plurality of analysis items provided by the analysis method corresponding to the user input. . The display 120 may display a symbol or text representing at least one of the location of each landmark 10, each analysis item, and the measurement value, overlapping the corresponding location on the overlapping image.

The processor 110 may obtain analysis results including a plurality of analysis items, a plurality of measurement values, and statistical values according to the analysis method, and the display 120 may display the analysis results on the third window 230. For example, the processor 110 accumulates measurement values (e.g., 10.52) of analysis items such as 'Facial Convexity' and each analysis item in the database according to the analysis method selected by the user (e.g., Integra by CJH). Statistical values (e.g., mean, standard deviation, median, etc.), differences between measured values and statistical values, etc. can be calculated using the images. Additionally, the display 120 may display analysis results including the calculated values on a third window 230 that is separate from the second window 220. In one embodiment, the analysis method may include Ricketts analysis technique, Steiner analysis technique, Downs analysis technique, etc.

The processor 110 may determine the possibility of an abnormality for a plurality of measured values based on a set analysis method. In one embodiment, the processor 110 may determine the probability of an anomaly for a plurality of measurement values based on the average and/or standard deviation for the analysis item. For example, the processor 110 may calculate the probability of an abnormality for each analysis item in proportion to the difference between the measured value and the statistical value.

The processor 110 may determine expected problem items corresponding to some of the plurality of measured values based on the probability of an abnormality, and the display 120 may display the predicted problem items on the third window 230 . Here, displaying expected problem items means deriving values that are outside the normal range through analysis of measured values and displaying them. In one embodiment, the problem prediction items are Convex Profile indicating a protruding face, Concave Facial indicating a depressed face, Protrusive Mx. indicating a protruding occlusion, and Pretrusive Mx indicating a distal occlusion. It may include, but is not limited to, various medical terms used in the treatment process. In one embodiment, the processor 110 may determine an expected problem item (eg, facial convexity) based on whether the probability of an abnormality for each analysis item is greater than or equal to a reference value. In another embodiment, the processor 110 predicts problem items (e.g., Concave Facial) based on the degree of likelihood of an abnormality, the direction of the difference (e.g., positive (+) difference, negative (-) difference), etc. can be decided.

In one embodiment, the processor 110 may perform a series of operations to display an image and may be electrically connected to the display 120 and other components to control data flow between them. To this end, the processor 110 may be implemented as a central processor unit (CPU) that controls the overall operation of the device 100. In addition, the display 120 may comprehensively refer to an image output device that displays an image, for example, a liquid crystal display, a thin film transistor-liquid crystal display, and an organic light emitting device. It may be an organic light-emitting diode, a flexible display, a 3D display, an electrophoretic display, etc.

Additionally, those skilled in the art can understand that other general-purpose components in addition to the components shown in FIG. 1 may be further included in the device 100. According to one embodiment, the device 100 further includes a communication unit for communicating with other devices through a wired or wireless network, a user interface for receiving user input, and a storage unit for storing data (e.g., memory, database, cloud), etc. It can be included. According to another embodiment, some of the components shown in FIG. 1 may be omitted.

FIG. 5 is a flowchart illustrating a method by which the device 100 displays an image according to an embodiment.

In step S510, the device 100 may acquire a 3D image of the object. In one embodiment, the 3D image may include a CBCT image.

In step S520, the device 100 may determine the positions of a plurality of landmarks 10 included in the 3D image. In one embodiment, the device 100 determines the positions of a plurality of landmarks 10 included in a 3D image using previously acquired learning data, and displays the positions of the plurality of landmarks 10 on the 3D image. It is possible to display and update the positions of a plurality of landmarks 10 based on user input received through a 3D image. In another embodiment, the device 100 may determine the positions of the plurality of landmarks 10 based on user input received through a 3D image.

In one embodiment, the device 100 may determine a plurality of landmarks 10 that are the target of location determination among all landmarks. Subsequently, the device 100 may provide a message including the name of a target landmark that is the target of location determination at the current stage among the plurality of landmarks 10. Subsequently, the device 100 may determine the location of the target landmark based on the user input for the message. For example, the device 100 may determine a plurality of landmarks 10 (eg, 10) that are the target of location determination, according to a user input regarding which landmark's location to determine. In addition, the device 100 displays a message requesting location input for each of the determined plurality of landmarks 10 (e.g., left orbit, right orbit, etc.), and displays each landmark 10 received through a 3D image. The location of each landmark 10 can be determined according to user input (e.g., touch, button click) about the location.

In step S530, the device 100 may specify the location of the bilateral landmark 11, which is a landmark for the bilateral structure, among the determined plurality of landmarks 10. For example, the device 100 may select the plurality of landmarks 10. When all user inputs for the positions of (10) are received, it can be determined whether each of the plurality of landmarks (10) is a bilateral landmark (11) or a non-bilateral landmark (12).

In step S540, the device 100 may determine the location of the reference point 20 for the bilateral landmark 11 based on the location of the designated bilateral landmark 11. In one embodiment, the device 100 may determine the first bilateral landmark 11a and the second bilateral landmark 11b constituting the bilateral landmark 11. Subsequently, the device 100 determines the average position of the position of the first bilateral landmark 11a and the second bilateral landmark 11b, and sets the position of the reference point 20 to correspond to the average position. You can decide. For example, the device 100 calculates the average position of the first bilateral landmark 11a and the second bilateral landmark 11b for each bilateral landmark 11 and calculates the average position of the corresponding bilateral landmark 11. The position of the reference point 20 with respect to the mark 11 can be determined. For example, since the left and right sides do not exactly match, analysis and decision-making regarding orthodontic diagnosis can be performed by providing information about the reference point 20 derived from the left and right average values.

In step S550, the device 100 may display a 2D image of the object in which reference points 20 for a plurality of landmarks 10 and bilateral landmarks 11 are displayed. In one embodiment, the positions of the plurality of landmarks 10 and reference points 20 displayed in the 2D image may be determined using perspective projection. For example, the device 100 may respectively acquire a 2D lateral image, a 2D front image, a 2D panoramic image, and a 2D SMV image from a 3D image using a plurality of landmarks 10. In addition, the device 100 may provide a 2D image in which the bilateral landmarks 11, the reference point 20 determined from the bilateral landmarks 11, and the non-bilateral landmarks 12 are all visually distinguished. You can.

According to an embodiment of the present disclosure, the device 100 produces various types of 2D images that accurately display the location of the landmark 10 and the location of the reference point 20 when the user inputs only the location of the landmark in the 3D image. User convenience can be improved by providing it immediately.

In addition, the device 100 performs tracing of landmark positions more accurately using 3D images, while providing 2D images obtained from 3D images, applying 2D analysis methods with high data reliability that have been accumulated over a long period of time to provide more accurate Analysis results can be provided.

Additionally, the device 100 can simultaneously provide a 3D image used to input a landmark location and a 2D image extracted from the 3D image. Accordingly, the user can check in real time where the landmark 10 determined in the 3D image is located in the 2D image.

Additionally, the device 100 can provide reference points for bilateral landmarks in 3D images and 2D images. Accordingly, there is an effect of enabling a more intuitive analysis of the left and right landmark positions of bilateral structures.

The order and combination of steps shown above is an example, and the order, combination, branch, function, and performer thereof may be added, omitted, or modified without departing from the essential characteristics of each component described in the specification. It can be seen that this can be implemented. In addition, throughout the specification, 'provision' includes the process of an object acquiring specific information or directly or indirectly sending and receiving it to a specific object, and can be interpreted to comprehensively include the performance of related operations required in this process.

Various embodiments of the present invention are implemented as software including one or more instructions stored in a storage medium (e.g., memory) that can be read by a machine (e.g., a display device or computer). It can be. For example, the device's processor (eg, processor 110) may call at least one instruction among one or more instructions stored from a storage medium and execute it. This allows the device to be operated to perform at least one function according to the at least one instruction called. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. A storage medium that can be read by a device may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' only means that the storage medium is a tangible device and does not contain signals (e.g. electromagnetic waves). This term refers to cases where data is stored semi-permanently in the storage medium. There is no distinction between temporary storage cases.

According to one embodiment, methods according to various embodiments disclosed in the present invention may be included and provided in a computer program product. Computer program products are commodities and can be traded between sellers and buyers. The computer program product may be distributed in the form of a machine-readable storage medium (e.g. compact disc read only memory (CD-ROM)) or via an application store (e.g. Play StoreTM) or on two user devices (e.g. It can be distributed (e.g. downloaded or uploaded) directly between smartphones) or online. In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

Those skilled in the art related to this embodiment will understand that the above-described base material can be implemented in a modified form without departing from the essential characteristics. Therefore, the disclosed methods should be considered from an explanatory rather than a restrictive perspective. The scope of the present invention is indicated in the claims rather than the foregoing description, and all differences within the equivalent scope should be construed as being included in the present invention.

100: device
110: Processor 120: Display
10: Multiple landmarks 11: Bilateral landmarks
11a: first bilateral landmark 11b: second bilateral landmark
20: reference point

Claims (17)

In a method of displaying an image,
Obtaining a 3D image of an object;
determining the positions of a plurality of landmarks included in the 3D image;
Specifying the location of a bilateral landmark, which is a landmark for a bilateral structure, among the determined plurality of landmarks;
determining a location of a reference point for the bilateral landmark based on the location of the designated bilateral landmark; and
Comprising: displaying a 2D image of the object in which reference points for the plurality of landmarks and the bilateral landmarks are displayed,
The step of determining the positions of the plurality of landmarks is
determining the plurality of landmarks that are subject to position determination among all landmarks for the object;
Providing a message confirming the location of a target landmark that is the target of location determination at the current stage according to a designated landmark order among the plurality of landmarks; and
Method comprising: determining the location of the target landmark based on the response to the message.
According to claim 1,
The step of determining the location of the reference point for the bilateral landmark is
determining a first bilateral landmark and a second bilateral landmark constituting the bilateral landmark;
determining an average position for the positions of the first bilateral landmarks and the positions of the second bilateral landmarks; and
Method comprising: determining a position of the reference point to correspond to the average position.
According to claim 1,
The location of reference points for the plurality of landmarks and the bilateral landmarks displayed in the 2D image are determined by at least one of a perspective projection method and an orthographic projection method.
delete According to claim 1,
The step of determining the location of the target landmark is
A step of determining the location of the target landmark at a point corresponding to the nasion, both orbits, and both porions based on user input,
The step of displaying a 2D image for the object is
determining a coordinate axis corresponding to the positions of the nasion, both orbits, and both porions as a setting axis; and
Comprising: acquiring the 2D image according to a perspective projection method from the 3D image using the plurality of landmarks obtained through 3D tracing on the set axis,
The location of the bilateral landmarks is determined to be the lowest point of the orbit relative to the bilateral structures representing the right and left eye bones,
Non-bilateral landmark locations are determined by pogonion points.
According to claim 1,
The step of determining the location of the target landmark is
Displaying the positions of each of the first and second bilateral landmarks constituting the bilateral landmarks on the 3D image simultaneously in different windows within one screen; and
The method further comprising: determining a location of the first bilateral landmark or the second bilateral landmark based on user input received through the different windows.
According to claim 1,
The step of determining the positions of the plurality of landmarks is
A method comprising: determining the positions of the plurality of landmarks included in the 3D image using previously acquired training data.
According to claim 7,
The step of determining the positions of the plurality of landmarks is
Displaying the positions of a plurality of landmarks determined through the learning data on the 3D image; and
Further comprising: updating the positions of the plurality of landmarks based on user input received through the 3D image,
The location of a specific landmark according to a plurality of analysis items based on the plurality of landmarks and the reference point, the position difference between one landmark and the other determined according to the location of the specific landmark, and the slope between the landmarks determining a plurality of measures of difference;
Determining statistical values representing the average value and standard deviation for each of the plurality of analysis items based on the plurality of images accumulated in the database;
determining a probability of an anomaly for the plurality of measured values based on the statistical values;
determining expected problem items corresponding to some of the plurality of measured values based on the abnormality probability; and
The method further comprising displaying the predicted problem item including numerical information in which the measured value is outside a normal range.
According to claim 1,
The 3D image and the 2D image are displayed simultaneously in distinct areas,
A method in which the positions of the plurality of landmarks displayed in the 2D image are updated in real time according to the positions of the plurality of landmarks determined based on user input for the 3D image.
According to claim 1,
The 2D image includes at least one of a lateral image, a front image, a panoramic image, and an SMV image for the object,
The method of claim 1, wherein the reference point is displayed in a different manner than the plurality of landmarks.
In a device that displays an image,
Obtaining a 3D image of the object, determining the location of a plurality of landmarks included in the 3D image, specifying the location of a bilateral landmark that is a landmark for a bilateral structure among the determined plurality of landmarks, and Determine the location of a reference point for the bilateral landmark based on the location of the designated bilateral landmark, determine the plurality of landmarks that are the target of positioning among all landmarks for the object, and determine the plurality of landmarks. a processor that provides a message confirming the location of a target landmark that is the target of location determination at the current stage according to a designated landmark order among landmarks, and determines the location of the target landmark based on a response to the message; and
A display that displays a 2D image of the object on which reference points for the plurality of landmarks and the bilateral landmarks are displayed.
According to claim 11,
The processor is
Determining a first bilateral landmark and a second bilateral landmark constituting the bilateral landmark,
Determining an average position for the positions of the first bilateral landmarks and the positions of the second bilateral landmarks,
A device that determines the location of the reference point to correspond to the average location.
According to claim 11,
The device wherein the positions of reference points for the plurality of landmarks and the bilateral landmarks displayed in the 2D image are determined by at least one of a perspective projection method and an orthographic projection method.
According to claim 11,
The processor is
Based on the user input, the location of the target landmark is determined at points corresponding to the nasion, both orbits, and both porions,
Determine a coordinate axis corresponding to the positions of the nasion, both orbits, and both porions as a setting axis,
Obtaining the 2D image according to a perspective projection method from the 3D image using the plurality of landmarks obtained through 3D tracing on the setting axis,
The location of the bilateral landmarks is determined to be the lowest point of the orbit relative to the bilateral structures representing the right and left eye bones,
Non-bilateral landmark locations are determined by the pogonion point, the device.
According to claim 14,
The display is
Displaying the positions of each of the first and second bilateral landmarks constituting the bilateral landmarks on the 3D image and displaying them simultaneously in different windows within one screen,
The processor is
A device that determines the location of the first bilateral landmark or the second bilateral landmark based on user input received through the different windows.
According to claim 11,
The processor is
Determine the positions of the plurality of landmarks included in the 3D image using previously acquired learning data,
The location of a specific landmark according to a plurality of analysis items based on the plurality of landmarks and the reference point, the position difference between one landmark and the other determined according to the location of the specific landmark, and the slope between the landmarks Determine a plurality of measures of difference,
Determine statistical values representing the average value and standard deviation for each of the plurality of analysis items based on the plurality of images accumulated in the database,
Determine the probability of an abnormality for the plurality of measured values based on the statistical values,
Based on the abnormality probability, determine expected problem items corresponding to some of the plurality of measurement values,
The display is
A device that displays the problem prediction item including numerical information in which the measured value is outside a normal range.
A computer-readable recording medium recording a program for executing the method of any one of claims 1 to 3 and 5 to 10 on a computer.