KR20230075178A - Method, device and recording medium for displaying a plurality of images - Google Patents

Abstract

According to an embodiment, a method of determining a modality of each of a plurality of images used for calibration diagnosis based on a learned neural network and displaying the plurality of images according to the modality is disclosed. Accordingly, it is possible to minimize a cumbersome procedure required of a user for classifying images related to calibration diagnosis, and provide an intuitive display interface to the user through a display according to modality.

Description

Method, device, and recording medium for displaying a plurality of images used for calibration diagnosis

The present disclosure relates to a method, a device, and a recording medium for displaying various images used for calibration diagnosis.

In general, when performing orthodontic treatment, diagnosis using various clinical images is required to diagnose a patient's problem. For example, not only radiographic images but also facial and intraoral photographs are taken to diagnose problems, and during orthodontic treatment, additional radiographic images and facial and intraoral photographs are taken to confirm changes in teeth and facial appearance. These various images should be classified into various treatment groups in consideration of what stage of the current patient's orthodontic treatment process and how they are being treated for efficient diagnosis by a dentist.

In the prior art, a user directly identifies various images necessary for orthodontic treatment while checking which treatment group of the current patient the corresponding image corresponds to and classifying them. This conventional technique has a disadvantage in that the user's convenience is poor because the user has to repeat the manual process of classifying newly captured images of the patient every time the user proceeds with the calibration analysis.

Accordingly, there is a need for a technique for solving the above problems and minimizing a user's cumbersome procedure before calibration diagnosis.

Korean Patent Publication No. 10-2006-0100737 (2006.09.21) Dental patient treatment service method and system using the Internet

One embodiment of the present disclosure is to solve the above-mentioned problems of the prior art, and to improve user convenience by minimizing cumbersome procedures required by users for classifying images related to calibration diagnosis.

The technical problem to be solved is not limited to the technical problems as described above, and various technical problems may be further included within a range obvious to those skilled in the art.

A method for displaying a plurality of images according to a first aspect of the present disclosure includes acquiring a plurality of images used for calibration diagnosis; obtaining an analysis result for the types of the plurality of images based on the learned neural network; determining a modality for each of the plurality of images based on a result of the analysis; and displaying the plurality of images in an order or position determined according to the modality.

The displaying of the plurality of images according to the modality may include classifying the plurality of images into one of a plurality of image groups according to the modality; and displaying the plurality of images according to image groups, wherein the plurality of image groups may include at least two of a projection image group, a facial image group, and an oral image group.

The area where the plurality of images are displayed may include a first area where one or more first images of which the modality is determined among the plurality of images are displayed; and a second area in which one or more second images of which the modality is not determined among the plurality of images are displayed.

In addition, the step of determining the modality of each of the plurality of images may include, when a first user input requesting the determination of the modality is received, the modality of each of the plurality of images through a CNN (Convolution Neural Network)-based classification model. determining; and when a second user input for selecting the modality is received, the modality of the one or more second images may be determined as a modality corresponding to the second user input without going through the classification model.

In the step of determining the modality of each of the plurality of images, when the user terminal receives the first user input, the user terminal requests the server to determine the modality of each of the plurality of images, and the server may respond with a result of determining the modality of each of the plurality of images through the classification model according to the request.

In addition, in the step of classifying and displaying the plurality of images, a third image obtained later from among the third image and the fourth image having the same modality may be displayed.

Also, in the classifying and displaying of the plurality of images, a UI image indicating that there is an additional image may be displayed in an area adjacent to the third image.

In the obtaining of the analysis result, the analysis result may be obtained based on a Residual Connection (ResNet) model among convolution neural networks (CNNs).

A device for displaying a plurality of images according to a second aspect of the present disclosure includes an image acquiring unit acquiring a plurality of images used for calibration diagnosis; a processor that obtains an analysis result for the types of the plurality of images based on a learned neural network, and determines a modality for each of the plurality of images based on the analysis result; and a display configured to display the plurality of images in an order or position determined according to the modality.

In addition, the processor classifies the plurality of images into one of a plurality of image groups according to the modality, and the display displays the plurality of images according to the image group, and the plurality of image groups include a projection image group, facial hair It may include at least two of an image group and an oral image group.

The area where the plurality of images are displayed may include a first area where one or more first images of which the modality is determined among the plurality of images are displayed; and a second area in which one or more second images of which the modality is not determined among the plurality of images are displayed.

In addition, the processor determines the modality of each of the plurality of images through a classification model based on a CNN (Convolution Neural Network) when a first user input requesting the determination of the modality is received, and selects the modality. When two user inputs are received, the modality of the at least one second image may be determined as a modality corresponding to the second user input without going through the classification model.

Also, the display may display a third image obtained later from among the third image and the fourth image having the same modality.

A third aspect of the present disclosure may provide a computer readable recording medium recording a program for executing the method according to the first aspect in a computer. Alternatively, the fourth aspect of the present disclosure may provide a computer program stored in a recording medium to implement the method according to the first aspect.

According to an embodiment, user convenience may be improved by minimizing cumbersome procedures required of a user for classifying images related to calibration diagnosis.

In addition, it is possible to reduce the time required for image classification by classifying images through artificial intelligence.

In addition, it is possible to improve work efficiency by improving classification accuracy through a trained neural network.

In addition, by appropriately classifying and displaying images according to modality, an intuitive display interface may be provided to the user.

The effects of the present disclosure are not limited to the above effects, and should be understood to include all effects that can be inferred from the configuration of the disclosure described in the detailed description or claims of the present disclosure.

1 is a block diagram illustrating an example of a configuration of a device according to an exemplary embodiment.
2 is a flowchart illustrating a method of displaying a plurality of images by a device according to an exemplary embodiment.
3 is a diagram illustrating an operation of receiving a user input requesting image acquisition by a device according to an exemplary embodiment.
4 is a diagram illustrating an operation in which a device receives a user input for selecting a folder for acquiring an image, according to an exemplary embodiment.
5 is a diagram illustrating a modality according to an embodiment.
6 is a diagram illustrating a projection image group, a facial image group, and an oral image group according to an exemplary embodiment.
7 is a diagram illustrating an operation of classifying and displaying a plurality of images according to an image group by a device according to an exemplary embodiment.
8 is a diagram illustrating an operation of classifying and displaying a plurality of images according to an image group by a device according to another embodiment.

The terms used in the embodiments have been selected as general terms that are currently widely used as much as possible while considering the functions in the present disclosure, but they may vary depending on the intention or precedent of a person skilled in the art, the emergence of new technologies, and the like. In addition, in a specific case, there is also a term arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the disclosure. Therefore, terms used in the present disclosure should be defined based on the meaning of the term and the general content of the present disclosure, not simply the name of the term.

In the entire specification, when a part is said to "include" a certain component, it means that it may further include other components, not excluding other components unless otherwise stated. In addition, as described in the specification, "... Terms such as “unit” refer to a unit that processes at least one function or operation, which may be implemented as hardware or software, or a combination of hardware and software.

Hereinafter, with reference to the accompanying drawings, embodiments of the present disclosure will be described in detail so that those skilled in the art can easily carry out the present disclosure. However, the present disclosure may be implemented in many different forms and is not limited to the embodiments described herein.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

1 is a block diagram illustrating an example of a configuration of a device 100 according to an embodiment. 2 is a flowchart illustrating a method of displaying a plurality of images by the device 100 according to an exemplary embodiment.

Referring to FIGS. 1 and 2 , the device 100 may include an image acquisition unit 110 , a processor 120 and a display 130 .

In step S210, the image acquisition unit 110 may acquire a plurality of images used for calibration diagnosis. Here, the image used for orthodontic diagnosis represents an image taken of a patient in a patient orthodontic diagnosis process. In one embodiment, the image used for orthodontic diagnosis includes a radiographic image obtained using an X-ray, etc., an image captured using an image sensor, etc. It can, but is not limited thereto. In an embodiment, the image used for orthodontic diagnosis may further include various other types of medical images, such as a scan image acquired through a 3D scanner and a computed tomography (CT) image.

In one embodiment, a plurality of images may correspond to patient information registered as a subject of corrective diagnosis. The image acquisition unit 110 receives or loads patient information (e.g., patient name, date of birth, sex, chart number, etc.) corresponding to a subject of corrective diagnosis according to a user input or loads from memory, and a plurality of images corresponding to the patient information. can be obtained.

In one embodiment, the image acquisition unit 110 may load a plurality of images stored in a memory (not shown). In another embodiment, the image capture unit 110 may receive an image captured by the image capture device (not shown) from the image capture device through interworking with the image capture device. In another embodiment, the image acquisition unit 110 may generate an image by capturing an image of the patient.

Hereinafter, embodiments of step S210 will be described with reference to FIGS. 3 and 4 .

3 is a diagram illustrating an operation in which the device 100 receives a user input requesting image acquisition, according to an exemplary embodiment.

Referring to FIG. 3 , the image acquisition unit 110 may receive a user input requesting acquisition of an image to be classified. For example, the image acquiring unit 110 provides a first interface screen 300 for providing classification of images, and a user input requesting image loading through the first interface screen 300 (eg, a right mouse button) After clicking, select 'Import Manager'). Throughout the specification, 'user input' may broadly mean a user's selection input, text input, and the like applied using a mouse, keyboard, joystick, or the like through a user interface.

4 is a diagram illustrating an operation in which the device 100 receives a user input for selecting a folder for image acquisition according to an exemplary embodiment.

Referring to FIG. 4 , the image acquisition unit 110 may provide a second interface screen 400 for folder selection when a user input for selecting a folder for image acquisition is received. In one embodiment, the second interface screen 400 includes a first window 410 displaying a folder user interface (UI) for selecting a folder, and a second window displaying information about images included in the selected folder. 420 and a third window 430 displaying classification results for the selected images.

The image acquisition unit 110 may receive a user input for a plurality of folders through the first window 410 . Also, the image acquisition unit 110 may determine one of a plurality of folders based on a user input, and acquire a plurality of images included in the determined folder. For example, upon receiving a user input (eg, clicking a left mouse button) for selecting a specific folder 411, the image acquisition unit 110 accesses a storage path corresponding to the specific folder 411 to access the corresponding folder. You can load multiple images stored in it. Subsequently, the image acquisition unit 110 may display summary information (eg, thumbnail images, file names) of the plurality of loaded images on the second window 420 .

The image acquisition unit 110 may receive a first user input requesting determination of modalities for a plurality of images included in the determined folder. For example, the image acquisition unit 110 receives a first user input through an import button 440 displayed at the bottom of the second interface screen 400, and as the first user input is received, a plurality of images are displayed. analysis can be performed.

In step S220, the processor 120 may obtain analysis results for the types of a plurality of images based on the learned neural network. In an embodiment, the processor 120 may analyze a plurality of images with the trained neural network to obtain analysis results of attributes such as a photographing method, a photographing direction, a region, and a subject associated with each type of image. In an embodiment, the trained neural network may be pre-stored in the device 100 or may be provided from a learning server that obtains the trained neural network by performing image-based machine learning and verification. In an embodiment, the trained neural network may be obtained based on a Residual Connection (ResNet) model among convolution neural networks (CNNs). Here, CNN corresponds to a type of deep neural network that extracts variable features using a convolutional layer and produces a desired output value. In addition, the ResNet model can prevent feature loss and improve learning performance by adding the input used in the previous layer to the output of the convolution layer. In another embodiment, the trained neural network may be obtained based on an AlexNet model or a DenseNet model among CNNs, and various other deep learning algorithms may be used.

In an embodiment, the processor 120 may obtain an analysis result for each type of image by analyzing the shape or format of the image based on the learned neural network. In one embodiment, when the learning server obtains a trained ResNet-based classification model by learning the similarity of features of images stored in the database using the CNN's ResNet model, the processor 120 performs a ResNet-based learning from the learning server. A trained neural network including a classification model of may be received. In addition, the processor 120 inputs a plurality of images imported from a folder for calibration diagnosis to the learned ResNet-based classification model, extracts features associated with the type of images, analyzes these features, and captures images. It is possible to identify the method, shooting direction, part, subject, etc. These analysis results can be used in the process of classifying the types of images in a later step.

In step S230, the processor 120 may determine a modality for each of the plurality of images based on the analysis result of the types of the plurality of images. For example, the processor 120 may determine a modality for each of a plurality of images according to identification values of attributes such as a photographing method, a photographing direction, a region, and a subject included in the analysis result.

Here, the modality represents the type of image used for calibration diagnosis. In one embodiment, the number of modalities may be less than or equal to a set number (eg, 19 or 20). Also, although the number of images may be large as images are captured several times within the same modality, the maximum number of modalities (eg, 19) may be a fixed value. Details regarding this will be further described with reference to FIG. 5 .

5 is a diagram illustrating a modality according to an embodiment.

Referring to FIG. 5 , a modality may be defined as an image type classified according to an image capturing method based on modality characteristics such as a capturing direction of an object (eg, a patient), a capturing region, and subject properties.

In one embodiment, the photographing method may include a projection image (Film X-ray), a facial image (Face Photo), an oral image (oral photo), a model image (Model Photo), and the like. For example, the projection image represents an image obtained using X-rays, the face image represents a face image (eg, ID photo), the oral image represents an image of the oral cavity, and the model image represents a plaster model modeled after the oral cavity. images can be displayed.

In an embodiment, modalities according to the direction of imaging of an object may include lateral, frontal (PA (Posterior, Anterior)/AP (Anterior, Posterior), Frontal), Panorama, Left, Right ( Right), CBCT (Cone Beam CT), SMV (SubMentoVertex), diagonal (a45), etc. may be included. For example, lateral denotes an image taken from behind looking to the right, and PA/AP denotes anterior posterior or posterior anterior frontal image. In addition, SMV represents an image obtained by photographing from under the chin in a state where the chin is lifted 90 degrees, and is used to find out the left-right asymmetry of the temporomandibular joint. In addition, a45 denotes an image acquired at an angle of 45 degrees from right to left, panorama denotes a panoramic image obtained in a foreground angle range, and CBCT denotes a 3D image obtained through CBCT imaging.

In one embodiment, the modality according to the region to be captured may include the lower jaw, the upper jaw, the wrist, and the like. In addition, the modality according to the object attribute may include a smile (eg, photographing a face in a smiling state) and the like.

In one embodiment, the modalities of the images are a side projection image, a frontal projection image, a panoramic projection image, a forearm projection image, an SMV projection image, a side facial image, a frontal facial image, and a diagonal facial image, as shown in FIG. 5(a). image, smile face image, right oral image, left oral image, frontal oral image, maxillary oral image, mandibular oral image, right model image, frontal model image, left model image, maxillary model image and mandibular model image. It can be one of the image types. In another embodiment, the modality of the image may be one of 20 image types further including a CBCT projection image, as shown in FIG. 5(b). However, it is not limited thereto, and in another embodiment, the modality may further include other image types. In addition, the modality name and characteristic information may be updated based on user input such as addition or deletion of modality.

In an embodiment, the processor 120 may classify the plurality of images into one of a plurality of image groups including at least one of a projection image group, a facial image group, and an oral image group according to the modality. For example, the processor 120 may classify each of the plurality of images into a projection image group, a facial image group, or an oral image group based on similarity for each modality characteristic. Details regarding this will be further described with reference to FIG. 6. // Unified by classification.

6 is a diagram illustrating a projection image group 610, a facial image group 620, and an oral image group 630 according to an exemplary embodiment.

Referring to FIG. 6 , the processor 120 may classify each of the plurality of images into a projection image group 610 , a facial image group 620 , or an oral image group 630 .

The projection image group 610 corresponds to an image group related to projection images, and may include at least one of a side projection image, a front projection image, a panoramic projection image, and an SMV projection image. The processor 120 may classify images corresponding to projection images among a plurality of images and having modalities of side, front, panorama, and SMV into the projection image group 610 .

The facial image group 620 corresponds to an image group related to facial images, and may include at least one of a side facial image, a frontal facial image, a diagonal facial image, and a smiling facial image. The processor 120 may classify images corresponding to facial images among a plurality of images and having modalities of side, front, diagonal, and smile into the facial image group 620 .

The oral image group 630 corresponds to an image group related to an oral image, and may include at least one of a right oral image, a left oral image, a front oral image, an upper jaw image, and a lower jaw image. In an embodiment, the processor 120 may classify images corresponding to an oral image or a model image and having modalities of right, left, front, upper jaw, and lower jaw among a plurality of images into the oral image group 630 . In another embodiment, the processor 120 may classify images corresponding to model images among a plurality of images and having modalities of right, left, front, upper jaw, and lower jaw into a separate model image group.

In one embodiment, the processor 120 may determine a modality for each of the plurality of images based on a setting value included in each of the plurality of images. When an image is captured, since the image is captured after setting in advance which part is to be captured, a setting value representing the type of the corresponding image may be stored as metadata of the image in each of the plurality of images. The processor 120 may obtain information about attributes such as a photographing method, a photographing direction, a region, and a subject from metadata of each image.

In one embodiment, the processor 120 may determine the modality of each of the plurality of images based on the file name of each of the plurality of images. For example, when a name of a specific modality or a word related thereto is included in text extracted from a file name of an image, the processor 120 may determine the modality of the corresponding image as the modality of the corresponding name.

In step S240, the display 130 may display a plurality of images according to the order or position determined according to the modality. For example, the display 130 may display a plurality of images on the third window 430 according to each modality so as to be sequentially arranged according to a preset order for each of the plurality of modalities. For another example, the display 130 may display a plurality of images so that an image having a corresponding modality is displayed at a position set in advance for each of the plurality of modalities within the third window 430 where the plurality of images are displayed. there is.

The display 130 may display a plurality of images according to image groups corresponding to the plurality of images. For example, the display 130 allows images in the same image group to be located adjacently and images in different image groups to be visually distinguished from each other in different regions of the screen according to an order or position determined according to the image group to be classified. A plurality of images can be displayed. This will be described later with reference to FIG. 7 .

7 is a diagram illustrating an operation of classifying and displaying a plurality of images according to an image group by the device 100 according to an exemplary embodiment.

Referring to FIG. 7 , the processor 120 may provide a result of classifying a plurality of images according to image groups to a third window 430 . In one embodiment, the processor 120 may display the representative image 720 preset in each modality on the third window 430 before the first user input for the import button 440 is received (FIG. 4). reference). Here, the representative image 720 means an image visualized as a thumbnail when displaying an image whose modality has not yet been determined. In one embodiment, the representative image 720 may include an icon that visually shapes each modality.

Also, after the first user input to the import button 440 is received, the processor 120 may classify and arrange a plurality of images for each image group according to the above-described analysis result, and display the images on the third window 430. Yes (see FIG. 7). When the modality is determined and the image is displayed according to each modality, the corresponding image may be displayed instead of the representative image 720 at a position corresponding to the corresponding modality within the third window 430 . If there is an image whose modality is not determined, the representative image 720 may be displayed as it is at a location corresponding to the modality. Here, a position corresponding to a modality may mean a position set to display an image having a corresponding modality within an area where a plurality of images are displayed.

The processor 120 may arrange and display a plurality of images according to priorities set for image groups. In one embodiment, the processor 120 may sort and display a plurality of images according to priority given to the projection image group 610, the facial image group 620, and the oral image group 630 in the order of high priority. . For example, as shown in FIG. 7 , the processor 120 exposes the images classified as the projection image group 610 at the top (or leftmost) side of the third window 430, and displays them as the facial image group 620. The classified images may be sequentially exposed to the next position of the projection image group 610 and the images classified to the oral image group 630 to the next position of the facial image group 620 . In an embodiment, a setting order for image groups may be determined differently according to the type of surgery.

The processor 120 may sort and display images corresponding to the same image group according to a modality setting order. In one embodiment, the order of setting modalities applied to each of the plurality of image groups may be different. For example, the images classified as the projection image group 610 may be sequentially arranged in the order of a side projection image, a front projection image, a panoramic projection image, a forearm projection image, and an SMV projection image. In addition, the images classified as the facial image group 620 may be sequentially arranged in the order of a side facial image, a frontal facial image, a diagonal facial image, and a smile facial image. In addition, the images classified as the oral image group 630 are sequentially arranged in the order of the oral image and the model image, and may be sequentially arranged in the order of right, front, left, upper jaw, and lower jaw for each of the oral image and the model image. .

In an embodiment, the setting sequence of modalities may be determined based on the similarity of modalities such that images having similar modalities are adjacently arranged. For example, the processor 120 determines the similarity of the modalities to be high when one of the photographing method of the image used to distinguish the 19 modalities, the photographing direction of the patient, the photographed part, and the subject property is the same or similar. can In one embodiment, the setting order of the modality may be updated according to a user input.

In this way, the processor 120 may provide a highly visible and intuitive display interface from a user's point of view by classifying and displaying a plurality of images according to image groups and modalities.

An area where a plurality of images are displayed may be composed of a first area associated with a modality and a second area not associated with a modality. In an embodiment, one or more first images of which the modality is determined among a plurality of images may be displayed in the first region associated with the modality. In addition, one or more second images of which the modality is not determined among the plurality of images may be displayed in the second area not associated with the modality. For example, the first area and the second area may be distinguished from each other within the third window 430, the first area may be disposed above, and the second area may be disposed below the first area. For example, when a user input requesting modality determination is received, step S230 is performed, and the first images whose modality is determined through step S230 may be displayed in a first area within the third window 430. . In addition, when a user input for selecting a modality is received for the second images whose modality has not been determined even though step S230 has been performed, the second image may be displayed in the second area according to the modality selected according to the corresponding user input. there is.

When a first user input requesting modality determination is received, the processor 120 may determine the modality of each of the plurality of images through a CNN-based classification model. For example, when the processor 120 receives the first user input through the import button 440, the above-described image capturing method, the capturing direction of the patient, and the capturing using the CNN-based classification model as described above. After obtaining the analysis result of identifying the part and the subject, the modality of the image can be determined through the analysis result, or the determined value for the modality can be obtained immediately as an output by inputting the image to a CNN-based classification model.

In one embodiment, when the device 100 receives the first user input, the device 100 requests the learning server to determine the modality of each of the plurality of images, and as a response to the request from the learning server, the device 100 uses a classification model to determine the plurality of images. A result of determining the modality for each image may be received. For example, the learning server executes the classification model previously acquired through learning and verification before step S210 according to the request of the device 100, and through the classification model as described above for a plurality of images received from the device 100. The modality of each of the plurality of images may be determined by analysis, and the determination result may be transmitted to the device 100 .

In another embodiment, when the device 100 receives the first user input, the device 100 executes a previously obtained classification model as described above, analyzes a plurality of images through the classification model, and determines a modality for each of the plurality of images. can

When the second user input for selecting the modality is received, the processor 120 selects the modality for the second image for which the modality is not determined among the plurality of images without going through the CNN-based classification model according to the second user input for selecting the modality. A modality of the second image may be determined. For example, for the second image whose modality has not been determined even though step S230 has been performed, the processor 120 moves the second image to a position corresponding to a specific modality in the third window 430 within the displayed screen. When a drag and drop input for moving the position of is received, the modality of the second image may be determined as the corresponding modality.

The processor 120 may display a third image obtained later from among the third image and the fourth image having the same modality. Here, the acquisition time may mean a time when an image is captured or a time when an image is stored in a corresponding folder. For example, if there are a plurality of images corresponding to the side projection images, the processor 120 may display only the most recently acquired image at a corresponding position of the third window 430 and omit display of the remaining images. there is. In one embodiment, an image other than an image of a late acquisition point may be used according to a user setting or a selection input.

In one embodiment, acquisition time points of the plurality of images may be included within a preset time range. The processor 120 may classify and store each image in a storage space (eg, memory, DB, etc.) accessed through one of a plurality of folders based on the acquisition time (eg, shooting time) of the plurality of images. . For example, the processor 120 may classify images of a specific patient on the same date or within a certain period of time into the same folder.

The processor 120 may display the first UI image 710 indicating that there is an additional image in an area adjacent to the third image. For example, the processor 120 may display the first UI image 710 in a form in which some of the remaining images are overlapped behind the most recent third image. For another example, the processor 120 may display the second UI image 730 in which the number of remaining images is displayed as a number next to the most recent image.

The processor 120 may determine the display order of images having the same modality through the UI images 710 and 730 . For example, when a user input of clicking the right mouse button on the first UI image 710 or the second UI image 730 is received, the processor 120 displays the first UI image 710 or the second UI. An image selection menu 740 including a list of file names and acquisition times of various images having the same modality as the image selected on the image 730 and having different acquisition times is superimposed on the third window 430 and displayed. , the first UI image 710 or the second UI image may be updated so that the image selected through the list is exposed at the top.

The processor 120 may update and display a classification result of a plurality of images based on a second user input for selecting a modality of an image whose modality has not been determined from among the plurality of images. For example, the processor 120 receives a drag-and-drop input for moving a specific image whose modality is not determined within the second window 420 to the position of the representative image 720 that is being displayed instead because the modality is not determined. In this case, the modality corresponding to the corresponding position may be matched with the corresponding image.

The processor 120 may update and display a classification result of a plurality of images based on a third user input for selecting a modality of an image whose modality is determined from among the plurality of images. For example, when a drag and drop input for moving an image from a location of a first modality to a location of a second modality is received in the third window 430, the processor 120 converts the modality of the corresponding image to the first modality. can be updated to the second modality.

When the processor 120 displays the corresponding image according to the setting order of the modality, if there is a modality that does not correspond to any one of the plurality of images, a modality corresponding to the corresponding modality is displayed in the third window 430. A representative image 720 indicating that the modality of the position has not yet been determined may be displayed. In one embodiment, the representative image 720 may be a blank image, a camera-shaped image, or a shape image corresponding to a corresponding modality. Accordingly, it may be intuitively notified that an image having a corresponding modality has not yet been obtained.

The processor 120 may apply a classification result to the plurality of images as a sixth user input requesting completion of classification of the plurality of images is received. For example, the processor 120 may receive a sixth user input through the completion button 450 displayed at the bottom of the second interface screen 400, and as shown in FIG. 8, a classification result for a plurality of images. may be applied to classify and display a plurality of images according to image groups on the first interface screen 300 .

When classification of the image is completed, the processor 120 may update file information about the image based on patient information. For example, the processor 120 may change the file name to include the contents of the chart number, patient name, and classified modality of a patient registered as a subject of corrective diagnosis. In addition, information on an image based on a corresponding file name may be displayed on a screen displayed for calibration diagnosis (eg, the first interface screen 300).

According to an embodiment of the present disclosure, since image groups are classified according to modalities by learning features of images based on deep learning, a high-level recognition result can be obtained and classification accuracy can be improved.

In one embodiment, the image acquisition unit 110 may be implemented as an image capture device or a communication device that receives an image from a memory. In addition, the processor 120 may perform a series of operations for displaying an image, and is electrically connected to the image acquisition unit 110, the display 130, and other components to control data flow between them. can The processor 120 may be implemented as a central processor unit (CPU) that controls overall operations of the device 100 for this purpose. The display 130 may comprehensively mean an image output device that displays an image, and includes, for example, a liquid crystal display, a thin film transistor-liquid crystal display, an organic light emitting diode (OLED). It may be an organic light-emitting diode, a flexible display, a 3D display, an electrophoretic display, and the like.

In addition, those of ordinary skill in the art may understand that other general-purpose components may be further included in the device 100 in addition to the components shown in FIG. 1 . According to an embodiment, the device 100 further includes a communication unit for communicating with other devices through a wired/wireless network, a user interface for receiving a user input, and a storage unit (eg, memory, database, cloud) for storing data. can include According to another embodiment, some of the components shown in FIG. 1 may be omitted.

The order and combination of the steps shown above is an embodiment, and the order, combination, branch, function, and subject of execution thereof may be added, omitted, or modified within the scope that does not deviate from the essential characteristics of each component described in the specification. It can be seen that this can be done. In addition, throughout the specification, 'providing' includes a process in which an object acquires specific information or directly or indirectly transmits/receives it to a specific object, and can be interpreted as comprehensively including the performance of related operations required in this process.

Various embodiments of the present disclosure are implemented as software including one or more instructions stored in a storage medium (eg, memory) readable by a machine (eg, a display device or a computer). It can be. For example, a processor (eg, processor 120 ) of the device may call at least one of one or more instructions stored from a storage medium and execute it. This enables the device to be operated to perform at least one function according to the invoked at least one instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' only means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and this term refers to the case where data is stored semi-permanently in the storage medium. It does not discriminate when it is temporarily stored.

According to one embodiment, the method according to various embodiments disclosed in the present disclosure may be included and provided in a computer program product. Computer program products may be traded between sellers and buyers as commodities. A computer program product is distributed in the form of a device-readable storage medium (e.g. compact disc read only memory (CD-ROM)), or through an application store (e.g. Play Store™) or on two user devices (e.g. It can be distributed (eg downloaded or uploaded) online, directly between smartphones. In the case of online distribution, at least part of the computer program product may be temporarily stored or temporarily created in a device-readable storage medium such as a manufacturer's server, an application store server, or a relay server's memory.

Those skilled in the art related to this embodiment will be able to understand that it can be implemented in a modified form within a range that does not deviate from the essential characteristics of the above description. Therefore, the disclosed methods are to be considered in an illustrative rather than a limiting sense. The scope of the present disclosure is shown in the claims rather than the foregoing description, and all differences within the equivalent scope will be construed as being included in the present disclosure.

100: device
110: image acquisition unit 120: processor
130: display
410: first window 420: second window
430: third window
610: projection image group 620: facial image group
630: oral image group

Claims (14)

A method for displaying a plurality of images,
acquiring a plurality of images used for calibration diagnosis;
obtaining an analysis result for the types of the plurality of images based on the learned neural network;
determining a modality for each of the plurality of images based on a result of the analysis; and
and displaying the plurality of images in an order or position determined according to the modality.
According to claim 1,
Displaying the plurality of images according to the modality
classifying the plurality of images into one of a plurality of image groups according to the modality; and
Including; displaying the plurality of images according to the image group,
The method of claim 1 , wherein the plurality of image groups include at least two of a projection image group, a facial image group, and an oral image group.
According to claim 1,
The area where the plurality of images are displayed is
a first area in which one or more first images of which the modality is determined among the plurality of images are displayed; and
and a second area in which one or more second images of which the modality is not determined among the plurality of images are displayed.
According to claim 3,
Determining the modality of each of the plurality of images
determining a modality of each of the plurality of images through a classification model based on a Convolution Neural Network (CNN) when a first user input requesting determination of the modality is received; and
determining a modality of the one or more second images as a modality corresponding to the second user input without going through the classification model when a second user input for selecting the modality is received.
According to claim 4,
Determining the modality of each of the plurality of images
When receiving the first user input, requesting a server to determine the modality of each of the plurality of images, and determining the modality of each of the plurality of images through the classification model as a response to the request from the server How to receive results.
According to claim 1,
Classifying and displaying the plurality of images
A method of displaying a third image having a later acquisition time point among the third image and the fourth image having the same modality.
According to claim 6,
Classifying and displaying the plurality of images
A method of displaying a UI image indicating that an additional image is present in an area adjacent to the third image.
According to claim 1,
Obtaining the analysis result
A method of obtaining the analysis result based on a Residual Connection (ResNet) model of a Convolution Neural Network (CNN).
In a device displaying a plurality of images,
an image acquiring unit acquiring a plurality of images used for calibration diagnosis;
a processor that obtains an analysis result for the types of the plurality of images based on a learned neural network, and determines a modality for each of the plurality of images based on the analysis result; and
A device comprising a; display for displaying the plurality of images according to an order or position determined according to the modality.
According to claim 9,
The processor
Classifying the plurality of images into one of a plurality of image groups according to the modality;
the display
displaying the plurality of images according to image groups;
The device of claim 1, wherein the plurality of image groups include at least two of a projection image group, a facial image group, and an oral image group.
According to claim 9,
The area where the plurality of images are displayed is
a first area in which one or more first images of which the modality is determined among the plurality of images are displayed; and
And a second area in which one or more second images of which the modality is not determined among the plurality of images are displayed.
According to claim 9,
The processor
When a first user input requesting the determination of the modality is received, a modality of each of the plurality of images is determined through a classification model based on a CNN (Convolution Neural Network),
and determining a modality of the one or more second images as a modality corresponding to the second user input without going through the classification model when a second user input for selecting the modality is received.
According to claim 9,
the display
A device that displays a third image having a later acquisition time point among the third image and the fourth image having the same modality.
A computer-readable recording medium recording a program for executing the method of any one of claims 1 to 8 in a computer.

Images