KR102477699B1 - A method, an imaging device and a recording medium for acquiring a lateral image for eye protrusion analysis - Google Patents

Abstract

The present application relates to a method for obtaining a lateral image for analyzing the protrusion of the eyeball. According to one embodiment, the method comprises: obtaining a front image of the face of a subject in a guided state to satisfy a first photographing condition; forming a panorama guide based on location information of first and second points extracted from the front image; guiding the movement of a photographing device to obtain a preview image corresponding to the panorama guide; and obtaining a side image of the face of the subject in a guided state to satisfy a second photographing condition. The first photographed image shows the iris region of the subject, and the second photographed image shows the corner and cornea of one eye of the subject. Accordingly, a lateral image of a face suitable for monitoring a change in eyeball protrusion can be obtained.

Description

A method for obtaining a lateral image for analyzing the projection of the eyeball, a photographing device and a recording medium performing the same

The present invention relates to a method for obtaining a side image for analyzing the protrusion of the eyeball, a photographing device and a recording medium for performing the same.

As artificial intelligence technology develops, technology is being actively developed to obtain information on the user's health condition by analyzing diagnostic images taken of the user, and to guide the user to visit the hospital so that the user can visit the hospital early. have.

For example, Tyroscope, the applicant of the present application, is a company that provides a thyroid management solution. Following the development of a software medical device that predicts the risk of thyroid dysfunction and guides visits to the hospital, it uses digital image learning of the face to improve thyroid safety. An artificial intelligence model that predicts disease has been developed, and a software medical device that predicts the risk of thyroid eye disease and guides visits to the hospital is about to be released.

As I conducted a consumer survey of a software medical device that predicts thyroid eye disease, I found out that it can be used not only for patients who want to check the timing of hospital visits, but also to monitor the treatment progress in the clinical process of thyroid eye disease treatment. In order to observe the treatment process of ophthalmopathy and leave it as data, it was required to develop a precise imaging technique that acquires lateral images that can analyze the projection of the eyeball.

One problem to be solved by the contents disclosed by the present application is to provide a method for acquiring a side image of a face, a photographing device for performing the same, and a recording medium so that the projection of the eyeball can be monitored without visiting a hospital.

Another problem to be solved by the contents disclosed by the present application is a method of acquiring an image so that a face image suitable for use as a diagnostic image can be obtained even in a state in which it is difficult to accurately detect the vertical rotation angle of the face, and performing this It is to provide a photographic device and a recording medium.

Another problem to be solved by the contents disclosed by this application is to be able to monitor the clinical activity score related to thyroid eye disease so that the general public or patients who are treating thyroid dysfunction can detect thyroid eye disease early. It is to provide a method and system.

The problem to be solved in this application is not limited to the above-mentioned problems, and problems not mentioned will be clearly understood from this application to those of ordinary skill in the art to which this application belongs.

According to one aspect of the present application, a method of acquiring a facial image of a subject to analyze protrusion of the eyeball is disclosed. The method acquires a frontal image of the subject's face in a guided state to satisfy a first photographing condition - satisfying the first photographing condition means that at least both eyes of the subject are located in a predetermined area of the first photographed image. Including being disposed-, forming a panorama guide based on the location information of the first point and the second point extracted from the front image-the panorama guide is formed in a horizontal direction based on the front of the face, and the The first point and the second point are vertically spaced apart from the front of the face - guide the movement of the photographing device so that a preview image corresponding to the panorama guide is obtained - at this time, extracted from the preview image The distance between the first point and the second point in the vertical direction is monitored so as to have a difference less than an error range determined based on the initial separation distance-, and the subject in a guided state to satisfy the second shooting condition. Obtaining a side image of the face of -The second shooting condition is, the separation distance in the vertical direction between the first point and the second point extracted from the second shooting image is an error determined based on the initial separation distance with a difference less than the range - inclusive. At this time, the iris region of the target is shown in the first captured image, and the outer corner of one eye and the cornea of the target are displayed in the second captured image so that the obtained image can be used to calculate the protrusion of the eyeball. is appearing

According to another aspect of the present application, a non-transitory computer readable medium storing one or more instructions is disclosed. When the non-transitory computer readable medium is executed by one or more processors of a computing device, it may cause the computing device to perform the above-described method.

According to another aspect of the present application, a photographing device that acquires a facial image of a subject to analyze the degree of protrusion of the eyeball is disclosed. The device includes a communication unit, a storage unit to store one or more instructions, and a control unit to execute the one or more instructions stored in the storage unit. By executing the one or more instructions, the controller acquires a frontal image of the face of the subject in a guided state to satisfy a first shooting condition, and positions of first and second points extracted from the frontal image. Forming a panorama guide based on the information, guiding movement of the photographing device so that a preview image corresponding to the panorama guide is obtained, and obtaining a side image of the subject's face in a guided state to satisfy a second photographing condition, The first photographing condition includes that at least both eyes of the subject are disposed in a predetermined area of a first photographed image, and the panorama guide is formed in a horizontal direction based on the front of the face, and the first point and the second The points are spaced apart in a vertical direction based on the front of the face, and the second photographing condition is at least a vertical distance between the first and second points extracted from the second photographed image. It may have a difference less than the error range determined based on the separation distance.

The solutions to the problems of the present application are not limited to the above-described solutions, and solutions not mentioned will be clearly understood by those skilled in the art from this application.

According to the contents disclosed by the present application, a side image of a face suitable for monitoring a change in protrusion of the eyeball may be obtained using a digital camera such as a smart phone usable by the general public.

Effects of the invention of the present application are not limited to the above-mentioned effects, and effects not mentioned will be clearly understood from this application to those of ordinary skill in the art to which this application belongs.

1 is a diagram for explaining an ophthalmopathy management system according to an exemplary embodiment.
2 is a diagram illustrating an ophthalmopathy management system including a user terminal.
3 is a block diagram of a user terminal according to an embodiment.
4 is a diagram for explaining the operation of an ophthalmopathy management system according to an embodiment.
5 is a diagram for explaining an image acquisition method according to an exemplary embodiment.
6 is a diagram for explaining the shape and operation of a photographing guide provided according to an embodiment.
7 is a diagram for explaining the shape and operation of a real-time shooting guide provided according to an embodiment.
8 is a diagram for describing a display screen after capturing an image of a user terminal according to an exemplary embodiment.
9 is a diagram for explaining an image acquisition method according to an exemplary embodiment.
10 is a diagram for explaining an operation of generating a reference guide according to an exemplary embodiment.
11 is a diagram for explaining an image acquisition method according to an exemplary embodiment.
12 and 13 are views for explaining a display screen providing a shooting guide and a reference guide according to an exemplary embodiment.
14 is a diagram for explaining an image acquisition method according to an exemplary embodiment.
15 is a diagram for explaining an image acquisition method according to an exemplary embodiment.
16 is a diagram illustrating a front image and a side image according to an exemplary embodiment.
17 is a flowchart of a method for predicting thyroid ophthalmopathy according to an embodiment.
18 is a diagram for explaining a display screen including a questionnaire input interface and confirmation interface of a user terminal according to an exemplary embodiment.
19 is a diagram for explaining an index for predicting thyroid ophthalmopathy according to an embodiment.
20 is a diagram for explaining a method of predicting prominence by analyzing a side image of a face, according to an exemplary embodiment.
21 is a diagram for explaining a diameter value of an iris region in a front image of a face and a protrusion degree of an eyeball in a side image of a face, according to an exemplary embodiment.

The foregoing objects, features and advantages of the present application will become more apparent from the following detailed description taken in conjunction with the accompanying drawings. However, the present application can apply various changes and can have various embodiments. Hereinafter, specific embodiments will be illustrated in the drawings and described in detail.

Like reference numerals designate essentially like elements throughout the specification. In addition, components having the same function within the scope of the same idea appearing in the drawings of each embodiment will be described using the same reference numerals, and overlapping descriptions thereof will be omitted.

If it is determined that a detailed description of a known function or configuration related to the present application may unnecessarily obscure the subject matter of the present application, the detailed description thereof will be omitted. In addition, numbers (eg, first, second, etc.) used in the description process of this specification are only identifiers for distinguishing one component from another component.

In addition, the suffixes "module" and "unit" for components used in the following embodiments are given or used interchangeably in consideration of ease of writing the specification, and do not have meanings or roles that are distinguished from each other by themselves.

In the following examples, expressions in the singular number include plural expressions unless the context clearly dictates otherwise.

In the following embodiments, terms such as include or have mean that features or components described in the specification exist, and do not preclude the possibility that one or more other features or components may be added.

In the drawings, the size of components may be exaggerated or reduced for convenience of explanation. For example, the size and thickness of each component shown in the drawings are arbitrarily shown for convenience of explanation, and the present invention is not necessarily limited to those shown.

If an embodiment is otherwise implementable, the order of specific processes may be performed differently from the order described. For example, two processes that are described in succession may be performed substantially concurrently, or may proceed in an order reverse to that described.

In the following embodiments, when components are connected, a case in which the components are directly connected as well as a case in which components are interposed between the components and connected indirectly is included. For example, when it is said that components are electrically connected in this specification, not only the case where the components are directly electrically connected, but also the case where the components are interposed and electrically connected indirectly is included.

According to one aspect of the present application, a method of acquiring a facial image of a subject to analyze protrusion of the eyeball is disclosed. The method acquires a frontal image of the subject's face in a guided state to satisfy a first photographing condition - satisfying the first photographing condition means that at least both eyes of the subject are located in a predetermined area of the first photographed image. Including being disposed-, forming a panorama guide based on the location information of the first point and the second point extracted from the front image-the panorama guide is formed in a horizontal direction based on the front of the face, and the The first point and the second point are vertically spaced apart from the front of the face - guide the movement of the photographing device so that a preview image corresponding to the panorama guide is obtained - at this time, extracted from the preview image The distance between the first point and the second point in the vertical direction is monitored so as to have a difference less than an error range determined based on the initial separation distance-, and the subject in a guided state to satisfy the second shooting condition. Obtaining a side image of the face of -The second shooting condition is, the separation distance in the vertical direction between the first point and the second point extracted from the second shooting image is an error determined based on the initial separation distance with a difference less than the range - inclusive. At this time, the iris region of the target is shown in the first captured image, and the outer corner of one eye and the cornea of the target are displayed in the second captured image so that the obtained image can be used to calculate the protrusion of the eyeball. is appearing

In some embodiments, satisfying the first photographing condition may include at least a horizontal rotation angle of the subject's face not exceeding a preset first reference value.

In some embodiments, satisfying the first photographing condition may include that a smile level of the subject does not exceed a preset second reference value based on at least facial expression information of the subject.

In some embodiments, satisfying the first photographing condition may include that at least a vertical rotation angle of the subject's face does not exceed a preset third reference value.

In some embodiments, satisfying the first photographing condition may include at least ambient brightness not exceeding a fourth reference value.

In some embodiments, satisfying the second photographing condition may include disposing at least the ears of the target in a predetermined area of the second photographed image.

In some embodiments, the first point may be an end point of the nose, and the second point may be an end point on the chin side of the outline of the face.

In some embodiments, guiding the movement of the photographing device to obtain a preview image corresponding to the panorama guide may include obtaining a preview image according to a preset frame rate and the first point extracted from the obtained preview image. and determining whether the vertical separation distance between the second points has a difference less than a predetermined error range based on the initial separation distance, and adjusting the distance between the photographing device and the face if it is determined that the distance is outside the predetermined error range. It may include outputting the guide.

In some embodiments, if it is determined that the distance is out of a predetermined error range, outputting a guide for adjusting the distance between the photographing device and the face includes the first point and the first point extracted from the obtained preview image. When it is determined that the vertical separation distance between the two points is large enough to exceed a predetermined error range compared to the initial separation distance, a guide to move the photographing device away from the face is output, and the above information extracted from the obtained preview image is output. When it is determined that the separation distance in the vertical direction between the first point and the second point is small enough to exceed a predetermined error range compared to the initial separation distance, outputting a guide instructing the photographing device to move closer to the face. can

In some embodiments, guiding the movement of the photographing device so that a preview image corresponding to the panorama guide is acquired is such that the position of the first point extracted from the acquired preview image is the location of the first point extracted from the front image. It is determined whether the position is moved by less than a fifth reference value compared to the position, and it is determined that the position of the first point extracted from the acquired preview image is moved by more than a fifth reference value compared to the position of the first point extracted from the front image. , outputting a guide instructing the photographing device to move upward or downward relative to the face.

In some embodiments, guiding the movement of the photographing device so that a preview image corresponding to the panorama guide is acquired is such that the location of the second point extracted from the acquired preview image is the location of the second point extracted from the front image. If it is determined whether the position of the second point extracted from the obtained preview image is moved by more than the sixth reference value compared to the position of the second point extracted from the front image, It may include outputting a guide instructing the photographing device to move upward or downward relative to the face.

In some embodiments, when the movement of the photographing device is guided so that a preview image corresponding to the panorama guide is acquired, at least some of the preview images are acquired as third captured images, the first captured image, and the second captured image. The image and the third captured image may be edited and stored as a panoramic image.

In some embodiments, a moving image may be stored by continuously acquiring images from a time when the first captured image is obtained to a time when the second captured image is acquired.

In some embodiments, satisfying the second photographing condition may include not detecting at least one eye of the subject in the second photographed image.

In some embodiments, the method may obtain a diameter value of the iris region detected in the first photographed image, and a distance between the outer corner of the eye detected in the second photographed image and the cornea located farthest from the outer corner of the eye. The method may include obtaining the projection of the eyeball by obtaining a value, obtaining a ratio between the distance value and the diameter value, and predicting the distance value by reflecting the eigenvalue of the iris region.

According to another aspect of the present application, a non-transitory computer readable medium storing one or more instructions is disclosed. When the non-transitory computer readable medium is executed by one or more processors of a computing device, it may cause the computing device to perform the above-described method.

According to another aspect of the present application, a photographing device that acquires a facial image of a subject to analyze the degree of protrusion of the eyeball is disclosed. The device includes a communication unit, a storage unit to store one or more instructions, and a control unit to execute the one or more instructions stored in the storage unit. By executing the one or more instructions, the controller acquires a frontal image of the face of the subject in a guided state to satisfy a first shooting condition, and positions of first and second points extracted from the frontal image. Forming a panorama guide based on the information, guiding movement of the photographing device so that a preview image corresponding to the panorama guide is obtained, and obtaining a side image of the subject's face in a guided state to satisfy a second photographing condition, The first photographing condition includes that at least both eyes of the subject are disposed in a predetermined area of a first photographed image, and the panorama guide is formed in a horizontal direction based on the front of the face, and the first point and the second The points are spaced apart in a vertical direction based on the front of the face, and the second photographing condition is at least a vertical distance between the first and second points extracted from the second photographed image. It may have a difference less than the error range determined based on the separation distance.

According to the present application, a method for capturing an image used to manage thyroid ophthalmopathy and a system for managing thyroid ophthalmopathy based thereon (eg, guidance to a hospital visit) are disclosed.

In the present specification, the term "eye" may refer to an area exposed to the outside through the eye area determined by the eyelid based on the eyeball. In other words, "eye" in this specification may be a concept included in "eyeball".

In the present specification, "iris region" may refer to an area in which the iris of the eye and the pupil of the eye are combined.

In the present specification, the term "subject" may refer to an object (eg, a person) that has become a subject in a photographed image acquired by the thyroid eye disease management system.

In the present specification, the term "preview image" may refer to an image obtained according to a frame rate determined at the time of capturing the image, and specifically, an image capturing operation according to a user's input or an automatic image capturing operation due to satisfying a pre-stored condition. It may refer to an image captured in a step before a photographing operation is initiated and a photographed image is stored in response thereto.

In this specification, the term "photographed image" may mean an image finally obtained according to a photographing operation of an image, and specifically, an image photographing operation according to a user's input or an automatic photographing operation resulting from satisfying a pre-stored condition. This may mean an image that is initiated and stored as a response thereto.

In this specification, the term "capturing guide" refers to a function for assisting image capturing, and for example, a preview image displayed on the display of the user terminal 1000 is output together in the form of lines or letters. It may be a shape, and as another example, it may be a form in which a voice assisting photographing is output through a speaker of the user terminal 1000, but is not limited thereto.

In the present specification, the "left and right rotation angle" of a face may refer to an angle (Yaw) of rotation based on an imaginary vertical axis passing through the center of the head side of the face. Specifically, when the face is facing the front, the left and right rotation angle of the face may be 0°, when the face is looking to the right, the left and right rotation angle of the face may be 90°, and when the face is looking to the left. In this case, the left and right rotation angle of the face may be -90°.

In this specification, the "up and down rotation angle" may mean a rotation angle (pitch) based on a virtual horizontal axis passing through the center of the ear side of the face. Specifically, when the face is facing the front, the vertical rotation angle of the face may be 0°, and when the face is facing upward (eg, the sky), the vertical rotation angle of the face may be 90°. When looking down (eg, the floor), the face's up and down rotation angle may be -90°.

In the present specification, "ophthalmia" may mean thyroid eye disease as an autoimmune eye disease known to occur in association with thyroid disease. Statistically, thyroid ophthalmopathy usually appears in conjunction with hyperthyroidism, but it may also appear before hyperthyroidism or when thyroid function is normal. Therefore, "eye disease" described herein should be interpreted as describing the disease to be managed using the eye disease management system as a disease name commonly referred to in the medical industry, but the mechanism of occurrence of the disease to be monitored is caused by the thyroid gland. It should not be interpreted to mean that

One. Eye disease management system

(One) system general

1 is a diagram for explaining an ophthalmopathy management system 10 according to an exemplary embodiment.

The ophthalmopathy management system 10 plays a role of predicting ophthalmopathy or confirming and managing the course of the disease after finding out the existence of ophthalmopathy.

According to an embodiment, the ophthalmopathy management system 10 may perform a role of acquiring an image of a target and predicting the possibility of ophthalmopathy by analyzing the obtained image. For example, the ophthalmopathy management system 10 acquires a front face image of the subject, analyzes the acquired front face image to predict a clinical activity score (CAS), and determines whether the predicted clinical activity score exceeds a reference value. It can be judged to predict the possibility of thyroid eye disease. For another specific example, the ophthalmopathy management system 10 may obtain a side face image of the subject, analyze the acquired side face image, predict the degree of projection of the eyeball, or monitor a change in the degree of projection of the eyeball.

2 is a diagram illustrating an ophthalmopathy management system including a user terminal.

In some embodiments, the user terminal 100 may be connected to other user terminals 100 and/or the eye disease management server 200 through a network. Generally, the network may be a communications network and/or a wide area network (WAN). In certain embodiments, the network may be the Internet.

The user terminal 100 may acquire an image of a body part of a target. The obtained image may be an image taken according to a preset condition so that it can be used for diagnosis in the ophthalmopathy management server 200 . The user terminal 100 may transmit the acquired image to the ophthalmopathy management server 200 .

The ophthalmopathy management server 200 may generate information about thyroid ophthalmia by analyzing the image. For example, the ophthalmopathy management server 200 may analyze the image to calculate a clinical utilization score, evaluate whether the clinical utilization score is 3 points or more, and induce visitation to the hospital for the purpose of treating thyroid ophthalmopathy at an appropriate time You can provide information to do so. As another example, the ophthalmopathy management server 200 may analyze the image to predict the projection degree of the eyeball, store the projection degree information of the eyeball, and provide information about the change in the projection degree of the eyeball to the user.

The ophthalmopathy management system 10 may be built in a shape in which at least some of the components shown in FIG. 2 are omitted, or may be built in a shape in which additional components not shown in FIG. 2 are included.

(2) Components of user terminal

3 is a block diagram of a user terminal according to an embodiment.

According to one embodiment, the user terminal 100 includes a terminal communication unit 110, a camera module 120, a terminal storage unit 130, an input/output module 140, and a terminal control unit 150. The user terminal 100 may be, for example, a smart phone, a tablet device, a laptop computer, a personal computer, or a combination thereof, but is not limited thereto.

The terminal communication unit 110 may include a wireless communication module and/or a wired communication module. The terminal communication unit 110 exemplarily includes a wired/wireless Local Area Network (LAN) module, a WAN module, an Ethernet module, a Bluetooth module, a Zigbee module, a Universal Serial Bus (USB) module, an IEEE 1394 module, It may be a WiFi module or a combination thereof, but is not limited thereto.

The camera module 120 is a digital camera and may include an image sensor and an image processor. An image sensor is a device that converts an optical image into an electrical signal, and may be composed of a chip in which a plurality of photo diodes are directly incorporated. Illustratively, the image sensor may include a Charge Coupled Device (CCD), Complementary Metal Oxide Semiconductor (CMOS), and the like. Meanwhile, the image processing unit may generate image information by processing a photographed result.

The terminal storage unit 130 is a memory, and the memory may be a storage means for storing data that can be read by a microprocessor. The terminal storage unit 130 may include a hard disk drive (HDD), solid state disk (SSD), silicon disk drive (SDD), ROM, RAM, and the like. The terminal storage unit 130 stores an operating system (OS) that drives a user terminal, various programs (and/or application programs 131) to be driven in the user terminal, and various data to be referred to by them. can be stored

The input/output module 140 may include a user input unit, an output unit, and/or an input/output interface. The user input unit receives a user's input for the user terminal 100 . The received input may be transmitted to the terminal control unit 150 . According to one embodiment, the user input unit may receive a user's input through a touch display. The output unit outputs various types of information according to the control command of the terminal control unit 150 . According to one embodiment, the output unit may include a display that visually outputs information to a user. Alternatively, the output unit may include a speaker that audibly outputs information to the user and a vibration motor that tactilely outputs information to the user.

The input/output interface may serve as an interface capable of transferring commands or data input from a user or other external device to other component(s) of the user terminal 100, for example. Also, the input/output interface may output commands or data received from other component(s) of the user terminal 100 to the user or other external devices.

The terminal controller 150 may include at least one processor. At this time, each processor may execute a predetermined operation by executing at least one instruction stored in the memory. According to an embodiment, the terminal controller 150 may control overall operations of elements included in the user terminal 100 . In other words, the user terminal 100 may be controlled or operated by the terminal control unit 150 . The terminal control unit 150 may, for example, execute calculations or data processing related to control and/or communication of at least one other component of the user terminal 100 .

The terminal control unit 150 may include one or more of a central processing unit (CPU), an application processor (AP), or a communication processor (CP).

The user terminal 100 may be provided in a form in which at least some of the components shown in FIG. 3 are omitted, or may be provided in a form in which components not shown in FIG. 3 are additionally included.

Hereinafter, unless otherwise noted, the operation of the user terminal 100 may be interpreted as being performed by the terminal control unit 150 .

2. Operation of Eye Disease Management System - Image Acquisition Method

4 is a diagram for explaining the operation of an ophthalmopathy management system according to an embodiment.

The ophthalmopathy management system 10 may acquire an image (S1000) and obtain ophthalmopathy information (S2000). An image acquisition method and an image analysis method will be described in more detail below.

According to an embodiment, both the image acquisition method and the image analysis method may be performed by the user terminal 100, and at this time, the user terminal 100 may be the only device constituting the eye disease management system 10. have.

According to another embodiment, the image acquisition method may be performed by the user terminal 100 and the image analysis method may be performed by the ophthalmopathy management server 200 . At this time, the eye disease management system 10 may include at least the electronic device 100 and the eye disease management server 200 . At this time, the user terminal 100 and the ophthalmopathy management server 200 are connected to each other to enable data exchange, and can transmit and receive necessary data to each other.

According to another embodiment, some operations of the image acquisition method may be performed by the user terminal 100 and other operations may be performed by the ophthalmopathy management server 200 . Also, some operations of the image analysis method may be performed by the user terminal 100 and other operations may be performed by the ophthalmopathy management server 200 .

Hereinafter, specific embodiments will be described on the assumption that the user terminal 100 independently performs the image acquisition method and the image analysis method independently performed by the ophthalmopathy management server 200 . However, this is only described as one embodiment for convenience of explanation, and does not mean that the invention disclosed in this specification is limited to being performed by the following subjects.

(1) Image Acquisition Method - First Embodiment

In a system for predicting thyroid ophthalmopathy by analyzing an image, it is necessary to capture an appropriate diagnosis image in order to improve the prediction accuracy of thyroid ophthalmopathy. To this end, a diagnostic image needs to be obtained in a state in which predetermined photographing conditions are satisfied. Therefore, in this embodiment, a method for obtaining an image used for analysis of thyroid eye disease will be described in detail.

In the ophthalmopathy management system 10 according to an exemplary embodiment, an image satisfying the first photographing condition may be obtained.

The first photographing condition is a criterion for image acquisition of the ophthalmopathy management system 10, whether both eyes of the subject are placed in appropriate positions on the diagnosis image, whether the vertical rotation angle of the subject's face is within an appropriate range, whether the subject's face is It can be determined by evaluating whether the left/right rotation angle is within an appropriate range, whether the subject's face occupies an appropriate portion on the image, whether the subject's smile level is within an appropriate range, and/or whether the ambient brightness is within an appropriate range. .

For example, satisfying the first photographing condition is that the eyes of the target are disposed in a predetermined area of the first photographed image, the horizontal rotation angle of the face of the target does not exceed a preset first reference value, and the target Based on the facial expression information of the subject, the smile level of the subject may not exceed a preset second reference value.

According to another embodiment, satisfying the first photographing condition is that both eyes of the target are disposed in a predetermined area of the first photographed image, a horizontal rotation angle of the face of the target does not exceed a preset first reference value, Based on the facial expression information of the target, the target's smile level may not exceed a preset second reference value, and the vertical rotation angle of the target's face may not exceed a preset third reference value.

According to another embodiment, satisfying the first photographing condition is that both eyes of the target are disposed in a predetermined area of the first photographed image, and a horizontal rotation angle of the face of the target does not exceed a preset first reference value. , the target's smile level does not exceed a preset second reference value based on the target's facial expression information, the vertical rotation angle of the target's face does not exceed a preset third reference value, and the ambient brightness 4 may not exceed the standard.

Without being limited thereto, the thyroid eye disease system 10 may determine whether the first shooting condition is satisfied by analyzing at least some of the above-described evaluation indicators or further analyzing additional indicators. , An embodiment of the image acquisition method of the present invention will be described with reference to FIG.

5 is a diagram for explaining an image acquisition method according to an exemplary embodiment.

The user terminal 100 may provide a photographing guide (SA1100). The terminal controller 150 may provide a photographing guide to the user through the input/output module 140 .

In a state in which a preview image is obtained and provided through the input/output module 140, the user terminal 100 may provide a photographing guide to overlap the preview image. In this case, the photographing guide is provided at a fixed position of the input/output module 140, and the position at which the photographing guide is provided does not change according to the characteristics of the acquired preview image.

The photographing guide may be an instruction provided to the user so that an image suitable for the above-described various evaluation indicators is obtained. The shooting guide may be an instruction for some of evaluation indicators set to be evaluated according to shooting conditions stored in the user terminal 100 . For example, the user terminal 100 may display points where both eyes of the target should be located, and may not provide a shooting guide corresponding to the remaining indicators. As another example, the user terminal 100 may display a point where both eyes of the target should be located and an area where the face of the target should be included, and may not provide a shooting guide corresponding to the remaining indicators. Meanwhile, the shooting guide may be an instruction for all evaluation indicators set to be evaluated according to shooting conditions stored in the user terminal 100 . An embodiment of the photographing guide of the present invention will be described below with reference to FIGS. 6 and 7 .

The user terminal 100 may obtain a preview image (SA1200). The terminal controller 150 may acquire a preview image using the camera module 120 (SA1200). The terminal controller 150 may acquire a preview image (SA1200) in a state in which a shooting guide is provided through the input/output module 140. A preview image may be acquired according to a preset frame rate.

The user terminal 100 may extract a plurality of indicators from the acquired preview image (SA1300). The terminal control unit 150 may extract a plurality of indicators using an algorithm stored in the terminal storage unit 130 (SA1300).

The user terminal 100 may perform step SA1300 with respect to at least some of the acquired preview images. This is because the time taken from acquiring the preview image to acquiring the next preview image may be shorter than the time required to perform SA1300 for one preview image.

The plurality of indicators extracted in step SA1300 may be information extracted to confirm the appropriateness of at least some of the aforementioned evaluation indicators. The plurality of indicators extracted in step SA1300 may be information about some of evaluation indicators set to be evaluated according to shooting conditions stored in the user terminal 100 . For a specific example, the plurality of indicators may include pupil position information, facial contour information, and the like, but are not limited thereto.

A plurality of indicators may be obtained through a landmark detection algorithm. For example, the user terminal 100 may check the eye position in the preview image using a landmark detection algorithm, and at this time, the checked eye coordinates may be an index related to the pupil position.

A plurality of indicators may be obtained using a segmentation model. For example, the user terminal 100 may input a preview image to the segmentation model to check the eye position in the preview image. In this case, the identified eye region may be an index related to the pupil position.

A plurality of indices may be obtained through a bounding box detection algorithm. For example, the user terminal 100 may input a preview image into a bounding box detection algorithm to check the positions of the eyes and face in the preview image. At this time, the checked eye area is an index related to the position of the pupil, and The face area may be an index related to a face ratio.

The indicators extracted in the SA1300 step may be related to indicators for some of the above-described evaluation indicators, and when there are two or more indicators to be extracted, the two indicators may be extracted by the same algorithm or extracted by different algorithms. It could be.

The user terminal 100 may determine whether the extracted indicator is at an appropriate location (SA1400). The terminal control unit 150 may evaluate whether the extracted indicator is at an appropriate position on the preview image by referring to the shooting conditions stored in the terminal storage unit 130 . In other words, the terminal control unit 150 may check whether the extracted indicator satisfies the criterion.

According to some embodiments, the user terminal 100 may compare the extracted index and the location of the shooting guide. For example, when the user terminal 100 acquires the coordinate values of pixels of the extracted index, it may evaluate whether the obtained coordinate values and the coordinate values of the photographing guide are separated from each other within a reference distance. Alternatively, according to some embodiments, the user terminal 100 may compare the extracted index with a pre-stored criterion. For example, when the user terminal 100 acquires left eye coordinates and right eye coordinates analyzed using a landmark algorithm as a plurality of indices, it is evaluated whether the obtained eye coordinates are within a pixel value range stored as an appropriate position. can

The user terminal 100 may evaluate the horizontal rotation angle of the face with respect to the acquired preview image (SA1500). The terminal control unit 150 calculates the left and right rotation angle of the face of the subject shown in the obtained preview image using an algorithm stored in the terminal storage unit 130, and the calculated angle is calculated by referring to the shooting conditions stored in the terminal storage unit 130. It may be determined whether the left and right rotation angle of the face is within an appropriate range.

The user terminal 100 may perform step SA1500 on at least some of the obtained preview images. This is because the time taken from acquiring the preview image to acquiring the next preview image may be shorter than the time required to perform SA1500 for one preview image.

The preview image on which step SA1300 is performed and the preview image on which step SA 1500 is performed may be the same. Alternatively, the preview image on which step SA1300 is performed may be different from the preview image on which step SA 1500 is performed. This is because the time taken to perform step SA1300 and the time taken to perform step SA1500 may be different.

The left and right rotation angle of the face may be obtained through a yaw estimation algorithm, but may also be calculated by comparing positions of body parts obtained using a landmark detection algorithm, but is not limited thereto.

The user terminal 100 may evaluate whether the horizontal rotation angle of the face is within an appropriate range (ie, the first reference value). For example, when evaluating whether or not the first reference value is within the range, it may be determined based on whether the horizontal rotation angle of the face is within ±2 degrees, but is not limited thereto.

The user terminal 100 may evaluate the vertical rotation angle of the face with respect to the acquired preview image (SA1600). The terminal controller 150 calculates the vertical rotation angle of the target's face shown in the obtained preview image using an algorithm stored in the terminal storage 130, and the calculated angle is calculated by referring to the shooting conditions stored in the terminal storage 130. It may be determined whether the vertical rotation angle of the face is within an appropriate range.

The user terminal 100 may perform step SA1600 on at least some of the obtained preview images. This is because the time taken from acquiring the preview image to acquiring the next preview image may be shorter than the time required to perform SA1600 for one preview image.

The preview image on which step SA1500 is performed and the preview image on which step SA 1600 is performed may be the same. Alternatively, the preview image on which step SA1500 is performed may be different from the preview image on which step SA 1600 is performed. This is because the time required to perform step SA1500 and the time required to perform step SA1600 may be different.

The vertical rotation angle of the face may be obtained through a pitch detection algorithm (Pitch Detection Algorithm, PDA), but may also be calculated by comparing positions of body parts obtained using a landmark detection algorithm, but is not limited thereto. .

The user terminal 100 may evaluate whether the vertical rotation angle of the face is within an appropriate range (ie, the third reference value). For example, when evaluating whether the face is within the third reference value, it may be determined based on whether the vertical rotation angle of the face is within ±2 degrees, but is not limited thereto.

The user terminal 100 may evaluate the degree of smiling with respect to the acquired image (SA1700). The terminal controller 150 calculates the degree of smile by analyzing the facial expression of the subject shown in the acquired preview image using an algorithm stored in the terminal storage unit 130, and by referring to the shooting conditions stored in the terminal storage unit 130. It can be determined whether the calculated degree of smile is within an appropriate range.

The user terminal 100 may perform step SA1700 with respect to at least some of the acquired preview images. This is because the time taken from acquiring the preview image to acquiring the next preview image may be shorter than the time required to perform SA1700 for one preview image.

The preview image on which step SA1700 is performed and the preview image on which step SA 1300 is performed may be the same. Alternatively, the preview image on which step SA1700 is performed may be different from the preview image on which step SA 1300 is performed. This is because the time taken to perform step SA1700 and the time taken to perform step SA1300 may be different.

The degree of smile may be obtained through a facial expression analysis algorithm (Face Reader), but may also be calculated by comparing positions of body parts obtained using a landmark detection algorithm, but is not limited thereto.

The user terminal 100 may evaluate whether the smile level of the face is within an appropriate range (ie, the second reference value). Here, the appropriate range (ie, the second reference value) may be an appropriate range set to prevent the deterioration of diagnosis accuracy due to deformation of the eye shape as a person's facial expression becomes brighter. For example, when evaluating whether the smile is within the second reference value, it may be determined based on whether the degree of smile is 0.1 or less, but is not limited thereto.

The user terminal 100 may capture an image (SA1800) when it is determined that all evaluation indicators set to be evaluated according to the photographing conditions stored in the user terminal 100 are satisfied.

The user terminal 100 does not capture an image unless it is determined that all of the evaluation indices set to be evaluated according to the stored photographing conditions are satisfied, even if all of the evaluation indices related to the photographing guide provided to the SA1100 are satisfied. In other words, the user terminal 100 determines whether all of the evaluation indices to be evaluated according to the shooting conditions are satisfied without being dependent on the evaluation indices provided as the shooting guide.

The user terminal 100 may capture an image (SA1800) when it is determined that evaluation indices set to be evaluated according to shooting conditions stored in the user terminal 100 are all satisfied with respect to one preview image. Alternatively, before capturing an image (SA1800), the user terminal 100 may capture an image (SA1800) when it is determined that each evaluation index set to be evaluated according to a photographing condition stored in the user terminal 100 is satisfied. can

Taking an image (SA1800) is to save a preview image obtained by the user terminal 100 as a captured image immediately after it is determined that a condition is satisfied, or to store a preview image when determining whether a condition is satisfied by the user terminal 100. This may mean saving as a captured image. Meanwhile, instead of automatically capturing an image, the user terminal 100 may activate a capturing button or provide a notification so that the user may capture an image when it is determined that the capturing condition is satisfied.

Hereinafter, for convenience of description, it is described that the user terminal 100 takes a picture when a condition according to each embodiment is satisfied, but this may be modified to activate a capture button or provide a notification to take a picture. can

Also, in some embodiments, the user terminal 100 may further perform a step of evaluating ambient brightness before capturing an image. The terminal control unit 150 may determine whether ambient brightness is in an appropriate range through a terminal sensor unit (not shown, for example, an ambient light sensor).

The user terminal 100 may evaluate whether the ambient brightness is within an appropriate range (ie, the fourth reference value). Here, an appropriate minimum criterion may be set so that the appropriate range (ie, the fourth reference value) has enough brightness to allow image analysis even without flash, and is appropriate to have enough brightness not to interfere with color analysis of the captured image. A maximum criterion may be included. For example, when evaluating whether it is within the fourth reference value, it may be determined based on whether the ambient brightness is 100 to 1000 lux, but is not limited thereto.

Also, in some embodiments, the user terminal 100 does not capture when it is determined that at least one evaluation index among several evaluation indexes determined based on the preview image does not satisfy a pre-stored condition, and at that time, the unsatisfied evaluation index is not satisfied. It can provide information about evaluation indicators. In this case, the provided information may be in the form of notifying the unsatisfactory evaluation index, or may be in the form of informing the user of a desirable action to satisfy the unsatisfactory evaluation index. For example, when the user terminal 100 determines that the level of smile of the face determined based on the preview image exceeds a predetermined criterion, the user terminal 100 may provide a message “don't laugh” through the display.

In addition, although the overall operation has been described by assuming that a photographing guide is provided in the above-described embodiment, the image acquisition method may be performed in a form in which the step of providing a photographing guide is omitted. Apart from that, the ophthalmopathy management system 10 disclosed in this specification requires image acquisition in a precisely controlled environment, so that the user terminal 100 appropriately induces the user's behavior through the shooting guide. More advanced ophthalmopathy management can be performed.

Also, according to another embodiment, the operation of the user terminal 100 described above may be performed by the ophthalmopathy management server 200 . An algorithm and related criteria for extracting a plurality of indices, an algorithm and related criteria for detecting vertical rotation angle, an algorithm and related criteria for detecting left and right rotation angles, an algorithm and related criteria for detecting degree of smile, etc. ) may be stored in The user terminal 100 may operate as an interface for receiving and providing data in relation to a user. The user terminal 100 and the ophthalmopathy management server 200 are interlocked in real time, and according to the subjective operation of the ophthalmopathy management server 200, the user terminal 100 provides an appropriate shooting guide and obtains a photographed image to treat ophthalmopathy. It may serve to transmit to the management server 200.

6 is a diagram for explaining the shape and operation of a photographing guide provided according to an embodiment. Hereinafter, it is assumed that the input/output module 140 is a display of a smart phone.

Referring to FIG. 6 , the user terminal 100 according to an embodiment may provide a fixed shooting guide.

The photographing guide may include a first guide G1 , a second guide G2 , a third guide G3 , and a fourth guide G4 . The first guide G1 may be output to indicate an appropriate position of the right eye of the subject. The second guide G2 may be output to indicate an appropriate position of the subject's left eye. The third guide G3 may be output to display an appropriate left/right rotation angle of the subject's face. The fourth guide G4 may be output to display an appropriate proportion of the target's face.

The first guide G1 may be output to directly or indirectly indicate an appropriate position of the right eye of the subject. For example, the first guide G1 is output at a point where the right eye of the subject is to be located, and may help the user align the right eye with the first guide G1. As shown in the drawing, the first guide G1 may have a shape that indicates the outline of the iris region when the target's eye is in an appropriate position, or the pupil of the target's eye when the target's eye is in an appropriate position. It may be a shape that displays a position or an outline of an eye of a target.

For a specific example, the first guide G1 may be provided in a cross shape. When the first guide G1 is provided in a cross shape, the first guide G1 may guide an image with the center of the eye disposed at an appropriate position to be captured without a pupil sensor. In other words, when the first guide G1 is provided in a cross shape, the first guide G1 may perform a function of inducing the user to match the center of the iris region of the left eye to the intersection of the cross shape, Accordingly, an appropriate diagnostic image may be acquired so that highly accurate analysis can be performed.

The second guide G2 may directly or indirectly display an appropriate position of the subject's left eye. The second guide G2 may have the same or similar shape as the first guide G1, although the region to be displayed is different. In other words, if the first guide G1 is output to display the right pupil in the shape of a cross, the second guide G2 can be output to display the left pupil in the shape of a cross.

The first guide G1 and the second guide G2 may be symmetrically provided on the display. The first guide G1 and the second guide G2 may be provided symmetrically with respect to the third guide G3.

The third guide G3 may directly or indirectly display the left and right rotation angle of the face of the target. The third guide G3 can display the horizontal rotation angle of the face as a numerical value or as a vertical line extending in the vertical direction of the display. In this case, the vertical line may be a vertical line passing through an area where the nose of the face should be located when both eyes of the target respectively correspond to the first guide G1 and the second guide G2 and the horizontal rotation angle of the face is 0. .

The fourth guide G4 may directly or indirectly display an appropriate proportion (portion) occupied by the face of the target on the image. The fourth guide G4 displays an area where the target's face should be located in a circular shape or displays the distance between the target's face and the user terminal 100 so that an image in which the target's face is occupied on the image is captured. can guide

The first guide G1 , the second guide G2 , and the fourth guide G4 may be related to the evaluation index to be determined in step SA1400 described above. The third guide G3 may be related to the evaluation index to be determined in the above-described step SA1500.

According to an embodiment, the user terminal 100 may provide a fixed shooting guide and a real-time shooting guide RG generated based on information obtained by analyzing a preview image.

The real-time shooting guide RG may be generated for evaluation index(s) related to the shooting guide provided to the user terminal 100 .

For example, the real-time shooting guide RG may be generated for all evaluation indexes related to shooting guides provided from the user terminal 100 . When the user terminal 100 provides shooting guides for the four evaluation indicators, real-time shooting guides RG for the four evaluation indicators may be provided. As another example, the real-time shooting guide RG may be generated for some of the evaluation indicators related to shooting guides provided from the user terminal 100 . When the user terminal 100 provides shooting guides for four evaluation indicators, real-time shooting guides RG for two evaluation indicators may be provided.

The real-time photographing guide RG may be provided in a shape corresponding to the photographing guide. In other words, if the shooting guide is in the shape of a vertical line, the corresponding real-time shooting guide RG may also be in the shape of a vertical line, and if the shooting guide is in the shape of a cross, the corresponding real-time shooting guide RG may also be in the shape of a cross.

At least one characteristic of the real-time photographing guide RG may be changed according to whether a corresponding evaluation index is satisfied. When it is determined that the evaluation index corresponding to the real-time shooting guide RG satisfies the pre-stored criterion, the user terminal 100 changes and provides the characteristics of the previously provided real-time shooting guide RG so that the user satisfies the condition. can be easily identified. The change in characteristics may be a form in which a dotted line is changed to a solid line, a form in which a color is changed, or a form in which the thickness of a line is changed, but is not limited thereto.

According to an embodiment, the user terminal 100 analyzes the preview image to obtain values corresponding to evaluation indicators (eg, a pupil position and a left/right rotation angle of a face), and values corresponding to the evaluation indicators. It is possible to evaluate whether or not the images satisfy the criterion, and provide a real-time photographing guide RG based on the evaluated value.

According to another embodiment, the user terminal 100 analyzes the preview image to obtain values corresponding to the evaluation indicators (eg, the position of the pupil and the left/right rotation angle of the face), and the corresponding real-time shooting guide ( RG) is provided, and by comparing the real-time recording guide RG and the recording guide, it is possible to evaluate whether values corresponding to evaluation indicators satisfy criteria.

7 is a diagram for explaining the shape and operation of a real-time shooting guide provided according to an embodiment. Hereinafter, an operation of providing the third guide G3 and the real-time guide RG associated with the third guide G3 through the user terminal 100 will be described.

According to one embodiment, the third guide G2 may be provided in a vertical line shape. According to some embodiments, the third guide G3 may be implemented in a form of providing a left and right rotation angle of the face as a numerical value, but this has a problem in that it does not intuitively reach the user how to correct the posture. To solve this problem, like the third guide G3, the horizontal rotation angle of the face may be displayed as a vertical line on the display. For example, the third guide G3 may be provided as a vertical line passing through the center of the display in the horizontal direction and extending in the vertical direction of the display.

In a state in which the third guide G3 is provided as a fixed photographing guide, the user terminal 100 may provide a real-time photographing guide RG for the left and right rotation angle of the face.

When the horizontal rotation angle of the face on the preview image is 0 degrees, the real-time shooting guide RG may be provided as a vertical line passing through the center of the display in the horizontal direction and extending in the vertical direction of the display. When the left and right rotation angle of the face on the preview image is 90 degrees, the real-time shooting guide RG may be provided as a vertical line extending in the vertical direction of the display passing through a quarter point based on the horizontal direction of the display. The user terminal 100 stores the position of the vertical line for each left and right rotation angle of the face so that points in the horizontal direction of the display can be equally displayed for each left and right rotation angle of the face on the preview image, and the real-time shooting guide RG of the display It may be provided in the form of a vertical line extending in the vertical direction to match the left and right rotation angle of the face.

Referring to FIG. 7(a) , the user terminal 100 may obtain a preview image in a state in which the third guide G3 is provided. The obtained preview image may be analyzed to calculate the horizontal rotation angle of the face, and the real-time photographing guide RG corresponding to the calculated facial horizontal rotation angle may be provided.

Since the subject's face is looking to the left, the real-time shooting guide RG is located on the right side relative to the horizontal direction of the user terminal 100 compared to the third guide G3. At this time, since the obtained horizontal rotation angle of the face does not satisfy an appropriate range (ie, the first reference value), the real-time photographing guide RG is displayed as a dotted line.

Since the left/right rotation angle of the face of the preview image does not satisfy the first reference value, the user terminal 100 may provide an operation to be performed by the user to satisfy the first reference value through the display. For example, the provided additional shooting guide may be "Move the vertical center line of the face left and right to align with the center line."

Referring to FIG. 7(b) , since the face of the subject is facing the front, the real-time shooting guide RG is located at the center of the horizontal direction of the user terminal 100 like the third guide G3. do. Since the acquired horizontal rotation angle of the face satisfies an appropriate range (ie, the first reference value), the real-time photographing guide RG is displayed as a solid line.

The user terminal 100 may selectively provide information to the user because the left and right rotation angle of the face of the preview image satisfies the first reference value. For example, the user terminal 100 may provide “automatically photographed” through a display while performing automatic photographing.

When an image for eye disease management is captured, the user terminal 100 may transmit the image to the eye disease management server 200 . Alternatively, the user terminal 100 may perform image analysis when an image for ophthalmopathy management is captured. A method of predicting the possibility of eye disease by analyzing the image and providing guidance to the user based on the analysis will be described later.

When an image for eye disease management (eg, prediction) is captured, the user terminal 100 may output the image through a display. For example, when an image is captured, the user terminal 100 may analyze the image to check the position of the eye on the image, crop the image to include the eye and the eye periphery, and output the image through a display.

In some embodiments, an operation of analyzing the image, detecting the eye, and cropping the image to include the eye and the eye periphery may be performed by the ophthalmopathy management server 200 . Alternatively, an operation of analyzing the image, detecting the eye, and cropping the image to include the eye and the eye periphery may be performed by the user terminal 100 .

In order to detect the eye, the user terminal 100 and/or the ophthalmopathy management server 200 may store a detection algorithm. Illustratively, the detection algorithm may be a landmark detection algorithm, a segmentation model, etc., but is not limited thereto.

In this embodiment, by providing the captured image to the user, the user can visually check once again whether the eye image is improperly captured, and re-photographed if necessary. Therefore, it is possible to solve the degradation of diagnosis accuracy due to the inappropriateness of the image.

In addition, in the present embodiment, by cropping and providing a photographed image, an advantage of allowing a user to check an image more intensively when checking an image with the naked eye can be derived. Therefore, it is possible to solve the degradation of diagnosis accuracy due to the inappropriateness of the image.

8 is a diagram for describing a display screen after capturing an image of a user terminal according to an exemplary embodiment.

The user terminal 100 may provide a crop image interface (CII), a confirmation interface (CI), a retake interface (RII), and a questionnaire input interface (QII) through a display.

The cropped image interface CII displays a partial area of the photographed image, and in this case, the partial area may include an eye.

Since it is important to maintain proper brightness in the analysis of eye disease, the confirmation interface CI may be an interface for checking whether the image is too bright or dark compared to the actual image and whether both eyes are photographed with the same brightness. The confirmation image CI may further include an area where a response for each item can be received. For example, a response area for whether the image is too bright or dark compared to the actual image and a response area for whether both eyes are photographed with the same brightness may be provided through the display.

The recapturing interface (RII) may be a selection interface provided so that the user can perform a recapture function when viewing the cropped image and/or confirmation items and proceeding with recapture. The recapture interface (RII) is activated regardless of whether there is a user input corresponding to the response area of the confirmation interface (CI), and when a user input for selecting the recapture interface (RII) is obtained in the user terminal 100, the recapture interface (RII) is activated. A shooting operation can be performed.

The questionnaire input interface (QII) may be a selection interface provided so that the user can input when he/she wants to go to the next step after viewing the cropped image and/or confirmation item. The questionnaire input interface QII may be an interface activated when there is a user input corresponding to the response area of the confirmation interface CI. Specifically, when it is confirmed that no appropriate response is input to the confirmation interface CI, the user terminal 100 may not perform an appropriate operation even if a user input for selecting the questionnaire input interface QII is obtained.

(2) Image Acquisition Method - Second Embodiment

In order to accurately predict thyroid ophthalmopathy, it is important to acquire an image in a state in which the horizontal and vertical rotation angles of the face and the vertical rotation angles of the face are maintained within appropriate ranges. This means that even if the same eye is photographed, the shape of the eye may be slightly different depending on the horizontal rotation angle of the face and/or the vertical rotation angle of the face at the time of capturing the image, which predicts the possibility of thyroid eye disease or protrudes. This is because it can act as a factor that impairs accuracy in analyzing .

However, when an image is obtained using a smartphone, the smartphone's CPU specifications (ex., operation speed) must be considered when selecting an algorithm for analyzing the horizontal rotation angle of the face and / or the vertical rotation angle of the face. Development of a method capable of consistently monitoring the horizontal rotation angle and/or the vertical rotation angle of the face using a low-throughput algorithm has been required.

In particular, in the case of the left-right rotation angle of the face, it can be grasped to some extent through the left-right symmetry of the face, but in the case of the vertical rotation angle of the face, the positions of the eyes, nose, mouth, etc. detected through the image (for example, through the landmark detection algorithm) detected body part), it is necessary to develop a method capable of consistently monitoring the vertical rotation angle of the face.

Accordingly, by selecting an image taken by the target in the correct posture as the reference image RI, generating a reference guide RG, providing it, and obtaining a satisfactory image, the horizontal rotation angle and/or the vertical rotation angle of the face can be determined. Disclosed is a method for obtaining a consistently maintained captured image.

However, in describing the following embodiments, an embodiment in which a reference guide RG for guiding the vertical rotation angle of the face is provided as a representative example, and a reference guide for guiding the horizontal rotation angle of the face ( RG) is not specifically described for an embodiment in which it is provided. However, the reference guide RG for guiding the left and right rotation angle of the face is implemented in a form in which reference guides RG related to a plurality of reference guide indicators spaced apart in the horizontal direction of the face are provided, and the operation thereof will be described later. It will be said that it is described enough to be practicable as it can be performed similarly to the operation of guiding the vertical rotation angle to be performed.

Similarly, although an embodiment in which reference guides RG are provided for guiding both the left and right rotation angles and up and down rotation angles of the face are not specifically described, references related to a plurality of reference guide indicators spaced apart in the vertical direction of the face It is implemented in a form in which a guide (RG) and a reference guide (RG) related to a plurality of reference guide indicators spaced apart in the horizontal direction of the face are provided, and the operation is performed similarly to the operation of guiding the vertical rotation angle to be described later It will be said that it is described to the extent that it can be fully implemented as much as possible.

9 is a diagram for explaining an image acquisition method according to an exemplary embodiment.

A user may capture a reference image RI (refer to Day 1 of FIG. 9 ).

The reference image RI may be captured in an environment suggested by the ophthalmopathy management system 10 . For example, the reference image RI may be an image captured while the user leans his back against the wall. The reference image RI may be an image captured according to the image acquisition method according to the first embodiment. The reference image RI may be an image captured by medical personnel (eg, nurses) in a hospital. The reference image RI may be an image evaluated ex post facto that satisfies the photographing condition among previously obtained diagnosis images (ie, photographed images).

A reference guide RG may be generated by analyzing the reference image RI. A method of generating the reference guide RG will be described in more detail below.

The reference guide RG (RI) may guide at least two or more points. For example, the reference guide RG may include a first reference guide RG1 and a second reference guide RG2.

The generated reference guide RG may be stored in the terminal storage unit 130 of the user terminal 100 .

The user terminal 100 may provide a reference guide RG (see Day 2 of FIG. 9 ). The user terminal 100 may provide a reference guide RG at the time of performing an image acquisition operation for ophthalmopathy management.

According to an embodiment, when performing an image acquisition operation, the user terminal 100 provides the stored reference guide RG through a display. According to another embodiment, the user terminal 100 may provide a reference guide RG and a photographing guide together when performing an image acquisition operation. A reference guide RG is provided on the preview image.

10 is a diagram for explaining an operation of generating a reference guide according to an exemplary embodiment.

The user terminal 10 may obtain a reference image RI (SB9100), extract a reference guide indicator based on the acquired reference image RI (SB9100), and generate a reference guide RG (SB9300). have.

Acquisition of the reference image RI (SB9100) may be performed through an image acquisition method according to an embodiment of the present specification. For example, the reference image RI may be obtained according to the image acquisition method described in FIG. 5 . According to some embodiments, the reference image RI may be an image taken before the onset of thyroid eye disease.

The user terminal 100 may analyze the reference image RI and extract reference guide indicators (S9200). The reference guide index is related to a body part that may appear on the front of the face, and at least two or more reference guide indexes may be obtained.

The reference guide indicator for guiding the vertical rotation angle of the face is related to at least two points spaced apart in the longitudinal direction of the face. The reference guide indicator for guiding the left/right rotation angle of the face is related to at least two points spaced apart in the horizontal direction of the face.

The reference guide indicator may be positional information corresponding to the tip of the nose and positional information corresponding to the eye. In the present specification, "location information" is information capable of confirming a location on an image, and may be, for example, coordinate values of body parts obtained through a landmark detection algorithm. Alternatively, the reference guide indicator may be positional information corresponding to the tip of the chin and positional information corresponding to the eye. However, since the disease monitored by the ophthalmopathy management system 10 is thyroid ophthalmopathy, in the case of the eye, shape changes may occur due to protrusion of the eyeball, and therefore, the reference guide index is set to a body part not related to the eye. is relatively preferable.

The reference guide indicator may be positional information corresponding to the tip of the nose and positional information corresponding to the lips. Alternatively, the reference guide indicator may be positional information corresponding to the tip of the chin and positional information corresponding to the lips. However, since the body part related to the reference guide indicator is monitored as a shooting condition in an image acquisition operation, it is relatively preferable that the reference guide indicator is set to a body part not related to a part with large movement such as lips.

The reference guide indicator may be positional information corresponding to the tip of the nose and positional information corresponding to the tip of the chin. Reference guide indicators for guiding the vertical rotation angle of the face may be positional information corresponding to the tip of the nose and positional information corresponding to the tip of the chin. However, it is not limited thereto, and the reference guide indicator is a body part that may appear in front of the face, and may be selected to include at least two or more points.

The user terminal 100 may generate a reference guide RG (SB9300). The reference guide RG may have a form in which a point corresponding to a reference guide index extracted from the reference image RI is displayed as a dot on a display, and a point corresponding to a reference guide index extracted from the reference image RI is displayed as a dot. It may be in the form of a passing line.

According to an embodiment, when the user terminal 100 provides a reference guide (RG) for guiding the vertical rotation angle of the face, it extends in the horizontal direction of the display while passing through a point corresponding to the first reference guide indicator. It may include a first reference guide RG1 and a second reference guide RG2 extending in the horizontal direction of the display while passing a point corresponding to the index of the second reference guide.

According to another embodiment, when the user terminal 100 provides a reference guide (RG) for guiding the left and right rotation angle of the face, it extends in the vertical direction of the display while passing through a point corresponding to the first reference guide index. It may include a first reference guide RG1 and a second reference guide RG2 extending in the vertical direction of the display while passing a point corresponding to the second reference guide index.

The user terminal 100 may store the reference guide RG obtained through the SB9100 to SB9300 in the terminal storage unit 130 . The stored reference guide RG may be provided on the display whenever the user terminal 100 performs an image acquisition operation for managing ophthalmia. Alternatively, the user terminal 100 may take an image for predicting ophthalmopathy in a state in which the reference guide RG is not provided, and analyze the captured image to evaluate whether the picture satisfies the reference guide RG ex post facto. have. If the photographed image determined ex post facto does not satisfy the criterion related to the criterion guide (RG), "The photographing condition was met, but it has a different characteristic from the last photograph, so it is doubtful whether the vertical rotation angle of the face is 0." can be displayed on the display. At this time, the user terminal 100 may provide a recapture interface (RII) through the display.

According to one embodiment, when the ophthalmopathy management application 131 downloaded to the user terminal 100 is executed, the user terminal 100 provides an interface to acquire the reference image RI, and the reference guide RG can create In other words, when the eye disease management application 131 is downloaded to the user terminal 100 and executed for the first time, the user terminal 100 provides an interface to acquire the reference image RI and provides the reference guide RG. can create Thereafter, the user terminal 100 may provide the stored reference guide RG whenever an image is captured through the ophthalmopathy management application 131 until a new reference image RI is obtained. In this embodiment, in particular, the reference guide index does not include the eye. The reason is that, as the eye protrudes, the distance between the eye and other indices can be closely analyzed, and thus distortion may occur through the reference guide RG.

According to another specific embodiment, when an image is captured through the ophthalmopathy management application 131 downloaded to the user terminal 100, a reference guide RG generated by analyzing the previous photographed image may be provided through the display. can When an image is captured while the reference guide RG is provided, the user terminal 100 may transmit the captured image to the eye disease management server 200 . The user terminal 100 and/or the eye disease management server 200 regenerates the reference guide RG using the image taken this time, and the criterion obtained with the image taken this time at the time of capturing the next image for managing eye disease. A guide (RG) may be provided. In other words, the user terminal 100 creates a reference guide RG whenever a captured image is acquired and provides the generated reference guide RG when acquiring the next captured image. The generated reference guide RG may be stored in the terminal storage unit 130 .

According to an embodiment, when performing an image acquisition operation, the user terminal 100 may provide the stored reference guide RG through a display. According to another embodiment, the user terminal 100 may provide a reference guide RG and a photographing guide together when performing an image acquisition operation.

11 is a diagram for explaining an image acquisition method according to an exemplary embodiment.

The user terminal 100 provides a photographing guide and a reference guide RG (SB1100), acquires a preview image (SB1200), extracts a plurality of indicators (SB1300), and determines whether the extracted indicators are at an appropriate position ( SB1400), evaluate the left and right rotation angle of the face (SB1500), and capture an image (SB1600).

Here, since SB1200, SB1500 and SB1600 of FIG. 11 can be performed similarly to SA1200, SA1500 and SA1800 of FIG. 5, duplicate descriptions will be omitted.

Referring to FIG. 12 , the user terminal 100 displays a first guide G1, a second guide G2, a third guide G3, a first reference guide RG1, and a second reference guide RG2. can be provided through The first guide G1, the second guide G2, and the third guide G3 have been described with reference to FIG. 6, and the first reference guide RG1 and the second reference guide RG2 are illustrated in FIG. Since it has been described while explaining 9, a detailed description will be omitted.

The first reference guide RG1 and the second reference guide RG2 may be displayed as parallel horizontal lines as shown in FIG. 9, but may be displayed to indicate positions corresponding to the reference guide indicators as dot guides. .

Referring again to FIG. 11 , the user terminal 100 may acquire a preview image in a state where a shooting guide and a reference guide RG are provided, and may extract a plurality of indices for the obtained preview image (SB1300).

The user terminal 100 may extract a plurality of indicators related to the reference guide RG from the preview image. The user terminal 100 may extract a plurality of indicators related to the reference guide indicator from the preview image.

Here, the plurality of indices related to the reference guide RG may be location information about the same body part corresponding to the reference guide RG and/or the reference guide indices, or location information about similar body parts. For example, if the reference guide RG is generated based on the position information of the tip of the nose, a plurality of indices related to the reference guide RG may be extracted as the position information of the tip of the nose. For another specific example, if the reference guide RG is generated based on eye position information, a plurality of indicators related to the reference guide RG may be extracted as pupil position information.

The user terminal 100 may extract at least one or more indicators corresponding to the shooting guide and extract all the indicators corresponding to the reference guide RG. 12, the user terminal 100 detects both eyes corresponding to the first guide G1 and the second guide G2, extracts a plurality of indicators, and first reference guide RG1. ) and the nose tip and chin tip corresponding to the second reference guide RG2 may be detected to extract a plurality of indices.

The user terminal 100 may determine whether the extracted indicator is at an appropriate location (SB1400). The terminal control unit 150 may evaluate whether the extracted indicator is at an appropriate position on the preview image by referring to the shooting conditions stored in the terminal storage unit 130 . In other words, the terminal control unit 150 may check whether the extracted indicator satisfies the criterion.

A method of determining whether the extracted index related to the photographing guide is at an appropriate position has already been described with reference to FIG. 5 , and thus a detailed description thereof will be omitted.

According to some embodiments, the user terminal 100 may compare an extracted indicator related to the reference guide RG with a position of the reference guide RG. For example, when the user terminal 100 acquires the coordinate values of pixels of the extracted index related to the reference guide RG, whether the obtained coordinate values and the coordinate values of the reference guide RG are separated within a reference distance. can be evaluated. Alternatively, according to some embodiments, the user terminal 100 may compare the distance between the extracted indicators with the distance between reference guide indicators. For example, when the user terminal 100 acquires the nose tip coordinates and the chin tip coordinates as a plurality of indicators, the separation distance between the two points is calculated, and the obtained separation distance differs from the separation distance between reference guide indicators within a certain range. can evaluate whether The user terminal 100 may capture an image (SB1600) when it is determined that the position of the plurality of indicators satisfies the criterion according to SB1400 and the horizontal rotation angle of the face satisfies the criterion according to SB1500. In this case, even if the vertical rotation angle of the face is not separately analyzed, a corresponding effect can be obtained.

The user terminal 100 may acquire a preview image in a state in which a photographing guide and a reference guide RG are provided. An image may be taken or an additional guide may be provided by analyzing the acquired preview image.

Referring to FIG. 13(a), since the subject's face is looking down, the distance between the tip of the nose and the tip of the chin extracted from the preview image is the same as the distance between the first reference guide RG1 and the second reference guide RG2. shorter than The location information of the tip of the nose in the preview image corresponds to the first reference guide RG1, but the location information of the tip of the chin in the preview image does not correspond to the second reference guide RG2.

Since the vertical rotation angle of the face of the preview image does not satisfy the third reference value, the user terminal 100 may provide an operation to be performed by the user through the display to satisfy the third reference value. As an example, the provided additional shooting guide may be "Rotate the face upward and align the tip of the nose and chin with the guide".

Referring to FIG. 13(b), since the subject's face is facing the front, the distance between the tip of the nose and the tip of the chin extracted from the preview image is the same as the distance between the first and second reference guides RG1 and RG2. same. Positional information of the tip of the nose in the preview image corresponds to the first reference guide RG1, and positional information of the tip of the chin in the preview image corresponds to the second reference guide RG2.

Since the vertical rotation angle of the face of the preview image satisfies the third reference value, the user terminal 100 can selectively provide information to the user. For example, the user terminal 100 may provide “automatically photographed” through a display while performing automatic photographing.

According to one embodiment, the operation of the user terminal 100 described above may be performed by the ophthalmopathy management server 200 . The user terminal 100 may operate as an interface for receiving and providing data in relation to a user. The user terminal 100 and the ophthalmopathy management server 200 are linked in real time, and according to the subjective operation of the ophthalmopathy management server 200, the user terminal 100 provides an appropriate reference guide RG and obtains a captured image. It can play a role of transmitting to the ophthalmopathy management server 200.

(3) Image Acquisition Method - Third Embodiment

The thyroid ophthalmopathy management system 10 may perform an operation of predicting the possibility of thyroid ophthalmopathy for health management and may perform an operation of monitoring the degree of protrusion due to thyroid ophthalmopathy. In particular, when the thyroid ophthalmopathy management system 10 is used during the clinical process of a pharmaceutical company developing a treatment for thyroid ophthalmopathy, it is necessary to record the 'treatment progress' according to medications, so that thyroid ophthalmopathy is predicted. It is recommended that the operation of monitoring the projection and the operation of monitoring the protrusion should be performed in parallel.

However, in order to confirm the degree of protrusion of thyroid ophthalmopathy with the naked eye or image analysis, a lateral image (SI) of the face must be obtained. It's a thing. In particular, in the prior art, in order to analyze the prominence of the face, images were acquired in the form of shooting a camera from the bottom of the face to the top, which made it impossible to take a consistent image, and furthermore, patients were able to capture images from the bottom to the top of the face. There were difficulties in commercialization and application of the service because they did not want to leave a record of the photos taken in the direction.

Below, an image acquisition method for acquiring front and side images SI of a face is specifically disclosed.

14 is a diagram for explaining an image acquisition method according to an exemplary embodiment.

This embodiment is a method of obtaining a side image SI in a monitored state based on the panoramic guide by first obtaining a front image FI and generating and providing a panorama guide accordingly.

In addition, the present embodiment describes that the user terminal 100 is guided to move according to the panorama guide generated when capturing the front image FI of the face. In this case, the distance between the user terminal 100 and the face when capturing the front image FI of the face may be maintained even when capturing the side image SI of the face, so that the front image FI Being able to utilize the size of the iris in , is a huge advantage. Accordingly, it is possible to predict the protrusion of the eyeball by analyzing the side image SI.

15 is a diagram for explaining an image acquisition method according to an exemplary embodiment.

The user terminal 100 may acquire the front image FI (SC1100). For example, the front image FI may be captured by the image acquisition method according to the first embodiment.

The front image FI may be acquired in a guided state to satisfy the first photographing condition. For example, satisfying the first photographing condition may be that both eyes of the target are disposed in a predetermined area of the first photographed image. As another example, satisfying the first photographing condition may be that both eyes of the target are disposed in a predetermined area of the first photographed image and that a horizontal rotation angle of the face of the target does not exceed a preset first reference value. As another example, satisfying the first photographing condition is that both eyes of the subject are disposed in a predetermined area of the first photographed image, that the left and right rotation angle of the subject's face does not exceed a preset first reference value, and that the subject's Based on the facial expression information, the target's smile level may not exceed a preset second reference value. As another example, satisfying the first photographing condition means that both eyes of the subject are disposed in a predetermined area of the first photographed image, that the left and right rotation angle of the subject's face does not exceed a preset first reference value, and that Based on the facial expression information, the target's smile level may not exceed a preset second reference value, and the vertical rotation angle of the target's face may not exceed a preset third reference value. As another example, satisfying the first photographing condition means that both eyes of the subject are disposed in a predetermined area of the first photographed image, that the left and right rotation angle of the subject's face does not exceed a preset first reference value, and that Based on facial expression information, the subject's smile level does not exceed a preset second reference value, the vertical rotation angle of the subject's face does not exceed a preset third reference value, and the ambient brightness exceeds a fourth reference value. may not be exceeded.

When the front image FI is obtained, the user terminal 100 may provide a panorama guide (SB1200). The panorama guide provided after the front image FI is acquired by the user terminal 100 may be generated based on the front image FI.

The panorama guide may be generated based on information extracted from the front image FI. According to an embodiment, the user terminal 100 may generate and provide a panorama guide based on location information of the first and second points extracted from the front image FI.

The user terminal 100 may extract a panorama guide indicator (SC1210). The panorama guide index is a body part that can appear in the frontal image FI of the face and a body part that can also appear in the side image SI of the face. The panorama guide index relates to at least two points vertically spaced apart from the front of the face.

The panorama guide indicator may be location information corresponding to the tip of the nose and location information corresponding to the eye. Alternatively, the panorama guide indicator may be positional information corresponding to the tip of the chin and positional information corresponding to the eye. However, since the right eye is not detected when the left side image (SI) of the face is taken, and the left eye is not detected when the right side image (SI) is taken, the panoramic guide index is It may be preferable not to include

The panorama guide index may be positional information corresponding to the tip of the nose and positional information corresponding to the lips. Alternatively, the reference guide indicator may be positional information corresponding to the tip of the chin and positional information corresponding to the lips. Since the body part related to the panorama guide index is monitored under shooting conditions until the side image SI is acquired, it is relatively preferable that the panoramic guide index is set to a body part not related to a part with large movement such as the lips. .

The panorama guide index may be location information corresponding to the tip of the nose and location information corresponding to the tip of the chin. However, it is not limited thereto, and the panorama guide indicator is a body part that may appear on the front and side of the face, and may be selected to include at least two or more points.

The user terminal 100 may generate a panorama guide based on the extracted panorama guide index (S1220). The panorama guide may be in the form of displaying a point corresponding to a panorama guide indicator extracted from the front image FI as a dot on the display. For example, the panorama guide may be provided in the form of a line passing through a point corresponding to a panorama guide indicator extracted from the front image FI, or an indicator for displaying a location.

According to an embodiment, when the user terminal 100 provides a panorama guide, the first panorama guide and second panorama guide indicators extending in the horizontal direction of the display pass through points corresponding to the first panorama guide indicators. It may be a form including a second panorama guide extending in the horizontal direction of the display while passing the point of doing.

The user terminal 100 may provide the generated panorama guide through a display (SC1230). The user terminal 100 may monitor the panorama guide (SC1300) while the panorama guide is provided through the display.

Specifically, the user terminal 100 may extract a plurality of indices from the preview image (SC1310). Here, the plurality of indicators may be indicators corresponding to the panorama guide indicators of S1210. Here, the plurality of indicators may be location information on the same body part corresponding to the panorama guide and/or the panorama guide indicator, or location information on similar body parts. For example, if the panorama guide is generated based on the location information of the tip of the nose, a plurality of indices may be extracted as the location information of the tip of the nose. For another specific example, if the panorama guide is generated based on eye position information, a plurality of indicators may be extracted as pupil position information.

According to some embodiments, when the user terminal 100 monitors the panorama guide (SC1300) and acquires the front image FI, it may not evaluate whether various evaluation indicators considered as photographing conditions are satisfied. For example, even if the positions of both eyes are monitored when taking a frontal picture, the positions of both eyes may not be monitored when the user terminal 100 is moved while monitoring the panorama guide.

According to some embodiments, when monitoring the panorama guide (SC1300), the user terminal 100 may continuously evaluate the vertical rotation angle of the face of the target in the preview image. In this case, it is possible to block variables such as a shortening of the separation distance between the tip of the nose and the tip of the chin of the target according to the vertical rotation angle of the face, so that the accuracy of constantly monitoring the distance between the target and the user terminal 100 is further improved. .

The user terminal 100 may determine whether the extracted indicator is at an appropriate location (SC1320). The user terminal 100 may provide a user terminal movement guide (SC1330).

The user terminal 100 may guide the movement of the user terminal 100 so that a preview image corresponding to the panorama guide is obtained.

The user terminal 100 may determine whether the extracted indicator is at an appropriate location (SC1320).

According to some embodiments, the user terminal 100 may compare the index extracted in step SC1320 with the location of the panorama guide. For example, when the user terminal 100 acquires coordinate values of a plurality of indices in step SC1320, it may be evaluated whether the obtained coordinate values and the corresponding coordinate values of the panorama guide are separated by a reference distance.

For a specific example, the user terminal 100 may extract location information of the tip of the nose and the tip of the chin of the face as a panorama guide indicator. The user terminal 100 may monitor whether movement of the user terminal 100 corresponding to the panorama guide is being performed while providing a panorama guide generated using a panorama guide indicator. The user terminal 100 may extract positions corresponding to the tip of the nose and the tip of the chin of the face with respect to the preview image, and correspond the extracted positions of the tip of the nose and the tip of the chin to the panorama guide index on a one-to-one basis to have a difference within a predetermined error range. can be monitored.

The user terminal 100 may output a guide for adjusting the position between the user terminal 100 and the face. Specifically, when it is determined that the first point (eg, the tip of the chin) extracted from the preview image is moved by an amount exceeding the fifth reference value compared to the first point extracted from the frontal image FI, the user terminal 100 may output a guide to move the user terminal 100 upward or downward relative to the face. In addition, when it is determined that the second point (eg, the tip of the nose) extracted from the preview image is moved by an amount exceeding the sixth reference value compared to the second point extracted from the frontal image FI, the user terminal 100 A guide instructing the user terminal 100 to move upward or downward relative to the face may be output.

Alternatively, according to some embodiments, the user terminal 100 may compare the separation distance between indicators extracted in step SC1320 with the separation distance between reference guide indicators. For example, when the user terminal 100 acquires the coordinates of the tip of the nose and the coordinates of the chin of a plurality of indicators in step SC1320, the separation distance between the two points is calculated, and the obtained separation distance is the initial separation distance between the panorama guide indicators and It can be evaluated whether the difference is within a certain range.

For a specific example, the user terminal 100 may extract positions of the tip of the nose and the tip of the chin of the face as panorama guide indicators. The user terminal 100 may monitor whether shooting according to the panorama guide is being performed while providing the panorama guide generated using the panorama guide indicator. The user terminal 100 may extract the positions of the tip of the nose and the tip of the chin of the face in the preview image to calculate the separation distance, and monitor the difference between the initial separation distance between the panorama guide indicators and a predetermined error range.

When it is determined that the user terminal 100 is out of a predetermined error range, the user terminal 100 may output a guide for adjusting the distance between the user terminal 100 and the face. Specifically, the vertical separation distance between the first point (eg, the tip of the chin) and the second point (eg, the tip of the nose) extracted from the preview image is the initial separation distance (ie, in the front image FI). If it is determined that the user terminal 100 is large enough to exceed a predetermined error range compared to the positions corresponding to the first and second points), the user terminal 100 may output a guide instructing the user terminal 100 to move away from the face. . Conversely, when it is determined that the vertical separation distance between the first and second points extracted from the preview image is smaller than the initial separation distance to exceed a predetermined error range, the user terminal 100 returns the user terminal 100 A guide to move closer to the face can be output.

The user terminal 100 may acquire the side image SI (SC1400). The user terminal 100 may acquire the side image SI in a guided state to satisfy the second photographing condition (SC1400).

In an environment where the panorama guide monitoring is performed, when the user terminal 100 moves to a position where the side image SI of the face is acquired, the user terminal 100 may acquire the side image SI (SC1400).

Satisfying the second photographing condition may be determined based on an index capable of recognizing that the side of the face is satisfied. In addition, satisfying the second photographing condition may be determined to satisfy a criterion for performing panorama guide monitoring.

For example, satisfying the second photographing condition means that the separation distance in the vertical direction between the first point and the second point extracted from the second photographed image has a difference less than an error range determined based on the initial separation distance. it could be Satisfying the second shooting condition means that the separation distance in the vertical direction between the first and second points extracted from the second shooting image has a difference less than the error range determined based on the initial separation distance and It may include disposing the ear in a predetermined area of the second photographed image. Satisfying the second photographing condition means that the separation distance in the vertical direction between the first and second points extracted from the second photographed image has a difference less than an error range determined based on the initial separation distance and the subject It may include that one eye of is not detected in the second captured image.

16 is a diagram illustrating a front image and a side image according to an exemplary embodiment.

The standard separation distance RSD of the front image FI acquired according to the image acquisition method according to the present embodiment has a value similar to the side separation distance SSD of the side image SI. That is, the front and side images SI may be acquired while maintaining a constant distance between the user terminal 100 and the object at the time of capturing the front image FI and the capturing time of the side image SI.

In order to perform the proptosis analysis method described below, a means capable of estimating the actual distance is required. It has been found that people's iris diameters are almost similar, so when acquiring front and side images SI according to the present embodiment, it is assumed that the diameter of the iris obtained from the front image FI is a pre-stored length, Since the degree of protrusion of the eyeball can be predicted, there is an advantage in that the degree of protrusion of the eyeball can be predicted without a separate absolute value acquisition means.

The image acquisition method described in FIGS. 14 to 16 may be performed to acquire an image SI of the right side of the face, may be performed to acquire an image SI of the left side of the face, and left and right sides of the face. It may be performed twice to obtain the lateral image SI.

Through the above-described image acquisition method, a front face image FI and a face side image SI may be obtained. Alternatively, a moving image including a frame corresponding to the front image FI of the face and a frame corresponding to the side image SI of the face may be acquired through the above-described image acquisition method. In this case, the moving image may be a moving image continuously photographed from the point of time when the front image FI of the face is obtained to the point of time when the side image SI of the face is obtained. Alternatively, a panoramic image edited to include the front face image FI and the face side image SI may be acquired through the above-described image acquisition method.

According to one embodiment, the operation of the user terminal 100 described above may be performed by the ophthalmopathy management server 200 . The user terminal 100 may operate as an interface for receiving and providing data in relation to a user. The user terminal 100 and the ophthalmopathy management server 200 are linked in real time, and according to the subjective operation of the ophthalmopathy management server 200, the user terminal 100 provides an appropriate panoramic guide, and the front and side images (SI ) and transmits them to the ophthalmopathy management server 200.

3. Operation of ophthalmopathy management system - image analysis method

(One) Image Analysis Method - First Embodiment

17 is a flowchart of a method for predicting thyroid eye disease according to an embodiment.

An image obtained from the user terminal 100 may be transmitted to the ophthalmopathy management server 200 .

The ophthalmopathy management server 200 may perform image analysis using the received image (SA2100). The value obtained through image analysis may be a prediction value regarding whether a person who is a subject of the received image has symptoms of conjunctival congestion, conjunctival edema, eyelid redness, eyelid edema, and lacrimal edema. For example, when the image includes a face image of the subject, the ophthalmopathy management server 200 analyzes the face image to determine whether conjunctival congestion is expected, conjunctival edema is expected, or eyelid redness is expected. information about whether the patient has eyelid edema, is expected to have eyelid edema, and/or is expected to have lacrimal edema.

A predicted value of at least one of conjunctival congestion, conjunctival edema, eyelid redness, eyelid edema, and lacrimal edema may be obtained through an artificial intelligence model. According to some embodiments, the ophthalmopathy management server 200 obtains a predictive value using at least five artificial intelligence models, and each artificial intelligence model is a predictive value of conjunctival congestion, conjunctival edema, eyelid redness, eyelid edema, or lacrimal edema. It can be learned to output. At this time, the image used as an input of the artificial intelligence model may be an image preprocessed using cropping, masking, inversion, and/or resizing of the face image.

The user terminal 100 may transmit a user response to the ophthalmopathy management server 200 . The user terminal 100 may output a user interface through the input/output module 140 to obtain user responses to whether there is voluntary pain in the back of the mouth or pain when moving the eyes. The ophthalmopathy management server 200 may obtain predicted values for voluntary pain in the posterior part of the mouth and pain when moving the eyes based on the received user response.

The ophthalmopathy management server 200 may calculate an ophthalmopathy prediction score (SA2300) using the predicted value obtained through the image analysis (SA2100) and the predicted value obtained through the user response (SA2200). Here, the ophthalmopathy prediction score is a value predicted by techniques such as image analysis of the ophthalmopathy management system for the clinical activity score, conjunctival congestion, conjunctival edema, eyelid redness, eyelid edema, lacrimal hillock edema, and spontaneous pain after mouth swelling. And it may be calculated by predicting the presence or absence of pain when moving the eyes as 1 point each.

The ophthalmopathy management server 200 may guide the user to visit the hospital (SA2400) when the ophthalmopathy prediction score is greater than or equal to a reference value. The server 200 may provide an eye disease prediction score to the user terminal 100 to guide the user to visit the hospital. The server 200 may transmit necessary data so that the user terminal 100 can output an ophthalmopathy prediction score and/or a visit guide.

The user terminal 100 may output a message recommending that the patient go to the hospital and receive treatment. The user terminal 100 may recommend that the patient take a drug used for treatment of thyroid eye disease. Accordingly, without being limited thereto, the user terminal 100 may guide the patient to perform procedures necessary for treating thyroid eye disease.

18 is a diagram for explaining a display screen including a questionnaire input interface and confirmation interface of a user terminal according to an exemplary embodiment.

The questionnaire input interface (PII) may be an interface for obtaining each user's response to whether there is voluntary pain in the posterior region or pain when moving the eyes. As an example, an interface for outputting “I feel pain as if pushing forward from the back of the eye” and “I feel stiffness when glancing up and down” and receiving a user's response to each of them is of the user terminal 100. It may be provided through the input/output module 140.

When user inputs for the two questionnaires are obtained through the user terminal 100 and “next” is pressed, a confirmation interface CI may be output.

The confirmation interface (CI) may perform a function of outputting items selected by the user in the survey input interface (PII) and verifying whether the user's intended input matches. The user's response obtained from the questionnaire input interface (PI) is of high importance because it is used to predict eye disease, and the 2-STEP input method is adopted to predict the result with high accuracy by confirming it to the user once more.

For the same reason, if only one response was obtained from the questionnaire input interface (PII) of "I feel a pushing pain from the back of my eye" and "I feel stiff when I glare my eyes up and down", even if I press the "Next" button It may not switch to the Confirmation Interface (CI).

19 is a diagram for explaining an index for predicting thyroid ophthalmopathy according to an embodiment.

According to another embodiment, the criterion for guiding visits for thyroid eye disease can be determined by calculating the clinical activity score based on 7 points and determining whether it is 3 points or more. However, in this embodiment, the guide for visiting the hospital for thyroid eye disease The criterion for determining whether a clinical activity score is 4 or higher by calculating the clinical activity score on a 10-point basis will be explained.

At this time, the ophthalmopathy management server 200 may analyze protrusion, calculate an eye movement angle, and obtain an estimated visual acuity.

A method of analyzing protrusion will be described in detail in "Image Analysis Method - Second Embodiment" below.

Eye movements can be evaluated by analyzing moving images. For example, while taking a frontal face image of the user, guides may be given to move the user's eyes in eight directions, and an eyeball movement angle may be obtained based on the maximum movement position. The eyeball movement angle can be calculated in the form of calculating a budget based on the theory that the size of the human eyeball and the iris are generally similar, with the diameter of the human eyeball or the diameter of the human iris as a constant. For example, the diameter of the eyeball may be 24 mm, the diameter of the iris may be set to 11 mm, or the eyeball movement angle may be calculated with different diameters of the iris depending on race, gender, and age.

The expected visual acuity compares the size of the iris obtained from the frontal image (FI) of the face with the absolute size of the iris, predicts the distance between the face and the user terminal 100, and converts the letters output through the user terminal 100 to the user. The visual acuity may be estimated by converting the size of the case in which ? is not answered correctly into the distance between the expected face and the user terminal 100 .

Alternatively, in order to exclude factors such as nearsightedness and hyperopia, the distance between the user terminal 100 and the face is set to 3m or 5m, and letters of an appropriate size are output through the display to evaluate whether or not the user answered correctly. eyesight can be saved. Even in this case, the distance between the user terminal 100 and the face may be calculated based on the size of the iris obtained from the frontal image FI of the face.

(2) Image Analysis Method - Second Embodiment

20 is a diagram for explaining a method of predicting prominence by analyzing a side image of a face, according to an exemplary embodiment.

The ophthalmopathy management server 200 may obtain a diameter value of the iris region from the front image FI (SB2310). For example, the ophthalmopathy management server 200 may obtain a pixel value corresponding to the diameter of the iris region through landmark extraction of the frontal image FI of the face. As another example, the ophthalmopathy management server 200 may obtain a pixel value corresponding to the diameter of the iris region by using the front image FI of the face and the segmentation model. As another example, the ophthalmopathy management server 200 may insert the frontal image FI of the face into a diameter prediction model of the iris region to obtain a corresponding pixel value or diameter prediction value.

When the ophthalmopathy management server 200 acquires a pixel value corresponding to the diameter of the iris region in the frontal image FI, it calculates an actual size value corresponding to one pixel based on the standard size of the iris region and calculates the actual size value. can be saved

The ophthalmopathy management server 200 may acquire a corner-corneal distance value from the side image SI (SB2320). For example, the ophthalmopathy management server 200 may obtain a pixel value corresponding to the distance between the outer corner of the eye and the cornea of the frontal image FI by using a segmentation model. As another example, the ophthalmopathy management server 200 may obtain a corresponding pixel value or an expected distance value by inserting the lateral image SI of the face into the caudal-corneal distance prediction model.

The ophthalmopathy management server 200 may predict the projection of the eyeball (SB2330). The ophthalmopathy management server 200 may predict the protrusion of the eyeball by considering the acquired pixel value corresponding to the distance between the outer corner of the eye and the cornea and the actual size value of one pixel calculated based on the standard size of the iris region. have.

21 is a diagram for explaining a diameter value of an iris region in a front image of a face and a protrusion degree of an eyeball in a side image of a face, according to an exemplary embodiment.

In the SB2310 of the above-described operation, the expected length of the diameter value (ID) of the iris region in the frontal image FI of the face may be calculated. In the SB2320 of the above-described operation, it is possible to calculate the expected length of the caudal-corneal distance in the side image SI of the face.

The ophthalmopathy management server 200 may provide the side picture of the face together with the date (and/or time) obtained through the user terminal 100 . The ophthalmopathy management server 200 maps and provides a cropped image and a date on which the image was acquired so that the eyeball-corneal distance of the face is displayed so that the user can check the change in the proptosis of the eyeball through the user terminal 100. can

The method according to the above embodiments can be written as a program that can be executed on a computer, and can be implemented in a general-purpose digital computer that operates the program using a computer-readable recording medium. In addition, the data structure, program command, or data file that can be used in the above-described embodiments of the present invention can be recorded on a computer-readable recording medium through various means. The computer-readable recording medium may include all types of storage devices in which data readable by a computer system is stored. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, floptical disks and A hardware device specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, etc. may be included. Also, the computer-readable recording medium may be a transmission medium that transmits signals designating program commands, data structures, and the like. Examples of the program instructions may include not only machine language codes generated by a compiler but also high-level language codes that can be executed by a computer using an interpreter or the like.

As described above, although the embodiments of the present application have been described with limited embodiments and drawings, the technical idea disclosed in this specification or the embodiments thereof are not limited to the above-described embodiments, which are common in the field to which the present invention belongs. Numerous modifications and variations from these descriptions are possible for those skilled in the art. Therefore, the scope of the present application is shown in the claims rather than the foregoing description, and all equivalent or equivalent modifications thereof will be said to belong to the scope of the technical spirit of the present invention.

10: eye disease management system
100: user terminal
200: ophthalmopathy management server
300: data storage server

Claims (17)

A method of acquiring a face image of a target performed by a controller to analyze the protrusion of an eyeball,
Obtaining a front image of the subject's face in a guided state to satisfy a first photographing condition - Satisfying the first photographing condition includes placing at least both eyes of the subject in a predetermined area of the first photographing image. box-;
A panorama guide is formed based on location information of a first point and a second point extracted from the frontal image - the panorama guide is formed in a horizontal direction based on the front of the face, and the first point and the second point are formed. are spaced apart in a vertical direction based on the front of the face-;
The movement of the photographing device is guided so that a preview image corresponding to the panorama guide is acquired - at this time, the vertical separation distance between the first and second points extracted from the preview image is based on the initial separation distance. Monitoring to have a difference less than the error range set by -; and
Obtaining a side image of the subject's face in a guided state to satisfy a second shooting condition - The second shooting condition is a vertical direction between the first point and the second point extracted from the second shooting image The separation distance to has a difference less than the error range determined based on the initial separation distance-; Including,
The iris region of the subject is shown in the first shot image, and the corner of the eye and the cornea of one eye of the subject are shown in the second shot image so that the obtained image can be used to calculate the protrusion of the eyeball. ,
How to acquire images.
According to claim 1,
Satisfying the first photographing condition includes at least a horizontal rotation angle of the subject's face not exceeding a preset first reference value.
How to acquire images.
According to claim 2,
Satisfying the first photographing condition includes that the degree of smiling of the subject does not exceed a predetermined second reference value based on at least facial expression information of the subject.
How to acquire images.
According to claim 3,
Satisfying the first photographing condition includes at least a vertical rotation angle of the subject's face not exceeding a preset third reference value.
How to acquire images.
According to claim 4,
Satisfying the first shooting condition includes at least ambient brightness not exceeding a fourth reference value.
How to acquire images.
According to claim 1,
Satisfying the second shooting condition includes at least the ear of the subject being disposed in a predetermined area of the second shooting image.
How to acquire images.
According to claim 1,
The first point is the end point of the nose, and the second point is the end point of the chin side of the outline of the face.
How to acquire images.
According to claim 1,
Guiding the movement of the photographing device so that a preview image corresponding to the panorama guide is obtained,
Acquiring a preview image according to a preset frame rate;
Determining whether a vertical separation distance between the first and second points extracted from the obtained preview image has a difference less than a predetermined error range based on the initial separation distance;
Including outputting a guide for adjusting the distance between the photographing device and the face when it is determined that it is out of the predetermined error range,
How to acquire images.
According to claim 8,
If it is determined that it is out of the set error range, outputting a guide for adjusting the distance between the photographing device and the face,
When it is determined that the vertical separation distance between the first and second points extracted from the obtained preview image is greater than the initial separation distance to exceed a predetermined error range, move the photographing device away from the face. outputs a guide,
When it is determined that the vertical separation distance between the first and second points extracted from the obtained preview image is smaller than the initial separation distance to exceed a predetermined error range, move the photographing device closer to the face. which outputs the guide,
How to acquire images.
According to claim 8,
Guiding the movement of the photographing device so that a preview image corresponding to the panorama guide is obtained,
Determining whether the position of the first point extracted from the obtained preview image is moved less than a fifth reference value compared to the position of the first point extracted from the front image;
When it is determined that the position of the first point extracted from the obtained preview image is moved by an amount exceeding the fifth reference value compared to the position of the first point extracted from the front image, the photographing device is moved upward or downward relative to the face. Shikira outputs a guide,
How to acquire images.
According to claim 10,
Guiding the movement of the photographing device so that a preview image corresponding to the panorama guide is obtained,
Determining whether the position of the second point extracted from the obtained preview image is moved less than a sixth reference value compared to the position of the second point extracted from the front image;
When it is determined that the position of the second point extracted from the obtained preview image is moved by an amount exceeding the sixth reference value compared to the position of the second point extracted from the front image, the photographing device is moved upward or downward relative to the face. Shikira outputs a guide,
How to acquire images.
According to claim 1,
When the motion of the photographing device is guided so that a preview image corresponding to the panorama guide is acquired, at least some of the preview images are acquired as third photographed images;
The first captured image, the second captured image, and the third captured image are edited and stored as a panoramic image.
How to acquire images.
According to claim 1,
Moving images are stored by continuously acquiring images from the time the first captured image is acquired to the time the second captured image is acquired.
How to acquire images.
According to claim 1,
Satisfying the second shooting condition includes that at least one eye of the subject is not detected in the second shooting image.
How to acquire images.
In the method of analyzing an image acquired by performing the image acquisition method according to claim 1,
obtaining a diameter value of the iris region detected in the first captured image;
obtaining a distance value between the tail of the eye detected in the second photographed image and the cornea located farthest from the tail of the eye;
Obtaining the projection of the eyeball by obtaining a ratio between the distance value and the diameter value and predicting the distance value by reflecting the eigenvalue of the iris region;
How to analyze images.
A non-transitory computer-readable medium storing one or more instructions, which, when executed by one or more processors of a computing device, cause the computing device to: A non-transitory computer readable medium for performing a method according to.
A photographing device for acquiring an image of a face of a target in order to analyze the protrusion of the eyeball,
communications department;
a storage unit for storing one or more instructions; and
And a control unit for executing the one or more instructions stored in the storage unit,
The controller, by executing the one or more instructions,
Obtaining a frontal image of the subject's face in a guided state to satisfy a first shooting condition;
Forming a panorama guide based on location information of a first point and a second point extracted from the front image;
Guide the movement of the photographing device so that a preview image corresponding to the panorama guide is acquired;
Obtaining a side image of the subject's face in a guided state to satisfy a second shooting condition;
The first photographing condition includes that at least both eyes of the subject are disposed in a predetermined area of a first photographed image,
The panoramic guide is formed in a horizontal direction based on the front of the face, and the first point and the second point are spaced apart in the vertical direction based on the front of the face,
In the second shooting condition, a separation distance in the vertical direction between the first and second points extracted from at least the second shooting image has a difference less than an error range determined based on the initial separation distance,
filming device.

Images