KR102501665B1 - Lip motion information analysis method and device - Google Patents

Abstract

The present invention relates to a method and apparatus for analyzing lip movement information. The method for analyzing lip movement information according to the present invention tracks an upper lip landmark corresponding to the upper lip and a lower lip landmark corresponding to the lower lip in a face photographed image. A lip movement tracking step, and visualizing lip movement information based on data obtained by tracking the upper lip landmark and the lower lip landmark. According to the present invention, lip movement information can be visualized and displayed when opening or closing the mouth, and by quantifying and displaying the maximum movement range of the lips, objectively analyzing the patient's lip movement information can be provided. .

Description

Lip motion information analysis method and device {Lip motion information analysis method and device}

The present invention relates to a method and apparatus for analyzing lip movement information.

The temporomandibular joint is a joint formed by the meeting of the lower jawbone and the skull bone, and plays a key role when chewing food or speaking. Problems with the jaw joint and muscles are called temporomandibular joint disease.

For accurate diagnosis of temporomandibular joint disease, X-ray, MRI, CT scan, and ultrasound imaging are performed. In addition, to find out the degree of restriction of jaw movement, the mouth opening range is measured by opening the mouth as much as possible, or the sound generated by the jaw joint when the mouth is opened or closed is examined.

If you have a disease in the temporomandibular joint, you may show symptoms of twisting your jaw from side to side and opening or closing your mouth. Conventionally, an examination is performed in such a way that a clinician directly observes and evaluates a patient's mouth opening or closing motion, and evaluation results may vary depending on the clinician.

Korean Patent Publication No. 2017-0052910 (published date: 2017-05-15)

Therefore, the technical problem to be solved by the present invention is to provide a lip movement information analysis method and apparatus that support the objective analysis of patient's lip movement information for diagnosing temporomandibular joint disease.

A lip movement information analysis method according to the present invention for solving the above technical problem includes a lip movement tracking step of tracking an upper lip landmark corresponding to the upper lip and a lower lip landmark corresponding to the lower lip in a face photographed image, and and visualizing lip motion information based on data obtained by tracking the upper lip landmark and the lower lip landmark.

The method may further include recognizing a plurality of predetermined landmarks in the face photographing image, and aligning face inclinations in a predetermined direction using at least some of the plurality of predetermined landmarks. .

The visualized lip motion information may include a graph representing a movement trajectory of the lower lip landmark with the upper lip landmark as a reference point or a graph representing a movement trajectory of the upper lip landmark with the lower lip landmark as a reference point. can

In the graph, a section in which the upper lip landmark and the lower lip landmark move in a direction away from each other and a section in which the landmark of the lower lip move in a direction closer to each other may be expressed in different colors or shapes.

The visualized lip motion information may include a horizontal maximum movement width and a vertical maximum movement width of the upper lip landmark or the lower lip landmark.

It may include a computer-readable recording medium on which a program for executing the method is recorded on a computer.

An apparatus for analyzing lip movement information according to the present invention for solving the above technical problem is an image acquisition unit that acquires a photographed face image, recognizes a plurality of predetermined landmarks in the photographed face image, and recognizes the plurality of landmarks. An image processing unit tracking an upper lip landmark corresponding to the upper lip and a lower lip landmark corresponding to the lower lip, and visualizing lip motion information based on data tracking the upper lip landmark and the lower lip landmark. It includes a visualization unit that

According to the present invention, lip movement information can be visualized and displayed when opening or closing the mouth, and by quantifying and displaying the maximum movement range of the lips, objectively analyzing the patient's lip movement information can be provided. .

1 is a block diagram of an apparatus for analyzing lip motion information according to an embodiment of the present invention.
2 illustrates a screen for visualizing and displaying lip movement information according to an embodiment of the present invention.
3 is a diagram provided to explain the maximum horizontal movement width and the maximum vertical movement width according to the present invention.
4 illustrates another example of a screen for visualizing and displaying lip movement information according to the present invention.
5 is an operation flowchart of an apparatus for analyzing lip motion information according to an embodiment of the present invention.

Then, with reference to the accompanying drawings, embodiments of the present invention will be described in detail so that those skilled in the art can easily practice the present invention.

1 is a block diagram of an apparatus for analyzing lip motion information according to an embodiment of the present invention.

Referring to FIG. 1 , the apparatus for analyzing lip motion information according to the present invention includes an image acquisition unit 110, an image processing unit 120, a visualization unit 130, a display unit 140, a storage unit 150, and a control unit 160. can include

The image acquisition unit 110 may obtain a video (hereinafter referred to as 'face photographing image') of a face of a person taking a motion of opening or closing the mouth. The image acquisition unit 110 may be implemented with a charge coupled device (CCD) camera or a complementary metal-oxide semiconductor (CMOS) camera.

The image processing unit 120 may perform face detection, landmark recognition, face tilt alignment, lip movement tracking, and the like in a face photographed image.

The image processing unit 120 may detect a human face region from a photographed face image. Various methods of detecting the human face area in a face-photographed image, such as extracting feature values for the symmetrical eye shape and the nose and mouth below it, attempting to detect them in the image, or extracting information on standard facial patterns A method of detecting a human face by extracting in advance and then extracting a correlation between an input image and a predefined template is known. Of course, various methods other than those mentioned here may be applied to detect the face region in the image processing unit 120 .

The image processing unit 120 may recognize a plurality of predetermined landmarks in the face area detected in the face photographing image. Landmarks detected in a face photographing image may vary according to embodiments.

The image processing unit 120 may use some of the plurality of recognized landmarks to align the inclination of the face detected in the face photographing image in a predetermined direction. For example, the inclination of a person's face may be aligned such that a line connecting the inner canthus of both eyes is horizontal. Of course, it is also possible to perform face tilt alignment by using the lateral canthus of both eyes for face tilt alignment or by using a landmark recognized in a part other than the eyes.

The image processing unit 120 may track an upper lip landmark corresponding to the upper lip and a lower lip landmark corresponding to the lower lip in the face photographing image. The upper lip landmark and the lower lip landmark may be determined as centers of the upper lip area and the lower lip area. According to embodiments, the upper lip landmark and the lower lip landmark may be determined as the uppermost center of the upper lip and the lowermost center of the lower lip. Of course, it is also possible to set the upper lip landmark and the lower lip landmark to different parts of the upper lip area and the lower lip area.

The visualization unit 130 may visualize lip motion information based on data obtained by tracking upper lip landmarks and lower lip landmarks in the image processing unit 120 and display the visualized lip motion information on the display unit 140 .

2 illustrates a screen for visualizing and displaying lip movement information according to an embodiment of the present invention.

Referring to FIG. 2 , the lip motion information visualization display screen may include a face capture image display area 10 , a graph display area 20 and a maximum movement range display area 30 . Depending on the screen mode, it is also possible to display only the graph display area 20 on the screen.

The face photographing image display area 10 may reproduce a face photographing image in which a mouth is opened and closed in real time or after being recorded. At least some of the landmarks recognized from the face may be displayed together with the face photographing image. 2 shows an example in which a landmark corresponding to the inner canthus of both eyes used for face tilt alignment, an upper lip landmark and a lower lip landmark used for lip movement tracking are displayed as green dots.

In the graph display area 20, a graph representing the movement trajectory of the landmark of the upper lip or the landmark of the lower lip may be displayed. 2 shows an example in which the position of the landmark of the upper lip is the reference point (0, 0) and the trajectory of movement of the point 23 corresponding to the landmark of the lower lip is displayed as graphs 21 and 22.

Different colors or shapes of the graphs 21 and 22 may be expressed for a section in which the upper lip landmark and the lower lip landmark move in a direction away from each other and a section in which the landmark moves in a direction closer to each other.

When the mouth is opened in a closed state, the point 23 moves downward from the reference point (0.0) and moves away from it. The portion of the graph 21 indicated in red in FIG. 2 shows the movement trajectory of the landmark of the lower lip when the mouth is opened.

On the other hand, if the mouth is opened and then closed, the point (23) moves upward and returns while approaching the reference point (0,0). The portion of the graph 22 indicated in blue in FIG. 2 shows the movement trajectory of the landmark of the lower lip when the mouth is closed.

Depending on the embodiment, the thickness of the line may be changed instead of the color, or the shape of the line (solid line, dotted line, etc.) may be expressed differently. Accordingly, it is possible to easily distinguish whether a corresponding graph portion corresponds to a closed mouth section or an open mouth section.

The maximum movement width display area 30 may display the maximum horizontal movement width and the maximum vertical movement width of the upper lip landmark or the lower lip landmark. Here, the horizontal maximum movement width represents the maximum distance from the upper lip landmark (or lower lip landmark) in a horizontal axis direction to the lower lip landmark (or upper lip landmark). Further, the vertical maximum movement width represents the maximum distance from the upper lip landmark (or lower lip landmark) in a vertical axis direction to the lower lip landmark (or upper lip landmark).

2 shows that the point 23 corresponding to the landmark of the lower lip is separated from the landmark of the upper lip by a maximum of 0.7 cm in the horizontal axis direction and by a maximum of 2.5 cm in the vertical axis direction. In FIG. 2 , the maximum horizontal movement width and the maximum vertical movement width are shown in 'cm' units, but the display unit may vary depending on the embodiment.

3 is a diagram provided to explain the maximum horizontal movement width and the maximum vertical movement width according to the present invention.

Referring to FIG. 3 , a rectangle 24 including all graphs corresponding to the movement trajectory of the point 23 may be defined. In FIG. 2 , the maximum horizontal movement width corresponds to the horizontal width A of the rectangle 24 , and the maximum vertical movement width corresponds to the vertical width B of the rectangle 24 in FIG. 2 .

4 illustrates another example of a screen for visualizing and displaying lip movement information according to the present invention.

If there is no disease in the temporomandibular joint, you can open and close your mouth wide without twisting it from side to side. As illustrated in FIG. 4 , a movement trajectory moving in a relatively small range in the horizontal axis direction and moving in a relatively large range in the vertical axis direction may be shown. 4 shows an example in which the maximum horizontal movement width and the maximum vertical movement width are 0.3 cm and 4.6 cm, respectively.

On the other hand, if you have some temporomandibular joint diseases, you may have symptoms that make it difficult to twist your mouth from side to side or open it wide. As illustrated in FIG. 3 above, a movement trajectory moving in a relatively large range in the horizontal axis direction and moving in a relatively small range in the vertical axis direction may be shown. 3 shows an example in which the maximum horizontal movement width and the maximum vertical movement width are 0.3 cm and 4.6 cm, respectively.

The display unit 140 may be implemented as a monitor device having display means such as an LCD panel or an LED panel. The display unit 140 may display various information and data related to the operation of the lip motion information analysis device on the screen.

The storage unit 150 may store various types of information and data related to the operation of the lip motion information analysis device. Specifically, the storage unit 150 includes the face photographed image acquired by the image acquisition unit 110, data used for landmark automatic recognition in the face photographed image, data tracking upper lip landmarks and lower lip landmarks, and lip movements. It can store information visualization data, etc.

The controller 160 controls the overall operation of the lip motion information analysis device according to the present invention. The controller 160 displays all of the face capture image display area 10, the graph display area 20, and the maximum movement width display area 30 on the screen according to the user's request, or selectively displays any one of them on the screen. You may.

5 is an operation flowchart of an apparatus for analyzing lip motion information according to an embodiment of the present invention.

Referring to FIGS. 1 and 5 , first, the image capture unit 110 may obtain a face photographing image in which a mouth is opened or closed (S510).

Next, the image processing unit 120 may detect a human face region from the face photographing image (S520).

Also, the image processing unit 120 may recognize a plurality of predetermined landmarks in the face area detected in the face photographing image (S530). In step S530, a landmark for use in face tilt alignment, an upper lip landmark, and a lower lip landmark may be recognized. A landmark, an upper lip landmark, a lower lip landmark, and the like to be used for face tilt alignment may be selected from the user among the landmarks recognized in step S530.

Thereafter, the image processing unit 120 may use some of the plurality of landmarks recognized in step S530 to align the slope of the face detected in the face photographing image in a predetermined direction (S540).

Meanwhile, the image processing unit 120 may track an upper lip landmark corresponding to the upper lip and a lower lip landmark corresponding to the lower lip in the face photographing image (S550). Step S540 may be performed in parallel while performing step S550.

Next, the visualization unit 130 may visualize the lip motion information based on the data of the upper lip landmark and the lower lip landmark tracking by the image processing unit 120 and display it on the display unit 140 (S560).

Finally, the storage unit 150 may store lip movement information visualization analysis data (S570). The lip movement information visualization analysis data may include lip movement information visualized and displayed in step S560, such as a movement trajectory of a lip landmark or a maximum movement width.

The embodiments described above may be implemented as hardware components, software components, and/or a combination of hardware components and software components. For example, the devices, methods and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate (FPGA). array), programmable logic units (PLUs), microprocessors, or any other device capable of executing and responding to instructions. A processing device may run an operating system (OS) and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of software. For convenience of understanding, there are cases in which one processing device is used, but those skilled in the art will understand that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it can include. For example, a processing device may include a plurality of processors or a processor and a controller. Other processing configurations are also possible, such as parallel processors.

Software may include a computer program, code, instructions, or a combination of one or more of the foregoing, which configures a processing device to operate as desired or processes independently or collectively. The device can be commanded. Software and/or data may be any tangible machine, component, physical device, virtual equipment, computer storage medium or device, intended to be interpreted by or provide instructions or data to a processing device. may be permanently or temporarily embodied in Software may be distributed on networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer readable media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. Program commands recorded on the medium may be specially designed and configured for the embodiment or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with limited drawings, those skilled in the art can apply various technical modifications and variations based on the above. For example, the described techniques may be performed in an order different from the method described, and/or components of the described system, structure, device, circuit, etc. may be combined or combined in a different form than the method described, or other components may be used. Or even if it is replaced or substituted by equivalents, appropriate results can be achieved.

Claims (11)

A lip movement tracking step of tracking an upper lip landmark corresponding to the upper lip and a lower lip landmark corresponding to the lower lip in a face photographing image, and
Visualizing lip motion information based on data tracking the upper lip landmark and the lower lip landmark
including,
The visualized lip movement information,
The method of analyzing lip motion information comprising a graph representing a movement trajectory of the lower lip landmark with the upper lip landmark as a reference point or a graph representing the movement trajectory of the upper lip landmark with the lower lip landmark as a reference point.
In claim 1,
recognizing a plurality of predetermined landmarks in the face photographing image; and
Aligning the facial inclination in a predetermined direction using at least some of the plurality of predetermined landmarks
Lip movement information analysis method further comprising a.
delete In claim 1,
The graph is
The lip movement information analysis method of claim 1 , wherein a color or shape of a section in which the upper lip landmark and the lower lip landmark move in a direction away from each other and a section in which the landmark moves in a direction closer to each other are expressed differently.
In claim 1,
The visualized lip movement information,
Lip motion information analysis method comprising a horizontal maximum movement width and a vertical maximum movement width of the upper lip landmark or the lower lip landmark.
A computer-readable recording medium recording a program for executing any one of the methods of claims 1, 2, 4, and 5 on a computer.
An image acquisition unit for acquiring a face photographed image;
An image processing unit recognizing a plurality of predetermined landmarks in the face photographing image and tracking an upper lip landmark corresponding to the upper lip and a lower lip landmark corresponding to the lower lip among the plurality of recognized landmarks; and
A visualization unit that visualizes lip motion information based on data tracking the upper lip landmark and the lower lip landmark
including,
The visualized lip movement information,
The lip motion information analysis device comprising a graph representing a movement trajectory of the lower lip landmark with the upper lip landmark as a reference point or a graph representing the movement trajectory of the upper lip landmark with the lower lip landmark as a reference point.
In paragraph 7,
The image processing unit,
A lip motion information analysis device for aligning facial inclinations using at least some of the plurality of predetermined landmarks.
delete In paragraph 7,
The graph is
The lip motion information analysis device in which the color or shape of a section in which the upper lip landmark and the lower lip landmark move in a direction away from each other and a section in which the landmark moves in a direction closer to each other are expressed differently.
In paragraph 7,
The visualized lip movement information,
Lip motion information analysis device comprising a horizontal maximum movement width and a vertical maximum movement width of the upper lip landmark or the lower lip landmark.

Images