JP7037159B2 - Systems, programs, and methods for measuring a subject's jaw movements - Google Patents

Description

The present invention relates to a system, program, and method for measuring jaw movements of a subject. In addition, it relates to systems, programs, and methods for constructing motion point trajectory trained models used to measure a subject's jaw motion.

One of the diagnostic actions in the dental field is the evaluation of jaw movement. Jaw movements include "marginal movement measurement" to check the range of movement of the jawbone when not chewing (during non-function) and "mastication jaw" to measure mandibular movement (mastication movement) during mastication (during functioning). There is "exercise measurement". In the field of dentistry, it is applied to patients with temporomandibular joint disease and patients who need prosthesis for missing teeth.

Patent Document 1 discloses a method for measuring jaw movement. This technique requires the patient to have a headframe and mandibular marker fixed. In the conventional method for measuring jaw movement as shown in Patent Document 1, the patient had to go directly to the hospital in order to attach the instrument necessary for the measurement to the patient. This can be a heavy burden and effort for patients, doctors and the like. Moreover, all of these appliances are dedicated and can be very expensive and costly.

Japanese Unexamined Patent Publication No. 2002-336282

An object of the present invention is to provide a system or the like capable of easily measuring the jaw movement of a subject.

In one embodiment of the invention, the system for measuring jaw movement is an acquisition means for acquiring a plurality of consecutive images of the subject's face during jaw movement and a correction that at least corrects the coordinate system of the subject's face. The means includes an extraction means for extracting at least a motion point in the lower jaw region of the face, and a generation means for generating at least motion point trajectory information indicating the trajectory of the motion point by tracking the motion point.

In one embodiment, the extraction means has a first extraction means for extracting motion points in the lower jaw region of the face and a second extraction means for extracting fixed points in the upper facial region of the face from the plurality of images. The correction means includes extraction means, and the correction means corrects the coordinate system based on the fixed point and a predefined face reference position template.

In one embodiment, the first extraction means extracts a plurality of feature portions in the plurality of images, and among the plurality of feature portions, a feature portion whose coordinate change within a predetermined period is within a predetermined range is moved. The second extraction means extracts a feature portion whose coordinate change within a predetermined period is less than a predetermined threshold value as a fixed point among the plurality of feature portions.

In one embodiment, the correction means has a first correction means for correcting the coordinate system of the subject's face, and a second correction means for generating fixed point locus information indicating the locus of the fixed point by tracking the fixed point. 2. The generation means and the second correction means for correcting the motion point locus information based on the fixed point locus information are provided.

In one embodiment, the correction means is a reference coordinate system trained model that has been processed to learn the reference coordinate system of the faces of a plurality of subjects, and the reference coordinate system trained model is an input image. It comprises a reference coordinate system trained model configured to correct the coordinate system of the subject's face inside to the reference coordinate system.

In one embodiment, the reference coordinate system trained model takes a difference between the coordinate system of the subject's face in the input image and the reference coordinate system, and the input is based on the difference. The coordinate system is corrected by converting the image.

In one embodiment, the conversion process comprises an affine transformation.

In one embodiment, the extraction means extracts a plurality of feature portions in the plurality of images, and extracts pixels among the plurality of feature portions whose coordinate changes within a predetermined period are within a predetermined range as motion points. ..

In one embodiment, the correction means reflects the subject's face in the plurality of images in the base face model, and the base face model generation means for generating the base face model of the subject's face. The correction means includes a jaw movement face model generation means for generating a jaw movement face model of a subject, and the correction means corrects the coordinate system of the jaw movement face model based on the coordinate system of the base face model. Correct the coordinate system.

In one embodiment, the extraction means extracts motion points in the jaw motion face model or the base face model.

In one embodiment, the system of the present invention further comprises an evaluation means that generates jaw movement evaluation information indicating the evaluation of the jaw movement of the subject based on at least the generated movement point trajectory information.

In one embodiment, the system of the present invention further comprises a reference point configured to be placed in the mandibular region of the subject, and the extraction means uses the reference point in the plurality of images as a motion point. Extract.

In one embodiment, the second extraction means further comprises a reference point configured to be installed in the upper facial region of the subject, and the second extraction means extracts the reference point in the plurality of images as a fixed point. ..

In one embodiment, the reference point is configured to represent a specific point on the reference point.

In one embodiment of the present invention, a program for measuring jaw movement is executed in a system including a processor unit, and the program acquires a plurality of consecutive images of a subject's face during jaw movement. Motion point trajectory information indicating the trajectory of the motion point by at least correcting the coordinate system of the subject's face, extracting at least the motion points in the lower jaw region of the face, and tracking the motion points. The processor unit is made to perform a process including at least generating.

In one embodiment of the invention, the method for measuring jaw movement is to acquire a plurality of consecutive images of the subject's face during jaw movement and to at least correct the coordinate system of the subject's face. , At least extracting the motion points in the lower jaw region of the face, and by tracking the motion points, at least generating motion point trajectory information indicating the trajectory of the motion points.

In one embodiment of the present invention, the system for measuring jaw movement is an acquisition means for acquiring a plurality of consecutive images of a subject's face during jaw movement, and a lower jaw of the face based on the plurality of images. Based on at least the extraction means for extracting at least the motion points in the region, the generation means for generating at least the motion point trajectory information indicating the trajectory of the motion points by tracking the motion points, and the motion point trajectory information. , An evaluation means for generating jaw movement evaluation information indicating the evaluation of the jaw movement of the subject, the evaluation means is a movement point trajectory learned model subjected to processing for learning the movement point trajectory information of a plurality of subjects. The model having learned the locus of motion points includes an evaluation means configured to correlate the input locus of motion points with the evaluation of jaw movement.

In one embodiment of the present invention, a program for measuring jaw movement is executed in a system including a processor unit, and the program acquires a plurality of consecutive images of a subject's face during jaw movement. Based on the plurality of images, at least the motion points in the lower jaw region of the face are extracted, and by tracking the motion points, at least the motion point trajectory information indicating the trajectory of the motion points is generated. , Jaw movement evaluation showing the evaluation of the jaw movement of the subject based on at least the movement point trajectory information by using the movement point trajectory learned model that has been processed to learn the movement point trajectory information of a plurality of subjects. The processor unit is subjected to processing including the generation of information, wherein the motion point trajectory trained model is configured to correlate the input motion point trajectory information with the evaluation of jaw movement. Let me do it.

In one embodiment of the invention, the method for measuring jaw movement is to acquire a plurality of consecutive images of the subject's face during jaw movement, and based on the plurality of images, the mandibular region of the face. Processing to learn at least the motion point trajectory information of a plurality of subjects by extracting at least the motion points in the motion point, generating at least the motion point trajectory information indicating the trajectory of the motion point by tracking the motion point, and learning the motion point trajectory information of a plurality of subjects. This is to generate jaw movement evaluation information indicating the evaluation of the jaw movement of the subject based on at least the movement point trajectory information by using the movement point trajectory learned model. The trained model includes that the input motion point trajectory information is configured to correlate with the evaluation of jaw motion.

In one embodiment of the present invention, the system for constructing a motion point trajectory trained model used for measuring the jaw motion of a subject is a motion point obtained by tracking the motion points of a plurality of subjects. An acquisition means for acquiring at least the motion point trajectory information indicating the trajectory of the motion point, and a construction means for constructing a motion point trajectory trained model by learning processing using the motion point trajectory information of at least the plurality of subjects as input teacher data. To prepare for.

In one embodiment, the learning process is supervised learning, the acquisition means acquires evaluations of jaw movements of the plurality of subjects, and the acquired evaluations of jaw movements are used as output teacher data. To.

In one embodiment, the learning process is unsupervised learning, and the system of the present invention further comprises a classification means for classifying the output of the constructed motion point trajectory trained model.

In one embodiment of the present invention, a program for constructing a motion point trajectory trained model used for measuring a subject's jaw motion is executed in a system including a processor unit, and the program is a plurality of subjects. By acquiring at least the motion point trajectory information indicating the trajectory of the motion point obtained by tracking the motion points of the above, and by learning processing using the motion point trajectory information of at least the plurality of subjects as input teacher data. The processor unit is made to perform a process including constructing a model in which the motion point trajectory has been learned.

In one embodiment, a method for constructing a motion point trajectory trained model used to measure a subject's jaw motion is to track the trajectory of the motion points obtained by tracking the motion points of a plurality of subjects. It includes at least acquiring the motion point trajectory information to be shown, and constructing a motion point trajectory trained model by a learning process using the motion point trajectory information of at least the plurality of subjects as input teacher data.

According to the present invention, it is possible to provide a system or the like capable of easily measuring the jaw movement of a subject. By using the system of the present invention, the user can measure jaw movements at any place, for example, at a company, at home, or the like, in addition to a medical institution.

The figure which shows an example of the flow 10 for simply measuring the jaw movement of a patient using one Embodiment of this invention. The figure which shows an example of the structure of the computer system 100 for measuring the jaw movement of a subject. The figure which shows an example of the structure of the processor part 120 in one Embodiment The figure which shows an example of the result of the extraction of a plurality of feature portions of a face by the extraction means 122. A diagram comparing the results of extracting facial features using Openface for two frames in a video when the angle between the face and the camera is shifted. The figure which shows an example of the structure of the processor part 130 in one Embodiment The figure which shows an example of the structure of the processor part 140 in another embodiment. The figure which shows an example of the structure of the processor part 150 in another embodiment. The figure which shows an example of the structure of the processor part 160 in another embodiment. The figure which shows an example of the structure of the neural network model 1610 which can be used by the evaluation means 161. A flowchart showing an example (process 800) of processing by the computer system 100 for measuring the jaw movement of the subject. A flowchart showing another example (process 900) of processing by the computer system 100 for measuring the jaw movement of the subject. A flowchart showing another example (process 1000) of processing by the computer system 100 for measuring the jaw movement of the subject. Diagram showing an example of a marker point Diagram showing an example of a marker point

(Definition)
As used herein, the term "mandible region" refers to the region on the mandible of the face.

As used herein, the term "upper facial region" refers to an region other than the mandibular region of the face. That is, the facial region is divided into a "mandibular region" and a "upper facial region".

As used herein, the term "face coordinate system" refers to a coordinate system defined for a face. The "face coordinate system" includes, for example, a horizontal system (x-axis), a sagittal system (y-axis), and a coronal system (z-axis). The horizontal system in the "face coordinate system" is defined along, for example, the eye-ear plane (Frankfurt plane), the nasal hearing line (or Campel plane), the hip plane, the occlusal plane, or both pupil lines. The sagittal system of the "face coordinate system" is defined, for example, along the median line, the mid-sagittal plane, and the like. The coronal system of the "face coordinate system" is defined, for example, along the orbital plane. The planes or lines that define the horizontal, sagittal, and coronal systems are not limited to those described above and can be defined along any plane or line.

As used herein, the term "face reference coordinate system" refers to the face coordinate system normally possessed by a person facing the front. The "face reference coordinate system" is derived by learning multiple facial images.

As used herein, the term "exercise point" refers to a point or region on a site that is exercised by jaw movement.

As used herein, the term "fixed point" refers to a point or region on a site that does not move due to jaw movement.

In the present specification, the "base face model" refers to a three-dimensional model of a face, and is a so-called "3D avatar". When the movement of the characteristic part extracted from the video of the actual jaw movement is reflected on the "base face model", the "jaw movement face model" that reflects the actual jaw movement is obtained. The "jaw movement face model" is a so-called "moving 3D avatar" according to the jaw movement in the moving image.

As used herein, "about" means ± 10% of the value that follows.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

1. 1. Flow for simply measuring a patient's jaw movement FIG. 1 shows an example of a flow 10 for simply measuring a patient's jaw movement using an embodiment of the present invention. In the flow 10, the dentist simply captures a video of the patient 20 chewing, and the jaw movement of the patient 20 is measured (that is, the jaw movement during chewing is measured), and the measurement result of the jaw movement is performed. Is provided to the dentist.

First, a dentist uses a terminal device 300 (for example, a smartphone, a tablet, etc.) to take a moving image of the patient 20 chewing. Since a moving image can be regarded as a plurality of continuous images (still images), the "moving image" and the "consecutive plurality of images" are used interchangeably in the present specification.

In step S1, the captured moving image is provided to the server device 30. It does not matter how the moving image is provided to the server device 30. For example, the moving image may be provided to the server device 30 via a network (eg, the Internet, LAN, etc.). For example, the moving image may be provided to the server device 30 via a storage medium (eg, removable media).

Next, in the server device 30, processing is performed on the moving image provided in step S1. The processing by the server device 30 generates the measurement result of the jaw movement of the patient 20. The measurement result of the jaw movement may be, for example, information representing the trajectory of the jaw movement. Alternatively, the measurement result of the jaw movement may be, for example, jaw movement evaluation information for evaluating the jaw movement of the patient 20 generated based on the information representing the trajectory of the jaw movement. The jaw movement evaluation information includes, for example, information on whether or not the trajectory of the jaw movement is a normal pattern.

In step S2, the measurement result of the jaw movement generated by the server device 30 is provided to the dentist. It does not matter which mode the measurement result of jaw movement is provided. For example, jaw movement measurements may be provided via a network or via a storage medium. Alternatively, for example, the measurement results of jaw movement may be provided via a paper medium.

The dentist confirms the measurement result of the jaw movement of the patient 20 provided in step S2.

As described above, the dentist can easily measure the jaw movement of the patient 20 simply by photographing the state of chewing of the patient 20 with the terminal device 300 without requiring a specific instrument.

In the above-mentioned example, it has been described that the jaw movement during mastication is measured in the flow 10, but the present invention is not limited to this. For example, in the flow 10, the limit motion measurement can be performed. For example, the dentist uses the terminal device 300 to take a video of the patient 20 moving his jaw greatly (for example, his mouth wide open, his jaw extended forward, etc.). It can be achieved by processing the captured moving image in the server device 30.

In the above-mentioned example, the dentist has described the flow 10 in which the measurement result of the jaw movement can be provided only by taking a moving image of the jaw movement of the patient 20, but the present invention is not limited thereto. For example, the patient 20 can use his / her own terminal device 300 to take a video of himself / herself chewing, or the patient 20 can use the terminal device 300 to another person and himself / herself. It is also possible to have the patient 20's jaw movement measurement results provided to the doctor, dentist, caregiver, or patient 20's family or 20 patients by simply having them take a video of the mastication. Is.

It is also possible for the terminal device 300 to perform the processing by the server device 30 described above. In this case, the server device 30 may be omitted and the terminal device 300 can operate standalone. In the above-mentioned example, in which the measurement result of the jaw movement of the patient 20 is provided to a doctor, a dentist, a caregiver, or a family member, the measurement result of the jaw movement is obtained from the terminal device 300 without going through the server device 30 by the doctor in the hospital. Alternatively, it may be provided to a dentist, a caregiver, or 20 family members or 20 patients themselves.

The above-mentioned flow 10 can be realized by using the computer system 100 of the present invention described later.

2. 2. Configuration of a Computer System for Measuring the Jaw Movement of a Subject FIG. 2 shows an example of the configuration of a computer system 100 for measuring the jaw movement of a subject.

The computer system 100 is connected to the database unit 200. Further, the computer system 100 may be connected to at least one terminal device 300 via the network 400.

The network 400 can be any kind of network. The network 400 may be, for example, the Internet or a LAN. The network 400 may be a wired network or a wireless network.

An example of the computer system 100 is, but is not limited to, a server device. An example of the terminal device 300 is, but is not limited to, a computer held by a user (for example, a terminal device) or a computer installed in a hospital (for example, a terminal device). Here, the computer (server device or terminal device) can be any type of computer. For example, the terminal device can be any type of terminal device such as a smartphone, tablet, personal computer, smart glasses and the like.

The computer system 100 includes an interface unit 110, a processor unit 120, and a memory unit 170.

The interface unit 110 exchanges information with the outside of the computer system 100. The processor unit 120 of the computer system 100 can receive information from the outside of the computer system 100 via the interface unit 110, and can transmit the information to the outside of the computer system 100. The interface unit 110 can exchange information in any format.

The interface unit 110 includes, for example, an input unit that enables information to be input to the computer system 100. It does not matter in what manner the input unit enables the information to be input to the computer system 100. For example, when the input unit is a touch panel, the user may input information by touching the touch panel. Alternatively, when the input unit is a mouse, the user may input information by operating the mouse. Alternatively, when the input unit is a keyboard, the user may input information by pressing a key on the keyboard. Alternatively, when the input unit is a microphone, the user may input information by inputting voice into the microphone. Alternatively, when the input unit is a camera, the information captured by the camera may be input. Alternatively, when the input unit is a data reading device, the information may be input by reading the information from the storage medium connected to the computer system 100. Alternatively, when the input unit is a receiver, the receiver may input information by receiving information from the outside of the computer system 100 via the network 400.

The interface unit 110 includes, for example, an output unit that enables information to be output from the computer system 100. It does not matter in what manner the output unit enables the information to be output from the computer system 100. For example, when the output unit is a display screen, information may be output to the display screen. Alternatively, when the output unit is a speaker, the information may be output by the voice from the speaker. Alternatively, when the output unit is a data writing device, the information may be output by writing the information to the storage medium connected to the computer system 100. Alternatively, when the output unit is a transmitter, the transmitter may output information by transmitting information to the outside of the computer system 100 via the network 400. In this case, the type of network does not matter. For example, the transmitter may transmit information via the Internet or may transmit information via LAN.

For example, the output unit can output the motion point trajectory information generated by the computer system 100. For example, the output unit can output the jaw movement evaluation information generated by the computer system 100.

The processor unit 120 executes the processing of the computer system 100 and controls the operation of the entire computer system 100. The processor unit 120 reads a program stored in the memory unit 170 and executes the program. This makes it possible to make the computer system 100 function as a system that performs a desired step. The processor unit 120 may be implemented by a single processor or may be implemented by a plurality of processors.

The memory unit 170 stores a program required for executing the processing of the computer system 100, data required for executing the program, and the like. The memory unit 170 includes a program for causing the processor unit 120 to perform a process for measuring the jaw movement of the subject (for example, a program for realizing the process shown in FIGS. 8 and 9 described later), and the subject's jaw. A process for constructing a motion point trajectory trained model used for measuring motion (for example, a program for realizing the process shown in FIG. 10 described later) may be stored. Here, it does not matter how the program is stored in the memory unit 170. For example, the program may be pre-installed in the memory unit 170. Alternatively, the program may be installed in the memory unit 170 by being downloaded via the network. In this case, the type of network does not matter. The memory unit 170 may be implemented by any storage means.

The database unit 200 may store, for example, a plurality of consecutive images of the faces of the subjects during jaw movement for each of the plurality of subjects. A plurality of continuous images of the subject's face during jaw movement may be, for example, transmitted from the terminal device 300 to the database unit 200 via the network 400, or may be, for example, a photographing means provided in the computer system 100. It may be taken by. A plurality of consecutive images of faces during jaw movement of a plurality of subjects can be used to construct a face reference coordinate system trained model described later.

In the database unit 200, for example, for each of a plurality of subjects, a plurality of consecutive images of the face during jaw movement or movement point trajectory information derived from those images is associated with the evaluation of the jaw movement of the subject. Can be stored. The stored data can be used to build a motion point trajectory trained model, which will be described later.

Further, the database unit 200 may store, for example, motion point trajectory information output by the computer system 100 or jaw motion evaluation information.

The terminal device 300 includes at least a shooting means such as a camera. The photographing means may have any configuration as long as it is possible to capture at least a plurality of consecutive images. The imaging means is used to capture a plurality of consecutive images of the subject's face during jaw movement. The image to be captured may be an image including two-dimensional information (vertical x horizontal) or an image including three-dimensional information (vertical x horizontal x depth).

FIG. 3 shows an example of the configuration of the processor unit 120 in one embodiment.

The processor unit 120 includes an acquisition unit 121, an extraction unit 122, and a generation unit 123.

The acquisition means 121 is configured to acquire a plurality of consecutive images of the face of the subject during jaw movement. The acquisition means 121 can acquire, for example, a plurality of consecutive images of the subject's face during jaw movement stored in the database unit 200 via the interface unit 110. Alternatively, the acquisition means 121 can acquire a plurality of continuous images received from the terminal device 300 via the interface unit 110, for example.

The acquisition means 121 is configured to acquire motion point trajectory information indicating the trajectory of the motion point obtained by tracking the motion points of a plurality of subjects in order to construct a model in which the motion point trajectory has been learned, which will be described later. You may. The motion point trajectory information may be, for example, motion point trajectory information acquired by using the computer system 100 of the present invention, or motion point trajectory information obtained from any known jaw motion measuring device. May be good. The acquisition means 121 can acquire, for example, the motion point trajectory information stored in the database unit 200 via the interface unit 110. The acquisition means 121 may be further configured to acquire evaluations of jaw movements of a plurality of subjects in order to construct a motion point trajectory trained model described later. The acquisition means 121 can acquire, for example, the evaluation of the jaw movement stored in the database unit 200 via the interface unit 110.

The extraction means 122 is configured to extract at least the motion points in the lower jaw region of the face. The extraction means 122 can, for example, extract a motion point from an image acquired by the acquisition means 121. The extraction means 122 can, for example, extract a motion point from the output of the correction means 131 described later. For example, the extraction means 122 may receive an input of where the motion point in the image is, and extract the motion point based on the input, or may automatically extract the motion point without receiving the input. May be extracted. The extraction means 122 can automatically extract, for example, a point or region on the midline of the face as a motion point. The points on the midline in the mandibular region of the face are suitable for assessing jaw movement because they move significantly with jaw movement. For example, the extraction means 122 may automatically extract a point at the tip of the lower jaw as a movement point.

In one embodiment, when the extraction means 122 automatically extracts the motion points, the extraction means 122 first detects a plurality of feature portions of the face for each of the plurality of images. One of the plurality of feature portions of the face can be represented, for example, as one or more pixels in the image. The plurality of characteristic parts of the face may be, for example, parts corresponding to eyes, eyebrows, nose, mouth, eyebrows, chin, ears, throat, contour and the like. The extraction of a plurality of feature portions of a face by the extraction means 122 can be performed, for example, by using a trained model that has been subjected to a process of learning the feature portions using facial images of a plurality of subjects. In one example, the extraction of a plurality of facial features by the extraction means 122 can be performed using the face recognition application "Openface" (Tadas Baltrusaitis, Peter Robinson, Louis-Philippe Morency, "OpenFace: an open source facial behavior". analysis toolkit ”, 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), pp.1-10, 2016). "Openface" is software that can output three-dimensional information (vertical x horizontal x depth) of a face from an image including two-dimensional information (vertical x horizontal) of the face.

For example, the extraction means 122 may perform color enhancement processing on an image so as to extract a portion having a color tone different from that of the surroundings as a feature portion. This is particularly preferable when extracting a portion having a color tone that is clearly different from the surroundings such as eyebrows.

FIG. 4 shows an example of the result of extraction of a plurality of feature portions of a face by the extraction means 122. The extraction means 122 uses the face recognition application "Openface" to extract a plurality of feature portions of a face. Each of the extracted feature portions is displayed with a black dot 4000. In "Open face", the contour is extracted as a feature portion, and it is not possible to extract a portion without a conspicuous feature inside the contour (for example, between the cheek, the mouth and the chin, etc.).

The extraction means 122 extracts the point 4100 in the mandibular region as a motion point from the extracted feature portions. The extraction means 122 can, for example, extract a portion of the extracted plurality of feature portions whose coordinate change within a predetermined period is within a predetermined range as a motion point. Here, the predetermined period may be, for example, all or a part of the period in which a plurality of consecutive images are taken. The predetermined range may be, for example, a range of about 5 mm to about 20 mm. This makes it possible to avoid erroneously extracting a feature portion whose coordinates change other than the movement due to jaw movement as a movement point. For example, it is possible to avoid erroneously extracting a large movement of the subject's body when photographing the subject.

For example, as shown in FIG. 4, the motion point 4100 at the lower end of the mandible in the mandibular region can be extracted.

With reference to FIG. 3 again, the generating means 123 is configured to generate at least the locus information of the locus indicating the locus of the locus by tracking the locus. The motion point locus information is information indicating the trajectory of the motion point within a predetermined period. Here, the predetermined period may be, for example, all or a part of the period in which a plurality of consecutive images are taken. The generation means 123 can generate motion point locus information by, for example, specifying the coordinates in an image of a motion point in each of a plurality of consecutive images and tracing the coordinates between the plurality of consecutive images. .. At this time, the coordinates of the motion point may be two-dimensional coordinates or three-dimensional coordinates. When the image is an image containing two-dimensional information (vertical x horizontal), the coordinates of the motion points are two-dimensional coordinates, and when the image is an image containing three-dimensional information (vertical x horizontal x depth), the coordinates are two-dimensional coordinates. The coordinates of the motion point are three-dimensional coordinates. Alternatively, even if the image contains 2D information (vertical x horizontal), when using an algorithm that outputs 3D information (vertical x horizontal x depth) from the 2D image, the coordinates of the motion point are It can be 3D coordinates.

The generated motion point trajectory information can be output to the outside of the computer system 100 via the interface unit 110.

In the extraction of the plurality of feature portions of the face by the extraction means 122 described above, it may be difficult to extract the same points on the face in each of the plurality of consecutive images. For example, in the above-mentioned Openface, since the feature portion is independently extracted from each of a plurality of consecutive images, the feature portion extracted by the first image among the continuous plurality of images and the continuous plurality of images The feature portion extracted in the second image is not always the same. Further, in Openface, eyes, eyebrows, nose, mouth, eyebrows, chin, ears, throat, contour, etc. are extracted as feature parts. Therefore, for example, when a point on the contour is a feature part corresponding to the lower jaw region. When the subject's body moves or tilts while taking a plurality of consecutive images, the contour of the subject's face in the image changes, and the extracted feature portion also changes. As a result, the plurality of featured portions of the extracted face are displaced between the plurality of consecutive images.

FIG. 5 is a diagram comparing the results of extracting facial feature portions using Openface for two frames in a moving image when the angle between the face and the camera is shifted. In FIG. 5, the white line is an extension of the nasal muscle and is a characteristic portion from which black dots are extracted. Since the distance between the feature portion on the mandible indicated by the arrow and the white line is different in the two frames, it is confirmed that the extracted feature portion is not the same in the two frames.

As described above, when the feature portions extracted from the plurality of consecutive images are different, the motion points extracted based on the feature portions are also different between the plurality of consecutive images. This makes it impossible to generate accurate motion point trajectory information. Therefore, it is preferable to correct the movement and inclination of the subject during imaging so that the extracted motion points do not differ between a plurality of consecutive images.

For example, by setting a gauge point in the mandibular region of the subject, the extraction means 122 may extract the motion point by recognizing the gauge point installed in the mandibular region. The reference point may be, for example, a sticker or an ink. By extracting the reference point as a motion point by the extraction means 122, the extracted motion points can be the same among a plurality of consecutive images.

In one embodiment, the gauge point may be configured to represent a particular point on the gauge point. If the gauge points have a certain size, the gauge points recognized by the extraction means 122 from each of the plurality of consecutive images may be slightly shifted between the plurality of consecutive images. This is because the extraction means 122 recognizes an arbitrary point in the reference point. In this case, the extracted motion points are slightly different between a plurality of consecutive images, which may lead to an error in the motion point trajectory information. By making the extraction means 122 recognize a specific point on the reference point, it is possible to eliminate the deviation of the recognized reference point, and make the extracted motion points the same among a plurality of consecutive images. be able to. The specific point is preferably of a size recognized as a point by the extraction means 122.

For example, the gauge point can have a pattern representing a specific point. The pattern can be, for example, a dot pattern (eg, an AR marker (eg, ArUco marker) as shown in FIG. 11A, a QR code®, etc.). The dot pattern represents the corner of each dot or the center of the pattern as a specific point. The pattern can be, for example, a center of gravity symbol in mechanical drawing (for example, a symbol in which a circle is divided into four equal parts and painted separately, as shown in FIG. 11B). The center of gravity symbol in the mechanical drawing represents the center as a specific point. For example, the reference point is color-coded (for example, the color of the specific point is the complementary color of the color of other parts of the reference point, or the color of the specific point is the complementary color of the skin color) to represent the specific point. You may do so. For example, the gauge point may represent a specific point by making its size sufficiently small.

For example, when the pattern possessed by the reference point is a dot pattern, the three-dimensional coordinates of the reference point can be derived by recognizing at least four points on the dot pattern. Derivation of three-dimensional coordinates can be achieved by using techniques known in the field of AR (augmented reality) and the like. The three-dimensional coordinates of the reference point correspond to the three-dimensional coordinates of the motion point.

The extraction means 122 may extract the feature portion by using an application other than "Open face". For example, an application that can extract a consistent feature portion between a plurality of images is preferable. This is because the same effect as the case of using the above-mentioned reference point, that is, the effect that the extracted motion points can be the same among a plurality of consecutive images can be obtained. Further, the extraction means 122 can, for example, extract a feature portion from an image including three-dimensional information (length x width x depth) of the face. The extraction means 122 also extracts a feature portion from an image including three-dimensional information (length x width x depth) of the face, and the extracted motion points can be the same among a plurality of consecutive images.

In this way, the extraction means 122 extracts the feature portion using an application capable of extracting the reference point as a motion point, or extracting a consistent feature portion among a plurality of images, or the face 3 By extracting a feature portion from an image containing dimensional information, the extracted motion points can be the same among a plurality of consecutive images. However, even in these cases, if the subject's movement or inclination during imaging is present, the subject's movement or inclination is included in the locus of the motion point, and accurate motion point trajectory information can be generated. become unable. Therefore, it is preferable to correct the movement and inclination of the subject during imaging.

FIG. 6A shows an example of the configuration of the processor unit 130 in one embodiment. The processor unit 130 may have a configuration for making corrections so that the extracted motion points do not differ between a plurality of consecutive images. The processor unit 130 is a processor unit included in the computer system 100 as an alternative to the processor unit 120 described above. In FIG. 6A, the same reference numbers as the elements shown in FIG. 3 are assigned the same reference numbers, and the description thereof will be omitted here.

The processor unit 130 includes an acquisition unit 121, a correction unit 131, an extraction unit 122, and a generation unit 123.

The correction means 131 is configured to at least correct the coordinate system of the subject's face based on the plurality of images acquired by the acquisition means 121.

In one embodiment, the correction means 131 can correct the coordinate system of the subject's face for each of the plurality of images so that the coordinate system of the subject's face matches for each of the plurality of images. The correction means 131 can correct the coordinate system of the face, for example, by performing an affine transformation on each of the plurality of images. When a plurality of images are images including three-dimensional information (length x width x depth), three-dimensional affine transformation can be performed. As a result, the motion points extracted from each of the plurality of images are matched between the plurality of images.

In another embodiment, the correction means 131 utilizes a reference coordinate system trained model that has been processed to learn the reference coordinate system of the faces of a plurality of subjects, for each of the plurality of images, of the subject's face. The coordinate system can be corrected. The reference coordinate system trained model is configured to output an image obtained by correcting the coordinate system of the subject's face in the input image to the reference coordinate system. The frame of reference trained model can be constructed using any machine learning model.

The frame of reference trained model can be constructed, for example, by supervised learning. In supervised learning, for example, an image of the subject's face can be used as input teacher data, and the reference coordinate system in the image can be used as output teacher data. By iteratively learning multiple images of multiple subjects (eg, for at least 50 people), the trained model can recognize a frame of reference that is statistically presumed to have the faces of multiple subjects. become. When an image of the subject's face is input to such a trained model, the difference between the subject's face coordinate system and the reference coordinate system is output. For example, the input image is converted (for example, scaling, rotation, shearing, translation, etc.) so that the difference between the coordinate system of the subject's face and the reference coordinate system becomes zero or less than a predetermined threshold. By constructing a trained model in, a frame of reference trained model can be generated. The image conversion process can be performed using, for example, an affine transformation.

The frame of reference trained model can be constructed, for example, by unsupervised learning. In unsupervised learning, for example, facial images of a plurality of subjects can be used as input teacher data. By repeatedly learning a large number of images of multiple subjects, the trained model can recognize the facial coordinate system common to many images as a reference coordinate system that is estimated to be statistically possessed by multiple subjects. become able to. When an image of the subject's face is input to such a trained model, the difference between the subject's face coordinate system and the reference coordinate system is output. For example, the input image is converted (for example, scaling, rotation, shearing, translation, etc.) so that the difference between the coordinate system of the subject's face and the reference coordinate system becomes zero or less than a predetermined threshold. By constructing a trained model in, a frame of reference trained model can be generated. The image conversion process can be performed using, for example, an affine transformation.

The image used for the input teacher data may be, for example, an image including two-dimensional information (vertical x horizontal), but is preferably an image including three-dimensional information (vertical x horizontal x depth). This is because the constructed reference coordinate system trained model can output an image including three-dimensional information. An image containing three-dimensional information (length x width x depth) can be acquired using, for example, an RGB-D camera. When using an image containing two-dimensional information, it is preferable to perform a process of estimating the depth information, add the depth information, and then use the image for the learning process.

The extraction means 122 extracts the motion points in the lower jaw region of the face in the coordinate system corrected by the correction means 131. The generation means 123 generates motion point locus information indicating the locus of the motion point by tracking the extracted motion points. Since the motion points extracted from each of the plurality of images are acquired in the same coordinate system, the generated motion point trajectory information becomes more accurate information.

FIG. 6B shows an example of the configuration of the processor unit 140 in another embodiment. The processor unit 140 may have a configuration for correcting the coordinate system of the subject's face by using a fixed point that does not move in jaw movement as a reference. The processor unit 140 is a processor unit included in the computer system 100 as an alternative to the processor unit 120 described above. In FIG. 6B, the same reference numbers are assigned to the same components as those described above in FIGS. 3 and 6A, and the description thereof will be omitted here.

The processor unit 140 includes an acquisition unit 121, a correction unit 131, an extraction unit 122, and a generation unit 123. The extraction means 122 includes a first extraction means 141 and a second extraction means 142.

The second extraction means 142 is configured to extract fixed points in the upper facial region of the face based on a plurality of images. The second extraction means 142 may, for example, receive an input of where the fixed point in the image is and extract the fixed point based on the input, or may automatically extract the fixed point without receiving the input. A fixed point may be extracted. For example, the fixed point can be a point or region on a site such as the forehead, glabellar, glabellar, nasal tip, etc. that does not move with jaw movement. For example, the fixed point can be a point or region on an anatomical feature (eg, eyebrows, glabellar, external eye angle (outer corner of the eye), internal eye angle, pupil, tragus, nose tip, etc.). The fixed point is preferably having at least 3 pixels on the image, and the fixed point may be, for example, one area having at least 3 pixels, or 3 points each having at least 1 pixel. May be good. This is because if there are at least 3 pixels, the correction means 131, which will be described later, can use these three as a reference and correct a plurality of images so as to be associated with the face reference position template. For example, the fixed points can be at least three points that are spaced apart from each other, in which case the coordinate system of the subject's face is based on the fixed points and a predefined face reference position template by the correction means 133 described below. It is possible to reduce the error that may occur when correcting the above. For example, the fixed point can be an area that covers at least a portion of the upper facial area of the face (eg, a curved surface that covers the forehead), in which case the fixed point and a pre-defined face reference by the correction means 133 described below. It is possible to reduce the error that may occur when correcting the coordinate system of the subject's face based on the position template.

In one embodiment, when the second extraction means 142 automatically extracts a fixed point, the second extraction means 142 first, like the extraction means 122 described above with reference to FIG. 3, first has a plurality of images. For each of the above, multiple feature parts of the face are detected. The extraction of a plurality of feature portions of a face by the second extraction means 142 can be performed, for example, by using a trained model that has been processed to learn the feature portions using facial images of a plurality of subjects. , Can be done using the face recognition application "Openface". The second extraction means 142 extracts a point or region in the upper facial region as a fixed point among the plurality of extracted feature portions. The second extraction means 142 can, for example, extract a portion of the extracted plurality of feature portions whose coordinate change within a predetermined period is less than a predetermined threshold value as a fixed point. Here, the predetermined period may be, for example, all or a part of the period in which a plurality of consecutive images are taken. The predetermined threshold can be, for example, about 1 mm. The second extraction means 142 may, for example, perform color enhancement processing on the image to extract a portion having a color tone different from that of the surroundings as a feature portion. This is particularly preferable when extracting a portion having a color tone that is clearly different from the surroundings such as eyebrows.

In one embodiment, by setting a reference point in the upper face region of the subject, the second extraction means 142 extracts the fixed point by recognizing the reference point installed in the upper face region. May be good. In the present specification and claims, a fixed point is also referred to as a fixed point to distinguish it from a point placed in the mandibular region. The fixed point station can have the same configuration as the point point installed in the mandibular region. The fixed point reference point may be, for example, a seal or ink. The fixed point point may be used separately from the point set in the mandibular region, but the fixed point point is preferably used in combination with the point set in the mandibular region.

In one embodiment, the fixed point gauge may be configured to represent a particular point on the fixed point gauge. For example, a fixed point fixed point can have a pattern representing a specific point. The pattern can be, for example, a dot pattern (eg, an AR marker (eg, ArUco marker) as shown in FIG. 11A, a QR code®, etc.). The dot pattern represents the corner of each dot or the center of the pattern as a specific point. The pattern can be, for example, a center of gravity symbol in mechanical drawing (for example, a symbol in which a circle is divided into four equal parts and painted separately, as shown in FIG. 11B). The center of gravity symbol in the mechanical drawing represents the center as a specific point. For example, a fixed point can be identified by color-coding (for example, the color of a specific point is the complementary color of the color of another part of the reference point, or the color of the specific point is the complementary color of the skin color). It may represent a point. For example, a fixed point fixed point may represent a specific point by making its size sufficiently small.

For example, when the pattern of the fixed point gauge point is a dot pattern, the three-dimensional coordinates of the fixed point gauge point can be derived by recognizing at least four points on the dot pattern. Derivation of three-dimensional coordinates can be achieved by using techniques known in the field of AR (augmented reality) and the like. The three-dimensional coordinates of the fixed point reference point correspond to the three-dimensional coordinates of the fixed point.

The correction means 131 is configured to correct the coordinate system of the subject's face based on the fixed point and a predefined face reference position template. For example, the correction means 131 transforms each of the plurality of images (eg, scaling, rotation, shearing, translation) so that each fixed point of the plurality of images moves to a corresponding point on the face reference position template. (Move, etc.). Alternatively, for example, the correction means 131 may each of the plurality of images so that the distance between each fixed point of the plurality of images and the corresponding point on the face reference position template is zero or less than a predetermined threshold. Is converted. Alternatively, for example, the correction means 131 transforms each of the plurality of images so that the plane defined by each fixed point of the plurality of images coincides with the corresponding plane on the face reference position template. The coordinate system of the subject's face in the plurality of images after the conversion process becomes the coordinate system of the corrected face. The image conversion process can be performed by, for example, an affine transformation. For example, when the fixed point is a region covering at least a part of the upper facial region of the face (for example, a curved surface covering the forehead), the correction means 131 has the region and the corresponding region on the face reference position template. Each of the plurality of images can be converted so that the error is minimized. This can be done, for example, using the method of least squares. The correction means 131 may, for example, correct the coordinate system of the subject's face by using the motion point in addition to the fixed point.

In one example, the face reference position template is a template that defines the position of the face (that is, the face reference position) when the face is facing the front. For example, a face reference position template can be anatomically defined. For example, the face reference position template is a template that defines the position of the face when the eye-ear plane (Frankfurt plane), the nasal hearing line (or Campel plane), the hip plane, the occlusal plane, or both pupil lines are horizontal. Can be. For example, the face reference position template can be a template that defines the position of the face in a state where the median line or the mid-sagittal plane is vertical. For example, a face reference position template can be a template that defines the position of the face with the orbital plane facing forward. For example, the face reference position template may be an ophthalmic plane (Frankfurt plane), a nasal hearing line (or Campel plane), a hip plane, an occlusal plane, or with both pupil lines horizontal and / or a midline. Alternatively, it may be a template that defines the position of the face with the midsaline plane perpendicular and / or with the orbital plane facing forward. The face reference position template can be used to confirm the position of each part of the face with the face facing forward. For example, the correction means 131 confirms the position of the face part in the face reference position template, and moves a plurality of corresponding parts (for example, a part where a fixed point is located) in the plurality of images to the position. Each of the images can be converted. For example, the correction means 131 confirms the orientation of various planes (eg, Frankfurt plane, Campel plane, etc.) in the face reference position template, and the orientation is defined by the corresponding planes (eg, fixed points) in the plurality of images. Each of the plurality of images can be converted so as to move the plane.

In one example, the face reference position template can be implemented as a template used when taking a plurality of consecutive images. For example, the image is displayed only when the face reference position template is displayed on the screen of the terminal device 300 when a plurality of images are taken by the terminal device 300, and the orientation of the subject's face matches the face reference position template. Can be taken. As a result, the appearance of the subject's face in each of the plurality of captured images is consistent, and the possibility that the extracted motion points differ among the plurality of consecutive images can be reduced. Limiting such captured images is also a type of correction by the computer system 100.

The first extraction means 141 may have the same configuration as the extraction means 122 described above with reference to FIG. The first extraction means 141 extracts the motion points in the lower jaw region of the face in the coordinate system corrected by the correction means 131. The generation means 123 generates motion point locus information indicating the locus of the motion point by tracking the extracted motion points. Since the motion points extracted from each of the plurality of images are acquired in the same coordinate system, the generated motion point trajectory information becomes more accurate information.

As described above, in the example shown in FIG. 6B, more accurate motion point trajectory information can be generated by correcting the coordinate system of the face using the fixed point as a reference. For example, when the fixed point moves independently of the jaw movement due to the movement of the subject's body when taking a plurality of consecutive images (for example, while performing the jaw movement when the eyebrows are set as the fixed point). If the fixed points do not match between multiple images due to the movement of the chin up and down or the skin in the upper facial area, etc.), correct the facial coordinate system using the fixed points as a reference. In addition, it is preferable to correct the movement of the fixed point so as to cancel it. This is because the error of the locus of the moving point due to the movement of the fixed point is reduced, and more accurate information on the locus of the moving point can be generated.

In order to compensate for offsetting the movement of the fixed point, the correction means 131 may further include a second generation means (not shown) and a second correction means (not shown).

The second generation means is configured to generate fixed point locus information indicating the locus of the fixed point by tracking the fixed point. The second generation means may, for example, generate fixed point trajectory information for each of at least 3 pixels, or at least 1 pixel out of at least 3 pixels (eg, the pixel closest to the midline, the uppermost). Fixed point trajectory information may be generated for pixels at, the bottom of the pixel, randomly selected pixels, etc.). The second generation means may generate fixed point locus information for, for example, a center of gravity of at least 3 pixels. The fixed point locus information is information indicating the locus of a fixed point within a predetermined period. Here, the predetermined period may be, for example, all or a part of the period in which a plurality of consecutive images are taken. The second generation means can generate fixed point locus information, for example, by tracking the coordinates in the image of the fixed point in each of a plurality of consecutive images.

The second correction means is configured to correct the motion point trajectory information based on the fixed point trajectory information. For example, the second correction means may correct the motion point locus information by subtracting the locus of the fixed point from the locus of the motion point. At this time, for example, among the fixed point locus information of each of at least 3 pixels, the fixed point locus information of the pixel having the smallest locus movement is used, and the locus of the fixed point is subtracted from the locus of the locus to move the locus. The trajectory information can be corrected. Alternatively, for example, by using the fixed point locus information of the pixel having the largest movement of the locus among the fixed point locus information of at least 3 pixels, the locus of the fixed point is subtracted from the locus of the moving point to obtain the locus of the moving point. Information can be corrected. Alternatively, for example, the motion point trajectory information can be corrected by subtracting the motion point trajectory from the motion point trajectory using the fixed point trajectory information of the center of gravity of at least 3 pixels.

For example, the second correction means may correct the motion point locus information by tracking the corrected motion point obtained by subtracting the coordinates of the fixed point from the coordinates of the motion point. At this time, for example, among the fixed point trajectory information of at least 3 pixels, the coordinates of the pixel having the smallest locus movement may be subtracted from the coordinates of the motion point to obtain the corrected motion point. Alternatively, for example, the coordinates of the pixel having the largest movement of the locus among the fixed point locus information of at least 3 pixels may be subtracted from the coordinates of the moving point to obtain the corrected moving point. Alternatively, for example, the coordinates of the center of gravity of at least 3 pixels may be subtracted from the coordinates of the motion point to obtain the corrected motion point.

The motion point trajectory information corrected in this way does not include an error due to the movement of the fixed point, and can be more accurate information.

In the above-mentioned example, it has been described that the motion point locus information is generated by correcting the coordinate system of the face, extracting the motion points in the corrected coordinate system, and tracking the motion points. , The timing of correction is not limited to this. For example, the motion point locus information may be generated by extracting the motion points, correcting the extracted motion points, and tracking the corrected motion points. For example, the motion point locus information may be generated by extracting the motion points and tracking the extracted motion points, and the generated motion point trajectory information may be corrected. This can be achieved by correcting the coordinate system of the moving point or the moving point locus information by the same method as the correction of the coordinate system described above.

That is, the motion point trajectory information generated by the generation means 123 includes information indicating motions other than the motion points of the lower jaw (for example, movement of the subject's body during imaging (for example, movement of the upper jaw) and noise due to tilt). In some cases, it may not contain information indicating movement other than the movement point of the lower jaw. In the former case, the motion point trajectory information is corrected after the motion point trajectory information is generated. In the latter case, the coordinate system is corrected to extract the motion points before the motion point trajectory information is generated, or the motion points are extracted and extracted before the motion point tracking information is generated. This is the case of correcting the motion points.

FIG. 6C shows an example of the configuration of the processor unit 150 in another embodiment. The processor unit 150 may have a configuration for correcting the coordinate system of the subject's face by using the subject's jaw movement face model. The processor unit 150 is a processor unit included in the computer system 100 as an alternative to the processor unit 120 described above. In FIG. 6C, the same reference numbers are assigned to the same components as those described above in FIGS. 3 and 6A, and the description thereof will be omitted here.

The processor unit 150 includes an acquisition unit 121, a correction unit 131, an extraction unit 122, and a generation unit 123. The correction means 131 includes a base face model generation means 151 and a jaw movement face model generation means 152.

The base face model generation means 151 is configured to generate a base face model of the subject's face. The base face model creating means 151 may generate a base face model from, for example, an image of the subject's face acquired in advance (for example, an image of the subject's face stored in the database unit 200). The base face model may be generated from the image of the subject's face received from the terminal device 300 via the interface unit 110. The face image used to generate the base face model is preferably a front-facing, stationary, expressionless image. This is because the generated base face model becomes an expressionless one facing the front, and it becomes easy to correspond to various movements. The base face model generation means 151 can generate a base face model by using any known method. The image used may be an image containing two-dimensional information (vertical x horizontal), but is preferably an image containing three-dimensional information (vertical x horizontal x depth). This is because an image containing three-dimensional information can generate a base face model more easily and with high accuracy.

The jaw movement face model generation means 152 is configured to generate a jaw movement face model of a subject by reflecting the face of the subject in a plurality of images in the base face model. The jaw movement face model generation means 152 generates a jaw movement face model of the subject by reflecting the face of the subject in a plurality of consecutive images received from the terminal device 300 via the interface unit 110 in the base face model. do. The jaw movement face model generation means 152 can generate a jaw movement face model by using any known method. For example, a jaw movement face model can be created by deriving the coordinates of each part of the subject's face in a plurality of consecutive images and mapping the coordinates of each part on the base face model. The generated jaw movement face model is a 3D avatar that moves according to the movement of the subject shown in a plurality of images.

In one example, the base face model generation means 151 and the jaw movement face model generation means 152 are subjected to the same processing as the processing for constructing the "animoji" implemented in iPhone (registered trademark) X, and the jaw movement face model is performed. Can be generated.

The correction means 131 corrects the difference in the coordinate system between the base face model and the jaw movement face model. The correction means 131 is configured to correct the coordinate system of the face by correcting the coordinate system of the jaw movement face model based on the coordinate system of the base face model. The correction means 131 corrects the coordinate system of the face by converting the coordinate system of the jaw movement face model to the coordinate system of the base face model, for example, by performing an arbitrary coordinate conversion process.

The correction means 131, for example, causes the jaw movement face model generation means 152 to perform coordinate conversion processing on the coordinates of each part of the subject's face in a plurality of consecutive images before the jaw movement face model generation means 152 generates the jaw movement face model. You may. As a result, the coordinate system of the generated jaw movement face model matches the coordinate system of the base face model.

The extraction means 122 extracts the motion points in the lower jaw region of the face in the coordinate system corrected by the correction means 131. The extraction means 122 may, for example, extract the movement points in the base face model, may extract the movement points in the jaw movement face model before the coordinate conversion, or may extract the movement points in the jaw movement after the coordinate conversion. The movement points may be extracted in the face model. When extracting motion points in the base face model, the motion points are reflected in the corresponding points of the jaw motion face model in the process of generating the jaw motion face model.

The generation means 123 generates motion point locus information indicating the locus of the motion point by tracking the extracted motion points. Since the motion points extracted from each of the plurality of images are acquired in the same coordinate system, the generated motion point trajectory information becomes more accurate information.

In the above example, it has been described that the motion point trajectory information is generated by tracking the motion points in the corrected coordinate system (that is, the coordinate system of the base face model), but in the present invention, the timing of the correction is determined. Not limited to this. For example, the motion point trajectory information may be generated by extracting the motion points in the coordinate system of the jaw motion face model, correcting the extracted motion points, and tracking the corrected motion points. For example, the corrected motion point trajectory information may be generated by extracting the motion points in the coordinate system of the jaw motion face model and correcting the motion point trajectory information generated by tracking the motion points. good. For example, motion point trajectory information generated by extracting motion points in multiple images, reflecting the extracted motion points in the jaw motion face model, and tracking the reflected motion points or the reflected motion points. The motion point trajectory information may be generated by the correction. This can be achieved by correcting the coordinate system of the moving point or the locus of the moving point by the same method as the coordinate conversion process described above.

That is, the motion point trajectory information generated by the generation means 123 includes information indicating motions other than the motion points of the lower jaw (for example, movement of the subject's body during imaging (for example, movement of the upper jaw) and noise due to tilt). In some cases, it may not contain information indicating movement other than the movement point of the lower jaw. In the former case, the motion point trajectory information is corrected after the motion point trajectory information is generated. In the latter case, the coordinate system is corrected to extract the motion points before the motion point trajectory information is generated, or the motion points are extracted and extracted before the motion point tracking information is generated. This is the case of correcting the motion points.

FIG. 6D shows an example of the configuration of the processor unit 160 in another embodiment. The processor unit 160 may have a configuration for evaluating the motion of the subject based on the generated motion point trajectory information. The processor unit 160 is a processor unit included in the computer system 100 as an alternative to the processor unit 120 described above. In FIG. 6D, the same reference figures are assigned to the same components as those described above in FIG. 3, and the description thereof will be omitted here.

The processor unit 160 includes an acquisition unit 121, an extraction unit 122, a generation unit 123, and an evaluation unit 161.

The evaluation means 161 is configured to generate jaw movement evaluation information indicating the evaluation of the jaw movement of the subject, based on at least the movement point trajectory information. The evaluation means 161 can generate jaw motion evaluation information by using the motion point trajectory learned model that has been processed to learn the motion point trajectory information of a plurality of subjects. The motion point trajectory trained model is configured to correlate the input motion point trajectory information with the evaluation of jaw motion. The motion point trajectory information generated by the generation means 123 may include noise due to the movement and inclination of the subject's body during imaging, and the motion point trajectory trained model includes the motion point trajectory including such noise. Since the processing for learning the information is performed, it is possible to accurately generate the jaw movement evaluation information regardless of the noise caused by the movement or tilt of the subject's body during imaging, which may be included in the movement point trajectory information. The motion point trajectory information used for learning may be two-dimensional information (information indicating the trajectory of the motion point in a certain plane) or three-dimensional information (information indicating the trajectory of the motion point in a certain space). ), Or four-dimensional information (information indicating the locus of the moving point and the speed of the moving point in a certain space). Here, the velocity may be a scalar quantity, but is preferably a vector quantity. By using higher-dimensional information, the accuracy of the constructed motion point trajectory trained model is improved. The motion point trajectory trained model can be constructed, for example, by learning thousands, tens of thousands, hundreds of thousands, or millions of motion point trajectory information. The more information you learn, the better the accuracy, but you need to be aware of overfitting.

The moving point trajectory trained model can be constructed by using any machine learning model. The motion point trajectory trained model can be, for example, a neural network model.

FIG. 7 shows an example of the structure of the neural network model 1610 that can be used by the evaluation means 161.

The neural network model 1610 has an input layer, at least one hidden layer, and an output layer. The number of nodes in the input layer of the neural network model 1610 corresponds to the number of dimensions of the input data. The hidden layer of the neural network model 1610 can contain any number of nodes. The number of nodes in the output layer of the neural network model 1610 corresponds to the number of dimensions of the output data. For example, when evaluating whether or not there is an abnormality in jaw movement, the number of nodes in the output layer may be 1. For example, when evaluating which of the seven patterns the jaw movement trajectory is, the number of nodes in the output layer can be seven.

The neural network model 1610 can be preliminarily trained using the information acquired by the acquisition means 121. The learning process is a process of calculating the weighting coefficient of each node of the hidden layer of the neural network model 1610 by using the data acquired by the acquisition means 121.

The learning process is, for example, supervised learning. In supervised learning, for example, the motion point trajectory information is used as input teacher data, the evaluation of the corresponding jaw movement is used as output teacher data, and the information of a plurality of subjects is used for each node of the hidden layer of the neural network model 1610. By calculating the weighting coefficient of, it is possible to construct a trained model that can correlate the motion point trajectory information with the evaluation of jaw motion.

For example, a set of (input teacher data, output teacher data) for supervised learning includes (first subject's motion point trajectory information, first subject's jaw motion evaluation), (second subject). (Evaluation of the jaw movement of the second subject), ... (Information on the movement point trajectory of the i-th subject, evaluation of the jaw movement of the i-th subject, ...). When the motion point trajectory information newly acquired from the subject is input to the input layer of the trained neural network model, the evaluation of the jaw motion of the subject is output to the output layer.

The learning process is, for example, unsupervised learning. In unsupervised learning, for example, for a plurality of subjects, the outputs are divided into a plurality of clusters by clustering a plurality of outputs when the motion point trajectory information is used as input teacher data. For each cluster of multiple clusters, characterize each cluster based on the assessment of jaw movements of the subject to which it belongs. As a result, a trained model capable of correlating the motion point trajectory information with the evaluation of jaw motion is constructed. Clustering can be performed using, for example, any known technique. When the motion point trajectory information newly acquired from the subject is input to the input layer of the trained neural network model, the evaluation of the jaw motion of the subject is output to the output layer.

The motion point trajectory information generated in the example described above with reference to FIGS. 6A-6C can be used, for example, to evaluate the jaw motion of the subject. In this case, the processor unit 120, 130, 140, or 150 provides an evaluation means (not shown) that generates jaw movement evaluation information indicating the evaluation of the jaw movement of the subject based on at least the generated movement point trajectory information. Further prepared. The evaluation means may have the same configuration as the evaluation means 161 included in the processor unit 160, or may have a different configuration. As the evaluation means, for example, the jaw motion evaluation information can be generated by using the motion point trajectory learned model that has been processed to learn the motion point trajectory information of a plurality of subjects.

In the above-mentioned example with reference to FIGS. 6A to 6C, the correction means 131 is used to deal with the movement and inclination of the subject during imaging so that the extracted motion points do not differ between a plurality of consecutive images. The configuration including is described. In the example shown in FIG. 6D, the movement and inclination of the subject during imaging are dealt with by using a learned model in which the locus of jaw movement including the body movement and inclination of the subject during imaging is learned.

In the example shown in FIG. 2, the database unit 200 is provided outside the computer system 100, but the present invention is not limited thereto. It is also possible to provide at least a part of the database unit 200 inside the computer system 100. At this time, at least a part of the database unit 200 may be implemented by the same storage means as the storage means for mounting the memory unit 170, or may be implemented by a storage means different from the storage means for mounting the memory unit 170. You may. In any case, at least a part of the database unit 200 is configured as a storage unit for the computer system 100. The configuration of the database unit 200 is not limited to a specific hardware configuration. For example, the database unit 200 may be composed of a single hardware component or may be composed of a plurality of hardware components. For example, the database unit 200 may be configured as an external hard disk device of the computer system 100, or may be configured as a storage on the cloud connected via a network.

In the example shown in FIGS. 3 and 6A to 6D described above, the components of the processor units 120, 130, 140, 150 and 160 are provided in the same processor unit 120, 130, 140, 150 and 160. However, the present invention is not limited to this. It is also within the scope of the present invention that the components of the processor units 120, 130, 140, 150, and 160 are distributed to a plurality of processor units. At this time, the plurality of processor units may be located in the same hardware component, or may be located in separate hardware components in the vicinity or remote. For example, it is preferable that the base face model generation means 151 of the processor unit 150 is implemented by a processor unit different from other components. This makes it possible to separately perform the heavy-duty process of creating a base model.

Each component of the computer system 100 described above may be composed of a single hardware component or may be composed of a plurality of hardware components. When it is composed of a plurality of hardware parts, the mode in which each hardware part is connected does not matter. Each hardware component may be connected wirelessly or may be connected by wire. The computer system 100 of the present invention is not limited to a specific hardware configuration. It is also within the scope of the present invention that the processor units 120, 130, 140, 150, 160 are configured by an analog circuit instead of a digital circuit. The configuration of the computer system 100 of the present invention is not limited to the above-mentioned one as long as the function can be realized.

3. 3. Processing by a computer system for measuring the jaw movement of a subject FIG. 8 is a flowchart showing an example (processing 800) of processing by a computer system 100 for measuring the jaw movement of a subject. The process 800 is executed, for example, in the processor unit 130, 140 or 150 in the computer system 100.

In step S801, the acquisition means 121 of the processor unit acquires a plurality of continuous images of the face of the subject during jaw movement. The acquisition means 121 can acquire, for example, a plurality of consecutive images of the subject's face during jaw movement stored in the database unit 200 via the interface unit 110. Alternatively, the acquisition means 121 can acquire a plurality of continuous images received from the terminal device 300 via the interface unit 110, for example.

In step S802, the correction means 131 of the processor unit at least corrects the coordinate system of the subject's face. The correction means 131 of the processor unit can correct the coordinate system of the subject's face, for example, based on the image acquired in step S802.

In step S803, the extraction means 122 of the processor unit extracts at least the motion points in the lower jaw region of the face in the coordinate system corrected in step S802 of each of the above-described embodiments. For example, the extraction means 122 can receive an input of where the motion point in the image is via the interface unit 110 (for example, from the terminal device 300), and can extract the motion point based on the input. Alternatively, the extraction means 122 can automatically extract the motion point without receiving an input, for example.

For example, the extraction means 122 detects a plurality of feature portions of the face for each of the plurality of images, and among the plurality of extracted feature portions, a portion whose coordinate change within a predetermined period is within a predetermined range. It can be extracted as a motion point. Here, the predetermined period may be, for example, all or a part of the period in which a plurality of consecutive images are taken. The predetermined range can be, for example, about 5 mm to about 20 mm.

In step S804, the generation means 123 of the processor unit traces the motion point to generate at least the motion point trajectory information indicating the trajectory of the motion point. The motion point locus information is information indicating the trajectory of the motion point within a predetermined period. Here, the predetermined period may be, for example, all or a part of the period in which a plurality of consecutive images are taken.

According to the process 800, since the motion points extracted from each of the plurality of images are acquired in the same coordinate system, the generated motion point trajectory information becomes more accurate information.

In one embodiment, in step S802, the correction means 131 of the processor unit 140 corrects the coordinate system of the subject's face based on the fixed point and the predefined face reference position template.

In this embodiment, step S8020 can be included before step S802. In step S8020, the second extraction means 142 of the extraction means 122 of the processor unit 140 extracts fixed points in the upper facial region of the face based on the plurality of images acquired in step S801. The second extraction means 142 receives, for example, an input of where the fixed point in the image is via the interface unit 110 (for example, from the terminal device 300), and extracts the fixed point based on the input. Can be done. Alternatively, the second extraction means 142 can automatically extract a fixed point without receiving an input, for example.

The second extraction means 142 detects, for example, a plurality of feature portions of the face for each of the plurality of images, and among the plurality of extracted feature portions, the coordinate change within a predetermined period is less than a predetermined threshold value. The part can be extracted as a fixed point. Here, the predetermined period may be, for example, all or a part of the period in which a plurality of consecutive images are taken. The predetermined range can be, for example, about 5 mm to about 20 mm. The second extraction means 142 can, for example, perform color enhancement processing on an image and extract a portion having a color tone different from the surroundings as a fixed point. The fixed point extracted in step S8020 can be considered to be one of those extracted in the extraction step of step S803, and thus step S8020 can be considered to be part of step S803.

In step S802, the correction means 131 of the processor unit 140 corrects the coordinate system of the subject's face based on the fixed point extracted in step S8020 and the predetermined face reference position template. For example, the correction means 131 converts each of the plurality of images (for example, for example) so that the fixed point extracted in step S8020 moves to the corresponding point on the face reference position template for each of the plurality of images. Scale, rotation, shear, translation, etc.). Alternatively, for example, the correction means 131 makes the distance between the fixed point extracted in step S8020 and the corresponding point on the face reference position template zero or less than a predetermined threshold for each of the plurality of images. In addition, each of the plurality of images is converted. Alternatively, for example, the correction means 131 transforms each of the plurality of images so that the plane defined by each fixed point of the plurality of images coincides with the corresponding plane on the face reference position template. Here, for example, the face reference position template is a template that defines the position of the face when the face is facing the front, and can be anatomically defined.

In another embodiment, in step S802, the correction means 131 of the processor unit 130 uses a reference coordinate system trained model that has been processed to learn the reference coordinate system of the faces of a plurality of subjects, and uses a plurality of images. The coordinate system of the subject's face can be corrected for each of the above. The reference coordinate system trained model is configured to output an image obtained by correcting the coordinate system of the subject's face in the input image to the reference coordinate system.

The frame of reference trained model can be constructed, for example, by supervised learning. In supervised learning, for example, an image of the subject's face can be used as input teacher data, and the reference coordinate system in the image can be used as output teacher data. By repeatedly learning a plurality of images of a plurality of subjects, the trained model becomes able to recognize a reference coordinate system that is estimated to be statistically possessed by the faces of the plurality of subjects. When an image of the subject's face is input to such a trained model, the difference between the subject's face coordinate system and the reference coordinate system is output. For example, the input image is converted (for example, scaling, rotation, shearing, translation, etc.) so that the difference between the coordinate system of the subject's face and the reference coordinate system becomes zero or less than a predetermined threshold. By constructing a trained model in, a frame of reference trained model can be generated. The image conversion process can be performed using, for example, an affine transformation.

The frame of reference trained model can be constructed, for example, by unsupervised learning. In unsupervised learning, for example, facial images of a plurality of subjects can be used as input teacher data. By repeatedly learning a large number of images of multiple subjects, the trained model can recognize the facial coordinate system common to many images as a reference coordinate system that is estimated to be statistically possessed by multiple subjects. become able to. When an image of the subject's face is input to such a trained model, the difference between the subject's face coordinate system and the reference coordinate system is output. For example, the input image is converted (for example, scaling, rotation, shearing, translation, etc.) so that the difference between the coordinate system of the subject's face and the reference coordinate system becomes zero or less than a predetermined threshold. By constructing a trained model in, a frame of reference trained model can be generated. The image conversion process can be performed using, for example, an affine transformation.

In step S802, the correction means 131 of the processor unit 130 inputs a plurality of images acquired in step S801 into the reference coordinate system trained model, and the coordinate system of the subject's face in the input image becomes the reference coordinate system. A corrected image can be obtained.

In yet another embodiment, in step S802, the processor unit 150 correction means 131 corrects the coordinate system of the subject's face using the subject's jaw movement face model.

In this embodiment, step S8021 and step S8022 can be included before step S802. In step S8021, the base face model generation means 151 of the correction means 131 of the processor unit 150 generates the base face model of the subject's face. The base face model creating means 151 may generate a base face model from, for example, an image of the subject's face acquired in advance (for example, an image of the subject's face stored in the database unit 200). The base face model may be generated from the image of the subject's face received from the terminal device 300 via the interface unit 110. If the base face model is generated in advance, step S8021 may be omitted.

In step S8022, the jaw movement face model generation means 152 of the correction means 131 of the processor unit 150 reflects the face of the subject in the plurality of images acquired in step S801 on the base face model, so that the jaw movement face of the subject is reflected. Generate a model. The generated jaw movement face model is a 3D avatar that moves according to the jaw movement of the subject shown in the plurality of images acquired in step S801.

In step S802, the correction means 131 of the processor unit 150 corrects the coordinate system of the jaw movement face model generated in step S8022 based on the coordinate system of the base face model generated in step S8021. The correction means 131 can convert the coordinate system of the jaw movement face model to the coordinate system of the base face model, for example, by performing an arbitrary coordinate conversion process.

When the coordinate system of the subject's face is corrected based on the fixed point and the face reference position template in step S802, the jaw movement is caused by the movement of the subject's body during the acquisition of a plurality of consecutive images. May move the fixed point independently. Therefore, the process 800 can include a process (step S805, step S806) for canceling the fixed point locus information that may be included in the motion point locus information.

In step S805, the second generation means of the correction means 131 of the processor unit 140 generates fixed point locus information indicating the locus of the fixed point by tracking the fixed point extracted in step S8010. The fixed point locus information is information indicating the locus of a fixed point within a predetermined period. Here, the predetermined period may be, for example, all or a part of the period in which a plurality of consecutive images are taken. The second generation means can generate fixed point locus information, for example, by tracking the coordinates in the image of the fixed point in each of a plurality of consecutive images.

In step S806, the second correction means of the correction means 131 of the processor unit 140 corrects the motion point trajectory information based on the fixed point trajectory information. For example, the second correction means can correct the motion point locus information by subtracting the locus of the fixed point from the locus of the motion point. Alternatively, for example, the second correction means may correct the motion point locus information by tracking the corrected motion point obtained by subtracting the coordinates of the fixed point from the coordinates of the motion point. The motion point trajectory information corrected in step S805 and step S806 can be regarded as one of those corrected in the correction step of step S802, and therefore step S805 and step S806 are a part of step S802. Can be considered to be.

The motion point trajectory information corrected in this way does not include an error due to the movement of the fixed point, and can be more accurate information.

In the above-mentioned example, it has been described that the motion point trajectory information is generated by tracking the motion points in the corrected coordinate system, but in the present invention, the timing of the correction is not limited to this. For example, in step S803 before step S802, motion points are extracted from a plurality of images acquired in step S801, and then in step S804, motion point trajectory information is generated by tracking the motion points, and then the step. In S802, the motion point locus information generated in step S804 may be corrected. Alternatively, for example, in step S803 before step S802, the motion points are extracted from the plurality of images acquired in step S801, then in step S802, the motion points extracted in step S803 are corrected, and then the step. In S804, the motion point trajectory information may be generated by tracking the corrected motion point.

That is, the motion point trajectory information generated by the generation means 123 includes information indicating motions other than the motion points of the lower jaw (for example, movement of the subject's body during imaging (for example, movement of the upper jaw) and noise due to tilt). In some cases, it may not contain information indicating movement other than the movement point of the lower jaw. The former is a case where the motion point trajectory information is corrected after the motion point trajectory information is generated. In the latter case, the coordinate system is corrected to extract the motion points before the motion point trajectory information is generated, or the motion points are extracted and the extracted motion points are extracted before the motion point tracking information is generated. This is the case for correction.

FIG. 9 is a flowchart showing another example (process 900) of processing by the computer system 100 for measuring the jaw movement of the subject. Process 900 is a process for evaluating the jaw movement of the subject. The process 900 is executed, for example, in the processor unit 160 in the computer system 100.

In step S901, the acquisition means 121 of the processor unit 160 acquires a plurality of continuous images of the subject's face during jaw movement. Step S901 is the same process as step S801.

In step S902, the extraction means 122 of the processor unit 160 extracts at least the motion points in the mandibular region of the face based on the plurality of images acquired in step S801. Extract the motion points in the mandibular region of the face. For example, the extraction means 122 can receive an input of where the motion point in the image is via the interface unit 110 (for example, from the terminal device 300), and can extract the motion point based on the input. Alternatively, the extraction means 122 can automatically extract the motion point without receiving an input, for example.

For example, the extraction means 122 detects a plurality of feature portions of the face for each of the plurality of images, and among the plurality of extracted feature portions, a portion whose coordinate change within a predetermined period is within a predetermined range. It can be extracted as a motion point. Here, the predetermined period may be, for example, all or a part of the period in which a plurality of consecutive images are taken. The predetermined range can be, for example, about 5 mm to about 20 mm.

In step S903, the generation means 123 of the processor unit 160 generates at least the motion point locus information indicating the trajectory of the motion point by tracking the motion points extracted in step S902. The motion point locus information is information indicating the trajectory of the motion point within a predetermined period. Here, the predetermined period may be, for example, all or a part of the period in which a plurality of consecutive images are taken.

In step S904, the evaluation means 161 of the processor unit 160 generates jaw movement evaluation information indicating the evaluation of the jaw movement of the subject based on at least the movement point trajectory information generated in step S903. The evaluation means 161 can generate jaw motion evaluation information by using the motion point trajectory learned model that has been processed to learn the motion point trajectory information of a plurality of subjects. The motion point trajectory trained model is configured to correlate the input motion point trajectory information with the evaluation of jaw motion. For example, when the motion point trajectory information generated in step S903 is input to the motion point trajectory learned model, the estimated evaluation of the jaw motion of the subject can be output.

The motion point trajectory information generated in step S903 may include noise due to the movement and inclination of the subject's body during imaging, and the motion point trajectory trained model includes motion point trajectory information including such noise. In step S904, the jaw movement evaluation information is accurately generated regardless of the noise caused by the movement or tilt of the subject's body during luck photography, which may be included in the movement point trajectory information. be able to.

For example, after performing steps S901 to S904, the exercise point locus trained model may be updated by performing the same processing as the learning processing described later.

FIG. 10 is a flowchart showing another example (process 1000) of processing by the computer system 100 for measuring the jaw movement of the subject. The process 1000 is a process for constructing a motion point trajectory trained model used for measuring the jaw motion of the subject. The process 1000 is executed, for example, in the processor unit 160 in the computer system 100.

In step S1001, the acquisition means 121 of the processor unit 160 acquires at least the motion point locus information indicating the locus of the motion points obtained by tracking the motion points of a plurality of subjects. The acquisition means 121 can acquire, for example, the motion point trajectory information stored in the database unit 200 via the interface unit 110. The motion point trajectory information may be, for example, motion point trajectory information acquired by using the computer system 100 of the present invention, or motion point trajectory information obtained from any known jaw motion measuring device. May be good.

In step S1001, the acquisition means 121 may further acquire an evaluation of the jaw movements of a plurality of subjects. The acquisition means 121 can acquire, for example, the evaluation of the jaw movement stored in the database unit 200 via the interface unit 110.

In step S1002, the processor unit 160 constructs a motion point trajectory learned model by learning processing using at least the motion point trajectory information of a plurality of subjects acquired in step S1001 as input teacher data. The motion point trajectory trained model can be, for example, a neural network model.

For example, when the moving point trajectory trained model is a neural network model, in step S1002, the weighting coefficient of each node in the hidden layer of the neural network model is set by using the data acquired in step S1001 by the learning process. It is calculated.

The learning process is, for example, supervised learning. In supervised learning, for example, the motion point trajectory information is used as input teacher data, the evaluation of the corresponding jaw movement is used as output teacher data, and the information of a plurality of subjects is used for each node of the hidden layer of the neural network model 1610. By calculating the weighting coefficient of, it is possible to construct a trained model that can correlate the motion point trajectory information with the evaluation of jaw motion.

The learning process is, for example, unsupervised learning. In unsupervised learning, for example, for a plurality of subjects, a plurality of outputs when the motion point trajectory information is used as input teacher data are classified. Classification can be performed using any known method, and a trained model capable of correlating motion point trajectory information with jaw motion evaluation by characterizing the classified output with jaw motion evaluation is available. Will be built.

Classification is done, for example, by clustering. For example, by clustering a plurality of outputs, the outputs are divided into a plurality of clusters. For each cluster of multiple clusters, characterize each cluster based on the assessment of jaw movements of the subject to which it belongs. As a result, a trained model capable of correlating the motion point trajectory information with the evaluation of jaw motion is constructed. Clustering can be performed using, for example, any known technique.

In the above-mentioned example with reference to FIGS. 8 to 10, it has been described that the processes are performed in a specific order, but the order of each process is not limited to the described one, and any order that is logically possible is not limited. Can be done in.

In the above-described example with reference to FIGS. 8 to 10, the processing of each step shown in FIGS. 8 to 10 includes a processor unit 120, a processor unit 130, a processor unit 140, a processor unit 150, or a processor unit 160 and a memory. Although it has been described that it is realized by the program stored in the unit 170, the present invention is not limited to this. At least one of the processes of each step shown in FIGS. 8 to 10 may be realized by a hardware configuration such as a control circuit.

In the above-mentioned example, the case where the computer system 100 is a server device connected to the terminal device 300 via the network 400 has been described as an example, but the present invention is not limited thereto. The computer system 100 can be any information processing device including a processor unit. For example, the computer system 100 can be a terminal device 300. Alternatively, for example, the computer system 100 may be a combination of a terminal device 300 and a server device.

In the above-mentioned example, the computer system 100 has been described as an implementation example of the present invention, but the present invention can also be implemented as, for example, a system including the computer system 100. The system comprises, for example, the computer system 100 described above and a gauge point configured to be placed in the mandibular region of the subject. The gauge point may be configured to represent a particular point on the gauge point, as described above. By using a plurality of continuous images taken with the reference point placed in the lower jaw region of the subject, the process of extracting the motion points in the computer system 100 becomes easy, and the number of motion points to be extracted is multiple. Will match between the images in. The system may further comprise a fixed point gauge configured to be placed in the upper facial area of the subject. Fixed point gauges may also be configured to represent specific points on the fixed point, as described above. By using a plurality of continuous images taken with the fixed point reference point installed in the upper face region of the subject, the fixed point extraction process in the computer system 100 becomes easy.

The present invention is not limited to the above-described embodiments, and various modifications can be made within the scope of the claims, and the embodiments obtained by appropriately combining the technical means disclosed in the different embodiments. Is also included in the technical scope of the present invention.

The present invention is useful as providing a system or the like capable of easily measuring the jaw movement of a subject.

100 Computer system 110 Interface part 120, 130, 140, 150, 160 Processor part 170 Memory part 200 Database part 300 Terminal device 400 Network

Claims (17)

A system for measuring jaw movement,
It is an acquisition means for acquiring a plurality of consecutive images of the subject's face during jaw movement, and the plurality of images include an image in which the subject's face is tilted, and the subject's face is included. The subject's face in the tilted image is tilted with respect to the subject's face in at least one of the plurality of images.
A correction means for at least correcting the coordinate system of the subject's face, wherein the correction means matches the coordinate system of the subject's face in each of the plurality of images including an image in which the subject's face is tilted. As a correction means for correcting the coordinate system of the subject's face,
An extraction means for extracting at least a motion point in the mandibular region of the face,
A system including a generation means for generating motion point locus information indicating at least the locus of the motion point by tracking the motion point. The extraction means is
A first extraction means for extracting a motion point in the lower jaw region of the face,
A second extraction means for extracting a fixed point in the upper facial region of the face from the plurality of images is provided.
The system of claim 1, wherein the correction means corrects the coordinate system based on the fixed point and a predefined face reference position template. The first extraction means extracts a plurality of feature portions in the plurality of images, and among the plurality of feature portions, the feature portions whose coordinate changes within a predetermined period are within a predetermined range are extracted as motion points.
The second extraction means extracts a feature portion whose coordinate change within a predetermined period is less than a predetermined threshold value as a fixed point among the plurality of feature portions.
The system according to claim 2. The correction means
The first correction means for correcting the coordinate system of the subject's face,
A second generation means for generating fixed point locus information indicating the locus of the fixed point by tracking the fixed point, and
The system according to any one of claims 2 to 3, further comprising a second correction means for correcting the motion point trajectory information based on the fixed point trajectory information. The correction means
It is a reference coordinate system trained model that has been processed to learn the reference coordinate system of the faces of a plurality of subjects, and the reference coordinate system trained model uses the coordinate system of the subject's face in the input image as described above. The system of claim 1, comprising a reference coordinate system trained model configured to be corrected to the reference coordinate system. The reference coordinate system trained model is
Taking the difference between the coordinate system of the subject's face in the input image and the reference coordinate system,
The system according to claim 5, wherein the coordinate system is corrected by performing a conversion process on the input image based on the difference. The system according to claim 6, wherein the conversion process includes an affine transformation. The extraction means extracts a plurality of feature portions in the plurality of images, and extracts pixels having a coordinate change within a predetermined period within a predetermined range as motion points from the plurality of feature portions. The system according to any one of 7. The correction means
A base face model generation means for generating a base face model of the subject's face,
A jaw movement face model generation means for generating a jaw movement face model of the subject by reflecting the face of the subject in the plurality of images in the base face model is provided.
The system according to claim 1, wherein the correction means corrects the coordinate system by correcting the coordinate system of the jaw movement face model based on the coordinate system of the base face model. The system according to claim 9, wherein the extraction means extracts motion points in the jaw motion face model or the base face model. The system according to any one of claims 1 to 10, further comprising an evaluation means for generating jaw movement evaluation information indicating the evaluation of the jaw movement of the subject based on at least the generated movement point trajectory information. Further equipped with a gauge point configured to be placed in the mandibular region of the subject.
The system according to any one of claims 1, 5 to 11, wherein the extraction means extracts the reference point in the plurality of images as a motion point. Further equipped with a gauge point configured to be placed in the mandibular region of the subject.
The system according to any one of claims 2 to 4, wherein the extraction means extracts the reference point in the plurality of images as a motion point. Further comprising a reference point configured to be placed in the upper facial area of the subject.
The system according to claim 13, wherein the second extraction means extracts the reference point in the plurality of images as a fixed point. The system according to any one of claims 12 to 14, wherein the reference point is configured to represent a specific point on the reference point. A program for measuring jaw movement, wherein the program is executed in a system including a processor unit, and the program is
It is to acquire a plurality of consecutive images of the subject's face during jaw movement, and the plurality of images include an image in which the subject's face is tilted, and the subject's face is included. The subject's face in the tilted image is tilted with respect to the subject's face in at least one of the plurality of images.
The correction is at least correcting the coordinate system of the subject's face, which means that the coordinate system of the subject's face matches in each of the plurality of images including the image in which the subject's face is tilted. As such, including correcting the coordinate system of the subject's face, and
Extracting at least the motion points in the mandibular region of the face and
A program for causing the processor unit to perform a process including at least generating motion point trajectory information indicating the trajectory of the motion point by tracking the motion point. A method for measuring jaw movement,
It is to acquire a plurality of consecutive images of the subject's face during jaw movement, and the plurality of images include an image in which the subject's face is tilted, and the subject's face is included. The subject's face in the tilted image is tilted with respect to the subject's face in at least one of the plurality of images.
The correction is at least correcting the coordinate system of the subject's face, which means that the coordinate system of the subject's face matches in each of the plurality of images including the image in which the subject's face is tilted. As such, including correcting the coordinate system of the subject's face, and
Extracting at least the motion points in the mandibular region of the face and
A method comprising tracking the motion point to at least generate motion point trajectory information indicating the trajectory of the motion point.

Images