TWI629662B - Method for realizing acupoint visualization by AR technology - Google Patents

Abstract

A method for realizing acupoint visualization by using AR technology, comprising a three-dimensional deformable model for establishing a trace on a human body, the model comprising a plurality of first marks and a point of acupuncture points; receiving an input image, marking the input image a second mark; fitting the three-dimensional deformable model according to the degree of deviation between the first mark and the coordinate information of the second mark to adjust the distribution of the first marks to approach the second a distribution of the markers; adjusting a coordinate of the at least one acupoint location point according to the coordinate variation of the first markers of the three-dimensional deformable model to obtain a corresponding estimated acupoint location point; and additionally estimating the acupoint location point to the input image Up to form an output image, and display the output image to achieve acupuncture visual effects.

Description

Method for realizing acupoint visualization by AR technology

The invention relates to a method for prompting the position of a human acupoint; in particular, a method for realizing acupuncture visualization by AR technology.

Traditional Chinese medicine acupuncture and acupoint massage are commonly used treatments in traditional Chinese medicine. According to the different symptoms of patients, acupuncture or massage can be performed on the acupuncture points corresponding to the symptoms, so as to treat or alleviate the discomfort of the patients.

However, unless professional training is performed for a period of time, it is difficult for the average person to find the location of the acupoint points corresponding to various symptoms and the corresponding uses. Furthermore, the exact position of the acupoints will vary depending on the individual body type, so For the general public, it is relatively unfamiliar to relieve their discomfort by acupuncture points.

Chinese Patent No. 104207931B discloses a precise positioning method for acupuncture points and acupuncture prescription learning methods, and the technical means thereof is to obtain the best fusion value of a certain acupoint point by a consistent information fusion algorithm combined with different acupuncture expert acupoint positioning measurement values. To achieve the objective function of acupuncture points. Among them, the acupuncture expert points collected by the aforementioned patents are positioned as two-dimensional data values. Therefore, when performing acupoint positioning, only a frontal photo of the face can be positioned. Secondly, the face matching performed by the aforementioned patent is to compare the face to be recognized with a plurality of face models to obtain the closest face model with the smallest Euclidean distance, however, if stored In the face model, the face to be recognized cannot be optimally matched, even if The fusion value of a certain point obtained by the closest facial model is actually different from the point of the acupuncture point of the face to be recognized, and has a place to be improved.

Furthermore, how to provide corresponding acupoint position points according to the difference of body shape or body characteristics of different users, in order to accurately locate the acupuncture points corresponding to different users, is also one of the problems that the inventors want to solve.

In view of this, the object of the present invention is to provide a method for realizing acupoint visualization by using AR technology, which can help different users quickly and accurately locate the corresponding acupuncture points.

In order to achieve the above object, the present invention provides a method for realizing acupoint visualization by using AR technology, comprising the following steps: A. providing a 3D Morphable Model based on a human body part, the three-dimensional deformable model including Having a plurality of first marks and at least one acupoint position points according to characteristics of the human body part; B, receiving an input image, and marking a plurality of second marks on the input image according to a human body part feature on the input image C, according to the degree of deviation between the first marks of the three-dimensional deformable model and the coordinate information of the second marks of the input image, the three-dimensional deformable model is fitted to adjust the first marks The distribution approaches the distribution of the second markers; D, adjusting the coordinates of the at least one acupoint location point according to the coordinate variation of the first markers of the three-dimensional deformable model to obtain a corresponding at least one estimated acupoint location Pointing; E, adding the at least one estimated acupoint position to the input image to form an output image; and F, displaying the output image.

The effect of the present invention is to obtain a corresponding estimated acupoint position point by comparing the difference between the three-dimensional deformable model and the input image, and estimate the position of the acupuncture point. The point is attached to the input image provided by the user, and the corresponding output image is output for the user to watch, thereby achieving the effect of visualizing the acupuncture point.

〔this invention〕

1‧‧‧Chinese medicine system

10‧‧‧Processing unit

20‧‧‧Input module

22‧‧‧ symptom input unit

24‧‧‧ acupoint input unit

26‧‧‧Image input unit

30‧‧‧Judgement module

32‧‧‧Database

34‧‧‧judging unit

40‧‧‧Image Processing Module

50‧‧‧ display module

60‧‧‧Smart mobile phones

70‧‧‧Server side

A‧‧‧ acupoint location

A’‧‧‧ acupoint location

A1‧‧‧ Estimated acupoint location

I‧‧‧ input image

L1‧‧‧ first mark

L11, L12, L13‧‧‧ first mark

L11’, L12’, L13’‧‧‧ first mark

L2‧‧‧ second mark

L21, L22, L23‧‧‧ second mark

M‧‧‧ deformable model

O‧‧‧ output image

v, v’‧‧‧ vertex

1 is a block diagram of a Chinese medicine system for implementing acupuncture visualization by AR technology according to an embodiment of the present invention.

2 is a schematic diagram of a situation in which an acupuncture point visualization Chinese medicine system is realized by AR technology according to an embodiment of the present invention.

FIG. 3 is a flow chart of a method for implementing acupuncture visualization by AR technology according to an embodiment of the present invention.

4 and 5 are schematic diagrams showing the second mark of the input image.

Figure 6 is a schematic diagram showing the degree of deviation between the input image and the deformable model.

7A and 7B are schematic diagrams of the initial and deformed three-dimensional deformable models projected on a two-dimensional plane, respectively.

Figure 8 is a schematic view of the plane coordinates, revealing the position of the moving acupoint to obtain the estimated point of the acupoint.

In order to explain the present invention more clearly, a preferred embodiment will be described in detail with reference to the drawings. Referring to FIG. 1 , a Chinese medicine system 1 for implementing a method for visualizing acupuncture points by using AR technology according to a preferred embodiment of the present invention includes a processing unit 10 , an input module 20 , and a The determination module 30, an image processing module 40 and a display module 50 are provided.

The input module 20 is connected to the processing unit 10. The input module 20 includes a symptom input unit 22 and/or a acupoint input unit 26, and an image input unit 26. The symptom input unit 22 is configured to receive a description of a symptom. For example, in an embodiment, the symptom input unit 22 can be a radio unit (such as a microphone) for receiving a user's voice description. The symptom is used to receive a sound file of the user, and the symptom of the user can be determined by analyzing the sound file. In addition, in an embodiment, the symptom input unit 22 can be an operation such as a touch screen, a keyboard, or the like. The interface is used for the user to input the description of the symptom; in addition, in an embodiment, the symptom input unit may also be a camera lens for capturing the image of the body part of the patient, and taking the feature of the image as an input. Description of the symptoms, but in other applications, not limited to this.

The acupoint input unit 24 is configured to receive a channel information. For example, in an embodiment, the acupoint input unit 24 can be a radio unit for recording the acupoint information input by the user in a voice manner. The acupoint information includes: acupuncture name, acupuncture function, and the like. In addition, in an embodiment, the acupoint input unit 24 can be an operation interface using a touch screen as an example, and multiple displays are displayed on the touch screen. A menu or button for acupuncture information can be selected by the user to input the selected acupoint message.

The image input unit 26 is configured to receive an input image. For example, the image input unit 26 can be a camera, a camera, or other lens-equipped mobile phone or wearable device (such as google glasses). The image input unit 24 can be operated to photograph a human body part to obtain an input image.

The determining module 30 is connected to the processing unit 10 and includes a database 32 and a determining unit 34. The database 32 is configured to store a plurality of disease data and a plurality of acupoint data associated with the foregoing conditions. In addition, in an embodiment, the database 32 may be an integrated medical library of Chinese medicine, except for storing a condition, In addition to the acupoint information, there are a variety of Information on medicinal materials, patient history information, and specialized hospital clinics recommended for each condition are not limited to this.

The determining unit 34 is configured to perform correlation matching with the disease data, acupoint data and the like stored in the database 32 according to the description of the symptom received by the input module 20 or the acupoint message, so as to obtain the corresponding symptoms. Acupoint information describing or acupuncture information.

The image processing module 40 is coupled to the processing unit 10 for attaching an estimated acupoint location point corresponding to the acupoint data to the input image to form an output image for display on the display module 50.

The foregoing input module 20 may be provided by one of an electronic device such as a notebook computer, a tablet computer, a desktop computer, a smart phone, a portable device, and a wearable device, or may be coordinated by the foregoing device. The operation of the above-mentioned symptom input unit 22, the acupoint input unit 24, and the image input unit 26, and the processing unit 10 may be the processing core of the foregoing device, but not limited thereto. The diagnostic module 30 can be seen in the foregoing electronic device, but not limited thereto, and can be built on another host or a server such as a cloud database, and can be through a network or other wireless or wired The communication method is connected with the aforementioned electronic device. The image processing module 40 may be provided by the aforementioned electronic device or provided by the server end. The display module 50 can be provided by the electronic device, or a display device can be additionally connected to the electronic device to implement the function of displaying the output image by the display module 50.

Referring to FIG. 1 and FIG. 2, in a use scenario, the Chinese medicine system includes a mobile device with a smart phone 60 as an example. The smart phone 60 has the processing unit 10, the input module 20, and the display module. The server 50 is provided with the determining module 30 and the image processing module 40.

First, the user can operate the smart phone 60 to connect to the server end 70, for example, connect to the server end 70 through a mobile application, and operate the smart phone 60 to input a description of the symptom or acupuncture point information. And uploading the description of the symptom or the acupoint message to the server end 70, and the server end receives the description of the symptom or the acupoint message, and can pass the determination unit 34 according to the description of the symptom or the acupoint message and the database. The data of the diseases stored in 32 and/or the acupoint data are correlated and the acupoint information corresponding to the description of the symptoms or the information of the acupoint is obtained. For example, after the correlation matching, the obtained acupoint data includes the Yingxiang and the giant acupoints.

Then, a two-dimensional input image provided by the user is received as a target image for subsequent acupoint visualization. In addition, in an embodiment, the server end 70 may further prompt the user to provide an input image of a specific body part feature corresponding to the matching result. For example, after the server end 70 performs the matching and confirming the corresponding acupoint data, the corresponding acupoint point position can be obtained together. For example, in the embodiment, the Yingxiang and the giant acupoints obtained by correlation matching are located. The face, then the server end 70 can further prompt the user to provide a facial image as an input image, for example, the head in the mannequin is represented by a different color or flashing, so that the prompt can be helpful. The user is to help the user to provide an image corresponding to the correct body part input.

After receiving the input image on the server end 70, the estimated acupoint position point is added to the input image according to the acupoint data obtained through the correlation matching to form an output image. For example, in the embodiment, the estimated acupoint position point A1 of the Yingxiang point and the giant acupoint point is attached to the input image to form the output image O, and is transmitted back to the smart phone 60. Displayed on the smart phone 60. The step of acquiring the input image and adding the estimated acupoint position point to the input image to form an output image can be repeatedly performed, so that the user can provide the instant acupoint view on the smart phone 60. The sensory output image O, the user can massage his own acupoint according to the estimated acupoint position point A1 of the output image O, thereby achieving the symptom relief effect, or the effect of learning and learning the specific acupoint position point.

Wherein, in order to make the displayed estimated acupoint position point A1 more accurately conform to the actual acupoint position on the input image provided by the user. As shown in FIG. 3, the method for acupoint visualization in the embodiment includes: step A, providing a 3D Morphable Model M based on a human body part (refer to FIG. 6), the three-dimensional deformable model M includes a plurality of first indicia L1 and at least one acupoint position point A distributed throughout the features of the body part. In the embodiment, the three-dimensional deformable model M is stored in the database 32, and the number of at least one acupoint location points is plural, and is respectively associated with the acupoint data in the database 32. The three-dimensional deformable model is constructed by collecting a large number of three-dimensional face scanning models in advance. Wherein, the human body part may be a part of the face, the hand, the leg, the sole of the foot, etc., but not limited thereto.

Wherein, in an embodiment, the three-dimensional deformable model is formed by Principal Component Analysis based on a plurality of three-dimensional face scanning models, and different face shapes can be determined according to different parameters. The three-dimensional deformable model M is composed of a plurality of vertices, each of which has coordinate information corresponding to x, y, and z, and the vertices include a first mark L 1 distributed according to the characteristics of the human body part and located at the vertices A plurality of acupoint position points A between the vertices. Wherein, when the mark of the three-dimensional deformable model M is performed, the vertex number of the first mark and the coordinate position of the acupoint point and at least three adjacent vertex numbers may be recorded together.

For example, assume that the three-dimensional deformable model M is composed of 3000 vertices, each of which has coordinate information of x, y, and z. If there are 60 parameters for controlling the face, the three-dimensional deformable model M has the following function. Relationship: NewFace(9000*1)= MeanShape (9000*1) + ShapeVector (9000 * 60) * Coeffs (60 * 1). Among them, MeanShape is a face synthesis model, ShapeVector is a parameter-changeable feature vector, and Coeffs is a parameter that can change the face shape. In this way, by giving different parameters, a three-dimensional deformable model of different face shapes can be obtained.

After constructing the three-dimensional deformable model M, step B can be performed to receive an input image, find a human body part to mark the acupuncture point, and mark the input image on the input image according to the characteristics of the human body part on the input image. Two marks. Wherein, the human body part may be a part of the face, the hand, the leg, the sole of the foot, etc., but not limited thereto. For example, as shown in FIG. 4 and FIG. 5, after receiving the input image I, the image processing module 40 is configured to perform image detection on the input image I, and according to the characteristics of the human body part on the input image I. A plurality of second marks L2 are indicated on the input image I. In the embodiment, the input image I is a face image, and the second marks L2 are arranged according to the feature points on the image such as the corners of the eyes, the corners of the mouth, the bridge of the nose, and the nose. .

In an embodiment, the method for finding a human body part may be to collect a large number of pictures of a human body part in advance, and obtain a feature of the human body part through a feature detection method, and obtain the characteristics, and then train through the classifier. Classified model. The feature detection method may be a Histogram of Oriented Gradient (HOG) feature, a Local Binary Pattern (LBP) feature, a Haar-like feature, a Convolutional Neural Network (CNN), or the like. One or a combination thereof, but not limited thereto; the classifier may be one of or a combination of a Support Vector Machine (SVM), a Relevant Vector Machine (RVM), or the like. , but not limited to this. For example, in an embodiment, a large number of facial images can be collected, and the HOG features are obtained to obtain features that can represent faces. After the features are acquired and aggregated, the classifier can be trained through the SVM.

Then, performing step C, fitting the three-dimensional deformable model according to the degree of deviation between the first marks of the three-dimensional deformable model and the coordinate information of the second marks of the input image, to adjust the The distribution of a marker approaches the distribution of the second markers. As shown in FIG. 6 , in an embodiment, in step C, the first mark L1 of the three-dimensional deformable model M is first projected to a two-dimensional plane parallel to the input image I, Adjusting the distribution of the first marks L1 to the distribution of the second marks L2 of the input image I on a two-dimensional plane; and storing the deformed three-dimensional deformable model M, wherein the three-dimensional deformable model M is stored The deformable model M may be a parameter for storing the corresponding deformation and/or each vertex number, coordinate position, and the like.

It is worth mentioning that, in an embodiment, since the input image provided by the user is not necessarily the image of the positive viewing angle, after receiving the input image I, the deformation model including the three-dimensional deformable model is included as an example. After the three-dimensional deformable model is aligned with the angle of the input image I, the first markers L1 are projected onto a two-dimensional plane parallel to the input image I, and the first ones are adjusted on the two-dimensional plane. The distribution of the mark L1 approaches the second mark L2. In an embodiment, regarding adjusting the distribution of the first marks L1 to approach the second marks L2, the parameters may be nested into the three-dimensional deformable model to obtain a new three-dimensional deformable model ( The new face model), and the feature phasor of the new three-dimensional deformable model that determines the first mark L1 is taken out and projected onto the two-dimensional plane and solved as a Singular Value Decomposition (SVD) to approximate Controlling the face parameter to determine whether the degree of deviation of the solution parameter is less than a predetermined threshold. For example, if the distance between the obtained parameter and the previous parameter is less than the predetermined threshold, the fitting of the three-dimensional deformable model is completed, otherwise, the fitting is completed. After adjusting the parameters of the three-dimensional deformable model, the previous step is performed again.

Among them, in order to find the acupoint points belonging to the deformed three-dimensional deformable model. Also included in step C is a first mark of the three-dimensional deformable model to be deformed and at least Projecting adjacent vertices of a point location point onto a two-dimensional plane, and projecting at least one acupoint position point of the initial three-dimensional deformable model and adjacent vertices of the at least one acupoint position point onto the two-dimensional plane; The adjacent vertex of the group is used as an image deformation control point, and the coordinates of the acupoint position point of the initial three-dimensional deformable model are adjusted to obtain a point position of the acupoint corresponding to the deformed three-dimensional deformable model. For example, please cooperate with FIG. 7A to project the initial three-dimensional deformable model to the first mark L11~L13 of the two-dimensional plane, the acupoint position point A, and the three vertices v adjacent to the acupoint position point A; 7B shows the first marks L11' to L12' of the deformed three-dimensional deformable model, the acupoint position point A', and the three vertices v' adjacent to the acupoint position point A'. The point A' of the acupoint position of the deformable model is the image deformation control point according to the adjacent vertices v and v' of the above two groups to adjust the acupoint position point A of the initial three-dimensional deformable model. The coordinates are obtained to obtain the acupoint position point A' of the three-dimensional deformable model corresponding to the deformation. Wherein the coordinate adjustment amount of the acupoint position point is based on a coordinate variation of at least three first marks and/or at least three vertices adjacent to the at least one acupoint position point, and according to the acupuncture point position, the distance is at least The distance between the three first markers and/or the at least three vertices is given a weight ratio, and the coordinates of the at least one acupoint location point are adjusted.

Then, in step D, the coordinates of the at least one acupoint position point are adjusted according to the coordinate change amount of the first marks of the three-dimensional deformable model to obtain corresponding at least one estimated acupoint position point. In order to find the position of the estimated acupoint point on the input image, the step D further includes: projecting the first mark of the deformed three-dimensional deformable model and the point of the acupoint point on the two-dimensional plane, and inputting the input The second marks of the image are used as control points, and the distribution of the first marks of the three-dimensional deformable model after the deformation is adjusted to approximate the distribution of the second marks, and the deformation is adjusted according to the deformation amounts of the first marks. A point of the acupuncture point is located to obtain the at least one estimated acupoint location point. For example, please refer to Figure 8, L11’, L12’, L13' is the first mark of the deformed three-dimensional deformable model, and L21, L22, and L23 are the second marks. As can be seen from the drawing, if the first marks are to be approximated to the second marks, adjustment is required. The first mark L13' is moved to the right, and then the position after the acupoint position point A is moved to the right is the estimated acupoint position point A1.

Then, step E is performed, the estimated acupoint position point A1 is attached to the input image I to form an output image O as shown in FIG. 2, and step F is displayed to display the output image O, thereby realizing acupoint visualization. Effect.

It is worth mentioning that the above-mentioned projection of the deformed three-dimensional deformable model to the two-dimensional plane is compared with the input image. The purpose of the input image may be changed, or the estimated three-dimensional deformability may be changed. There is an error in the angle between the rotation angle of the model and the input image. Therefore, in order to eliminate the aforementioned influence, the deformed three-dimensional deformable model is again compared with the input image to obtain the estimated acupoint position point.

The connection between each unit and the module described above generally refers to connection or connection through wired or wireless means, and is not limited to direct connection. Wherein, the aforementioned three-dimensional deformable model may also be marked with only a single acupoint position point, and is not limited to a plurality of acupoint position points. The above is only a preferred embodiment of the present invention, and equivalent changes to the scope of the present invention and the scope of the patent application are intended to be included in the scope of the present invention.

Claims (6)

A method for realizing acupoint visualization by using AR technology includes the following steps: A. providing a 3D Morphable Model based on a human body part, the three-dimensional deformable model including a plurality of features according to the body part a first mark and at least one acupoint position point; B, receiving an input image, and marking a plurality of second marks on the input image according to a human body part feature on the input image; C, according to the three-dimensional deformable model And comparing the degree of deviation of the first mark from the coordinate information of the second marks of the input image, and fitting the three-dimensional deformable model to adjust a distribution of the first marks to approximate a distribution of the second marks; D. Adjusting coordinates of the at least one acupoint position point according to the coordinate change amount of the first markers of the three-dimensional deformable model to obtain corresponding at least one estimated acupoint position point; E, adding the at least one estimated acupoint Positioning on the input image to form an output image; and F, displaying the output image; wherein the input image is a two-dimensional image; And first projecting the first marks of the three-dimensional deformable model to a two-dimensional plane parallel to the input image, so as to adjust the distribution of the first marks on the two-dimensional plane to approximate the distribution of the second marks; Storing the deformed three-dimensional deformable model; wherein the three-dimensional deformable model includes a plurality of vertices having the first marks, the at least one acupoint position point being located between the vertices; in step C The method includes: projecting a first mark of the deformed three-dimensional deformable model and adjacent vertices of at least one of the acupoint position points onto a two-dimensional plane, and at least one acupoint position point of the initial three-dimensional deformable model and the Projecting adjacent vertices of at least one acupoint location point onto the two-dimensional plane; The adjacent vertices of the above two groups are used as image deformation control points, and the coordinates of the acupoint position points of the initial three-dimensional deformable model are adjusted to obtain the acupoint position points of the three-dimensional deformable model corresponding to the deformation. The method for implementing acupoint visualization in the AR technology according to claim 1, comprising: projecting the first mark of the deformed three-dimensional deformable model and the point of the acupoint point on the two-dimensional plane, and using the input image The second mark is used as a control point, and the distribution of the first mark of the three-dimensional deformable model after the deformation is adjusted to approximate the distribution of the second marks, and the coordinates of the point position of the acupuncture point are adjusted according to the deformation amount of the first marks. To obtain the at least one estimated acupoint location point. The method for implementing acupoint visualization in the AR technology according to claim 1, wherein the step C includes: rotating the three-dimensional deformable model, and the three-dimensional deformable model is consistent with the input image angle, and then A labeled distribution is projected onto a two-dimensional plane. The method for implementing acupoint visualization according to the AR technology according to claim 1, wherein before the fitting in step C, the method includes: comparing whether the degree of deviation is greater than a predetermined threshold, and if greater than the predetermined threshold, re-adjusting the After the parameters of the three-dimensional deformable model, step C is performed again. The method for implementing acupuncture visualization by AR technology according to claim 1, wherein in step C, according to a coordinate variation of at least three first markers adjacent to the at least one acupoint location point, and according to the at least one acupoint location The points are respectively spaced from the distance of the at least three first marks to give a weight ratio, and the coordinates of the at least one acupoint position point are adjusted. The method for implementing acupoint visualization in AR technology according to claim 1, wherein the three-dimensional deformable model includes a plurality of vertices, wherein the vertices have the first markers, and the at least one acupoint location points are located at the vertices In step C, according to the coordinate variation of at least three vertices adjacent to the at least one acupoint location point, and according to the at least one acupoint location point The distances from the at least three vertices are respectively given a weight ratio, and the coordinates of the at least one acupoint position point are adjusted.