WO2023048153A1

WO2023048153A1 - Information processing method, computer program, and information processing device

Info

Publication number: WO2023048153A1
Application number: PCT/JP2022/035045
Authority: WO
Inventors: 俊彦西村; 康之本間; 雄太吉田
Original assignee: テルモ株式会社
Priority date: 2021-09-24
Filing date: 2022-09-21
Publication date: 2023-03-30

Abstract

Provided are an information processing method, a computer program, and an information processing device which can be expected to suppress a drop in the determination accuracy of a health condition or the like due to a shadow included in an image which captures a face.　By the information processing method according to this embodiment, an information processing device acquires a face image, extracts facial feature points from the acquired face image, determines whether there is a shadow on the face image, and corrects the facial feature points of the shadowed portion if a determination was made that there is a shadow. The information processing device determines whether a correction is necessary according to the position of the shadowed portion with respect to the face, and if a correction is determined to be necessary, may correct the facial feature points of the shadowed portion.

Description

Information processing method, computer program and information processing apparatus

The present invention relates to an information processing method, a computer program, and an information processing apparatus for handling an image of a person's face taken with a camera or the like.

Devices equipped with cameras, such as smartphones or tablet terminals, are widely used. In recent years, research and development have been made on techniques for determining the health condition of a person based on an image of the person captured by a camera.

In Patent Literature 1, an image of a user captured by a user terminal is transmitted to a server, and the server calculates a health level indicating the health condition of the user based on the face image included in the image and transmits the health level to the user terminal. , a health condition determination system in which a user terminal outputs a health level has been proposed. In this health condition determination system, the server creates graphs showing the health level, skin condition, and facial beauty rate in chronological order, correlation images showing the correlation with the health level, advice information, facial image videos, etc. , to the user terminal.

Japanese Patent Application Laid-Open No. 2020-52505

With technology that determines the health condition, etc., based on an image of the user's face, there is a risk that the surrounding environment at the time of shooting will affect the determination accuracy. For example, shadows may appear on the user's face due to external light, indoor lighting, or the like, and the shadows included in the photographed image of the face may reduce the accuracy of determining the health condition or the like.

The present invention has been made in view of such circumstances, and its object is to provide information that can be expected to suppress deterioration in the accuracy of determining health conditions, etc. due to shadows contained in images of faces. An object of the present invention is to provide a processing method, a computer program, and an information processing apparatus.

An information processing method according to one embodiment includes an information processing apparatus that acquires a facial image, extracts facial feature points from the acquired facial image, determines whether or not a shadow is present in the facial image, and extracts a shadow. When it is determined that , the facial feature points in the shaded portion are corrected.

According to one embodiment, it can be expected to suppress deterioration in the accuracy of determination of the health condition, etc., due to shadows contained in the photographed image of the face.

1 is a schematic diagram for explaining an overview of an information processing system according to an embodiment; FIG. 1 is a block diagram showing the configuration of a server device according to an embodiment; FIG. 2 is a block diagram showing the configuration of a terminal device according to this embodiment; FIG. 6 is a flowchart showing the procedure of health level determination processing performed by the server device according to the present embodiment; 6 is a flow chart showing a procedure of shadow correction processing relating to the outline of a face, which is performed by the server device according to the present embodiment; FIG. 10 is a schematic diagram for explaining shadow correction processing relating to the contour of the face; 7 is a flowchart showing a procedure of shadow correction processing relating to nasolabial folds or a nose performed by the server device according to the present embodiment; FIG. 10 is a schematic diagram for explaining shadow correction processing for nasolabial folds or a nose; FIG. 10 is a schematic diagram for explaining shadow correction processing for nasolabial folds or a nose; 7 is a flow chart showing a procedure of shadow correction processing for the eyes or mouth performed by the server device according to the present embodiment; FIG. 10 is a schematic diagram for explaining shadow correction processing for eyes or mouth; FIG. 10 is a schematic diagram for explaining shadow correction processing for eyes or mouth;

A specific example of the information processing system according to the embodiment of the present invention will be described below with reference to the drawings. The present invention is not limited to these exemplifications, but is indicated by the scope of the claims, and is intended to include all modifications within the meaning and scope of equivalents to the scope of the claims.

<System configuration>
FIG. 1 is a schematic diagram for explaining an outline of an information processing system according to this embodiment. The information processing system according to the present embodiment is a system that analyzes a subject's face image to determine the health condition, and includes a server device 1, a terminal device 3, and the like. A user who uses this system takes a picture of a person's face using a terminal device 3 such as a smart phone or a tablet-type terminal device, and transmits the taken face image from the terminal device 3 to the server device 1 . In this figure, the user uses the terminal device 3 to photograph his/her own face, but the present invention is not limited to this. good.

The server device 1 receives the face image transmitted by the terminal device 3, analyzes the features of the face image, determines the health level of the subject, and transmits the determined health level and information related thereto to the terminal device 3. do. Based on the face image, the server device 1 determines, for example, the subject's fatigue level, stress level, positive (negative) emotion, presence and degree of facial paralysis, presence and degree of signs of stroke, and the like. can be determined as Note that the degree of health determined by the server device 1 is not limited to the above example, and various indices related to the health of the subject may be employed.

The server device 1 extracts facial feature points (so-called keypoints, landmarks, etc.) from the facial image using, for example, a facial feature point extraction model machine-learned in advance, and based on the extracted facial feature points, the degree of health is calculated. judgment is made. In the information processing system according to the present embodiment, the server device 1 is configured to suppress the deterioration of the health level determination accuracy due to shadows that occur in the face image captured by the user using the terminal device 3. Processing is performed to correct the extraction result of the facial feature points for the shaded portion. The server device 1 determines, for example, whether or not a shadowed portion is included in the facial image, and if the shadowed portion is included, identifies the position and range of the shadowed portion, and identifies facial feature points included in the shadowed portion. The correction suppresses deterioration in the accuracy of determination of the health level based on the facial feature points.

<Device configuration>
FIG. 2 is a block diagram showing the configuration of the server device 1 according to this embodiment. The server device 1 according to the present embodiment includes a processing unit 11, a storage unit (storage) 12, a communication unit (transceiver) 13, and the like. In this embodiment, the explanation is given assuming that the processing is performed by one server device, but the processing may be performed by a plurality of server devices in a distributed manner.

The processing unit 11 includes an arithmetic processing unit such as a CPU (Central Processing Unit), MPU (Micro-Processing Unit), GPU (Graphics Processing Unit) or quantum processor, ROM (Read Only Memory), RAM (Random Access Memory), etc. It is configured using The processing unit 11 reads out and executes the server program 12a stored in the storage unit 12 to perform processing of acquiring a facial image of the subject from the terminal device 3, processing of extracting facial feature points from the acquired facial image, Various processes such as a process of correcting the facial feature points for the shaded portion and a process of judging the health level of the subject from the facial feature points are performed.

The storage unit 12 is configured using a large-capacity storage device such as a hard disk. The storage unit 12 stores various programs executed by the processing unit 11 and various data required for processing by the processing unit 11 . In the present embodiment, the storage unit 12 stores a server program 12a executed by the processing unit 11. FIG. The storage unit 12 stores a facial feature point extraction model 12b for extracting facial feature points from a facial image, and is provided with a facial feature point DB (database) 12c for storing information on the extracted facial feature points.

In the present embodiment, the server program (program product) 12a is provided in a form recorded in a recording medium 99 such as a memory card or an optical disk, and the server device 1 reads the server program 12a from the recording medium 99 and stores it in the storage unit 12. Remember. However, the server program 12a may be written in the storage unit 12 during the manufacturing stage of the server device 1, for example. Further, for example, the server program 12a may be delivered by another remote server device or the like, and the server device 1 may acquire the program through communication. For example, the server program 12 a may be recorded in the recording medium 99 and read by a writing device and written in the storage unit 12 of the server device 1 . The server program 12 a may be provided in the form of distribution via a network, or may be provided in the form of being recorded on the recording medium 99 .

The facial feature point extraction model 12b is a learning model that has undergone machine learning in advance so as to extract and output the facial feature points of the target person in response to the input of the target person's face image. The facial feature point extraction model 12b is a trained model that extracts facial feature points using, for example, Open Pose technology. The facial feature point extraction model 12b is not limited to the Open Pose learning model, and various learning models for extracting facial feature points using other techniques may be employed. Extraction of facial feature points using a learning model based on machine learning is an existing technique, and thus details such as the configuration and generation method of the learning model will be omitted.

The facial feature point DB 12c is a database that stores and accumulates information on facial feature points extracted from the facial image of the subject. The facial feature point DB 12c includes, for example, identification information such as the name or ID of the subject, the facial image of the subject, the date and time when the facial image was taken, information on the facial feature points extracted from the facial image, and the facial feature points. is stored in association with the determination result of the degree of health based on The server device 1 compares, for example, the facial feature points extracted from the latest facial image with facial feature points from a predetermined period of time, such as one month or one year ago, and based on changes in the facial feature points during the predetermined period. can determine the degree of health.

The communication unit 13 communicates with various devices via a network N including a mobile phone communication network, a wireless LAN (Local Area Network), the Internet, and the like. In the present embodiment, the communication unit 13 communicates with one or more terminal devices 3 via the network N. FIG. The communication unit 13 transmits the data given from the processing unit 11 to other devices, and gives the data received from the other devices to the processing unit 11 .

The storage unit 12 may be an external storage device connected to the server device 1. The server device 1 may be a multicomputer including a plurality of computers, or may be a virtual machine virtually constructed by software. The server device 1 is not limited to the above configuration, and may include, for example, a reading unit that reads information stored in a portable storage medium, an input unit that receives operation inputs, or a display unit that displays images. .

In the server device 1 according to the present embodiment, the server program 12a stored in the storage unit 12 is read out and executed by the processing unit 11, so that the facial image acquisition unit 11a, the facial feature point extraction unit 11b, the shadow determination unit A portion 11c, a shadow portion specifying portion 11d, a shadow correction portion 11e, a health level determination portion 11f, a notification portion 11g, and the like are implemented in the processing portion 11 as software functional portions. In this figure, as functional units of the processing unit 11, functional units related to determination of the health level from the face image are illustrated, and functional units related to other processes are omitted.

The face image acquisition unit 11a performs processing for acquiring a face image of the target person's face from the terminal device 3. The facial image acquisition unit 11 a communicates with the terminal device 3 through the communication unit 13 , receives facial image data transmitted from the terminal device 3 , and stores the data in the storage unit 12 . Further, when the image obtained from the terminal device 3 is, for example, a photograph of the entire body of the subject, the face image obtaining section 11a may perform a process of extracting an image region corresponding to the face of the subject from the image. . In this case, the face image acquisition unit 11a performs face detection processing on the image acquired from the terminal device 3, and cuts out an image area including the detected face from the original image, thereby extracting the face image of the subject. can be obtained. Since face detection processing is an existing technology, detailed description thereof will be omitted.

The facial feature point extraction unit 11b extracts a plurality of points indicating facial features as feature points (keypoints, landmarks, etc.) from the facial image acquired by the facial image acquisition unit 11a. In the present embodiment, the facial feature point extraction unit 11b uses the facial feature point extraction model 12b stored in the storage unit 12 to extract facial feature points from the facial image. The facial feature point extracting unit 11b inputs the facial image of the subject to the facial feature point extraction model 12b, and obtains the information of the facial feature points output by the facial feature point extraction model 12b in response to the facial image. Extract a plurality of facial feature points from The facial feature point extraction model 12b outputs, for example, the coordinates of the facial feature points in the input facial image, and the number of facial feature points output by the facial feature point extraction model 12b is, for example, several to several hundred.

The shadow determination unit 11c performs processing for determining whether or not the face image acquired by the face image acquisition unit 11a contains a shadow. For example, the shadow determination unit 11c compares the luminance (or brightness, etc.) of each pixel forming the face image with a predetermined threshold value, and determines whether or not there is a pixel whose luminance is lower (darker) than the threshold value. determines whether or not the face image includes a shadow. The threshold used for shadow determination may be determined, for example, at the design stage of the present system, or may be calculated, for example, based on the average value of luminance of the entire face image.

The shadow portion identification unit 11d performs a process of identifying an image area containing a shadow present in the face image as a shadow portion. Similar to the shadow determination unit 11c, the shadow portion identification unit 11d compares the brightness of each pixel of the face image with a predetermined threshold, and identifies pixels whose brightness is lower than the threshold. The threshold used for this process may be the same as the threshold used by the shadow determination unit 11c for determination. may be used. The shaded portion specifying unit 11d specifies a rectangular image region including pixels whose brightness is smaller than a threshold value, and specifies this image region as a shaded portion. Note that the shadow portion identifying section 11d may identify a plurality of shadow portions from the face image.

In addition, the shadow part specifying unit 11d specifies which part of the subject's face corresponds to the shadow part included in the face image. The shadow portion specifying unit 11d compares, for example, the coordinates of the facial feature points extracted by the facial feature point extraction unit 11b with the range of the coordinates of the shadow portion, and determines whether the facial feature points existing within or near the shadow portion are identified as the face. It is determined which part of the face the shaded part corresponds to by determining which feature of the point. In the present embodiment, the shaded portion specifying unit 11d specifies whether the shaded portion corresponds to any part of the contour of the face, the nasolabial fold, the nose, the eyes, or the mouth, or does not correspond to any part. .

The shadow correction unit 11e performs processing for correcting facial feature points included in the shadow portion specified by the shadow portion specifying unit 11d. In the present embodiment, the shadow correction section 11e corrects the shadow portion using a correction method determined according to which part of the face the shadow portion corresponds to. In the present embodiment, a method for correcting a shadow portion corresponding to the contour of the face, a method for correcting a shadow portion corresponding to nasolabial folds or a nose, and a method for correcting a shadow portion corresponding to the eyes or mouth are described. The shadow correction section 11e selects an appropriate method according to the shadow portion from the three types of correction methods and performs correction. Details of each correction method will be described later.

The health level determination unit 11f performs processing for determining the health level of the subject based on a plurality of facial feature points extracted from the face image and corrected for shadow portions. The health degree determination unit 11f according to the present embodiment determines, for example, the presence or absence or degree of facial paralysis of the subject. A designer or the like of an information processing system according to the present embodiment collects information about facial feature points of a person with symptoms of facial paralysis, predetermines conditions for determining whether or not the person has facial paralysis, The determination conditions are stored in the device 1 . The health degree determination unit 11f determines whether or not the facial feature points of the subject have the characteristics of facial paralysis based on pre-stored determination conditions, and outputs the presence or absence of facial paralysis or the degree thereof as a determination result. do. The health level determination unit 11f outputs, for example, a decimal value ranging from 0 (no facial paralysis characteristic) to 1 (facial paralysis characteristic) as a determination result.

The information processing system according to the present embodiment is intended to accurately extract facial feature points that can be used to determine various degrees of health. It may be used for determining the degree of health. For this reason, in the present embodiment, a detailed description of the method of determining the degree of health, such as the method of determining the presence or absence of facial paralysis and the degree of facial paralysis by the health degree determination unit 11f, will be omitted.

The health degree determination unit 11f may determine the presence or absence or degree of facial paralysis from the subject's facial feature points using a machine-learned learning model in advance. The learning model performs machine learning using, for example, teacher data in which information on facial feature points and information on the presence or absence of facial paralysis are associated, and outputs the presence or absence of facial paralysis or its degree in response to the input of facial feature points. do.

In addition, the health level determination unit 11f may determine the degree of progression or improvement of, for example, the symptom of facial paralysis based on the facial feature point information stored in the facial feature point DB 12c of the storage unit 12. Further, the health degree determination unit 11f predicts the degree of facial paralysis at a future point in time, such as one month or one year ahead, based on the time-series facial feature point information stored in the facial feature point DB 12c. good too. In the present embodiment, the health degree determination unit 11f determines the presence or absence of facial paralysis of the subject or the degree thereof, but is not limited to this. The presence or absence of cerebral apoplexy or the like may be determined as the degree of health.

The notification unit 11g performs a process of notifying the user of the result of health level determination by the health level determination unit 11f. The notification unit 11g may notify, for example, when an abnormality is detected in the subject's health level, or may notify, for example, when an improvement in the subject's health level is detected. The judgment result may be notified regardless of the presence or absence of improvement. In the present embodiment, for example, when the health level determination unit 11f determines that the target person has a symptom of facial paralysis, the notification unit 11g notifies the user of that effect. The notification unit 11g transmits a message for notifying the determination result to one or a plurality of notification destinations preset in association with the target person. In the present embodiment, the notification unit 11g notifies the terminal device 3 of the user who has transmitted the face image. However, the notification destination is not limited to the terminal device 3 of the user who transmitted the face image, and may be the terminal device 3 of a different user (for example, the user's family member or doctor in charge). good.

FIG. 3 is a block diagram showing the configuration of the terminal device 3 according to this embodiment. The terminal device 3 according to this embodiment includes a processing unit 31, a storage unit (storage) 32, a communication unit (transceiver) 33, a display unit (display) 34, an operation unit 35, and the like. The terminal device 3 is a device used by a user such as a target person who wants to determine the health level, or a related person such as a family member or a doctor in charge of the target person, for example, a smartphone, a tablet terminal device, a personal computer, etc. information processing apparatus. The terminal device 3 does not have to be a portable device, and may be a device such as an AI speaker or a surveillance camera installed in the target person's house. It is desirable that the terminal device 3 is equipped with a camera for photographing the face of the subject. A device without a camera may be used.

The processing unit 31 is configured using an arithmetic processing unit such as a CPU or MPU, a ROM, and the like. The processing unit 31 reads out and executes the program 32a stored in the storage unit 32 to perform processing related to photographing of the subject, processing of transmitting the photographed image to the server device 1 and requesting determination of the degree of health, and , acquires the determination result of the health level from the server device 1, and performs various processes such as a process of displaying it to the user.

The storage unit 32 is configured using, for example, a non-volatile memory device such as a flash memory or a storage device such as a hard disk. The storage unit 32 stores various programs executed by the processing unit 31 and various data required for processing by the processing unit 31 . In the present embodiment, the storage unit 32 stores a program 32a executed by the processing unit 31. FIG. In the present embodiment, the program (program product) 32 a is distributed by a remote server device or the like, and the terminal device 3 acquires it through communication and stores it in the storage unit 32 . However, the program 32a may be written in the storage unit 32 during the manufacturing stage of the terminal device 3, for example. For example, the program 32a may be stored in the storage unit 32 after the terminal device 3 reads the program 32a recorded in the recording medium 98 such as a memory card or an optical disk. For example, the program 32 a may be recorded in the recording medium 98 and read by a writing device and written in the storage unit 32 of the terminal device 3 . The program 32 a may be provided in the form of distribution via a network, or may be provided in the form of being recorded on the recording medium 98 .

The communication unit 33 communicates with various devices via a network N including a mobile phone communication network, a wireless LAN, the Internet, and the like. In the present embodiment, the communication unit 33 communicates with the server device 1 via the network N. FIG. The communication unit 33 transmits data received from the processing unit 31 to other devices, and provides the processing unit 31 with data received from other devices.

The display unit 34 is configured using a liquid crystal display or the like, and displays various images, characters, etc. based on the processing of the processing unit 31. The operation unit 35 receives a user's operation and notifies the processing unit 31 of the received operation. For example, the operation unit 35 receives a user's operation using an input device such as mechanical buttons or a touch panel provided on the surface of the display unit 34 . Further, for example, the operation unit 35 may be an input device such as a mouse and a keyboard, and these input devices may be detachable from the terminal device 3 . The camera 36 is configured using an image sensor such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor), and an optical element such as a lens. Give to 31.

In the terminal device 3 according to the present embodiment, the program 32a stored in the storage unit 32 is read out and executed by the processing unit 31, so that the photographing processing unit 31a, the display processing unit 31b, etc. function as software functional units. It is implemented in the processing unit 31 . The program 32a may be a program dedicated to the information processing system according to the present embodiment, or may be a general-purpose program such as an Internet browser or web browser.

The photographing processing unit 31a performs processing related to photographing of the subject's face image by the camera 36. The photographing processing unit 31a performs photographing by the camera 36 according to, for example, the user's operation on the operation unit 35, and acquires data of the photographed image. At this time, the photographing processing unit 31a may perform, for example, a message display or a photographing guide display to assist the user in photographing an appropriate face image. The imaging processing unit 31a transmits an image (face image) obtained by imaging to the server device 1, and requests determination of the health level.

The display processing unit 31b performs processing for displaying information on the health level determination result received from the server device 1 on the display unit 34. The display processing unit 31b may display, for example, the presence or absence of facial paralysis determined by the server device 1 or its degree, and, for example, when a determination result indicating that there is a characteristic of facial paralysis is obtained, a push notification to that effect is obtained. For example, the time-series change in the degree of facial paralysis may be displayed as a graph or the like, or various displays other than these may be performed.

<Shadow Correction Processing>
FIG. 4 is a flow chart showing the procedure of health level determination processing performed by the server device 1 according to the present embodiment. The facial image acquisition unit 11a of the processing unit 11 of the server device 1 according to the present embodiment communicates with the terminal device 3 through the communication unit 13, and receives facial image data transmitted by the terminal device 3. , a face image of the subject is obtained (step S1). The facial feature point extraction unit 11b of the processing unit 11 inputs the facial image acquired in step S1 to the facial feature point extraction model 12b stored in the storage unit 12, and extracts the face image output by the facial feature point extraction model 12b. By obtaining information on the feature points, the feature points are extracted from the face image of the subject (step S2).

The shadow determination unit 11c of the processing unit 11 determines whether or not the face image acquired in step S1 includes a shadow (step S3). At this time, the shadow determination unit 11c determines whether or not there is a pixel having a brightness lower than the threshold based on the result of comparing the brightness of each pixel constituting the face image with the threshold, thereby determining whether or not there is a shadow on the face image. Determine whether it contains If it is determined that the face image does not contain a shadow (S3: NO), the shadow determining section 11c advances the process to step S7.

When it is determined that the face image includes a shadow (S3: YES), the shadow portion specifying unit 11d of the processing unit 11 specifies an image region including the shadow in the face image as a shadow portion, and identifies the shadow portion. It is specified which part of the subject's face the part corresponds to (step S4). At this time, the shadow portion specifying unit 11d determines which portion of the face the shadow portion corresponds to based on the positional relationship between the region specified as the shadow portion and the coordinates of the facial feature points extracted in step S2. to identify

The shadow portion specifying unit 11d determines whether or not the shadow portion specified in step S4 needs to be corrected (step S5). In the present embodiment, the shadow portion specifying unit 11d determines that correction is necessary when the shadow portion of the face image corresponds to any part of the subject's facial contour, nasolabial fold, nose, eyes, or mouth. If it does not correspond to these parts, it is determined that there is no need for correction. If it is determined that correction is not necessary (S5: NO), the shadow portion specifying section 11d advances the process to step S7.

If it is determined that correction is necessary (S5: YES), the shadow correction section 11e of the processing section 11 performs correction processing on the shadow portion of the face image identified by the shadow portion identification section 11d (step S6). At this time, the shadow corrector 11e is determined according to whether the shadowed portion is a portion corresponding to the outline of the face, a portion corresponding to the nasolabial fold or nose, or a portion corresponding to the eyes or mouth. Use the correction method to correct the shaded area.

The health level determination unit 11f of the processing unit 11 determines the health level of the subject based on the facial feature points extracted from the facial image of the subject and the facial feature points corrected with respect to the shaded portion as necessary. (step S7). In the present embodiment, the health degree determination unit 11f determines the presence or absence of facial paralysis and its degree based on the facial feature points of the subject. Various health measures such as a person's degree of fatigue, stress level, degree of positive (negative) emotions, or presence and degree of signs of stroke may be determined. The health level determination unit 11f transmits information on the facial feature points extracted from the face image of the subject and information on the facial feature points corrected with respect to the shaded portion as necessary, together with the health level determination result in step S7. Stored in the facial feature point DB 12c of the storage unit 12 (step S8).

The notification unit 11g of the processing unit 11 notifies the determination result of the health level by transmitting the information about the health level of the target person determined in step S7 to the destination preset for the target person (step S9), the process ends. For example, the server device 1 stores, in a database or the like, identification information such as the subject's name or ID, and information such as the e-mail address of the recipient of the determination result of the health level in association with each other. When transmitting the face image of the target person to the server device 1 , the terminal device 3 transmits the identification information of the target person to the server device 1 together with the face image. The notification unit 11g of the server device 1 acquires the information of the transmission destination to which the health level determination result is transmitted from the database based on the subject identification information received together with the face image, and sends the acquired transmission destination for the health level determination. You can send the results.

(1) Shadow Correction Processing Concerning Face Contour FIG. 5 is a flowchart showing the procedure of shadow correction processing concerning the contour of the face performed by the server device 1 according to the present embodiment. The shadow correction processing shown in FIG. 5 is processing that can be performed in step S6 of the flowchart of FIG. Also, FIG. 6 is a schematic diagram for explaining the shadow correction processing regarding the contour of the face.

The shadow correction unit 11e of the processing unit 11 of the server device 1 according to the present embodiment performs correction to increase the brightness of the shadow portion of the face image identified in step S4 of the flowchart shown in FIG. S21). The upper part of FIG. 6 shows an example of a face image and a shaded portion 101 corresponding to the contour of the face specified from this face image. The middle part of FIG. 6 shows an example of the image after the correction for increasing the luminance of the shaded portion 101 has been performed. The shadow correction unit 11e performs luminance correction only on the shadow portion specified from the face image, and does not have to perform luminance correction on portions other than the shadow portion (although it may be performed).

The shadow correction unit 11e examines, for example, the color distribution of the image of the shaded portion with the increased luminance, and identifies image regions with similar colors to determine the region corresponding to human skin (skin region) and the other regions. (background area) and (step S22). The lower part of FIG. 6 shows an example in which the shaded area is separated into a skin area 101a and a background area 101b. In the present embodiment, the shadow correction unit 11e separates the skin region and the background region by specifying regions with similar colors, but the region separation method is not limited to this. Another method may be used to separate regions, such as detecting a boundary line from a portion by processing such as edge extraction and separating the region based on this boundary line.

The shadow correction unit 11e acquires a portion corresponding to the boundary line between the skin area and the background area based on the separation result of the skin area and the background area in step S22 (step S23). The shadow correction unit 11e acquires some points as feature points from among the acquired plurality of points on the boundary line (step S24), and ends the shadow correction process. The shadow correction unit 11e replaces the feature points included in the shadow portion among the feature points extracted in step S2 of the flowchart of FIG. 4 with the feature points acquired in step S24, thereby completing the shadow correction processing. .

(2) Shadow Correction Processing for Nasolabial Lines or Nose FIG. 7 is a flowchart showing a procedure for shadow correction processing for nasolabial folds or nose performed by the server device 1 according to the present embodiment. The shadow correction processing shown in FIG. 7 is processing that can be performed in step S6 of the flowchart of FIG. 8 and 9 are schematic diagrams for explaining the shadow correction processing for nasolabial folds or nose.

The shadow correction unit 11e of the processing unit 11 of the server device 1 according to the present embodiment performs correction to increase the brightness of the shadow portion of the face image identified in step S4 of the flowchart shown in FIG. S31). The upper part of FIG. 8 shows an example of a face image and a shadow portion 102 corresponding to the nasolabial folds of the face identified from this face image. Further, the lower part of FIG. 8 shows an example of the image after the correction for increasing the luminance of the shaded portion 101 is performed. The shadow correction unit 11e performs luminance correction only on the shadow portion specified from the face image, and does not have to perform luminance correction on portions other than the shadow portion (although it may be performed).

The shadow correction unit 11e compares the brightness of each pixel in the shaded portion of the image of the shaded portion with the increased brightness with a predetermined threshold value, thereby extracting regions where the brightness is lower than the threshold value (step S32). The upper part of FIG. 9 shows an example of the shadow portion 102 identified from the face image, and the lower part of FIG. there is In addition, in the lower graph of FIG. 9, a region where the brightness is lower than the threshold value is shown hatched, and this region can be estimated as a portion corresponding to the nasolabial folds of the face. In FIG. 9, the nasolabial folds are taken as an example, but the same applies to the nose, and regions where the brightness is lower than the threshold value correspond to the boundaries of the nose or concave portions in the unevenness of the nose.

The shadow correction unit 11e acquires some points as feature points from among the points included in the low-luminance region extracted in step S32 (step S33), and ends the shadow correction process. For example, the shadow correction unit 11e replaces the feature points included in the shadow portion among the feature points extracted in step S2 of the flowchart of FIG. 4 with the feature points acquired in step S33, thereby completing the shadow correction processing. .

(3) Shadow Correction Processing for Eyes or Mouth FIG. 10 is a flowchart showing a procedure for shadow correction processing for eyes or mouth performed by the server device 1 according to the present embodiment. The shadow correction processing shown in FIG. 10 is processing that can be performed in step S6 of the flowchart of FIG. 11 and 12 are schematic diagrams for explaining the shadow correction processing for the eyes or mouth.

The shadow correction unit 11e of the processing unit 11 of the server device 1 according to the present embodiment performs correction to increase the brightness of the shadow portion of the face image identified in step S4 of the flowchart shown in FIG. S41). The upper part of FIG. 11 shows an example of a face image and a shadow portion 103 corresponding to the eyes identified from this face image. Further, the lower part of FIG. 11 shows an example of the image after the correction for increasing the brightness of the shaded portion 103 is performed. The shadow correction unit 11e performs luminance correction only on the shadow portion specified from the face image, and does not have to perform luminance correction on portions other than the shadow portion (although it may be performed).

The shadow correction unit 11e performs processing for extracting a closed curve surrounding the eyes or mouth from the image of the shadow portion with the increased brightness (step S42). At this time, the shadow correction unit 11e extracts, for example, pixels corresponding to edges from the image, extracts a portion where the pixels of the edge are connected to form a curve, and extracts a portion where the curve is closed. can be extracted. Extraction of closed curves surrounding the eyes or mouth can be done using active contour methods such as Snakes or Level Set. Since the active contour method is an existing technique, detailed description thereof will be omitted. Note that this method of extracting a closed curve is only an example and is not limited to this, and extraction of a closed curve by the shadow correction unit 11e may be performed by any method.

The shadow correction unit 11e performs processing for detecting corners of the closed curve extracted in step S42 (step S43). At this time, the shadow correction unit 11e detects a crooked portion of the closed curve, and if the internal angle at this portion is smaller than a predetermined angle (eg, 90° or 60°), this portion is determined to be a corner portion. can be done. Note that this angle detection method is an example and is not limited to this, and the angle detection by the shadow correction unit 11e may be performed by any method. The upper part of FIG. 12 shows a closed curve 104 detected from a shaded portion 103 corresponding to the eye, and a corner portion 105 of this closed curve 104 . In the lower part of FIG. 12, a closed curve 104 detected from a shaded portion 103 corresponding to the mouth and a corner portion 105 of this closed curve 104 are shown.

The shadow correction unit 11e acquires the point corresponding to the corner detected in step S43 as a feature point (step S44), and ends the shadow correction process. The shadow correction unit 11e completes the shadow correction processing by, for example, replacing the feature points included in the shadow portion among the feature points extracted in step S2 of the flowchart of FIG. 4 with the feature points acquired in step S44. .

<Summary>
In the information processing system according to the present embodiment having the configuration described above, the server device 1 acquires a face image of a subject photographed from the terminal device 3, extracts facial feature points from the acquired face image, and extracts facial feature points from the face image. To determine whether or not a shadow occurs, and correct facial feature points in the shadow portion when the shadow occurs. As a result, in the information processing system according to the present embodiment, it is possible to suppress deterioration in accuracy in extracting facial feature points due to shadows included in a photographed image of the face, and to improve accuracy in determining health conditions and the like based on facial feature points. It can be expected that the decrease will be suppressed.

The server device 1 according to the present embodiment also determines whether or not correction is necessary according to which part of the face the shaded portion included in the face image corresponds to, that is, the position of the shaded portion with respect to the face. As a result, the server device 1 performs correction when a part included in the facial image corresponds to a part that affects the extraction of facial feature points, for example, and corrects a part that does not affect the extraction of facial feature points. By not performing correction, it is expected that the load of correction processing can be reduced.

Further, the server device 1 according to the present embodiment corrects facial feature points related to the outline of the face included in the shaded portion. At this time, the server device 1 separates the face portion (facial skin region) and the background portion (background region) included in the shaded portion, extracts the boundary line between the face portion and the background portion, and Correction is performed by using the points as facial feature points. As a result, the server apparatus 1 can be expected to accurately extract facial feature points even from a facial image in which shadows are produced on the outline of the face and its surroundings.

In addition, the server device 1 according to the present embodiment corrects feature points related to nasolabial folds or noses of the face included in the shaded portion. At this time, the server device 1 performs correction by extracting a portion in which the brightness of each pixel included in the shaded portion is smaller than a threshold, and using points included in the extracted portion as facial feature points. As a result, the server device 1 can be expected to extract facial feature points with high accuracy even in a face image in which the nasolabial folds or the nose and its surroundings are shaded.

In addition, the server device 1 according to the present embodiment corrects feature points related to the eyes or mouth of the face included in the shaded portion. At this time, the server device 1 extracts a closed curve surrounding the eyes or mouth included in the shaded portion, detects the corner of the extracted closed curve, and uses points on the detected corner as facial feature points for correction. conduct. As a result, the server device 1 can be expected to accurately extract facial feature points even from a facial image in which the eyes or mouth and their surroundings are shaded.

In addition, the server device 1 according to the present embodiment determines the health level of the subject depicted in the face image based on the feature points extracted from the face image and the feature points corrected as necessary. The degree of health to be determined includes, for example, the presence or absence and degree of facial paralysis of the subject, the degree of fatigue of the subject, the stress level, the degree of positive (negative) emotions, or the presence or absence and degree of signs of stroke. can be adopted. The server device 1 notifies, for example, the terminal device 3 of the determination result of the health level. Accordingly, the user can obtain information about the health level of the target person, such as himself or his family, by a simple operation of photographing and transmitting a face image.

In addition, the server device 1 according to the present embodiment stores in the storage unit 12 information on the facial feature points extracted from the facial image of the subject and information on the facial feature points corrected with respect to the shaded portion as necessary. do. The server device 1 compares, for example, past facial feature points stored in the storage unit 12 with the latest facial feature points extracted based on the facial image received from the terminal device 3, and determines whether or not there is a change in the facial feature points. The degree of health may be determined based on the degree of change or the like. As a result, the server apparatus 1 can be expected to perform more accurate health level determination based on the facial feature point extraction results obtained at a plurality of points in time.

In the present embodiment, the server device 1 determines the degree of health based on the face image of the subject photographed by the camera 36 or the like of the terminal device 3. The information obtained from the above may be used together to correct the shaded portion and determine the degree of health. For example, the terminal device 3 is equipped with a sensor such as a depth sensor to measure the surface shape of the subject's face, and information about the surface shape of the face measured by the sensor together with the face image of the subject photographed by the camera 36. may be transmitted to the server device 1 . When a face image includes a shadow portion, the server apparatus 1 can correct the facial feature points related to the shadow portion based on information on the surface shape of the face measured by the sensor.

Further, for example, the terminal device 3 is equipped with a sensor that detects infrared light, detects infrared light on the face of the subject, and detects the infrared light detected by the sensor together with the face image of the subject taken by the camera 36. information may be transmitted to the server device 1 . When there is a shadow portion in the face image, the server device 1 can correct the facial feature points related to the shadow portion based on the infrared light information detected by the sensor.

The embodiments disclosed this time are illustrative in all respects and should be considered not restrictive. The scope of the present invention is indicated by the scope of the claims rather than the meaning described above, and is intended to include all changes within the meaning and scope equivalent to the scope of the claims.

<Appendix>
(Appendix 1)
The information processing device
get face image
extracting facial feature points from the obtained facial image;
Determining whether or not there is a shadow in the face image,
correcting the facial feature points in the shaded portion when it is determined that the shadow occurs;
Information processing methods.
(Appendix 2)
Determining whether or not correction is necessary according to the position of the shaded portion with respect to the face,
correcting facial feature points in the shaded portion when it is determined that correction is necessary;
The information processing method according to appendix 1.
(Appendix 3)
correcting facial feature points related to the contour of the face included in the shaded portion;
The information processing method according to Supplementary Note 1 or Supplementary Note 2.
(Appendix 4)
Separating the face part and the background part included in the shadow part,
extracting a boundary line between the face portion and the background portion based on the separation result;
The point on the extracted boundary line is set as the facial feature point;
The information processing method according to appendix 3.
(Appendix 5)
correcting facial nasolabial folds or feature points related to the nose included in the shaded portion;
The information processing method according to any one of appendices 1 to 4.
(Appendix 6)
extracting a portion where the brightness of the shaded portion is smaller than a threshold;
The points included in the extracted portion are set as the facial feature points;
The information processing method according to appendix 5.
(Appendix 7)
correcting feature points related to the eyes or mouth of the face included in the shaded portion;
The information processing method according to any one of appendices 1 to 6.
(Appendix 8)
extracting a closed curve surrounding the eyes or mouth included in the shaded portion;
Detecting a corner portion of the extracted closed curve,
The point of the detected corner portion is set as the facial feature point;
The information processing method according to appendix 7.
(Appendix 9)
Determining the health level of the person depicted in the facial image based on the facial feature points;
The information processing method according to any one of appendices 1 to 8.
(Appendix 10)
Notifying the user of information about the determined health level;
The information processing method according to appendix 9.
(Appendix 11)
storing the facial feature points in a storage unit;
Determining the health level based on changes over time in a plurality of stored feature points;
The information processing method according to appendix 9 or appendix 10.
(Appendix 12)
to the computer,
get face image
extracting facial feature points from the obtained facial image;
Determining whether or not there is a shadow in the face image,
A computer program for executing processing for correcting facial feature points in a shadowed portion when it is determined that a shadow is present.
(Appendix 13)
an acquisition unit that acquires a face image;
an extraction unit that extracts facial feature points from the acquired facial image;
a determination unit that determines whether or not there is a shadow in the face image;
An information processing apparatus, comprising: a correction unit that corrects facial feature points in a shaded portion when it is determined that a shadow is present.

1 server device 3 terminal device 11 processing unit 11a facial image acquisition unit 11b facial feature point extraction unit 11c shadow determination unit 11d shadow part identification unit 11e shadow correction unit 11f health level determination unit 11g notification unit 12 storage unit 12a server program 12b facial features Point extraction model 12c Facial feature point DB
13 communication unit 31 processing unit 31a photographing processing unit 31b display processing unit 32 storage unit 32a program 33 communication unit 34 display unit 35 operation unit 36

camera

98, 99 recording medium N network

Claims

The information processing device
get face image
extracting facial feature points from the obtained facial image;
Determining whether or not there is a shadow in the face image,
correcting the facial feature points in the shaded portion when it is determined that the shadow occurs;
Information processing methods.
Determining whether or not correction is necessary according to the position of the shaded portion with respect to the face,
correcting facial feature points in the shaded portion when it is determined that correction is necessary;
The information processing method according to claim 1 .
correcting facial feature points related to the contour of the face included in the shaded portion;
The information processing method according to claim 1 .
Separating the face part and the background part included in the shadow part,
extracting a boundary line between the face portion and the background portion based on the separation result;
The point on the extracted boundary line is set as the facial feature point;
The information processing method according to claim 3.
correcting facial nasolabial folds or feature points related to the nose included in the shaded portion;
The information processing method according to claim 1 .
extracting a portion where the brightness of the shaded portion is smaller than a threshold;
The points included in the extracted portion are set as the facial feature points;
The information processing method according to claim 5.
correcting feature points related to the eyes or mouth of the face included in the shaded portion;
The information processing method according to claim 1 .
extracting a closed curve surrounding the eyes or mouth included in the shaded portion;
Detecting a corner portion of the extracted closed curve,
The point of the detected corner portion is set as the facial feature point;
The information processing method according to claim 7.
Determining the health level of the person depicted in the facial image based on the facial feature points;
The information processing method according to claim 1 .
Notifying the user of information about the determined health level;
The information processing method according to claim 9 .
storing the facial feature points in a storage unit;
Determining the health level based on changes over time in a plurality of stored feature points;
The information processing method according to claim 9 .
to the computer,
get face image
extracting facial feature points from the obtained facial image;
Determining whether or not there is a shadow in the face image,
A computer program for executing processing for correcting facial feature points in a shadowed portion when it is determined that a shadow is present.
an acquisition unit that acquires a face image;
an extraction unit that extracts facial feature points from the acquired facial image;
a determination unit that determines whether or not there is a shadow in the face image;
An information processing apparatus, comprising: a correction unit that corrects facial feature points in a shaded portion when it is determined that a shadow is present.