WO2021044249A1 - 情報処理装置 - Google Patents

情報処理装置 Download PDF

Info

Publication number
WO2021044249A1
WO2021044249A1 PCT/IB2020/057918 IB2020057918W WO2021044249A1 WO 2021044249 A1 WO2021044249 A1 WO 2021044249A1 IB 2020057918 W IB2020057918 W IB 2020057918W WO 2021044249 A1 WO2021044249 A1 WO 2021044249A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
information processing
pupil
size
processing device
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/IB2020/057918
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
小國哲平
岡野達也
秋元健吾
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Semiconductor Energy Laboratory Co Ltd
Original Assignee
Semiconductor Energy Laboratory Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Semiconductor Energy Laboratory Co Ltd filed Critical Semiconductor Energy Laboratory Co Ltd
Priority to JP2021543605A priority Critical patent/JP7548914B2/ja
Priority to US17/638,359 priority patent/US12333859B2/en
Publication of WO2021044249A1 publication Critical patent/WO2021044249A1/ja
Anticipated expiration legal-status Critical
Priority to JP2024150460A priority patent/JP7836865B2/ja
Priority to US19/215,713 priority patent/US20250329192A1/en
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/18—Eye characteristics, e.g. of the iris
    • G06V40/193—Preprocessing; Feature extraction
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/174—Facial expression recognition
    • G06V40/176—Dynamic expression
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B3/00—Apparatus for testing the eyes; Instruments for examining the eyes
    • A61B3/10—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions
    • A61B3/11—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions for measuring interpupillary distance or diameter of pupils
    • A61B3/112—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions for measuring interpupillary distance or diameter of pupils for measuring diameter of pupils
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/16—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state
    • A61B5/165—Evaluating the state of mind, e.g. depression, anxiety
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/044—Recurrent networks, e.g. Hopfield networks
    • G06N3/0442—Recurrent networks, e.g. Hopfield networks characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G06N3/0455—Auto-encoder networks; Encoder-decoder networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0475—Generative networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/09—Supervised learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/094—Adversarial learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/60—Analysis of geometric attributes
    • G06T7/62—Analysis of geometric attributes of area, perimeter, diameter or volume
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/103—Measuring devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
    • A61B5/11—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor or mobility of a limb
    • A61B5/1103—Detecting muscular movement of the eye, e.g. eyelid movement
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/16—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state
    • A61B5/168—Evaluating attention deficit, hyperactivity
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20048—Transform domain processing
    • G06T2207/20061—Hough transform
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20076—Probabilistic image processing
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30196—Human being; Person
    • G06T2207/30201—Face

Definitions

  • One aspect of the present invention relates to an information processing device. Further, one aspect of the present invention relates to an information processing system. Further, one aspect of the present invention relates to an information processing method. Further, one aspect of the present invention relates to an information terminal.
  • Patent Document 1 discloses an eye fatigue detection device and a detection method.
  • the pupil diameter changes depending on the presence or absence of fatigue, drowsiness, etc. For example, in the case of fatigue and drowsiness, the pupil diameter is smaller than in the case of no fatigue and drowsiness. In general, the pupil diameter fluctuates periodically, but in the case of fatigue and drowsiness, the fluctuation cycle of the pupil diameter becomes longer than in the case of no fatigue and drowsiness.
  • the operation of the information terminal can be changed according to the presence or absence of the user's fatigue, drowsiness, etc. Therefore, it is preferable.
  • the information terminal itself used has a function of estimating user fatigue, drowsiness, etc.
  • a dedicated device is required.
  • One aspect of the present invention is to provide an information processing device having a function of detecting user fatigue, drowsiness, etc. in real time. Alternatively, one aspect of the present invention is to provide an information processing device having a function of accurately estimating user fatigue, drowsiness, etc. Alternatively, one aspect of the present invention is to provide an information processing device having a function of estimating user fatigue, drowsiness, etc. by a simple method. Alternatively, one aspect of the present invention is to provide an information processing device having a function of estimating user fatigue, drowsiness, etc. in a short time.
  • One aspect of the present invention is to provide an information processing system having a function of detecting user fatigue, drowsiness, etc. in real time. Alternatively, one aspect of the present invention is to provide an information processing system having a function of accurately estimating user fatigue, drowsiness, etc. Alternatively, one aspect of the present invention is to provide an information processing system having a function of estimating user fatigue, drowsiness, etc. by a simple method. Alternatively, one aspect of the present invention is to provide an information processing system having a function of estimating user fatigue, drowsiness, etc. in a short time.
  • One aspect of the present invention includes an imaging unit and an arithmetic unit having a function of performing calculation by machine learning, and the imaging unit has a function of acquiring a moving image which is a set of two or more frames of images.
  • the calculation unit has a function of detecting a first object from each of two or more images among the images included in the moving image, and the calculation unit has a function of detecting a second object from each of the detected first objects.
  • the calculation unit has a function of calculating the size of the second object for each of the detected second objects, and the calculation unit has the function of calculating the size of the second object over time. It is an information processing device that has a function of performing machine learning using changes.
  • machine learning may be performed by a neural network.
  • the moving image may include a face
  • the first object may be an eye
  • the second object may be a pupil
  • one aspect of the present invention is based on the learning result obtained by performing learning using the time course of the size of the first object shown in the two or more first images included in the first moving image.
  • An information processing device having a function of performing inference based on the information processing device, the information processing device has a function of acquiring a second moving image, and the information processing device has two or more second images included in the second moving image.
  • the information processing device has a function of detecting a third object from each of the detected second objects, and the information processing device has a function of detecting a third object from each of the detected second objects.
  • Each of the third objects has a function of calculating the size of the third object, and the information processing device has a function of making an inference based on the learning result with respect to the time course of the size of the third object. It is an information processing device having.
  • learning and inference may be performed by a neural network, and the learning result may include a weighting coefficient.
  • the first moving image includes the first face
  • the second moving image includes the second face
  • the first and third objects are pupils
  • the second The object may be an eye
  • the information processing device may have a function of estimating fatigue of a person having a second face.
  • an information processing device having a function of detecting user fatigue, drowsiness, etc. in real time.
  • an information processing system having a function of detecting user fatigue, drowsiness, etc. in real time.
  • FIG. 1 is a block diagram showing a configuration example of an information processing system.
  • FIG. 2 is a flowchart showing an example of an operation method of the information processing apparatus.
  • FIG. 3 is a flowchart showing an example of an operation method of the information processing apparatus.
  • FIG. 4 is a flowchart showing an example of an operation method of the information processing apparatus.
  • 5A, 5B, and 5C are schematic views showing an example of an operation method of the information processing apparatus.
  • 6A and 6B are schematic views showing an example of an operation method of the information processing apparatus.
  • 7A1 and 7A2, and 7B1 and 7B2 are schematic views showing an example of an operation method of the information processing apparatus.
  • 8A and 8B are schematic views showing an example of an operation method of the information processing apparatus.
  • FIGS. 10B1 and 10B2 are schematic views showing an example of an operation method of the information processing apparatus.
  • FIG. 11 is a diagram illustrating AnoGAN that can be applied to one aspect of the present invention.
  • the components are classified by function and the block diagram is shown as blocks independent of each other. However, it is difficult to completely separate the actual components by function, and one component is used. It is possible that a component is involved in a plurality of functions, or that one function is realized by a plurality of components.
  • Embodiment 1 an information processing system according to one aspect of the present invention and an information processing method using the information processing system will be described.
  • the information processing system and information processing method of one aspect of the present invention it is possible to estimate fatigue, drowsiness, etc. of a user of an information terminal such as a smartphone or a tablet.
  • FIG. 1 is a block diagram showing a configuration example of an information processing system 10 which is an information processing system of one aspect of the present invention.
  • the information processing system 10 includes an information processing device 20 and an information processing device 30.
  • the information processing device 20 includes an imaging unit 21, a display unit 22, a calculation unit 23, a main storage unit 24, an auxiliary storage unit 25, and a communication unit 26. Data and the like can be transmitted between the components of the information processing apparatus 20 via the transmission line 27. Further, the information processing device 30 includes an imaging unit 31, a display unit 32, a calculation unit 33, a main storage unit 34, an auxiliary storage unit 35, and a communication unit 36. Data and the like can be transmitted between the components of the information processing apparatus 30 via the transmission line 37.
  • the imaging unit 21 and the imaging unit 31 have a function of performing imaging and acquiring imaging data.
  • the display unit 22 and the display unit 32 have a function of displaying an image.
  • the calculation unit 23 and the calculation unit 33 have a function of performing calculation processing.
  • the calculation unit 23 has a function of performing predetermined calculation processing on data transmitted from the image pickup unit 21, the main storage unit 24, the auxiliary storage unit 25, or the communication unit 26 to the calculation unit 23 via the transmission line 27, for example.
  • the calculation unit 33 has a function of performing a predetermined calculation process on the data transmitted from the imaging unit 31, the main storage unit 34, the auxiliary storage unit 35, or the communication unit 36 to the calculation unit 33 via the transmission line 37, for example.
  • the calculation unit 23 and the calculation unit 33 have a function of performing a calculation by machine learning. For example, it has a function of performing an operation using a neural network.
  • the calculation unit 23 and the calculation unit 33 can have, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and the like.
  • the main storage unit 24 and the main storage unit 34 have a function of storing data, programs, and the like.
  • the calculation unit 23 can read the data stored in the main storage unit 24, a program, and the like, and execute the calculation process. For example, the calculation unit 23 can execute a predetermined calculation process on the data read from the main storage unit 24 by executing the program read from the main storage unit 24.
  • the calculation unit 33 can read the data stored in the main storage unit 34, a program, and the like, and execute the calculation process. For example, the calculation unit 33 can execute a predetermined calculation process on the data read from the main storage unit 34 by executing the program read from the main storage unit 34.
  • the main storage unit 24 and the main storage unit 34 preferably operate at a higher speed than the auxiliary storage unit 25 and the auxiliary storage unit 35.
  • the main storage unit 24 and the main storage unit 34 can have, for example, a DRAM (Dynamic Random Access Memory), a SRAM (Static Random Access Memory), or the like.
  • the auxiliary storage unit 25 and the auxiliary storage unit 35 have a function of storing data, programs, and the like for a longer period of time than the main storage unit 24 and the main storage unit 34.
  • the auxiliary storage unit 25 and the auxiliary storage unit 35 may have, for example, an HDD (Hard Disk Drive), an SSD (Solid State Drive), or the like.
  • the auxiliary storage unit 25 and the auxiliary storage unit 35 include ReRAM (Resetive Random Access Memory, also referred to as resistance change type memory), PRAM (Phase change Random Access Memory), FeRAM (Random Access Memory), and FeRAM (Random Access Memory). It may have a non-volatile memory such as Access Memory (also referred to as magnetic resistance type memory) or flash memory.
  • the communication unit 26 has a function of transmitting / receiving data or the like to a device or the like provided outside the information processing device 20.
  • the communication unit 36 has a function of transmitting / receiving data or the like to a device or the like provided outside the information processing device 30. For example, by supplying data or the like from the communication unit 26 to the communication unit 36, the information processing device 20 can supply the data or the like to the information processing device 30. Further, the communication unit 26 and the communication unit 36 can have a function of supplying data or the like to the network and a function of acquiring data or the like from the network.
  • the calculation unit 23 and the calculation unit 33 have a function of performing a calculation by machine learning
  • the calculation unit 23 can perform learning and the learning result can be supplied from the information processing device 20 to the information processing device 30.
  • the calculation unit 23 learns to acquire a weight coefficient or the like, and the weight coefficient or the like is information from the information processing device 20. It can be supplied to the processing device 30.
  • the arithmetic unit 33 provided in the information processing apparatus 30 does not perform learning, the data input to the arithmetic unit 33 is inferred based on the learning result by the arithmetic unit 23 provided in the information processing apparatus 20. It can be performed. Therefore, the arithmetic processing capacity of the arithmetic unit 33 can be made lower than that of the arithmetic unit 23.
  • the information processing device 20 can be provided in, for example, a server.
  • the information processing device 20 does not have to be provided with the image pickup unit 21 and the display unit 22. That is, the imaging unit 21 and the display unit 22 may be provided outside the information processing device 20.
  • the information processing device 30 can be provided in an information terminal such as a smartphone, a tablet, or a personal computer. Further, at least a part of the components of the information processing device 20 and at least a part of the components of the information processing device 30 may be provided in the server.
  • the calculation unit 23 and the calculation unit 33 may be provided in the server. In this case, for example, the data acquired by the information terminal is supplied to the calculation unit 33 via the network, and the calculation unit 33 provided in the server makes an inference or the like for the data. Then, by supplying the inference result to the information terminal via the network, the information terminal can acquire the inference result.
  • Example of information processing method> an example of an information processing method using the information processing system 10 will be described. Specifically, an example of a method of estimating fatigue, drowsiness, etc. of a user of an information terminal provided with an information processing device 30 included in the information processing system 10 by calculation using machine learning will be described.
  • FIG. 2 and 3 are flowcharts illustrating an example of a method of estimating fatigue, drowsiness, etc. by calculation using machine learning. Learning is shown in FIG. 2, and inference is shown in FIG.
  • the imaging unit 21 captures a moving image.
  • a moving image including a human face is captured (step S01).
  • the moving image indicates a set of images having two or more frames.
  • learning data is created based on the moving image captured by the imaging unit 21, and the calculation unit 23 performs learning. Therefore, for example, when the imaging unit 21 captures a moving image including a human face, it is preferable that the imaging unit 21 captures the moving image for a large number of people having different genders, races, physiques, and the like.
  • Image processing may be performed on the moving image captured by the imaging unit 21. For example, noise removal, grayscale conversion, normalization, contrast adjustment, and the like can be performed. Further, the image included in the moving image may be binarized or the like. By performing such processing, the subsequent steps can be performed with high accuracy. For example, the detection of the first object performed in step S02, which will be described later, can be performed with high accuracy.
  • the calculation unit 23 detects the first object from each of the captured images.
  • the first object can be an eye, for example, when a moving image of the face is captured in step S01 (step S02).
  • the first object can be detected, for example, by a cascade classifier. For example, it can be detected by Haar Cascades. If the first object is the eye and both eyes are included in one image, only one eye can be detected.
  • the calculation unit 23 detects the second object from each of the detected first objects.
  • the second object can be the pupil (step S03).
  • the pupil can be detected from the eye by circular extraction. Details of the method for detecting the pupil from the eye will be described later.
  • the pupil is a hole surrounded by an iris and can be called a "black eye".
  • the pupil has the function of adjusting the amount of light projected onto the retina.
  • the iris is, for example, a thin film between the cornea and the crystalline lens, which can be a colored portion of the eye.
  • the calculation unit 23 calculates the size of the second object for each of the detected second objects (step S04). For example, when the second object is detected by circular extraction, the radius or diameter of the second object can be set as the size of the second object. When the shape of the second object is extracted into an elliptical shape, the length of the major axis and the length of the minor axis can be set as the size of the second object. Further, the area of the second object can be the size of the second object.
  • the calculation unit 23 performs learning using the size of the second object and acquires the learning result (step S05). Specifically, the learning result is acquired based on the time-dependent change in the size of the second object. Learning can be performed using, for example, a neural network. In this case, the learning result can be a weighting coefficient or the like as described above. The details of the learning method will be described later.
  • the information processing device 20 supplies the learning result to the information processing device 30 (step S06). Specifically, the learning result acquired by the calculation unit 23 is transmitted to the communication unit 26 via the transmission line 27, and then supplied from the communication unit 26 to the communication unit 36.
  • the learning result supplied to the communication unit 36 can be stored in the auxiliary storage unit 35. Further, the learning result may be stored in the auxiliary storage unit 25.
  • the imaging unit 31 captures a moving image. For example, a moving image including the face of a user of an information terminal provided with an information processing device 30 is captured (step S11).
  • step S01 shown in FIG. 2 when image processing is performed on the moving image captured by the imaging unit 21, the same image processing is performed on the moving image captured by the imaging unit 31 to make the inference accurate. It is preferable because it can be done well.
  • the calculation unit 33 detects the first object from each image included in the captured moving image.
  • the first object can be an eye, for example, when a moving image of the face is captured in step S11 (step S12).
  • the first object can be detected by the same method as the detection method used in step S02 shown in FIG.
  • the calculation unit 33 detects the second object from each of the detected first objects.
  • the second object can be the pupil (step S13).
  • the second object can be detected by the same method as the detection method used in step S03 shown in FIG.
  • the calculation unit 33 calculates the size of the second object for each of the detected second objects (step S14).
  • the size calculation method the same method as that used in step S04 shown in FIG. 2 can be used.
  • the calculation unit 33 into which the learning result acquired by the calculation unit 23 in step S05 shown in FIG. 2 is input makes an inference.
  • the calculation unit 33 can estimate the fatigue, drowsiness, etc. of the user (step S15). The details of the inference method will be described later.
  • step S01 shown in FIG. 2 it is preferable to take a plurality of moving images of the face of the same person, for example, with different brightness of the environment. Thereby, for example, regardless of the brightness of the environment, it is possible to accurately estimate the fatigue, drowsiness, etc. of the user of the information terminal provided with the information processing device 30.
  • the information processing device 30 having a function of estimating fatigue, drowsiness, etc. is provided in an information terminal such as a smartphone, a tablet, or a personal computer.
  • an information terminal such as a smartphone, a tablet, or a personal computer.
  • FIG. 4 is a flowchart showing an example of a pupil detection method.
  • the calculation unit acquires an image 41, which is an image including the detected eyes (step S31).
  • FIG. 5A is a schematic diagram illustrating step S31.
  • the calculation unit acquires an image 41 including eyes detected from the image captured by the imaging unit.
  • the calculation unit 23 acquires an image including the eyes detected by the calculation unit 23 in step S02 as an image 41.
  • the calculation unit 33 acquires an image including the eyes detected by the calculation unit 33 in step S12 as an image 41.
  • the calculation unit may convert the image 41 to grayscale after the calculation unit acquires the image 41.
  • FIG. 5B is a schematic diagram illustrating an expansion process and a contraction process.
  • the calculation unit subtracts the image 43 from the image 41 and acquires the image 44 (step S33). That is, the image 44 is an image represented by the difference between the image 41 and the image 43.
  • the calculation unit acquires the image 44 by performing the Black-hat conversion using the image 41 and the image 43.
  • the calculation unit adds the image 41 acquired in step S31 and the image 44 acquired in step S33 to acquire the image 45 (step S34).
  • the image 41 is converted to grayscale in step S31
  • the image 41 converted to grayscale and the image 44 can be added together in step S34.
  • steps S32 to S34 may not be performed in whole or in part. Further, a process other than the processes shown in steps S32 to S34 may be performed.
  • the calculation unit performs image processing on the image 45 and acquires the image 46 (step S35).
  • the arithmetic unit performs processing such as noise removal and smoothing on the image 45.
  • processing such as edge detection and binarization is performed. Specifically, for example, after removing noise with an intermediate value filter and smoothing the image 45 with a Gaussian filter, edge detection by the Canny method and binarization processing are performed.
  • the noise may be removed by, for example, a moving average filter.
  • the smoothing may be performed by, for example, a moving average filter or a median filter.
  • edge detection may be performed by, for example, a Laplacian filter.
  • the calculation unit detects the iris 47.
  • the iris 47 can be detected by using the Hough transform.
  • the Hough transform is used, for example, the iris 47 can be detected in a circular shape. Alternatively, for example, the iris 47 can be detected in an elliptical shape.
  • the iris 47 may be detected by using the generalized Hough transform.
  • the calculation unit acquires the image 49 including the detected iris 47 (step S36).
  • the image 49 is extracted from the image 46 based on the coordinates of the detected iris 47 in the image 46.
  • FIG. 5C is a schematic diagram illustrating step S36.
  • the image 49 can be a rectangle whose four sides are in contact with the iris 47, as shown in FIG. 5C.
  • the image 49 can be a square whose four sides are in contact with the iris 47. It should be noted that each side of the image 49 does not have to be in contact with the iris 47.
  • an image having a predetermined number of pixels centered on the iris 47 may be an image 49.
  • the calculation unit detects the pupil 48 from the image 49 (step S37).
  • the pupil 48 is detected from the image 49 by an operation using a neural network.
  • Step S37 is performed using a generator that has been trained in advance.
  • the generator is a program that performs calculations by machine learning, and has a function of outputting data corresponding to the input data. Specifically, the generator can make inferences on the data input to the generator by performing learning.
  • FIG. 6A is a schematic diagram illustrating the above learning.
  • the generator that performs learning is referred to as the generator 50.
  • the generator 50 can use a convolutional neural network (CNN).
  • CNN convolutional neural network
  • U-net the input image is downsampled by convolution, and then upsampled by deconvolution using the features obtained by downsampling. It can be said that the calculation unit 23 and the calculation unit 33 have a function as a generator 50.
  • the learning of the generator 50 can be performed by supervised learning using the data 51 and the data 52.
  • the data 51 can be a set of images 59.
  • Image 59 includes the iris 57 and the pupil 58.
  • the image 59 can be acquired by the information processing apparatus 20 in the same manner as in steps S01 and S02 shown in FIG. 2 and steps S31 to S36 shown in FIG.
  • the imaging unit 21 captures a moving image of the face, but when acquiring the image 59, it is not necessary to capture the moving image.
  • the imaging unit 21 may capture one image (one frame) per person.
  • the data 52 is data showing the coordinates of the pupil 58 included in the image 59. Specifically, it is possible to obtain a binary image in which the color of the portion of the pupil 58 is different from the color of the other portion.
  • the data 52 can be obtained, for example, by filling the pupil 58 included in the image 59.
  • the data 52 can be acquired by acquiring the image including the eyes by the same method as in step S31 shown in FIG. 4 and then filling the pupil 58 of the image including the eyes.
  • the learning of the generator 50 is performed so that when the data 51 is input to the generator 50, the output data approaches the data 52. That is, the generator 50 is trained using the data 52 as the correct answer data.
  • the generator 50 performs learning, the generator 50 generates a learning result 53.
  • the learning result 53 can be a weighting coefficient or the like.
  • the learning of the generator 50 that is, the generation of the learning result 53 can be performed by, for example, the arithmetic unit 23 of the information processing apparatus 20. Then, by supplying the learning result 53 from the information processing device 20 to the information processing device 30, the calculation unit 33 can perform the same inference as the calculation unit 23.
  • the learning result 53 generated by the calculation unit 23 can be stored in, for example, the auxiliary storage unit 25. Further, the learning result 53 generated by the calculation unit 23 and supplied to the information processing device 30 can be stored in, for example, the auxiliary storage unit 35.
  • FIG. 6B is a schematic diagram illustrating step S37. That is, FIG. 6B is a schematic diagram illustrating the detection of the pupil 48 from the image 49.
  • step S37 the image 49 acquired by the calculation unit in step S36 is input to the generator 50 in which the learning result 53 is read.
  • the generator 50 can make an inference to the image 49 and output data indicating the coordinates of the pupil 48.
  • the generator 50 can output a binary image in which the color of the pupil 48 is different from the color of other parts.
  • the pupil can be detected in step S03 or step S13 from the eyes detected in step S02 or step S12.
  • the pupil By detecting the pupil by calculation using machine learning, it is possible to detect the pupil in a shorter time than, for example, visually detecting the pupil. Further, for example, even if the surrounding landscape is reflected in the pupil, the pupil can be detected with high accuracy.
  • the method of detecting the pupil 48 in step S37 is not limited to the method shown in FIGS. 6A and 6B.
  • the image 49 may be a color image
  • the color image 49 may be grayscaled
  • the edge of the pupil 48 may be detected. Then, after performing edge detection, the pupil 48 may be detected.
  • the grayscale of the image 49 can be performed using, for example, partial least squares (PLS) regression.
  • PLS partial least squares
  • the edge detection of the pupil 48 can be performed by, for example, the Canny method or the Laplacian filter. Further, the pupil 48 can be detected after the edge detection, for example, by using the Hough transform. When the Hough transform is used, the pupil 48 can be detected in a circular shape, for example. Alternatively, for example, the pupil 48 can be detected in an elliptical shape. The pupil 48 may be detected by using the generalized Hough transform.
  • imaging can be performed using infrared rays in step S01 or step S11.
  • the iris reflects infrared rays.
  • the pupil does not reflect infrared rays. Therefore, in step S01 or step S11, the iris and the pupil can be clearly distinguished by performing imaging using infrared rays. Therefore, the pupil can be detected with high accuracy.
  • Example of estimation method for fatigue, drowsiness, etc._1 Next, an example of a method of estimating fatigue, drowsiness, etc. of the user of the information terminal provided with the information processing device 30 by calculation using machine learning will be described. Specifically, an example of the learning method using the size of the pupil, which is performed in step S05, will be described. In addition, an example of a method for estimating fatigue, drowsiness, etc. by inference based on the learning result, which is performed in step S15, will be described. In the following, the second object will be described as a pupil.
  • FIG. 7A1 is a schematic diagram illustrating step S05.
  • the generator 60 which is a program that performs calculations by machine learning, is learned.
  • the generator 60 can use a neural network. Details will be described later, but time-series data such as a change in pupil size over time is input to the generator 60. Therefore, when a neural network is used as the generator 60, it is preferable to use a recursive neural network (RNN) as the generator 60. Alternatively, it is preferable to use long / short-term memory (Long Short-Term Memory: LSTM) as the generator 60. Alternatively, it is desirable to use a gated recurrent unit (GRU).
  • RNN recursive neural network
  • LSTM Long Short-Term Memory
  • GRU gated recurrent unit
  • the learning of the generator 60 can be performed using the data 61 and the data 62.
  • the data 61 is the data acquired in step S04, and can be the time-dependent change in the size of the pupil.
  • the radius or diameter of the pupil can be used as the size of the pupil.
  • the length of the major axis and the length of the minor axis can be used as the size of the pupil.
  • the area of the pupil can be the size of the pupil.
  • the time-dependent change in the size of the pupil at time 1 to n-1 (n is an integer of 3 or more) is used as data 61.
  • the data 61 may be a change over time in the ratio of the size of the pupil to the size of the iris.
  • the iris and the pupil are extracted into the same type of shape.
  • the pupil is also extracted in a circular shape.
  • the pupil is also extracted in an elliptical shape.
  • the image 49 including the iris 47 and the pupil 48 when the pupil is detected by the method shown in step S37, for example, by setting the ratio of the pupil size to the iris size with time, the image 49 including the iris 47 and the pupil 48.
  • the resolutions can be different from each other.
  • the resolution of the image 49 including the first human iris 47 and the pupil 48 and the resolution of the image 49 including the second human iris 47 and the pupil 48 can be different from each other.
  • Data 62 is the size of the pupil at time n. That is, it is the size of the pupil at a time after the time when the size of the pupil included in the data 61 is measured.
  • the data 61 is the time-dependent change in the ratio of the pupil size to the iris size
  • the data 62 is also the ratio of the pupil size to the iris size.
  • FIG. 7A2 is a diagram showing an example of the relationship between the pupil diameter and the time.
  • the black circles indicate the measured values of the pupil diameter.
  • the measured points may be indicated by black circles.
  • the data 62 can be the size of the pupil at a time after the time when the size of the pupil included in the data 61 is measured.
  • the data 62 can be the size of the pupil at the time following the last time when the size of the pupil included in the data 61 is measured.
  • the generator 60 has a function of estimating the presence or absence of fatigue
  • the change with time of the pupil size of the person with fatigue is not included in the data 61 and the data 62. That is, the data 61 is the time-dependent change in the size of the pupil of the person without fatigue, and the data 62 is the size of the pupil of the person without fatigue.
  • the generator 60 has a function of estimating the presence or absence of drowsiness
  • the change with time of the pupil size of the drowsy person is not included in the data 61 and the data 62. That is, the data 61 is the time-dependent change in the size of the pupil of the person without drowsiness, and the data 62 is the size of the pupil of the person without drowsiness.
  • the learning of the generator 60 is performed so that when the data 61 is input to the generator 60, the output data approaches the data 62. That is, the generator 60 is trained using the data 62 as the correct answer data.
  • the generator 60 performs learning, the generator 60 generates a learning result 63.
  • the learning result 63 can be a weighting coefficient or the like.
  • step S15 is schematic views for explaining step S15, and show an example of a method of estimating fatigue, drowsiness, etc. of a user of an information terminal provided with an information processing device 30 by using a generator 60. It is a figure.
  • step S15 first, as shown in FIG. 7B1, the data 64 indicating the change in pupil size with time acquired in step S14 is input to the generator 60 in which the learning result 63 is read.
  • the time-dependent change in pupil size at times 1 to n-1 is used as input data when learning the generator 60 in step S05, times 1 to n are also used during the inference in step S15.
  • the time-dependent change in the size of the pupil in -1 is used as the input data.
  • the data 64 is the time-dependent change in the size of the pupil of the user of the information terminal provided with the information processing device 30 at times 1 to n-1.
  • the generator 60 makes an inference to the data 64 and outputs the data 65.
  • data 65 which is inference data at time n, may be used, data at times 2 to n may be used as input data, and data at time n + 1 may be inferred.
  • the data 64 is also the time-dependent change in the ratio between the pupil size and the iris size.
  • the resolution of the image 49 including the iris 47 and the pupil 48 when the pupil is detected by the method shown in step S37, for example, by using the data 64 as the time-dependent change in the ratio of the pupil size to the iris size. Can be different from each other.
  • the resolution of the image 49 acquired by the arithmetic unit to calculate the ratio of and can be made different from each other.
  • the data 65 is an estimated value of the pupil size at a time after the time when the pupil size included in the data 64 is measured, which is calculated by inferring the data 64 based on the learning result 63. Is.
  • the data 65 can be the size of the pupil at time n.
  • the magnitude of the measured value of the pupil at time 1 to n-1 it is described as x 1 to x n-1, respectively.
  • the estimated value of the pupil size at time n is described as x n (E).
  • the data 64 is the time-dependent change in the ratio between the pupil size and the iris size
  • the data 65 is the ratio between the pupil size and the iris size.
  • the data 66 representing the measured value of the pupil size at time n is compared with the data 65 which is the data output from the generator 60. That is, for example, the measured value of the pupil size at time n is compared with the estimated value.
  • the presence or absence of fatigue, drowsiness, etc. is estimated from the comparison results.
  • the generator 60 has a function of estimating the presence or absence of fatigue
  • learning of the generator 60 is performed using, for example, a change over time in the size of the pupil of a person without fatigue as input data. Therefore, when the user of the information terminal provided with the information processing device 30 is in a state without fatigue, the data 65 is close to the data 66.
  • the difference between the data 65 and the data 66 is small.
  • the difference between the data 66 and the data 65 is determined by the user of the information terminal provided with the information processing device 30. It is larger than the case without fatigue. From the above, by comparing the data 66 and the data 65, it is possible to estimate the presence or absence of fatigue of the user of the information terminal provided with the information processing device 30. The same applies when estimating the presence or absence of drowsiness.
  • the data 65 is an estimated value of the ratio between the pupil size and the iris size
  • the data 66 is an actually measured value of the ratio between the pupil size and the iris size.
  • the function as the generator 60 can be provided to both the calculation unit 23 and the calculation unit 33.
  • the arithmetic unit 23 of the information processing device 20 learns the generator 60 to generate the learning result 63, and the learning result 63 can be supplied from the information processing device 20 to the information processing device 30.
  • the data input to the arithmetic unit 33 is inferred based on the learning result by the arithmetic unit 23 provided in the information processing apparatus 20. It can be performed. Therefore, the arithmetic processing capacity of the arithmetic unit 33 can be made lower than that of the arithmetic unit 23.
  • the learning result 63 can be stored in the auxiliary storage unit 25 and the auxiliary storage unit 35.
  • FIG. 8A and 8B are schematic views for explaining step S05, and are examples of a generator learning method different from the above method.
  • FIG. 8A is a diagram showing an example of a method of creating data input to the generator as learning data
  • FIG. 8B is a diagram showing an example of a learning method of the generator 80.
  • the generator 80 is a program that performs calculations by machine learning.
  • a neural network can be used as the generator 80.
  • the data 81 shown in FIG. 8A is the data acquired in step S04, and can be the time-dependent change in the size of the pupil.
  • the radius or diameter of the pupil can be used as the size of the pupil.
  • the length of the major axis and the length of the minor axis can be used as the size of the pupil.
  • the area of the pupil can be the size of the pupil. Similar to the learning method shown in FIG. 7A1, the data 81 can be a change over time in the ratio of the size of the pupil to the size of the iris.
  • the data 82 is generated by performing a Fourier transform on the data 81. As shown in FIG. 8A, the change with time of the pupil diameter can be converted into the frequency characteristic of the pupil diameter by the Fourier transform.
  • the data 82 can be a frequency characteristic of the ratio of the pupil size to the iris size.
  • the learning of the generator 80 can be performed using the data 82 and the data 83 as shown in FIG. 8B.
  • the data 82 represents the frequency characteristics of the pupil diameter as described above.
  • the data 83 can be a label indicating the presence or absence of fatigue.
  • the data 83 includes both the frequency characteristic of the pupil size of the person with fatigue and the frequency characteristic of the pupil size of the person without fatigue. Then, the frequency characteristic of the pupil size of the person with fatigue is associated with the label "with fatigue", and the frequency characteristic of the pupil size of the person without fatigue is associated with the label "no fatigue".
  • the data 83 may be used as a label indicating the presence or absence of drowsiness.
  • the learning of the generator 80 is performed so that when the data 82 is input to the generator 80, the output data approaches the data 83. That is, the generator 80 is trained using the data 83 as the correct answer data.
  • the generator 80 performs learning, the generator 80 generates a learning result 84.
  • the learning result 84 can be a weighting coefficient or the like.
  • the data input to the generator 80 can be made into data that is not time series data.
  • the generator 80 can perform learning and inference without using the RNN as the generator 80.
  • 9A and 9B are schematic views for explaining step S15, and show an example of a method of estimating fatigue, drowsiness, etc. of a user of an information terminal provided with an information processing device 30 by using a generator 80. It is a figure.
  • the data 85 shown in FIG. 9A is the data acquired in step S04, and can be the time-dependent change in the size of the pupil.
  • the radius or diameter of the pupil can be used as the size of the pupil.
  • the length of the major axis and the length of the minor axis can be used as the size of the pupil.
  • the area of the pupil can be the size of the pupil.
  • the data 86 is generated by performing the Fourier transform on the data 85. As shown in FIG. 9A, the change with time of the pupil diameter can be converted into the frequency characteristic of the pupil diameter by the Fourier transform.
  • the data 85 is a change over time in the ratio of the pupil size to the iris size
  • the data 86 can be a frequency characteristic of the ratio of the pupil size to the iris size.
  • the data 86 after the Fourier transform is input to the generator 80.
  • the generator 80 can infer the data 86 and output the data 87 indicating the presence or absence of fatigue.
  • the data 87 shown in FIG. 9B is a label indicating the presence or absence of drowsiness
  • the data 87 output by the generator 80 can be data indicating the presence or absence of drowsiness.
  • the function as the generator 80 can be provided to both the calculation unit 23 and the calculation unit 33 in the same manner as the function as the generator 60 and the function as the generator 70. As a result, the arithmetic processing capacity of the arithmetic unit 33 can be made lower than that of the arithmetic unit 23.
  • FIG. 10A is a schematic diagram illustrating step S05, and is an example of a generator learning method different from the above method.
  • the generator 70 is trained.
  • the generator 70 is a program that performs calculations by machine learning.
  • a neural network can be used, and for example, an autoencoder can be used.
  • the data 71 is the data acquired in step S04, and can be the time-dependent change in the size of the pupil.
  • the generator 70 has a function of estimating the presence or absence of fatigue
  • the time-dependent change in the size of the pupil of the person with fatigue is not included in the data 71. That is, the data 71 is a change over time in the size of the pupil of a person without fatigue.
  • the generator 70 has a function of estimating the presence or absence of drowsiness
  • the change with time of the pupil size of the drowsy person is not included in the data 71. That is, the data 71 is a change over time in the size of the pupil of a person without drowsiness.
  • the radius or diameter of the pupil can be used as the size of the pupil.
  • the length of the major axis and the length of the minor axis can be used as the size of the pupil.
  • the area of the pupil can be the size of the pupil.
  • the data 71 can be a change over time in the ratio of the size of the pupil to the size of the iris. Further, as in the case shown in FIG. 8A, for example, a Fourier transform of the change with time of the pupil size may be used as the data 71.
  • the learning of the generator 70 is performed so that when the data 71 is input to the generator 70, the data 72, which is the output data, approaches the input data 71. That is, the generator 70 is trained so that the data 71 and the data 72 are equal to each other.
  • the generator 70 performs learning, the generator 70 generates a learning result 73.
  • the learning result 73 can be a weighting coefficient or the like.
  • step S15 are schematic views for explaining step S15, and show an example of a method of estimating fatigue, drowsiness, etc. of a user of an information terminal provided with an information processing device 30 by using a generator 70. It is a figure.
  • step S15 first, as shown in FIG. 10B1, the data 74 indicating the change in pupil size with time acquired in step S14 is input to the generator 70 in which the learning result 73 is read. As a result, the generator 70 makes an inference to the data 74 and outputs the data 75.
  • the data 74 is also the time-dependent change in the ratio between the pupil size and the iris size.
  • the data after the Fourier transform is used as the data 71
  • the data after the Fourier transform is also used for the data 74.
  • the data 74 is also assumed to be the Fourier transform of the time course of the pupil size.
  • the data 74 which is the data input to the generator 70
  • the data 75 which is the data output from the generator 70
  • the presence or absence of fatigue, drowsiness, etc. is estimated from the comparison results.
  • the generator 70 has a function of estimating the presence or absence of fatigue
  • learning of the generator 70 is performed, for example, by using the time course of the pupil size of a person without fatigue. Therefore, when the user of the information terminal provided with the information processing device 30 is in a state without fatigue, the data 75, which is the output data from the generator 70, is close to the data 74, which is the input data to the generator 70. It becomes a thing.
  • the difference between the data 74 and the data 75 is small.
  • the difference between the data 74 and the data 75 is determined by the user of the information terminal provided with the information processing device 30. It is larger than the case without fatigue. From the above, by comparing the data 74 and the data 75, it is possible to estimate the presence or absence of fatigue of the user of the information terminal provided with the information processing device 30. The same applies when estimating the presence or absence of drowsiness.
  • the function as the generator 70 can be provided to both the calculation unit 23 and the calculation unit 33 in the same manner as the function as the generator 60. As a result, the arithmetic processing capacity of the arithmetic unit 33 can be made lower than that of the arithmetic unit 23.
  • the learning performed in step S05 and the inference based on the learning result performed in step S15 may be performed using a hostile generative network (GAN).
  • GAN hostile generative network
  • it may be performed using AnnoGAN (Anormary GAN).
  • FIG. 11 is a diagram illustrating AnoGAN capable of performing the above learning and inference.
  • the AnoGAN shown in FIG. 11 has a generator 91 and a discriminator 92.
  • the generator 91 and the discriminator 92 can be configured by a neural network.
  • the discriminator 92 is input with data 93, which is time-series data representing changes over time in the size of the pupil of a person who does not have fatigue, drowsiness, etc., acquired by imaging.
  • data 95 which is time-series data generated by the generator 91 to which the data 94 is input, is input to the discriminator 92.
  • the discriminator 92 has a function of determining whether the input data is the data 93 acquired by imaging or the data 95 generated by the generator 91 (also referred to as authenticity determination).
  • the data 93 may be Fourier-transformed time-series data representing changes in the pupil size of a person without fatigue, drowsiness, etc., acquired by imaging.
  • the determination result is output as data 96.
  • the data 96 can be, for example, a continuous value between 0 and 1.
  • the discriminator 92 outputs a value close to 1 as the data 96 when the input data is the data 93 acquired by imaging after the learning is completed, and the input data is the generator.
  • a value close to 0 may be output as the data 96.
  • the data 94 is a multidimensional random number (also referred to as a latent variable).
  • the latent variable represented by the data 94 is defined as the latent variable z.
  • the generator 91 has a function of generating data as similar as possible to data representing changes in the size of the pupil of a person who does not have fatigue, drowsiness, etc., based on such data 94.
  • the learning of the discriminator 92 and the learning of the generator 91 are alternately performed. That is, when the discriminator 92 is trained, the weighting coefficient of the neural network constituting the generator 91 is fixed. Further, when learning the generator 91, the weighting coefficient of the neural network constituting the discriminator 92 is fixed.
  • the discriminator 92 At the time of learning the discriminator 92, the data 93 acquired by imaging or the data 95 generated by the generator 91 is input to the discriminator 92. A correct label is given to the data input to the discriminator 92.
  • the correct label can be determined for the data 96 output by the discriminator 92 as follows. For example, when the data 93 is input to the discriminator 92, the correct answer label is set to “1”, and when the data 95 is input to the discriminator 92, the correct answer label is set to “0”.
  • the discriminator 92 can perform the authenticity determination.
  • the generator 91 At the time of learning the generator 91, the data 94 representing the latent variable z is input to the generator 91. Then, the generator 91 generates the data 95 based on the input data 94. The correct label of the data 96 is "1". Then, the generator 91 is trained so that the value of the data 96 output from the discriminator 92 becomes “1”. As the learning of the generator 91 progresses, the generator 91 can generate data similar to the data 93 acquired by imaging as the data 95.
  • the generator 91 can generate the data 95 similar to the data 93 acquired by the imaging regardless of what latent variable z is input as the data 94.
  • the space of the latent variable is searched, and the latent variable z1 that generates the data most similar to the data of the pupil size of the person who does not have fatigue, drowsiness, etc. is found by the gradient descent method or the like.
  • the generator has a function of generating data very similar to the data representing the time-dependent change in the size of the pupil of a person who does not have fatigue, drowsiness, etc. by learning 91. Therefore, the data generated by the generator from the latent variable z1 and the data obtained by photographing and representing the change over time in the size of the pupil of a person without fatigue, drowsiness, etc. are very similar.
  • the generator 91 has the ability to generate data very close to the data representing the time course of the pupil size of a person without fatigue, drowsiness, etc. by learning, but the pupil size of a person with fatigue, drowsiness, etc. It does not have the ability to generate data that resembles data that represents changes over time.
  • the data generated by the generator from the latent variable z2 and the data representing the time-dependent change in the size of the pupil of a person with fatigue, drowsiness, etc. obtained by imaging are not very similar data. From the above, the presence or absence of fatigue, drowsiness, etc. can be estimated by the generator 91.
  • the information processing device 30 estimates fatigue, drowsiness, and the like of the user of the information terminal provided with the information processing device 30 by calculation using machine learning.
  • machine learning for example, fatigue, drowsiness, etc. can be accurately measured without manually setting the time-dependent change of the feature amount when presuming that there is fatigue and the time-dependent change of the feature amount when estimating that there is no fatigue.
  • fatigue, drowsiness, etc. can be detected without manually setting the time-dependent change in pupil size when presumed to be fatigued and the time-dependent change in pupil size when estimated to be non-fatigue. It can be estimated accurately.
  • fatigue, drowsiness, etc. can be estimated without manually setting the time-dependent change of the feature amount when presuming that there is fatigue and the time-dependent change of the feature amount when presuming that there is no fatigue.
  • Fatigue, drowsiness, etc. can be estimated by a simple method.
  • an alarm indicating that fatigue, drowsiness, etc. is occurring may be displayed on the display unit of the information terminal provided with the information processing device 30. It can. Thereby, for example, the user of the information terminal can be urged to stop using the information terminal as soon as possible. Alternatively, the power of the information terminal provided with the information processing device 30 can be turned off. As a result, it is possible to suppress the occurrence of health damage caused by continuing to use the information terminal even though the user of the information terminal is tired, drowsy, or the like.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • General Physics & Mathematics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Molecular Biology (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Computational Linguistics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Artificial Intelligence (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Human Computer Interaction (AREA)
  • Multimedia (AREA)
  • Ophthalmology & Optometry (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Medical Informatics (AREA)
  • Surgery (AREA)
  • Animal Behavior & Ethology (AREA)
  • Public Health (AREA)
  • Veterinary Medicine (AREA)
  • Psychiatry (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Geometry (AREA)
  • Child & Adolescent Psychology (AREA)
  • Educational Technology (AREA)
  • Hospice & Palliative Care (AREA)
  • Psychology (AREA)
  • Social Psychology (AREA)
  • Pathology (AREA)
  • Developmental Disabilities (AREA)
  • Image Analysis (AREA)
  • Traffic Control Systems (AREA)
PCT/IB2020/057918 2019-09-06 2020-08-25 情報処理装置 Ceased WO2021044249A1 (ja)

Priority Applications (4)

Application Number Priority Date Filing Date Title
JP2021543605A JP7548914B2 (ja) 2019-09-06 2020-08-25 情報処理装置
US17/638,359 US12333859B2 (en) 2019-09-06 2020-08-25 Information processor performing arithmetic operation using machine learning
JP2024150460A JP7836865B2 (ja) 2019-09-06 2024-09-02 情報処理装置
US19/215,713 US20250329192A1 (en) 2019-09-06 2025-05-22 Information processor

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2019-162687 2019-09-06
JP2019162687 2019-09-06

Related Child Applications (2)

Application Number Title Priority Date Filing Date
US17/638,359 A-371-Of-International US12333859B2 (en) 2019-09-06 2020-08-25 Information processor performing arithmetic operation using machine learning
US19/215,713 Continuation US20250329192A1 (en) 2019-09-06 2025-05-22 Information processor

Publications (1)

Publication Number Publication Date
WO2021044249A1 true WO2021044249A1 (ja) 2021-03-11

Family

ID=74852843

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/IB2020/057918 Ceased WO2021044249A1 (ja) 2019-09-06 2020-08-25 情報処理装置

Country Status (3)

Country Link
US (2) US12333859B2 (https=)
JP (2) JP7548914B2 (https=)
WO (1) WO2021044249A1 (https=)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2023023226A3 (en) * 2021-08-18 2023-04-06 Diagnosys LLC System comprising integrated dichoptic flash and pupillometry monitoring, and method for using the same
US12076154B2 (en) 2015-11-10 2024-09-03 Diagnosys LLC Method and apparatus for the assessment of electrophysiological signals

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR102615650B1 (ko) * 2022-08-09 2023-12-20 저지앙 대학 생성적 적대 신경망 기반의 군중 이동 예측 방법
CN115937595A (zh) * 2022-12-20 2023-04-07 中交公路长大桥建设国家工程研究中心有限公司 一种基于智能数据处理的桥梁表观异常识别方法及系统
CN117077086B (zh) * 2023-07-25 2024-12-27 中国人民解放军空军军医大学 一种超前脑疲劳预警系统、设备和存储介质

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2007295946A (ja) * 2006-04-27 2007-11-15 Konica Minolta Holdings Inc 酒気帯び検知システム及び酒気帯び検知方法
JP2014050649A (ja) * 2012-09-10 2014-03-20 Nippon Telegr & Teleph Corp <Ntt> 疲労感評価装置、疲労感評価方法、及びプログラム
JP2015132783A (ja) * 2014-01-16 2015-07-23 日本電信電話株式会社 音の顕著度推定装置、その方法、及びプログラム

Family Cites Families (30)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH09218375A (ja) 1996-02-08 1997-08-19 Canon Inc 疲労判定方法及びそれを用いた観察装置
KR100224889B1 (ko) * 1996-11-18 1999-10-15 윤종용 디스크 플레이어의 디스크 트레이 위치 감지장치
JP4971340B2 (ja) 2007-03-29 2012-07-11 パナソニック株式会社 炭化珪素半導体素子の製造方法
JP4974788B2 (ja) 2007-06-29 2012-07-11 キヤノン株式会社 画像処理装置、画像処理方法、プログラム、及び記憶媒体
KR20120093952A (ko) 2009-11-06 2012-08-23 가부시키가이샤 한도오따이 에네루기 켄큐쇼 반도체 소자 및 반도체 장치 제조 방법과, 성막 장치
JP2012073876A (ja) * 2010-09-29 2012-04-12 Fujitsu Ltd 電子機器、状態制御プログラム、および状態制御方法
JP2012212990A (ja) 2011-03-30 2012-11-01 Brother Ind Ltd ヘッドマウントディスプレイ
EP2807605A2 (en) * 2012-01-26 2014-12-03 Umoove Services Ltd. Eye tracking
KR101224889B1 (ko) * 2012-07-03 2013-01-22 한국해양과학기술원 선원 피로도 측정 방법 및 장치
WO2014046222A1 (en) 2012-09-24 2014-03-27 Semiconductor Energy Laboratory Co., Ltd. Display device
CN104013414B (zh) 2014-04-30 2015-12-30 深圳佑驾创新科技有限公司 一种基于移动智能手机的驾驶员疲劳检测系统
CN105513278A (zh) * 2014-09-23 2016-04-20 付宁婧 一种汽车疲劳驾驶预警器
CN106137181A (zh) * 2015-04-13 2016-11-23 上海帝仪科技有限公司 用于获取用户的疲劳特征的系统、方法和设备
JP6582604B2 (ja) * 2015-06-23 2019-10-02 富士通株式会社 瞳孔検出プログラム、瞳孔検出方法、瞳孔検出装置および視線検出システム
US9888845B2 (en) * 2015-06-30 2018-02-13 Antonio Visconti System and method for optical detection of cognitive impairment
JP2017064325A (ja) * 2015-10-02 2017-04-06 富士通株式会社 縮瞳検出方法及び縮瞳検出装置
US11194405B2 (en) 2015-10-08 2021-12-07 Panasonic Intellectual Property Corporation Of America Method for controlling information display apparatus, and information display apparatus
KR20170049997A (ko) * 2015-10-29 2017-05-11 에스케이플래닛 주식회사 안구 상태로부터 피로도를 판단하여 추천장소를 제공하는 웨어러블 디바이스, 추천장소 제공서버 및 방법
CN105354985B (zh) * 2015-11-04 2018-01-12 中国科学院上海高等研究院 疲劳驾驶监控装置及方法
JP2017130201A (ja) 2016-01-20 2017-07-27 株式会社半導体エネルギー研究所 入力システム、および電子機器
JP6714402B2 (ja) 2016-03-18 2020-06-24 国立大学法人大阪大学 眼疲労検査装置及び眼疲労検査方法
WO2018020368A1 (en) 2016-07-29 2018-02-01 Semiconductor Energy Laboratory Co., Ltd. Display method, display device, electronic device, non-temporary memory medium, and program
US20190159716A1 (en) 2016-08-03 2019-05-30 Akili Interactive Labs, Inc. Cognitive platform including computerized evocative elements
US11049147B2 (en) 2016-09-09 2021-06-29 Sony Corporation System and method for providing recommendation on an electronic device based on emotional state detection
WO2018100875A1 (ja) * 2016-11-30 2018-06-07 ソニー株式会社 情報処理装置、情報処理方法およびプログラム
WO2019025895A1 (ja) 2017-07-31 2019-02-07 株式会社半導体エネルギー研究所 表示装置
US10521661B2 (en) * 2017-09-01 2019-12-31 Magic Leap, Inc. Detailed eye shape model for robust biometric applications
CN111372735A (zh) * 2017-11-17 2020-07-03 三菱电机株式会社 3维空间监视装置、3维空间监视方法及3维空间监视程序
JP7350735B2 (ja) 2018-06-22 2023-09-26 株式会社半導体エネルギー研究所 画像表示装置
JP7014129B2 (ja) * 2018-10-29 2022-02-01 オムロン株式会社 推定器生成装置、モニタリング装置、推定器生成方法及び推定器生成プログラム

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2007295946A (ja) * 2006-04-27 2007-11-15 Konica Minolta Holdings Inc 酒気帯び検知システム及び酒気帯び検知方法
JP2014050649A (ja) * 2012-09-10 2014-03-20 Nippon Telegr & Teleph Corp <Ntt> 疲労感評価装置、疲労感評価方法、及びプログラム
JP2015132783A (ja) * 2014-01-16 2015-07-23 日本電信電話株式会社 音の顕著度推定装置、その方法、及びプログラム

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US12076154B2 (en) 2015-11-10 2024-09-03 Diagnosys LLC Method and apparatus for the assessment of electrophysiological signals
WO2023023226A3 (en) * 2021-08-18 2023-04-06 Diagnosys LLC System comprising integrated dichoptic flash and pupillometry monitoring, and method for using the same

Also Published As

Publication number Publication date
JP2024161623A (ja) 2024-11-19
JP7836865B2 (ja) 2026-03-27
US12333859B2 (en) 2025-06-17
US20250329192A1 (en) 2025-10-23
JP7548914B2 (ja) 2024-09-10
JPWO2021044249A1 (https=) 2021-03-11
US20220292880A1 (en) 2022-09-15

Similar Documents

Publication Publication Date Title
JP7548914B2 (ja) 情報処理装置
US11503998B1 (en) Method and a system for detection of eye gaze-pattern abnormalities and related neurological diseases
US11113842B2 (en) Method and apparatus with gaze estimation
US9076030B2 (en) Liveness detection
EP3271869B1 (fr) Procédé de traitement d&#39;un signal asynchrone
CN110211021B (zh) 图像处理装置、图像处理方法和存储介质
CN104182720A (zh) 瞳孔检测方法和装置
WO2020222785A1 (en) Facial action unit detection
CN110059666B (zh) 一种注意力检测方法及装置
JP2017522104A (ja) 目状態決定システム
JP2020170252A (ja) 画像処理装置、情報処理方法及びプログラム
JPWO2021044249A5 (https=)
EP3145405B1 (fr) Procédé de détermination d&#39;au moins un paramètre comportemental
JP7552698B2 (ja) 画像処理装置、画像処理方法、及び、記録媒体
JP6377545B2 (ja) 個人識別方法、個人識別装置、プログラム
CN112528713B (zh) 一种注视点估计方法、系统、处理器及设备
CN116311465A (zh) 筛查方法及相关设备
CN108495584A (zh) 用于通过触觉界面确定眼球运动的装置和方法
CA3177238C (en) A method and a system for detection of eye gaze-pattern abnormalities and related neurological diseases
FR3092423A1 (fr) Cartographie de saillance d&#39;images pendant une classification d&#39;images par intelligence artificielle
CN112073597A (zh) 视觉稳定度检测方法、装置、设备以及存储介质
Stankiewicz et al. Using performance efficiency for testing and optimization of visual attention models
CN112528714A (zh) 基于单光源的注视点估计方法、系统、处理器及设备
JP2017202048A (ja) 特徴量抽出装置、推定装置、それらの方法、およびプログラム
Masluha et al. Improving Classification of Medical Images Using ESRGAN-Based Upscaling and MobileNetV2

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20860113

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2021543605

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20860113

Country of ref document: EP

Kind code of ref document: A1

WWG Wipo information: grant in national office

Ref document number: 17638359

Country of ref document: US