JP6665203B2 - ビデオシーケンスからのパルス信号の決定 - Google Patents

ビデオシーケンスからのパルス信号の決定 Download PDF

Info

Publication number
JP6665203B2
JP6665203B2 JP2017560151A JP2017560151A JP6665203B2 JP 6665203 B2 JP6665203 B2 JP 6665203B2 JP 2017560151 A JP2017560151 A JP 2017560151A JP 2017560151 A JP2017560151 A JP 2017560151A JP 6665203 B2 JP6665203 B2 JP 6665203B2
Authority
JP
Japan
Prior art keywords
frame
video
image frames
subset
pixels
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Expired - Fee Related
Application number
JP2017560151A
Other languages
English (en)
Japanese (ja)
Other versions
JP2018520418A (ja
JP2018520418A5 (OSRAM
Inventor
ハーン ジェラルド デ
ハーン ジェラルド デ
ウェンジン ワン
ウェンジン ワン
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Koninklijke Philips NV
Original Assignee
Koninklijke Philips NV
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Koninklijke Philips NV filed Critical Koninklijke Philips NV
Publication of JP2018520418A publication Critical patent/JP2018520418A/ja
Publication of JP2018520418A5 publication Critical patent/JP2018520418A5/ja
Application granted granted Critical
Publication of JP6665203B2 publication Critical patent/JP6665203B2/ja
Expired - Fee Related legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/0059—Measuring for diagnostic purposes; Identification of persons using light, e.g. diagnosis by transillumination, diascopy, fluorescence
    • A61B5/0077—Devices for viewing the surface of the body, e.g. camera, magnifying lens
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/02—Detecting, measuring or recording for evaluating the cardiovascular system, e.g. pulse, heart rate, blood pressure or blood flow
    • A61B5/024—Measuring pulse rate or heart rate
    • A61B5/02416—Measuring pulse rate or heart rate using photoplethysmograph signals, e.g. generated by infrared radiation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/213—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
    • G06F18/2134—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods based on separation criteria, e.g. independent component analysis
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/22—Matching criteria, e.g. proximity measures
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G06T7/0014—Biomedical image inspection using an image reference approach
    • G06T7/0016—Biomedical image inspection using an image reference approach involving temporal comparison
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/56—Extraction of image or video features relating to colour
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/762—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using clustering, e.g. of similar faces in social networks
    • G06V10/763—Non-hierarchical techniques, e.g. based on statistics of modelling distributions
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/7715—Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/12—Fingerprints or palmprints
    • G06V40/1382—Detecting the live character of the finger, i.e. distinguishing from a fake or cadaver finger
    • G06V40/1388—Detecting the live character of the finger, i.e. distinguishing from a fake or cadaver finger using image processing
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/15—Biometric patterns based on physiological signals, e.g. heartbeat, blood flow
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/161—Detection; Localisation; Normalisation
    • G06V40/162—Detection; Localisation; Normalisation using pixel segmentation or colour matching
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2218/00—Aspects of pattern recognition specially adapted for signal processing
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20216—Image averaging
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30076—Plethysmography
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/40—Spoof detection, e.g. liveness detection
    • G06V40/45—Detection of the body part being alive

Landscapes

  • Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • General Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • Medical Informatics (AREA)
  • Biophysics (AREA)
  • Cardiology (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • Human Computer Interaction (AREA)
  • Animal Behavior & Ethology (AREA)
  • Surgery (AREA)
  • Molecular Biology (AREA)
  • Public Health (AREA)
  • Veterinary Medicine (AREA)
  • Biomedical Technology (AREA)
  • Pathology (AREA)
  • Data Mining & Analysis (AREA)
  • Software Systems (AREA)
  • Physiology (AREA)
  • Databases & Information Systems (AREA)
  • Computing Systems (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Radiology & Medical Imaging (AREA)
  • Quality & Reliability (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Probability & Statistics with Applications (AREA)
  • Image Analysis (AREA)
  • Measuring Pulse, Heart Rate, Blood Pressure Or Blood Flow (AREA)
JP2017560151A 2015-05-21 2016-05-09 ビデオシーケンスからのパルス信号の決定 Expired - Fee Related JP6665203B2 (ja)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
EP15168575 2015-05-21
EP15168575.7 2015-05-21
PCT/EP2016/060250 WO2016184705A1 (en) 2015-05-21 2016-05-09 Determining a pulse signal from a video sequence

Publications (3)

Publication Number Publication Date
JP2018520418A JP2018520418A (ja) 2018-07-26
JP2018520418A5 JP2018520418A5 (OSRAM) 2019-04-11
JP6665203B2 true JP6665203B2 (ja) 2020-03-13

Family

ID=53268656

Family Applications (1)

Application Number Title Priority Date Filing Date
JP2017560151A Expired - Fee Related JP6665203B2 (ja) 2015-05-21 2016-05-09 ビデオシーケンスからのパルス信号の決定

Country Status (5)

Country Link
US (1) US10290104B2 (OSRAM)
EP (1) EP3298536A1 (OSRAM)
JP (1) JP6665203B2 (OSRAM)
CN (1) CN107666853A (OSRAM)
WO (1) WO2016184705A1 (OSRAM)

Families Citing this family (36)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9609385B2 (en) * 2014-08-28 2017-03-28 The Nielsen Company (Us), Llc Methods and apparatus to detect people
WO2016184703A1 (en) * 2015-05-21 2016-11-24 Koninklijke Philips N.V. Identifying living skin tissue in a video sequence
JP6665203B2 (ja) * 2015-05-21 2020-03-13 コーニンクレッカ フィリップス エヌ ヴェＫｏｎｉｎｋｌｉｊｋｅ Ｐｈｉｌｉｐｓ Ｎ．Ｖ． ビデオシーケンスからのパルス信号の決定
US10242441B2 (en) * 2015-05-21 2019-03-26 Koninklijke Philips N.V. Identifying living skin tissue in a video sequence using color and spatial similarities
US11328152B2 (en) 2019-06-17 2022-05-10 Pixart Imaging Inc. Recognition system employing thermal sensor
US10335045B2 (en) 2016-06-24 2019-07-02 Universita Degli Studi Di Trento Self-adaptive matrix completion for heart rate estimation from face videos under realistic conditions
US10709354B2 (en) * 2016-09-06 2020-07-14 Photorithm, Inc. Generating a breathing alert
US11389119B2 (en) * 2016-09-06 2022-07-19 Photorithm, Inc. Generating a breathing alert
CN108073933B (zh) * 2016-11-08 2021-05-25 杭州海康威视数字技术股份有限公司 一种目标检测方法及装置
JP6767247B2 (ja) * 2016-11-29 2020-10-14 株式会社日立製作所 生体情報検出装置及び生体情報検出方法
JP2020508713A (ja) * 2017-02-27 2020-03-26 コーニンクレッカ フィリップス エヌ ヴェＫｏｎｉｎｋｌｉｊｋｅ Ｐｈｉｌｉｐｓ Ｎ．Ｖ． 信号変動増幅による静脈穿刺及び動脈ラインガイド
EP3440991A1 (en) 2017-08-08 2019-02-13 Koninklijke Philips N.V. Device, system and method for determining a physiological parameter of a subject
DE102017118636B3 (de) 2017-08-15 2018-12-06 Technische Universität Chemnitz Verfahren zur kontaktlosen Bestimmung der Herzfrequenz einer Person
US11311202B2 (en) 2017-11-14 2022-04-26 Arizona Board Of Regents On Behalf Of Arizona State University Robust real-time heart rate monitoring method based on heartbeat harmonics using small-scale radar
EP3749180A4 (en) * 2018-02-08 2021-11-24 Gill R. Tsouri OPPORTUNISTIC PLETHYSMOGRAPHY USING VIDEO CAMERAS
CN110236509A (zh) * 2018-03-07 2019-09-17 台北科技大学 于视频中实时分析生理特征的方法
TWI712909B (zh) * 2018-04-17 2020-12-11 和碩聯合科技股份有限公司 輔助臉部辨識的防僞方法及系統
US11259710B2 (en) * 2018-05-16 2022-03-01 Mitsubishi Electric Research Laboratories, Inc. System and method for remote measurements of vital signs
EP3570186B1 (en) * 2018-05-17 2021-11-17 FRAUNHOFER-GESELLSCHAFT zur Förderung der angewandten Forschung e.V. Device and method for detecting partial matches between a first time varying signal and a second time varying signal
CN111374646B (zh) * 2018-12-28 2021-08-03 清华大学 一种基于智能手机的非硬件依赖型脉象信息采集系统和方法
US11771380B2 (en) 2019-03-19 2023-10-03 Arizona Board Of Regents On Behalf Of Arizona State University Vital sign monitoring system using an optical sensor
US11783483B2 (en) 2019-03-19 2023-10-10 Arizona Board Of Regents On Behalf Of Arizona State University Detecting abnormalities in vital signs of subjects of videos
EP3941342A4 (en) 2019-03-20 2022-12-07 Arizona Board of Regents on behalf of Arizona State University RADAR CARDIGRAPHY: PROCEDURE FOR ACCURATE RECONSTRUCTION OF HEART DATA
KR102444459B1 (ko) * 2019-11-21 2022-09-19 주식회사 지비소프트 비접촉식 졸음감지 방법 및 장치 방법
WO2021100994A1 (ko) * 2019-11-21 2021-05-27 주식회사 지비소프트 비접촉식 생체 지수 측정 방법
US11103144B2 (en) 2019-11-21 2021-08-31 Gb Soft Inc. Method of measuring physiological parameter of subject in contactless manner
KR20220104052A (ko) 2019-12-05 2022-07-25 알리바바 그룹 홀딩 리미티드 크로마 샘플링을 위한 방법 및 장치
EP3989106B1 (en) * 2020-10-26 2024-05-15 Robert Bosch GmbH Unsupervised training of a video feature extractor
KR102601683B1 (ko) * 2021-02-25 2023-11-13 주식회사 지비소프트 생체 지수 측정 방법
TWI861371B (zh) * 2021-03-17 2024-11-11 鉅怡智慧股份有限公司 利用非接觸式生理量測技術之健康管理系統
US12322205B1 (en) * 2021-04-02 2025-06-03 Affective Software, Inc. Methods and systems that determine heartbeat rate from video recordings
CN113420612B (zh) * 2021-06-02 2022-03-18 深圳中集智能科技有限公司 基于机器视觉的生产节拍计算方法
EP4098183A1 (en) 2021-06-02 2022-12-07 Koninklijke Philips N.V. Blood pressure detection
CN113378798B (zh) * 2021-07-16 2023-06-23 福建师范大学 基于视频微运动放大的脉搏视频脉搏提取方法及系统
CN116311175A (zh) * 2021-12-10 2023-06-23 上海商汤临港智能科技有限公司 一种生理状态检测方法、装置、电子设备及存储介质
CN117643461B (zh) * 2024-01-30 2024-04-02 吉林大学 基于人工智能的心率智能监测系统及方法

Family Cites Families (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2008056638A1 (fr) * 2006-11-06 2008-05-15 Fujifilm Ri Pharma Co., Ltd. Procédé d'aide au diagnostic à partir d'une image du cerveau, logiciel, et procédé d'enregistrement
CN101689394B (zh) * 2007-02-01 2014-03-26 耶路撒冷希伯来大学伊森姆研究发展有限公司 用于视频索引和视频概要的方法和系统
US8494608B2 (en) * 2008-04-18 2013-07-23 Medtronic, Inc. Method and apparatus for mapping a structure
EP3142071A1 (en) * 2009-10-06 2017-03-15 Koninklijke Philips N.V. Method and system for obtaining a first signal for analysis to characterize at least one periodic component thereof
JP5856960B2 (ja) * 2009-10-06 2016-02-10 コーニンクレッカ フィリップス エヌ ヴェＫｏｎｉｎｋｌｉｊｋｅ Ｐｈｉｌｉｐｓ Ｎ．Ｖ． 第１の信号の少なくとも一つの周期的成分を特徴付けるための分析のため第１の信号を得るための方法及びシステム
CN101695136B (zh) * 2009-10-22 2011-07-20 北京交通大学 一种自动式视频颜色协调处理方法和处理系统
JP5195741B2 (ja) * 2009-12-25 2013-05-15 株式会社デンソー 生体活動計測装置
KR101642897B1 (ko) * 2011-07-13 2016-07-26 주식회사 고영테크놀러지 검사방법
US8838209B2 (en) * 2012-02-21 2014-09-16 Xerox Corporation Deriving arterial pulse transit time from a source video image
US8768438B2 (en) * 2012-06-25 2014-07-01 Xerox Corporation Determining cardiac arrhythmia from a video of a subject being monitored for cardiac function
US9412024B2 (en) * 2013-09-13 2016-08-09 Interra Systems, Inc. Visual descriptors based video quality assessment using outlier model
US9336594B2 (en) * 2014-03-07 2016-05-10 Xerox Corporation Cardiac pulse rate estimation from source video data
WO2016184703A1 (en) * 2015-05-21 2016-11-24 Koninklijke Philips N.V. Identifying living skin tissue in a video sequence
US10242441B2 (en) * 2015-05-21 2019-03-26 Koninklijke Philips N.V. Identifying living skin tissue in a video sequence using color and spatial similarities
JP6665203B2 (ja) * 2015-05-21 2020-03-13 コーニンクレッカ フィリップス エヌ ヴェＫｏｎｉｎｋｌｉｊｋｅ Ｐｈｉｌｉｐｓ Ｎ．Ｖ． ビデオシーケンスからのパルス信号の決定

Also Published As

Publication number Publication date
CN107666853A (zh) 2018-02-06
US20160343135A1 (en) 2016-11-24
JP2018520418A (ja) 2018-07-26
EP3298536A1 (en) 2018-03-28
WO2016184705A1 (en) 2016-11-24
US10290104B2 (en) 2019-05-14

Similar Documents

Publication Publication Date Title
JP6665203B2 (ja) ビデオシーケンスからのパルス信号の決定
JP6549734B2 (ja) ビデオ・シーケンスにおける生体皮膚組織の識別
JP6343107B1 (ja) ビデオ・シーケンスにおける生体皮膚組織の識別
EP3057487B1 (en) Device and method for obtaining a vital sign of a subject
Botina-Monsalve et al. RTrPPG: An ultra light 3DCNN for real-time remote photoplethysmography
Wang et al. Unsupervised subject detection via remote PPG
Wang et al. Living-skin classification via remote-PPG
JP6521845B2 (ja) 心拍に連動する周期的変動の計測装置及び計測方法
EP3440991A1 (en) Device, system and method for determining a physiological parameter of a subject
Tran et al. A robust real time system for remote heart rate measurement via camera
Speth et al. Unifying frame rate and temporal dilations for improved remote pulse detection
Macwan et al. Periodic variance maximization using generalized eigenvalue decomposition applied to remote photoplethysmography estimation
Jaiswal et al. Heart rate estimation network from facial videos using spatiotemporal feature image
WO2023055862A1 (en) Systems and methods for platform-agnostic, real-time physiologic vital sign detection from video stream data
Akamatsu et al. Heart rate and oxygen saturation estimation from facial video with multimodal physiological data generation
Li A temporal encoder-decoder approach to extracting blood volume pulse signal morphology from face videos
Slapnicar et al. Contact-free monitoring of physiological parameters in people with profound intellectual and multiple disabilities
Tu et al. Standardized rPPG signal generation based on generative adversarial networks
Raseena et al. A maximum likelihood formulation to exploit heart rate variability for robust heart rate estimation from facial video
Jian et al. Remote Heart Rate Estimation Based on ROI Extraction
Gupta et al. Heart-Rate Evaluation Using Remote Photoplethysmography–A Case Study
WO2024246626A1 (en) System and method for estimating at least one physiological sign of a subject in dark environments

Legal Events

Date Code Title Description
A521 Request for written amendment filed

Free format text: JAPANESE INTERMEDIATE CODE: A523

Effective date: 20190228

A621 Written request for application examination

Free format text: JAPANESE INTERMEDIATE CODE: A621

Effective date: 20190228

A871 Explanation of circumstances concerning accelerated examination

Free format text: JAPANESE INTERMEDIATE CODE: A871

Effective date: 20190228

A977 Report on retrieval

Free format text: JAPANESE INTERMEDIATE CODE: A971007

Effective date: 20190711

A975 Report on accelerated examination

Free format text: JAPANESE INTERMEDIATE CODE: A971005

Effective date: 20190806

A131 Notification of reasons for refusal

Free format text: JAPANESE INTERMEDIATE CODE: A131

Effective date: 20190819

TRDD Decision of grant or rejection written
A01 Written decision to grant a patent or to grant a registration (utility model)

Free format text: JAPANESE INTERMEDIATE CODE: A01

Effective date: 20200121

A61 First payment of annual fees (during grant procedure)

Free format text: JAPANESE INTERMEDIATE CODE: A61

Effective date: 20200219

R150 Certificate of patent or registration of utility model

Ref document number: 6665203

Country of ref document: JP

Free format text: JAPANESE INTERMEDIATE CODE: R150

LAPS Cancellation because of no payment of annual fees