JP6549734B2 - ビデオ・シーケンスにおける生体皮膚組織の識別 - Google Patents

ビデオ・シーケンスにおける生体皮膚組織の識別 Download PDF

Info

Publication number
JP6549734B2
JP6549734B2 JP2017559799A JP2017559799A JP6549734B2 JP 6549734 B2 JP6549734 B2 JP 6549734B2 JP 2017559799 A JP2017559799 A JP 2017559799A JP 2017559799 A JP2017559799 A JP 2017559799A JP 6549734 B2 JP6549734 B2 JP 6549734B2
Authority
JP
Japan
Prior art keywords
video
frame
similarity
subsequence
pixels
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Expired - Fee Related
Application number
JP2017559799A
Other languages
English (en)
Japanese (ja)
Other versions
JP2018515854A (ja
JP2018515854A5 (enExample
Inventor
ワン，ウェンジン
ハーン，ヘラルト デ
ハーン，ヘラルト デ
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Koninklijke Philips NV
Original Assignee
Koninklijke Philips NV
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Koninklijke Philips NV filed Critical Koninklijke Philips NV
Publication of JP2018515854A publication Critical patent/JP2018515854A/ja
Publication of JP2018515854A5 publication Critical patent/JP2018515854A5/ja
Application granted granted Critical
Publication of JP6549734B2 publication Critical patent/JP6549734B2/ja
Expired - Fee Related legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/02—Detecting, measuring or recording for evaluating the cardiovascular system, e.g. pulse, heart rate, blood pressure or blood flow
    • A61B5/024—Measuring pulse rate or heart rate
    • A61B5/02416—Measuring pulse rate or heart rate using photoplethysmograph signals, e.g. generated by infrared radiation
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/0059—Measuring for diagnostic purposes; Identification of persons using light, e.g. diagnosis by transillumination, diascopy, fluorescence
    • A61B5/0077—Devices for viewing the surface of the body, e.g. camera, magnifying lens
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/74—Details of notification to user or communication with user or patient; User input means
    • A61B5/7475—User input or interface means, e.g. keyboard, pointing device, joystick
    • A61B5/748—Selection of a region of interest, e.g. using a graphics tablet
    • A61B5/7485—Automatic selection of region of interest
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/213—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
    • G06F18/2134—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods based on separation criteria, e.g. independent component analysis
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/22—Matching criteria, e.g. proximity measures
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/762—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using clustering, e.g. of similar faces in social networks
    • G06V10/763—Non-hierarchical techniques, e.g. based on statistics of modelling distributions
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/7715—Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/12—Fingerprints or palmprints
    • G06V40/1382—Detecting the live character of the finger, i.e. distinguishing from a fake or cadaver finger
    • G06V40/1388—Detecting the live character of the finger, i.e. distinguishing from a fake or cadaver finger using image processing
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/15—Biometric patterns based on physiological signals, e.g. heartbeat, blood flow
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/161—Detection; Localisation; Normalisation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/161—Detection; Localisation; Normalisation
    • G06V40/162—Detection; Localisation; Normalisation using pixel segmentation or colour matching
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/168—Feature extraction; Face representation
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B2562/00—Details of sensors; Constructional details of sensor housings or probes; Accessories for sensors
    • A61B2562/02—Details of sensors specially adapted for in-vivo measurements
    • A61B2562/0233—Special features of optical sensors or probes classified in A61B5/00
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B2576/00—Medical imaging apparatus involving image processing or analysis
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2218/00—Aspects of pattern recognition specially adapted for signal processing
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2218/00—Aspects of pattern recognition specially adapted for signal processing
    • G06F2218/12—Classification; Matching
    • G06F2218/16—Classification; Matching by matching signal segments
    • G06F2218/18—Classification; Matching by matching signal segments by plotting the signal segments against each other, e.g. analysing scattergrams
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20021—Dividing image into blocks, subimages or windows
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30088—Skin; Dermal
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/40—Spoof detection, e.g. liveness detection
    • G06V40/45—Detection of the body part being alive
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H30/00—ICT specially adapted for the handling or processing of medical images
    • G16H30/40—ICT specially adapted for the handling or processing of medical images for processing medical images, e.g. editing

Landscapes

  • Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Health & Medical Sciences (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Medical Informatics (AREA)
  • Multimedia (AREA)
  • Biophysics (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • Human Computer Interaction (AREA)
  • Molecular Biology (AREA)
  • Pathology (AREA)
  • Veterinary Medicine (AREA)
  • Public Health (AREA)
  • Animal Behavior & Ethology (AREA)
  • Surgery (AREA)
  • Biomedical Technology (AREA)
  • Cardiology (AREA)
  • Software Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Physiology (AREA)
  • Computing Systems (AREA)
  • Databases & Information Systems (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Probability & Statistics with Applications (AREA)
  • Quality & Reliability (AREA)
  • Radiology & Medical Imaging (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Measuring And Recording Apparatus For Diagnosis (AREA)
  • Image Analysis (AREA)
JP2017559799A 2015-05-21 2016-05-20 ビデオ・シーケンスにおける生体皮膚組織の識別 Expired - Fee Related JP6549734B2 (ja)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
EP15168566 2015-05-21
EP15168566.6 2015-05-21
PCT/EP2016/061376 WO2016185004A1 (en) 2015-05-21 2016-05-20 Identifying living skin tissue in a video sequence

Publications (3)

Publication Number Publication Date
JP2018515854A JP2018515854A (ja) 2018-06-14
JP2018515854A5 JP2018515854A5 (enExample) 2019-01-31
JP6549734B2 true JP6549734B2 (ja) 2019-07-24

Family

ID=53189742

Family Applications (1)

Application Number Title Priority Date Filing Date
JP2017559799A Expired - Fee Related JP6549734B2 (ja) 2015-05-21 2016-05-20 ビデオ・シーケンスにおける生体皮膚組織の識別

Country Status (5)

Country Link
US (1) US10242441B2 (enExample)
EP (1) EP3298539A1 (enExample)
JP (1) JP6549734B2 (enExample)
CN (1) CN107646113A (enExample)
WO (1) WO2016185004A1 (enExample)

Families Citing this family (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP6665203B2 (ja) * 2015-05-21 2020-03-13 コーニンクレッカ フィリップス エヌ ヴェＫｏｎｉｎｋｌｉｊｋｅ Ｐｈｉｌｉｐｓ Ｎ．Ｖ． ビデオシーケンスからのパルス信号の決定
JP6343107B1 (ja) * 2015-05-21 2018-06-13 コーニンクレッカ フィリップス エヌ ヴェKoninklijke Philips N.V. ビデオ・シーケンスにおける生体皮膚組織の識別
US10242441B2 (en) * 2015-05-21 2019-03-26 Koninklijke Philips N.V. Identifying living skin tissue in a video sequence using color and spatial similarities
US10335045B2 (en) 2016-06-24 2019-07-02 Universita Degli Studi Di Trento Self-adaptive matrix completion for heart rate estimation from face videos under realistic conditions
US10515199B2 (en) 2017-04-19 2019-12-24 Qualcomm Incorporated Systems and methods for facial authentication
US11311202B2 (en) 2017-11-14 2022-04-26 Arizona Board Of Regents On Behalf Of Arizona State University Robust real-time heart rate monitoring method based on heartbeat harmonics using small-scale radar
JP6826733B2 (ja) * 2018-03-26 2021-02-10 国立大学法人東北大学 信号制御装置、信号制御プログラム、及び信号制御方法
WO2020072297A1 (en) 2018-10-03 2020-04-09 Arizona Board Of Regents On Behalf Of Arizona State University Direct rf signal processing for heart-rate monitoring using uwb impulse radar
US11783483B2 (en) 2019-03-19 2023-10-10 Arizona Board Of Regents On Behalf Of Arizona State University Detecting abnormalities in vital signs of subjects of videos
US11771380B2 (en) 2019-03-19 2023-10-03 Arizona Board Of Regents On Behalf Of Arizona State University Vital sign monitoring system using an optical sensor
EP3941342A4 (en) 2019-03-20 2022-12-07 Arizona Board of Regents on behalf of Arizona State University RADAR CARDIGRAPHY: PROCEDURE FOR ACCURATE RECONSTRUCTION OF HEART DATA
CN111860079B (zh) * 2019-04-30 2024-05-14 北京嘀嘀无限科技发展有限公司 活体图像检测方法、装置以及电子设备
US11356679B2 (en) * 2019-12-05 2022-06-07 Alibaba Group Holding Limited Method and apparatus for chroma sampling
US11317137B2 (en) * 2020-06-18 2022-04-26 Disney Enterprises, Inc. Supplementing entertainment content with ambient lighting
US12322205B1 (en) * 2021-04-02 2025-06-03 Affective Software, Inc. Methods and systems that determine heartbeat rate from video recordings
EP4098183A1 (en) 2021-06-02 2022-12-07 Koninklijke Philips N.V. Blood pressure detection
CN114913608B (zh) * 2022-06-17 2025-03-21 支付宝(杭州)信息技术有限公司 活体检测方法和系统
CN115410138A (zh) * 2022-11-02 2022-11-29 山东省人工智能研究院 基于特征编码器分类和定位串行的视频动作检测方法

Family Cites Families (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090043210A1 (en) * 2005-10-12 2009-02-12 Konica Minolta Holdings, Inc. Data detection device and data detection method
WO2008093321A1 (en) * 2007-02-01 2008-08-07 Yissum Research Development Company Of The Hebrew University Of Jerusalem Method and system for video indexing and video synopsis
JP2007295946A (ja) * 2006-04-27 2007-11-15 Konica Minolta Holdings Inc 酒気帯び検知システム及び酒気帯び検知方法
EP2486539B1 (en) * 2009-10-06 2016-09-07 Koninklijke Philips N.V. Method and system for obtaining a first signal for analysis to characterize at least one periodic component thereof
WO2011042858A1 (en) * 2009-10-06 2011-04-14 Koninklijke Philips Electronics N.V. Method and system for processing a signal including at least a component representative of a periodic phenomenon in a living being
CN101702198B (zh) * 2009-11-19 2011-11-23 浙江大学 基于背景比对的视频和活体人脸的鉴别方法
CN101807246A (zh) * 2010-03-17 2010-08-18 上海大学 多特征活体识别方法及装置
DE102011054658A1 (de) * 2011-10-20 2013-04-25 Bioid Ag Verfahren zur Unterscheidung zwischen einem realen Gesicht und einer zweidimensionalen Abbildung des Gesichts in einem biometrischen Erfassungsprozess
CN102622588B (zh) * 2012-03-08 2013-10-09 无锡中科奥森科技有限公司 双验证人脸防伪方法及装置
US8768438B2 (en) * 2012-06-25 2014-07-01 Xerox Corporation Determining cardiac arrhythmia from a video of a subject being monitored for cardiac function
BR112015006006A2 (pt) * 2012-09-21 2017-07-04 Koninklijke Philips Nv dispositivo para extrair informações fisiológicas da radiação eletromagnética remotamente detectada emitida ou refletida por um sujeito, e método para extrair informações fisiológicas da radiação eletromagnética remotamente detectada emitida ou refletida por um sujeito
US9336594B2 (en) * 2014-03-07 2016-05-10 Xerox Corporation Cardiac pulse rate estimation from source video data
CN104239869B (zh) * 2014-09-25 2018-03-16 武汉华和机电技术有限公司 一种智能指纹识别装置及方法
US10242441B2 (en) * 2015-05-21 2019-03-26 Koninklijke Philips N.V. Identifying living skin tissue in a video sequence using color and spatial similarities
JP6343107B1 (ja) * 2015-05-21 2018-06-13 コーニンクレッカ フィリップス エヌ ヴェKoninklijke Philips N.V. ビデオ・シーケンスにおける生体皮膚組織の識別
JP6665203B2 (ja) * 2015-05-21 2020-03-13 コーニンクレッカ フィリップス エヌ ヴェＫｏｎｉｎｋｌｉｊｋｅ Ｐｈｉｌｉｐｓ Ｎ．Ｖ． ビデオシーケンスからのパルス信号の決定

Also Published As

Publication number Publication date
EP3298539A1 (en) 2018-03-28
JP2018515854A (ja) 2018-06-14
WO2016185004A1 (en) 2016-11-24
US10242441B2 (en) 2019-03-26
CN107646113A (zh) 2018-01-30
US20160343130A1 (en) 2016-11-24

Similar Documents

Publication Publication Date Title
JP6549734B2 (ja) ビデオ・シーケンスにおける生体皮膚組織の識別
US10290104B2 (en) Determining a pulse signal from concatenated video sub-sequences generated from a video sequence
JP6343107B1 (ja) ビデオ・シーケンスにおける生体皮膚組織の識別
Sun et al. Contrast-phys: Unsupervised video-based remote physiological measurement via spatiotemporal contrast
Boccignone et al. An open framework for remote-PPG methods and their assessment
Gupta et al. Radiant: Better rppg estimation using signal embeddings and transformer
Wang et al. Unsupervised subject detection via remote PPG
Wang et al. Living-skin classification via remote-PPG
Botina-Monsalve et al. RTrPPG: An ultra light 3DCNN for real-time remote photoplethysmography
Maraci et al. Searching for structures of interest in an ultrasound video sequence
Speth et al. Unifying frame rate and temporal dilations for improved remote pulse detection
Macwan et al. Periodic variance maximization using generalized eigenvalue decomposition applied to remote photoplethysmography estimation
CN106203255B (zh) 一种基于时间对齐的行人重识别方法及系统
Lampier et al. A deep learning approach to estimate pulse rate by remote photoplethysmography
Wang et al. Accurate face alignment and adaptive patch selection for heart rate estimation from videos under realistic scenarios
CN118247708A (zh) 一种去除非规律性运动伪迹的非接触式心率检测方法
Ibrahim et al. Analysis of non-invasive video based heart rate monitoring system obtained from various distances and different facial spot
Srinivasan et al. Frequency based gait gender identification
Li A temporal encoder-decoder approach to extracting blood volume pulse signal morphology from face videos
Tu et al. Standardized rPPG signal generation based on generative adversarial networks
Gupta et al. Heart-Rate Evaluation Using Remote Photoplethysmography–A Case Study
Luukkonen Exploring remote photoplethysmography signals for deepfake detection in facial videos
Zhang et al. Adaptive projection plane and feature selection for robust non-contact heart rate detection
Demirezen Remote Heart Rate Estimation Using Non-Contact Photoplethysmography
Feng et al. Remote Heart Rate Estimation Based on Convolutional Neural Network and Regional Adaptive Weighting

Legal Events

Date Code Title Description
A521 Request for written amendment filed

Free format text: JAPANESE INTERMEDIATE CODE: A523

Effective date: 20171122

A521 Request for written amendment filed

Free format text: JAPANESE INTERMEDIATE CODE: A523

Effective date: 20181212

A621 Written request for application examination

Free format text: JAPANESE INTERMEDIATE CODE: A621

Effective date: 20181212

A871 Explanation of circumstances concerning accelerated examination

Free format text: JAPANESE INTERMEDIATE CODE: A871

Effective date: 20181212

A975 Report on accelerated examination

Free format text: JAPANESE INTERMEDIATE CODE: A971005

Effective date: 20181225

A131 Notification of reasons for refusal

Free format text: JAPANESE INTERMEDIATE CODE: A131

Effective date: 20190305

A521 Request for written amendment filed

Free format text: JAPANESE INTERMEDIATE CODE: A523

Effective date: 20190411

TRDD Decision of grant or rejection written
A01 Written decision to grant a patent or to grant a registration (utility model)

Free format text: JAPANESE INTERMEDIATE CODE: A01

Effective date: 20190604

A61 First payment of annual fees (during grant procedure)

Free format text: JAPANESE INTERMEDIATE CODE: A61

Effective date: 20190627

R150 Certificate of patent or registration of utility model

Ref document number: 6549734

Country of ref document: JP

Free format text: JAPANESE INTERMEDIATE CODE: R150

LAPS Cancellation because of no payment of annual fees