JP6323184B2 - 画像処理装置、画像処理方法、並びにプログラム - Google Patents

画像処理装置、画像処理方法、並びにプログラム Download PDF

Info

Publication number
JP6323184B2
JP6323184B2 JP2014115769A JP2014115769A JP6323184B2 JP 6323184 B2 JP6323184 B2 JP 6323184B2 JP 2014115769 A JP2014115769 A JP 2014115769A JP 2014115769 A JP2014115769 A JP 2014115769A JP 6323184 B2 JP6323184 B2 JP 6323184B2
Authority
JP
Japan
Prior art keywords
image
surgical instrument
forceps
unit
surgical
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
JP2014115769A
Other languages
English (en)
Japanese (ja)
Other versions
JP2015228955A (ja
JP2015228955A5 (cg-RX-API-DMAC7.html
Inventor
岳志 宮井
岳志 宮井
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sony Corp
Original Assignee
Sony Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sony Corp filed Critical Sony Corp
Priority to JP2014115769A priority Critical patent/JP6323184B2/ja
Priority to PCT/JP2015/002748 priority patent/WO2015186335A1/en
Priority to US15/306,775 priority patent/US10827906B2/en
Priority to EP15728186.6A priority patent/EP3151719B1/en
Publication of JP2015228955A publication Critical patent/JP2015228955A/ja
Publication of JP2015228955A5 publication Critical patent/JP2015228955A5/ja
Application granted granted Critical
Publication of JP6323184B2 publication Critical patent/JP6323184B2/ja
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/00002—Operational features of endoscopes
    • A61B1/00004—Operational features of endoscopes characterised by electronic signal processing
    • A61B1/00009—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope
    • A61B1/000095—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope for image enhancement
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/00002—Operational features of endoscopes
    • A61B1/00043—Operational features of endoscopes provided with output arrangements
    • A61B1/00045—Display arrangement
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/00163—Optical arrangements
    • A61B1/00193—Optical arrangements adapted for stereoscopic vision
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/00163—Optical arrangements
    • A61B1/00194—Optical arrangements adapted for three-dimensional imaging
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B23/00—Telescopes, e.g. binoculars; Periscopes; Instruments for viewing the inside of hollow bodies; Viewfinders; Optical aiming or sighting devices
    • G02B23/24—Instruments or systems for viewing the inside of hollow bodies, e.g. fibrescopes
    • G02B23/2407—Optical details
    • G02B23/2415—Stereoscopic endoscopes
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B23/00—Telescopes, e.g. binoculars; Periscopes; Instruments for viewing the inside of hollow bodies; Viewfinders; Optical aiming or sighting devices
    • G02B23/24—Instruments or systems for viewing the inside of hollow bodies, e.g. fibrescopes
    • G02B23/2476—Non-optical details, e.g. housings, mountings, supports
    • G02B23/2484—Arrangements in relation to a camera or imaging device
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B19/00—Programme-control systems
    • G05B19/02—Programme-control systems electric
    • G05B19/18—Numerical control [NC], i.e. automatically operating machines, in particular machine tools, e.g. in a manufacturing environment, so as to execute positioning, movement or co-ordinated operations by means of programme data in numerical form
    • G05B19/402—Numerical control [NC], i.e. automatically operating machines, in particular machine tools, e.g. in a manufacturing environment, so as to execute positioning, movement or co-ordinated operations by means of programme data in numerical form characterised by control arrangements for positioning, e.g. centring a tool relative to a hole in the workpiece, additional detection means to correct position
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H30/00—ICT specially adapted for the handling or processing of medical images
    • G16H30/20—ICT specially adapted for the handling or processing of medical images for handling medical images, e.g. DICOM, HL7 or PACS
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H30/00—ICT specially adapted for the handling or processing of medical images
    • G16H30/40—ICT specially adapted for the handling or processing of medical images for processing medical images, e.g. editing
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/204—Image signal generators using stereoscopic image cameras
    • H04N13/239—Image signal generators using stereoscopic image cameras using two 2D image sensors having a relative position equal to or related to the interocular distance
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/50—Constructional details
    • H04N23/555—Constructional details for picking-up images in sites, inaccessible due to their dimensions or hazardous conditions, e.g. endoscopes or borescopes
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/56—Cameras or camera modules comprising electronic image sensors; Control thereof provided with illuminating means
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/63—Control of cameras or camera modules by using electronic viewfinders
    • H04N23/633—Control of cameras or camera modules by using electronic viewfinders for displaying additional information relating to control or operation of the camera
    • H04N23/635—Region indicators; Field of view indicators
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/67—Focus control based on electronic image sensor signals
    • H04N23/673—Focus control based on electronic image sensor signals based on contrast or high frequency components of image signals, e.g. hill climbing method
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B2219/00—Program-control systems
    • G05B2219/30—Nc systems
    • G05B2219/45—Nc applications
    • G05B2219/45123—Electrogoniometer, neuronavigator, medical robot used by surgeon to operate
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N2013/0074—Stereoscopic image analysis
    • H04N2013/0081—Depth or disparity estimation from stereoscopic image signals

Landscapes

  • Health & Medical Sciences (AREA)
  • Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Surgery (AREA)
  • Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • Radiology & Medical Imaging (AREA)
  • Medical Informatics (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Public Health (AREA)
  • Optics & Photonics (AREA)
  • Signal Processing (AREA)
  • Multimedia (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Pathology (AREA)
  • Molecular Biology (AREA)
  • Animal Behavior & Ethology (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Veterinary Medicine (AREA)
  • Epidemiology (AREA)
  • Primary Health Care (AREA)
  • General Physics & Mathematics (AREA)
  • Astronomy & Astrophysics (AREA)
  • Human Computer Interaction (AREA)
  • Manufacturing & Machinery (AREA)
  • Automation & Control Theory (AREA)
  • Endoscopes (AREA)
JP2014115769A 2014-06-04 2014-06-04 画像処理装置、画像処理方法、並びにプログラム Active JP6323184B2 (ja)

Priority Applications (4)

Application Number Priority Date Filing Date Title
JP2014115769A JP6323184B2 (ja) 2014-06-04 2014-06-04 画像処理装置、画像処理方法、並びにプログラム
PCT/JP2015/002748 WO2015186335A1 (en) 2014-06-04 2015-06-01 Image processing apparatus, image processing method, and program
US15/306,775 US10827906B2 (en) 2014-06-04 2015-06-01 Endoscopic surgery image processing apparatus, image processing method, and program
EP15728186.6A EP3151719B1 (en) 2014-06-04 2015-06-01 Image processing apparatus and program

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
JP2014115769A JP6323184B2 (ja) 2014-06-04 2014-06-04 画像処理装置、画像処理方法、並びにプログラム

Publications (3)

Publication Number Publication Date
JP2015228955A JP2015228955A (ja) 2015-12-21
JP2015228955A5 JP2015228955A5 (cg-RX-API-DMAC7.html) 2017-02-23
JP6323184B2 true JP6323184B2 (ja) 2018-05-16

Family

ID=53373516

Family Applications (1)

Application Number Title Priority Date Filing Date
JP2014115769A Active JP6323184B2 (ja) 2014-06-04 2014-06-04 画像処理装置、画像処理方法、並びにプログラム

Country Status (4)

Country Link
US (1) US10827906B2 (cg-RX-API-DMAC7.html)
EP (1) EP3151719B1 (cg-RX-API-DMAC7.html)
JP (1) JP6323184B2 (cg-RX-API-DMAC7.html)
WO (1) WO2015186335A1 (cg-RX-API-DMAC7.html)

Families Citing this family (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP6707083B2 (ja) * 2015-06-29 2020-06-10 オリンパス株式会社 画像処理装置、内視鏡システム、画像処理方法、及び画像処理プログラム
JP6753081B2 (ja) 2016-03-09 2020-09-09 ソニー株式会社 内視鏡手術システム、画像処理方法及び医療用観察システム
JP2017164007A (ja) * 2016-03-14 2017-09-21 ソニー株式会社 医療用画像処理装置、医療用画像処理方法、プログラム
US11653853B2 (en) * 2016-11-29 2023-05-23 Biosense Webster (Israel) Ltd. Visualization of distances to walls of anatomical cavities
US11418700B2 (en) * 2017-03-27 2022-08-16 Sony Olympus Medical Solutions Inc. Control device, endoscope system, processing method, and program
JPWO2019181212A1 (ja) * 2018-03-23 2021-03-11 ソニー・オリンパスメディカルソリューションズ株式会社 医療用表示制御装置、および表示制御方法
WO2019187502A1 (ja) * 2018-03-29 2019-10-03 ソニー株式会社 画像処理装置と画像処理方法およびプログラム
US11147629B2 (en) * 2018-06-08 2021-10-19 Acclarent, Inc. Surgical navigation system with automatically driven endoscope
JP7392654B2 (ja) * 2018-10-18 2023-12-06 ソニーグループ株式会社 医療用観察システム、医療用観察装置及び医療用観察方法
JP7286948B2 (ja) * 2018-11-07 2023-06-06 ソニーグループ株式会社 医療用観察システム、信号処理装置及び医療用観察方法
US12210665B2 (en) * 2019-02-12 2025-01-28 Intuitive Surgical Operations, Inc. Systems and methods for facilitating optimization of an imaging device viewpoint during an operating session of a computer-assisted operation system
US11625834B2 (en) * 2019-11-08 2023-04-11 Sony Group Corporation Surgical scene assessment based on computer vision
EP4205691A4 (en) * 2020-10-15 2024-02-21 Sony Olympus Medical Solutions Inc. MEDICAL IMAGE PROCESSING DEVICE AND MEDICAL OBSERVATION SYSTEM

Family Cites Families (40)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP3036287B2 (ja) * 1992-12-15 2000-04-24 富士ゼロックス株式会社 動画像シーン検出装置
JPH07154687A (ja) * 1993-11-29 1995-06-16 Olympus Optical Co Ltd 画像合成装置
US5649021A (en) * 1995-06-07 1997-07-15 David Sarnoff Research Center, Inc. Method and system for object detection for instrument control
DE19529950C1 (de) * 1995-08-14 1996-11-14 Deutsche Forsch Luft Raumfahrt Verfahren zum Nachführen eines Stereo-Laparoskops in der minimalinvasiven Chirurgie
JPH11309A (ja) * 1997-06-12 1999-01-06 Hitachi Ltd 画像処理装置
JP2006215105A (ja) * 2005-02-01 2006-08-17 Fuji Photo Film Co Ltd 撮像装置
KR100567391B1 (ko) 2005-02-04 2006-04-04 국방과학연구소 수은램프와 할로겐램프 조합 방식을 이용한 솔라 시뮬레이터
JP4980625B2 (ja) * 2006-02-21 2012-07-18 富士フイルム株式会社 体腔内観察装置
US9718190B2 (en) * 2006-06-29 2017-08-01 Intuitive Surgical Operations, Inc. Tool position and identification indicator displayed in a boundary area of a computer display screen
US7794396B2 (en) * 2006-11-03 2010-09-14 Stryker Corporation System and method for the automated zooming of a surgical camera
US9826200B2 (en) * 2007-04-30 2017-11-21 Mobileye Vision Technologies Ltd. Rear obstruction detection
US8264542B2 (en) * 2007-12-31 2012-09-11 Industrial Technology Research Institute Methods and systems for image processing in a multiview video system
JP5410021B2 (ja) * 2008-01-22 2014-02-05 株式会社日立メディコ 医用画像診断装置
JP5385163B2 (ja) 2010-01-06 2014-01-08 オリンパスメディカルシステムズ株式会社 内視鏡システム
JP5380348B2 (ja) * 2010-03-31 2014-01-08 富士フイルム株式会社 内視鏡観察を支援するシステムおよび方法、並びに、装置およびプログラム
JP5551957B2 (ja) * 2010-03-31 2014-07-16 富士フイルム株式会社 投影画像生成装置およびその作動方法、並びに投影画像生成プログラム
JP5535725B2 (ja) * 2010-03-31 2014-07-02 富士フイルム株式会社 内視鏡観察支援システム、並びに、内視鏡観察支援装置、その作動方法およびプログラム
JP2012075507A (ja) * 2010-09-30 2012-04-19 Panasonic Corp 手術用カメラ
JP2012075508A (ja) * 2010-09-30 2012-04-19 Panasonic Corp 手術用カメラ
DE112010006052T5 (de) * 2010-12-08 2013-10-10 Industrial Technology Research Institute Verfahren zum Erzeugen stereoskopischer Ansichten von monoskopischen Endoskopbildern und Systeme, die diese verwenden
WO2012127618A1 (ja) * 2011-03-22 2012-09-27 株式会社モルフォ 動体検出装置、動体検出方法、動体検出プログラム、動体追跡装置、動体追跡方法及び動体追跡プログラム
JP5855358B2 (ja) * 2011-05-27 2016-02-09 オリンパス株式会社 内視鏡装置及び内視鏡装置の作動方法
US9204939B2 (en) * 2011-08-21 2015-12-08 M.S.T. Medical Surgery Technologies Ltd. Device and method for assisting laparoscopic surgery—rule based approach
US9081488B2 (en) * 2011-10-19 2015-07-14 Boston Scientific Neuromodulation Corporation Stimulation leadwire and volume of activation control and display interface
JP6103827B2 (ja) * 2012-06-14 2017-03-29 オリンパス株式会社 画像処理装置および立体画像観察システム
TWI517828B (zh) * 2012-06-27 2016-01-21 國立交通大學 影像追蹤系統及其影像追蹤方法
US9877633B2 (en) * 2012-07-25 2018-01-30 Intuitive Surgical Operations, Inc Efficient and interactive bleeding detection in a surgical system
GB2505926A (en) * 2012-09-14 2014-03-19 Sony Corp Display of Depth Information Within a Scene
JP2014147630A (ja) * 2013-02-04 2014-08-21 Canon Inc 立体内視鏡装置
JP6265627B2 (ja) * 2013-05-23 2018-01-24 オリンパス株式会社 内視鏡装置及び内視鏡装置の作動方法
WO2015029318A1 (ja) * 2013-08-26 2015-03-05 パナソニックＩｐマネジメント株式会社 ３次元表示装置および３次元表示方法
EP3031386B1 (en) * 2013-09-10 2021-12-29 Sony Group Corporation Image processing device and program
US20150080652A1 (en) * 2013-09-18 2015-03-19 Cerner Innovation, Inc. Lesion detection and image stabilization using portion of field of view
ES2900181T3 (es) * 2014-02-27 2022-03-16 Univ Surgical Associates Inc Pantalla interactiva para cirugía
US10350009B2 (en) * 2014-03-28 2019-07-16 Intuitive Surgical Operations, Inc. Quantitative three-dimensional imaging and printing of surgical implants
JP6609616B2 (ja) * 2014-03-28 2019-11-20 インテュイティブ サージカル オペレーションズ， インコーポレイテッド マルチポートの視点からの手術シーンの定量的な３次元イメージング
KR101599129B1 (ko) * 2014-05-20 2016-03-02 박현준 내시경 상 보이는 병변의 크기 측정 방법 및 컴퓨터 판독 가능한 기록매체
JP6381313B2 (ja) * 2014-06-20 2018-08-29 キヤノン株式会社 制御装置、制御方法、およびプログラム
DE112014007145T5 (de) * 2014-12-02 2017-08-03 Olympus Corporation Scharfstellsteuerungsvorrichtung, Endoskopvorrichtung und Verfahren zum Steuern der Scharfstellsteuerungsvorrichtung
DE112014007147T5 (de) * 2014-12-02 2017-08-03 Olympus Corporation Scharfstellsteuerungsvorrichtung, Endoskopvorrichtung und Verfahren zum Steuern der Scharfstellsteuerungsvorrichtung

Also Published As

Publication number Publication date
EP3151719B1 (en) 2021-02-17
JP2015228955A (ja) 2015-12-21
US20170042407A1 (en) 2017-02-16
EP3151719A1 (en) 2017-04-12
WO2015186335A1 (en) 2015-12-10
US10827906B2 (en) 2020-11-10

Similar Documents

Publication Publication Date Title
JP6323184B2 (ja) 画像処理装置、画像処理方法、並びにプログラム
JP6323183B2 (ja) 画像処理装置および画像処理方法
JP5576739B2 (ja) 画像処理装置、画像処理方法、撮像装置及びプログラム
JP6758287B2 (ja) 制御装置及び医療用撮像システム
JP7226325B2 (ja) 焦点検出装置および方法、並びにプログラム
US11822089B2 (en) Head wearable virtual image module for superimposing virtual image on real-time image
JP2017164007A (ja) 医療用画像処理装置、医療用画像処理方法、プログラム
US11426052B2 (en) Endoscopic system
US20110178371A1 (en) Endoscope apparatus and method of measuring subject
KR101818005B1 (ko) 얼굴 데이터 관리 시스템 및 그 방법
WO2019106712A1 (ja) 内視鏡画像処理装置及び内視鏡画像処理方法
JP2022136184A (ja) 制御装置、内視鏡システムおよび制御装置の作動方法
JP2017205343A (ja) 内視鏡装置、内視鏡装置の作動方法
JP2017176773A (ja) 手術支援システム、手術支援方法、手術支援プログラム
US20250072731A1 (en) Image processing system, image processing device, and image processing method
JP2011170276A (ja) 内視鏡装置およびプログラム
US12266126B2 (en) Measuring method and a measuring device for measuring and determining the size and dimension of structures in scene
CN117670971A (zh) 图像处理装置、图像处理方法和存储介质
US12295543B2 (en) Medical image processing device and medical observation system
JPWO2014208224A1 (ja) オートフォーカス装置
JPWO2018179875A1 (ja) 撮像装置とフォーカス制御方法およびフォーカス判定方法
JPWO2018047465A1 (ja) 内視鏡装置

Legal Events

Date Code Title Description
A521 Request for written amendment filed

Free format text: JAPANESE INTERMEDIATE CODE: A523

Effective date: 20170118

A621 Written request for application examination

Free format text: JAPANESE INTERMEDIATE CODE: A621

Effective date: 20170118

A131 Notification of reasons for refusal

Free format text: JAPANESE INTERMEDIATE CODE: A131

Effective date: 20171130

A521 Request for written amendment filed

Free format text: JAPANESE INTERMEDIATE CODE: A523

Effective date: 20180129

TRDD Decision of grant or rejection written
A01 Written decision to grant a patent or to grant a registration (utility model)

Free format text: JAPANESE INTERMEDIATE CODE: A01

Effective date: 20180313

A61 First payment of annual fees (during grant procedure)

Free format text: JAPANESE INTERMEDIATE CODE: A61

Effective date: 20180326

R151 Written notification of patent or utility model registration

Ref document number: 6323184

Country of ref document: JP

Free format text: JAPANESE INTERMEDIATE CODE: R151