JP6386194B2 - ステレオカメラと構造化光とを使用したヘッドマウントディスプレイでの深度マッピング - Google Patents

ステレオカメラと構造化光とを使用したヘッドマウントディスプレイでの深度マッピング Download PDF

Info

Publication number
JP6386194B2
JP6386194B2 JP2017549761A JP2017549761A JP6386194B2 JP 6386194 B2 JP6386194 B2 JP 6386194B2 JP 2017549761 A JP2017549761 A JP 2017549761A JP 2017549761 A JP2017549761 A JP 2017549761A JP 6386194 B2 JP6386194 B2 JP 6386194B2
Authority
JP
Japan
Prior art keywords
light pattern
cells
cell
structured light
laser
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Expired - Fee Related
Application number
JP2017549761A
Other languages
English (en)
Japanese (ja)
Other versions
JP2018509632A5 (enExample
JP2018509632A (ja
Inventor
ロマーノ、ニタイ
グロッシンガー、ナダブ
アルパーン、ヤイル
アロン、エミル
ラズ、ガイ
Original Assignee
フェイスブック,インク.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by フェイスブック,インク. filed Critical フェイスブック,インク.
Publication of JP2018509632A publication Critical patent/JP2018509632A/ja
Publication of JP2018509632A5 publication Critical patent/JP2018509632A5/ja
Application granted granted Critical
Publication of JP6386194B2 publication Critical patent/JP6386194B2/ja
Expired - Fee Related legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20Image signal generators
    • H04N13/271Image signal generators wherein the generated image signals comprise depth maps or disparity maps
    • GPHYSICS
    • G06COMPUTING OR CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/0304Detection arrangements using opto-electronic means
    • G06F3/0325Detection arrangements using opto-electronic means using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
    • GPHYSICS
    • G06COMPUTING OR CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/0304Detection arrangements using opto-electronic means
    • G06F3/0317Detection arrangements using opto-electronic means in co-operation with a patterned surface, e.g. absolute position or relative movement detection for an optical mouse or pen positioned with respect to a coded surface
    • GPHYSICS
    • G06COMPUTING OR CALCULATING; COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00Arrangements for image or video recognition or understanding
    • G06V10/10Image acquisition
    • G06V10/12Details of acquisition arrangements; Constructional details thereof
    • G06V10/14Optical characteristics of the device performing the acquisition or on the illumination arrangements
    • G06V10/145Illumination specially adapted for pattern recognition, e.g. using gratings
    • GPHYSICS
    • G06COMPUTING OR CALCULATING; COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20Movements or behaviour, e.g. gesture recognition
    • G06V40/28Recognition of hand or arm movements, e.g. recognition of deaf sign language
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20Image signal generators
    • H04N13/204Image signal generators using stereoscopic image cameras
    • H04N13/25Image signal generators using stereoscopic image cameras using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20Image signal generators
    • H04N13/204Image signal generators using stereoscopic image cameras
    • H04N13/254Image signal generators using stereoscopic image cameras in combination with electromagnetic radiation sources for illuminating objects
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30Image reproducers
    • H04N13/332Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
    • H04N13/344Displays for viewing with the aid of special glasses or head-mounted displays [HMD] with head-mounted left-right displays
    • GPHYSICS
    • G01MEASURING; TESTING
    • G01NINVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/84Systems specially adapted for particular applications
    • G01N21/88Investigating the presence of flaws or contamination
    • G01N21/8806Specially adapted optical and illumination features
    • GPHYSICS
    • G06COMPUTING OR CALCULATING; COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V2201/00Indexing scheme relating to image or video recognition or understanding
    • G06V2201/12Acquisition of 3D measurements of objects
    • HELECTRICITY
    • H01ELECTRIC ELEMENTS
    • H01SDEVICES USING THE PROCESS OF LIGHT AMPLIFICATION BY STIMULATED EMISSION OF RADIATION [LASER] TO AMPLIFY OR GENERATE LIGHT; DEVICES USING STIMULATED EMISSION OF ELECTROMAGNETIC RADIATION IN WAVE RANGES OTHER THAN OPTICAL
    • H01S5/00Semiconductor lasers
    • H01S5/005Optical components external to the laser cavity, specially adapted therefor, e.g. for homogenisation or merging of the beams or for manipulating laser pulses, e.g. pulse shaping
    • H01S5/0085Optical components external to the laser cavity, specially adapted therefor, e.g. for homogenisation or merging of the beams or for manipulating laser pulses, e.g. pulse shaping for modulating the output, i.e. the laser beam is modulated outside the laser cavity
    • HELECTRICITY
    • H01ELECTRIC ELEMENTS
    • H01SDEVICES USING THE PROCESS OF LIGHT AMPLIFICATION BY STIMULATED EMISSION OF RADIATION [LASER] TO AMPLIFY OR GENERATE LIGHT; DEVICES USING STIMULATED EMISSION OF ELECTROMAGNETIC RADIATION IN WAVE RANGES OTHER THAN OPTICAL
    • H01S5/00Semiconductor lasers
    • H01S5/40Arrangement of two or more semiconductor lasers, not provided for in groups H01S5/02 - H01S5/30
    • H01S5/42Arrays of surface emitting lasers
    • H01S5/423Arrays of surface emitting lasers having a vertical cavity
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N2013/0074Stereoscopic image analysis
    • H04N2013/0081Depth or disparity estimation from stereoscopic image signals

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Signal Processing (AREA)
  • Human Computer Interaction (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Electromagnetism (AREA)
  • Artificial Intelligence (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Psychiatry (AREA)
  • Social Psychology (AREA)
  • Length Measuring Devices By Optical Means (AREA)
JP2017549761A 2015-03-22 2016-03-22 ステレオカメラと構造化光とを使用したヘッドマウントディスプレイでの深度マッピング Expired - Fee Related JP6386194B2 (ja)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US201562136549P 2015-03-22 2015-03-22
US62/136,549 2015-03-22
PCT/US2016/023611 WO2016154218A1 (en) 2015-03-22 2016-03-22 Depth mapping with a head mounted display using stereo cameras and structured light

Publications (3)

Publication Number Publication Date
JP2018509632A JP2018509632A (ja) 2018-04-05
JP2018509632A5 JP2018509632A5 (enExample) 2018-07-12
JP6386194B2 true JP6386194B2 (ja) 2018-09-05

Family

ID=56977613

Family Applications (1)

Application Number Title Priority Date Filing Date
JP2017549761A Expired - Fee Related JP6386194B2 (ja) 2015-03-22 2016-03-22 ステレオカメラと構造化光とを使用したヘッドマウントディスプレイでの深度マッピング

Country Status (6)

Country Link
US (1) US10031588B2 (enExample)
EP (1) EP3274653B1 (enExample)
JP (1) JP6386194B2 (enExample)
KR (1) KR101904181B1 (enExample)
CN (1) CN107637074B (enExample)
WO (1) WO2016154218A1 (enExample)

Families Citing this family (23)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10337860B2 (en) * 2016-12-07 2019-07-02 Magik Eye Inc. Distance sensor including adjustable focus imaging sensor
US10620316B2 (en) * 2017-05-05 2020-04-14 Qualcomm Incorporated Systems and methods for generating a structured light depth map with a non-uniform codeword pattern
US10733748B2 (en) 2017-07-24 2020-08-04 Hand Held Products, Inc. Dual-pattern optical 3D dimensioning
US20190349569A1 (en) * 2018-05-10 2019-11-14 Samsung Electronics Co., Ltd. High-sensitivity low-power camera system for 3d structured light application
JP7114387B2 (ja) * 2018-07-30 2022-08-08 キヤノン株式会社 撮像装置、撮像装置の制御方法およびプログラム
US11474245B2 (en) 2018-06-06 2022-10-18 Magik Eye Inc. Distance measurement using high density projection patterns
CN110855961A (zh) * 2018-08-20 2020-02-28 奇景光电股份有限公司 深度感测装置及其操作方法
US10909373B1 (en) * 2018-08-24 2021-02-02 Snap Inc. Augmented reality system using structured light
US10896516B1 (en) * 2018-10-02 2021-01-19 Facebook Technologies, Llc Low-power depth sensing using dynamic illumination
US11483503B2 (en) 2019-01-20 2022-10-25 Magik Eye Inc. Three-dimensional sensor including bandpass filter having multiple passbands
US12031808B2 (en) * 2019-02-20 2024-07-09 Trinamix Gmbh Detector with a projector for illuminating at least one object
US11474209B2 (en) 2019-03-25 2022-10-18 Magik Eye Inc. Distance measurement using high density projection patterns
EP3788545B1 (en) * 2019-03-26 2022-12-07 Shenzhen Goodix Technology Co., Ltd. Multizone illumination for optical face identification
CN111566663B (zh) * 2019-03-26 2023-05-02 深圳市汇顶科技股份有限公司 用于光学脸部识别的多区域照明
CN112036222B (zh) * 2019-06-04 2023-12-29 星宸科技股份有限公司 脸部辨识系统及方法
US11422638B2 (en) 2019-07-08 2022-08-23 Apple Inc. Input devices that use self-mixing interferometry to determine movement within an enclosure
WO2021045685A1 (en) * 2019-09-04 2021-03-11 Ams Sensors Singapore Pte. Ltd. Designing and constructing dot projectors for three-dimensional sensor modules
US11409365B2 (en) * 2019-09-06 2022-08-09 Apple Inc. Self-mixing interferometry-based gesture input system including a wearable or handheld device
US11419546B2 (en) 2019-09-24 2022-08-23 Apple Inc. Wearable self-mixing interferometry device used to sense physiological conditions
US11450083B2 (en) 2019-09-27 2022-09-20 Honeywell International Inc. Dual-pattern optical 3D dimensioning
US11639846B2 (en) 2019-09-27 2023-05-02 Honeywell International Inc. Dual-pattern optical 3D dimensioning
US12386188B2 (en) 2019-10-07 2025-08-12 Trinamix Gmbh Projector for illuminating at least one object
CN111240009B (zh) 2019-12-31 2020-12-29 嘉兴驭光光电科技有限公司 可用于投射斜线的衍射光学元件、投射装置及其设计方法

Family Cites Families (64)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE4115445C2 (de) 1990-07-05 1994-02-17 Reinhard Malz Verfahren zum Aufnehmen eines dreidimensionalen Bildes eines Objektes nach dem aktiven Triangulationsprinzip und Vorrichtung hierzu
US5212555A (en) * 1991-12-17 1993-05-18 Texas Instruments Incorporated Image capture with spatial light modulator and single-cell photosensor
US5335288A (en) 1992-02-10 1994-08-02 Faulkner Keith W Apparatus and method for biometric identification
JP3531014B2 (ja) 1994-06-22 2004-05-24 株式会社トプコン レーザ照準装置
US6102552A (en) 1996-10-18 2000-08-15 Hewlett-Packard Company Laser-array based digital illuminator
WO1999058930A1 (en) 1998-05-14 1999-11-18 Metacreations Corporation Structured-light, triangulation-based three-dimensional digitizer
US6040910A (en) 1998-05-20 2000-03-21 The Penn State Research Foundation Optical phase-shift triangulation technique (PST) for non-contact surface profiling
JP2000267800A (ja) 1999-01-12 2000-09-29 Takenaka Komuten Co Ltd 動作認識装置
US7006236B2 (en) 2002-05-22 2006-02-28 Canesta, Inc. Method and apparatus for approximating depth of an object's placement onto a monitored region with applications to virtual interface devices
JP2004523031A (ja) 2001-01-08 2004-07-29 ヴィーケービー インコーポレイテッド データ入力装置
US7176440B2 (en) 2001-01-19 2007-02-13 Honeywell International Inc. Method and apparatus for detecting objects using structured light patterns
US6841780B2 (en) 2001-01-19 2005-01-11 Honeywell International Inc. Method and apparatus for detecting objects
US6714234B1 (en) 2001-04-11 2004-03-30 Applied Minds, Inc. Maintaining eye-contact in teleconferencing using structured light
US6947575B2 (en) 2001-05-24 2005-09-20 Trw Inc. Apparatus and method for determining vehicle occupant characteristic utilizing imaging with provided light
US8300042B2 (en) 2001-06-05 2012-10-30 Microsoft Corporation Interactive video display system using strobed light
JP3975892B2 (ja) 2002-05-02 2007-09-12 富士ゼロックス株式会社 位置計測システム
US7440590B1 (en) * 2002-05-21 2008-10-21 University Of Kentucky Research Foundation System and technique for retrieving depth information about a surface by projecting a composite image of modulated light patterns
US20040119833A1 (en) 2002-07-25 2004-06-24 Duncan Donald D. Three-dimensional context sensitive scanner
CN1176351C (zh) 2002-10-09 2004-11-17 天津大学 动态多分辨率的三维数字成像的方法及装置
US7064810B2 (en) 2003-09-15 2006-06-20 Deere & Company Optical range finder with directed attention
US7834850B2 (en) 2005-11-29 2010-11-16 Navisense Method and system for object control
EP1983402A4 (en) 2006-02-03 2013-06-26 Panasonic Corp INPUT DEVICE AND METHOD THEREFOR
US20090002342A1 (en) 2006-02-03 2009-01-01 Tomohiro Terada Information Processing Device
CN101957994B (zh) 2006-03-14 2014-03-19 普莱姆传感有限公司 三维传感的深度变化光场
JP5592070B2 (ja) 2006-03-14 2014-09-17 プライム センス リミティド 三次元検知のために深度変化させる光照射野
EP2041640B1 (en) 2006-07-16 2012-01-25 I. Cherradi Free fingers typing technology
US8144121B2 (en) 2006-10-11 2012-03-27 Victor Company Of Japan, Limited Method and apparatus for controlling electronic appliance
US20080256494A1 (en) 2007-04-16 2008-10-16 Greenfield Mfg Co Inc Touchless hand gesture device controller
US7907781B2 (en) 2007-06-21 2011-03-15 Mitsubishi Electric Research Laboraties, Inc. System and method for determining geometries of scenes
US8726194B2 (en) 2007-07-27 2014-05-13 Qualcomm Incorporated Item selection using enhanced control
US20090189858A1 (en) 2008-01-30 2009-07-30 Jeff Lev Gesture Identification Using A Structured Light Pattern
US8319832B2 (en) 2008-01-31 2012-11-27 Denso Corporation Input apparatus and imaging apparatus
US8107083B2 (en) * 2008-03-05 2012-01-31 General Electric Company System aspects for a probe system that utilizes structured-light
FR2932562B1 (fr) * 2008-06-12 2010-08-27 Univ Pasteur Dispositif de projection de lumiere structuree au moyen de vcsel et de composants optiques diffractifs de phase.
EP2166305B1 (de) 2008-09-23 2012-05-16 Sick Ag Beleuchtungseinheit und Verfahren zur Projektion eines Beleuchtungsmusters
CN101451826B (zh) 2008-12-17 2010-06-09 中国科学院上海光学精密机械研究所 物体三维轮廓测量装置及测量方法
TWI409502B (zh) 2009-01-23 2013-09-21 Univ Nat Taipei Technology 相位資訊擷取方法及其三維形貌量測系統
US20100309391A1 (en) * 2009-06-03 2010-12-09 Honeywood Technologies, Llc Multi-source projection-type display
JP2011018090A (ja) 2009-07-07 2011-01-27 Panasonic Corp 電子ペン及び電子ペンシステム
US8654234B2 (en) 2009-07-26 2014-02-18 Massachusetts Institute Of Technology Bi-directional screen
CN102656543A (zh) * 2009-09-22 2012-09-05 泊布欧斯技术有限公司 计算机装置的远程控制
US9075184B2 (en) * 2012-04-17 2015-07-07 Milan Momcilo Popovich Compact edge illuminated diffractive display
JP2011127932A (ja) * 2009-12-15 2011-06-30 Nippon Telegr & Teleph Corp <Ntt> 三次元形状測定装置、三次元形状測定方法、三次元形状測定プログラム、および記録媒体
US8320621B2 (en) * 2009-12-21 2012-11-27 Microsoft Corporation Depth projector system with integrated VCSEL array
EP2378394A3 (en) 2010-04-15 2015-03-25 Electronics and Telecommunications Research Institute User interface device and method for recognizing user interaction using same
BR112013000904A8 (pt) * 2010-07-16 2018-02-06 Koninklijke Philips Electronics Nv Projetor de luz e sistema de visão
US20120056982A1 (en) * 2010-09-08 2012-03-08 Microsoft Corporation Depth camera based on structured light and stereo vision
US9870068B2 (en) * 2010-09-19 2018-01-16 Facebook, Inc. Depth mapping with a head mounted display using stereo cameras and structured light
US9008515B2 (en) * 2010-10-07 2015-04-14 Alcatel Lucent Direct laser modulation
KR20120071551A (ko) 2010-12-23 2012-07-03 한국전자통신연구원 구조 영상을 이용한 사용자 인터랙션 장치 및 방법
US20120200829A1 (en) 2011-02-09 2012-08-09 Alexander Bronstein Imaging and projecting devices and methods
EP2500687A3 (en) * 2011-03-14 2013-02-20 Heliotis SA Structured light projector
KR101446902B1 (ko) 2011-08-19 2014-10-07 한국전자통신연구원 사용자 인터랙션 장치 및 방법
KR101272573B1 (ko) * 2011-11-17 2013-06-10 재단법인대구경북과학기술원 구조광 패턴 기반의 깊이 정보 추정 장치 및 방법
KR101272574B1 (ko) * 2011-11-18 2013-06-10 재단법인대구경북과학기술원 구조광 패턴 기반의 3차원 영상 정보 추정 장치 및 방법
WO2013127974A1 (en) * 2012-03-01 2013-09-06 Iee International Electronics & Engineering S.A. Spatially coded structured light generator
KR20150035513A (ko) * 2012-03-26 2015-04-06 맨티스비전 리미티드 3차원 카메라 및 투사기
WO2014083485A1 (en) * 2012-11-29 2014-06-05 Koninklijke Philips N.V. Laser device for projecting a structured light pattern onto a scene
JP6070274B2 (ja) * 2013-02-28 2017-02-01 旭硝子株式会社 拡散素子、照明光学系および計測装置
KR20140133093A (ko) 2013-05-09 2014-11-19 한국전자통신연구원 인식 장치 및 그 방법
US10019843B2 (en) 2013-08-08 2018-07-10 Facebook, Inc. Controlling a near eye display
BR112016009202A8 (pt) * 2013-10-23 2020-03-24 Oculus Vr Llc aparelhos e método para gerar um padrão de luz estruturada
US9749513B2 (en) 2014-04-29 2017-08-29 Facebook, Inc. System and method for generating a light pattern for object illumination
JP6417702B2 (ja) 2014-05-01 2018-11-07 富士通株式会社 画像処理装置、画像処理方法および画像処理プログラム

Also Published As

Publication number Publication date
US20180157342A1 (en) 2018-06-07
KR20170129882A (ko) 2017-11-27
WO2016154218A1 (en) 2016-09-29
EP3274653A1 (en) 2018-01-31
KR101904181B1 (ko) 2018-10-04
EP3274653A4 (en) 2018-11-14
CN107637074B (zh) 2020-09-18
US10031588B2 (en) 2018-07-24
EP3274653B1 (en) 2020-11-25
JP2018509632A (ja) 2018-04-05
CN107637074A (zh) 2018-01-26

Similar Documents

Publication Publication Date Title
JP6386194B2 (ja) ステレオカメラと構造化光とを使用したヘッドマウントディスプレイでの深度マッピング
US11962748B2 (en) Three dimensional depth mapping using dynamic structured light
US9870068B2 (en) Depth mapping with a head mounted display using stereo cameras and structured light
US9826216B1 (en) Systems and methods for compact space-time stereo three-dimensional depth sensing
US8761495B2 (en) Distance-varying illumination and imaging techniques for depth mapping
CN102870121B (zh) 图像捕捉装置
US20200182974A1 (en) Vertical cavity surface emitting laser-based projector
TWI869399B (zh) 將三維深度地圖資料映射到二維圖像上的方法、非暫時性機器可讀儲存媒體以及裝置
CN104024833A (zh) 图像处理装置、摄像装置、显微镜装置、图像处理方法以及图像处理程序
JP4973009B2 (ja) プロジェクタ及び画像投写方法
CN115867765A (zh) 漫射照明和多模式照明装置
CN110836647B (zh) 三维扫描系统
TWI719383B (zh) 多影像投影機以及具有多影像投影機之電子裝置
CN209690703U (zh) 投影装置及其光源和设备

Legal Events

Date Code Title Description
A521 Request for written amendment filed

Free format text: JAPANESE INTERMEDIATE CODE: A523

Effective date: 20180604

A621 Written request for application examination

Free format text: JAPANESE INTERMEDIATE CODE: A621

Effective date: 20180604

A871 Explanation of circumstances concerning accelerated examination

Free format text: JAPANESE INTERMEDIATE CODE: A871

Effective date: 20180604

A975 Report on accelerated examination

Free format text: JAPANESE INTERMEDIATE CODE: A971005

Effective date: 20180619

TRDD Decision of grant or rejection written
A01 Written decision to grant a patent or to grant a registration (utility model)

Free format text: JAPANESE INTERMEDIATE CODE: A01

Effective date: 20180717

A61 First payment of annual fees (during grant procedure)

Free format text: JAPANESE INTERMEDIATE CODE: A61

Effective date: 20180808

R150 Certificate of patent or registration of utility model

Ref document number: 6386194

Country of ref document: JP

Free format text: JAPANESE INTERMEDIATE CODE: R150

S111 Request for change of ownership or part of ownership

Free format text: JAPANESE INTERMEDIATE CODE: R313113

R350 Written notification of registration of transfer

Free format text: JAPANESE INTERMEDIATE CODE: R350

RD02 Notification of acceptance of power of attorney

Free format text: JAPANESE INTERMEDIATE CODE: R3D02

RD02 Notification of acceptance of power of attorney

Free format text: JAPANESE INTERMEDIATE CODE: R3D02

R250 Receipt of annual fees

Free format text: JAPANESE INTERMEDIATE CODE: R250

S533 Written request for registration of change of name

Free format text: JAPANESE INTERMEDIATE CODE: R313533

R350 Written notification of registration of transfer

Free format text: JAPANESE INTERMEDIATE CODE: R350

R250 Receipt of annual fees

Free format text: JAPANESE INTERMEDIATE CODE: R250

R250 Receipt of annual fees

Free format text: JAPANESE INTERMEDIATE CODE: R250

LAPS Cancellation because of no payment of annual fees