CN104280886A - Microscopic system and microscopic method based on in-situ three-dimensional enhanced display - Google Patents
Microscopic system and microscopic method based on in-situ three-dimensional enhanced display Download PDFInfo
- Publication number
- CN104280886A CN104280886A CN201410498047.5A CN201410498047A CN104280886A CN 104280886 A CN104280886 A CN 104280886A CN 201410498047 A CN201410498047 A CN 201410498047A CN 104280886 A CN104280886 A CN 104280886A
- Authority
- CN
- China
- Prior art keywords
- true
- bore hole
- view
- original position
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B30/00—Optical systems or apparatus for producing three-dimensional [3D] effects, e.g. stereoscopic images
- G02B30/20—Optical systems or apparatus for producing three-dimensional [3D] effects, e.g. stereoscopic images by providing first and second parallax images to an observer's left and right eyes
- G02B30/34—Stereoscopes providing a stereoscopic pair of separated images corresponding to parallactically displaced views of the same object, e.g. 3D slide viewers
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B21/00—Microscopes
- G02B21/36—Microscopes arranged for photographic purposes or projection purposes or digital imaging or video purposes including associated control and data processing arrangements
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
- H04N13/111—Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
- H04N13/139—Format conversion, e.g. of frame-rate or size
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- H04N13/302—Image reproducers for viewing without the aid of special glasses, i.e. using autostereoscopic displays
- H04N13/305—Image reproducers for viewing without the aid of special glasses, i.e. using autostereoscopic displays using lenticular lenses, e.g. arrangements of cylindrical lenses
Landscapes
- Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- General Physics & Mathematics (AREA)
- Optics & Photonics (AREA)
- Chemical & Material Sciences (AREA)
- Analytical Chemistry (AREA)
Abstract
The invention discloses a microscopic system and microscopic method based on in-situ three-dimensional enhanced display. By means of the microscopic system and microscopic method, a user can clearly and unceasingly observe an entity within a large view field range. The microscopic system comprises an entity observation device, a naked-eye three-dimensional display device and the in-situ three-dimensional display device. The entity observation device is aligned with an entity area and used for making the user directly obtain an entity visual image in real time. The naked-eye three-dimensional display device is used for providing a naked-eye true three-dimensional image related to the content of the entity visual image. The in-situ three-dimensional display device is connected with the entity observation device and the naked-eye three-dimensional display device and used for displaying the entity visual image and the naked-eye true three-dimensional image to the user after conducting in-situ three-dimensional perspective and fusion on the entity visual image and the naked-eye true three-dimensional image. The microscopic system has the advantages of being simple in structure, high in operation flexibility, capable of meeting different requirements, capable of achieving high-resolution three-dimensional fusion display or system portability and miniaturization, and the like.
Description
Technical field
The present invention relates to microtechnic, be specifically related to a kind of microscopic system and the microscopic method that strengthen display based on original position solid.
Background technology
Microscopic system is more and more extensive in the utilization of the research fields such as clinical medicine, biology, pharmacy.Traditional micro-display is directly obtained by microscope.For product OPMI PENTERO 900 surgical operation microscope of Zeiss company and Axio Imager 2 positively fixed type research microscope platform, its observing system is made up of principal goods mirror, binocular lens tube and zoom unit, is reached the requirement of different entities enlargement factor and operating distance by the arrange in pairs or groups eyepiece of different multiplying and object lens.In mirror, projection function allows user in entity, merge the certain additional image information of display simultaneously, guides the operation of user.But this microscopical visual field diameter is very little, and only can carry out straight line observation, easily produces and observe dead angle.Meanwhile, the additional image information of display can be projected in mirror very limited, adopt existing plane display mode, although valuable various required image informations can be provided for trier, shortcoming intuitive.
For solving the problem such as the visual field and image information display, people have carried out large quantity research to three-dimensional head mounted display.The solid " stethoscope " utilizing perspective head mounted display to make as: North Carolina University and the head-wearing device for endoscopy planning and preoperative simulation.Head mounted display places micro-display respectively in the respective visual field of wearer's right and left eyes, the principle of stereoscopic vision can be produced when observing different images based on right and left eyes, utilization can obtain stereopsis image by autozoom high definition stereo camera apparatus, observation entity is combined with the 3 d image data of entity, in head-mounted display, presents the realtime graphic that augmented reality merges display.Report is pointed out, under 3-D display, the understanding accuracy of user and operating accuracy increase, and the running time also can shorten.Such as, but there is significantly restriction in the development of current head mounted display, when improving the resolution requirement of situ fusion image, the size of micro-display also can correspondingly increase, and Whole Equipment dimension and weight also can increase; In addition, head mounted display can cause converge like the spokes of a wheel at the hub inconsistent with focal adjustments of user.In order to ensure three-dimensional imaging true to nature, head mounted display also exists the problems such as volume is comparatively large, width concentrate around one point, as spokes on acis and focal adjustments be inconsistent when observing, and very easily produces the phenomenon such as visual fatigue, dizzy discomfort when long-time use.Converge like the spokes of a wheel at the hub during eye-observation real-world object is consistent with focal adjustments, and when watching stereopsis, eyes regulate and are on screen, and on converge like the spokes of a wheel at the hub virtual three-dimensional image being in parallax synthesis, therefore long-term viewing will inevitably cause visual fatigue.
As from the foregoing, though current microscopic system directly can observe entity, area of visual field is less, is also difficult to carry out the display of 3-D view In-sltu reinforcement; And though head mounted display can carry out the fusion of high definition 3-D view, be difficult to the sharpness simultaneously ensureing user's comfort and fused images, easily have sense of discomfort.There is no both meeting requirements on three-dimensional stereo-picture In-sltu reinforcement at present show and ensure the micro-display system in enough visuals field, the true microscopy information strengthening the entity situ fusion shown directly perceived, three-dimensional cannot be provided for user.
Summary of the invention
The present invention is intended to solve one of technical matters in correlation technique at least to a certain extent.For this reason, the object of the invention is to propose a kind of to there is the microscopic system and the microscopic method that strengthen display based on original position solid that structure is simple, flexible operation degree is large.
According to the microscopic system strengthening display based on original position solid of the embodiment of the present invention, can comprise: physical inspection device, described physical inspection device aims at entity area, obtains stereopsis image in real time, directly for allowing user; Naked eye three-dimensional display device, for providing the bore hole true 3-D view relevant to described stereopsis picture material; Original position 3 d display device, described original position 3 d display device is connected with described naked eye three-dimensional display device with described physical inspection device respectively, after described stereopsis image and the true 3-D view of described bore hole are carried out original position volume rendering fusion, be shown to user.
In addition, microscopic system according to the above embodiment of the present invention can also have following additional technical feature:
Alternatively, described naked eye three-dimensional display device comprises the true three-dimensional image source of bore hole and projection part, described projection part be used for according to described bore hole true three-dimensional image source in space Projection Display go out the corresponding true 3-D view of bore hole, wherein, when naked eye three-dimensional display device is based on 3 D full-figure technique type, described projection part is lens arra; When naked eye three-dimensional display device is poly-talented based on light field, described projection part is lens arra; When naked eye three-dimensional display device is based on holographic technique type, described projection part is interference device.
Alternatively, described original position 3 d display device comprises: the first fixture, for the projection part in fixing described naked eye three-dimensional display device; Second fixture, for fixing described physical inspection device; Semi-transparent semi-reflecting device, the relative position of described semi-transparent semi-reflecting device and described first fixture and described second fixture meets following condition: described stereopsis image and the true 3-D view of described bore hole are shown to user after described semi-transparent semi-reflecting device place carries out original position volume rendering fusion, wherein, described stereopsis image is shown to user through after described semi-transparent semi-reflecting device, and the true 3-D view of described bore hole is shown to user after described semi-transparent semi-reflecting device reflection.
Alternatively, in described original position 3 d display device, also comprise between the true 3-D view of described bore hole and described semi-transparent semi-reflecting device: aerial image integrated unit and the controlled reflector element of light path, merge projection structure to realize easily extensible close-coupled.
Alternatively, also comprise: in solid images amplifying unit, true 3-D view unit for scaling and fused images unit for scaling one of at least, wherein, described solid images amplifying unit is for amplifying the size of the described stereopsis image before carrying out original position volume rendering fusion, described true 3-D view unit for scaling carries out the size of the true 3-D view of described bore hole before original position volume rendering fusion for convergent-divergent, and described fused images unit for scaling carries out the picture size after original position volume rendering fusion for stereopsis image described in convergent-divergent and the true 3-D view of described bore hole.
Alternatively, also comprise: aberration correction processing module, described aberration correction processing module is used for aberration correction.
Alternatively, in described original position 3 d display device, be shown to user after carrying out home position manifestation after described stereopsis image and the true 3-D view of described bore hole are carried out registration, wherein the mode of registration is the registration of registration based on monumented point or no marks point.
According to the microscopic system strengthening display based on original position solid of the embodiment of the present invention, can comprise the following steps: A. obtains stereopsis image in real time, directly; B., the true 3-D view of bore hole relevant to described stereopsis picture material is provided; C. user is shown to after described stereopsis image and the true 3-D view of described bore hole being carried out original position volume rendering fusion.
In addition, microscopic system according to the above embodiment of the present invention can also have following additional technical feature:
Alternatively, described step B specifically comprises: provide bore hole true three-dimensional image source, by projection part in space Projection Display go out the corresponding true 3-D view of bore hole, wherein, when naked eye three-dimensional display packing is based on 3 D full-figure technique type, described projection part is lens arra; When naked eye three-dimensional display packing is poly-talented based on light field, described projection part is lens arra; When naked eye three-dimensional display packing is based on holographic technique type, described projection part is interference device.
Alternatively, in described step C, user is shown to after described stereopsis image and the true 3-D view of described bore hole being carried out original position volume rendering fusion by semi-transparent semi-reflecting device, wherein, described stereopsis image is shown to user through after described semi-transparent semi-reflecting device, and the true 3-D view of described bore hole is shown to user after described semi-transparent semi-reflecting device reflection.
Alternatively, easily extensible close-coupled is adopted to merge projection structure when carrying out light path design.
Alternatively, in also comprising the following steps one of at least: the size of amplifying the described stereopsis image before carrying out original position volume rendering fusion; Convergent-divergent carries out the size of the true 3-D view of described bore hole before original position volume rendering fusion; Stereopsis image described in convergent-divergent and the true 3-D view of described bore hole carry out the picture size after original position volume rendering fusion.
Alternatively, also step is comprised: carry out aberration correction process.
Alternatively, in described step C, be shown to user after carrying out home position manifestation after described stereopsis image and the true 3-D view of described bore hole are carried out registration, wherein the mode of registration is the registration of registration based on monumented point or no marks point.
According to the microscopic system and the microscopic method that strengthen display based on original position solid of the embodiment of the present invention, while making user can carry out microexamination and operation to entity in larger field range, for observer provides very three-dimensional bore hole volumetric body In-sltu reinforcement to show image, observe at auxiliary lower clear, the directly perceived and continuous print realized the inside and outside of entity combines of perspective fused images.The present invention also has that structure is simple, flexible operation degree large, can meet different demand, can realize the display of high-resolution stereoscopic fusion or can realize the advantages such as system Portable small.
Accompanying drawing explanation
Fig. 1 is the structured flowchart strengthening the microscopic system of display based on original position solid of the embodiment of the present invention.
Fig. 2 is the structural representation of the original position stereomicroscopy device of the embodiment of the present invention.
Fig. 3 is that the easily extensible close-coupled of the embodiment of the present invention merges projection structure schematic diagram.
Fig. 4 a is the schematic diagram of the microscopic system that the size of true for bore hole 3-D view reduces by true 3-D view unit for scaling; Fig. 4 b is the schematic diagram of the microscopic system that the size of true for bore hole 3-D view is amplified by true 3-D view unit for scaling.
Fig. 5 a is the schematic diagram strengthening the microscopic system of display based on original position solid not carrying out anaberration process; Fig. 5 b is the schematic diagram strengthening the microscopic system of display based on original position solid carrying out anaberration process.
Fig. 6 is the process flow diagram strengthening the microscopic method of display based on original position solid of the embodiment of the present invention.
Embodiment
Be described below in detail embodiments of the invention, the example of described embodiment is shown in the drawings, and wherein same or similar label represents same or similar element or has element that is identical or similar functions from start to finish.Be exemplary below by the embodiment be described with reference to the drawings, be intended to for explaining the present invention, and can not limitation of the present invention be interpreted as.
The microscopic system based on original position solid enhancing display of first aspect present invention embodiment as shown in Figure 1, comprises physical inspection device 10, naked eye three-dimensional display device 20 and original position 3 d display device 30.
Physical inspection device 10 aims at entity area, obtains stereopsis image in real time, directly for allowing user.Called entity visual pattern, image when referring to use in eyesight scope in one's power.Such as, if patients head needs to carry out operative treatment during operation technique, then the content such as the hair of this patients head of being directly observed visually of doctor, skin, bone, blood vessel, organ (below several open cranium after visible) all belongs to the content of stereopsis image.The simplest physical inspection device 10 can be one by one or more optical lens, this optical lens aims at entity area so that the direct visual inspection of user obtains stereopsis image.Preferably, physical inspection device 10 also can use the array configuration of display screen and image collecting device, observes entity realtime graphic for observer provides.Certainly, physical inspection device 10 also can be equipped with the accessory structure such as light shield, dust-proof translucent cover, does not repeat herein.
Naked eye three-dimensional display device 20 is for providing the bore hole true 3-D view relevant to stereopsis picture material.Here mean the two-layer meaning: the first, the image that naked eye three-dimensional display device 20 provides, be formed in space, bore hole and look that there is relief, three-dimensional virtual image.Naked eye three-dimensional display device 20 refers to based on optical imaging concept, can demonstrate the device being seen the picture with three-dimensional stereopsis by user's bore hole in space.The second, the content of the true 3-D view of bore hole is relevant to the content of stereopsis image.Such as, in certain surgical procedure, entity area is patient's head, stereopsis image is the image of seen head directly perceived, now, the content of the true 3-D view of the bore hole that naked eye three-dimensional display device 20 provides also should for the image relevant to head, instead of the image relevant to other positions such as thoracic cavity.
It should be noted that, the true 3-D view of bore hole is used for providing abundant data message to user (such as doctor), is convenient to user and makes analysis judgement.The true three-dimensional image source of bore hole for generating the true 3-D view of bore hole can be the volume data that the methods such as computer tomography (CT), Computerized three-dimensional modeling generate, also can be the two-dimensional ct image that the formation method such as X-ray transmission, optical coherence tomography (OCT) is produced, can also be the skin texture images that the method such as camera, structured light generates.In addition, the content of the true 3-D view of bore hole both can be obtain before operation, also can be Real-time Obtaining in operation, also can be carry out the result after real-time registration fusion by before operation with the image in operation.
It should be noted that, naked eye three-dimensional display device 20 comprises the true three-dimensional image source 21 of bore hole and projection part 22.Namely bore hole true three-dimensional image source 21 provides the parts of the content sources of the true 3-D view of bore hole, it can be some image data acquisition device (such as X-ray projection scanister and associated picture processing module), also can be the memory device (such as storing the hard disk of two-dimensional ultrasonic image) storing original image corresponding to the true 3-D view of bore hole, those skilled in the art can select as required flexibly, do not limit herein.Projection part 22 for according to bore hole true three-dimensional image source 21 in space Projection Display go out the true 3-D view of bore hole.Such as, using the light emitting display of one piece of two dimensional surface as the true three-dimensional image source 21 of bore hole, allow this light emitting display present pre-designed pattern, the light that then this light emitting display sends projects in space after projection part 22 has the true 3-D view of relief bore hole.The ins and outs of naked eye three-dimensional display device 20 are those skilled in the art's known knowledge, can select flexibly as required.When naked eye three-dimensional display device 20 is based on 3 D full-figure technique type, projection part 22 is lens arra.When naked eye three-dimensional display device 20 is poly-talented based on light field, projection part 22 is lens arra.When naked eye three-dimensional display device 20 is based on holographic technique type, projection part 22 is interference device.It should be noted that, other the naked eye three-dimensional display device of type well known by persons skilled in the art can also be selected, do not change principle of the present invention.And it should be noted that, the algorithm of the original image that the true 3-D view of design bore hole is corresponding is relevant with the type of naked eye three-dimensional display device.Such as, when naked eye three-dimensional display device 20 is based on 3 D full-figure technique type, need first to write computer program, utilize the rays pass through lens array that in the method simulated three-dimensional space object of volume drawing or iso-surface patch, certain is a bit launched, process obtains original image corresponding to the true 3-D view of bore hole.For the dynamic realtime realizing the true 3-D view of bore hole in operation upgrades, the acceleration that the parallelization processing modes such as GPU can be used to carry out original image corresponding to the true 3-D view of bore hole is played up.
Original position 3 d display device 30 is connected with naked eye three-dimensional display device 20 with physical inspection device 10 respectively, is shown to user after carrying out situ fusion for the stereopsis image that content is relative to each other and the true 3-D view of bore hole.Original position 3 d display device 30 forms by ad hoc fashion arrangement primarily of a series of optical element (lens, catoptron, semi-transparent semi-reflecting device etc.), that usually also comprise some necessity, that play support fixation housing or support etc. simultaneously.Concrete form the present invention of original position 3 d display device 30 is not construed as limiting, only need to guarantee to allow user see simultaneously stereopsis image and the true 3-D view of bore hole carry out home position manifestation after image.
Fig. 2 shows the structural representation of the original position 3 d display device 30 of the present invention's specific embodiment.As shown in Figure 2, this original position 3 d display device 30 comprises: the first fixture 31, second fixture 32 and semi-transparent semi-reflecting device 33.First fixture 31 is for the projection part in fixing naked eye three-dimensional display device 20.Second fixture 32 is for permanent entity finder 10.Semi-transparent semi-reflecting device 33 meets following condition with the relative position of physical inspection device 10, first fixture 31 and the second fixture 32: stereopsis image and the true 3-D view of bore hole are shown to user after semi-transparent semi-reflecting device 33 place original position volume rendering is merged, wherein, stereopsis image is shown to user through after semi-transparent semi-reflecting device 33, and the true 3-D view of bore hole is shown to user after semi-transparent semi-reflecting device 33 reflects.It should be noted that, should ensure that semi-transparent semi-reflecting device 33 arrives the consistance of the light path of semi-transparent semi-reflecting device 33 to light path and the naked eye three-dimensional display device 20 of entity.Preferably, semi-transparent semi-reflecting device 33 can adopt the semi-transparent semi-reflecting device of liquid crystal display formula.The semi-transparent semi-reflecting device of this liquid crystal display formula is to realize controlled transmittance and reflectance.Liquid crystal display itself also can be used as display screen, in certain applications, can be used as the quantity of information that additional flat-panel screens increases system in-situ image, or gathers user's operation as touch-screen, strengthen the interaction capabilities of system and user.
The mode of linear projection is adopted to complete paths between naked eye three-dimensional display device 20 to semi-transparent semi-reflecting device 33 in original position 3 d display device 30 embodiment illustrated in fig. 2, cause device to take up space comparatively large, cause the inconvenience that device controls and user operates when limited space.For this situation, applicant also proposes a kind of original position 3 d display device adopting easily extensible close-coupled to merge projection structure, describes in detail below in conjunction with Fig. 3.
In the original position 3 d display device 30 of the microscopic system based on original position solid enhancing display of one embodiment of the present of invention, aerial image integrated unit 34 and the controlled reflector element 35 of light path is also comprised, as shown in Figure 3 between naked eye three-dimensional display device 20 to semi-transparent semi-reflecting device 33.Wherein: aerial image fusion structure 34 is based on most Tou Ban deinstall mechanism, realize that several are independent in projection arrangement front end by optical hardware mode, the Space integration of three-dimensional bore hole space image side by side, the registration realizing simple multi-modal image information merges, and elevator system shows the rich of information; Meanwhile, the image co-registration based on optical hardware is very quick, to be applied to occasion image taking speed being had to high request.The controlled reflection configuration 35 of light path, for reducing optical axis length, makes system architecture compacter.By adopting the zoom lens Array Design in light path in this embodiment, and the catoptron group that spacing is adjustable, realize ensureing light path consistance when reducing system bulk, and expand visual range.
In one embodiment of the invention, the microscopic system strengthening display based on original position solid can also comprise in solid images amplifying unit 41, true 3-D view unit for scaling 42 and fused images unit for scaling 43 one of at least.Wherein: solid images amplifying unit 41 is for amplifying the size of the stereopsis image before carrying out original position volume rendering fusion, and such as, solid images amplifying unit 41 can for being positioned at lens between entity area and semi-transparent semi-reflecting device 33 or lens combination.True 3-D view unit for scaling 42 carries out the size of the true 3-D view of bore hole before original position volume rendering fusion for convergent-divergent.Such as, true 3-D view unit for scaling 42 can comprise lens between projection part and semi-transparent semi-reflecting device 33 or lens combination.Fused images unit for scaling 43 carries out the picture size after original position volume rendering fusion for convergent-divergent stereopsis image and the true 3-D view of bore hole.Such as, fused images unit for scaling 43 can comprise lens between user's eyes and semi-transparent semi-reflecting device 33 or lens combination.As from the foregoing, user can by the respective convergent-divergent multiple of comprehensive regulation solid images amplifying unit 41, true 3-D view unit for scaling 42 and fused images unit for scaling 43 to obtain best observation effect.
It should be noted that, in solid images amplifying unit 41, true 3-D view unit for scaling 42 and fused images unit for scaling 43 except lens or lens combination, also to there is powder operation device supporting with it.Control powder operation device to be used for lens or lens combination to move to suitable working position.Powder operation device can be arranged flexibly, can be the automatic actuating of motor form, also can be manual actuating of guide rail form etc.
It should be noted that, the convergent-divergent meaning of true 3-D view unit for scaling 42 pairs of true 3-D views of bore hole is the most great.Illustrate below in conjunction with Fig. 4 a and Fig. 4 b.
In an embodiment of the invention, the size of true for bore hole 3-D view can reduce by true 3-D view unit for scaling 42.As shown in fig. 4 a, true for initial bore hole 3-D view is labeled as A, this A is through true 3-D view unit for scaling 42 minification, and this is equivalent to the virtual representation A ' defining minification in space, and last A ' is mirrored in user's eye through semi-transparent semi-reflecting device 33 reflection.In this embodiment, the true 3-D view of script large scale bore hole can be carried out reducing focus on after small size with stereopsis image co-registration, reduce the resolution requirement to the true three-dimensional image source of original bore hole, improve the readability of fused image.
In another embodiment, the size of true for bore hole 3-D view can be amplified by true 3-D view unit for scaling 42.As shown in Figure 4 b, true for initial bore hole 3-D view is labeled as B, this B is through true 3-D view unit for scaling 42 up-sizing, and this is equivalent to the virtual representation B ' defining up-sizing in space, and last B ' is mirrored in user's eye after semi-transparent semi-reflecting device 33 reflects.In this embodiment, the true 3-D view of script small size bore hole can be carried out amplification expand to after large scale with stereopsis image co-registration, can be reduced the area requirements of projection part in naked eye three-dimensional display device, be conducive to the hardware miniaturization of whole microscopic system.
In one embodiment of the invention, strengthen based on original position solid the microscopic system shown also to comprise: aberration correction processing module 50, this aberration correction processing module 50 is for eliminating aberration.Due to the imperfection of actual optical components, the defects such as imaging generation is fuzzy, distortion can be caused.As shown in Figure 5 a.The true 3-D view C of original bore hole, after reducing process, creates the C ' with aberration, to such an extent as to has occurred obvious aberration in the fused images observed of user, at this moment needs to carry out aberration correction process to imaging system.Here computer simulation technique can be used quantitatively to calculate the aberration value of optical system, aberration correction process generally has two kinds of approach: be 1. optimized initial optical design parameter based on aberration value, readjust the parameters such as the focal length of each optics in relevant unit for scaling, radius, spacing, minimum to reach aberration.2. based on aberration value, bore hole stereoscopically displaying images is adjusted, make bore hole stereo-picture aberration after the imaging of lens combination reach minimum.Wherein a kind of aberration correcting method can be selected according to embody rule situation, also two kinds of methods combining can be used.Fig. 5 b provides an embodiment, the basis of Fig. 5 a changes the lens in true 3-D view unit for scaling 42 into gummed mirror, and the true 3-D view D of bore hole having predeformation and compensate is provided, this D obtains aberrationless virtual representation D ' after gummed mirror, and user finally observes aberrationless three-dimensional fused images.
In position in 3 d display device 30, when carrying out image co-registration, need stereopsis image and the true 3-D view of bore hole to carry out registration, to ensure that the image that user observes can correctly merge in position.Wherein the mode of registration comprises two kinds: based on the registration of monumented point and the registration of no marks point.When using the registration Algorithm based on monumented point, the gauge point can determined based on automatic or manual, uses Optimum Theory to calculate the registration relation with stereo-picture.Need Intraoperative position equipment carries out in art lasting tracking to physical object, the real-time update of registration information during realize target motion simultaneously.When using the registration Algorithm of no marks point, the method for reconstructing of three-dimensional scenic can be used, using Optimum Theory to calculate the registration relation with stereo-picture based on three-dimensional surface profile information.Meanwhile, need to use Intraoperative position equipment carries out in art lasting tracking to physical object, the real-time update of registration information during realize target motion.Based on the above-mentioned registration result having monumented point or no marks point, calculate optimal spatial coordinate transformation relation between the two, and based on the size that physical object adjusted size stereo-picture shows.This step is the known knowledge of those skilled in the art, does not repeat herein.
It should be noted that, if the enlargement factor of system changes in using, or when physical object pose changes, need the registration re-starting stereoscopic fusion image and physical object entity, upgrade stereoscopic fusion image displaying contents.When enlargement factor changes, according to multiple, convergent-divergent is carried out to display image; When physical object position changes, the lasting tracking in needing to use Intraoperative position equipment to use physical object, carries out registration again according to target surface monumented point or surface profile information, again upgrades display image according to registration result.
In sum, the microscopic system strengthening display based on original position solid of the embodiment of the present invention, make user in larger field range, clear, continuous print can be carried out to entity and stereoscopic fusion image to observe, have that structure is simple, flexible operation degree is large, different demand can be met, the display of high-resolution stereoscopic fusion can be realized or the advantages such as system Portable small can be realized.
The microscopic method strengthening display based on original position solid of second aspect present invention embodiment, as shown in Figure 6, comprises the following steps:
A. in real time, directly stereopsis image is obtained;
B., the true 3-D view of bore hole relevant to stereopsis picture material is provided;
C. user is shown to after stereopsis image and the true 3-D view of bore hole being carried out original position volume rendering fusion.
In one embodiment of the invention, step B specifically comprises: provide bore hole true three-dimensional image source, by projection part in space Projection Display go out the corresponding true 3-D view of bore hole.The method of carrying out naked eye three-dimensional display has many kinds, can select flexibly as required.When naked eye three-dimensional display packing is based on 3 D full-figure technique type, projection part is lens arra; When naked eye three-dimensional display packing is poly-talented based on light field, projection part is lens arra; When naked eye three-dimensional display packing is based on holographic technique type, projection part is interference device.
In one embodiment of the invention, in step C, after stereopsis image and the true 3-D view of bore hole being carried out original position volume rendering fusion by semi-transparent semi-reflecting device, be shown to user.Wherein, stereopsis image is shown to user through after semi-transparent semi-reflecting device, and the true 3-D view of bore hole is shown to user after semi-transparent semi-reflecting device reflection.
In one embodiment of the invention, easily extensible close-coupled is adopted to merge projection structure when carrying out light path design.Like this can reduction equipment volume, reduce fabricating yard limited size degree.
In one embodiment of the invention, convergent-divergent can also be carried out to image.Particularly, in can comprising the following steps one of at least: the size of amplifying the stereopsis image before carrying out original position volume rendering fusion; Convergent-divergent carries out the size of the true 3-D view of bore hole before original position volume rendering fusion; Convergent-divergent stereopsis image and the true 3-D view of bore hole carry out the picture size after original position volume rendering fusion.As from the foregoing, user can by comprehensive regulation stereopsis image, the true 3-D view of bore hole and fused images three convergent-divergent multiple separately to obtain best observation effect.
In one embodiment of the invention, also comprise: carry out aberration correction process.After aberration correction process, user can be allowed finally to observe aberrationless three-dimensional fused images.
In one embodiment of the invention, be shown to user after carrying out home position manifestation after stereopsis image and the true 3-D view of bore hole are carried out registration, wherein the mode of registration is the registration of registration based on monumented point or no marks point.
In sum, the microscopic method strengthening display based on original position solid of the embodiment of the present invention, make user in larger field range, clear, continuous print can be carried out to entity and stereoscopic fusion image to observe, have that flow process is simple, flexible operation degree is large, different demand can be met, the display of high-resolution stereoscopic fusion can be realized or the advantages such as system Portable small can be realized.
In describing the invention, it will be appreciated that, term " " center ", " longitudinal direction ", " transverse direction ", " length ", " width ", " thickness ", " on ", D score, " front ", " afterwards ", " left side ", " right side ", " vertically ", " level ", " top ", " end " " interior ", " outward ", " clockwise ", " counterclockwise ", " axis ", " radial direction ", orientation or the position relationship of the instruction such as " circumference " are based on orientation shown in the drawings or position relationship, only the present invention for convenience of description and simplified characterization, instead of indicate or imply that the device of indication or element must have specific orientation, with specific azimuth configuration and operation, therefore limitation of the present invention can not be interpreted as.
In addition, term " first ", " second " only for describing object, and can not be interpreted as instruction or hint relative importance or imply the quantity indicating indicated technical characteristic.Thus, be limited with " first ", the feature of " second " can express or impliedly comprise at least one this feature.In describing the invention, the implication of " multiple " is at least two, such as two, three etc., unless otherwise expressly limited specifically.
In the present invention, unless otherwise clearly defined and limited, the term such as term " installation ", " being connected ", " connection ", " fixing " should be interpreted broadly, and such as, can be fixedly connected with, also can be removably connect, or integral; Can be mechanical connection, also can be electrical connection; Can be directly be connected, also indirectly can be connected by intermediary, can be the connection of two element internals or the interaction relationship of two elements, unless otherwise clear and definite restriction.For the ordinary skill in the art, above-mentioned term concrete meaning in the present invention can be understood as the case may be.
In the present invention, unless otherwise clearly defined and limited, fisrt feature second feature " on " or D score can be that the first and second features directly contact, or the first and second features are by intermediary indirect contact.And, fisrt feature second feature " on ", " top " and " above " but fisrt feature directly over second feature or oblique upper, or only represent that fisrt feature level height is higher than second feature.Fisrt feature second feature " under ", " below " and " below " can be fisrt feature immediately below second feature or tiltedly below, or only represent that fisrt feature level height is less than second feature.
Describe and can be understood in process flow diagram or in this any process otherwise described or method, represent and comprise one or more for realizing the module of the code of the executable instruction of the step of specific logical function or process, fragment or part, and the scope of the preferred embodiment of the present invention comprises other realization, wherein can not according to order that is shown or that discuss, comprise according to involved function by the mode while of basic or by contrary order, carry out n-back test, this should understand by embodiments of the invention person of ordinary skill in the field.
In the description of this instructions, specific features, structure, material or feature that the description of reference term " embodiment ", " some embodiments ", " example ", " concrete example " or " some examples " etc. means to describe in conjunction with this embodiment or example are contained at least one embodiment of the present invention or example.In this manual, to the schematic representation of above-mentioned term not must for be identical embodiment or example.And the specific features of description, structure, material or feature can combine in one or more embodiment in office or example in an appropriate manner.In addition, when not conflicting, the feature of the different embodiment described in this instructions or example and different embodiment or example can carry out combining and combining by those skilled in the art.
Although illustrate and describe embodiments of the invention above, be understandable that, above-described embodiment is exemplary, can not be interpreted as limitation of the present invention, and those of ordinary skill in the art can change above-described embodiment within the scope of the invention, revises, replace and modification.
Claims (14)
1. strengthen a microscopic system for display based on original position solid, it is characterized in that, comprising:
Physical inspection device, described physical inspection device aims at entity area, obtains stereopsis image in real time, directly for allowing user;
Naked eye three-dimensional display device, for providing the bore hole true 3-D view relevant to described stereopsis picture material;
Original position 3 d display device, described original position 3 d display device is connected with described naked eye three-dimensional display device with described physical inspection device respectively, after described stereopsis image and the true 3-D view of described bore hole are carried out original position volume rendering fusion, be shown to user.
2. microscopic system according to claim 1, it is characterized in that, described naked eye three-dimensional display device comprises the true three-dimensional image source of bore hole and projection part, described projection part be used for according to described bore hole true three-dimensional image source in space Projection Display go out the corresponding true 3-D view of bore hole, wherein
When naked eye three-dimensional display device is based on 3 D full-figure technique type, described projection part is lens arra;
When naked eye three-dimensional display device is poly-talented based on light field, described projection part is lens arra;
When naked eye three-dimensional display device is based on holographic technique type, described projection part is interference device.
3. microscopic system according to claim 2, is characterized in that, described original position 3 d display device comprises:
First fixture, for the projection part in fixing described naked eye three-dimensional display device;
Second fixture, for fixing described physical inspection device;
Semi-transparent semi-reflecting device, the relative position of described semi-transparent semi-reflecting device and described first fixture and described second fixture meets following condition: described stereopsis image and the true 3-D view of described bore hole are shown to user after described semi-transparent semi-reflecting device place carries out original position volume rendering fusion, wherein, described stereopsis image is shown to user through after described semi-transparent semi-reflecting device, and the true 3-D view of described bore hole is shown to user after described semi-transparent semi-reflecting device reflection.
4. microscopic system according to claim 3, it is characterized in that, in described original position 3 d display device, also comprise between the true 3-D view of described bore hole and described semi-transparent semi-reflecting device: aerial image integrated unit and the controlled reflector element of light path, merge projection structure to realize easily extensible close-coupled.
5. microscopic system according to claim 3, it is characterized in that, also comprise: solid images amplifying unit, in true 3-D view unit for scaling and fused images unit for scaling one of at least, wherein, described solid images amplifying unit is for amplifying the size of the described stereopsis image before carrying out original position volume rendering fusion, described true 3-D view unit for scaling carries out the size of the true 3-D view of described bore hole before original position volume rendering fusion for convergent-divergent, described fused images unit for scaling carries out the picture size after original position volume rendering fusion for stereopsis image described in convergent-divergent and the true 3-D view of described bore hole.
6. microscopic system according to claim 1, is characterized in that, also comprises: aberration correction processing module, and described aberration correction processing module is used for aberration correction.
7. microscopic system according to claim 1, it is characterized in that, in described original position 3 d display device, be shown to user after carrying out home position manifestation after described stereopsis image and the true 3-D view of described bore hole are carried out registration, wherein the mode of registration is the registration of registration based on monumented point or no marks point.
8. strengthen a microscopic method for display based on original position solid, it is characterized in that, comprise the following steps:
A. in real time, directly stereopsis image is obtained;
B., the true 3-D view of bore hole relevant to described stereopsis picture material is provided;
C. user is shown to after described stereopsis image and the true 3-D view of described bore hole being carried out original position volume rendering fusion.
9. microscopic method according to claim 8, is characterized in that, described step B specifically comprises: provide bore hole true three-dimensional image source, by projection part in space Projection Display go out the corresponding true 3-D view of bore hole, wherein,
When naked eye three-dimensional display packing is based on 3 D full-figure technique type, described projection part is lens arra;
When naked eye three-dimensional display packing is poly-talented based on light field, described projection part is lens arra;
When naked eye three-dimensional display packing is based on holographic technique type, described projection part is interference device.
10. microscopic method according to claim 9, it is characterized in that, in described step C, user is shown to after described stereopsis image and the true 3-D view of described bore hole being carried out original position volume rendering fusion by semi-transparent semi-reflecting device, wherein, described stereopsis image is shown to user through after described semi-transparent semi-reflecting device, and the true 3-D view of described bore hole is shown to user after described semi-transparent semi-reflecting device reflection.
11. microscopic methods according to claim 10, is characterized in that, adopt easily extensible close-coupled to merge projection structure when carrying out light path design.
12. microscopic methods according to claim 10, is characterized in that, in also comprising the following steps one of at least:
Amplify the size of the described stereopsis image before carrying out original position volume rendering fusion;
Convergent-divergent carries out the size of the true 3-D view of described bore hole before original position volume rendering fusion;
Stereopsis image described in convergent-divergent and the true 3-D view of described bore hole carry out the picture size after original position volume rendering fusion.
13. microscopic methods according to claim 8, is characterized in that, also comprise step: carry out aberration correction process.
14. microscopic methods according to claim 8, it is characterized in that, in described step C, be shown to user after carrying out home position manifestation after described stereopsis image and the true 3-D view of described bore hole are carried out registration, wherein the mode of registration is the registration of registration based on monumented point or no marks point.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410498047.5A CN104280886A (en) | 2014-09-25 | 2014-09-25 | Microscopic system and microscopic method based on in-situ three-dimensional enhanced display |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410498047.5A CN104280886A (en) | 2014-09-25 | 2014-09-25 | Microscopic system and microscopic method based on in-situ three-dimensional enhanced display |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN104280886A true CN104280886A (en) | 2015-01-14 |
Family
ID=52255920
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410498047.5A Pending CN104280886A (en) | 2014-09-25 | 2014-09-25 | Microscopic system and microscopic method based on in-situ three-dimensional enhanced display |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104280886A (en) |
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104965302A (en) * | 2015-06-26 | 2015-10-07 | 深圳市虚拟现实科技有限公司 | Enhanced reality microscope |
| CN105093552A (en) * | 2015-09-17 | 2015-11-25 | 清华大学深圳研究生院 | Three-dimensional perspective method and system |
| CN105125177A (en) * | 2015-09-28 | 2015-12-09 | 郑州麦德杰医疗科技有限公司 | Semi-transparent visual guidance glasses for intravenous puncture |
| CN106873169A (en) * | 2015-12-10 | 2017-06-20 | 上海交通大学 | Three dimensional display |
| CN107203047A (en) * | 2017-07-28 | 2017-09-26 | 广州弥德科技有限公司 | The augmented reality system shown based on bore hole 3D |
| CN107907987A (en) * | 2017-12-26 | 2018-04-13 | 深圳科创广泰技术有限公司 | 3D microscopes based on mixed reality |
| CN108652824A (en) * | 2018-05-18 | 2018-10-16 | 深圳市莫廷影像技术有限公司 | Ophthalmic operating microscope system |
| CN108694730A (en) * | 2017-04-06 | 2018-10-23 | 赫克斯冈技术中心 | It is manipulated using the near field of the AR devices of image trace |
| CN109963141A (en) * | 2017-12-25 | 2019-07-02 | 广东虚拟现实科技有限公司 | Vision display system and method and head-wearing display device |
| CN110007455A (en) * | 2018-08-21 | 2019-07-12 | 腾讯科技（深圳）有限公司 | Pathology microscope, display module, control method, device and storage medium |
| CN111045202A (en) * | 2019-12-31 | 2020-04-21 | 浙江未来技术研究院(嘉兴) | Operating microscope |
| CN111239085A (en) * | 2019-03-06 | 2020-06-05 | 南昌工程学院 | Microscopic vision servo control method based on deep learning |
| WO2021057422A1 (en) * | 2019-09-25 | 2021-04-01 | 腾讯科技（深圳）有限公司 | Microscope system, smart medical device, automatic focusing method and storage medium |
| CN116719082A (en) * | 2023-04-25 | 2023-09-08 | 中国地质大学（北京） | Deep unfavorable geology multisource information fusion and in-situ stereoscopic perspective system and method |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103968829A (en) * | 2014-05-13 | 2014-08-06 | 清华大学 | Three-dimensional space orientation tracking method and three-dimensional space orientation tracking system based on virtual marker |
| CN103969838A (en) * | 2014-05-27 | 2014-08-06 | 清华大学 | Three-dimensional stereoscopic imaging method and device |
| US20140228653A1 (en) * | 2012-02-14 | 2014-08-14 | Marianna Kiraly | Device and method for displaying fetal positions and fetal biological signals using portable technology |
| US20140232746A1 (en) * | 2013-02-21 | 2014-08-21 | Hyundai Motor Company | Three dimensional augmented reality display apparatus and method using eye tracking |
-
2014
- 2014-09-25 CN CN201410498047.5A patent/CN104280886A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140228653A1 (en) * | 2012-02-14 | 2014-08-14 | Marianna Kiraly | Device and method for displaying fetal positions and fetal biological signals using portable technology |
| US20140232746A1 (en) * | 2013-02-21 | 2014-08-21 | Hyundai Motor Company | Three dimensional augmented reality display apparatus and method using eye tracking |
| CN103968829A (en) * | 2014-05-13 | 2014-08-06 | 清华大学 | Three-dimensional space orientation tracking method and three-dimensional space orientation tracking system based on virtual marker |
| CN103969838A (en) * | 2014-05-27 | 2014-08-06 | 清华大学 | Three-dimensional stereoscopic imaging method and device |
Non-Patent Citations (2)
| Title |
|---|
| HONGEN LIAO，TAKASHI INOMATA,TAKEYOSHI DOHI: "3-D Augmented Reality for MRI-Guided Surgery Using Intergral Videography Autostereoscopic Image Overlay", 《TRANSACTIONS ON BIOMEDICAL ENGINEERING》 * |
| HUY HOANG TRAN等: "Interactive 3D Navigation System for Iamge-guided Surgery", 《THE INTERNATIONAL JOURNAL OF VIRTUAL REALITY》 * |
Cited By (18)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104965302A (en) * | 2015-06-26 | 2015-10-07 | 深圳市虚拟现实科技有限公司 | Enhanced reality microscope |
| CN105093552A (en) * | 2015-09-17 | 2015-11-25 | 清华大学深圳研究生院 | Three-dimensional perspective method and system |
| CN105125177A (en) * | 2015-09-28 | 2015-12-09 | 郑州麦德杰医疗科技有限公司 | Semi-transparent visual guidance glasses for intravenous puncture |
| CN106873169A (en) * | 2015-12-10 | 2017-06-20 | 上海交通大学 | Three dimensional display |
| CN108694730A (en) * | 2017-04-06 | 2018-10-23 | 赫克斯冈技术中心 | It is manipulated using the near field of the AR devices of image trace |
| CN107203047A (en) * | 2017-07-28 | 2017-09-26 | 广州弥德科技有限公司 | The augmented reality system shown based on bore hole 3D |
| CN109963141A (en) * | 2017-12-25 | 2019-07-02 | 广东虚拟现实科技有限公司 | Vision display system and method and head-wearing display device |
| CN109963141B (en) * | 2017-12-25 | 2024-04-26 | 广东虚拟现实科技有限公司 | Visual display system and method and head-mounted display device |
| CN107907987A (en) * | 2017-12-26 | 2018-04-13 | 深圳科创广泰技术有限公司 | 3D microscopes based on mixed reality |
| CN108652824A (en) * | 2018-05-18 | 2018-10-16 | 深圳市莫廷影像技术有限公司 | Ophthalmic operating microscope system |
| CN110007455A (en) * | 2018-08-21 | 2019-07-12 | 腾讯科技（深圳）有限公司 | Pathology microscope, display module, control method, device and storage medium |
| WO2020038245A1 (en) * | 2018-08-21 | 2020-02-27 | 腾讯科技（深圳）有限公司 | Pathological microscope, display module, control method and device, and storage medium |
| US11982803B2 (en) | 2018-08-21 | 2024-05-14 | Tencent Technology (Shenzhen) Company Limited | Pathologic microscope, display module, control method and apparatus, and storage medium |
| CN111239085A (en) * | 2019-03-06 | 2020-06-05 | 南昌工程学院 | Microscopic vision servo control method based on deep learning |
| CN111239085B (en) * | 2019-03-06 | 2022-11-22 | 南昌工程学院 | Microscopic vision servo control method based on deep learning |
| WO2021057422A1 (en) * | 2019-09-25 | 2021-04-01 | 腾讯科技（深圳）有限公司 | Microscope system, smart medical device, automatic focusing method and storage medium |
| CN111045202A (en) * | 2019-12-31 | 2020-04-21 | 浙江未来技术研究院(嘉兴) | Operating microscope |
| CN116719082A (en) * | 2023-04-25 | 2023-09-08 | 中国地质大学（北京） | Deep unfavorable geology multisource information fusion and in-situ stereoscopic perspective system and method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104280886A (en) | Microscopic system and microscopic method based on in-situ three-dimensional enhanced display | |
| US11217028B2 (en) | Surgeon head-mounted display apparatuses | |
| Rolland et al. | Optical versus video see-through head-mounted displays in medical visualization | |
| US11628038B2 (en) | Multi-option all-digital 3D surgery visualization system and control | |
| JP6058290B2 (en) | Image processing system, apparatus, method, and medical image diagnostic apparatus | |
| JP5909055B2 (en) | Image processing system, apparatus, method and program | |
| JP6116754B2 (en) | Device for stereoscopic display of image data in minimally invasive surgery and method of operating the device | |
| WO2014037953A2 (en) | Digital system for surgical video capturing and display | |
| US11092929B2 (en) | Visualization system and method for generating holographic presentations from optical signals | |
| JP6147464B2 (en) | Image processing system, terminal device and method | |
| JP7404534B2 (en) | Surgical applications using integrated visualization camera and optical coherence tomography | |
| US20200371338A1 (en) | Microscope assembly for capturing and displaying three-dimensional images of a sample | |
| US11698535B2 (en) | Systems and methods for superimposing virtual image on real-time image | |
| JP2012242906A (en) | Image processing system, device and method | |
| JP2013009040A (en) | Image processing system, device, and method | |
| US20130172735A1 (en) | Optical coherent imaging medical device | |
| Liao et al. | Intra-operative real-time 3-D information display system based on integral videography | |
| Rolland et al. | Optical versus video see-through head-mounted displays | |
| CN110169821A (en) | A kind of image processing method, apparatus and system | |
| JP5921102B2 (en) | Image processing system, apparatus, method and program | |
| Van der Jeught et al. | Real-time structured light-based otoscopy for quantitative measurement of eardrum deformation | |
| US20240070970A1 (en) | Visualization system with stereo-volumetric intraoperative oct and stereoscopic camera | |
| Dunkin et al. | 3d in the minimally invasive surgery (mis) operating room: Cameras and displays in the evolution of mis | |
| Qian | Augmented Reality Assistance for Surgical Interventions Using Optical See-through Head-mounted Displays | |
| TW202418223A (en) | Head wearable virtual image module for superimposing virtual image on real-time image |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20150114 |