CN109088803A - A kind of AR remote control device, intelligent domestic remote control system and method - Google Patents
A kind of AR remote control device, intelligent domestic remote control system and method Download PDFInfo
- Publication number
- CN109088803A CN109088803A CN201811102831.4A CN201811102831A CN109088803A CN 109088803 A CN109088803 A CN 109088803A CN 201811102831 A CN201811102831 A CN 201811102831A CN 109088803 A CN109088803 A CN 109088803A
- Authority
- CN
- China
- Prior art keywords
- smart home
- instruction
- gesture
- remote control
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 38
- 230000033001 locomotion Effects 0.000 claims description 62
- 230000003993 interaction Effects 0.000 claims description 32
- 238000004891 communication Methods 0.000 claims description 12
- 230000006399 behavior Effects 0.000 claims description 10
- 230000000007 visual effect Effects 0.000 claims description 10
- 230000004424 eye movement Effects 0.000 claims description 7
- 230000002452 interceptive effect Effects 0.000 claims description 5
- 230000006855 networking Effects 0.000 claims description 3
- 238000005516 engineering process Methods 0.000 abstract description 22
- 210000003811 finger Anatomy 0.000 description 26
- 238000012790 confirmation Methods 0.000 description 23
- 210000003128 head Anatomy 0.000 description 19
- 239000011521 glass Substances 0.000 description 15
- 238000010586 diagram Methods 0.000 description 9
- 238000004378 air conditioning Methods 0.000 description 8
- 230000000694 effects Effects 0.000 description 8
- 238000010304 firing Methods 0.000 description 7
- 230000003287 optical effect Effects 0.000 description 6
- 230000006870 function Effects 0.000 description 5
- 230000006872 improvement Effects 0.000 description 5
- 230000009286 beneficial effect Effects 0.000 description 4
- 210000003813 thumb Anatomy 0.000 description 4
- 238000004458 analytical method Methods 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 3
- 238000005286 illumination Methods 0.000 description 3
- 238000007654 immersion Methods 0.000 description 3
- 241000287181 Sturnus vulgaris Species 0.000 description 2
- 230000004075 alteration Effects 0.000 description 2
- 230000003190 augmentative effect Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 230000005611 electricity Effects 0.000 description 2
- 238000005265 energy consumption Methods 0.000 description 2
- 230000005764 inhibitory process Effects 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 238000005057 refrigeration Methods 0.000 description 2
- 238000010408 sweeping Methods 0.000 description 2
- 230000009897 systematic effect Effects 0.000 description 2
- 238000012800 visualization Methods 0.000 description 2
- 239000011800 void material Substances 0.000 description 2
- 241000251468 Actinopterygii Species 0.000 description 1
- 238000005273 aeration Methods 0.000 description 1
- 230000003321 amplification Effects 0.000 description 1
- 210000001367 artery Anatomy 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 210000000988 bone and bone Anatomy 0.000 description 1
- 235000008429 bread Nutrition 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 229910052799 carbon Inorganic materials 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- QVFWZNCVPCJQOP-UHFFFAOYSA-N chloralodol Chemical compound CC(O)(C)CC(C)OC(O)C(Cl)(Cl)Cl QVFWZNCVPCJQOP-UHFFFAOYSA-N 0.000 description 1
- 235000019504 cigarettes Nutrition 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000009977 dual effect Effects 0.000 description 1
- 210000004247 hand Anatomy 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 230000006698 induction Effects 0.000 description 1
- 238000007689 inspection Methods 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 210000003041 ligament Anatomy 0.000 description 1
- 210000004932 little finger Anatomy 0.000 description 1
- 235000012054 meals Nutrition 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 210000003205 muscle Anatomy 0.000 description 1
- 210000005036 nerve Anatomy 0.000 description 1
- 238000003199 nucleic acid amplification method Methods 0.000 description 1
- 238000011022 operating instruction Methods 0.000 description 1
- 238000000746 purification Methods 0.000 description 1
- 238000009738 saturating Methods 0.000 description 1
- 238000012163 sequencing technique Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 238000005406 washing Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L12/00—Data switching networks
- H04L12/28—Data switching networks characterised by path configuration, e.g. LAN [Local Area Networks] or WAN [Wide Area Networks]
- H04L12/2803—Home automation networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L12/00—Data switching networks
- H04L12/28—Data switching networks characterised by path configuration, e.g. LAN [Local Area Networks] or WAN [Wide Area Networks]
- H04L12/2803—Home automation networks
- H04L12/2816—Controlling appliance services of a home automation network by calling their functionalities
Landscapes
- Engineering & Computer Science (AREA)
- Automation & Control Theory (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Selective Calling Equipment (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
The invention discloses a kind of AR remote control device, intelligent domestic remote control system and methods, are related to the field AR, which includes: that AR shows equipment, for capturing the image of smart home；Instruction control equipment sends control command code signal to the smart home for receiving control instruction, and according to the control instruction.The present invention is based on the remote controler of AR technology, the user of any remote controler is allowed, all feel succinctly and conveniently, be Gospel emperorship especially to the sufferer for losing language competence；It assists more allowing old man and child user to feel warm with language semantic identification technology again.
Description
Technical field
The present invention relates to field of remote control more particularly to a kind of remote control device based on AR technology, based on AR technology
Intelligent domestic remote control system and remote control implementation method.
Background technique
The remote control of electronics/smart home, is generally completed by infrared remote controller, for different electronics/intelligence man
It occupies and needs different remote controlers, often will appear and hold the case where air-conditioning remote control operates TV, or look for not for the moment
The phenomenon that going mad to desired remote controler.
Have in the prior art and is uniformly controlled by wisdom speaker, but the disadvantage is that, the spoken command of user, with specific electricity
Son/smart home control instruction may not match, for example user instruction is equivocal or ambiguous, it is therefore desirable to intelligence
Intelligent speaker has the neural network retrieval technique of AI chip, carries out the language semantic analysis of a deep step, could go back the operation of original subscriber
Instruction, and AI chip introducing undoubtedly can larger production cost.
In addition, having vision and consciousness, but the not user of language competence, intelligent family can not be just carried out by wisdom speaker
The operation in residence.
Application No. is 201810050821.4 inventions, provide a kind of aobvious optical system of the see-through head of nearly eye, including first
Lens, the second lens and a miniature image display, the first lens are all bonded with miniature image display with the second lens, and
First lens and the second lens are all equal thick free-form surface lens.Optics is shown by the nearly see-through head of eye provided by the present invention
System architecture can also eliminate miniature image and show other than it can reduce the number that light reflects in this optical system framework
The aberration of the light that device is issued in all directions, makes it see that image not will cause aberration with angle in all directions.
Application No. is 201821172477.8 utility model, an AR is provided on the basis of above-mentioned optical system shows and set
Standby, wherein the AR shows that equipment includes at least one display/projector and at least one Optical devices, real image passes through the light
Device building is learned, the virtual image is constructed in the Optical devices by the display/projector, and the real image and the virtual image combine and occur
In same visual field, wherein the Optical devices include at least one main lens and at least one free-form surface lens, it is described
Free-form surface lens, the image projected for reflecting the display/projector；The main lens, for described in secondary reflection again
The image that display/projector projects.The AR equipment can accomplish big visual angle, small size, light weight, and wearing user can have
The experience sense of immersion AR.
Application No. is 201810994555.0 inventions, further provide one on the basis of aforementioned AR shows equipment
Formula actual situation interactive device and actual situation exchange method are worn, wherein the wear-type actual situation interactive device includes wearing AR to show equipment,
For by least showing 3D virtual image in described wear in the visual field that AR shows equipment；Device is interacted with actual situation, for obtaining
AR is worn described in wearing shows the behavior act of equipment person perhaps control instruction and according to the behavior act or control instruction
And change or control the content of the 3D virtual image.
Summary of the invention
Can be on the Nonopaque type AR technology of above-mentioned referenced patents or the basis of image-type AR technology, the present invention provides one
Kind AR remote control device, intelligent domestic remote control system and method allow, and it is an object of the present invention to provide a kind of new remote control device and remote control mode
User is more convenient on the basis of being based on AR technology, quickly uses remote controler, and suitable crowd is wide.
Technical solution provided by the invention is as follows:
A kind of AR remote control device, comprising: AR shows equipment, for capturing smart home/intelligent household appliances image；Instruction control
Control equipment sends control command code letter to the smart home for receiving control instruction, and according to the control instruction
Number.
In the above-mentioned technical solutions, instruction control equipment and AR are shown that equipment combines and AR remote control device is made, visualized
Remote control device, any smart home in family can be driven, it is easy to use.
Further, described instruction control equipment includes: that module occurs for instruction, for receiving the control instruction；Communicate mould
Block, for sending control command code signal to the smart home according to the control instruction.
Further, described instruction occur module include: gesture interaction component, head take aim at component (Helmet Sighting), eye movement with
Track component (eye-tracking camera), wisdom gloves, wisdom ring, control handle and/or language control component；The gesture interaction portion
Part, for when the gesture motion for detecting that user makes is identical as default gesture, then it is assumed that receive the control of user's input
Instruction；Language control component, for receiving the control instruction of voice input.
In the above-mentioned technical solutions, there are many control modes on AR remote control device, it is greatly enriched AR remote control device
Realization may and application range.
Further, the gesture interaction component detects that the gesture motion that user makes is identical as default gesture for working as
When specifically: the gesture interaction component, for when detect first gesture movement that user makes with it is pre- in default gesture
If the movement of first gesture is identical, user makes second gesture is identical as the default second gesture in default gesture and described the
Time difference between one gesture motion and second gesture movement is in preset time range, then it is assumed that detects that user makes
Gesture motion it is identical as default gesture.
In the above-mentioned technical solutions, input of the linkage gesture as control instruction, accuracy are high.
Further, the control instruction includes gesture instruction, phonetic order, head takes aim at instruction, eye movement instructs and/or menu refers to
It enables.
Further, the camera in AR remote controler further comprises: depth camera, RGB camera, TOF camera, fish
Eye imaging head and/or wide-angle camera.AR remote control device further include: picture recognition module, for obtaining the intelligence in environment
The image or identification code of energy household；Image comparison module, for being compared according to the image or identification code of the smart home
Determine the type, brand and model of the smart home.
Further, AR remote control device further include: memory module determines described in type, brand and model for pre-stored
Image or identification code (bar code, two dimensional code, the AR identification trigger flag attached on smart home main body cover of smart home
Deng), it is called when being compared for described image comparison module；The memory module is also used to pre-stored determining type, brand and model
The smart home control command code signal, control equipment for described instruction and send institute to the specific smart home
State control command code signal.
In the above-mentioned technical solutions, it is stored with the data of various smart homes in AR remote control device, makes the control of smart home
Make more humane, more convenientization.
Further, AR remote control device further include: Web crawler, for searching for described in determining type, brand and model
The image or identification code of smart home call when comparing for described image comparison module；The Web crawler is also used to search for
The control command code signal for determining the smart home of type, brand and model, for described instruction control equipment to
The specific smart home sends the control command code signal.
In the above-mentioned technical solutions, if local not stored, can voluntarily network downloading, and intelligence degree is higher, user experience
More preferably.
Further, AR remote control device further include: virtual objects generation module, for assisting described instruction control equipment apparent
The control instruction.
Further, during the virtual objects that the virtual objects generation module generates include: virtual controlling command menu, are virtual
Jie's object, the Virtual Space 3D environment or 3D map image.
In the above-mentioned technical solutions, virtual objects generation module further assists the clear control instruction of user, and control is more
Precision, convenient purification, diversification.
The present invention also provides a kind of intelligent domestic remote control systems, comprising: smart home and AR remote control device, the AR remote control
Device, for issuing control command code signal to the smart home.
Further, intelligent domestic remote control system further include: second camera module, for obtaining the intelligent family at a distance
The image in residence, and it is sent to the AR remote control device.
Further, the second camera module further comprises: IP Camera, unmanned aerial vehicle onboard camera or nobody
Vehicle vehicle-mounted camera.
Further, the AR remote control device includes instruction control equipment, for receiving the control instruction, and according to institute
It states control instruction and sends control command code signal to the smart home.
Further, the control instruction includes gesture instruction, phonetic order, head takes aim at instruction, eye movement instructs and/or menu refers to
It enables.
Further, described instruction control equipment includes: that module occurs for instruction, for receiving the control instruction；Communicate mould
Block, for sending control command code signal to the smart home according to the control instruction.
Further, it includes: that gesture interaction component, head take aim at component, eye-tracking component, wisdom hand that module, which occurs, for described instruction
Set, wisdom ring, control handle and/or language control component, institute's predicate control component, for receiving the control instruction of voice input；AR
Remote control device further includes the semantics recognition module, is further used for detect that the behavior of user meets preset trigger condition
When, issue inquiry message；And receive the object confirmation instruction that user inputs according to the inquiry message voice.
The gesture interaction component, for when the gesture motion for detecting that user makes is identical as default gesture, then recognizing
For the control instruction for receiving user's input.
Further, intelligent domestic remote control system further include: picture recognition module, for obtaining the intelligent family in environment
The image or identification code in residence；Image comparison module is determined for being compared according to the image or identification code of the smart home
Type, brand and the model of the smart home.
Further, the intelligent domestic remote control system includes: memory module and/or Web crawler；The memory module,
For the image or identification code of the pre-stored smart home for determining type, brand and model, for described image comparison module
It is called when comparison；The memory module, the control for being also used to the pre-stored smart home for determining type, brand and model refer to
Encoded signal is enabled, equipment is controlled for described instruction and sends the control command code signal to the specific smart home；Institute
Web crawler is stated, the image or identification code of the smart home for searching for determining type, brand and model, for the figure
As being called when comparison module comparison；The Web crawler is also used to search for the intelligent family of determining type, brand and model
The control command code signal occupied controls equipment for described instruction and refers to the specific smart home transmission control
Enable encoded signal.
The present invention also provides a kind of remote control methods of smart home, comprising: S100: obtaining and show the shadow of smart home
Picture；S300: (i.e. according to the object to be controlled to the smart home being specifically directed towards, shown in the image of the smart home
When receiving the smart home confirmation instruction of (user) input, is confirmed according to the smart home and instructed, confirm the institute of display
State the smart home to be controlled in the image of smart home, i.e., object to be controlled)；S500: according to the object to be controlled
Control instruction, which generates corresponding control command code signal, (ought receive the event assigned according to the smart home to be controlled
When control instruction, corresponding control command code signal is generated according to event control instruction)；S600: the control is referred to
It enables encoded signal be sent to the object to be controlled, the object to be controlled is made to execute corresponding behaviour according to the control signal
Make.
Further, before the S300 step further include: S200: when the gesture motion and default hand for detecting that user makes
When gesture is identical, it is determined that the smart home being specifically directed towards in the image of the smart home (is thought to receive user's input
Smart home confirm instruction).
Further, the S200 step specifically: S210: when detecting (user makes) first gesture movement and default
Default first gesture in gesture is identical, the movement of (user makes) second gesture and the default second gesture phase in default gesture
Time difference between the same and described first gesture movement and second gesture movement is in preset time range, then it is assumed that inspection
It is identical as default gesture to measure the gesture motion that user makes.
Further, the S300 step includes: S310: according to the phonetic order to the smart home being specifically directed towards,
Show in the image of the smart home object to be controlled (i.e. when receive voice input smart home confirmation instruction when,
The smart home to be controlled in the image for the smart home that confirmation command is shown is confirmed according to the smart home).
Further, after the S100 step further include: S150: when the behavior of detecting meets preset trigger condition, hair
Inquiry message out；The S300 step includes: S330: according to the phonetic order to the smart home, showing the shadow
As in object to be controlled (i.e. when receive according to the inquiry message voice input smart home confirmation instruction when, according to
The smart home to be controlled in the image that the smart home confirmation confirmation command is shown).
Further, the image for smart home being obtained in the S100 step includes: S110: obtaining AR remote control by camera
The image of the smart home around device visual field；Or, S120: being taken the photograph by receiving the auxiliary that second camera module is sent
As the image around module visual field.
Further, between the S300 step and the S500 step further include: S470: search in the preset database to
Type, brand and the model (the virtual controlling command menu of the i.e. described smart home to be controlled) of control object, and described in display
Virtual controlling command menu；The S500 step specifically: S510: dish is instructed according to the virtual controlling to the object to be controlled
The control instruction singly assigned, generating the corresponding control command code signal (ought receive user according to display
When the event control instruction that the virtual controlling command menu of the smart home to be controlled is assigned, instructed according to the event control
Generate the corresponding control command code signal).
Further, before the S470 step further include: S410: when not finding object to be controlled in the preset database
When the type, brand and model, the identification information for scanning the object to be controlled (does not find the smart home to be controlled
Virtual controlling command menu when, scan the identification information of the smart home to be controlled)；S430: according to the identification information
The control command code signal and/or virtual controlling command menu that the object to be controlled is downloaded in networking are to the preset data
Library.
Further, the S100 step includes: S180: according to phonetic order, showing that the relevant 3D of the phonetic order is empty
Quasi- space environment or 3D map image (i.e. when receiving the phonetic control command that user assigns, obtain and show the voice
The relevant Virtual Space the 3D environment of control instruction or 3D map image)；The S300 step includes: S350: according to referring specifically to
To the smart home, show that the object to be controlled in the Virtual Space 3D environment or 3D map image (ought receive
When user confirms instruction according to the smart home that the Virtual Space 3D environment or 3D map image input, according to the intelligent family
Occupy the smart home to be controlled in the Virtual Space the 3D environment or 3D map image that confirmation confirmation command is shown).
Compared with prior art, AR remote control device of the invention, intelligent domestic remote control system and method technical effect or
Person's beneficial effect is: be visually remotely controlled various intelligent electric appliance/smart homes, overcome existing single voice remote controller or
The inadequate natural endowment that infrared single remote controller is directly remotely controlled, an AR remote controler can drive any wisdom electric appliance/intelligence man in family
ON/OFF, operation and the adjusting in residence.
Detailed description of the invention
Below by clearly understandable mode, preferred embodiment is described with reference to the drawings, to a kind of AR remote control device, intelligence
Home remote control system and above-mentioned characteristic, technical characteristic, advantage and its implementation of method are further described.
Fig. 1 is AR remote control device structural schematic diagram of the present invention；
Fig. 2 is the structural schematic diagram of image-type AR head-mounted display of the present invention with gesture interaction component；
Fig. 3 is the structural schematic diagram of Nonopaque type AR head-mounted display of the present invention with gesture interaction component；
Fig. 4 is the structural schematic diagram of gesture interaction component induction finger tip of the present invention；
Fig. 5 is intelligent domestic remote control system first embodiment schematic diagram of the present invention；
Fig. 6 is the flow chart of Intelligent household remote control method first embodiment of the present invention；
Fig. 7 is the flow chart of the deformation of Intelligent household remote control method first embodiment of the present invention；
Fig. 8 is the schematic diagram of intelligent domestic remote control system second embodiment of the present invention；
Fig. 9 is the schematic diagram of image in head-up visual field in apartment space of the present invention；
Figure 10 is the flow chart of Intelligent household remote control method second embodiment of the present invention；
Figure 11 is the schematic diagram of intelligent domestic remote control system 3rd embodiment of the present invention；
Figure 12 is the flow chart of Intelligent household remote control method 3rd embodiment of the present invention；
Figure 13 is the flow chart of Intelligent household remote control method 3rd embodiment deformation of the present invention；
Figure 14 is the three dimensional top schematic diagram of a scenario in virtual apartment space of the invention.
Drawing reference numeral explanation:
100.AR remote control device/device, 1.AR show equipment, 2. instruction control equipment, and module, 22. communications occur for 21. instructions
Module, 3. picture recognition modules, 4. image comparison modules, 5. memory modules, 6. Web crawlers, 7. virtual objects generate mould
Block, 8. gesture interaction components, 200. smart homes/intelligent household appliances, 300. second camera modules.
Specific embodiment
In order to more clearly explain the embodiment of the invention or the technical proposal in the existing technology, Detailed description of the invention will be compareed below
A specific embodiment of the invention.It should be evident that drawings in the following description are only some embodiments of the invention, for
For those of ordinary skill in the art, without creative efforts, it can also be obtained according to these attached drawings other
Attached drawing, and obtain other embodiments.
To make simplified form, part related to the present invention is only schematically shown in each figure, they are not represented
Its practical structures as product.In addition, there is identical structure or function in some figures so that simplified form is easy to understand
Component only symbolically depicts one of those, or has only marked one of those.Herein, "one" is not only indicated
" only this ", can also indicate the situation of " more than one ".
Embodiment one
In an Installation practice according to the present invention, as shown in Figure 1, a kind of AR remote control device (remote controler) 100, packet
Include: AR shows equipment 1, for capturing smart home/intelligent household appliances image；Instruction control equipment 2 shows that equipment 1 is logical with AR
Letter connection sends control command code letter to the smart home for receiving control instruction, and according to the control instruction
Number.
Specifically, AR (Augmented Reality, augmented reality) is a kind of position for calculating camera image in real time
And angle and plus respective image, video, 3D model technology, the target of this technology is on the screen virtual world " set "
In real world and interacted.
AR shows that equipment 1 includes: that immersion helmet-type shows equipment, AR all-in-one machine, digital glass mirror surface, digital glass table
Face, display, Intelligent mobile equipment (smart phone, Intelligent flat etc.), alternatively, the AR head of the Intelligent mobile equipment can be connected
Helmet；The immersion helmet-type shows that equipment includes image-type head-mounted display, transmission-type head-mounted display, alternatively, image is saturating
Penetrate combined type head-mounted display；Wearable smart machine, such as: AR glasses, AR helmet etc..
Instruction control equipment is subjected to the control instruction of user (voice/gesture) input, and control instruction is had by content point: intelligence
Energy household confirmation instruction, control instruction and event control instruction etc..
Smart home confirmation instruction/control instruction determines that user wants which smart home controlled, and event control instruction is control
What smart home processed does, such as: open/close TV, sound equipment, headlamp, air-conditioning, humidifier, electric fan, air cleaning
Device, intelligent curtain, sweeping robot, IP Camera, intelligent aeration, gate inhibition etc..
Instruction control equipment can be according to the control instruction and smart home to be controlled (i.e. some intelligence being specifically designated received
Energy household) corresponding control command code signal is generated, and it is sent to it, it controls its and executes corresponding operation.
Optionally, instruction control equipment 2 includes: that module 21 occurs for instruction, for receiving the control instruction；Communication module
22, for sending control command code signal to the smart home according to the control instruction.
Specifically, communication module includes: infrared emission component, (Near Field Communication, near field are logical by NFC
Letter), Wi-Fi, bluetooth, the communication standards such as ZigBee, Li-Fi, Z-Wave chip.
The visible light wireless communication of Li-Fi--Light Fidelity, light online, i.e. illumination are surfed the Internet -- with LED illumination lamp
The light of sending is to utilize quick light pulse wireless transmission information as the delivery means of network signal.That is it is not necessarily to Wi-
Fi signal, putting a LED light can surf the Internet.Its principle be according to different rates in light encoded information, such as LED game clock shows 1,
Closing indicates 0, can transmit information by quick HF switch.LED light network by visible light come transmission network signal, can be with
Directly dual is completed using the existing energy consumption output such as flashlight, headlamp, street lamp, room lighting and public illumination
Business.The characteristics of light is surfed the Internet is Low emissivity, low energy consumption, low-carbon environment-friendly.
It includes: that gesture interaction component, head are taken aim at component, eye-tracking component, wisdom gloves, wisdom and guarded against that module 21, which occurs, for instruction
Refer to, control handle and/or language control component.Different components, are divided into control instruction by input mode: gesture instruction, phonetic order,
Head takes aim at instruction, eye movement instruction and/or menu instructions.
When head takes aim at component applied to AR glasses or the AR helmet, the IMU such as the gyroscope in AR glasses or the AR helmet are relied on
Sensor determines orientation.
It is mainly the positioning and operating function for realizing mouse that each component in module, which occurs, for instruction.
Gesture interaction component, for when the gesture motion for detecting that user makes is identical as default gesture, then it is assumed that connect
Receive the control instruction of user's input；Specifically, gesture interaction component is the body-sensing sensor about gesture, and such as: LEAP
MOTION, uSens Fingo etc..
Language control component, for receiving the control instruction of voice input.Language control component includes: microphone and semantic phonetic decision
Submodule.
Preferably, gesture interaction component 8, for the tool when the gesture motion for detecting that user makes is identical as default gesture
Body are as follows: the gesture interaction component, for when detecting first gesture movement that user makes and default the in default gesture
The second gesture movement that one gesture is identical, user makes is identical and described first-hand as the default second gesture in default gesture
Time difference between gesture movement and second gesture movement is in preset time range, then it is assumed that detects the hand that user makes
Gesture movement is identical as default gesture.
AR remote control device 100 further include: picture recognition module 3, for obtaining the image of the smart home in environment
Or identification code (identification code can identify trigger flag etc. for bar code, two dimensional code, AR)；Image comparison module 4 is known with image
Other module 3 is electrically connected, for determined according to the comparison of the image or identification code of the smart home smart home type,
Brand and model.
Memory module 5, is electrically connected with image comparison module 4 and instruction control equipment, determines type, brand for pre-stored
With the image or identification code of the smart home of model, called when being compared for described image comparison module；The memory module,
Being also used to the control command code signal of the pre-stored smart home for determining type, brand and model, (and virtual controlling refers to
Enable menu), equipment, which is controlled, for described instruction sends the control command code signal to the specific smart home.
Specifically, being explained and illustrated using the working principle of LEAP MOTION as gesture interaction component 8: with LEAP
For MOTION, it is a manual interactive device of micron order 3D, can track as low as 0.01 millimeter of movement, possess 150
The visual angle of degree, can track the movement of 10 fingers of 1 people, and maximum frequency is 290 frame per second.What LEAP MOTION was built
Gesture is captured in space, shows that hand and arm skeleton, the hand of people have 29 pieces of bones, 29 joints, 123 ligaments, 48
Nerve and 30 arteries.LEAP MOTION controller also almost grasps this technology, that is to say, that it can not be detected
Non- manpower energy and gesture motion.
LEAP MOTION space overlapping enters AR three-dimensional display space, and the object in gesture and AR three-dimensional display space carries out
Interaction/feedback, LEAP MOTION controller can be tracked accurately.Technically, this is 8 cubic feet interacts
Formula 3d space.10 fingers of the traceable whole people of Leap Motion controller, precision are up to 1/100 millimeter.It is more than existing
Movement control technology it is more accurate.The spatial field of view of 150 ° of ultra-wides, can be as in real world with the sky for being intended to 3D
Between mobile both hands.LEAP MOTION controller is mobile with the hand of the Velocity Pursuit user more than 200 frame per second, realizes perfect
It is synchronous.
Identified object includes thumb (Thumb finger), index finger (Index finger), middle finger (Middle
Finger), the third finger (Ring finger), little finger (Pinky finger), its available initial position of each finger
Start (X, Y, Z), final position end (X, Y, Z) and direction direction (pitch, roll, yaw).
Fig. 2 is the AR head-mounted display (image-type HMD) with gesture interaction component, and, Fig. 3 is that another has hand
The AR head-mounted display (Nonopaque type HMD) of gesture interactive component.
Identify that the specific example of control instruction is as follows using gesture interaction component 8:
1. utilizing Unity3D engine, the virtual scene space of a 3D is built, creates certain 3D void in the virtual scene space
Quasi- object, such as virtual controlling command menu, virtual controlling command menu.
2. accessing SDK (Software Development Kit (the foreign language acronym: SDK, foreign language of the six degree of freedom of high pass production
Full name: Software Development Kit) be typically all some software engineers be specific software package, software frame,
The set of developing instrument when hardware platform, operating system etc. establish application software.).Number is positioned by the gyroscope in equipment
According to, position of the calculating equipment in virtual scene, while being mapped in the virtual scene space of Unity3D creation, to realize
The 6DOF effect that can be rotated and walk in the 3 d space.
3. accessing the SDK for the identification gesture (spatial parameter) that Leap Motion is provided, hand is added in virtual scene space
Model (including hand and arm).The driving and hardware device for needing exist for Leap Motion are supported.LeapSDK can be
It drives the gesture information parameter detected to be transmitted to Unity3D. to map in these information to hand model, can be realized will be true
Hand is modeled to virtual hand, and is presented in virtual 3D scene space.
4. the information of gesture is carried out analytical calculation in Unity3D engine, some specific gesture shapes are obtained, such as
The movement of " firing finger " of the movement of " in the referring to " of index finger finger and middle finger and thumb on hand, " in referring to " indicate similar and use mouse
Mark is chosen, and " firing finger " indicates to use mouse-click.
Analysis obtains the beginning and end acted " in referring to ".At the beginning and end of prolonging according to index finger tip or finger tip
The spatial position for being directed toward smart home or virtual controlling command menu of long line (as shown in Figure 4).When index finger tip distance intelligence
It is to enter the state of " in referring to " that the space length of electric appliance or virtual controlling command menu, which is less than some threshold value, is eaten after waiting the several seconds
Refer to that finger tip is greater than some threshold value apart from the distance of intelligent electric appliance or virtual controlling command menu, as leave the state " in referring to ",
An above-mentioned whole flow process is recognized and is acted " in referring to ".
It is noted that if the object of " finger " is distant, if concrete intelligence household image not enough understands, Ke Yitong
The wide-angle camera in AR equipment is crossed, further the smart home of amplification display remotely.Then it carries out operating " in referring to " again.
5. can identify " in referring to " act after, analysis " firing fingers " act beginning and end, at the beginning and end of according to
It is to enter " to beat when the distance between two finger tips is less than some threshold value according to middle fingertip at a distance from thumb tip (as shown in Figure 4)
The state of snap " is greater than some threshold value, as leaves the state of " firing finger "." in referring to " and " firing finger " is a joint operation,
The completion that the movement of the latter is completed with previous movement has interval time requirement, more than the time of setting, even if this 2 movements
It successively completes, any operation will not be caused.Certainly, the movement of " determination " can freely be set, however it is not limited to above-mentioned gesture.
If being recognized within interval time " in referring to ", the movement of " firing finger ", then it is assumed that gesture instruction is received, according to
The intelligence that " in referring to " is directed toward occupies, virtual controlling command menu, determines smart home to be controlled and to want smart home to be controlled to do assorted
Thing.
6. using platform (including Android, iOS, PSP, xbox and PC etc.) at corresponding hardware by the publication of Unity3D engine
Application program.
AR shows the virtual controlling command menu that equipment is shown, draws from specific smart home to be controlled, i.e., from AR
Display equipment sees the scene that virtual object and physical objects be combined with each other in show, and the presentation mode of virtual controlling command menu can be with
It is text push button, icon formula, is also possible to one section of animation or video image broadcasting.Preferably, AR remote control device also wraps
Loudspeaker is included, there is the feedback of sound or sound equipment, reinforces user mutual with virtual controlling command menu, virtual controlling command menu
Experience effect when dynamic.
Other than head takes aim at the control technologies such as (Helmet Sighting), eye-tracking, wisdom gloves, AR glasses can be in conjunction with another 2
Kind control technology, one is (unarmed) gesture control (cooperation for needing body-sensing sensor hardware), the other is sound control (needs body
The cooperation of the hardware such as microphone and voice, Semantic judgement module).The AR remote controler of the present embodiment refers to according to the control that user inputs
It enables, controls corresponding smart home, instead of original remote controler, and user can be used different modes and send control instruction,
Multiple choices are given, the range of target user is improved.
According to the first systematic embodiment that this product is invented, as shown in Figure 5, comprising: smart home 200 is (i.e. in Fig. 5
Each intelligent household appliances 1-n) and AR remote control device 100 (i.e. AR remote controler in Fig. 5), AR remote control device 100, for the intelligence
Household issues control command code signal.
AR remote control device in the present embodiment is the AR remote control device in above-described embodiment, and identical part is no longer superfluous herein
It states.Smart home/intelligent household appliances include but is not limited to: TV, sound equipment, headlamp, air-conditioning, washing machine, refrigerator, humidifier, electricity
Fan, air purifier, intelligent curtain, sweeping robot, IP Camera, intelligence lead to/air draft mechanism, gate inhibition, micro-wave oven, electric meal
Pot, oven, range hood, egg-whisk, bread producing machine, wall-breaking machine, computer, PAD, mobile phone, socket (row), alarm clock, cigarette sense, air
Switch etc..
AR remote control device 100 includes instruction control equipment, is referred to for receiving the control instruction, and according to the control
It enables to the smart home and sends control command code signal.Control instruction include gesture instruction, phonetic order, head take aim at instruction,
Eye movement instruction and/or menu instructions.
Instruction control equipment includes: that module occurs for instruction, for receiving the control instruction；Communication module is used for basis
The control instruction sends control command code signal to the smart home.
Optionally, it includes: that gesture interaction component, head take aim at component (Helmet Sighting), eye-tracking portion that module, which occurs, for instruction
Part, wisdom gloves, wisdom ring, control handle and/or language control component, institute's predicate control component, for receiving the control of voice input
System instruction；Gesture interaction component, for when the gesture motion for detecting that user makes is identical as default gesture, then it is assumed that receive
The control instruction inputted to user.
Intelligent domestic remote control system further include: picture recognition module, for obtaining the shadow of the smart home in environment
Picture or identification code；Image comparison module, for determining the intelligence according to the comparison of the image or identification code of the smart home
Type, brand and the model of energy household；
And memory module, image or identification for the pre-stored smart home for determining type, brand and model
Code calls when comparing for described image comparison module；Memory module is also used to the pre-stored institute for determining type, brand and model
The control command code signal for stating smart home controls equipment for described instruction and sends the control to the specific smart home
Instruction encoding signal processed；
And Web crawler, for searching for image or the identification of the smart home of determining type, brand and model
Code calls when comparing for described image comparison module；
The Web crawler is also used to search for the control of the smart home of determining type, brand and model
Instruction encoding signal controls equipment for described instruction and sends the control command code signal to the specific smart home；
And virtual objects generation module, for assisting described instruction to control the apparent control instruction of equipment；It is virtual right
As generation module includes: virtual controlling command menu.
In other embodiments, intelligent domestic remote control system can only include a memory module or Web crawler.
AR remote control device part identical with the AR remote control device in above-mentioned apparatus embodiment in the present embodiment is herein not
It repeats again.Smart home 200 and AR remote control device 100 communicate to connect.
The intelligent domestic remote control system of the present embodiment can control smart home using AR remote control device, the scope of application
Extensively, easy to use, substantially increase the usage experience of user.
First embodiment of the method according to the present invention, the smart home 200 and AR applied to the above system embodiment are distant
Device 100 is controlled, as shown in fig. 6, this method comprises:
S100: obtaining and shows the image of smart home；When the image of smart home smaller (farther out), taken the photograph by wide-angle
As head, it can further and amplify the image of smart home；
S200: when detecting that gesture motion is identical as default gesture, then it is assumed that receive the smart home of user's input
Confirm instruction/control instruction；
S300: when the smart home for receiving input confirms instruction/control instruction, referred to according to smart home confirmation
Order/control instruction confirms the object to be controlled (smart home to be controlled chosen) in the image of the smart home of display；
S410: when the virtual controlling for not finding the smart home to be controlled in presetting database (local library) instructs dish
Dan Shi scans the identification information of the smart home to be controlled；S430: described to be controlled according to identification information networking downloading
The virtual controlling command menu of smart home is to the presetting database；
S470: the virtual controlling command menu of the smart home to be controlled is searched in the preset database, and shows institute
State virtual controlling command menu；
S500: when receiving the event control assigned according to the smart home to be controlled instruction, according to the event
Control instruction generates corresponding control command code signal.Optionally, S500 step specifically: S510: when receiving user's root
When being instructed according to the event control that the virtual controlling command menu of the smart home to be controlled of display is assigned, according to the event
Control instruction generates the corresponding control command code signal, and (input mode of event control instruction can be voice, gesture
Etc. modes, determined according to the mode that AR remote controler is supported)；
S600: the control command code signal is sent to the smart home to be controlled, makes smart home to be controlled
Corresponding operation is executed according to control command code signal.
Preferably, S200 step specifically:
S210: when detecting that first gesture movement, second gesture identical with the default first gesture in default gesture act
Time between the first gesture movement identical and described with the default second gesture in default gesture and second gesture movement
Difference is in preset time range, then it is assumed that detects that gesture motion is identical as default gesture.
It specifically describes in detail, for showing equipment, gesture interaction component using AR glasses as AR:
User is directed at certain intelligent electric appliance by (depth or wide-angle) camera on AR glasses/helmet, and user passes through hand
Gesture mode inputs smart home confirmation instruction/control instruction.Such as: refrigerator is chosen in referring to, firing finger indicates that confirmation selects this
Refrigerator, a whole operation indicate that smart home confirms instruction/control instruction, confirm refrigerator as smart home to be controlled.
By the intelligent electric appliance database (i.e. local presetting database before this) in the memory on comparison AR glasses and take the photograph
As the graph image/identification code for the smart home to be controlled that head transmits, concrete intelligence household is determined；If you need to can further obtain
To corresponding virtual controlling command menu, menu shows more detailed operational order and/or operating instruction, and user passes through again
Gesture interaction component enters the virtual controlling command menu of the smart home, is inquired, the event controls such as ON/OFF or adjusting refer to
The input of order, the short haul connection chip (i.e. instruction sender) after generating corresponding control command code signal, on AR glasses
Control command code signal is sent to smart home, also having corresponding short haul connection chip on smart home, (instruction receives
Side), the intelligent electric appliance after receiving control command code signal carries out relevant operation, is able to the control instruction of feedback user.
When AR glasses can not find corresponding virtual controlling command menu, control command code in intelligent electric appliance database
When signal, it can be referred to according to the graph image/identification code downloading control command code signal and virtual controlling of smart home to be controlled
Enable menu.
In other embodiments, it can local presetting database, straight-forward network not search for；Or only have local preset
Database, the not function of web search.
It should be noted that if Fig. 7 is the flow chart institute of the deformation of Intelligent household remote control method first embodiment of the present invention
Show, the S200 and S300 of the present embodiment can be supplemented or be replaced, S310 by S310: when the smart home confirmation for receiving voice input
When instruction/control instruction, the image of instruction/control instruction confirmation display smart home is confirmed according to the smart home
In smart home to be controlled.
The remote control of basic smart home in the present embodiment be visualization remote control intelligent electric appliance, overcome voice remote control or
The inadequate natural endowment that person's remote controler is directly remotely controlled, an AR remote controler can drive the switch of any wisdom electric appliance in family, operation with
And adjustment.Based on the remote controler of AR technology, no matter short distance or local area network and wide area network, remote control effect is more intuitive, efficient.
Specifically, one of the present embodiment deforms, user can also be inputted smart home confirmation by way of voice and be referred to
Order/control instruction, similarly, the mode that voice can also be used in (event) control instruction input.Because having had in AR glasses/helmet
The display or prompt of virtual selection menu, so the auxiliary of sound control is more convenient, such as " page turning ", " next line ", " to
Left/right ", " determination ", that is to say, that if user is stranger to the concrete syntax control command of this set system, Ke Yitong
It crosses under the display of virtual menu, then carries out simple language operation, keep whole operation process more smooth, for certain user
It is more efficient.
The image graphic data library of various smart homes is all stored in AR remote control device or in network, control instruction is compiled
Code signal and user's usage record etc., these make electrical equipment control more humane all in order to further develop and improve user experience,
More convenientization provides strong background support.
The AR remote control device of the present embodiment, the technical effect of intelligent domestic remote control system and method or beneficial effect exist
In: short-range wisdom appliance remote control, it is versatile, it brings the wisdom electric appliance of remote control system into, need to be only remotely controlled by an AR
Device achieves that remote control, and virtual menu function facilitates remote control and implements, and voice miscellaneous function enriches remote control means.
Void (virtually selecting menu) based on AR technology real (specific wisdom electric appliance) combines, and allows the use of any remote controler
Person feels succinctly and conveniently, especially the sufferer to (temporary) forfeiture language competence or muscle control ability, is Gospel mind
Device, then assist more allowing old man and child user to feel warm with language semantic identification technology.
Embodiment two
According to the abovementioned embodiments of the present invention, the improvement including device, system embodiment and embodiment of the method, except identical
Part is outer, as illustrated in figures 1 and 8, intelligent domestic remote control system, further includes: second camera module 300 is remotely controlled with the AR and fills
100 communication connections are set, for obtaining the image of the remote smart home/intelligent household appliances, and are sent to the AR remote control
Device 100.For not controlled in intelligent electric appliance object nearby, it is necessary to introduce network technology, can be used at home
Carrier of the WIFI signal as control instruction；
Second camera module 300 further comprises: IP Camera, unmanned aerial vehicle onboard camera or unmanned vehicle is vehicle-mounted takes the photograph
As head.The wide area network or IoT technology of intelligent electric appliance needs in remote control man of office.As shown in Figure 9.Net at home
Network camera need to cooperate AR glasses to be manipulated, i.e., AR glasses first pass through local area network or wide area network connection as in the room
IP Camera checks specific intelligent electric appliance by IP Camera and shows the virtual selection menu of the intelligent electric appliance, so
The operations such as laggard wardrobe take aim at, voice or gesture, control centre of the router at order forwarding in family, the wisdom respectively networked
Electric appliance receives the command signal from router.
By internet environment, identifiable wisdom electric appliance object will be expanded significantly, also can access major brand manufacturers
Cloud database finds newest remote manipulation instruction, carries out the remote operation of internet of things (LoRa or Nb-IoT), certainly each
Internet of Things communication chip is embedded in wisdom electric appliance, all to receive the command signal from Internet of Things.
Various wisdom electric appliances have oneself LOGO, the identification by AR equipment to LOGO, it is easier to brand manufacturers is locked,
The cloud database for more easily accessing brand manufacturers, determines the model or style of specific wisdom electric appliance.
A kind of Intelligent household remote control method, as Figure 10 shows, comprising: the image of acquisition smart home includes: in S100 step
S110: the image of the smart home around AR robot view field is obtained by camera；Or, S120: being turned by receiving
The image of the smart home around the second camera module visual field of hair.
Specifically, by taking unmanned aerial vehicle onboard camera or the communication connection of unmanned vehicle vehicle-mounted camera and AR remote controler as an example:
Airborne camera can move under control of the user, and the image that will acquire is sent to AR remote controler, therefore, wear AR remote controler
The remote controlled smart home of user, realize smart home of the user in space/room A remote-control space/room B.
In addition, intelligent mobile small machine people is then flexibly many, it has with it compared to the IP Camera fixed at one
The sensor of indoor positioning drives it that can enter any room in family, is then searched by its eyes (camera)
Rope observation, such as Fig. 9 can carry out AR " capture " to intelligent domestic/electric appliance, then realize remote control.
Intelligent mobile terminal (mobile phone or PAD) can also pass through short message, wechat, multimedia message, APP or cellular mobile communication side
The wireless remote control command signal of formula sending intelligent electric appliance.
Remote control mode on AR remote control device is also varied, including head takes aim at (Helmet Sighting), eye-tracking, sound
Sound, wisdom gloves, control handle, free hand gestures, and be remotely controlled with the interaction mode of virtual caretaker, all it is greatly enriched
The realization possibility and application range of AR remote control device.
The AR remote control device of the present embodiment, the technical effect of intelligent domestic remote control system and method or beneficial effect exist
In: in the present embodiment, the remote control based on internet or Internet of Things over long distances is realized by second camera module, for family
In any smart home/wisdom electric appliance in any space, visualization remote control operation can be carried out, it is accurate, accurate, specific, straight
It sees.Based on the remote controler of AR technology, no matter short distance or local area network and wide area network, remote control effect is more intuitive, efficient.
Image graphic data library, the telecommand data library of various wisdom electric appliances are all stored in AR equipment or in network
With user's usage record etc., these all in order to further develop and improve user experience, make electrical equipment control more humane, are more convenient
Change, provides strong background support.
Embodiment three
Above-mentioned two device/system embodiments according to the present invention, based on the improvement of first AR remote control device 100, except with
Outside its identical part, virtual objects generation module further include: virtual controlling command menu, virtual intermediary object, the Virtual Space 3D
Environment and/or 3D map image.
Language control component, for receiving the control instruction of user speech input；And when the behavior for detecting user meet it is pre-
If when trigger condition, issuing inquiry message；And receive the control instruction that user inputs according to inquiry message voice.
Based on the improvement of first systematic embodiment, in addition to same part, as shown in figure 11, Intelligent household remote control
Virtual objects generation module in system further include: virtual intermediary object, the Virtual Space 3D environment and/or 3D map image；And
Language control component, for receiving the control instruction of user speech input；And when the behavior for detecting user meets default triggering item
When part, inquiry message is issued；And receive the control instruction that user inputs according to inquiry message voice.
According to above method embodiment, based on the improvement of first embodiment of the method, a kind of remote control method of smart home,
It is controlled based on virtual intermediary object, as shown in figure 12, comprising:
After S100 step further include: S150: when the behavior of detecting meet preset trigger condition (such as: the gesture of user
Identical, user stares at the same place and has seen certain time, ambient light lower than timing etc. with default gesture for movement) when, it is (empty
Quasi- intermediary meeting) issue inquiry message.
S300 step includes: S330: confirming instruction/control according to the smart home of inquiry message voice input when receiving
When instruction, the smart home to be controlled in instruction/control instruction confirmation display image is confirmed according to the smart home
Or its virtual image.
It should be noted that SS410, S430 and S470 are optional step.
Specifically, for showing the AR remote controler of equipment using the AR helmet as AR:
User wears the AR helmet, and the AR helmet shows the smart home image that can be captured around it, when virtual intermediary object
(such as: house keeper smart) stares at the duration that air-conditioning is seen by eye-tracking camera detection to user and reaches preset time
(such as: 5 seconds), then it is assumed that the behavior of user meets preset trigger condition, and house keeper's spirit issues inquiry message, and " owner, you need
Open the virtual controlling command menu of air-conditioning? "；User can input smart home by different modes such as gesture, voices
Confirmation instruction, such as user answer: "Yes", i.e., smart home confirmation instruction here is that the virtual controlling of air-conditioning instructs dish
It is single, corresponding virtual controlling command menu can be searched in presetting database or on the net and be shown to user and seen, user according to
Virtual controlling command menu assigns event control instruction.
For another example: the luminosity sensor at window frame sense luminous intensity it is weak to certain value when, house keeper's spirit issues inquiry
Information " owner, it is dark, and whether I all pull up the roomed curtain of institute ", after being confirmed, house keeper's spirit is to all
All wisdom curtain in room assign the instruction for drawing curtain, and smart home confirmation herein instructs corresponding smart home to be controlled
For roomed wisdom curtain；Event control instruction is drawing curtain.
It should be noted that S100, S150, S330 in the present embodiment can be replaced by following S180 and S350.
As shown in figure 13, it S180: when receiving the phonetic control command that user assigns, obtains and shows the voice control
The system instruction relevant Virtual Space 3D environment or 3D map image；S350: when receiving user according to the Virtual Space 3D ring
When the smart home of border or the input of 3D map image confirms instruction, confirm that confirmation command shows according to the smart home described in
Smart home to be controlled in the Virtual Space 3D environment or 3D map image.
Specifically, in some cases, the phonetic control command that user assigns is not clear enough, for example user says: " house keeper's essence
Spirit please opens air-conditioning, refrigeration mode, and 26 degrees Celsius." will appear in the AR helmet house keeper's spirit and the apartment Figure 14 plane or
3 dimension scenes, house keeper's spirit can point to some rooms (such as the user family be between 3 rooms apartment) say: " owner, what you said is
The air-conditioning in this room? " after obtaining the denying of user, house keeper's spirit can point to another room and continue to inquire.It is obtaining
After the confirmation of user, smart home to be controlled is determined, house keeper's spirit is instructed according to event control before, the sky in this room
It adjusts and opens, and determine refrigeration mode and 26 degrees Celsius.
3 (as shown in figure 14) combine of dimension scene in house keeper's spirit and apartment occur, or such as Fig. 9 spatial scene visual field institute
Show.In the present embodiment on AR remote control device, generate virtual management intermediary --- house keeper is smart, well with the user of remote controler
Good interaction is carried out, clear and apparent user instruction object, specific instruction and instruction degree, this greatly simplifies technologies
It realizes, intelligence degree is stronger, and high reliablity has a wide range of application.
The AR remote control device of the present embodiment, the technical effect of intelligent domestic remote control system and method or beneficial effect exist
In: in AR equipment, including AR all-in-one machine, AR glasses, smart phone or PAD, generate virtual management intermediary --- house keeper's essence
Spirit carries out good interaction, clear and apparent user instruction object, specific instruction and instruction journey with the user of remote controler well
Degree, this greatly simplifies the realizations of technology, and intelligence degree is stronger, and high reliablity has a wide range of application.
Remote control mode in AR equipment is also varied, including head takes aim at (Helmet Sighting), eye-tracking, sound, intelligence
Intelligent gloves, control handle, free hand gestures, and be remotely controlled with the interaction mode of virtual caretaker, it is distant to be all greatly enriched AR
Control the realization possibility and application range of equipment.
It should be noted that the step identifier in patent specification in each flow chart, limits, Ke Yihe without sequence
Sequencing is adjusted under manage bar part.
It should be noted that above-described embodiment can be freely combined as needed.The above is only of the invention preferred
Embodiment, it is noted that for those skilled in the art, in the premise for not departing from the principle of the invention
Under, several improvements and modifications can also be made, these modifications and embellishments should also be considered as the scope of protection of the present invention.
Claims (29)
1. a kind of AR remote control device characterized by comprising
AR shows equipment, for capturing the image of smart home；
Instruction control equipment, for receiving control instruction, and sends to the smart home according to the control instruction and controls
Instruction encoding signal.
2. AR remote control device as described in claim 1, which is characterized in that described instruction controls equipment and includes:
Module occurs for instruction, for receiving the control instruction；
Communication module, for sending control command code signal to the smart home according to the control instruction.
3. AR remote control device as claimed in claim 2, which is characterized in that it includes: gesture interaction portion that module, which occurs, for described instruction
Part, head take aim at component, eye-tracking component, wisdom gloves, wisdom ring, control handle and/or language control component；
The gesture interaction component, for when the gesture motion for detecting that user makes is identical as default gesture, then it is assumed that connect
Receive the control instruction of input；
Institute's predicate control component, for receiving the control instruction of voice input.
4. AR remote control device as claimed in claim 3, which is characterized in that the gesture interaction component, in one's hands when detecting
When gesture movement is identical as default gesture specifically:
The gesture interaction component, for when detect first gesture movement it is identical as the default first gesture in default gesture,
Second gesture acts the first gesture movement identical and described with the default second gesture in default gesture and the second gesture is dynamic
Time difference between work is in preset time range, then it is assumed that detects that gesture motion is identical as default gesture.
5. AR remote control device as claimed in claim 2, which is characterized in that the control instruction includes that gesture instruction, voice refer to
It enables, head takes aim at instruction, eye movement instructs and/or menu instructions.
6. AR remote control device as described in claim 1, which is characterized in that further include:
Picture recognition module, for obtaining the image or identification code of the smart home in environment；
Image comparison module, for determining the kind of the smart home according to the comparison of the image or identification code of the smart home
Class, brand and model.
7. AR remote control device as claimed in claim 6, which is characterized in that further include: memory module, for pre-stored determining kind
The image or identification code of the smart home of class, brand and model call when comparing for described image comparison module；And/or
The memory module, the control command code letter for the pre-stored smart home for determining type, brand and model
Number, equipment, which is controlled, for described instruction sends the control command code signal to the specific smart home.
8. AR remote control device as claimed in claims 6 or 7, which is characterized in that further include: Web crawler, for searching for really
The image or identification code for determining the smart home of type, brand and model call when comparing for described image comparison module；
And/or
The Web crawler is compiled for searching for the control instruction of the smart home of determining type, brand and model
Code signal controls equipment for described instruction and sends the control command code signal to the specific smart home.
9. AR remote control device as claimed in claim 7, which is characterized in that further include: virtual objects generation module, for assisting
Described instruction controls the apparent control instruction of equipment.
10. AR remote control device as claimed in claim 9, which is characterized in that the virtual objects generation module generates virtual
Object includes: virtual controlling command menu, virtual intermediary object, the Virtual Space 3D environment or 3D map image.
11. a kind of intelligent domestic remote control system characterized by comprising smart home and AR remote control device, the AR remote control dress
It sets, for issuing control command code signal to the smart home.
12. intelligent domestic remote control system as claimed in claim 11, which is characterized in that further include: second camera module is used for
The image of the smart home at a distance is obtained, and is sent to the AR remote control device.
13. intelligent domestic remote control system as claimed in claim 12, which is characterized in that the second camera module includes: net
Network camera, unmanned aerial vehicle onboard camera or unmanned vehicle vehicle-mounted camera.
14. the intelligent domestic remote control system as described in any in claim 11 to 13, which is characterized in that the AR remote control dress
Set, IP Camera, unmanned plane or unmanned vehicle include instruction control equipment, for receiving the control instruction, and according to
The control instruction sends control command code signal to the smart home.
15. intelligent domestic remote control system as claimed in claim 14, which is characterized in that the control instruction includes that gesture refers to
It enables, phonetic order, head take aim at instruction, eye movement instructs and/or menu instructions.
16. intelligent domestic remote control system as claimed in claim 14, which is characterized in that described instruction controls equipment and includes:
Module occurs for instruction, for receiving the control instruction；
Communication module, for sending control command code signal to the smart home according to the control instruction.
17. intelligent domestic remote control system as claimed in claim 16, which is characterized in that it includes: hand that module, which occurs, for described instruction
Gesture interactive component, head take aim at component, eye-tracking component, wisdom gloves, wisdom ring, control handle and/or language control component,
Institute's predicate control component, for receiving the control instruction of voice input；
The gesture interaction component, for when the gesture motion for detecting that user makes is identical as default gesture, then it is assumed that connect
Receive the control instruction of input.
18. the intelligent domestic remote control system as described in any in claim 11 to 13, which is characterized in that further include:
Picture recognition module, for obtaining the image or identification code of the smart home in environment；
Image comparison module, for determining the kind of the smart home according to the comparison of the image or identification code of the smart home
Class, brand and model.
19. intelligent domestic remote control system as claimed in claim 18, which is characterized in that the intelligent domestic remote control system packet
It includes: memory module；
The memory module is supplied for the image or identification code of the pre-stored smart home for determining type, brand and model
Described image comparison module is called when comparing；And/or
The memory module, the control command code letter for the pre-stored smart home for determining type, brand and model
Number, equipment, which is controlled, for described instruction sends the control command code signal to the specific smart home.
20. intelligent domestic remote control system as claimed in claim 18, which is characterized in that the intelligent domestic remote control system packet
It includes: Web crawler；
The Web crawler, the image or identification code of the smart home for searching for determining type, brand and model supply
Described image comparison module is called when comparing；And/or
The Web crawler is compiled for searching for the control instruction of the smart home of determining type, brand and model
Code signal controls equipment for described instruction and sends the control command code signal to the specific smart home.
21. a kind of remote control method of smart home characterized by comprising
S100: obtaining and shows the image of the smart home；
S300: according to the smart home being specifically directed towards, the object to be controlled in the image of the smart home is shown；
S500: corresponding control command code signal is generated according to the control instruction to the object to be controlled；
S600: the control command code signal is sent to the object to be controlled.
22. the remote control method of smart home as claimed in claim 21, which is characterized in that also wrapped before the S300 step
It includes:
S200: when detecting that gesture motion is identical as default gesture, it is determined that be specifically directed towards in the image of the smart home
The smart home.
23. the remote control method of smart home as claimed in claim 22, which is characterized in that the S200 step specifically:
S210: when detect the movement of first gesture movement, second gesture identical with the default first gesture in default gesture in advance
If the default second gesture in gesture is identical and time difference between first gesture movement and second gesture movement exists
In preset time range, then it is assumed that detect that gesture motion is identical as default gesture.
24. the remote control method of smart home as claimed in claim 21, which is characterized in that the S300 step includes:
S310: according to the phonetic order to the smart home being specifically directed towards, show in the image of the smart home to
Control object.
25. the remote control method of smart home as claimed in claim 24, it is characterised in that:
After the S100 step further include:
S150: when the behavior of detecting meets preset trigger condition, inquiry message is issued；
The S300 step includes:
S330: according to the phonetic order to the smart home, the object to be controlled in the image is shown.
26. the remote control method of smart home as claimed in claim 21, which is characterized in that obtain intelligence in the S100 step
The image of household includes:
S110: the image of the smart home around the AR robot view field is obtained by camera；Alternatively,
S120: the image of the smart home around the second camera module visual field by receiving forwarding.
27. the remote control method of smart home as claimed in claim 21, which is characterized in that the S300 step and the S500
Between step further include:
S470: searching type, brand and the model of the object to be controlled in the preset database, and shows corresponding virtual control
Command menu processed；
The S500 step specifically:
S510: the control instruction assigned according to the virtual controlling command menu to the object to be controlled generates corresponding
The control command code signal.
28. the remote control method of smart home as claimed in claim 27, which is characterized in that S470 step described in smart home it
Before further include:
S410: when not finding the type, brand and model of the object to be controlled in the preset database, scanning it is described to
The identification information of control object；
S430: the control command code signal for downloading the object to be controlled is searched for extremely according to networking to the identification information
The presetting database.
29. the remote control method of smart home as claimed in claim 21, it is characterised in that；
The S100 step includes:
S180: according to phonetic order, the relevant Virtual Space the 3D environment of the phonetic order or 3D map image are shown；
The S300 step includes:
S350: it according to the smart home being specifically directed towards, shows in the Virtual Space 3D environment or 3D map image
Object to be controlled.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811102831.4A CN109088803B (en) | 2018-09-20 | 2018-09-20 | AR remote control device, intelligent home remote control system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811102831.4A CN109088803B (en) | 2018-09-20 | 2018-09-20 | AR remote control device, intelligent home remote control system and method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109088803A true CN109088803A (en) | 2018-12-25 |
| CN109088803B CN109088803B (en) | 2023-11-07 |
Family
ID=64842061
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811102831.4A Active CN109088803B (en) | 2018-09-20 | 2018-09-20 | AR remote control device, intelligent home remote control system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109088803B (en) |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109905299A (en) * | 2019-02-18 | 2019-06-18 | 珠海格力电器股份有限公司 | A kind of control method and electronic equipment of household appliance |
| CN110109370A (en) * | 2019-06-05 | 2019-08-09 | 重庆邮电大学 | Intelligent appliance control system based on GAN algorithm and MR platform |
| CN110321003A (en) * | 2019-05-30 | 2019-10-11 | 苏宁智能终端有限公司 | Smart home exchange method and device based on MR technology |
| CN111176126A (en) * | 2019-12-30 | 2020-05-19 | 创维集团有限公司 | Equipment control method, system and storage medium based on voice recognition |
| CN112017418A (en) * | 2020-08-27 | 2020-12-01 | 上海博泰悦臻电子设备制造有限公司 | Sunroof control method, system, medium, and apparatus for vehicle |
| CN112115855A (en) * | 2020-09-17 | 2020-12-22 | 四川长虹电器股份有限公司 | Intelligent household gesture control system and control method based on 5G |
| CN112751734A (en) * | 2019-10-29 | 2021-05-04 | 珠海市一微半导体有限公司 | Household appliance control method based on cleaning robot, cleaning robot and chip |
| US11170540B1 (en) | 2021-03-15 | 2021-11-09 | International Business Machines Corporation | Directional based commands |
| WO2022028537A1 (en) * | 2020-08-05 | 2022-02-10 | 华为技术有限公司 | Device recognition method and related apparatus |
| CN114296361A (en) * | 2021-12-28 | 2022-04-08 | 广州河东科技有限公司 | Intelligent household equipment configuration method and device, electronic equipment and storage medium |
| CN115225419A (en) * | 2022-06-17 | 2022-10-21 | 宜百科技(深圳)有限公司 | AR system capable of controlling smart home |
| CN116540613A (en) * | 2023-06-20 | 2023-08-04 | 威海海鸥智能电子科技有限责任公司 | Intelligent control equipment for electric appliance |
Citations (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN202931503U (en) * | 2012-10-31 | 2013-05-08 | 康佳集团股份有限公司 | Hand gesture recognition remote control system and television |
| WO2015090185A1 (en) * | 2013-12-20 | 2015-06-25 | 乐视致新电子科技（天津）有限公司 | Method for generating wireless gesture remote control instruction and wireless remote controller |
| CN104977904A (en) * | 2014-04-04 | 2015-10-14 | 浙江大学 | Visible and controllable intelligent household control system and control method thereof |
| CN105807624A (en) * | 2016-05-03 | 2016-07-27 | 惠州Tcl移动通信有限公司 | Method for controlling intelligent home equipment through VR equipment and VR equipment |
| CN105955042A (en) * | 2016-05-27 | 2016-09-21 | 浙江大学 | Virtuality reality type visible and controllable intelligent household control system and method |
| CN106054650A (en) * | 2016-07-18 | 2016-10-26 | 汕头大学 | Novel intelligent household system and multi-gesture control method thereof |
| CN106249611A (en) * | 2016-09-14 | 2016-12-21 | 深圳众乐智府科技有限公司 | A kind of Smart Home localization method based on virtual reality, device and system |
| CN106445156A (en) * | 2016-09-29 | 2017-02-22 | 宇龙计算机通信科技(深圳)有限公司 | Method, device and terminal for intelligent home device control based on virtual reality |
| CN106502118A (en) * | 2016-12-21 | 2017-03-15 | 惠州Tcl移动通信有限公司 | A kind of intelligent home furnishing control method and system based on AR photographic head |
| CN206075026U (en) * | 2016-07-18 | 2017-04-05 | 汕头大学 | A kind of intelligent household control terminal based on many gesture controls |
| CN106681354A (en) * | 2016-12-02 | 2017-05-17 | 广州亿航智能技术有限公司 | Flight control method and flight control device for unmanned aerial vehicles |
| CN106713082A (en) * | 2016-11-16 | 2017-05-24 | 惠州Tcl移动通信有限公司 | Virtual reality method for intelligent home management |
| CN106896732A (en) * | 2015-12-18 | 2017-06-27 | 美的集团股份有限公司 | The methods of exhibiting and device of household electrical appliance |
| CN107168076A (en) * | 2017-05-19 | 2017-09-15 | 上海青研科技有限公司 | A kind of eye control intelligent domestic system and method based on AR or VR |
| CN107272902A (en) * | 2017-06-23 | 2017-10-20 | 深圳市盛路物联通讯技术有限公司 | Smart home service end, control system and control method based on body feeling interaction |
| CN107705539A (en) * | 2016-12-21 | 2018-02-16 | 深圳中盛智兴科技有限公司 | Intelligent remote controller and intelligent main equipment, intelligent distant control method and system |
| CN108509049A (en) * | 2018-04-19 | 2018-09-07 | 北京华捷艾米科技有限公司 | The method and system of typing gesture function |
-
2018
- 2018-09-20 CN CN201811102831.4A patent/CN109088803B/en active Active
Patent Citations (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN202931503U (en) * | 2012-10-31 | 2013-05-08 | 康佳集团股份有限公司 | Hand gesture recognition remote control system and television |
| WO2015090185A1 (en) * | 2013-12-20 | 2015-06-25 | 乐视致新电子科技（天津）有限公司 | Method for generating wireless gesture remote control instruction and wireless remote controller |
| CN104977904A (en) * | 2014-04-04 | 2015-10-14 | 浙江大学 | Visible and controllable intelligent household control system and control method thereof |
| CN106896732A (en) * | 2015-12-18 | 2017-06-27 | 美的集团股份有限公司 | The methods of exhibiting and device of household electrical appliance |
| CN105807624A (en) * | 2016-05-03 | 2016-07-27 | 惠州Tcl移动通信有限公司 | Method for controlling intelligent home equipment through VR equipment and VR equipment |
| CN105955042A (en) * | 2016-05-27 | 2016-09-21 | 浙江大学 | Virtuality reality type visible and controllable intelligent household control system and method |
| CN206075026U (en) * | 2016-07-18 | 2017-04-05 | 汕头大学 | A kind of intelligent household control terminal based on many gesture controls |
| CN106054650A (en) * | 2016-07-18 | 2016-10-26 | 汕头大学 | Novel intelligent household system and multi-gesture control method thereof |
| CN106249611A (en) * | 2016-09-14 | 2016-12-21 | 深圳众乐智府科技有限公司 | A kind of Smart Home localization method based on virtual reality, device and system |
| CN106445156A (en) * | 2016-09-29 | 2017-02-22 | 宇龙计算机通信科技(深圳)有限公司 | Method, device and terminal for intelligent home device control based on virtual reality |
| CN106713082A (en) * | 2016-11-16 | 2017-05-24 | 惠州Tcl移动通信有限公司 | Virtual reality method for intelligent home management |
| CN106681354A (en) * | 2016-12-02 | 2017-05-17 | 广州亿航智能技术有限公司 | Flight control method and flight control device for unmanned aerial vehicles |
| CN106502118A (en) * | 2016-12-21 | 2017-03-15 | 惠州Tcl移动通信有限公司 | A kind of intelligent home furnishing control method and system based on AR photographic head |
| CN107705539A (en) * | 2016-12-21 | 2018-02-16 | 深圳中盛智兴科技有限公司 | Intelligent remote controller and intelligent main equipment, intelligent distant control method and system |
| CN107168076A (en) * | 2017-05-19 | 2017-09-15 | 上海青研科技有限公司 | A kind of eye control intelligent domestic system and method based on AR or VR |
| CN107272902A (en) * | 2017-06-23 | 2017-10-20 | 深圳市盛路物联通讯技术有限公司 | Smart home service end, control system and control method based on body feeling interaction |
| CN108509049A (en) * | 2018-04-19 | 2018-09-07 | 北京华捷艾米科技有限公司 | The method and system of typing gesture function |
Non-Patent Citations (2)
| Title |
|---|
| 傅大梅;倪瑛;: "基于Kinect的智能家居体感控制系统的研究与设计", 价值工程, no. 32, pages 174 - 176 * |
| 杨栋;张建强;曹鹏;徐国祥;: "智能家居中体感技术的应用和前景", 电视技术, no. 21, pages 71 - 75 * |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109905299A (en) * | 2019-02-18 | 2019-06-18 | 珠海格力电器股份有限公司 | A kind of control method and electronic equipment of household appliance |
| CN110321003A (en) * | 2019-05-30 | 2019-10-11 | 苏宁智能终端有限公司 | Smart home exchange method and device based on MR technology |
| CN110109370A (en) * | 2019-06-05 | 2019-08-09 | 重庆邮电大学 | Intelligent appliance control system based on GAN algorithm and MR platform |
| CN112751734A (en) * | 2019-10-29 | 2021-05-04 | 珠海市一微半导体有限公司 | Household appliance control method based on cleaning robot, cleaning robot and chip |
| CN111176126A (en) * | 2019-12-30 | 2020-05-19 | 创维集团有限公司 | Equipment control method, system and storage medium based on voice recognition |
| WO2022028537A1 (en) * | 2020-08-05 | 2022-02-10 | 华为技术有限公司 | Device recognition method and related apparatus |
| CN112017418A (en) * | 2020-08-27 | 2020-12-01 | 上海博泰悦臻电子设备制造有限公司 | Sunroof control method, system, medium, and apparatus for vehicle |
| CN112115855A (en) * | 2020-09-17 | 2020-12-22 | 四川长虹电器股份有限公司 | Intelligent household gesture control system and control method based on 5G |
| CN112115855B (en) * | 2020-09-17 | 2022-11-01 | 四川长虹电器股份有限公司 | Intelligent household gesture control system and control method based on 5G |
| US11170540B1 (en) | 2021-03-15 | 2021-11-09 | International Business Machines Corporation | Directional based commands |
| CN114296361A (en) * | 2021-12-28 | 2022-04-08 | 广州河东科技有限公司 | Intelligent household equipment configuration method and device, electronic equipment and storage medium |
| CN115225419A (en) * | 2022-06-17 | 2022-10-21 | 宜百科技(深圳)有限公司 | AR system capable of controlling smart home |
| CN116540613A (en) * | 2023-06-20 | 2023-08-04 | 威海海鸥智能电子科技有限责任公司 | Intelligent control equipment for electric appliance |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109088803B (en) | 2023-11-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109088803A (en) | A kind of AR remote control device, intelligent domestic remote control system and method | |
| US11314399B2 (en) | Adaptive graphic user interfacing system | |
| US10016334B2 (en) | System and method to assist users having reduced visual capability utilizing lighting device provided information | |
| EP2703836B1 (en) | TOF illuminating system and TOF camera and method for operating, with control means for driving electronic devices located in the scene | |
| CN103460256B (en) | In Augmented Reality system, virtual image is anchored to real world surface | |
| CN109324693A (en) | AR searcher, the articles search system and method based on AR searcher | |
| JP2014535154A (en) | Light control method and lighting apparatus using the light control method | |
| CN105843054A (en) | Method for controlling household devices, intelligent household system and mobile device | |
| CN208723929U (en) | A kind of AR remote controler | |
| EP3225082B1 (en) | Controlling lighting dynamics | |
| CN105527944A (en) | Intelligent control system capable of time partition | |
| US11514705B2 (en) | Information processing apparatus and non-transitory computer readable medium to allow operation without contact | |
| US11265428B2 (en) | Information processing apparatus and non-transitory computer readable medium for operating a target object in a real space through a virtual interface by detecting a motion of a user between a display surface displaying the virtual interface and the user | |
| JP2018142090A (en) | Character image generating device, character image generating method, program, recording medium and character image generating system | |
| CN106569409A (en) | Graph capturing based household equipment control system, device and method | |
| CN109901698A (en) | A kind of intelligent interactive method, wearable device and terminal and system | |
| CN109857129A (en) | A kind of method, apparatus of controlling electronic devices, mobile terminal and storage medium | |
| CN112363406A (en) | Intelligent control method and system based on visual sensor | |
| CN209514548U (en) | AR searcher, the articles search system based on AR searcher | |
| JP7271909B2 (en) | DISPLAY DEVICE AND CONTROL METHOD OF DISPLAY DEVICE | |
| CN112888118B (en) | Lighting lamp control method and device, electronic equipment and storage medium | |
| CN109711282A (en) | Light adjusting method and device | |
| KR20220009244A (en) | Control device and control method for controlling a real IoT device in a virtual environment | |
| CN115997388A (en) | Information processing terminal, remote control method, and program | |
| CN205880633U (en) | Indoor scene's control end |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |