EP3108289A1 - Kopfmontierte anzeige und verfahren zur steuerung davon - Google Patents
Kopfmontierte anzeige und verfahren zur steuerung davonInfo
- Publication number
- EP3108289A1 EP3108289A1 EP14883433.6A EP14883433A EP3108289A1 EP 3108289 A1 EP3108289 A1 EP 3108289A1 EP 14883433 A EP14883433 A EP 14883433A EP 3108289 A1 EP3108289 A1 EP 3108289A1
- Authority
- EP
- European Patent Office
- Prior art keywords
- augmented reality
- mode
- reality image
- gaze
- external device
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/013—Eye tracking input arrangements
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/0093—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/017—Head mounted
- G02B27/0172—Head mounted characterised by optical features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0487—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
- G06F3/1423—Digital output to display device ; Cooperation and interconnection of the display device with other functional units controlling a plurality of local displays, e.g. CRT and flat panel display
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/006—Mixed reality
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/0101—Head-up displays characterised by optical features
- G02B2027/0138—Head-up displays characterised by optical features comprising image capture systems, e.g. camera
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/0101—Head-up displays characterised by optical features
- G02B2027/014—Head-up displays characterised by optical features comprising information/image processing systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2200/00—Indexing scheme for image data processing or generation, in general
- G06T2200/28—Indexing scheme for image data processing or generation, in general involving image processing hardware
Definitions
- the present specification relates to a head mounted display (HMD), and more particularly, to an HMD for displaying an augmented reality image and a method for controlling the same.
- HMD head mounted display
- An augmented reality service enables a user to view a real image and an augmented reality image to provide real information and virtual information to the user.
- a head mounted display HMD
- HMD head mounted display
- the present specification is directed to a head mounted display (HMD) and a method for controlling the same that substantially obviate one or more problems due to limitations and disadvantages of the related art.
- HMD head mounted display
- An object of the present specification is to provide an HMD for executing a function corresponding to an augmented reality image according to alignment between an external device and an augmented reality image or alignment between a content image displayed on an external device and an augmented reality image, and a method for controlling the same.
- a head mounted display includes a camera unit configured to detect an external device, a display unit configured to display an augmented reality image, a gaze detection unit configured to detect a gaze of a user, and a processor configured to control the camera unit, the display unit and the gaze detection unit, wherein the processor is further configured to detect the external device, display the augmented reality image as a first mode, wherein the first mode is a mode in which a display position of the augmented reality image depends on a position of the external device, change the first mode to a second mode when the gaze of the user to the augmented reality image is detected in the first mode, wherein the second mode is a mode in which the display position of the augmented reality image remains fixed even when the position of the external device is changed, and execute a function corresponding to the augmented reality image when the augmented reality image and the external device are aligned in the second mode.
- the first mode is a mode in which a display position of the augmented reality image depends on a position of the external device
- a method for controlling a head mounted display includes detecting an external device, displaying an augmented reality image as a first mode, wherein the first mode is a mode in which a display position of the augmented reality image depends on a position of the external device, changing the first mode to a second mode when the gaze of the user to the augmented reality image is detected in the first mode, wherein the second mode is a mode in which the display position of the augmented reality image remains fixed even when the position of the external device is changed, and executing a function corresponding to the augmented reality image when the augmented reality image and the external device are aligned in the second mode.
- a HMD may display an augmented reality image as two modes, that is, a first mode or a second mode, depending on whether the augmented reality image depends on the position of the external device.
- the HMD may change the first mode to the second mode or change the second mode to the first mode according to the gaze of the user to the augmented reality image.
- a function corresponding to the augmented reality image may be executed. Accordingly, the HMD may execute the function corresponding to the augmented reality image using a more convenient input method, thereby providing a user-friendly augmented reality service to the user.
- FIG. 1 is a diagram showing a method for detecting an external device and a gaze of a user and displaying an augmented reality image, at a head mounted display (HMD) according to one embodiment of the present specification;
- HMD head mounted display
- FIG. 2 is a diagram showing a method for displaying an augmented reality image as a first mode or a second mode, at an HMD according to one embodiment of the present specification
- FIG. 3 is a diagram showing a method for executing a function corresponding to an augmented reality image according to alignment between the augmented reality image and an external device, at an HMD according to one embodiment of the present specification;
- FIGs. 4a and 4b are diagrams showing a method for executing a function corresponding to an augmented reality image according to alignment between the augmented reality image and an external device, at an HMD according to another embodiment of the present specification;
- FIG. 5 is a diagram showing a method for providing an execution screen of a function corresponding to an augmented reality image, at an HMD according to one embodiment of the present specification
- FIG. 6 is a diagram showing a method for displaying an augmented reality image when a function corresponding to an augmented reality image is terminated, at an HMD according to one embodiment of the present specification
- FIG. 7 is a diagram showing a method for displaying a gaze holding area, at an HMD according to one embodiment of the present specification
- FIG. 8 is a block diagram showing the configuration of an HMD according to one embodiment of the present specification.
- FIG. 9 is a flowchart illustrating a method for controlling an HMD according to one embodiment of the present specification.
- a head mounted display may mean a wearable device worn on a body of a user to display visual information.
- the HMD may be a wearable device which is worn on the head of a user to display an augmented reality image.
- the augmented reality image is a virtual image and may include graphic images, text, symbols and emoticons.
- FIG. 1 is a diagram showing a method for detecting an external device and a gaze of a user and displaying an augmented reality image, at a head mounted display (HMD) according to one embodiment of the present specification.
- HMD head mounted display
- FIG. 1 shows a method for, at an HMD 100, detecting an external device 200 using a camera unit 110 and the lower side of FIG. 1 shows a method for, at the HMD 100, displaying an augmented reality image 10 using a display unit and detecting a gaze 20 of a user using a gaze detection unit.
- the HMD 100 may detect the external device 200 using the camera unit 110.
- the HMD 100 may detect position change of the external device 200 using the camera unit 110. More specifically, the HMD 100 may detect the external device 200 and position change of the external device 200 from the image captured by the camera unit 110.
- the external device 200 is a display device for displaying at least one content image.
- the HMD 100 may display the augmented reality image 10 using the display unit. If the display unit is composed of an optical see-through display panel, the HMD 100 may display the augmented reality image 10 on the periphery of the external device 200 of the real world viewed by the user via the see-through display panel.
- the HMD 100 may display the augmented reality image 10 corresponding to the external device 200.
- the HMD 100 may display the augmented reality image 10 corresponding to a display function of the external device 200. That is, the HMD 100 may display the augmented reality image 10 corresponding to a moving image playback button of the external device 200.
- the HMD 100 may display the augmented reality image 10 corresponding to the HMD 100.
- the HMD 100 may display the augmented reality image 10 corresponding to a phone function or a message transfer function of the HMD 100.
- the above-described embodiments are only examples of the augmented reality image 10 and the HMD 100 of the present specification is not limited by the embodiment.
- the HMD 100 may detect the gaze 20 of the user using the gaze detection unit.
- the HMD 100 may detect the gaze 20 of the user into the augmented reality image 10 using the gaze detection unit.
- the HMD 100 may detect position change of the gaze 20 of the user using the gaze detection unit.
- FIG. 2 is a diagram showing a method for displaying an augmented reality image as a first mode or a second mode, at an HMD according to one embodiment of the present specification.
- the HMD may display the augmented reality image 10 as the first mode.
- the first mode may mean a mode in which the display position of the augmented reality image 10 depends on the position of the external device 200. That is, in the first mode, when the position of the external device 200 is changed, the display position of the augmented reality image 10 is changed based on the changed position of the external device 200.
- the HMD may change the position of the augmented reality image 10 displayed as the first mode from the left to the right based on the changed position of the external device 200. Accordingly, the HMD may provide a user with the augmented reality image 20 is tied with the external device 200 and moved together with the external device 200.
- the HMD may display the augmented reality image 10 as the second mode.
- the second mode may mean a mode in which the display position of the augmented reality image 10 remains fixed even when the position of the external device 200 is changed.
- the HMD may fix the display position of the augmented reality image 10.
- the HMD may provide a user with the augmented reality image 10 which is independent of the external device 200. That is, the HMD may provide a user with the augmented reality image 10 is not tied with the external device 200.
- the HMD may fix the augmented reality image displayed as the second mode in a display area of the HMD. That is, the HMD may fix the augmented reality image 10 displayed as the second mode at a specific position of the display area of the HMD (e.g., specific coordinates of the display unit). In this case, a user who wears the HMD can continuously view the augmented reality image 20 at a specific position of the display area of the HMD even when the position of the HMD is changed. In addition, the HMD may fix the augmented reality image 10 displayed as the second mode in a peripheral space of the HMD.
- the HMD may fix the augmented reality image 10 displayed as the second mode at a specific position of a peripheral space of the HMD (e.g., specific coordinates of the space).
- a user who wears the HMD may not continuously view the augmented reality image 10 when the position of the HMD is changed while viewing the augmented reality image 20 but may view the augmented reality image 10 again when the HMD is returned to the original position.
- FIG. 3 is a diagram showing a method for executing a function corresponding to an augmented reality image according to alignment between the augmented reality image and an external device, at an HMD according to one embodiment of the present specification.
- the HMD may display the augmented reality image 10 as the first mode.
- the HMD may display the augmented reality image 10 on the periphery of the upper end of the external device 200 as the first mode.
- the HMD may display the augmented reality image 10 as the first mode.
- the HMD may display the augmented reality image 10 as the first mode.
- the HMD may display the augmented reality image 10 as the first mode.
- the HMD may change the first mode to the second mode. More specifically, when a first gaze of the user into a first gaze area of the augmented reality image 10 is detected in the first mode, the HMD may change the first mode to the second mode.
- the first gaze area may be equal to the display area of the augmented reality image 10.
- the first gaze area may be greater than the display area of the augmented reality image 10.
- the first gaze area may include the display area of the augmented reality image 10.
- the HMD may acquire a first gaze holding period when the first gaze is held. If the first gaze holding period is equal to or greater than a first time period, the HMD may change the first mode to the second mode. If the first gaze holding period is less than the first time period, the HMD may retain the first mode. Accordingly, the HMD may not change the mode when the user glances at the augmented reality image 10.
- the first time period may be set by the user, or may be set upon the fabrication of the HMD.
- the HMD may acquire a second gaze holding period when the second gaze is held. If the second gaze holding period is equal to or greater than a second time period, the HMD may change the second mode to the first mode again. If the second gaze holding period is less than the second time period, the HMD may retain the second mode. Accordingly, the HMD may provide the augmented reality image 10 which depends on the external device, when the gaze of the user deviates from the augmented reality image 10.
- the second time period may be set by the user, or may be set upon the fabrication of the HMD.
- first gaze holding period and the second gaze holding period may be different.
- the first gaze holding period may be greater than the second gaze holding period.
- the HMD may detect alignment between the augmented reality image 10 and the external device 200 in the second mode. More specifically, the HMD may detect the position change of the external device 100 in the second mode and detect alignment between the augmented reality image 10 and the external device 200 according to the position change of the external device 200. At this time, the HMD may detect the position change of the external device 200 from the image captured by the camera unit.
- Alignment between the augmented reality image 10 and the external device 200 may be substantial alignment satisfying a predetermined condition. For example, when about 90% of the display area of the augmented reality image 10 overlaps the area of the external device 200, the HMD may detect alignment between the augmented reality image 10 and the external device 200.
- the HMD may execute a function corresponding to the augmented reality image 10.
- the function corresponding to the augmented reality image 10 may mean a variety of functions associated with the HMD or the external device. For example, if the augmented reality image 10 corresponds to a message transfer function of the HMD, the HMD may execute the message transfer function when the augmented reality image 10 and the external device 200 are aligned in the second mode.
- the HMD may execute a function corresponding to the augmented reality image 10. That is, the HMD may execute a function corresponding to the augmented reality image 10 if the gaze 20 of the user is further aligned.
- the HMD may execute a function corresponding to the augmented reality image 10.
- the HMD can prevent the function corresponding to the augmented reality image 10 from being executed regardless of user intention.
- the HMD may display an execution augmented reality image 30 for providing an execution screen of the function.
- the HMD may display the execution augmented reality image 30 for providing a message transfer screen.
- the display position of the execution augmented reality image 30 may depend on the display position of the augmented reality image 10. That is, when the display position of the augmented reality image 10 is changed, the HMD may change the display position of the execution augmented reality image 30 based on the changed position of the augmented reality image 10.
- FIGs. 4a and 4b are diagrams showing a method for executing a function corresponding to an augmented reality image according to alignment between the augmented reality image and an external device, at an HMD according to another embodiment of the present specification.
- FIGs. 4a and 4b show a method for, at the HMD, executing a function corresponding to the augmented reality image 10 when the augmented reality image 10 and a content image displayed on the external device 200 are aligned.
- the HMD may display the augmented reality image 10 as the first mode.
- the HMD may display the augmented reality image 10 on the periphery of the upper end of the external device 200.
- the augmented reality image 20 may correspond to a selection function.
- the external device 200 may display at least one content image.
- the external device 200 may display a first content image 210 and a second content image 220.
- the content image may include a video content image, a photo content image and an audio content image.
- the first content image 210 may be a video content image and the second content image 220 may be a photo content image.
- the content image is not limited to the above-described embodiment.
- the HMD may change the first mode to the second mode. This is equal to that described with reference to the right upper side of FIG. 3 and thus a description thereof will be omitted.
- the HMD may detect alignment between the augmented reality image 10 and any one of at least one content image displayed on the external device 200 in the second mode. For example, as shown, when a first content image 210 and a second content image 220 are displayed on the external device 200, the HMD may detect alignment between the augmented reality image 10 and any one of the first content image 210 and the second content image 220.
- alignment between the augmented reality image 10 and the content image displayed on the external device 200 may be substantial alignment satisfying a predetermined condition. For example, when about 90% of the display area of the augmented reality image 10 overlaps the display area of the content image, the HMD may detect alignment between the augmented reality image 10 and the content image.
- the HMD may execute a function corresponding to the augmented reality image 10. For example, as shown in FIG. 4a, when alignment between the augmented reality image 10 and the first content image 210 is detected, the HMD may execute a function corresponding to the augmented reality image 10. That is, as shown in FIG. 4b, when alignment between the augmented reality image 10 and the first content image 210 is detected, the HMD may execute a selection function corresponding to the augmented reality image 10.
- the HMD may display an execution augmented reality image 30 for providing an execution screen of the function.
- the HMD may display the execution augmented reality image 30 for providing the execution screen of the function, which is associated with the aligned content image.
- the HMD may display the execution augmented reality image 30 for providing an execution screen for selecting and playing video content back.
- the HMD may display the execution augmented reality image 30 for providing a message transfer screen attached with a photo content image.
- the display position of the execution augmented reality image 30 may depend on the display position of the augmented reality image 10. That is, even when the position of the external device 200 is changed, the display position of the execution augmented reality image 30 may remain fixed.
- FIG. 5 is a diagram showing a method for providing an execution screen of a function corresponding to an augmented reality image, at an HMD according to one embodiment of the present specification.
- FIG. 5 shows a method for, at the HMD, providing an execution screen of a function corresponding to the augmented reality image 10 as an execution augmented reality image 30 or as an execution image 40 displayed on the external device 200 according to settings.
- the HMD may execute a function corresponding to the augmented reality image 10. This is equal to that described in the right lower side of FIG. 3 and thus a detailed description thereof will be omitted.
- the HMD may display the execution augmented reality image 30 for providing the execution screen of the function. This is equal to that described in the left lower side of FIG. 3 and thus a detailed description thereof will be omitted.
- the HMD may control the external device 200 to display the execution image of the function on the display unit.
- the HMD may transmit control information for controlling the external device 200 to display the execution image 40 of the function to the external device 200 using a communication unit.
- the external device 200 may display the execution image 40 of the function corresponding to the augmented reality image 10 on the display unit of the external device 200 using the control information.
- the HMD may terminate display of the augmented reality image.
- FIG. 6 is a diagram showing a method for displaying an augmented reality image when a function corresponding to an augmented reality image is terminated, at an HMD according to one embodiment of the present specification.
- FIG. 6 shows a method for, at the HMD, displaying the augmented reality image 10 according function execution end when the execution augmented reality image 30 is displayed and the lower side of FIG. 6 shows a method for, at the HMD, displaying the augmented reality image 10 according to function execution end when the execution image 40 is displayed on the external device 200.
- the HMD may terminate display of the augmented reality image 10 when execution of the function corresponding to the augmented reality image 10 is terminated.
- the HMD may change the second mode to the first mode again.
- the HMD may terminate display of the execution image 40 displayed on the external device 200 when execution of the function corresponding to the augmented reality image 10 is terminated. More specifically, when execution of the function is terminated, the HMD may transmit control information for terminating display of the execution image 40 displayed on the external device 200 to the external device 200 using the communication unit.
- the HMD may display the augmented reality image 10 as the first mode again.
- FIG. 7 is a diagram showing a method for displaying a gaze holding area, at an HMD according to one embodiment of the present specification.
- the HMD may change the first mode to the second mode.
- the first gaze area may be equal to the display area of the augmented reality image 10. This is equal to that described in the right upper side of FIG. 3 and thus a detailed description thereof will be omitted.
- the HMD may display the second gaze area 50 of the augmented reality image 10 when the mode is changed to the second mode.
- the second gaze area 50 may mean a gaze holding area in which the second mode is held.
- the HMD may retain the second mode.
- the second gaze area 50 may be wider than the first gaze area.
- the second gaze area 50 may include the first gaze area.
- the HMD may change the second mode to the first mode again.
- the HMD may terminate display of the second gaze area 50 of the augmented reality image 10.
- FIG. 8 is a block diagram showing the configuration of an HMD according to one embodiment of the present specification.
- a detailed description of parts similar or equal to those of the HMD 800 shown in FIGs. 1 to 7 will be omitted.
- processor 850 Operations of the below-described processor 850 correspond to operations of the HMD 800 described above with reference to the figures. That is, in the present specification, the processor 850 is described as equating to the HMD.
- the HMD 800 may include a camera unit 810, a gaze detection unit 830, a communication unit 840 and a processor 850.
- the communication unit 840 is optional.
- the camera unit 810 may capture an image within an angle of view.
- the camera unit 810 may detect the external device. More specifically, the camera unit 810 may capture an image located in a front direction of the HMD 800 and deliver the captured image to the processor 850. At this time, the captured image may include the image of the external device.
- the processor 850 may detect the external device from the captured image.
- the display unit 820 may display the image.
- the display unit 820 may display an augmented reality image.
- the augmented reality image may be a virtual image associated with an external device or the HMD.
- the gaze detection unit 830 may detect a gaze of a user. More specifically, the gaze detection unit 830 may detect the gaze of the user and deliver the detected result to the processor 850. In addition, the gaze detection unit 830 may detect gaze change of the user. At this time, the gaze detection unit 830 may be a rear camera unit 810 provided on a rear surface of the HMD 800 to capture a gaze image. This is only an example of the gaze detection unit 830 and the HMD 800 of the present specification is not limited to the embodiment.
- the communication unit 840 may communicate with the external device.
- the communication unit 840 may pair the HMD 800 with the external device. Pairing may mean wired/wireless connection between the HMD 800 and the external device.
- the method for connecting the HMD 800 and the external device by wire or wirelessly is not limited.
- the communication unit 840 may transmit control information for displaying an execution image on the external device or for terminating display of the execution image to the external device.
- the processor 850 may control at least one unit included in the HMD 800. More specifically, the processor 850 may control the above-described units and control data transmission and/or reception between the units.
- the processor 850 may detect the external device, as described with reference to FIG. 1.
- the processor 850 may detect the external device using the camera unit 810.
- the processor 850 may detect position change of the external device using the camera unit 810. More specifically, the processor 850 may detect the external device and position change of the external device from the image captured by the camera unit 810.
- the processor 850 may display the augmented reality image as the first mode or the second mode as described with reference to FIG. 2.
- the display position of the augmented reality image may depend on the position of the external device. That is, the first mode may mean a mode in which, when the position of the external device is changed, the display position of the augmented reality image is changed based on the changed position of the external device.
- the second mode may mean a mode in which, even when the position of the external device is changed, the display position of the augmented reality image remains fixed.
- the processor 850 may fix the augmented reality image in the display area of the HMD 800.
- the processor 850 may fix the augmented reality image in a peripheral space of the HMD 800.
- the processor 850 may change the first mode to the second mode. More specifically, when a first gaze of the user into the first gaze area of the augmented reality image is detected in the first mode, the processor 850 may change the first mode to the second mode.
- the first gaze area may be equal to the display area of the augmented reality image. In addition, the first gaze area may be greater than the display area of the augmented reality image.
- the processor 850 may detect alignment between the augmented reality image and the external image in the second mode. More specifically, the processor 850 may detect position change of the external device in the second mode and detect alignment between the augmented reality image and the external device according to position change of the external device.
- the processor 850 may detect alignment between the augmented reality image and any one of at least one content image displayed on the external device in the second mode.
- the processor 850 may execute the function corresponding to the augmented reality image.
- the processor 850 may display the execution augmented reality image for providing the execution screen of the function.
- the processor 850 may control the external device to display the execution image of the function on the display unit 820.
- the processor 850 may terminate display of the execution augmented reality image or the execution image.
- FIG. 9 is a flowchart illustrating a method for controlling an HMD according to one embodiment of the present specification. In this flowchart, a detailed description of parts similar or equal to those shown in FIGs. 1 to 8 will be omitted.
- the HMD may display the augmented reality image as the first mode.
- the first mode may mean a mode in which the display position of the augmented reality image depends on the position of the external device. That is, in the first mode, when the position of the external device is changed, the display position of the augmented reality image is changed based on the changed position of the external device. Accordingly, the HMD can provide the user with the augmented reality image moved together with the external device.
- the HMD may display the augmented reality image in the first mode when the external device is detected (S910). According to another embodiment, the HMD may display the augmented reality image in the first mode when the gaze of the user into the external device is detected.
- the HMD may determine whether the gaze of the user to the augmented reality image is detected in the first mode (S920). According to one embodiment, the HMD may detect whether the first gaze of the user into the first gaze area of the augmented reality image is detected in the first mode.
- the first gaze area may be equal to the display area of the augmented reality image.
- the first gaze area may be greater than the display area of the augmented reality image.
- the first gaze area may include the display area of the augmented reality image.
- the HMD may change the first mode to the second mode (S930). More specifically, when the first gaze of the user into the first gaze area of the augmented reality image is detected in the first mode, the HMD may change the first mode to the second mode.
- the first gaze area may be equal to the display area of the augmented reality image. In addition, the first gaze area may be greater than the display area of the augmented reality image.
- the second mode may mean a mode in which, even when the position of the external device is changed, the display position of the augmented reality image remains fixed. Accordingly, the HMD can provide the user with the augmented reality image which is independent of the external device.
- the HMD may acquire the first gaze holding period when the first gaze is held. If the first gaze holding period is equal to or greater than a first time period, the HMD may change the first mode to the second mode. If the first gaze holding period is less than the first time period, the HMD may retain the first mode. Accordingly, the HMD may not change the mode when the user glances at the augmented reality image.
- the HMD may acquire the second gaze holding period when the second gaze is held. If the second gaze holding period is equal to or greater than a second period, the HMD may change the second mode to the first mode again. If the second gaze holding period is less than the second time period, the HMD may retain the second mode. Accordingly, the HMD may provide the augmented reality image which depends on the external device, when the gaze of the user deviates from the augmented reality image.
- the first gaze holding period and the second gaze holding period may be different.
- the first gaze holding period may be greater than the second gaze holding period.
- the HMD may detect alignment between the augmented reality image and the external device in the second mode (S940). More specifically, the HMD may detect the position change of the external device in the second mode and detect alignment between the augmented reality image and the external device according to the position change of the external device. At this time, the HMD may detect the position change of the external device from the image captured by the camera unit. Alignment between the augmented reality image and the external device may be substantial alignment satisfying a predetermined condition. For example, when about 90% of the display area of the augmented reality image overlaps the area of the external device, the HMD may detect alignment between the augmented reality image and the external device. This is equal to that described with reference to FIG. 3.
- the HMD may detect alignment between the augmented reality image and any one of at least one content image displayed on the external device in the second mode.
- alignment between the augmented reality image and the content image displayed on the external device (hereinafter, referred to as the content image) may be substantial alignment satisfying a predetermined condition. For example, when about 90% of the display area of the augmented reality image overlaps the display area of the content image, the HMD may detect alignment between the augmented reality image and the content image. This is equal to that described with reference to FIGs. 4a and 4b.
- the function corresponding to the augmented reality image may be executed (S950).
- the HMD may display the execution augmented reality image for providing the execution screen of the function.
- the display position of the execution augmented reality image may depend on the display position of the augmented reality image. This is equal to that described with reference to FIG. 3.
- the HMD may execute the function corresponding to the augmented reality image when the augmented reality image and the content image are aligned in the second mode. This is equal to that described with reference to FIGs. 4a and 4b.
- the HMD and the method for controlling the same of the present specification may be implemented as code that can be written to a processor-readable recording medium and can thus be read by a processor.
- the processor-readable recording medium may be any type of recording device in which data can be stored in a processor-readable manner. Examples of the processor-readable recording medium include a ROM, a RAM, a magnetic tape, a floppy disk, an optical data storage, and a carrier wave (e.g., data transmission over the Internet).
- the processor-readable recording medium can be distributed over a plurality of computer systems connected to a network so that processor-readable code is written thereto and executed therefrom in a decentralized manner.
- angle may indicate an accurate angle, distance and direction or include a substantial angle, distance and direction of a predetermined range, respectively. That is, the angle, distance and direction of the present specification may indicate a substantial angle, distance and direction with errors.
- the present invention is totally or partially applicable to electronic devices.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Optics & Photonics (AREA)
- Computer Graphics (AREA)
- Computer Hardware Design (AREA)
- Software Systems (AREA)
- User Interface Of Digital Computer (AREA)
- Controls And Circuits For Display Device (AREA)
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
KR1020140019522A KR102182161B1 (ko) | 2014-02-20 | 2014-02-20 | Hmd 및 그 제어 방법 |
US14/254,497 US9239618B2 (en) | 2014-02-20 | 2014-04-16 | Head mounted display for providing augmented reality and interacting with external device, and method for controlling the same |
PCT/KR2014/004165 WO2015126007A1 (en) | 2014-02-20 | 2014-05-09 | Head mounted display and method for controlling the same |
Publications (3)
Publication Number | Publication Date |
---|---|
EP3108289A1 true EP3108289A1 (de) | 2016-12-28 |
EP3108289A4 EP3108289A4 (de) | 2017-11-15 |
EP3108289B1 EP3108289B1 (de) | 2018-12-12 |
Family
ID=53798115
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP14883433.6A Not-in-force EP3108289B1 (de) | 2014-02-20 | 2014-05-09 | Kopfmontierte anzeige und verfahren zur steuerung davon |
Country Status (5)
Country | Link |
---|---|
US (2) | US9239618B2 (de) |
EP (1) | EP3108289B1 (de) |
KR (1) | KR102182161B1 (de) |
CN (1) | CN105917268B (de) |
WO (1) | WO2015126007A1 (de) |
Families Citing this family (39)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10254856B2 (en) | 2014-01-17 | 2019-04-09 | Osterhout Group, Inc. | External user interface for head worn computing |
US9939934B2 (en) | 2014-01-17 | 2018-04-10 | Osterhout Group, Inc. | External user interface for head worn computing |
US20170017323A1 (en) * | 2015-07-17 | 2017-01-19 | Osterhout Group, Inc. | External user interface for head worn computing |
US11003246B2 (en) | 2015-07-22 | 2021-05-11 | Mentor Acquisition One, Llc | External user interface for head worn computing |
US9865091B2 (en) * | 2015-09-02 | 2018-01-09 | Microsoft Technology Licensing, Llc | Localizing devices in augmented reality environment |
US10824253B2 (en) | 2016-05-09 | 2020-11-03 | Mentor Acquisition One, Llc | User interface systems for head-worn computers |
US10684478B2 (en) | 2016-05-09 | 2020-06-16 | Mentor Acquisition One, Llc | User interface systems for head-worn computers |
US10466491B2 (en) | 2016-06-01 | 2019-11-05 | Mentor Acquisition One, Llc | Modular systems for head-worn computers |
EP3242228A1 (de) * | 2016-05-02 | 2017-11-08 | Artag SARL | Verwaltung der anzeige von vermögenswerten in einem erweiterten realitätsmodus |
US10198874B2 (en) * | 2016-05-13 | 2019-02-05 | Google Llc | Methods and apparatus to align components in virtual reality environments |
WO2018026893A1 (en) * | 2016-08-03 | 2018-02-08 | Google Llc | Methods and systems for determining positional data for three-dimensional interactions inside virtual reality environments |
US11172005B2 (en) * | 2016-09-09 | 2021-11-09 | Nokia Technologies Oy | Method and apparatus for controlled observation point and orientation selection audiovisual content |
US20180095635A1 (en) * | 2016-10-04 | 2018-04-05 | Facebook, Inc. | Controls and Interfaces for User Interactions in Virtual Spaces |
US11094017B1 (en) | 2016-12-01 | 2021-08-17 | Wells Fargo Bank, N.A. | Visualizing investment changes using augmented reality |
KR102656425B1 (ko) | 2016-12-07 | 2024-04-12 | 삼성전자주식회사 | 영상을 표시하는 전자 장치 및 방법 |
US10437343B2 (en) | 2017-01-06 | 2019-10-08 | Samsung Electronics Co., Ltd. | Augmented reality control of internet of things devices |
KR101874111B1 (ko) * | 2017-03-03 | 2018-07-03 | 클릭트 주식회사 | 가상현실영상 재생방법 및 이를 이용한 프로그램 |
CN106970467B (zh) * | 2017-03-28 | 2020-03-24 | 联想(北京)有限公司 | 信息显示方法及头戴式显示设备 |
US11494986B2 (en) * | 2017-04-20 | 2022-11-08 | Samsung Electronics Co., Ltd. | System and method for two dimensional application usage in three dimensional virtual reality environment |
KR101965785B1 (ko) | 2017-05-15 | 2019-04-04 | 경희대학교 산학협력단 | 증강현실 헤드마운트 디스플레이 시스템 |
CN109101120B (zh) * | 2017-06-21 | 2021-09-28 | 腾讯科技(深圳)有限公司 | 图像显示的方法和装置 |
US10216333B2 (en) * | 2017-06-30 | 2019-02-26 | Microsoft Technology Licensing, Llc | Phase error compensation in single correlator systems |
KR101941880B1 (ko) | 2017-07-18 | 2019-01-24 | 경희대학교 산학협력단 | 자유 초점 디스플레이 장치 |
CN110869880A (zh) * | 2017-08-24 | 2020-03-06 | 麦克赛尔株式会社 | 头戴式显示装置 |
US10338766B2 (en) * | 2017-09-06 | 2019-07-02 | Realwear, Incorporated | Audible and visual operational modes for a head-mounted display device |
KR102340665B1 (ko) * | 2017-09-29 | 2021-12-16 | 애플 인크. | 프라이버시 스크린 |
US10732826B2 (en) * | 2017-11-22 | 2020-08-04 | Microsoft Technology Licensing, Llc | Dynamic device interaction adaptation based on user engagement |
CN108038916B (zh) * | 2017-12-27 | 2022-12-02 | 上海徕尼智能科技有限公司 | 一种增强现实的显示方法 |
EP3676656A4 (de) | 2018-01-18 | 2020-11-25 | Samsung Electronics Co., Ltd. | Verfahren und vorrichtung zur einstellung von inhalt mit erweiterter realität |
CN108415386A (zh) * | 2018-03-12 | 2018-08-17 | 范业鹏 | 用于智能车间的增强现实系统及其工作方法 |
KR102638956B1 (ko) | 2018-11-26 | 2024-02-22 | 삼성전자주식회사 | 전자 장치 및 증강 현실 서비스를 제공하는 증강 현실 장치와 그 동작 방법 |
KR20200098034A (ko) * | 2019-02-11 | 2020-08-20 | 삼성전자주식회사 | 가상 현실 유저 인터페이스를 제공하기 위한 전자 장치 및 그의 동작 방법 |
JP7238456B2 (ja) * | 2019-02-21 | 2023-03-14 | セイコーエプソン株式会社 | 表示システム、情報処理装置の制御プログラム、及び情報処理装置の制御方法 |
KR20210133674A (ko) * | 2020-04-29 | 2021-11-08 | 삼성전자주식회사 | 증강 현실 장치 및 그 제어 방법 |
CN114527864B (zh) * | 2020-11-19 | 2024-03-15 | 京东方科技集团股份有限公司 | 增强现实文字显示系统、方法、设备及介质 |
KR102513434B1 (ko) | 2021-03-18 | 2023-03-22 | 충북대학교 산학협력단 | 홀로그램 광학 소자 마이크로 미러 어레이를 이용한 3차원 홀로그래픽 도파관 타입 증강현실 디스플레이 시스템 |
WO2023092375A1 (en) * | 2021-11-25 | 2023-06-01 | Citrix Systems, Inc. | Computing device with live background and related method |
KR102642625B1 (ko) | 2021-12-30 | 2024-02-29 | 충북대학교 산학협력단 | 홀로그램 프린터에서의 홀로그래픽 광학 소자 렌즈 어레이 제작 방법 |
US11861801B2 (en) * | 2021-12-30 | 2024-01-02 | Snap Inc. | Enhanced reading with AR glasses |
Family Cites Families (19)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP3407383B2 (ja) | 1993-06-15 | 2003-05-19 | オリンパス光学工業株式会社 | 頭部装着型ディスプレイシステム |
GB2377147A (en) * | 2001-06-27 | 2002-12-31 | Nokia Corp | A virtual reality user interface |
JP4203279B2 (ja) | 2002-07-26 | 2008-12-24 | 独立行政法人科学技術振興機構 | 注目判定装置 |
JP5235122B2 (ja) | 2008-10-17 | 2013-07-10 | みずほ情報総研株式会社 | 解析支援システム、解析支援方法及び解析支援プログラム |
JP5236423B2 (ja) | 2008-10-17 | 2013-07-17 | Kddi株式会社 | 表示システム、表示方法およびプログラム |
US8860729B2 (en) | 2009-07-22 | 2014-10-14 | Imagemovers Digital Llc | Gaze intent estimation for retargeting of characters |
JP5494153B2 (ja) * | 2010-04-08 | 2014-05-14 | ソニー株式会社 | 頭部装着型ディスプレイにおける画像表示方法 |
KR101252169B1 (ko) * | 2011-05-27 | 2013-04-05 | 엘지전자 주식회사 | 휴대 단말기 및 그 동작 제어방법 |
KR101315303B1 (ko) * | 2011-07-11 | 2013-10-14 | 한국과학기술연구원 | 착용형 디스플레이 장치 및 컨텐츠 디스플레이 방법 |
US20130083003A1 (en) * | 2011-09-30 | 2013-04-04 | Kathryn Stone Perez | Personal audio/visual system |
US9286711B2 (en) * | 2011-09-30 | 2016-03-15 | Microsoft Technology Licensing, Llc | Representing a location at a previous time period using an augmented reality display |
US8611015B2 (en) | 2011-11-22 | 2013-12-17 | Google Inc. | User interface |
US20130147686A1 (en) * | 2011-12-12 | 2013-06-13 | John Clavin | Connecting Head Mounted Displays To External Displays And Other Communication Networks |
US9024844B2 (en) | 2012-01-25 | 2015-05-05 | Microsoft Technology Licensing, Llc | Recognition of image on external display |
US9165381B2 (en) | 2012-05-31 | 2015-10-20 | Microsoft Technology Licensing, Llc | Augmented books in a mixed reality environment |
US9202313B2 (en) * | 2013-01-21 | 2015-12-01 | Microsoft Technology Licensing, Llc | Virtual interaction with image projection |
US20140282220A1 (en) * | 2013-03-14 | 2014-09-18 | Tim Wantland | Presenting object models in augmented reality images |
JP2014191718A (ja) * | 2013-03-28 | 2014-10-06 | Sony Corp | 表示制御装置、表示制御方法および記録媒体 |
US20150123966A1 (en) * | 2013-10-03 | 2015-05-07 | Compedia - Software And Hardware Development Limited | Interactive augmented virtual reality and perceptual computing platform |
-
2014
- 2014-02-20 KR KR1020140019522A patent/KR102182161B1/ko active IP Right Grant
- 2014-04-16 US US14/254,497 patent/US9239618B2/en not_active Expired - Fee Related
- 2014-05-09 US US15/114,022 patent/US9990036B2/en not_active Expired - Fee Related
- 2014-05-09 EP EP14883433.6A patent/EP3108289B1/de not_active Not-in-force
- 2014-05-09 WO PCT/KR2014/004165 patent/WO2015126007A1/en active Application Filing
- 2014-05-09 CN CN201480073675.7A patent/CN105917268B/zh not_active Expired - Fee Related
Also Published As
Publication number | Publication date |
---|---|
EP3108289A4 (de) | 2017-11-15 |
US20150234456A1 (en) | 2015-08-20 |
EP3108289B1 (de) | 2018-12-12 |
US20160342204A1 (en) | 2016-11-24 |
WO2015126007A1 (en) | 2015-08-27 |
CN105917268B (zh) | 2019-03-12 |
US9239618B2 (en) | 2016-01-19 |
CN105917268A (zh) | 2016-08-31 |
KR20150098361A (ko) | 2015-08-28 |
KR102182161B1 (ko) | 2020-11-24 |
US9990036B2 (en) | 2018-06-05 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2015126007A1 (en) | Head mounted display and method for controlling the same | |
WO2015108232A1 (en) | Portable device and method for controlling the same | |
WO2015126006A1 (en) | Head mounted display and method for controlling the same | |
WO2015122565A1 (en) | Display system for displaying augmented reality image and control method for the same | |
WO2015064846A1 (en) | Apparatus and method for head mounted display indicating process of 3d printing | |
WO2015030307A1 (en) | Head mounted display device and method for controlling the same | |
WO2015030304A1 (en) | Portable device supporting videotelephony of a head mounted display and method of controlling therefor | |
WO2015046677A1 (en) | Head-mounted display and method of controlling the same | |
WO2015030301A1 (en) | Smartwatch and control method thereof | |
WO2015046676A1 (en) | Head-mounted display and method of controlling the same | |
WO2014025108A1 (en) | Head mounted display for adjusting audio output and video output in relation to each other and method for controlling the same | |
WO2015030303A1 (en) | Portable device displaying augmented reality image and method of controlling therefor | |
WO2014148696A1 (en) | Display device detecting gaze location and method for controlling thereof | |
WO2010150976A2 (en) | Receiving system and method of providing 3d image | |
WO2015046837A1 (ko) | 컨텐츠 공유 장치 및 방법 | |
EP2872945A1 (de) | Kopfmontierte anzeige und verfahren zur ausgabe von inhalten damit | |
WO2015105234A1 (en) | Head mounted display and method for controlling the same | |
WO2015122566A1 (en) | Head mounted display device for displaying augmented reality image capture guide and control method for the same | |
EP2939062A1 (de) | Kopfmontierte anzeige und verfahren zur videokommunikation damit | |
WO2016017855A1 (en) | Wearable device and method of controlling therefor | |
WO2015122564A1 (en) | Display apparatus and control method thereof | |
WO2014109434A1 (en) | Device for transmitting and receiving data using earphone and method for controlling the same | |
WO2015147374A1 (en) | Display device providing history information and method for controlling the same | |
WO2015088101A1 (en) | Display device and method of controlling the same | |
WO2015046683A1 (en) | Digital device and control method thereof |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE |
|
17P | Request for examination filed |
Effective date: 20160830 |
|
AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
AX | Request for extension of the european patent |
Extension state: BA ME |
|
DAX | Request for extension of the european patent (deleted) | ||
A4 | Supplementary search report drawn up and despatched |
Effective date: 20171017 |
|
RIC1 | Information provided on ipc code assigned before grant |
Ipc: G06F 3/14 20060101ALI20171011BHEP Ipc: H04N 5/64 20060101ALI20171011BHEP Ipc: G02B 27/01 20060101ALI20171011BHEP Ipc: G06F 3/01 20060101ALI20171011BHEP Ipc: G02B 27/00 20060101ALI20171011BHEP Ipc: G02B 27/02 20060101AFI20171011BHEP Ipc: G06F 3/0487 20130101ALI20171011BHEP |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R079 Ref document number: 602014038023 Country of ref document: DE Free format text: PREVIOUS MAIN CLASS: G02B0027020000 Ipc: G06F0003010000 |
|
GRAP | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOSNIGR1 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: GRANT OF PATENT IS INTENDED |
|
RIC1 | Information provided on ipc code assigned before grant |
Ipc: G06F 3/0487 20130101ALI20180618BHEP Ipc: G06F 3/14 20060101ALI20180618BHEP Ipc: G02B 27/01 20060101ALI20180618BHEP Ipc: G06F 3/01 20060101AFI20180618BHEP Ipc: G02B 27/00 20060101ALI20180618BHEP Ipc: G02B 27/02 20060101ALI20180618BHEP |
|
INTG | Intention to grant announced |
Effective date: 20180710 |
|
GRAS | Grant fee paid |
Free format text: ORIGINAL CODE: EPIDOSNIGR3 |
|
GRAA | (expected) grant |
Free format text: ORIGINAL CODE: 0009210 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE PATENT HAS BEEN GRANTED |
|
AK | Designated contracting states |
Kind code of ref document: B1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
REG | Reference to a national code |
Ref country code: GB Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: EP |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: REF Ref document number: 1076856 Country of ref document: AT Kind code of ref document: T Effective date: 20181215 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R096 Ref document number: 602014038023 Country of ref document: DE |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: NL Ref legal event code: MP Effective date: 20181212 |
|
REG | Reference to a national code |
Ref country code: LT Ref legal event code: MG4D |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181212 Ref country code: NO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190312 Ref country code: FI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181212 Ref country code: ES Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181212 Ref country code: LV Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181212 Ref country code: BG Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190312 Ref country code: HR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181212 |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: MK05 Ref document number: 1076856 Country of ref document: AT Kind code of ref document: T Effective date: 20181212 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181212 Ref country code: AL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181212 Ref country code: GR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190313 Ref country code: RS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181212 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: NL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181212 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: PT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190412 Ref country code: CZ Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181212 Ref country code: PL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181212 Ref country code: IT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181212 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SM Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181212 Ref country code: IS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190412 Ref country code: RO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181212 Ref country code: EE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181212 Ref country code: SK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181212 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R097 Ref document number: 602014038023 Country of ref document: DE |
|
PLBE | No opposition filed within time limit |
Free format text: ORIGINAL CODE: 0009261 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: DK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181212 Ref country code: AT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181212 Ref country code: SI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181212 |
|
26N | No opposition filed |
Effective date: 20190913 |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: PL |
|
GBPC | Gb: european patent ceased through non-payment of renewal fee |
Effective date: 20190509 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LI Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20190531 Ref country code: CH Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20190531 Ref country code: MC Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181212 |
|
REG | Reference to a national code |
Ref country code: BE Ref legal event code: MM Effective date: 20190531 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LU Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20190509 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: TR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181212 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: GB Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20190509 Ref country code: IE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20190509 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: BE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20190531 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: FR Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20190531 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: CY Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181212 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: HU Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO Effective date: 20140509 Ref country code: MT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181212 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: DE Payment date: 20210406 Year of fee payment: 8 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181212 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R119 Ref document number: 602014038023 Country of ref document: DE |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: DE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20221201 |