CN116744027A - Meta universe live broadcast system - Google Patents
Meta universe live broadcast system Download PDFInfo
- Publication number
- CN116744027A CN116744027A CN202310693037.6A CN202310693037A CN116744027A CN 116744027 A CN116744027 A CN 116744027A CN 202310693037 A CN202310693037 A CN 202310693037A CN 116744027 A CN116744027 A CN 116744027A
- Authority
- CN
- China
- Prior art keywords
- data
- cloud server
- meta
- edge cloud
- universe
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000009471 action Effects 0.000 claims abstract description 44
- 230000003993 interaction Effects 0.000 claims abstract description 25
- 230000004424 eye movement Effects 0.000 claims abstract description 15
- 230000002452 interceptive effect Effects 0.000 claims abstract description 11
- 238000012545 processing Methods 0.000 claims description 16
- 238000009877 rendering Methods 0.000 claims description 8
- 238000000034 method Methods 0.000 claims description 5
- 230000008569 process Effects 0.000 claims description 2
- 238000004891 communication Methods 0.000 abstract description 18
- 230000008901 benefit Effects 0.000 abstract description 4
- 238000005516 engineering process Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 3
- 239000011521 glass Substances 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/21—Server components or server architectures
- H04N21/218—Source of audio or video content, e.g. local disk arrays
- H04N21/2187—Live feed
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs
- H04N21/23412—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs for generating or manipulating the scene composition of objects, e.g. MPEG-4 objects
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/41—Structure of client; Structure of client peripherals
- H04N21/422—Input-only peripherals, i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS]
- H04N21/42201—Input-only peripherals, i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS] biosensors, e.g. heat sensor for presence detection, EEG sensors or any limb activity sensors worn by the user
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs
- H04N21/44012—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs involving rendering scenes according to scene graphs, e.g. MPEG-4 scene graphs
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/47—End-user applications
- H04N21/478—Supplemental services, e.g. displaying phone caller identification, shopping application
- H04N21/4788—Supplemental services, e.g. displaying phone caller identification, shopping application communicating with other users, e.g. chatting
Abstract
The invention discloses a meta-universe living broadcast system, which relates to the technical field of meta-universe data interaction and comprises the following components: the system comprises head display equipment, limb action recognition equipment, an edge cloud server and a cloud server; the limb action recognition device and the head display device respectively acquire limb action data, voice and eye movement data of a user in real time, the limb action data, voice and eye movement data of the user are collected and primarily processed by the edge cloud server, and the data sent to the cloud server and other users are interactively processed and then are returned to the head display device for interactive display through the edge cloud server. The invention solves the problem of data interaction between the edge cloud server and the limb action recognition device, realizes that the virtual image is controlled to enter the virtual world to watch live broadcast by wearing the limb action recognition device and the meta-universe display device, realizes that the anchor and the audience carry out face-to-face communication in all communication forms consistent with the display world through languages, actions and the like, changes the original communication mode, and has considerable economic benefit.
Description
Technical Field
The invention relates to the technical field of meta-universe data interaction, in particular to a meta-universe live broadcast system.
Background
At present, the existing live broadcast and short video platform is mainly a second generation internet product, the main presentation mode is 2D plane pictures and videos, the main hardware carrier is a mobile phone or a tablet personal computer, the main man-machine interaction mode is based on a virtual keyboard and a capacitive touch screen on the mobile phone or the tablet personal computer, the interaction mode between audiences and a host is mainly characters, the communication mode between audiences and the audience is also mainly characters, and the host can only unilaterally face the language communication of the audiences.
Therefore, the metauniverse living broadcast system is provided, based on artificial intelligent limb action recognition and natural language recognition technology, virtual images are controlled to enter a virtual world to watch living broadcast by wearing limb action recognition equipment and metauniverse display equipment, and the fact that a host and a spectator conduct face-to-face communication in all communication forms consistent with a display world through languages, actions and the like is a problem to be solved by those skilled in the art.
Disclosure of Invention
In view of the above, the invention provides a meta-universe living broadcast system for realizing face-to-face communication between a host and a spectator in a communication form consistent with the real world through language, action and the like, and in order to realize the purposes, the invention adopts the following technical scheme:
a meta-cosmic living system, comprising: the system comprises head display equipment, limb action recognition equipment, an edge cloud server and a cloud server;
the limb action recognition device and the head display device respectively acquire limb action data, voice and eye movement data of a user in real time, the limb action data, voice and eye movement data of the user are collected and primarily processed by the edge cloud server, and the data sent to the cloud server and other users are interactively processed and then are returned to the head display device for interactive display through the edge cloud server.
Optionally, the limb motion recognition device and the head display device respectively acquire and recognize limb motion data, voice and eye movement data of the user in real time.
Optionally, the edge cloud server is used as a live broadcast software operation platform, and is used for performing data collection, preliminary processing and operation on the limb motion data, voice and eye motion data of the user, which are acquired by the head display device and the limb motion recognition device.
Optionally, the performing preliminary processing by the edge cloud server includes performing real-time operation and rendering on the user data and the local image, so as to share image operation and rendering pressure of the cloud server.
Optionally, the edge cloud server is installed in a local area network, one end of the edge cloud server is connected with the cloud server for transmitting user data, and the other end of the edge cloud server is networked in the local area network to realize multi-person interaction in a local area range.
Optionally, the cloud server is configured to aggregate and process action data voice data, man-machine interaction data, indoor space positioning data and world coordinate data of all users within the wide area network, so as to implement real-time interaction between all users.
Optionally, the head display device is provided with a display screen, and the display screen is in data connection with the server and is used for displaying the interactive picture.
Compared with the prior art, the invention discloses a meta-universe living broadcast system, which has the following beneficial effects:
the edge cloud server is used as a data preliminary processing port, and the data processing pressure of the cloud server is relieved when a large number of users access the edge cloud server simultaneously. The invention solves the problem of data interaction between the edge cloud server and the limb action recognition device, controls the virtual image to enter the virtual world to watch live broadcast by wearing the limb action recognition device and the meta-universe display device, realizes face-to-face communication between the anchor and the audience in a communication form consistent with the display world through all of languages, actions and the like, changes the original communication mode, and has considerable economic benefit.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required to be used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only embodiments of the present invention, and that other drawings can be obtained according to the provided drawings without inventive effort for a person skilled in the art.
Fig. 1 is a schematic flow chart of a meta-universe living broadcast system provided by the invention.
Fig. 2 is a schematic diagram of a meta-universe living broadcast system provided by the invention.
Fig. 3 is a schematic diagram of a unit structure of a meta-universe living broadcast system provided by the invention.
Fig. 4 is a schematic diagram of the overall structure of the meta-universe living broadcast system provided by the invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The embodiment of the invention discloses a meta-universe live broadcast system, which is shown in figure 1 and comprises the following steps: the system comprises head display equipment, limb action recognition equipment, an edge cloud server and a cloud server;
the limb action recognition device and the head display device respectively acquire limb action data, voice and eye movement data of a user in real time, the limb action data, voice and eye movement data of the user are collected and primarily processed by the edge cloud server, and the data sent to the cloud server and other users are interactively processed and then are returned to the head display device for interactive display through the edge cloud server.
Further, the limb motion recognition device and the head display device respectively acquire and recognize limb motion data, voice and eye movement data of a user in real time. The limb movement recognition device recognizes limb movement and behavior data, and the helmet recognizes voice and eye movement data.
Further, the edge cloud server is used as a live broadcast software operation platform and is used for collecting, primarily processing and operating the limb action data, the voice and the eye movement data of the user, which are acquired by the head display device and the limb action recognition device. The data communication may be accomplished using the own protocol stack of bluetooth, WIFI or independent channels.
Further, the edge cloud server performs preliminary processing including performing real-time operation and rendering on the user data and the local picture, so as to share picture operation and rendering pressure of the cloud server.
Furthermore, the edge cloud server is installed in a local area network, one end of the edge cloud server is connected with the cloud server and used for transmitting user data, and the other end of the edge cloud server is networked in the local area network to realize multi-person interaction in a local area range. The multi-person interaction here refers to that the edge cloud server collects data uploaded to the edge cloud server by one or more users who are experiencing the local wearing helmet and the limb motion recognition device, and the local or external network user can see the behavior of the local user in the virtual world.
Furthermore, the cloud server is used for summarizing and processing man-machine interaction data such as action data, voice data and the like of all users in the wide area network range, indoor space positioning data and world coordinate data, and realizing real-time interaction among all users.
Further, the head display device is provided with a display screen, and the display screen is in data connection with the server and is used for displaying the interactive picture.
In a specific embodiment, a meta-cosmic living system, as shown in fig. 2, includes: all users, including the audience of the anchor or the anchor group and the whole watching live broadcast, enter the meta universe virtual world by wearing the head display device and the limb action recognition device, the meta universe software runs on the edge cloud server and the cloud server, the lightweight interactive data processed by the data through the edge cloud server is transmitted into the cloud server from the edge cloud server, and the lightweight interactive data are used for summarizing and processing man-machine interactive data such as action data voice data and the like of all users in the wide area network range and indoor space positioning data and world coordinate data, so that real-time interaction among all users is realized. The main broadcasting or the main broadcasting group can see the interactive personnel including the main broadcasting equipment, can be all meta-universe audiences including the first person or the third person, and can carry out the immersive conversation and interaction with the meta-universe audiences.
In a specific embodiment, a meta-cosmic living system is constructed, including:
s1, constructing a data interaction channel between an edge cloud server and head display equipment and between the edge cloud server and limb action recognition equipment;
s2, developing live broadcast software to be installed on an edge cloud server and a cloud end, so that multi-user multithreading is realized, and a server end and a user end coexist;
s3, construction and maintenance of the cloud server and collection, processing and distribution of interaction data.
In a specific embodiment, a meta-universe living broadcast system is constructed, as shown in fig. 3 and fig. 4, including:
(1) The limb motion recognition device is mainly used for acquiring limb motion data of a user.
(2) The head display device is mainly used for displaying pictures, voice recognition, eye movement recognition and the like.
(3) The edge cloud server is used as a live broadcast software operation platform and is used for collecting, primarily processing and operating the limb action data, the voice and the eye movement data of the user, which are acquired by the head display equipment and the limb action recognition equipment; the edge cloud server performs preliminary processing including real-time operation and rendering on user data and local pictures, and sharing picture operation and rendering pressure of the cloud server; the edge cloud server is installed in a local area network, one end of the edge cloud server is connected with the cloud server and used for transmitting user data, and the other end of the edge cloud server is networked in the local area network to realize multi-person interaction in a local area range.
(4) The cloud server is used for summarizing and processing man-machine interaction data such as action data voice data and the like of all users in the wide area network range, indoor space positioning data and world coordinate data, and realizing real-time interaction among all users.
The edge cloud server is used as a data preliminary processing port, and the data processing pressure of the cloud server is relieved when a large number of users access the edge cloud server simultaneously. The invention solves the problem of data interaction between the edge cloud server and the limb action recognition device, controls the virtual image to enter the virtual world to watch live broadcast by wearing the limb action recognition device and the meta-universe display device, realizes face-to-face communication between the anchor and the audience in a communication form consistent with the display world through all of languages, actions and the like, changes the original communication mode, and has considerable economic benefit.
The hardware carrier of the content presentation is changed in subversion in the metauniverse era, the VR helmet and the AR/MR glasses are changed, and the man-machine interaction mode is updated and iterated correspondingly, mainly based on the artificial intelligent limb action recognition GR & MC, natural language recognition NLP, idea control and other technologies.
The invention aims to develop a set of meta-universe living broadcast system by integrating the VR/AR/MR technology with GR & MC technology, NLP, idea control and other basic technologies. The method has the advantages that a host can wear limb action recognition equipment and meta-universe display equipment (VR helmets and AR/MR glasses) to control a character model 1:1 of the host to enter a 3D meta-universe for live broadcasting, a person watching live broadcasting can wear limb action recognition equipment and meta-universe display equipment to control an avatar of the host to enter a virtual world for live broadcasting, and the host and spectators can realize face-to-face communication in the meta-universe and communicate in all communication forms consistent with the real world through language, actions and the like. The invention can also be applied to various entertainment scenes such as online teaching, online work, live broadcasting and rebroadcasting of events, virtual concert evening, and the like.
In the present specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different point from other embodiments, and identical and similar parts between the embodiments are all enough to refer to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant points refer to the description of the method section.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (7)
1. A meta-cosmic living system, comprising: the system comprises head display equipment, limb action recognition equipment, an edge cloud server and a cloud server;
the limb action recognition device and the head display device respectively acquire limb action data, voice and eye movement data of a user in real time, the limb action data, voice and eye movement data of the user are collected and primarily processed by the edge cloud server, and the data sent to the cloud server and other users are interactively processed and then are returned to the head display device for interactive display through the edge cloud server.
2. The meta space living broadcast system according to claim 1, wherein the limb motion recognition device and the head display device acquire limb motion data and voice and eye movement data of the recognized user in real time, respectively.
3. The meta-universe living broadcast system of claim 1 wherein the edge cloud server is used as a living broadcast software operation platform for collecting, primarily processing and operating the limb motion data, voice and eye motion data of the user acquired by the head display device and the limb motion recognition device.
4. A meta-cosmic living system according to claim 3, wherein the performing of preliminary processing by the edge cloud server includes performing real-time computation and rendering on user data and local pictures, sharing the picture computation and rendering pressure of the cloud server.
5. The meta-universe living broadcast system of claim 1 wherein the edge cloud server is installed in a local area network, one end of the edge cloud server is connected with the cloud server for transmitting user data, and the other end of the edge cloud server is networked in the local area network to realize multi-person interaction in a local area range.
6. The meta-universe living broadcast system of claim 1 wherein the cloud server is configured to aggregate and process action data voice data, man-machine interaction data, indoor space positioning data and world coordinate data of all users within a wide area network, so as to implement real-time interaction between all users.
7. The meta space live broadcast system of claim 1, wherein the head display device has a display screen thereon, and the display screen is in data connection with the server for displaying interactive pictures.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310693037.6A CN116744027A (en) | 2023-06-12 | 2023-06-12 | Meta universe live broadcast system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310693037.6A CN116744027A (en) | 2023-06-12 | 2023-06-12 | Meta universe live broadcast system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116744027A true CN116744027A (en) | 2023-09-12 |
Family
ID=87916310
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310693037.6A Pending CN116744027A (en) | 2023-06-12 | 2023-06-12 | Meta universe live broadcast system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116744027A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117221633A (en) * | 2023-11-09 | 2023-12-12 | 北京申信达成科技有限公司 | Virtual reality live broadcast system based on meta universe and digital twin technology |
-
2023
- 2023-06-12 CN CN202310693037.6A patent/CN116744027A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117221633A (en) * | 2023-11-09 | 2023-12-12 | 北京申信达成科技有限公司 | Virtual reality live broadcast system based on meta universe and digital twin technology |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110213601A (en) | A kind of live broadcast system and live broadcasting method based on cloud game, living broadcast interactive method | |
| CN109889855A (en) | Intelligent panoramic net cast networked shopping system and method based on mobile APP | |
| CN103269423B (en) | Can expansion type three dimensional display remote video communication method | |
| CN109976690A (en) | AR glasses remote interaction method, device and computer-readable medium | |
| CN109874021A (en) | Living broadcast interactive method, apparatus and system | |
| CN102340690A (en) | Interactive television program system and realization method | |
| US9661273B2 (en) | Video conference display method and device | |
| CN102866919A (en) | Cloud drawing-based large-sized three-dimensional scene multi-person collaborative creation method | |
| CN108322474B (en) | Virtual reality system based on shared desktop, related device and method | |
| CN116744027A (en) | Meta universe live broadcast system | |
| Zerman et al. | User behaviour analysis of volumetric video in augmented reality | |
| Viola et al. | Vr2gather: A collaborative social vr system for adaptive multi-party real-time communication | |
| Zhang et al. | Semantic sensing and communications for ultimate extended reality | |
| CN113489938B (en) | Virtual conference control method, intelligent device and terminal device | |
| WO2024027611A1 (en) | Video live streaming method and apparatus, electronic device and storage medium | |
| CN108320331B (en) | Method and equipment for generating augmented reality video information of user scene | |
| CN115494962A (en) | Virtual human real-time interaction system and method | |
| KR20130067855A (en) | Apparatus and method for providing virtual 3d contents animation where view selection is possible | |
| KR20160136160A (en) | Virtual Reality Performance System and Performance Method | |
| CN116962746A (en) | Online chorus method and device based on continuous wheat live broadcast and online chorus system | |
| KR102404130B1 (en) | Device for transmitting tele-presence image, device for receiving tele-presence image and system for providing tele-presence image | |
| KR102190388B1 (en) | Method, system and non-transitory computer-readable recording medium for providing contents including augmented reality object using multi-devices | |
| CN108989327B (en) | Virtual reality server system | |
| CN115424156A (en) | Virtual video conference method and related device | |
| Sun et al. | Video Conference System in Mixed Reality Using a Hololens |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |