CN109911140A - A kind of water-area navigation information enhancement device, system and method - Google Patents
A kind of water-area navigation information enhancement device, system and method Download PDFInfo
- Publication number
- CN109911140A CN109911140A CN201910278629.5A CN201910278629A CN109911140A CN 109911140 A CN109911140 A CN 109911140A CN 201910278629 A CN201910278629 A CN 201910278629A CN 109911140 A CN109911140 A CN 109911140A
- Authority
- CN
- China
- Prior art keywords
- information
- water
- navigation
- area
- virtual
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Navigation (AREA)
- Traffic Control Systems (AREA)
Abstract
A kind of water-area navigation information enhancement device, includes at least: water environment detector and augmented reality display, the water environment detector, for obtaining the information in kind of water environment；The augmented reality display, the information in kind of the water environment for obtaining according at least to the water environment detector generate visual navigation virtual information, and Overlapping display is in the visual field of the augmented reality display.Water-area navigation information enhancement device in the application has merged augmented reality display technology, AI machine recognition technology, edge calculations technology, cloud computing technology and technology of Internet of things etc., obtain numerous ship-handling information, so can on the normal visual field of driver display information in real time, facilitate driver to operate, guarantees driving safety.
Description
Technical field
The invention belongs to shipping logistics, intelligent equipment field, and in particular to a kind of water-area navigation information enhancement device is
System, while the application further relates to a kind of water-area navigation information enhancement method.
Background technique
Waterway transportation be using ship include as transportation base, with waters as main conveyance, using harbour or port station ocean,
River and lake are a kind of means of transportation of movement operation range, and water transport is still that world many countries and area are most important so far
One of means of transportation.Waterway transportation is to rise earliest, the longest means of transportation of history in current each prevailing traffic mode.Its technology
Economic characteristics are that loading capacity is big, at low cost, with low investment, but flexibility is small, and continuity is also poor, are well suited to undertake large, low value, stupid
The middle and long distance of weight and various bulk goods transports.
Waterway transportation includes coastal transport, coastal waters transport, transport by sea, inland water transport etc., wherein is needed in inland water transport
There is auxiliary tool of the inland river electronic channel chart as navigation when.
Inland river electronic channel chart refers to using technologies such as computer, geography information, and each element information of cruiseway is carried out
Technical treatment, the standardization being made, digitized thematic maps.Inland river electronic channel chart being capable of real-time display navigation mark, water
The navigation channels element informations such as depth, periphery ship, the underwater obstruction information such as shipwreck, reef, bridge, harbour etc. face the building letter that crosses the river
Breath, the public service informations such as notice to navigator and maintenance dimension of channel, and can be automatically updated most by wirelessly or non-wirelessly network communication
New data, for users to use.
Ship automatic identification system (Automatic Identification System, abbreviation AIS system) is by bank base
(base station) facility and ship-borne equipment collectively constitute, be a kind of novel collection network technology, modern communications technology, computer technology,
The digital navaid and equipment that electronic information display technology is integrated.Ship automatic identification system (AIS) cooperates global location
System (GPS) is by the Ship dynamic situations combination such as ship physical location, ship's speed, rate of changing course and course name of vessel, catchword, drinking water, ship
The ships such as oceangoing ship scale and dangerous cargo static data is broadcasted from very high frequency(VHF) (VHF) channel to Its Adjacent Waters ship and base station, can also
To be opened to the public by maritime sector by information service platform.
Inland waterway navigation aids are used to guide ship's navigation, positioning and mark obstruction and the artificial target for indicating warning, are various
The facility or system of water activity offer security information.Set on navigable waters or its nearby, with indicate navigation channel, anchorage, beach danger and
The position of other obstructions indicates the depth of water, landscape, commands the traffic of narrow waters.
The more advanced method of cruiseway marine navigation at present are as follows: inland river electronic channel chart system by with GPS system,
The multisystems such as AIS system, radar system blend, and for ship, user provides navigation feature.But cruiseway marine navigation by
The influence factors such as navigation channel size, water level, underwater obstacle, reef, bridge, periphery ship, complexity is higher, weather weather shadow
Sound is larger, also has higher requirements to deck officer.
All in all, cruiseway marine navigation presence at present has the disadvantage that: 1. traditional cruiseway marine navigation systems
System is only able to display ship in the path of interior korneforos, and the region that ship should travel can not intuitively be expressed.2. at night or
Under severe weather conditions, deck officer only by conventional navigation, for identification navigation area, barrier, bridge, navigation mark,
Other ships etc. are more difficult.3. traditional cruiseway navigation needs deck officer to drive ship on one side, observation navigation on one side
Screen navigates not intuitive enough, and has higher requirements to the driving experience of deck officer.
Based on problems some present on, the application is further studied water-area navigation with ancillary technique.
Summary of the invention
For the above deficiency in the prior art, the present invention provides a kind of water-area navigation information enhancement apparatus and system,
Good water-area navigation miscellaneous function can be played, while additionally providing a kind of water-area navigation information enhancement method, it is highly-safe.
In order to solve the above-mentioned technical problem, the present invention is addressed by following technical proposals.
A kind of water-area navigation information enhancement device, includes at least: water environment detector and augmented reality display, described
Water environment detector, for obtaining the information in kind of water environment；The augmented reality display, for according at least to described
The information in kind for the water environment that water environment detector obtains generates visual navigation virtual information, and Overlapping display is in institute
In the visual field for stating augmented reality display.
Augmented reality (Augmented Reality, abbreviation AR) is a kind of position for calculating camera image in real time
Set and angle and plus respective image, video, 3D model technology, the target of this technology is on the screen virtual world set
In real world and interacted.Augmented reality display in the application can be head-mounted display, or thoroughly
Bright/semi-transparent display, or projection display equipment, for image to be superimposed upon in the visual field of reality displays, or
It is superimposed upon on the glass of drive window, forms augmented reality effect, can so believe reality mark on the normal visual field of driver
The virtual images such as breath, navigation channel line, facilitate driver to operate, and guarantee driving safety.
Preferably, the water environment detector includes: wide-angle camera, zoom camera, RGB camera, depth
Camera, laser detector, millimetre-wave radar, sonar contact device, water speed survey meter, thermometer, hygrometer, anemoscope, wind speed
One or more of instrument, the RGB camera, for obtaining the image information in kind of the water environment；The depth
Camera, for obtaining the real space dimension information of the water environment；The laser detector, for obtaining the waters
The range information in kind of environment；The millimetre-wave radar, for obtaining the distance in kind of the water environment in rain and fog weather
Information；The sonar contact device, for obtaining the underwater environment information of the water environment；The water speed survey meter, for obtaining
Take the water velocity information of the water environment；The thermometer, for obtaining the temperature information of the water environment；It is described wet
Degree meter, for obtaining the humidity information of the water environment；The anemoscope, the wind direction for obtaining the water environment are believed
Breath；The anemobiagraph, for obtaining the wind speed information of the water environment.
Above water environment detector can obtain the corresponding information in external environment, be shown in the visual field of driver
On, facilitate driver to check, it is highly-safe.
Preferably, further including sailing device on water (such as this ship) detector, the sailing device on water detector, for obtaining
The information in kind for obtaining sailing device on water itself, is shown on the visual field of driver, driver is facilitated to check, highly-safe.
Preferably, the sailing device on water detector includes: one of magnetometer, gyroscope, navigator, draft gauge
Or a variety of, the magnetometer, for obtaining the direction letter of the sailing device on water where the water-area navigation information enhancement device
Breath；The gyroscope, for obtaining the directional information of the sailing device on water where the water-area navigation information enhancement device；It is described
Navigator, for obtaining the latitude and longitude information of the sailing device on water where the water-area navigation information enhancement device；The drinking water
Instrument, for obtaining the depth under water information of the sailing device on water where the water-area navigation information enhancement device.
Preferably, further include interactive component, the interactive component, for the water-area navigation information enhancement device into
Row interaction.
Preferably, the interactive component includes: that gesture identification component, wisdom gloves, head take aim at component, eye-tracking portion
One or more of part, voice control component, control handle；The gesture identification component or wisdom gloves, for detecting
Gesture motion receives the control instruction of input；The head takes aim at component or eye-tracking component, for simulating the control of mouse
System instruction；The voice control component, for receiving the control instruction of voice input；The control handle, for receiving
The control instruction of input.By above interactive component, driver can be operated the virtual signage being shown in the visual field,
Further information is obtained, information can also be inputted.
Preferably, further include AI picture recognition module, the AI picture recognition module, for according to the acquisition waters
The information in kind of environment identifies the water surface, waterfront, contact ship, bridge, danger zone, shallow water area etc..
Preferably, further including virtual object generation module, the letter such as virtual navigation channel line, warning line, danger zone can be generated
Breath.The virtual object generation module, for being shown according to the recognition result of the AI picture recognition module in the augmented reality
One of Overlapping display instruction information, guidance information, prompting message, environmental information and information in kind or more in the visual field of device
Kind, the instruction information includes at least river width information, bridge pier width information, the sailing device on water width information and warp
One or more of Position Latitude information；The guidance information includes at least contact navigation channel cut-off rule information, described waterborne
One of the location information that other sailing device on water navigate by water in aircraft flight direction and velocity information, cartographic information and map
Or it is a variety of；The prompting message, including at least in shipwreck area information, shallow water area information, the high information of bridge, anticollision information
One or more；The environmental information, including at least Weather information, wind speed/wind direction information, temperature ,/humidity information rises/is fallen
One or more of tidewater breath, water (flow) direction/velocity information；The material object information, includes at least ship query information, port
One or more of mouth query information, route query information.
Preferably, further include loudspeaker and/or attention device, the loudspeaker, for instruction information described in voice broadcast,
One or more of guidance information, prompting message, environmental information and information in kind；The attention device is used for alarm song/light
Mode prompts important attention information.
A kind of water-area navigation information enhancement system involved in the application, comprising: water-area navigation information enhancement device also wraps
Include sailing device on water information management subsystem, electronic channel chart subsystem, ship automatic identification subsystem, global location subsystem
(Position Fixing Navigation Systems such as GPS, Beidou, GLONASS), intelligent waters material object recognition subsystem and weather forecast reality man who brings news of appointment's system
One or more of；The water-area navigation information enhancement device, for from the sailing device on water information management subsystem,
Electronic channel chart subsystem, ship automatic identification subsystem, global location subsystem, intelligent waters material object recognition subsystem, day
Gas forecasts one of real man who brings news of appointment's system or a variety of middle information in kind for obtaining water environment, generates the virtual letter of visual navigation
Breath, Overlapping display is in the visual field of the water-area navigation information enhancement device.
Further a kind of water-area navigation information enhancement system, wherein water-area navigation information enhancement device includes communication mould
Block, the communication module, be used to load from local (edge calculations node) or the network equipment (including cloud calculate node
And/or edge calculations node) load the electronic channel chart subsystem, ship automatic identification subsystem, global location subsystem
The information in kind that water environment is obtained in system, intelligent waters material object recognition subsystem and/or weather forecast reality man who brings news of appointment's system, generates
Visual navigation virtual information.
Further a kind of water-area navigation information enhancement system, wherein water-area navigation information enhancement device includes virtual object
Generation module, the virtual object generation module, for it is empty to generate visual navigation according to the information in kind for obtaining water environment
Quasi- information, the Overlapping display in the visual field of the augmented reality display.
A kind of further water-area navigation information enhancement system, wherein the navigation virtual information include: indicate information,
One or more of guidance information, prompting message, environmental information and information in kind, the instruction information includes at least river
One of road width information, bridge pier width information, the sailing device on water width information and longitude and latitude positional information are more
Kind；The guidance information, include at least contact navigation channel cut-off rule information, the sailing device on water navigation direction and velocity information,
One or more of the location information that other sailing device on water navigate by water in cartographic information and map；The prompting message, until
It less include one or more of shipwreck area information, shallow water area information, the high information of bridge, anticollision information；The environment
Information, including at least Weather information, wind speed/wind direction information, temperature/humidity information, rise/ebb tide information, water (flow) direction/velocity information
One or more of；The material object information, includes at least ship query information, harbour query information, route query information
One or more of.
A kind of water-area navigation information enhancement method involved in the application, according to the information in kind of water environment, in waters
Overlapping display visually navigates by water virtual information in the visual field of the augmented reality display of sail information enhancement device, including virtual
Navigation channel line, virtual warning line, virtual danger zone mark etc., judge convenient for driver, improve the safety of driving.
Preferably, further comprising the steps of: from the water environment detector of water-area navigation information enhancement device, obtaining institute
The information in kind of water environment is stated, visual navigation virtual information is generated.
Preferably, further comprising the steps of: the electronic channel chart loaded from the locally loaded or network equipment
Subsystem, ship automatic identification subsystem, global location subsystem, intelligent waters material object recognition subsystem and/or weather forecast
The information in kind that water environment is obtained in real man who brings news of appointment's system generates visual navigation virtual information.
Preferably, further comprising the steps of: by interactive component, to water-area navigation information enhancement device and/or network
Equipment issues the control instruction for obtaining the information in kind of the water environment.
Preferably, further comprising the steps of: passing through water-area navigation information enhancement device and/or the AI image of the network equipment
Identification module obtains the information in kind of water environment, identifies the water surface, waterfront, contact ship, bridge, danger zone, phytal zone
Domain；According to the recognition result of the AI picture recognition module, pass through the void of water-area navigation information enhancement device and/or the network equipment
Quasi- object generation module generates the visual navigation virtual information.
Preferably, the navigation virtual information includes: instruction information, guidance information, prompting message, environmental information and reality
One or more of object information, the instruction information include at least river width information, bridge pier width information, the water
One or more of upper aircraft width information and longitude and latitude positional information；The guidance information includes at least contact boat
Other sailing device on water in road cut-off rule information, sailing device on water navigation direction and velocity information, cartographic information and map
One or more of location information of navigation；The prompting message includes at least shipwreck area information, shallow water area is believed
One or more of breath, the high information of bridge, anticollision information；The environmental information includes at least Weather information, wind speed/wind
To information, temperature/humidity information, rise/ebb tide information, one or more of water (flow) direction/velocity information；The letter in kind
Breath includes at least one or more of ship query information, harbour query information, route query information.
Preferably, this method further include: indicate information, guidance information, prompting message, environmental information described in voice broadcast
One or more of with information in kind；And/or sound/light alert methods prompt important attention information.
Compared with prior art, the present invention program and/or embodiment one of at least have the advantages that: (1) providing
A kind of sail information enhancement device in waterway transportation uses for driver or commander, can by front water
Domain, river, lake surface, information is shown in the visual field of driver in the form of virtual identifying on ocean surface, is facilitated driver or is referred to
The person of waving makes corresponding operation (order), improves the safety in shipping process.(2) a kind of water-area navigation information enhancement is provided
System can collect the much information in water environment, and be presented to driver in the form of virtual image, be able to carry out void
Quasi- interactive operation, improves that operation of the shipping personnel in driving procedure be visual, convenience, enhances safety.(3) one is provided
Kind water-area navigation information enhancement method passes through subsystem information collection, information virtualization, reality enhancing display, interactive operation etc.
Step greatly facilitates operability of the shipping personnel in driving procedure, improves driving safety performance.
Detailed description of the invention
Fig. 1 is the system module schematic diagram in the present invention.
Fig. 2 is the vision figure under normal condition on daytime in the prior art.
Fig. 3 is vision figure of the daytime under normal condition in the present invention after augmented reality.
Fig. 4 is virtual sign title comparative diagram involved in Fig. 3/Fig. 7/Figure 10.
Fig. 5 is the schematic diagram being described in detail occur after clicking ship title of coming and going in visual field in a kind of embodiment.
Fig. 6 is the schematic diagram being described in detail occur after clicking the ship that comes and goes in visual field in a kind of embodiment.
Fig. 7 is the schematic diagram being described in detail occur after clicking menu in a kind of embodiment.
Fig. 8 is the schematic diagram that menu bar dragging is carried out in a kind of embodiment.
Fig. 9 is the vision figure under night in the prior art or dense fog state.
Figure 10 is the night in the present invention after augmented reality or the vision figure under dense fog state.
Figure 11 is vision figure when this ship is passed a bridge in the prior art.
Figure 12 is vision figure when this ship in the present invention after augmented reality is passed a bridge.
Figure 13 is a kind of schematic diagram of the water-area navigation information enhancement device in embodiment.
Specific embodiment
Present invention is further described in detail with specific embodiment with reference to the accompanying drawing.
In following implementation, same or similar label indicates same or similar original part or has identical from beginning to end
Or the original part of similar functions, it is exemplary below by way of the embodiment being described with reference to the drawings, for explaining only the invention, and
It is not considered as limiting the invention.
In the description that the present invention retouches, it is to be understood that term: center, longitudinally, laterally, length, width, thickness, it is upper,
Under, front, rear, left and right, vertical, level, top, bottom, the indicating positions such as inside and outside, clockwise, counterclockwise or positional relationship for based on
Orientation or positional relationship shown in the drawings, is merely for convenience of description of the present invention and simplification of the description, therefore should not be understood as to this
The limitation of invention.In addition, term: first, second etc. is used for description purposes only, and is not understood to indicate or imply relatively important
Property or the quantity for implicitly indicating shown technical characteristic.In description of the invention unless specifically defined or limited otherwise, term:
Installation, connected, connection etc. shall be understood in a broad sense, and those skilled in the art can understand above-mentioned term as the case may be
Concrete meaning in this practicability.
Referring to figs. 1 to Figure 13, a kind of water-area navigation information enhancement device involved in the application is included at least: waters ring
Border detector and augmented reality display, the water environment detector, for obtaining the information in kind of water environment；The increasing
Strong reality displays, the information in kind of the water environment for obtaining according at least to the water environment detector generate visual
The navigation virtual information of change, Overlapping display is in the visual field of the augmented reality display.
Specifically, the water environment detector includes: that wide-angle camera, zoom camera, RGB camera, depth are taken the photograph
As head, laser detector, millimetre-wave radar, sonar contact device, water speed survey meter, thermometer, hygrometer, anemoscope, anemobiagraph
One or more of, the RGB camera, for obtaining the image information in kind of the water environment；The depth is taken the photograph
As head, for obtaining the real space dimension information of the water environment；The laser detector, for obtaining the waters ring
The range information in kind in border；The millimetre-wave radar, in rain and fog weather, obtaining the distance letter in kind of the water environment
Breath；The sonar contact device, for obtaining the underwater environment information of the water environment；The water speed survey meter, for obtaining
The water velocity information of the water environment；The thermometer, for obtaining the temperature information of the water environment；The humidity
Meter, for obtaining the humidity information of the water environment；The anemoscope, for obtaining the wind direction information of the water environment；
The anemobiagraph, for obtaining the wind speed information of the water environment.
Above water environment detector can obtain the corresponding information in external environment, be shown in the visual field of driver
On, facilitate driver to check, it is highly-safe.
It further include sailing device on water detector, the sailing device on water detector, for obtaining sailing device on water itself
Information in kind, is shown on the visual field of driver, and driver is facilitated to check, highly-safe.The sailing device on water detector packet
It includes: one or more of magnetometer, gyroscope, navigator, draft gauge, the magnetometer, for obtaining the waters boat
The orientation information of sailing device on water where row information enhancement device；The gyroscope, for obtaining the water-area navigation information
The directional information of sailing device on water where enhancement device；The navigator, for obtaining the water-area navigation information enhancement dress
The latitude and longitude information of sailing device on water where setting；The draft gauge, for obtaining water-area navigation information enhancement device institute
Sailing device on water depth under water information.
It further include interactive component, the interactive component, for being interacted with the water-area navigation information enhancement device.Institute
Stating interactive component includes: that gesture identification component, wisdom gloves, head take aim at component, eye-tracking component, voice control component, manipulation
One or more of handle；The gesture identification component or wisdom gloves act for detection gesture, receive the control of input
System instruction；The head takes aim at component or eye-tracking component, for simulating the control instruction of mouse；The voice control portion
Part, for receiving the control instruction of voice input；The control handle, control instruction for receiving input.By with
On interactive component, driver can operate the virtual signage being shown in the visual field, obtain further information, can also
To input information.
It further include AI picture recognition module, the AI picture recognition module, for according to the reality for obtaining water environment
Object information identifies the water surface, waterfront, contact ship, bridge, danger zone, shallow water area.
Further include virtual object generation module, the information such as virtual navigation channel line, warning line, danger zone can be generated.The void
Quasi- object generation module, for the recognition result according to the AI picture recognition module in the visual field of the augmented reality display
Overlapping display indicates one or more of information, guidance information, prompting message, environmental information and information in kind, the finger
Show information, includes at least river width information, bridge pier width information, the sailing device on water width information and longitude and latitude position letter
One or more of breath；The guidance information includes at least contact navigation channel cut-off rule information, sailing device on water navigation
One or more of the location information that other sailing device on water navigate by water in direction and velocity information, cartographic information and map；
The prompting message, include at least one of shipwreck area information, shallow water area information, the high information of bridge, anticollision information or
Person is a variety of；The environmental information, including at least Weather information, wind speed/wind direction information, temperature/humidity information, rise/ebb tide information, water
Flow one or more of direction/velocity information；The material object information includes at least ship query information, harbour inquiry letter
One or more of breath, route query information.
Further include loudspeaker and/or attention device, the loudspeaker, for instruction information, guidance information described in voice broadcast,
One or more of prompting message, environmental information and information in kind；The attention device is prompted for alarm song/light mode
Important attention information.
A kind of water-area navigation information enhancement system involved in the application, comprising: water-area navigation information enhancement device also wraps
Include sailing device on water information management subsystem, electronic channel chart subsystem, ship automatic identification subsystem, global location subsystem
System, intelligent one or more of waters material object recognition subsystem and weather forecast reality man who brings news of appointment's system；The water-area navigation letter
Enhancement device is ceased, for sub from the sailing device on water information management subsystem, electronic channel chart subsystem, ship automatic identification
One of system, global location subsystem, intelligent waters material object recognition subsystem, weather forecast reality man who brings news of appointment's system or it is a variety of in
The information in kind of water environment is obtained, visual navigation virtual information is generated, Overlapping display increases in the water-area navigation information
In the visual field of intensity device.
Further, wherein water-area navigation information enhancement device includes communication module, the communication module, is used for from local
It is (edge calculations node) loading or the network equipment (including cloud calculate node and/or edge calculations node) loading described
Electronic channel chart subsystem, ship automatic identification subsystem, global location subsystem, intelligent waters material object recognition subsystem and/
Or the information in kind of water environment is obtained in weather forecast reality man who brings news of appointment's system, generate visual navigation virtual information.
Further, wherein water-area navigation information enhancement device includes virtual object generation module, and the virtual object generates mould
Block, for generating visual navigation virtual information, being shown in the augmented reality according to the information in kind for obtaining water environment
Overlapping display in the visual field of device.
Further, the navigation virtual information includes: instruction information, guidance information, prompting message, environmental information and reality
One or more of object information, the instruction information include at least river width information, bridge pier width information, the water
One or more of upper aircraft width information and longitude and latitude positional information；The guidance information includes at least contact boat
Other sailing device on water in road cut-off rule information, sailing device on water navigation direction and velocity information, cartographic information and map
One or more of location information of navigation；The prompting message includes at least shipwreck area information, shallow water area is believed
One or more of breath, the high information of bridge, anticollision information；The environmental information includes at least Weather information, wind speed/wind
To information, temperature/humidity information, rise/ebb tide information, one or more of water (flow) direction/velocity information；The letter in kind
Breath includes at least one or more of ship query information, harbour query information, route query information.
A kind of water-area navigation information enhancement method involved in the application, according to the information in kind of water environment, in waters
Overlapping display visually navigates by water virtual information in the visual field of the augmented reality display of sail information enhancement device, including virtual
Navigation channel line, virtual warning line, virtual danger zone mark etc., judge convenient for driver, improve the safety of driving.
Preferably, further comprising the steps of: from the water environment detector of water-area navigation information enhancement device, obtaining institute
The information in kind of water environment is stated, visual navigation virtual information is generated.
Preferably, further comprising the steps of: the electronic channel chart loaded from the locally loaded or network equipment
Subsystem, ship automatic identification subsystem, global location subsystem, intelligent waters material object recognition subsystem and/or weather forecast
The information in kind that water environment is obtained in real man who brings news of appointment's system generates visual navigation virtual information.
Preferably, further comprising the steps of: by interactive component, to water-area navigation information enhancement device and/or network
Equipment issues the control instruction for obtaining the information in kind of the water environment.
Preferably, further comprising the steps of: passing through water-area navigation information enhancement device and/or the AI image of the network equipment
Identification module obtains the information in kind of water environment, identifies the water surface, waterfront, contact ship, bridge, danger zone, phytal zone
Domain；According to the recognition result of the AI picture recognition module, pass through the void of water-area navigation information enhancement device and/or the network equipment
Quasi- object generation module generates the visual navigation virtual information.
Preferably, the navigation virtual information includes: instruction information, guidance information, prompting message, environmental information and reality
One or more of object information, the instruction information include at least river width information, bridge pier width information, the water
One or more of upper aircraft width information and longitude and latitude positional information；The guidance information includes at least contact boat
Other sailing device on water in road cut-off rule information, sailing device on water navigation direction and velocity information, cartographic information and map
One or more of location information of navigation；The prompting message includes at least shipwreck area information, shallow water area is believed
One or more of breath, the high information of bridge, anticollision information；The environmental information includes at least Weather information, wind speed/wind
To information, temperature/humidity information, rise/ebb tide information, one or more of water (flow) direction/velocity information；The letter in kind
Breath includes at least one or more of ship query information, harbour query information, route query information.
Preferably, this method further include: indicate information, guidance information, prompting message, environmental information described in voice broadcast
One or more of with information in kind；And/or sound/light alert methods prompt important attention information.
It illustrates by the following examples.
Embodiment one
As shown in Fig. 2, in the case of daytime normal weather, the visual field of driver during sailing.Visual field optical driver window
Edge 8, there are also window frame 6 in drive window, the lower part of drive window is control area 11.Go out in terms of drive window it can be seen that beacon 1,
Ship 2, buoy light 3, fore 4, water front 5.Daytime, these scenes were as it can be seen that but cannot show the boundary of water front 5 well, drive boat
The information such as road, ship title, can not show danger zone.
As shown in figure 3, driver uses the water-area navigation information enhancement device (such as wear-type AR glasses) in the application, depending on
The scene of Yezhong is changed, and many virtual images are superimposed upon in display scene, e.g., overrides obstruction vision with background image
Window frame 6, can show in the visual field and display water front 5 is reinforced by color, driver is made clearly to see the boundary in waters；In the visual field
It can show virtual course line, cut-off rule, warning line etc., make the information such as the location of clear ship of driver, direction of advance,
It avoids driving into danger zone；Title mark, such as beacon 1, ship title, buoy light 3 can be carried out to actual object in the visual field,
Play the purpose of warning alert driver；In addition, the lower part in the visual field can show virtual action bar, phase can be shown after click
The information answered, such as: operational configuration, weather, time, route information, harbour inquiry, system function；It can also be shown in the visual field
Optimal course-and-bearing arrow, facilitates driver to operate.Respective fictional shows the meaning title of object in Fig. 3, refers to Fig. 4.
In Fig. 3, i.e., in the visual field of water-area navigation information enhancement device, clicked by interactive device as past in Fig. 5
The display Name for carrying out ship will further display the relevant information of its navigation；It is clicked by interactive device such as the contact ship in Fig. 6
Image only will further display the relevant information of its navigation；Chain is gone back in water-area navigation information enhancement system is connected to ship registration
Information query system, ship photo, ship parameter when including factory etc..By AI image comparison identification technology, can pass through
The shape of ship in ship register information inquiry system so that obtain the relevant information of the ship.Even, by linking port
Mouth information system, can inquire the information such as starting port and port of destination, the ship visitor's loading number of passenger boat；The starting of freighter can be inquired
The information such as port and port of destination, charging burden.
In Fig. 3, i.e., in the visual field of water-area navigation information enhancement device, clicked by interactive device such as the shape in Fig. 7
Button in state information bar, the information of the navigation areas such as the relevant information of available ship and weather；Or even port of destination
Relevant information.The information further shown after click, as shown in figure 8, can also carry out arbitrarily dragging ease to stop in visual field.
Embodiment two
As shown in figure 9, in the case where night or dense fog, the visual field of driver during sailing.In the visual field only
Point point light source 7 and window frame 6, but do not know be what object of where light source, driver/steersman micro-judgment can only be relied on, very
It is dangerous.
As shown in Figure 10, driver uses the water-area navigation information enhancement device (such as wear-type AR glasses) in the application,
Scene in the visual field is changed, and many virtual images are superimposed upon in display scene, e.g., can be shown in the visual field and be passed through color
Display display water front 5, makes driver clearly see the boundary in waters；Virtual course line, cut-off rule, warning can be shown in the visual field
Line etc. makes the information such as the location of clear ship of driver, direction of advance, avoids driving into danger zone；In the visual field, user
Title mark, such as beacon 1, ship title, buoy light 3 can be carried out to actual object in real time, play warning alert driver's
Purpose；In addition, the lower part in the visual field can show virtual action bar, corresponding information can be shown after click, such as: navigation shape
State, weather, time, route information, harbour inquiry, system function etc.；Optimal course-and-bearing arrow can also be shown in the visual field,
Driver is facilitated to operate.
Embodiment three
As shown in figure 11, in the case of normal weather on daytime, the visual field of the driver when sailing to pass a bridge.Only have in the front in the visual field
Bridge 10, driver need to drive ship and pass through between two bridge piers 9, need experience and operation by experienced driver.
As shown in figure 12, driver uses this water-area navigation information enhancement device (such as wear-type AR glasses, projection arrangement),
Scene in the visual field is changed, and many virtual images are superimposed upon in display scene, e.g., can be shown in the visual field and be passed through color
Display display water front 5, makes driver clearly see the boundary in waters；Virtual course line, cut-off rule, warning can be shown in the visual field
Line etc. makes the information such as the location of clear ship of driver, direction of advance, can also understand this ship in navigation by beam line
Width position, beam line is can be extended, and by the relationship of beam line and pier location, driver adjusts this ship in time
Driving direction avoids driving into danger zone；The height of title, bridge from the water surface for being labelled with bridge in the visual field simultaneously (can basis
Real-time measurement can also convert according to same day tidal condition), the letter such as the entire length of the distance between two bridge piers, bridge
Breath converts to obtain this ship water surface elevation information further according to this ship elevation information, drinking water information, driver is facilitated to judge and pass through
Bridge opening.
In above embodiments, used information data comes from each information collection to sail information enhancement device at work
System, such as water environment detector, sailing device on water detector, AI picture recognition module, further include virtual object generation module
And other subsystems, such as AIS system, electronic navigation drawing system, GPS system, information acquisition system, these information are by processing
Showing after system processing and is superimposed upon in the visual field of driver, processing system can integrate in sail information enhancement device,
It can be the self-contained unit on ship, communication connection is kept between each equipment, subsystem.In addition, being driven by interactive component
Personnel can also interact with water-area navigation information enhancement device.
In above embodiments sail information enhancing system by processing system (CPU), electronic channel chart system, AIS system,
GPS system, information acquisition system and AR display system composition, as shown in Figure 1.Electronic channel chart system, AIS system, GPS system
It can be read with the information in information acquisition system with system processed (CPU), after the information that processing system (CPU) processing obtains,
By calculating and handling, virtual navigation channel line, beam line, navigation index line, barrier, phytal zone, bridge dimension mark are generated
The information such as line, bridge pier danger area, are shown by AR display system.Such as, processing system (CPU) provides navigation mark, depth of water navigation channel
The underwater obstruction information such as element shipwreck, reef, bridge, harbour etc. face the information such as the building that crosses the river.AIS system is processing system
(CPU) periphery ship information is provided.GPS system is that processing system (CPU) provides ship's fix information.Information acquisition system can be with
Using equipment such as camera, infrored equipment radars, it is installed on stem and the ship side position of ship, real-time acquisition water surface ship,
The information such as navigation mark, bridge, harbour, obstacle information and submerged wreck, reef, barrier, the depth of water.Have the advantage that 1.
Clear, intuitive navigation is provided for cruiseway deck officer, reduces the operation difficulty and technical threshold of deck officer.2.
Effective method is provided for the standardization of cruiseway ship running, safe.3. under night or inclement weather conditions, can be
Cruiseway deck officer provides and clearly navigates.
In a kind of embodiment, the specific treatment process of system is as follows:
1. processing system (CPU) is according to the information such as width, length, the trend in river in course line whole in electronic channel chart,
Generate virtual bank line identical with practical riverbank.
2. if processing system (CPU) is according to the boat in electronic channel chart there is the dividing line in navigation channel in electronic channel chart
Road dividing line generates the virtual warning navigation channel line, virtual central breaks and virtual common navigation channel line of coincidence respectively.If electronics
There is no the dividing line in navigation channel in navigation channel chart, then generates virtual warning navigation channel line, virtual central breaks and void according to following methods
Intend common navigation channel line.
(1) processing system (CPU) calculates navigation channel according to the drinking water of electronic channel chart midchannel water depth distribution and ship
Be not suitable for the phytal zone of traveling, phytal zone and travelable region with virtual warning navigation channel line segmentation.
(2) the warning navigation channel line virtual according to navigation channel two sides, calculates virtual central breaks, does not go together for dividing
Sail the navigation area in direction.
(3) navigation area of same driving direction generates between virtual central breaks and virtual warning navigation channel line
Virtual common navigation channel line, to distinguish the different navigation areas of same driving direction.Virtual common navigation channel line number amount is according to boat
Road management requires to determine.
3. processing system (CPU) travels the information of ship according to the cruiseway provided in AIS system or electronic channel chart,
In conjunction with GPS system, information of the ship around travelled, such as Hull Number, travel speed, the dimension of ship of ship etc. are identified, and
Generate the virtual ship threedimensional model that the respective corresponding coincidence of ship is travelled with surrounding.
4. processing system (CPU) reads electronic channel chart Air China target position and information, such as crosses for indicating navigation channel
The river in range and course line mark, land mark, beacon, crossing leading marks, head and stern leading marks, bridge mark；Triangle for indicating barrier is floating
Mark, floating drum, light boat, left and right navigation buoy, stick buoy etc.；It is used to indicate navigation channel depth, aerial and underwater line, forecast landscape
With the depth signal pole of commander's narrow fairway traffic, right of way signal platform, whistle post, boundary mark, cable mark, lateral flow buoy, wind news
Signal lever etc..According to physical location in electronic channel chart, the virtual navigation mark threedimensional model being overlapped with practical navigation mark is generated, and right
Virtual instruction information should be generated.
Information acquisition system for acquiring ship peripheral information, such as underwater obstruction zone position and range, buoy position in real time
It sets, the information such as surrounding dimension of ship, bridge height and pier location, processing system (CPU) is transferred to, for correcting and assisting electricity
Sub- navigation channel chart.
According to AR location technology, the virtual navigation channel cartographic information of generation is folded with Information locating corresponding in electronic channel chart
It closes.
In the application, water-area navigation information enhancement device can both be set the projection on image projecting to driver's cabin window
It is standby, or wear-type AR glasses or mobile intelligent terminal (mobile phone or PAD).It is a kind of embodiment shown in Figure 13
In wear-type AR glasses form water-area navigation information enhancement device, wear-type AR glasses include main body, the front of main body
Equipped with display eyeglass 92 and front end camera 96, it is respectively used to display superimposed image and obtains the image data of front；Main body
Side is equipped with gesture-capture camera 94, for identification with capture operation by human hand, realizes interactive function；Earphone is additionally provided in main body
93 and headset 95, for playing voice, input voice, realize human-computer interaction.
The above, the present invention provides a kind of sail information enhancement device and method in waterway transportation, for
Driver use, the much information in water environment can be collected, and driver is presented in the form of virtual image, will before
Square waters, river, lake surface, information is shown in the visual field of driver in the form of virtual identifying on ocean surface, and driver is facilitated to do
Corresponding operation out improves the safety in shipping process；It is able to carry out virtual interacting operation, improves shipping personnel in drive the cross
Operation ease in journey enhances safety.
Protection scope of the present invention includes but is not limited to embodiment of above, and protection scope of the present invention is with claims
Subject to, replacement, deformation, the improvement that those skilled in the art that any pair of this technology is made is readily apparent that each fall within of the invention
Protection scope.
Claims (10)
1. a kind of water-area navigation information enhancement device, which is characterized in that include at least: water environment detector and augmented reality are aobvious
Show device,
The water environment detector, for obtaining the information in kind of water environment；
The augmented reality display, the letter in kind of the water environment for being obtained according at least to the water environment detector
Breath generates visual navigation virtual information, and Overlapping display is in the visual field of the augmented reality display.
2. a kind of water-area navigation information enhancement device as described in claim 1, which is characterized in that further include that sailing device on water is visited
Survey device, the sailing device on water detector, for obtaining the information in kind of sailing device on water itself.
3. a kind of water-area navigation information enhancement device as described in claim 1, which is characterized in that it further include interactive component, institute
Interactive component is stated, for being interacted with the water-area navigation information enhancement device.
4. such as a kind of water-area navigation information enhancement device any one of claims 1 to 3, which is characterized in that further include AI
Picture recognition module, the AI picture recognition module, for according to the information in kind for obtaining water environment, identification water outlet
Face, waterfront, contact ship, bridge, danger zone, shallow water area.
5. a kind of water-area navigation information enhancement device as claimed in claim 4, which is characterized in that further include that virtual object generates mould
Block, the virtual object generation module, for being shown according to the recognition result of the AI picture recognition module in the augmented reality
One of Overlapping display instruction information, guidance information, prompting message, environmental information and information in kind or more in the visual field of device
Kind,
The instruction information includes at least river width information, bridge pier width information, the sailing device on water width information and warp
One or more of Position Latitude information；
The guidance information, include at least contact navigation channel cut-off rule information, the sailing device on water navigation direction and velocity information,
One or more of the location information that other sailing device on water navigate by water in cartographic information and map；
The prompting message, including at least one in shipwreck area information, shallow water area information, the high information of bridge, anticollision information
Kind is a variety of；
The environmental information, including at least Weather information, wind speed/wind direction information, temperature/humidity information, rise/ebb tide information, water flow
One or more of direction/velocity information；
The material object information, includes at least one of ship query information, harbour query information, route query information or more
Kind.
6. a kind of water-area navigation information enhancement system, comprising: water-area navigation information enhancement device, which is characterized in that further include water
Upper aircraft information management subsystem, electronic channel chart subsystem, ship automatic identification subsystem, global location subsystem, intelligence
It can one or more of waters material object recognition subsystem and weather forecast reality man who brings news of appointment's system；
The water-area navigation information enhancement device, for sub from the sailing device on water information management subsystem, electronic channel chart
System, ship automatic identification subsystem, global location subsystem, intelligent waters material object recognition subsystem, weather forecast reality man who brings news of appointment
One of system or a variety of middle information in kind for obtaining water environment, generate visual navigation virtual information, Overlapping display
In the visual field of the water-area navigation information enhancement device.
7. a kind of water-area navigation information enhancement system as claimed in claim 6, which is characterized in that water-area navigation information enhancement dress
It sets including communication module, the communication module, the electronic channel chart for being loaded from the locally loaded or network equipment
Subsystem, ship automatic identification subsystem, global location subsystem, intelligent waters material object recognition subsystem and/or weather forecast
The information in kind that water environment is obtained in real man who brings news of appointment's system generates visual navigation virtual information.
8. a kind of water-area navigation information enhancement system as claimed in claims 6 or 7, which is characterized in that water-area navigation information increases
Intensity device includes virtual object generation module, the virtual object generation module, for according to the information in kind for obtaining water environment, life
At visual navigation virtual information, the Overlapping display in the visual field of the augmented reality display.
9. a kind of water-area navigation information enhancement system as claimed in claim 8, which is characterized in that the navigation virtual information packet
It includes: one or more of instruction information, guidance information, prompting message, environmental information and information in kind,
The instruction information includes at least river width information, bridge pier width information, the sailing device on water width information and warp
One or more of Position Latitude information；
The guidance information, include at least contact navigation channel cut-off rule information, the sailing device on water navigation direction and velocity information,
One or more of the location information that other sailing device on water navigate by water in cartographic information and map；
The prompting message, including at least one in shipwreck area information, shallow water area information, the high information of bridge, anticollision information
Kind is a variety of；
The environmental information, including at least Weather information, wind speed/wind direction information, temperature/humidity information, rise/ebb tide information, water flow
One or more of direction/velocity information；
The material object information, includes at least one of ship query information, harbour query information, route query information or more
Kind.
10. a kind of water-area navigation information enhancement method, which is characterized in that according to the information in kind of water environment, in water-area navigation
Overlapping display visually navigates by water virtual information in the visual field of the augmented reality display of information enhancement device.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910278629.5A CN109911140A (en) | 2019-04-09 | 2019-04-09 | A kind of water-area navigation information enhancement device, system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910278629.5A CN109911140A (en) | 2019-04-09 | 2019-04-09 | A kind of water-area navigation information enhancement device, system and method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109911140A true CN109911140A (en) | 2019-06-21 |
Family
ID=66969071
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910278629.5A Pending CN109911140A (en) | 2019-04-09 | 2019-04-09 | A kind of water-area navigation information enhancement device, system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109911140A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110837299A (en) * | 2019-11-11 | 2020-02-25 | 上海萃钛智能科技有限公司 | Activity management intelligent device, system and method |
| US10739141B2 (en) | 2013-09-03 | 2020-08-11 | Fugro Chance, Inc. | Interactive remote guidance system for seaborne vessels |
| CN113044180A (en) * | 2021-03-08 | 2021-06-29 | 杭州华雁数码电子有限公司 | Ship navigation monitoring system based on big data center and safety monitoring method thereof |
| CN113445987A (en) * | 2021-08-05 | 2021-09-28 | 中国铁路设计集团有限公司 | Railway drilling auxiliary operation method based on augmented reality scene under mobile terminal |
| CN114115264A (en) * | 2021-11-19 | 2022-03-01 | 四方智能(武汉)控制技术有限公司 | Unmanned ship surveying and mapping navigation system and control method thereof |
| CN114115548A (en) * | 2022-01-29 | 2022-03-01 | 亿海蓝（北京）数据技术股份公司 | Ship information display method and system, electronic device and readable storage medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103975576A (en) * | 2011-07-21 | 2014-08-06 | 韩国海洋研究院 | Augmented reality system using moving ceiling transparent display for ship and method for enabling same |
| CN104781873A (en) * | 2012-11-13 | 2015-07-15 | 索尼公司 | Image display device and image display method, mobile body device, image display system, and computer program |
| CN105593641A (en) * | 2013-10-01 | 2016-05-18 | 戴姆勒股份公司 | Method and device for augmented display |
| US9751607B1 (en) * | 2015-09-18 | 2017-09-05 | Brunswick Corporation | Method and system for controlling rotatable device on marine vessel |
| CN206557751U (en) * | 2016-08-31 | 2017-10-13 | 上海鹰觉科技有限公司 | Naval vessels augmented reality integrated information display device |
| CN108550281A (en) * | 2018-04-13 | 2018-09-18 | 武汉理工大学 | A kind of the ship DAS (Driver Assistant System) and method of view-based access control model AR |
| CN109491497A (en) * | 2018-10-19 | 2019-03-19 | 华中科技大学 | A kind of human assistance assembly application system based on augmented reality |
-
2019
- 2019-04-09 CN CN201910278629.5A patent/CN109911140A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103975576A (en) * | 2011-07-21 | 2014-08-06 | 韩国海洋研究院 | Augmented reality system using moving ceiling transparent display for ship and method for enabling same |
| CN104781873A (en) * | 2012-11-13 | 2015-07-15 | 索尼公司 | Image display device and image display method, mobile body device, image display system, and computer program |
| CN105593641A (en) * | 2013-10-01 | 2016-05-18 | 戴姆勒股份公司 | Method and device for augmented display |
| US9751607B1 (en) * | 2015-09-18 | 2017-09-05 | Brunswick Corporation | Method and system for controlling rotatable device on marine vessel |
| CN206557751U (en) * | 2016-08-31 | 2017-10-13 | 上海鹰觉科技有限公司 | Naval vessels augmented reality integrated information display device |
| CN108550281A (en) * | 2018-04-13 | 2018-09-18 | 武汉理工大学 | A kind of the ship DAS (Driver Assistant System) and method of view-based access control model AR |
| CN109491497A (en) * | 2018-10-19 | 2019-03-19 | 华中科技大学 | A kind of human assistance assembly application system based on augmented reality |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10739141B2 (en) | 2013-09-03 | 2020-08-11 | Fugro Chance, Inc. | Interactive remote guidance system for seaborne vessels |
| US10746553B2 (en) | 2013-09-03 | 2020-08-18 | Fugro Chance, Inc. | Interactive remote guidance system for seaborne vessels |
| US10775176B2 (en) | 2013-09-03 | 2020-09-15 | Fugro Chance, Inc. | Interactive remote guidance system for seaborne vessels |
| US10859381B2 (en) | 2013-09-03 | 2020-12-08 | Fugro Usa Marine, Inc. | Interactive remote guidance system for seaborne vessels |
| CN110837299A (en) * | 2019-11-11 | 2020-02-25 | 上海萃钛智能科技有限公司 | Activity management intelligent device, system and method |
| CN113044180A (en) * | 2021-03-08 | 2021-06-29 | 杭州华雁数码电子有限公司 | Ship navigation monitoring system based on big data center and safety monitoring method thereof |
| CN113445987A (en) * | 2021-08-05 | 2021-09-28 | 中国铁路设计集团有限公司 | Railway drilling auxiliary operation method based on augmented reality scene under mobile terminal |
| CN114115264A (en) * | 2021-11-19 | 2022-03-01 | 四方智能(武汉)控制技术有限公司 | Unmanned ship surveying and mapping navigation system and control method thereof |
| CN114115264B (en) * | 2021-11-19 | 2024-04-30 | 四方智能(武汉)控制技术有限公司 | Unmanned ship survey and drawing navigation system and control method thereof |
| CN114115548A (en) * | 2022-01-29 | 2022-03-01 | 亿海蓝（北京）数据技术股份公司 | Ship information display method and system, electronic device and readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109911140A (en) | A kind of water-area navigation information enhancement device, system and method | |
| CN107748561B (en) | Unmanned ship local obstacle avoidance system and method based on multiple sensing parameters | |
| US9401049B2 (en) | Augmented reality system using moving ceiling transparent display for ship and method for enabling same | |
| KR101289349B1 (en) | Marine navigation system | |
| CN101214851B (en) | Intelligent all-weather actively safety early warning system and early warning method thereof for ship running | |
| US20210206459A1 (en) | Video sensor fusion and model based virtual and augmented reality systems and methods | |
| US11892298B2 (en) | Navigational danger identification and feedback systems and methods | |
| KR101894674B1 (en) | Ship navigation apparatus and method for providing route information for ship | |
| KR101799216B1 (en) | Ship navigation apparatus and method for providing route information for ship | |
| KR101719142B1 (en) | Ship navigation apparatus and method for providing route information for ship | |
| US20200369351A1 (en) | Marine docking and object awareness system | |
| US20220172464A1 (en) | Water non-water segmentation systems and methods | |
| US20220392211A1 (en) | Water non-water segmentation systems and methods | |
| JP2022173157A (en) | Tidal current information display device | |
| JP2011225208A (en) | Image display system, image display method and program | |
| CN116824912A (en) | Digital twin inland river ship navigation management and control system | |
| CN117232520A (en) | Ship intelligent navigation system and navigation method suitable for offshore navigation | |
| US20220214689A1 (en) | Autonomous Vessel and Infrastructure for Supporting an Autonomous Vessel on Inland Waterways | |
| KR102464207B1 (en) | Ship safety management system using augmented reality | |
| US20230195118A1 (en) | Autonomous marine autopilot system | |
| CN209641094U (en) | Water-area navigation information enhancement glasses | |
| KR102392134B1 (en) | Maritime accident prevention guide system for ships in sailing by using mobile terminal | |
| Xiaoxia et al. | Electronic chart display and information system | |
| Okazaki et al. | Development of sea route display system by using augmented reality | |
| KR102249156B1 (en) | Sailing assistance device using augmented reality image |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |