CN111866464B - Agricultural tractor remote control system based on virtual reality technology - Google Patents
Agricultural tractor remote control system based on virtual reality technology Download PDFInfo
- Publication number
- CN111866464B CN111866464B CN202010735665.2A CN202010735665A CN111866464B CN 111866464 B CN111866464 B CN 111866464B CN 202010735665 A CN202010735665 A CN 202010735665A CN 111866464 B CN111866464 B CN 111866464B
- Authority
- CN
- China
- Prior art keywords
- scene
- simulation
- agricultural tractor
- driving
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
- H04N7/181—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a plurality of remote sources
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/20—Scenes; Scene-specific elements in augmented reality scenes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- General Engineering & Computer Science (AREA)
- Signal Processing (AREA)
- Human Computer Interaction (AREA)
- Control Of Position, Course, Altitude, Or Attitude Of Moving Bodies (AREA)
Abstract
The invention provides a virtual reality technology-based agricultural tractor remote control system. The system comprises a scene recognition module, a state monitoring module, a data transmission module, a cloud scene simulation module and an AR remote control module. The invention has the beneficial effects that: according to the method, through scene recognition, the driving image is a comprehensive image in six directions, so that scene elements of the agricultural tractor can be comprehensively acquired, the driving scene is determined, and the running environment is comprehensively monitored. The state monitoring aims at monitoring the real-time state of the agricultural tractor and judging the body data of the agricultural tractor. During data transmission, different data transmission modes are adopted according to the number of different agricultural tractors, and the comprehensiveness of the efficiency of data transmission is improved conveniently. And (4) simulating a cloud scene. The real state of the agricultural tractor can be reflected based on the cloud big data technology, and then the real-time state of the agricultural tractor is remotely observed through the AR far-end control module, real-time monitoring is carried out, and real-time remote control is realized.
Description
Technical Field
The invention relates to the technical field, in particular to an agricultural tractor remote control system based on a virtual reality technology.
Background
At present, a tractor is used as a high-horsepower device, is generally used for agricultural transportation and agricultural planting, and is mostly directly used for agriculture, rural areas and farmers and is equipment for agricultural modernization. The development level of agricultural equipment directly influences the technical level and the economic benefit of agricultural departments in China. Agricultural equipment is largely used in modern agricultural production, so that agricultural productivity and agricultural product commercialization rate are improved, grain production safety is guaranteed, and farmer income is increased, and the most agricultural equipment is a tractor.
When the tractor is in use. In the face of the requirements of various agricultural products, complex scene conditions, and huge requirements on the state, performance, service life, cost and the like of a tractor product, the agricultural scene simulation and remote control are realized along with the development of the internet technology, the scene simulation technology and the sensing technology, so that the high automation of agricultural work is promoted, and in recent years, along with the continuous development of the virtual reality technology, an operation and control system with immersion is established, and the remote operation and control is a new development trend.
Disclosure of Invention
The invention provides an agricultural tractor remote control system based on a virtual reality technology, which is used for solving the problems.
A farm tractor remote control system based on virtual reality technology, characterized by comprising:
a scene recognition module: the system comprises a driving image acquisition module, a driving scene determination module and a driving scene determination module, wherein the driving image acquisition module is used for acquiring a driving image of an operating agricultural tractor, extracting image elements and determining the driving scene;
a state monitoring module: the agricultural tractor is used for determining a driving state through a sensing device arranged on the agricultural tractor;
a data transmission module: the system comprises a data transmission mode determining module, a cloud scene simulation module and a driving state simulation module, wherein the data transmission mode is used for transmitting the driving scene and the driving state to the cloud scene simulation module according to the data transmission mode;
cloud scene simulation module: the simulation system is used for constructing a simulation scene of the agricultural tractor according to the driving scene and the driving state and generating a real-time simulation video;
the AR far-end control module: and the real-time simulation video is displayed through the AR equipment, simulation parameters are determined, the running scene of the agricultural tractor is optimized according to the simulation parameters, and a control instruction is generated.
As an embodiment of the present invention, a scene recognition module includes:
an image pickup unit: the image acquisition device is used for acquiring a scene image through the image acquisition device arranged on the agricultural tractor; wherein the content of the first and second substances,
the number of the camera devices is not less than 5; wherein the content of the first and second substances,
at least 2 camera devices with 120-degree camera angles are arranged on two sides of the tail of the agricultural tractor;
at least 2 camera devices with 180-degree camera angles are arranged on two sides of the middle part of the agricultural tractor;
at least 1 camera device with a 180-degree camera angle is arranged on two sides of the agricultural tractor head;
an element extraction unit: the image processing device is used for judging whether image elements in the scene image influence the driving behavior of the agricultural tractor or not according to the scene image, and extracting the image elements when the image elements influence the driving behavior of the agricultural tractor;
the image elements include road surface elements, obstacle elements, traffic indication elements, and weather elements;
a scene determination unit: the image element is used for comparing with a historical driving scene to determine the driving scene; wherein the content of the first and second substances,
and when the image element does not exist in the historical driving scene, transmitting the image element to a user terminal, determining the driving scene, and storing the driving scene.
As an embodiment of the present invention, the state monitoring module includes:
a sensor unit: the agricultural tractor is used for acquiring state data according to preset sensing equipment on the agricultural tractor; wherein the content of the first and second substances,
the sensing device includes: temperature sensing equipment, speed sensing equipment, a position sensor, a liquid level sensor, an energy consumption sensor, a speed sensor, an acceleration sensor, a ray radiation sensor, a thermosensitive sensor, a vibration sensor and a humidity sensor;
a signal strength unit: the system comprises an AR remote control module, a verification data acquisition module, a signal processing module and a signal processing module, wherein the AR remote control module is used for acquiring the verification data of the agricultural tractor and the signal processing module;
a data processing unit: the system is used for determining the correlation among the state data and determining a dynamic state model according to the correlation;
threshold unit: the device parameter setting module is used for acquiring the device parameters of the agricultural tractor and setting a threshold model according to the device parameters;
a state judgment unit: the dynamic state model is used for comparing the threshold model with the dynamic state model, determining the driving difference, calculating the state loss value of the agricultural tractor under the driving difference according to the signal intensity, and determining the driving state.
As an embodiment of the present invention, a data transmission module includes:
a transmission path judgment unit: the system is used for acquiring a data transmission mode and verifying whether a data transmission channel is synchronous transmission or not;
a mode setting unit: for setting a data transmission mode according to the strength of the signal strength in the driving scene, wherein,
the data transmission single-channel data transmission mode and the multi-channel data transmission mode;
a mode selection unit: and the data transmission mode is determined according to the priority.
As an embodiment of the present invention, the cloud scene simulation module includes:
a data acquisition unit: the system comprises a driving scene and a driving state, wherein the driving scene and the driving state are used for determining scene data and state data according to the driving scene and the driving state;
a scene processing unit: the scene simulation system is used for establishing a scene simulation model by taking the scene data as a data source and determining a scene video;
a state processing unit: the state data is used as a data source and is substituted into a preset simulated agricultural tractor to determine a state video;
a fusion unit: the system is used for fusing the state video into a scene video and determining a simulation video;
a real-time processing unit: the real-time simulation video fusion unit is used for determining difference data according to the data acquisition unit and the fusion unit, substituting the difference data into the simulation video according to the type of the difference data, and determining the real-time simulation video.
As an embodiment of the present invention, the AR remote control module includes:
an apparatus control unit: the system is used for acquiring the simulation video and pushing the simulation video to the AR equipment;
a parameter extraction unit: the simulation system is used for determining a simulation program package and a simulation language according to the simulation video, determining a simulation object parameter of the agricultural tractor according to the simulation program package, and determining a simulation structure parameter of the agricultural tractor according to the simulation language;
an apparatus manipulation unit: the simulation system is used for generating an object regulation and control parameter according to the simulation object parameter, generating an object regulation and control window according to the simulation structure parameter and the object regulation and control parameter, and receiving a first regulation and control action input by a user;
an autonomous regulation unit: the ant colony optimization algorithm is used for processing the simulation object parameters to obtain autonomous regulation and control parameters and generate a second regulation and control behavior;
an instruction generation unit: the regulating and controlling intention is judged according to the first regulating and controlling behavior and the second regulating and controlling behavior, and a regulating and controlling instruction is generated; wherein the content of the first and second substances,
the first regulatory behavior is higher in priority than the second regulatory behavior;
an instruction implementation unit: and the control instruction is sent to the agricultural tractor, control information is determined, and control operation is executed.
As an embodiment of the present invention, the signal strength determining unit determines the signal strength by:
step 1: determining a channel feature set and a base station feature set between the agricultural tractor and the AR remote control module based on the agricultural tractor and the AR remote control module
：
Wherein, the
Is shown as
A plurality of channels; the above-mentioned
Is shown as
Characteristics of individual base stations; the above-mentioned
Is shown as having
A channel, in common
A base station;
step 2: substituting the channel characteristics and the base station characteristics into a relation model to determine the implementation probability of any channel and any base station
：
Wherein, the
Representing a mean value of channel characteristics; the above-mentioned
Representing and calculating a characteristic mean value of the base station; the above-mentioned
Is shown as
A channel and a
Implementation probability of each base station;
and step 3: determining the implementation capability of the channel according to the implementation probability
：
Wherein, the
Is shown as
Transmission capabilities of individual channels;
and 4, step 4: constructing a strength model according to the implementation capacity
：
Wherein, the
Represents the signal strength;
setting intensity threshold
When said
When, it indicates that the signal strength is strong; when said
When it is, the signal strength is weak.
As an embodiment of the present invention, the scene recognition module further includes a cloud control unit and an AI recognition unit; wherein
The cloud control unit comprises a cloud server, and the cloud server is used for constructing an AI quantitative model, an AI mode identification model and an AI analysis model in the AI identification unit through a big data technology and a general AI model; the cloud server is also used for constructing a special communication channel between the AI identification unit and the agricultural tractor through a cloud network;
the AI identification unit comprises an AI identification server, and the AI identification server is used for receiving the scene image and generating a full scene three-dimensional space; wherein the content of the first and second substances,
the AI recognition server performs the following operations:
importing the received scene image into the AI analysis model to obtain image elements;
importing the image elements into an AI quantitative model to generate a corresponding quantitative analysis mode; in
The quantitative analysis mode at least comprises ground analysis, traffic signal analysis and service type analysis;
and importing the quantitative analysis mode into the AI mode recognition model, and determining historical driving data and real-time driving data according to the quantitative analysis mode by the AI mode recognition model.
As an embodiment of the present invention, the AR remote control module further includes:
a path optimization unit: the real-time simulation video is used for optimizing the running track of the agricultural tractor according to the real-time simulation video; wherein the content of the first and second substances,
the running track optimization step of the agricultural tractor comprises the following steps:
acquiring an initial simulation track path, reducing the error rate and the convergence rate of the simulation track path based on multi-model control, and determining a first simulation optimization track path; wherein the content of the first and second substances,
the multi-model control comprises gain control, sliding mode control and artificial intelligence control;
setting an expected change value of the error rate change, and judging whether the difference value between the change value of the error rate and the expected change value is higher than a preset expected change threshold value or not;
when the variation value of the error rate is larger than the preset expected variation threshold value, stabilizing the variation value of the error rate in a differential control mode, and determining a target simulation optimization track path;
when the variation value of the error rate is smaller than the preset expected variation threshold, firstly adopting proportional model control to increase the variation value of the error rate, and when the variation value of the error rate is larger than the preset expected variation threshold, adopting a differential control mode to stabilize the variation value of the error rate and determining a target simulation optimization track path.
As an embodiment of the present invention, the manner of the differential control includes:
wherein, the
Representing a preset expected change threshold; the above-mentioned
Is shown as
A variation value of an error rate at a time; the above-mentioned
An error coefficient representing an error rate; the above-mentioned
An actual variation value representing the error rate;
when in use
When the error rate is high, the variation value indicating the error rate is stable;
the proportional model control includes: determining the parameters of the influence of the gain control on the variation of the error rate
Sliding mode control standThe influence parameter of the variation value of the error rate
And controlling the parameters influencing the variation value of the error rate by artificial intelligence
And constructing a proportion control model:
adjusting an influence parameter of the gain control on a variation value of the error rate according to the proportional control model
Influence parameter of sliding mode control on variation value of error rate
And controlling the parameters influencing the variation value of the error rate by artificial intelligence
When said
Is greater than
It is indicated that the variation value of the error rate can be stabilized by means of the differential control.
The invention has the beneficial effects that: according to the method, through scene recognition, the driving image is a comprehensive image in six directions, so that scene elements of the agricultural tractor can be comprehensively acquired, the driving scene is determined, and the running environment is comprehensively monitored. The state monitoring aims at monitoring the real-time state of the agricultural tractor and judging the body data of the agricultural tractor. During data transmission, different data transmission modes are adopted according to the number of different agricultural tractors, and the comprehensiveness of the efficiency of data transmission is improved conveniently. And (4) simulating a cloud scene. The real state of the agricultural tractor can be reflected based on the cloud big data technology, and then the real-time state of the agricultural tractor is remotely observed through the AR far-end control module, real-time monitoring is carried out, and real-time remote control is realized.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and drawings.
The technical solution of the present invention is further described in detail by the accompanying drawings and embodiments.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the principles of the invention and not to limit the invention.
In the drawings:
fig. 1 is a system composition diagram of a farm tractor remote control system based on virtual reality technology in an embodiment of the present invention.
Detailed Description
The preferred embodiments of the present invention will be described in conjunction with the accompanying drawings, and it will be understood that they are described herein for the purpose of illustration and explanation and not limitation.
As shown in fig. 1, a virtual reality technology-based agricultural tractor remote control system includes:
a scene recognition module: the system comprises a driving image acquisition module, a driving scene determination module and a driving scene determination module, wherein the driving image acquisition module is used for acquiring a driving image of an operating agricultural tractor, extracting image elements and determining the driving scene; the driving image is a comprehensive drawing of six directions, so that a driving scene can be comprehensively determined, and the driving scene at least comprises a comprehensive drawing of five directions of the agricultural tractor. Image elements such as the ground, the presence of dirt, cement, asphalt, and non-road lawns, land, mountainous terrain, among other scene elements include, among others, obstacle elements such as people, stones, trees, poles, traffic signs, etc. near agricultural tractors.
A state monitoring module: the agricultural tractor is used for determining a driving state through a sensing device arranged on the agricultural tractor; the method is mainly used for acquiring the state of the tractor, such as the temperature state, the humidity state, the diesel oil level state, the speed state and the like.
A data transmission module: the system comprises a data transmission mode determining module, a cloud scene simulation module and a driving state simulation module, wherein the data transmission mode is used for transmitting the driving scene and the driving state to the cloud scene simulation module according to the data transmission mode; the data transmission module is mainly used for single-channel data transmission when only one agricultural tractor is available, and a multi-channel and multi-channel synchronous data transmission mode is adopted when a plurality of agricultural tractors are available.
Cloud scene simulation module: the simulation system is used for constructing a simulation scene of the agricultural tractor according to the driving scene and the driving state and generating a real-time simulation video; the method is mainly used for simulating the operation scene of the agricultural tractor, and further the simulated state video of the agricultural tractor comprises a state track and a scene page.
The AR far-end control module: and the real-time simulation video is displayed through the AR equipment, simulation parameters are determined, the running scene of the agricultural tractor is optimized according to the simulation parameters, and a control instruction is generated. The method is used for predicting and optimizing the predicted behavior of the agricultural tractor according to the simulation video and providing the optimal predicted behavior based on the business problem to be solved.
The invention has the beneficial effects that: according to the method, through scene recognition, the driving image is a comprehensive image in six directions, so that scene elements of the agricultural tractor can be comprehensively acquired, the driving scene is determined, and the running environment is comprehensively monitored. The state monitoring aims at monitoring the real-time state of the agricultural tractor and judging the body data of the agricultural tractor. During data transmission, different data transmission modes are adopted according to the number of different agricultural tractors, and the comprehensiveness of the efficiency of data transmission is improved conveniently. And (4) simulating a cloud scene. The real state of the agricultural tractor can be reflected based on the cloud big data technology, and then the real-time state of the agricultural tractor is remotely observed through the AR far-end control module, real-time monitoring is carried out, and real-time remote control is realized.
As an embodiment of the present invention, a scene recognition module includes:
an image pickup unit: the image acquisition device is used for acquiring a scene image through the image acquisition device arranged on the agricultural tractor; wherein the content of the first and second substances,
the number of the camera devices is not less than 5; wherein the content of the first and second substances,
at least 2 camera devices with 120-degree camera angles are arranged on two sides of the tail of the agricultural tractor;
at least 2 camera devices with 180-degree camera angles are arranged on two sides of the middle part of the agricultural tractor;
at least 1 camera device with a 180-degree camera angle is arranged on two sides of the agricultural tractor head; the camera device can be a snapshot camera, a video camera, or the like; not less than five, in order to carry out comprehensive control to the tractor. Each camera shooting device and the adjacent device have overlapped parts at the set camera shooting angle, so that a camera shooting picture cannot be lacked, and image elements cannot be lost.
An element extraction unit: the image processing device is used for judging whether image elements in the scene image influence the driving behavior of the agricultural tractor or not according to the scene image, and extracting the image elements when the image elements influence the driving behavior of the agricultural tractor; the influence is defined as positive or negative influence on the behavior of the agricultural tractor, under which the image element is determined.
The image elements include road surface elements, obstacle elements, traffic indication elements, and weather elements;
a scene determination unit: the image element is used for comparing with a historical driving scene to determine the driving scene; where historical driving scenarios, i.e. similar situations have been encountered, there are many of the same situations in the agricultural field.
And when the image element does not exist in the historical driving scene, transmitting the image element to a user terminal, determining the driving scene, and storing the driving scene. The user terminal is a behavior for manually judging a driving scene, so that when a machine cannot identify the driving scene, the identification behavior cannot influence the determination of the scene.
As an embodiment of the present invention, the state monitoring module includes:
a sensor unit: the agricultural tractor is used for acquiring state data according to preset sensing equipment on the agricultural tractor; wherein, the sensor is mainly used for obtaining the state of the agricultural tractor.
The sensing device includes: temperature sensing equipment, speed sensing equipment, a position sensor, a liquid level sensor, an energy consumption sensor, a speed sensor, an acceleration sensor, a ray radiation sensor, a thermosensitive sensor, a vibration sensor and a humidity sensor;
a signal strength unit: the system comprises an AR remote control module, a verification data acquisition module, a signal processing module and a signal processing module, wherein the AR remote control module is used for acquiring the verification data of the agricultural tractor and the signal processing module; the method comprises the steps of representing that preset verification data are repeatedly transmitted in a time period, transmitting a time back and forth according to a transmission distance of the verification data transmission on a time axis, and judging the signal strength according to the time, so that the signal strength can be improved under the condition of low signal strength.
A data processing unit: the system is used for determining the correlation among the state data and determining a dynamic state model according to the correlation; for determining relationships between the collected data and generating a dynamic state model representing the operating state of the agricultural tractor.
Threshold unit: the device parameter setting module is used for acquiring the device parameters of the agricultural tractor and setting a threshold model according to the device parameters; any equipment has a limit value, so that the state threshold value of the agricultural tractor can be judged conveniently and visually, and the agricultural tractor can be adjusted.
A state judgment unit: the dynamic state model is used for comparing the threshold model with the dynamic state model, determining the driving difference, calculating the state loss value of the agricultural tractor under the driving difference according to the signal intensity, and determining the driving state. The method is mainly used for judging whether the agricultural tractor is in a normal running state or has a fault, and is convenient for predicting the fault of the agricultural tractor.
As an embodiment of the present invention, a data transmission module includes:
a transmission path judgment unit: the system is used for acquiring a data transmission mode and verifying whether a data transmission channel is synchronous transmission or not; the synchronous transmission means that the monitoring information and the state of the agricultural tractor observed by the user through the AR equipment present a synchronous state, what behavior is actually on site, and what behavior is displayed by the AR equipment.
A mode setting unit: for setting a data transmission mode according to the strength of the signal strength in the driving scene, wherein,
the data transmission single-channel data transmission mode and the multi-channel data transmission mode;
the method is used for preventing data transmission from being untimely, reducing communication resource waste through single-path data transmission when the signal is strong, and preventing data loss through multi-path synchronous transmission and data complementation when the signal is weak.
A mode selection unit: and the data transmission mode is determined according to the priority.
As an embodiment of the present invention, the cloud scene simulation module includes:
a data acquisition unit: the system comprises a driving scene and a driving state, wherein the driving scene and the driving state are used for determining scene data and state data according to the driving scene and the driving state; scene data is mainly data of elements in a scene, such as: data of obstacles (stones, people and other obstacles on the road). The state data mainly comprise operating state parameters of the agricultural tractor, such as: velocity data, temperature data, liquid level data, and the like.
A scene processing unit: the scene simulation system is used for establishing a scene simulation model by taking the scene data as a data source and determining a scene video; a scene framework is used for the scene simulation model and is based on simulation software in the prior art.
A state processing unit: the state data is used as a data source and is substituted into a preset simulated agricultural tractor to determine a state video;
a fusion unit: the system is used for fusing the state video into a scene video and determining a simulation video;
a real-time processing unit: the real-time simulation video fusion unit is used for determining difference data according to the data acquisition unit and the fusion unit, substituting the difference data into the simulation video according to the type of the difference data, and determining the real-time simulation video.
The beneficial effects of the above technical scheme are that: through the integration of state video and scene video, the action of the remote monitoring agricultural tractor of definite agricultural tractor that can be comprehensive does not have the data disappearance moreover, can also let the long-range real-time behavior state of personally observing agricultural tractor of user, the manual control of being convenient for.
As an embodiment of the present invention, the AR remote control module includes:
an apparatus control unit: the system is used for acquiring the simulation video and pushing the simulation video to the AR equipment; AR devices, such as AR helmets or AR eyes, facilitate the user's observation.
A parameter extraction unit: the simulation system is used for determining a simulation program package and a simulation language according to the simulation video, determining a simulation object parameter of the agricultural tractor according to the simulation program package, and determining a simulation structure parameter of the agricultural tractor according to the simulation language; the simulation package determines the logic of the guidelines and the simulated data parameters. The simulation language determines the technical architecture at the time of simulation.
An apparatus manipulation unit: the simulation system is used for generating an object regulation and control parameter according to the simulation object parameter, generating an object regulation and control window according to the simulation structure parameter and the object regulation and control parameter, and receiving a first regulation and control action input by a user; the first regulation and control behavior is mainly the regulation and control behavior of the user, and the priority is higher.
An autonomous regulation unit: the ant colony optimization algorithm is used for processing the simulation object parameters to obtain autonomous regulation and control parameters and generate a second regulation and control behavior; after the user regulates and controls, a regulation and control reference is determined, and the regulation and control instruction is optimized through an ant colony optimization algorithm to determine the optimal regulation and control mode.
An instruction generation unit: the regulating and controlling intention is judged according to the first regulating and controlling behavior and the second regulating and controlling behavior, and a regulating and controlling instruction is generated; wherein the content of the first and second substances,
the first regulatory behavior is higher in priority than the second regulatory behavior;
an instruction implementation unit: and the control instruction is sent to the agricultural tractor, control information is determined, and control operation is executed. The agricultural tractor is provided with signal receiving and identifying equipment in advance, and can analyze and clearly control information according to a control instruction so as to implement implementation.
As an embodiment of the present invention, the signal strength determining unit determines the signal strength by:
step 1: determining a channel feature set and a base station feature set between the agricultural tractor and the AR remote control module based on the agricultural tractor and the AR remote control module
：
Wherein, the
Is shown as
A plurality of channels; the above-mentioned
Is shown as
Characteristics of individual base stations; the above-mentioned
Is shown as having
A channel, in common
A base station;
step 2: substituting the channel characteristics and the base station characteristics into a relation model to determine the implementation probability of any channel and any base station
：
Wherein, the
Representing a mean value of channel characteristics; the above-mentioned
Representing and calculating a characteristic mean value of the base station; the above-mentioned
Is shown as
A channel and a
Implementation probability of each base station;
and step 3: determining the implementation capability of the channel according to the implementation probability
：
Wherein, the
Is shown as
Transmission capacity of individual channels
；
And 4, step 4: constructing a strength model according to the implementation capacity
：
Wherein, the
Represents the signal strength;
setting intensity threshold
When said
When, it indicates that the signal strength is strong; when said
When it is, the signal strength is weak.
The principle and the beneficial effects of the technical scheme are as follows: the method is determined according to the data of the base station and the channel between the agricultural tractor and the AR remote control module on the basis of the judgment of the signal intensity. The base station has base stations of different operators, but the channels have no difference between the operators, and only have difference between channel stability and transmission quantity. The invention determines the implementation probability when each channel is matched with the base station based on the relation model through the channel characteristic set and the base station characteristic set, further determines the implementation capability according to the transmission capability of the channel, and finally substitutes the implementation capability into the strength model of the signal strength to determine the final signal strength.
As an embodiment of the present invention, the scene recognition module further includes a cloud control unit and an AI recognition unit; wherein
The cloud control unit comprises a cloud server, and the cloud server is used for constructing an AI quantitative model, an AI mode identification model and an AI analysis model in the AI identification unit through a big data technology and a general AI model; the cloud server is also used for constructing a special communication channel between the AI identification unit and the agricultural tractor through a cloud network; the method realizes intelligent and accurate analysis and generation of scene data through three steps of isomorphic accurate analysis, quantitative analysis and instruction generation of the constructed AI quantitative model, the AI mode recognition model and the AI analysis model.
The AI identification unit comprises an AI identification server, and the AI identification server is used for receiving the scene image and generating a full scene three-dimensional space; wherein the content of the first and second substances,
the AI recognition server performs the following operations:
importing the received scene image into the AI analysis model to obtain image elements;
importing the image elements into an AI quantitative model to generate a corresponding quantitative analysis mode; in
The quantitative analysis mode at least comprises ground analysis, traffic signal analysis and service type analysis;
and importing the quantitative analysis mode into the AI mode recognition model, and determining historical driving data and real-time driving data according to the quantitative analysis mode by the AI mode recognition model. According to the method, the driving data are accurately converted from the assumed image through an AI technology intelligent identification technology and a big data technology data analysis and calculation function, historical data are extracted, and intelligent judgment of scenes is achieved.
As an embodiment of the present invention, the AR remote control module further includes:
a path optimization unit: the real-time simulation video is used for optimizing the running track of the agricultural tractor according to the real-time simulation video; wherein the content of the first and second substances,
the step of optimizing the running track of the agricultural tractor comprises the following steps:
acquiring an initial simulation track path, reducing the error rate and the convergence rate of the simulation track path based on multi-model control, and determining a first simulation optimization track path; wherein the content of the first and second substances,
the multi-model control comprises gain control, sliding mode control and artificial intelligence control;
the error rate of the simulation track path is reduced through various control models during multi-model control, and the convergence rate is reduced through an error iteration adjustment sequence.
Setting an expected change value of the error rate change, and judging whether the difference value between the change value of the error rate and the expected change value is higher than a preset expected change threshold value or not; and the expected variation value of the error rate is a preset variation condition threshold value.
When the variation value of the error rate is larger than the preset expected variation threshold value, stabilizing the variation value of the error rate in a differential control mode, and determining a target simulation optimization track path;
the differential control is performed to stabilize the change value of the error after the optimization. The proportional model is used for increasing the change value of the error so as to further realize the regulation and control of the error change rate, and when the change rate of the error is larger, the error is easier to regulate; because the invention constructs the adjustment model based on the variation value of the error rate, the smaller the variation value is, the more difficult the adjustment is, and the larger the variation value is, the more the model in accordance with the invention is, the optimization adjustment is convenient.
When the variation value of the error rate is smaller than the preset expected variation threshold, firstly adopting proportional model control to increase the variation value of the error rate, and when the variation value of the error rate is larger than the preset expected variation threshold, adopting a differential control mode to stabilize the variation value of the error rate and determining a target simulation optimization track path.
As an embodiment of the present invention, the manner of the differential control includes:
wherein, the
Representing a preset expected change threshold; the above-mentioned
Is shown as
A variation value of an error rate at a time; the above-mentioned
An error coefficient representing an error rate (i.e., an error coefficient between the actual agricultural tractor and the simulated agricultural tractor); the above-mentioned
An actual variation value representing the error rate;
when in use
When the error rate is high, the variation value indicating the error rate is stable;
the proportional model control includes: determining the parameters of the influence of the gain control on the variation of the error rate
Influence parameter of sliding mode control on variation value of error rate
And controlling the parameters influencing the variation value of the error rate by artificial intelligence
And constructing a proportion control model:
adjusting an influence parameter of the gain control on a variation value of the error rate according to the proportional control model
Influence parameter of sliding mode control on variation value of error rate
And controlling the parameters influencing the variation value of the error rate by artificial intelligence
When said
Is greater than
It is indicated that the variation value of the error rate can be stabilized by means of the differential control.
In the above technical solution, the differential control mode is mainly an optimization control, mainly for adjusting the variation value of the error rate, and the proportional control model is for adjusting the variation value of the error rate to a condition that can be adjusted and controlled.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present invention without departing from the spirit and scope of the invention. Thus, if such modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include such modifications and variations.
Claims (6)
1. A farm tractor remote control system based on virtual reality technology, characterized by comprising:
a scene recognition module: the system comprises a driving image acquisition module, a driving scene determination module and a driving scene determination module, wherein the driving image acquisition module is used for acquiring a driving image of an operating agricultural tractor, extracting image elements and determining the driving scene;
a state monitoring module: the agricultural tractor is used for determining a driving state through a sensing device arranged on the agricultural tractor;
a data transmission module: the system comprises a data transmission mode determining module, a cloud scene simulation module and a driving state simulation module, wherein the data transmission mode is used for transmitting the driving scene and the driving state to the cloud scene simulation module according to the data transmission mode;
cloud scene simulation module: the simulation system is used for constructing a simulation scene of the agricultural tractor according to the driving scene and the driving state and generating a real-time simulation video;
the AR far-end control module: the real-time simulation video is displayed through AR equipment, simulation parameters are determined, the running scene of the agricultural tractor is optimized according to the simulation parameters, and a control instruction is generated; wherein the content of the first and second substances,
the AR remote control module comprises:
an apparatus control unit: the system is used for acquiring the simulation video and pushing the simulation video to the AR equipment;
a parameter extraction unit: the simulation system is used for determining a simulation program package and a simulation language according to the simulation video, determining a simulation object parameter of the agricultural tractor according to the simulation program package, and determining a simulation structure parameter of the agricultural tractor according to the simulation language;
an apparatus manipulation unit: the simulation system is used for generating an object regulation and control parameter according to the simulation object parameter, generating an object regulation and control window according to the simulation structure parameter and the object regulation and control parameter, and receiving a first regulation and control action input by a user;
an autonomous regulation unit: the ant colony optimization algorithm is used for processing the simulation object parameters to obtain autonomous regulation and control parameters and generate a second regulation and control behavior;
an instruction generation unit: the regulating and controlling intention is judged according to the first regulating and controlling behavior and the second regulating and controlling behavior, and a regulating and controlling instruction is generated; wherein the content of the first and second substances,
the first regulatory behavior is higher in priority than the second regulatory behavior;
an instruction implementation unit: the control instruction is sent to the agricultural tractor, control information is determined, and control operation is executed;
the state monitoring module includes:
a sensor unit: the agricultural tractor is used for acquiring state data according to preset sensing equipment on the agricultural tractor; wherein the content of the first and second substances,
the sensing device includes: temperature sensing equipment, speed sensing equipment, a position sensor, a liquid level sensor, an energy consumption sensor, a speed sensor, an acceleration sensor, a ray radiation sensor, a thermosensitive sensor, a vibration sensor and a humidity sensor;
a signal strength unit: the system comprises an AR remote control module, a verification data acquisition module, a signal processing module and a signal processing module, wherein the AR remote control module is used for acquiring the verification data of the agricultural tractor and the signal processing module;
a data processing unit: the system is used for determining the correlation among the state data and determining a dynamic state model according to the correlation;
threshold unit: the device parameter setting module is used for acquiring the device parameters of the agricultural tractor and setting a threshold model according to the device parameters;
a state judgment unit: the dynamic state model is used for comparing the threshold model with the dynamic state model, determining the driving difference, calculating the state loss value of the agricultural tractor under the driving difference according to the signal intensity, and determining the driving state.
2. An agricultural tractor remote control system based on virtual reality technology according to claim 1, characterized in that the scene recognition module includes:
an image pickup unit: the image acquisition device is used for acquiring a scene image through the image acquisition device arranged on the agricultural tractor; wherein the content of the first and second substances,
the number of the camera devices is not less than 5; wherein the content of the first and second substances,
at least 2 camera devices with 120-degree camera angles are arranged on two sides of the tail of the agricultural tractor;
at least 2 camera devices with 180-degree camera angles are arranged on two sides of the middle part of the agricultural tractor;
at least 1 camera device with a 180-degree camera angle is arranged on two sides of the agricultural tractor head;
an element extraction unit: the image processing device is used for judging whether image elements in the scene image influence the driving behavior of the agricultural tractor or not according to the scene image, and extracting the image elements when the image elements influence the driving behavior of the agricultural tractor;
the image elements include road surface elements, obstacle elements, traffic indication elements, and weather elements;
a scene determination unit: the image element is used for comparing with a historical driving scene to determine the driving scene; wherein the content of the first and second substances,
and when the image element does not exist in the historical driving scene, transmitting the image element to a user terminal, determining the driving scene, and storing the driving scene.
3. An agricultural tractor remote control system based on virtual reality technology as claimed in claim 1, characterized in that, the data transmission module includes:
a transmission path judgment unit: the system is used for acquiring a data transmission mode and verifying whether a data transmission channel is synchronous transmission or not;
a mode setting unit: for setting a data transmission mode according to the strength of the signal strength in the driving scene, wherein,
the data transmission modes include: a single-channel data transmission mode and a multi-channel data transmission mode;
a mode selection unit: and the data transmission mode is determined according to the priority.
4. The virtual reality technology-based agricultural tractor remote control system of claim 1, wherein the cloud scene simulation module comprises:
a data acquisition unit: the system comprises a driving scene and a driving state, wherein the driving scene and the driving state are used for determining scene data and state data according to the driving scene and the driving state;
a scene processing unit: the scene simulation system is used for establishing a scene simulation model by taking the scene data as a data source and determining a scene video;
a state processing unit: the state data is used as a data source and is substituted into a preset simulated agricultural tractor to determine a state video;
a fusion unit: the system is used for fusing the state video into a scene video and determining a simulation video;
a real-time processing unit: the real-time simulation video fusion unit is used for determining difference data according to the data acquisition unit and the fusion unit, substituting the difference data into the simulation video according to the type of the difference data, and determining the real-time simulation video.
5. An agricultural tractor remote control system based on virtual reality technology according to claim 1, characterized in that the scene recognition module comprises a cloud control unit and an AI recognition unit; wherein the content of the first and second substances,
the cloud control unit comprises a cloud server, and the cloud server is used for constructing an AI quantitative model, an AI mode identification model and an AI analysis model in the AI identification unit through a big data technology and a general AI model; the cloud server is also used for constructing a special communication channel between the AI identification unit and the agricultural tractor through a cloud network;
the AI identification unit comprises an AI identification server, and the AI identification server is used for receiving the scene image and generating a full scene three-dimensional space; wherein the content of the first and second substances,
the AI recognition server performs the following operations:
importing the received scene image into the AI analysis model to obtain image elements;
importing the image elements into an AI quantitative model and generating a corresponding quantitative analysis mode; wherein the content of the first and second substances,
the quantitative analysis mode at least comprises ground analysis, traffic signal analysis and service type analysis;
and importing the quantitative analysis mode into the AI mode recognition model, and determining historical driving data and real-time driving data according to the quantitative analysis mode by the AI mode recognition model.
6. An agricultural tractor remote control system based on virtual reality technology as claimed in claim 1, wherein the AR remote control module further comprises:
a path optimization unit: the real-time simulation video is used for optimizing the running track of the agricultural tractor according to the real-time simulation video; wherein the content of the first and second substances,
the running track optimization step of the agricultural tractor comprises the following steps:
acquiring an initial simulation track path, reducing the error rate and the convergence rate of the simulation track path based on multi-model control, and determining a first simulation optimization track path; wherein the content of the first and second substances,
the multi-model control comprises gain control, sliding mode control and artificial intelligence control;
setting an expected change value of the error rate change, and judging whether the difference value between the change value of the error rate and the expected change value is higher than a preset expected change threshold value or not;
when the variation value of the error rate is larger than the preset expected variation threshold value, stabilizing the variation value of the error rate in a differential control mode, and determining a target simulation optimization track path;
when the variation value of the error rate is smaller than the preset expected variation threshold, firstly adopting proportional model control to increase the variation value of the error rate, and when the variation value of the error rate is larger than the preset expected variation threshold, adopting a differential control mode to stabilize the variation value of the error rate and determining a target simulation optimization track path.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010735665.2A CN111866464B (en) | 2020-07-28 | 2020-07-28 | Agricultural tractor remote control system based on virtual reality technology |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010735665.2A CN111866464B (en) | 2020-07-28 | 2020-07-28 | Agricultural tractor remote control system based on virtual reality technology |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111866464A CN111866464A (en) | 2020-10-30 |
| CN111866464B true CN111866464B (en) | 2021-08-24 |
Family
ID=72947853
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010735665.2A Active CN111866464B (en) | 2020-07-28 | 2020-07-28 | Agricultural tractor remote control system based on virtual reality technology |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111866464B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112883501A (en) * | 2021-04-06 | 2021-06-01 | 河南科技大学 | Method for constructing digital research and development platform system architecture of tractor |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106710360A (en) * | 2016-02-03 | 2017-05-24 | 北京易驾佳信息科技有限公司 | Intelligent driving training system and method based on augment virtual reality man-machine interaction |
| CN107505944A (en) * | 2017-09-27 | 2017-12-22 | 驭势科技（北京）有限公司 | A kind of method and apparatus for being used to carry out vehicle remote assistance |
| CN108797669A (en) * | 2018-06-20 | 2018-11-13 | 清华大学 | A kind of autonomous 3D excavations construction robot |
| CN109003335A (en) * | 2018-07-05 | 2018-12-14 | 上海思致汽车工程技术有限公司 | A kind of virtual driving sensory perceptual system and its scene generating method based on VR technology |
| WO2019112799A1 (en) * | 2017-12-07 | 2019-06-13 | Steeringz, Inc. | Improving safety of autonomous vehicles using a virtual augmented support environment |
| CN110103221A (en) * | 2019-05-21 | 2019-08-09 | 深圳市超时空机器人有限公司 | A kind of long-range drive manner, equipment and its system |
| CN110176159A (en) * | 2019-05-28 | 2019-08-27 | 北京精英智通科技股份有限公司 | A kind of display systems and host of traffic accident case |
| CN110239563A (en) * | 2019-04-27 | 2019-09-17 | 深圳市赛亿科技开发有限公司 | A kind of long-range driving control method and system |
| CN110262469A (en) * | 2019-07-23 | 2019-09-20 | 江苏常发农业装备股份有限公司 | A kind of automatic ride control system and method applied to agricultural vehicle |
| CN111045425A (en) * | 2019-12-05 | 2020-04-21 | 中国北方车辆研究所 | Auxiliary teleoperation driving method for ground unmanned vehicle |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7920071B2 (en) * | 2006-05-26 | 2011-04-05 | Itt Manufacturing Enterprises, Inc. | Augmented reality-based system and method providing status and control of unmanned vehicles |
| US10936050B2 (en) * | 2014-06-16 | 2021-03-02 | Honda Motor Co., Ltd. | Systems and methods for user indication recognition |
| US11669653B2 (en) * | 2017-05-02 | 2023-06-06 | The Regents Of The University Of Michigan | Simulated vehicle traffic for autonomous vehicles |
| CN109034639B (en) * | 2018-08-09 | 2021-03-23 | 北京智行者科技有限公司 | Scheduling control method |
| CN110047150B (en) * | 2019-04-24 | 2023-06-16 | 大唐环境产业集团股份有限公司 | Complex equipment operation on-site simulation system based on augmented reality |
-
2020
- 2020-07-28 CN CN202010735665.2A patent/CN111866464B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106710360A (en) * | 2016-02-03 | 2017-05-24 | 北京易驾佳信息科技有限公司 | Intelligent driving training system and method based on augment virtual reality man-machine interaction |
| CN107505944A (en) * | 2017-09-27 | 2017-12-22 | 驭势科技（北京）有限公司 | A kind of method and apparatus for being used to carry out vehicle remote assistance |
| WO2019112799A1 (en) * | 2017-12-07 | 2019-06-13 | Steeringz, Inc. | Improving safety of autonomous vehicles using a virtual augmented support environment |
| CN108797669A (en) * | 2018-06-20 | 2018-11-13 | 清华大学 | A kind of autonomous 3D excavations construction robot |
| CN109003335A (en) * | 2018-07-05 | 2018-12-14 | 上海思致汽车工程技术有限公司 | A kind of virtual driving sensory perceptual system and its scene generating method based on VR technology |
| CN110239563A (en) * | 2019-04-27 | 2019-09-17 | 深圳市赛亿科技开发有限公司 | A kind of long-range driving control method and system |
| CN110103221A (en) * | 2019-05-21 | 2019-08-09 | 深圳市超时空机器人有限公司 | A kind of long-range drive manner, equipment and its system |
| CN110176159A (en) * | 2019-05-28 | 2019-08-27 | 北京精英智通科技股份有限公司 | A kind of display systems and host of traffic accident case |
| CN110262469A (en) * | 2019-07-23 | 2019-09-20 | 江苏常发农业装备股份有限公司 | A kind of automatic ride control system and method applied to agricultural vehicle |
| CN111045425A (en) * | 2019-12-05 | 2020-04-21 | 中国北方车辆研究所 | Auxiliary teleoperation driving method for ground unmanned vehicle |
Non-Patent Citations (1)
| Title |
|---|
| "基于模糊加权的多模态切换控制器设计与仿真";朱华勇，张庆杰，沈林成;《系统仿真学报》;20070930;第19卷(第18期);第4212-4215页全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111866464A (en) | 2020-10-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108694367B (en) | Method, device and system for establishing driving behavior model | |
| US9043129B2 (en) | Method for governing a speed of an autonomous vehicle | |
| CN110458032B (en) | Whole-process litchi growth condition monitoring method and system, cloud server and storage medium | |
| CN113920474B (en) | Internet of things system and method for intelligently supervising citrus planting situation | |
| CN113283324B (en) | Forest fire prevention early warning method and system based on dynamic image | |
| CN111866464B (en) | Agricultural tractor remote control system based on virtual reality technology | |
| CN104123847A (en) | ACP (artificial societies, computational experiments and parallel execution) method-based traffic signal recommendation system and corresponding method | |
| CN115546672A (en) | Forest picture processing method and system based on image processing | |
| Petkovic et al. | IoT devices VS. drones for data collection in agriculture | |
| CN112817331A (en) | Intelligent forestry information monitoring system based on multi-machine cooperation | |
| CN114115020A (en) | Intelligent control system and control method for height of unmanned aerial vehicle | |
| CN117193347B (en) | Unmanned aerial vehicle flight height control method and device, electronic equipment and storage medium | |
| CN116757509B (en) | Information service management method and management system for ecological agriculture informatization | |
| CN113349188A (en) | Lawn and forage grass precise weeding method based on cloud weeding spectrum | |
| CN112509321A (en) | Unmanned aerial vehicle-based driving control method and system for urban complex traffic situation and readable storage medium | |
| CN116739739A (en) | Loan amount evaluation method and device, electronic equipment and storage medium | |
| CN114545833B (en) | Intelligent interactive processing system of facility agriculture based on internet of things | |
| CN110781865A (en) | Crop growth control system | |
| WO2022131176A1 (en) | Control device, program, system and method | |
| CN115115275A (en) | AI-based unattended crop disease and pest monitoring and early warning system | |
| CN114091774A (en) | Crop yield estimation method and device | |
| CN113159644A (en) | Forestry management system and method based on unmanned aerial vehicle remote sensing technology and GIS data processing | |
| CN113705282A (en) | Information acquisition device, system and method and information acquisition vehicle | |
| CN117197701B (en) | Method and system for analyzing environmental spatial features of protected area based on image analysis | |
| CN116295662B (en) | Crop growth state monitoring method and device, electronic equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |