CN115328131A - Obstacle avoidance method and device for unmanned ship, unmanned ship and storage medium - Google Patents
Obstacle avoidance method and device for unmanned ship, unmanned ship and storage medium Download PDFInfo
- Publication number
- CN115328131A CN115328131A CN202210969477.5A CN202210969477A CN115328131A CN 115328131 A CN115328131 A CN 115328131A CN 202210969477 A CN202210969477 A CN 202210969477A CN 115328131 A CN115328131 A CN 115328131A
- Authority
- CN
- China
- Prior art keywords
- obstacle
- unmanned ship
- determining
- state information
- underwater
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 66
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 claims abstract description 81
- 230000000007 visual effect Effects 0.000 claims abstract description 27
- 238000004590 computer program Methods 0.000 claims description 7
- 238000006073 displacement reaction Methods 0.000 claims description 7
- 238000004891 communication Methods 0.000 claims description 3
- 239000000126 substance Substances 0.000 claims 1
- 238000012544 monitoring process Methods 0.000 abstract description 6
- 238000001514 detection method Methods 0.000 description 16
- 238000005259 measurement Methods 0.000 description 16
- 238000012545 processing Methods 0.000 description 8
- 230000008569 process Effects 0.000 description 7
- 238000012549 training Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 5
- 238000013473 artificial intelligence Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 230000004888 barrier function Effects 0.000 description 3
- 230000004927 fusion Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000007613 environmental effect Effects 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 229910000831 Steel Inorganic materials 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 239000003814 drug Substances 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 238000007667 floating Methods 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 230000002427 irreversible effect Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 239000010959 steel Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
- G05D1/02—Control of position or course in two dimensions
- G05D1/0206—Control of position or course in two dimensions specially adapted to water vehicles
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S15/00—Systems using the reflection or reradiation of acoustic waves, e.g. sonar systems
- G01S15/86—Combinations of sonar systems with lidar systems; Combinations of sonar systems with systems not using wave reflection
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S15/00—Systems using the reflection or reradiation of acoustic waves, e.g. sonar systems
- G01S15/88—Sonar systems specially adapted for specific applications
- G01S15/93—Sonar systems specially adapted for specific applications for anti-collision purposes
Landscapes
- Engineering & Computer Science (AREA)
- Radar, Positioning & Navigation (AREA)
- Remote Sensing (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Aviation & Aerospace Engineering (AREA)
- Automation & Control Theory (AREA)
- Acoustics & Sound (AREA)
- Measurement Of Velocity Or Position Using Acoustic Or Ultrasonic Waves (AREA)
Abstract
The embodiment of the invention provides an obstacle avoidance method and device for an unmanned ship, the unmanned ship and a storage medium, wherein the method is applied to the unmanned ship, a camera and a sonar sensor are installed on the unmanned ship, and the method comprises the following steps: acquiring an overwater image frame and underwater scanning data which are synchronously acquired by the camera and the sonar sensor in the current area; determining obstacles in the current area and current state information related to the obstacles according to the overwater image frames and the underwater scanning data; and controlling the unmanned ship to avoid the obstacle by combining a preset obstacle avoidance strategy according to the current state information. By utilizing the method, the visual camera and the sonar sensor are combined to collect multi-source data, the identification and monitoring of the obstacles in the above-water and underwater modes can be realized, and compared with the prior art, the obstacle identification preparation rate can be improved, the error identification is reduced, and the adaptability of an operation scene is improved.
Description
Technical Field
The invention relates to the technical field of unmanned ships, in particular to an obstacle avoidance method and device for an unmanned ship, the unmanned ship and a storage medium.
Background
The unmanned ship is an intelligent, motorized, unmanned and networked water surface vehicle, and plays an important role in the fields of marine medicine, maritime patrol and the like. The unmanned ship can meet various obstacles in the process of running on the water surface, and irreversible danger can be caused to the unmanned ship if the unmanned ship can not avoid the obstacles in the water in time.
The existing unmanned ship can detect the obstacle by adopting a millimeter wave radar, but the mode has the advantages of short effective measuring range, complex data processing and high hardware cost, and can only acquire distance information for the front object, and the object cannot be sensed by information such as whether the object is the obstacle or only a floater. In addition, effective distance measurement cannot be realized on the ship by adopting a monocular visual angle identification mode, and the effective distance measurement is shorter by adopting a binocular visual angle identification mode.
Disclosure of Invention
The embodiment of the invention provides an obstacle avoidance method and device for an unmanned ship, the unmanned ship and a storage medium, which are used for realizing the identification and monitoring of obstacles on water and under water in two modes, improving the identification preparation rate of the obstacles, reducing the false identification and improving the adaptability of an operation scene.
In a first aspect, the present embodiment provides an obstacle avoidance method for an unmanned ship, which is applied to the unmanned ship, where a camera and a sonar sensor are installed on the unmanned ship, and the method includes:
acquiring an overwater image frame and underwater scanning data which are synchronously acquired by the camera and the sonar sensor in the current area;
determining obstacles in the current area and current state information related to the obstacles according to the overwater image frames and the underwater scanning data;
and controlling the unmanned ship to avoid the obstacle by combining a preset obstacle avoidance strategy according to the current state information.
In a second aspect, this embodiment provides an obstacle avoidance device of unmanned ship, integrate in unmanned ship, install camera and sonar sensor on the unmanned ship, the device includes:
the data acquisition module is used for acquiring the overwater image frame and the underwater scanning data which are acquired by the camera and the sonar sensor synchronously in the current area;
the state information determining module is used for determining obstacles in the current area and determining current state information related to the obstacles according to the overwater image frames and the underwater scanning data;
and the obstacle avoidance control module is used for controlling the unmanned ship to avoid the obstacle by combining a preset obstacle avoidance strategy according to the current state information.
In a third aspect, the present embodiment provides an unmanned ship, including:
an unmanned ship hull;
the camera, the millimeter wave radar and the sonar sensor are arranged on the hull of the unmanned ship;
a controller in communication connection with the camera, millimeter wave radar, and sonar sensor, the controller comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein, the first and the second end of the pipe are connected with each other,
the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the method for avoiding obstacles of an unmanned ship according to any embodiment of the present invention.
In a fourth aspect, this embodiment provides a computer-readable storage medium, where computer instructions are stored, and the computer instructions are configured to, when executed by a processor, implement the obstacle avoidance method for an unmanned ship according to any embodiment of the present invention.
The embodiment of the invention provides an obstacle avoidance method and device for an unmanned ship, the unmanned ship and a storage medium, the method is applied to the unmanned ship, a camera and a sonar sensor are installed on the unmanned ship, and the method comprises the following steps: acquiring an overwater image frame and underwater scanning data which are synchronously acquired by the camera and the sonar sensor in the current area; determining obstacles in the current area and current state information related to the obstacles according to the overwater image frames and the underwater scanning data; and controlling the unmanned ship to avoid the obstacle by combining a preset obstacle avoidance strategy according to the current state information. By utilizing the method, the visual camera and the sonar sensor are combined to collect multi-source data, the identification and monitoring of the obstacles in two modes of water and underwater can be realized, and compared with the prior art, the obstacle identification preparation rate can be improved, the false identification is reduced, and the adaptability of an operation scene is improved.
It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present invention, nor do they necessarily limit the scope of the invention. Other features of the present invention will become apparent from the following description.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the description below are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a schematic flow chart of an obstacle avoidance method for an unmanned ship according to an embodiment of the present invention;
fig. 2 is a schematic flow chart of an obstacle avoidance method for an unmanned ship according to a second embodiment of the present invention;
fig. 2a is a flowchart illustrating an obstacle avoidance performing step of an unmanned ship in a certain scenario according to a second embodiment of the present invention;
fig. 3 is a schematic structural diagram of an obstacle avoidance device of an unmanned ship according to a third embodiment of the present invention;
fig. 4 is a schematic structural diagram of an unmanned ship according to a fourth embodiment of the present invention.
Detailed Description
In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that the terms "original", "target", and the like in the description and claims of the present invention and the drawings described above are used for distinguishing similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
In the prior art, the detection and identification of obstacles of unmanned ships are mainly realized in the following ways. One obstacle identification method is as follows: the identification is carried out by only depending on millimeter wave radar or a distance sensor and the like. Firstly, the operable distance of the millimeter wave radar is mostly a short distance, which is tens of meters, and the operation is greatly limited at a relatively long distance; in addition, the data processing complexity of the millimeter waves is relatively high, the hardware cost is high, only distance information can be acquired for the front object, and information such as whether the object is an obstacle or a floating object cannot be perceived.
Another obstacle recognition method is: the recognition is performed by means of camera vision. The applications of camera vision on unmanned boats are divided into two categories: video monitoring and identification of obstacles. The video monitoring is to record the water area around the ship body to obtain the situation in front, judge the identification of the auxiliary obstacle by human eyes for the situation of the obstacle, apply the artificial intelligence technology to identify the obstacle, transmit the picture shot by the camera to the processing module, and then carry out artificial intelligence reasoning to obtain whether there is the obstacle and the information of the obstacle.
In the existing application scene, the vision is relatively independent from the application of a laser radar, a millimeter wave radar and a sensor, and a camera can control the effective visual field distance of the camera within a required range by selecting a visual field angle and a focal length, so that whether an obstacle exists or not can be acquired within the visual field range. Increasing the range of binocular range by increasing the baseline can have several consequences: the large baseline can cause the accuracy of calibration to be reduced; the lower limit of the distance measurement is increased, and the influence on the distance measurement in a short distance is caused; the interference is formed by the continuous non-textured surfaces such as specular reflection of the water surface, so that the binocular vision cannot measure the distance. The millimeter wave radar can perform accurate distance calculation in a short distance, and the effective distance for distance measurement is limited under the existing condition, so that the obstacle avoidance requirement in a medium-long distance cannot be met, and the data processing difficulty is higher. In view of the above problems, a method is needed to realize accurate obstacle avoidance of the unmanned ship.
Example one
Fig. 1 is a schematic flow diagram of an obstacle avoidance method for an unmanned ship according to an embodiment of the present invention, where the method is applicable to a situation where obstacles on water and underwater in a current area are avoided in a driving process of the unmanned ship, and the method is applied to the unmanned ship, where a camera and a sonar sensor are installed on the unmanned ship, and the method can be executed by an obstacle avoidance device of the unmanned ship, and the obstacle avoidance device of the unmanned ship can be implemented in a hardware and/or software manner.
As shown in fig. 1, the obstacle avoidance method for an unmanned ship provided in this embodiment may specifically include the following steps:
and S110, acquiring an overwater image frame and underwater scanning data which are acquired by a camera and a sonar sensor synchronously in the current area.
It is understood that obstacles such as reefs, shoals, abandoned steel pipes, etc. may be encountered when the unmanned ship is traveling at sea. In order to avoid collision between the unmanned ship and the obstacle, it is necessary to determine in advance whether the unmanned ship has the obstacle around the unmanned ship and to obtain information such as a distance and a direction of the obstacle with respect to the unmanned ship, and the information is used for the unmanned ship to avoid the obstacle. In this embodiment, a camera and a sonar sensor are installed on the unmanned ship, wherein the current position where the unmanned ship runs is used as a center, and a range where the camera and the sonar sensor on the unmanned ship can shoot or detect is determined as a current area. The camera is used for obtaining an overwater image frame of a current area where the unmanned ship runs, and the sonar sensor is used for obtaining underwater scanning data of the current area of the unmanned ship. Preferably, the above-water image frames shot in the driving direction of the unmanned ship and the underwater scan data in the driving direction of the unmanned ship are acquired.
It should be noted that, when the unmanned ship is in a driving process, the unmanned ship may encounter obstacles only on the water, obstacles only on the underwater part, and obstacles on both the water and the underwater parts. Therefore, in this embodiment, the camera and the sonar sensor perform synchronous acquisition on the current area to obtain the above-water image frames and the underwater scanning data, which are used for judging whether the above-water part and the underwater part of the current area have obstacles. Preferably, the camera may be a monocular camera.
And S120, determining the obstacles in the current area and determining the current state information related to the obstacles according to the overwater image frames and the underwater scanning data.
The overwater image frame represents the shooting condition of the overwater part of the current area, and the underwater scanning data represents the obstacle condition of the underwater part of the current area. Therefore, whether an obstacle exists in the overwater part of the current area or not can be determined according to the overwater image frames, and if the obstacle exists, obstacle related information such as the type of the obstacle, the distance between the obstacle and the unmanned ship and the like can be determined. Whether an obstacle exists in the underwater part of the current area or not can be determined according to the underwater scanning data, and if the obstacle exists, obstacle related information such as the type of the obstacle and the distance between the obstacle and the unmanned ship is determined.
In this embodiment, the obstacle in the above-water portion may be determined by inputting the acquired above-water image frames into a predetermined obstacle detection model, and determining whether there is an obstacle in the above-water portion according to an output result of the obstacle detection model. And if the output result is null, determining that the overwater image frame does not contain the obstacle, and further determining that the overwater part of the current area does not contain the obstacle. And if the output result of the obstacle is the obstacle category, determining that the overwater image frame contains the obstacle, and further determining that the overwater part of the current area contains the obstacle. The obstacle detection model can be obtained based on a large number of image samples containing obstacles through training. Here, the training process of the obstacle detection model is not specifically described.
It can be understood that if it is determined that the above-water part of the current area contains an obstacle, the distance between the current obstacle and the unmanned ship and the ship speed of the unmanned ship relative to the obstacle can be further determined according to the above-water image frames. In this embodiment, the above-water image frames captured by the camera may be presented in a video stream manner, and any one frame of the captured video stream is the above-water image frame. Based on each frame of the overwater image frame, a depth distance picture can be generated by applying artificial intelligence, and the distance between the front obstacle and the unmanned ship and the ship speed of the unmanned ship are further determined based on the depth distance picture.
The sonar transducer utilizes the transmission and reflection characteristics of sound waves in water to determine whether an obstacle exists through electroacoustic conversion and information processing. In this embodiment, the distance between the obstacle in the underwater portion and the unmanned ship and the ship speed may be determined by converting the acquired underwater scan data into picture data, inputting the picture data into a predetermined obstacle detection model, and determining whether the obstacle exists in the underwater portion according to an output result of the obstacle detection model. Note that the obstacle detection model is different from the obstacle detection model for the image frames on water. And if the output result is null, determining that the overwater image frame does not contain the obstacle, and further determining that the underwater part of the current area does not contain the obstacle. And if the output result of the obstacle is the obstacle type, determining that the underwater image frame contains the obstacle, and further determining that the underwater part of the current area contains the obstacle. The obstacle detection model can be obtained based on a large number of image samples containing obstacles through training. Here, the training process of the obstacle detection model is not specifically described.
It can be understood that if it is determined that the underwater portion of the current region includes an obstacle, the distance between the current obstacle and the unmanned ship and the ship speed of the unmanned ship relative to the obstacle can be further determined according to the underwater scan data. In this embodiment, the distance between the obstacle in front and the unmanned ship and the ship speed of the unmanned ship are determined based on each frame of image data.
In this embodiment, the ship speed of the unmanned ship is calculated according to the above-water obstacle and the underwater obstacle, and the two speeds are comprehensively processed to obtain the ship speed of the unmanned ship. In this embodiment, the above-water obstacle type, the below-water obstacle type, the distance between the above-water obstacle and the unmanned ship, the distance between the below-water obstacle and the unmanned ship, and the ship speed of the unmanned ship are used as the current state information related to the obstacle. And the current state information related to the obstacle is used as a data basis for determining that the unmanned ship avoids the obstacle according to the current state information.
And S130, controlling the unmanned ship to avoid the obstacle by combining a preset obstacle avoidance strategy according to the current state information.
The current state information comprises the type of the above-water obstacle, the type of the underwater obstacle, the distance between the above-water obstacle and the unmanned ship, the distance between the underwater obstacle and the unmanned ship and the ship speed of the unmanned ship. The preset obstacle avoidance strategy comprises obstacle avoidance modes of the unmanned ship under all conditions, for example, if the obstacle exists only in the overwater part in the current area, the underwater part does not need to be considered, only the ship speed and the driving direction need to be controlled, and the unmanned ship is prevented from colliding with the overwater obstacle. If the current area only has the obstacle in the underwater part, the overwater part does not need to be considered, and only the ship speed and the running direction need to be controlled, so that the unmanned ship is prevented from colliding the underwater obstacle. If the obstacle exists in the overwater part and the underwater part of the current area, the distance between the overwater obstacle and the underwater obstacle and the distance between the unmanned ship and the underwater obstacle need to be comprehensively considered, the ship speed and the running direction need to be controlled, and the unmanned ship is prevented from colliding with the overwater obstacle and the underwater obstacle.
In this embodiment, the parameter values such as the offset required for obstacle avoidance of the unmanned ship are determined, the control of the driving state of the unmanned ship mainly includes the ship speed of the unmanned ship and the driving direction of the unmanned ship, and the unmanned ship is enabled to avoid the obstacle by controlling the ship speed and the driving direction of the unmanned ship.
The embodiment of the invention provides an obstacle avoidance method of an unmanned ship, which is applied to the unmanned ship, wherein the unmanned ship is provided with a camera and a sonar sensor, and the method comprises the following steps: acquiring an overwater image frame and underwater scanning data which are acquired by a camera and a sonar sensor synchronously in a current area; determining obstacles in the current area and current state information related to the obstacles according to the overwater image frames and the underwater scanning data; and controlling the unmanned ship to avoid the obstacle by combining a preset obstacle avoidance strategy according to the current state information. By utilizing the method, the visual camera and the sonar sensor are combined to collect multi-source data, the identification and monitoring of the obstacles in two modes of water and underwater can be realized, and compared with the prior art, the obstacle identification preparation rate can be improved, the false identification is reduced, and the adaptability of an operation scene is improved.
As an alternative embodiment of the present invention, on the basis of the above embodiment, the method is further defined to further include: and if the current area is determined to have no obstacle, controlling the unmanned ship to run according to the current running state.
In this embodiment, if it is determined that no obstacle exists in the above-water part and the below-water part of the current area, the current running state of the unmanned ship does not need to be changed, and the unmanned ship is controlled to run according to the current running state.
Example two
Fig. 2 is a schematic flow diagram of an obstacle avoidance method for an unmanned ship according to a second embodiment of the present invention, and this embodiment is further optimized in the foregoing embodiment, and in this embodiment, the determining of the obstacle in the current area and the determining of the current state information associated with the obstacle according to the overwater image frame and the underwater scanning data are further specifically: determining obstacles on the water part of the current area and determining first state information related to the obstacles according to the water image frame; determining obstacles of the underwater part of the current area and second state information related to the obstacles according to the underwater scanning data; and determining the current state information associated with the obstacle according to the first state information and the second state information.
As shown in fig. 2, the second embodiment provides an obstacle avoidance method for an unmanned ship, which specifically includes the following steps:
s210, acquiring an overwater image frame and underwater scanning data which are acquired by a camera and a sonar sensor synchronously in the current area.
And S220, determining the obstacle on the water part of the current area and determining first state information related to the obstacle according to the water image frame.
Wherein, state information that is relevant with the obstacle on part on water is marked as first state information, and wherein, first state information mainly includes: the distance of the above-water obstacle from the unmanned ship, and the ship speed of the unmanned ship. In this embodiment, the image frames on the water are input into a pre-constructed obstacle recognition model, and whether an obstacle exists in the part on the water is determined according to an output result. If the obstacle exists, the output result is the obstacle type of the water obstacle.
And if the obstacle exists on the part above water, determining a depth distance map according to the image frame above water. And determining the distance between the above-water obstacle and the unmanned ship and the ship speed of the unmanned ship according to the depth distance map.
Optionally, in this embodiment, according to the image frames in water, the obstacle determination of the water portion in the current area and the determination of the first current state information associated with the obstacle may be implemented by the following steps, which may be specifically expressed as:
a1, inputting the overwater image frame into a first class recognition model which is constructed in advance, and if the output result is empty, determining that no obstacle exists in the overwater part of the current area.
The method comprises the steps of recording a water obstacle recognition model as a first class recognition model, wherein the first class recognition model is obtained in advance based on image training of a large number of obstacles containing various types. Specifically, the overwater image frame is input into the first class identification model, and if the output result is null, it is determined that no obstacle exists in the overwater part of the current area.
b1, otherwise, determining the obstacle category of the part of obstacles on the water in the current area, and determining first state information related to the obstacles according to the water image frame.
If the output result is not null, the obstacle type, namely the obstacle type of the water obstacle in the current area can be output. Further, after the obstacle in the water part is determined, first state information related to the obstacle is determined, wherein the first state information comprises the distance between the water obstacle and the unmanned ship and the ship speed of the unmanned ship.
Further, still install the millimeter wave radar on the unmanned ship, according to image frame on the water, confirm the first status information that the barrier is relevant, include:
and b11, inputting the overwater image frame into a pre-constructed depth map generation model, and generating a depth distance map corresponding to the overwater image frame.
The overwater image frame is a picture shot by a camera, and the depth-distance map is an image with the distance depth from each point in an actual scene to the camera as a pixel value. And recording a model for generating a depth distance map according to the overwater image frames as a depth map generation model, wherein the depth map generation model is obtained by training on the basis of a large number of images and depth distance maps. Specifically, the overwater image frame is input into a depth map generation model, and a depth distance map corresponding to the overwater image frame is generated. The monocular distance measurement of the objects in the visual field range is realized, and a depth distance map is generated.
And b12, determining the visual angle ranging of the obstacle from the unmanned ship according to the depth distance map.
It will be appreciated that the camera may obtain far-away marine image frames, and thus the distance of the far-away obstacle from the unmanned ship, from the depth-distance map. Specifically, the distance between the obstacle and the unmanned ship is determined according to the value of each pixel point in the depth distance map, and the distance is recorded as the visual angle ranging.
And b13, determining a first distance between the obstacle and the unmanned ship according to the visual angle ranging and the radar ranging measured by the millimeter wave radar.
In this embodiment, the unmanned ship is further provided with a millimeter wave radar, and the millimeter wave radar has the advantages of high obstacle distance measurement precision, strong environmental adaptability and short effective range measurement. In the step, the first distance between the obstacle and the unmanned ship is determined by comprehensively considering the radar ranging of the view angle ranging and the millimeter wave radar measurement.
Further, determining a first distance of the obstacle from the unmanned ship according to the angle of view ranging and the radar ranging measured by the millimeter wave radar, comprising:
b131, when the ranging angle is larger than the set distance threshold, taking the ranging angle data as a first distance between the obstacle and the unmanned ship.
The distance threshold may be preset, and the distance threshold is used to distinguish the determination method of the first distance. It is understood that the perspective range determined from the depth-distance map may determine a longer distance, and for a shorter distance, the accuracy determined from the millimeter-wave radar may be higher.
Specifically, because the millimeter wave radar range is short, and for longer distance, the millimeter wave radar measurement accuracy is inaccurate, when the visual angle range is greater than the set distance threshold, only the visual angle range data determined according to the distance depth map is considered, and the visual angle range data is used as the first distance from the obstacle to the unmanned ship.
And b132, when the range of the angle of view is smaller than or equal to the set distance threshold, controlling the millimeter wave radar to turn to the current area and obtaining the radar range of the obstacle from the unmanned ship.
Specifically, since the millimeter wave radar ranging range is short, and the millimeter wave radar measurement accuracy is high in the short ranging range, when the view angle ranging is less than or equal to the set distance threshold, the view angle ranging data determined according to the depth-distance map and the radar ranging of the obstacle distance unmanned ship measured by the millimeter wave radar are comprehensively considered. And calculating the actual direction of the obstacle according to the pixel position and the rotation angle information of the obstacle and moving the millimeter wave radar to rotate to perform obstacle ranging.
And b133, carrying out weighting processing on the view angle ranging and the radar ranging, and determining a first distance between the obstacle and the unmanned ship.
Specifically, when the range of the angle of view is smaller than or equal to the set distance threshold, different weights are distributed to the range of the angle of view and the range of the radar, weighting processing is carried out, and the first distance from the obstacle to the unmanned ship is determined.
And b14, determining the first ship speed of the unmanned ship according to the displacement of the pixels in the two adjacent depth distance maps and the frame time interval.
Specifically, the movement displacement of the unmanned ship within the frame time interval can be determined according to the movement of the pixel position between the adjacent depth distance frames, and then divided by the frame time interval, the marine ship speed can be calculated, which is denoted as the first ship speed in this embodiment.
And b15, taking the first distance and the first ship speed as first state information.
Specifically, the first distance and the first ship speed are taken as the first state information.
And S230, determining the obstacle of the underwater part of the current area and determining second state information related to the obstacle according to the underwater scanning data.
The state information related to the obstacle of the underwater part is recorded as second state information, wherein the second state information mainly comprises: the distance of the underwater obstacle from the unmanned ship, and the ship speed of the unmanned ship. In this embodiment, the underwater scan data is converted into image data, the image data is input into a pre-constructed obstacle recognition model, and whether an obstacle exists in the underwater portion is determined according to an output result. And if the obstacle exists, outputting the obstacle type of the underwater obstacle as an output result.
And if the underwater part has the obstacle, converting the underwater scanning data into image data. And determining the distance between the underwater obstacle and the unmanned ship and the ship speed of the unmanned ship according to the image data.
Optionally, in this embodiment, according to the underwater scanning data, the obstacle determination of the underwater portion of the current area and the determination of the second state information associated with the obstacle may be implemented by the following steps, which may be specifically expressed as:
and a2, converting the underwater scanning data collected within the set time into picture data.
Specifically, underwater scanning data acquired by the sonar sensor is acquired in real time, presented in a row line data mode and converted into picture data.
And b2, inputting the picture data into a pre-constructed second type identification model, and if the output result is empty, determining that no obstacle exists in the underwater part of the current area.
The underwater obstacle recognition model is marked as a second type recognition model, and the second type recognition model is obtained in advance based on a large number of images containing various obstacles through training. Specifically, the overwater picture data are input into the second type identification model, and if the output result is null, it is determined that no obstacle exists in the underwater part of the current area.
And c2, otherwise, determining the obstacle category of the underwater partial obstacle in the current area, and determining second state information related to the obstacle according to the underwater scanning data.
If the output result is not empty, the obstacle type, namely the obstacle type of the underwater partial obstacle in the current area can be output. Further, after the obstacle is determined to exist in the underwater part, second state information related to the obstacle is determined, wherein the second state information comprises the distance between the underwater obstacle and the unmanned ship and the ship speed of the unmanned ship.
Further, determining second state information associated with the obstacle from the underwater scan data includes:
and c21, determining a second distance between the obstacle and the unmanned ship according to the underwater scanning data.
Specifically, according to underwater scanning data, data conditions of sound waves sent and received by the sonar sensor, and the time difference of sending and receiving the sound waves, the second distance between the obstacle and the unmanned ship can be calculated.
And c22, determining a second ship speed of the unmanned ship according to the distance difference between the set time interval barrier and the unmanned ship.
Specifically, according to the distance difference between the obstacle and the unmanned ship at the set time interval, the displacement of the unmanned ship within the set time interval can be determined, and the ship speed of the unmanned ship is calculated according to the displacement and the time interval, which is denoted as the second ship speed in this embodiment.
And c23, taking the second distance and the second ship speed as second state information.
Specifically, the second distance and the second ship speed are taken as the second state information.
And S240, determining the current state information related to the obstacle according to the first state information and the second state information.
The first state information comprises a first distance between the above-water obstacle and the unmanned ship and the ship speed of the unmanned ship. The second state information comprises a second distance between the underwater obstacle and the unmanned ship and the ship speed of the unmanned ship. After the obstacle is detected on the underwater water, the overwater image frame is intercepted, the overwater image frame and the sonar sensor are kept, the same-frequency data of the millimeter wave radar are fused and returned to the category and the distance of the overwater and underwater obstacle, and the overwater ship speed is calculated according to the movement variable of the pixel position between adjacent frames, weighted fusion is carried out, and ship speed information and obstacle azimuth information are obtained.
And carrying out same-frequency and same-frame-rate fusion on the underwater sonar detection result, the above-water monocular vision detection and ranging and millimeter wave radar ranging, and generating information corresponding to the obstacle distance category, speed, direction and the like for obstacle avoidance calculation.
Further, determining the current state information associated with the obstacle according to the first state information and the second state information includes:
and a3, weighting the first ship speed in the first state information and the second ship speed in the second state information to determine the ship speed of the unmanned ship.
Specifically, different weights are respectively given to the first ship speed and the second ship speed, and the first ship speed in the first state information and the second ship speed in the second state information are subjected to weighting processing to determine the ship speed of the unmanned ship.
And b3, taking the obstacle type, the first distance in the first state information, the second distance in the second state information and the ship speed as the current state information related to the obstacle.
And taking the obstacle type of the above-water obstacle, the first distance between the above-water obstacle and the unmanned ship, the obstacle type of the underwater obstacle, the first distance between the underwater obstacle and the unmanned ship and the ship speed of the unmanned ship as the current state information related to the obstacle.
And S250, controlling the unmanned ship to avoid the obstacle by combining a preset obstacle avoidance strategy according to the current state information.
In the optional embodiment, functions such as visual ranging, visual detection, speed measurement and the like are completed through an artificial intelligence algorithm and a multi-source fusion technology, so that parameter values such as offset required by an obstacle avoidance strategy are provided for a control end. Compared with the prior art, the method has good environmental adaptability, can cover short-range and medium-range distance measurement detection and distance measurement, and covers the overwater and underwater distance measurement obstacle avoidance functions; the working efficiency is improved, and the visual ranging is added in the equipment aspect, so that the cost is reduced, and the actual operation efficiency is improved; the resource utilization rate is high, the visual scheme and the millimeter wave sensor are fused, more effective information of the ship body movement is obtained, and the high efficiency and the accuracy of obstacle avoidance are convenient to realize.
In order to more clearly express the obstacle avoidance method for the unmanned ship provided by the embodiment of the invention, the obstacle avoidance step of the unmanned ship is described by taking a certain scene as an example. Fig. 2a is a flowchart illustrating steps of performing obstacle avoidance on an unmanned ship in a certain scenario according to a second embodiment of the present invention, where as shown in fig. 2a, the specific steps include:
s1, acquiring an overwater image frame and underwater scanning data which are acquired by a camera and a sonar sensor synchronously in a current area.
S2, inputting the overwater image frame into a first class recognition model which is constructed in advance, judging whether an output result is empty, if so, executing the step S3, otherwise, executing the steps S4-S11.
And S3, determining that no obstacle exists in the water part of the current area.
And S4, determining the obstacle type of the part of obstacles on the water in the current area.
And S5, inputting the overwater image frame into a pre-constructed depth map generation model to generate a depth distance map corresponding to the overwater image frame.
And S6, determining the visual angle ranging of the barrier from the unmanned ship according to the depth distance map.
And S7, judging whether the visual angle ranging is larger than a set distance threshold value, if so, executing the step S8, and otherwise, executing the steps S9-S10.
And S8, when the range of the angle of view is larger than a set distance threshold, taking the range data of the angle of view as a first distance from the obstacle to the unmanned ship.
And S9, when the visual angle ranging is smaller than or equal to the set distance threshold, controlling the millimeter wave radar to turn to the current area and obtaining the radar ranging of the obstacle from the unmanned ship.
And S10, weighting the view angle ranging and the radar ranging, and determining a first distance between the obstacle and the unmanned ship.
S11, determining a first ship speed of the unmanned ship according to the displacement of the pixels in the two adjacent depth distance maps and the frame time interval.
And S12, taking the first distance and the first ship speed as first state information.
Steps S2 to S12 are related determinations for the above-water obstacle, and steps S13 to S21 are related determinations for the underwater obstacle.
And S13, converting the underwater scanning data acquired within the set time into picture data.
And S14, inputting the picture data into a pre-constructed second type recognition model, judging whether an output result is empty, if so, executing the step S15, otherwise, executing the step S16.
S15, determining that no obstacle exists in the underwater part of the current area.
And S16, determining the obstacle type of the underwater partial obstacle in the current area.
And S17, determining a second distance between the obstacle and the unmanned ship according to the underwater scanning data.
And S18, determining a second ship speed of the unmanned ship according to the distance difference between the obstacle and the unmanned ship at the set time interval.
And S19, taking the second distance and the second ship speed as second state information.
And S20, weighting the first ship speed in the first state information and the second ship speed in the second state information to determine the ship speed of the unmanned ship.
And S21, taking the obstacle type, the first distance in the first state information, the second distance in the second state information and the ship speed as current state information related to the obstacle.
And S22, controlling the unmanned ship to avoid the obstacle by combining a preset obstacle avoidance strategy according to the current state information.
And S23, if it is determined that no obstacle exists in the water part and the underwater part of the current area, controlling the unmanned ship to run according to the current running state.
Fig. 2a shows control of the driving state of the unmanned ship when there is an obstacle in both the underwater and marine sections and there is no obstacle in both the underwater and marine sections. If only the obstacle exists in the above-water part or only the obstacle exists in the underwater part, the driving state of the unmanned ship is controlled based on only the current state information of the above-water obstacle or the current state information of the underwater obstacle, which is not specifically shown in the figure.
EXAMPLE III
Fig. 3 is a schematic structural view of an obstacle avoidance device for an unmanned ship according to a third embodiment of the present invention, which is applicable to a situation where an unmanned ship avoids obstacles above and below water in a current driving area. As shown in fig. 3, the apparatus includes: a data acquisition module 31, a status information determination module 32, and an obstacle avoidance control module 33, wherein,
the data acquisition module 31 is configured to acquire an image frame on water and underwater scan data acquired by the camera and the sonar sensor synchronously in a current area;
the state information determining module 32 is configured to determine an obstacle in a current area and current state information associated with the obstacle according to the above-water image frames and the underwater scanning data;
and the obstacle avoidance control module 33 is used for controlling the unmanned ship to avoid obstacles by combining a preset obstacle avoidance strategy according to the current state information.
Further, the status information determining module 32 may specifically include:
the first state information determining unit is used for determining the obstacle of the water part of the current area and determining the first state information related to the obstacle according to the water image frame;
the second state information determining unit is used for determining the obstacles of the underwater part of the current area and determining the second state information related to the obstacles according to the underwater scanning data;
and the current state information determining unit is used for determining the current state information related to the obstacle according to the first state information and the second state information.
Further, the first state information determination unit may include:
the overwater obstacle identification subunit is used for inputting the overwater image frame into a first type identification model which is constructed in advance, and if the output result is empty, determining that no obstacle exists in the overwater part of the current area;
and the overwater obstacle type determining subunit is used for determining the obstacle type of the part of the obstacles in the current area in the water and determining first state information related to the obstacles according to the overwater image frame if the obstacle type of the part of the obstacles in the water is not the same as the obstacle type of the part of the obstacles in the current area in the water.
Further, the above-water obstacle category determining subunit may specifically be configured to:
inputting the overwater image frames into a pre-constructed depth map generation model to generate a depth distance map corresponding to the overwater image frames;
determining the visual angle ranging of the obstacle from the unmanned ship according to the depth distance map;
determining a first distance between the obstacle and the unmanned ship according to the visual angle ranging and the radar ranging measured by the millimeter wave radar;
determining a first ship speed of the unmanned ship according to the displacement of pixels in the two adjacent depth distance maps and the frame time interval;
the first distance and the first ship speed are used as first state information.
Further, the above-water obstacle category determination subunit is configured to perform radar ranging according to the perspective ranging and the millimeter wave radar measurement, and the step of determining the first distance from the unmanned ship to the obstacle may include:
when the range of the angle of view is larger than a set distance threshold, taking the range data of the angle of view as a first distance from the unmanned ship to the obstacle;
when the range of the angle of view is smaller than or equal to the set distance threshold, controlling the millimeter wave radar to turn to the current area and obtaining the radar range of the unmanned ship from the obstacle;
and weighting the view angle ranging and the radar ranging to determine a first distance between the obstacle and the unmanned ship.
Further, the second state information determining unit may specifically include:
the data conversion unit is used for converting underwater scanning data acquired within set time into picture data;
the underwater obstacle identification subunit is used for inputting the picture data into a second type identification model which is constructed in advance, and if the output result is empty, determining that no obstacle exists in the underwater part of the current area;
and the underwater obstacle type determining subunit is used for determining the obstacle types of the underwater partial obstacles in the current area and determining second state information related to the obstacles according to the underwater scanning data if the underwater partial obstacles are not in the current area.
Further, the underwater obstacle category determination subunit may be specifically configured to:
determining a second distance from the obstacle to the unmanned ship according to the underwater scanning data;
determining a second ship speed of the unmanned ship according to the distance difference between the obstacle and the unmanned ship at the set time interval;
and taking the second distance and the second ship speed as second state information.
Further, the current state information determining unit may specifically be configured to:
weighting the first ship speed in the first state information and the second ship speed in the second state information to determine the ship speed of the unmanned ship;
and taking the obstacle category, the first distance in the first state information, the second distance in the second state information and the ship speed as the current state information related to the obstacle.
Further, the apparatus further includes a state maintaining control module, specifically configured to:
and if the obstacle does not exist in the current area, controlling the unmanned ship to run according to the current running state.
The obstacle avoidance device of the unmanned ship provided by the embodiment of the invention can execute the obstacle avoidance method of the unmanned ship provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.
Example four
Fig. 4 is a schematic structural diagram of an unmanned ship according to a fourth embodiment of the present invention. As shown in fig. 4, an unmanned ship according to a fourth embodiment of the present invention includes: unmanned ship hull (not shown in the figures); the camera 1 is installed on the hull of the unmanned ship; the millimeter wave radar 1 is installed on the hull of the unmanned ship; the sonar sensor 3 is arranged on the hull of the unmanned ship; and the controller 4 is in communication connection with the camera 1, the millimeter wave radar 2 and the sonar sensor 3.
The controller 4 includes: one or more processors 41 and storage 42; the processor 41 in the controller 4 may be one or more, and one processor 41 is taken as an example in fig. 4; storage 42 is used to store one or more programs; the one or more programs are executed by the one or more processors 41, so that the one or more processors 41 implement the object detection and positioning method according to any one of the embodiments of the present invention.
The processor 41 and the storage device 42 in the controller 4 may be connected by a bus or other means, and the bus connection is exemplified in fig. 4.
The storage device 42 in the controller 4 is used as a computer-readable storage medium, and can be used to store one or more programs, which may be software programs, computer-executable programs, and modules, and program instructions/modules corresponding to the obstacle avoidance method for the unmanned ship according to one or two embodiments of the present invention (for example, modules in the obstacle avoidance apparatus for the unmanned ship shown in fig. 3 include the data acquisition module 31, the status information determination module 32, and the obstacle avoidance control module 33). The processor 41 executes various functional applications and data processing of the controller 4 by running software programs, instructions and modules stored in the storage device 42, so as to implement the obstacle avoidance method of the unmanned ship in the above method embodiment.
The storage device 42 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of the controller 4, and the like. Further, the storage 42 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some examples, storage 42 may further include memory located remotely from processor 41, which may be connected to the device over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The input device 43 may be used to receive input numeric or character information and generate key signal inputs related to user settings and function control of the controller 4. The output device 44 may include a display device such as a display screen.
And, when the one or more programs included in the above-mentioned controller 4 are executed by the one or more processors 41, the programs perform the following operations:
acquiring an overwater image frame and underwater scanning data which are synchronously acquired by the camera and the sonar sensor in the current area;
determining obstacles in the current area and current state information related to the obstacles according to the overwater image frames and the underwater scanning data;
and controlling the unmanned ship to avoid the obstacle by combining a preset obstacle avoidance strategy according to the current state information.
EXAMPLE five
An embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, where the computer program is used, when executed by a processor, to execute an obstacle avoidance method for an unmanned ship, where the method includes:
acquiring an overwater image frame and underwater scanning data which are synchronously acquired by the camera and the sonar sensor in the current area;
determining obstacles in the current area and current state information related to the obstacles according to the overwater image frames and the underwater scanning data;
and controlling the unmanned ship to avoid the obstacle by combining a preset obstacle avoidance strategy according to the current state information.
Optionally, the program, when executed by the processor, may be further configured to execute the obstacle avoidance method for the unmanned ship according to any embodiment of the present invention.
Computer storage media for embodiments of the invention may employ any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a Read Only Memory (ROM), an Erasable Programmable Read Only Memory (EPROM), a flash Memory, an optical fiber, a portable CD-ROM, an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. A computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take a variety of forms, including, but not limited to: an electromagnetic signal, an optical signal, or any suitable combination of the foregoing. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, radio Frequency (RF), etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in some detail by the above embodiments, the invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the invention, and the scope of the invention is determined by the scope of the appended claims.
Claims (12)
1. An obstacle avoidance method of an unmanned ship is applied to the unmanned ship, a camera and a sonar sensor are installed on the unmanned ship, and the method comprises the following steps:
acquiring an overwater image frame and underwater scanning data which are synchronously acquired by the camera and the sonar sensor in the current area;
determining obstacles in the current area and current state information related to the obstacles according to the overwater image frames and the underwater scanning data;
and controlling the unmanned ship to avoid the obstacle by combining a preset obstacle avoidance strategy according to the current state information.
2. The method of claim 1, wherein said determining an obstacle for the current region and current status information associated with the obstacle from the above-water image frames and the underwater scan data comprises:
determining obstacles on the water part of the current area and determining first state information related to the obstacles according to the water image frame;
determining obstacles of the underwater part of the current area and second state information related to the obstacles according to the underwater scanning data;
and determining the current state information associated with the obstacle according to the first state information and the second state information.
3. The method of claim 2, wherein said determining an obstacle in the marine portion of the current region and a first current state information associated with the obstacle from the marine image frames comprises:
inputting the overwater image frame into a first class recognition model which is constructed in advance, and if the output result is empty, determining that no obstacle exists in the overwater part of the current area;
otherwise, determining the obstacle category of the above-water partial obstacles in the current area, and determining first state information related to the obstacles according to the above-water image frame.
4. The method of claim 3, wherein the drone is further equipped with a millimeter wave radar, and wherein determining the first status information associated with the obstacle from the marine image frames comprises:
inputting the overwater image frame into a pre-constructed depth map generation model to generate a depth distance map corresponding to the overwater image frame;
according to the depth distance map, determining the visual angle ranging of the obstacle from the unmanned ship;
determining a first distance from the obstacle to the unmanned ship according to the view angle ranging and the radar ranging measured by the millimeter wave radar;
determining a first ship speed of the unmanned ship according to the displacement of pixels in two adjacent depth distance maps and a frame time interval;
and taking the first distance and the first ship speed as the first state information.
5. The method of claim 4, wherein determining a first distance of the obstacle from the unmanned vessel based on the perspective ranging and the radar ranging measured by the millimeter wave radar comprises:
when the visual angle ranging data is larger than a set distance threshold value, taking the visual angle ranging data as a first distance between the obstacle and the unmanned ship;
when the view angle ranging is smaller than or equal to a set distance threshold value, controlling the millimeter wave radar to turn to the current area and obtaining radar ranging from an obstacle to the unmanned ship;
and weighting the view ranging and the radar ranging to determine a first distance between the obstacle and the unmanned ship.
6. The method of claim 2, wherein said determining an obstacle in the underwater portion of the current region and determining second status information associated with the obstacle from the underwater scan data comprises:
converting the underwater scanning data collected within a set time into picture data;
inputting the picture data into a second type recognition model which is constructed in advance, and if the output result is empty, determining that no obstacle exists in the underwater part of the current area;
otherwise, determining the obstacle type of the underwater partial obstacle in the current area, and determining second state information related to the obstacle according to underwater scanning data.
7. The method of claim 6, wherein determining second status information associated with the obstacle from the underwater scan data comprises:
determining a second distance from the obstacle to the unmanned ship according to underwater scanning data;
determining a second ship speed of the unmanned ship according to the distance difference between the obstacle and the unmanned ship at the set time interval;
and taking the second distance and the second ship speed as the second state information.
8. The method of claim 2, wherein determining the current state information associated with the obstacle based on the first state information and the second state information comprises:
weighting a first ship speed in the first state information and a second ship speed in the second state information to determine the ship speed of the unmanned ship;
and taking the type of the obstacle, the first distance in the first state information, the second distance in the second state information and the ship speed as the current state information related to the obstacle.
9. The method of claim 1, further comprising:
and if the current area is determined to have no obstacle, controlling the unmanned ship to run according to the current running state.
10. The utility model provides an unmanned ship's obstacle avoidance device which characterized in that, integrates in unmanned ship, install camera and sonar sensor on the unmanned ship, the device includes:
the data acquisition module is used for acquiring the overwater image frame and the underwater scanning data which are acquired by the camera and the sonar sensor synchronously in the current area;
the state information determining module is used for determining obstacles in the current area and determining current state information related to the obstacles according to the overwater image frames and the underwater scanning data;
and the obstacle avoidance control module is used for controlling the unmanned ship to avoid the obstacle by combining a preset obstacle avoidance strategy according to the current state information.
11. An unmanned ship, comprising:
an unmanned ship hull;
the camera, the millimeter wave radar and the sonar sensor are arranged on the unmanned ship body;
a controller in communication with the camera, the millimeter wave radar, and the sonar sensor, the controller includes:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores a computer program executable by the at least one processor, the computer program being executable by the at least one processor to enable the at least one processor to perform the method of unmanned ship avoidance according to any one of claims 1-9.
12. A computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions for causing a processor to implement the obstacle avoidance method for an unmanned ship according to any one of claims 1-9 when executed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210969477.5A CN115328131A (en) | 2022-08-12 | 2022-08-12 | Obstacle avoidance method and device for unmanned ship, unmanned ship and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210969477.5A CN115328131A (en) | 2022-08-12 | 2022-08-12 | Obstacle avoidance method and device for unmanned ship, unmanned ship and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115328131A true CN115328131A (en) | 2022-11-11 |
Family
ID=83924455
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210969477.5A Pending CN115328131A (en) | 2022-08-12 | 2022-08-12 | Obstacle avoidance method and device for unmanned ship, unmanned ship and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115328131A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116540776A (en) * | 2023-06-05 | 2023-08-04 | 深圳市华赛睿飞智能科技有限公司 | Unmanned aerial vehicle vision obstacle avoidance method and system |
-
2022
- 2022-08-12 CN CN202210969477.5A patent/CN115328131A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116540776A (en) * | 2023-06-05 | 2023-08-04 | 深圳市华赛睿飞智能科技有限公司 | Unmanned aerial vehicle vision obstacle avoidance method and system |
| CN116540776B (en) * | 2023-06-05 | 2023-11-07 | 深圳市华赛睿飞智能科技有限公司 | Unmanned aerial vehicle vision obstacle avoidance method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9354314B2 (en) | Underwater detection device | |
| KR20200039589A (en) | Device and method for monitoring ship and port | |
| CN104297758B (en) | A kind of auxiliary berthing device and its method based on 2D pulse type laser radars | |
| EP4089660A1 (en) | Method and device for monitoring port and ship in consideration of sea level | |
| CN110580044A (en) | unmanned ship full-automatic navigation heterogeneous system based on intelligent sensing | |
| CN112526524B (en) | Underwater fishing net detection method based on forward-looking sonar image and AUV platform | |
| CN102193089B (en) | Method and device for processing echo signal, radar device and echo signal processing program | |
| KR102530847B1 (en) | Method and device for monitoring harbor and ship | |
| KR102520844B1 (en) | Method and device for monitoring harbor and ship considering sea level | |
| KR102466804B1 (en) | Autonomous navigation method using image segmentation | |
| CN111323756B (en) | Marine radar target detection method and device based on deep learning | |
| CN115328131A (en) | Obstacle avoidance method and device for unmanned ship, unmanned ship and storage medium | |
| KR20230074438A (en) | Device and method for monitoring ship and port | |
| Wang et al. | Estimation of ship berthing parameters based on Multi-LiDAR and MMW radar data fusion | |
| CN117075112A (en) | Unmanned ship radar photoelectric fusion method for azimuth track matching | |
| CN115424233A (en) | Target detection method and target detection device based on information fusion | |
| CN111710192A (en) | Ship bridge collision accident early warning and recording method, device and system | |
| CN110412583B (en) | Underwater unidentified moving target detection system based on multi-ROV cooperation | |
| CN115082811A (en) | Method for identifying and measuring distance of marine navigation ship according to image data | |
| KR101833240B1 (en) | ISAR Signal Processing Method and Recording Medium Storing Computer Program thereof | |
| PD et al. | Vision of the seas: open visual perception framework for autonomous sailing vessels | |
| WO2023112349A1 (en) | Target monitoring device, target monitoring method, and program | |
| Muduli et al. | A Review On Recent Advancements In Signal Processing and Sensing Technologies for AUVs | |
| CN115995032B (en) | Multi-mode feature fusion ship intelligent identification method and system | |
| CN117974792B (en) | Ship target detection positioning method based on vision and AIS data cooperative training |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |