CN112907728B - Ship scene restoration and positioning method and system based on camera and edge calculation - Google Patents
Ship scene restoration and positioning method and system based on camera and edge calculation Download PDFInfo
- Publication number
- CN112907728B CN112907728B CN202110111326.1A CN202110111326A CN112907728B CN 112907728 B CN112907728 B CN 112907728B CN 202110111326 A CN202110111326 A CN 202110111326A CN 112907728 B CN112907728 B CN 112907728B
- Authority
- CN
- China
- Prior art keywords
- target
- images
- camera
- processed
- targets
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration by the use of more than one image, e.g. averaging, subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
- Y02A10/00—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE at coastal zones; at river basins
- Y02A10/40—Controlling or monitoring, e.g. of flood or hurricane; Forecasting, e.g. risk assessment or mapping
Abstract
The embodiment of the application provides a ship scene restoration and positioning method and system based on a camera and edge calculation, which are applied to the technical field of communication and can acquire a plurality of images to be processed; analyzing the multiple images to be processed by utilizing a computer vision technology to obtain targets in the multiple images to be processed and distances between the targets and a camera for shooting the targets; and synthesizing the plurality of images to be processed according to the target and the distance between the target and a camera for shooting the target to obtain a three-dimensional scene image of the ship. By obtaining the three-dimensional scene image of the ship, a ship driver can conveniently judge the channel and the surrounding environment of the ship according to the three-dimensional scene image of the ship.
Description
Technical Field
The application relates to the technical field of communication, in particular to a ship scene restoration and positioning method and system based on a camera and edge calculation.
Background
With the development of water traffic, the number of channel openings is increased day by day, and the navigation route of the ship is gradually complicated. During the running process of the ship, a ship driver needs to know the specific position of the channel where the ship driver is located and the related information around the channel to ensure the safety and high efficiency of the ship navigation.
However, current channel monitoring systems often perform channel monitoring by acquiring images through a single camera mounted on a ship or on the shore. However, when the distance measurement is performed by a single camera, the measurement accuracy is often not high, and the driver needs to know the installation position of the camera, so that the experience effect is not good.
Disclosure of Invention
An object of the embodiment of the application is to provide a method and a system for restoring and positioning a ship scene based on a camera and edge calculation, so as to solve the problem that the user experience effect of the current ship monitoring system is not good.
The specific technical scheme is as follows:
in a first aspect of an embodiment of the present application, a method for restoring and positioning a ship scene based on a camera and edge calculation is provided first, where the method includes:
acquiring a plurality of images to be processed, wherein the images to be processed comprise images of a shore shot by a camera on a ship and images of the ship shot by the camera on the shore;
analyzing the multiple images to be processed by utilizing a computer vision technology to obtain targets in the multiple images to be processed and distances between the targets and a camera for shooting the targets, wherein the targets comprise the ship and the shore;
and synthesizing the plurality of images to be processed according to the target and the distance between the target and a camera for shooting the target to obtain a three-dimensional scene image of the ship.
Optionally, the analyzing the multiple to-be-processed images by using a computer vision technology to obtain the targets in the multiple to-be-processed images and the distances between the targets and a camera for shooting the targets includes:
utilizing a computer vision technology to preprocess the plurality of images to be processed;
inputting the preprocessed images into a plurality of pre-trained deep learning network models for target recognition to obtain targets in the images to be processed recognized by the models and recognition distances between the targets recognized by the models and a camera for shooting the targets;
and weighting and summing the recognition distance between the target recognized by each model and the camera shooting the target through a preset balance function to obtain the distance between the target and the camera shooting the target.
Optionally, the preprocessing the multiple images to be processed by using the computer vision technology includes:
carrying out translation correction and angle correction on the plurality of images to be processed by utilizing a computer vision technology;
and carrying out filtering and denoising treatment on the corrected image.
Optionally, the inputting the preprocessed image into a plurality of pre-trained deep learning network models for target recognition to obtain targets in the plurality of images to be processed recognized by the models and recognition distances between the targets recognized by the models and a camera for shooting the targets includes:
inputting the preprocessed images into a plurality of pre-trained deep learning network models for target recognition to obtain targets in the plurality of images to be processed, which are recognized by the models;
acquiring pixel positions of the target in the multiple images to be processed, which are identified by the models, according to the target in the multiple images to be processed;
according to the pixel position, obtaining a first identification distance between the target and a camera for shooting the target through a triangulation algorithm;
according to the pixel position, a second identification distance between the target and a camera for shooting the target is obtained through a monocular depth estimation algorithm;
the obtaining the distance between the target and the camera shooting the target by performing weighted summation on the recognition distance between the target recognized by each model and the camera shooting the target through a preset balance function includes:
and carrying out weighted summation on the first identification distance and the second identification distance through a preset balance function to obtain the distance between the target and a camera for shooting the target.
Optionally, after analyzing the multiple images to be processed by using a computer vision technology to obtain the targets in the multiple images to be processed and the distances between the targets and a camera for shooting the targets, the method further includes:
acquiring a current channel scene two-dimensional image, wherein the current channel scene two-dimensional image is an image obtained by mapping calculation through the current channel scene image in advance;
according to the targets in the multiple images to be processed and the distance between the target and a camera for shooting the target, performing distance matching on the target to obtain a proportional scale;
and marking the target in the two-dimensional image of the current channel scene according to the proportional scale to obtain a two-dimensional scene image of the ship.
In a second aspect of the embodiments of the present application, there is provided a system for restoring and positioning a ship scene based on a camera and edge calculation, the system including:
the bank camera is used for shooting an image of the ship;
the ship camera is used for shooting images of the shore;
the system comprises edge computing equipment, a processing unit and a processing unit, wherein the edge computing equipment is used for acquiring a plurality of images to be processed, and the plurality of images to be processed comprise images of a shore shot by a camera on a ship and images of the ship shot by the camera on the shore; analyzing the plurality of images to be processed by utilizing a computer vision technology to obtain targets in the plurality of images to be processed and distances between the targets and a camera for shooting the targets, wherein the targets comprise the ship and the shore; and synthesizing the plurality of images to be processed according to the target and the distance between the target and a camera for shooting the target to obtain a three-dimensional scene image of the ship.
Optionally, the edge computing device is further configured to perform preprocessing on the multiple images to be processed by using a computer vision technology; inputting the preprocessed images into a plurality of pre-trained deep learning network models for target recognition to obtain targets in the images to be processed recognized by the models and recognition distances between the targets recognized by the models and a camera for shooting the targets; and weighting and summing the identification distances between the target identified by each model and the camera shooting the target through a preset balance function to obtain the distance between the target and the camera shooting the target.
Optionally, the edge computing device is further configured to perform, by using a computer vision technology, translational correction and angular correction on the multiple images to be processed; and carrying out filtering and denoising treatment on the corrected image.
Optionally, the edge computing device is further configured to input the preprocessed image into a plurality of pre-trained deep learning network models for target recognition, so as to obtain targets in the plurality of to-be-processed images recognized by the models; acquiring pixel positions of the target in the multiple images to be processed, which are identified by the models, according to the target in the multiple images to be processed; according to the pixel position, obtaining a first identification distance between the target and a camera for shooting the target through a triangulation algorithm; according to the pixel position, obtaining a second identification distance between the target and a camera for shooting the target through a monocular depth estimation algorithm; and carrying out weighted summation on the first identification distance and the second identification distance through a preset balance function to obtain the distance between the target and a camera for shooting the target.
Optionally, the edge computing device is further configured to obtain a current channel scene two-dimensional image, where the current channel scene two-dimensional image is an image obtained by performing mapping calculation on the current channel scene image in advance; according to the targets in the multiple images to be processed and the distance between the target and a camera for shooting the target, performing distance matching on the target to obtain a proportional scale; and marking the target in the two-dimensional image of the current channel scene according to the proportional scale to obtain a two-dimensional scene image of the ship.
On the other hand, the embodiment of the present application further provides an electronic device, which includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus;
a memory for storing a computer program;
and the processor is used for realizing any one of the ship scene restoration and positioning method based on the camera and the edge calculation when executing the program stored in the memory.
In another aspect of the embodiments of the present application, a computer-readable storage medium has a computer program stored therein, and the computer program, when executed by a processor, implements any one of the above-mentioned methods for restoring and positioning a ship scene based on camera and edge calculation.
In another aspect of the embodiments of the present application, there is also provided a computer program product including instructions, which when run on a computer, causes the computer to execute any one of the above methods for restoring and positioning a ship scene based on camera and edge calculation.
The embodiment of the application has the following beneficial effects:
according to the ship scene restoration and positioning method and system based on the camera and the edge calculation, a plurality of images to be processed can be obtained, wherein the images to be processed comprise images of a shore shot by the camera on the ship and images of the ship shot by the camera on the shore; analyzing the plurality of images to be processed by utilizing a computer vision technology to obtain targets in the plurality of images to be processed and distances between the targets and a camera for shooting the targets, wherein the targets comprise the ship and the shore; and synthesizing the plurality of images to be processed according to the target and the distance between the target and a camera for shooting the target to obtain a three-dimensional scene image of the ship.
The multiple images to be processed are synthesized according to the target and the distance between the target and the camera for shooting the target to obtain the three-dimensional scene image of the ship, so that a ship driver can conveniently judge a channel and the surrounding environment of the ship according to the three-dimensional scene image of the ship.
Of course, not all advantages described above need to be achieved at the same time in the practice of any one product or method of the present application.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the embodiments or the prior art descriptions will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present application, and other embodiments can be obtained by those skilled in the art without creative efforts.
Fig. 1 is a schematic flowchart of a ship scene restoration and positioning method based on a camera and edge calculation according to an embodiment of the present disclosure;
fig. 2a is a schematic view of a river bank camera provided in an embodiment of the present application;
fig. 2b is a schematic view of a shipborne camera provided in an embodiment of the present application;
fig. 3 is a schematic flowchart of analyzing a plurality of images to be processed according to an embodiment of the present disclosure;
fig. 4 is a schematic flowchart of a recognition distance between a recognition target and a camera for shooting the target according to an embodiment of the present disclosure;
fig. 5 is a schematic flowchart of generating a two-dimensional scene image of a ship according to an embodiment of the present application;
FIG. 6 is a diagram illustrating an example of a method for restoring and positioning a ship scene based on camera and edge calculation according to an embodiment of the present disclosure;
FIG. 7 is a diagram of an example of a camera and edge calculation based marine scene restoration and positioning system provided in an example of the present application;
fig. 8 is a schematic structural diagram of a ship scene restoration and positioning system based on a camera and edge calculation according to an embodiment of the present disclosure;
fig. 9 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments that can be derived by one of ordinary skill in the art from the description herein are intended to be within the scope of the present disclosure.
In a first aspect of an embodiment of the present application, a method for restoring and positioning a ship scene based on a camera and edge calculation is provided first, where the method includes:
acquiring a plurality of images to be processed, wherein the images to be processed comprise images of a shore shot by a camera on a ship and images of the ship shot by the camera on the shore;
analyzing the multiple images to be processed by utilizing a computer vision technology to obtain targets in the multiple images to be processed and distances between the targets and a camera for shooting the targets, wherein the targets comprise ships and shores;
and synthesizing the plurality of images to be processed according to the target and the distance between the target and a camera for shooting the target to obtain a three-dimensional scene image of the ship.
Therefore, according to the ship scene restoration and positioning method based on the camera and the edge calculation, the multiple images to be processed can be synthesized to obtain the three-dimensional scene image of the ship according to the target and the distance between the target and the camera for shooting the target, and the ship driver can sense the channel and the surrounding environment of the ship more conveniently and conveniently due to the fact that the three-dimensional scene image of the ship is more visual, so that the experience of the driver is improved.
Referring to fig. 1, fig. 1 is a schematic flowchart of a ship scene restoration and positioning method based on a camera and edge calculation according to an embodiment of the present application, including:
and step S11, acquiring a plurality of images to be processed.
The images to be processed comprise images of the shore shot by a camera on the ship and images of the ship shot by the camera on the shore. For example, images of the shore captured by a plurality of cameras on the ship and images of the ship captured by a plurality of cameras on the shore may be used. In the in-service use process, the camera on the boats and ships can include the multiunit camera, and this multiunit camera can be installed with the same or different positions of boats and ships. For example, referring to fig. 2a and 2b, the riparian camera 101 comprises two sets of cameras 1011 and 1012, and the onboard camera 102 comprises 1021, 1022, 1023, 1024, 1025, 1026.
The ship scene restoration and positioning method based on the camera and the edge calculation in the embodiment of the application can be implemented through an intelligent terminal, and the specific intelligent terminal can be a computer or a server and the like. In the actual use process, the intelligent terminal can be installed on a ship or on the shore.
And S12, analyzing the multiple images to be processed by utilizing a computer vision technology to obtain targets in the multiple images to be processed and distances between the targets and a camera for shooting the targets.
Wherein the targets include ships and shore. The method comprises the steps of analyzing a plurality of images to be processed by utilizing a computer vision technology, and analyzing the plurality of images to be processed by a pre-trained deep learning neural network to obtain targets in the plurality of images to be processed and distances between the targets and a camera for shooting the targets.
The training process of the deep learning neural network may include:
1. acquiring a ship image sample in a channel through a bank camera, and acquiring a channel bank image sample through a shipborne camera;
2. preprocessing an image sample, including gray processing, image filtering, translation correction, angle correction and data enhancement;
3. labeling a target object in the image sample;
4. and taking the marked image samples as a training data set, and training to obtain a plurality of deep learning neural networks.
In the actual use process, a plurality of images to be processed are analyzed by using a computer vision technology, and the ship and river bank images shot by a plurality of cameras at different angles and heights can be input into the trained deep learning neural network to obtain ship and river bank target identification results at different angles and heights.
And S13, synthesizing a plurality of images to be processed according to the target and the distance between the target and a camera for shooting the target to obtain a three-dimensional scene image of the ship.
The synthesis of the plurality of images to be processed can be realized by a method, such as a three-dimensional reconstruction algorithm of DynamicFusion, bundleFusion and the like, according to the target and the distance between the target and a camera for shooting the target.
The images of the shore shot by the plurality of cameras on the ship and the images of the ship shot by the plurality of cameras on the shore can be images of the shore shot by at least three cameras on the ship and images of the ship shot by at least three cameras on the shore. When a plurality of images to be processed are synthesized according to the target and the distance between the target and the camera shooting the target, the position of the target can be determined by a triangulation method according to the position of each camera and the distance between the target and the camera shooting the target, so that the plurality of images to be processed are synthesized, and the three-dimensional scene image of the ship is obtained.
After the three-dimensional scene image is generated, the display of the three-dimensional scene image may be performed by a display device installed on a ship or on the shore. The display device can be in wireless or wired communication with the intelligent terminal, for example, when the display device is located on a ship and the intelligent terminal is located on the shore, the generated three-dimensional scene image can be transmitted to the display device through the intelligent terminal through wireless communication to be displayed.
Therefore, according to the ship scene restoration and positioning method based on the camera and the edge calculation, the multiple images to be processed can be synthesized to obtain the three-dimensional scene image of the ship according to the target and the distance between the target and the camera for shooting the target, and the ship driver can sense the channel and the surrounding environment of the ship more conveniently and conveniently due to the fact that the three-dimensional scene image of the ship is more visual, so that the experience of the driver is improved.
Optionally, referring to fig. 3, in step S12, analyzing the multiple images to be processed by using a computer vision technique to obtain the targets in the multiple images to be processed and the distances between the targets and the cameras for capturing the targets, including:
and step S121, preprocessing a plurality of images to be processed by utilizing a computer vision technology.
Optionally, the preprocessing is performed on the multiple images to be processed by using a computer vision technology, and includes: carrying out translation correction and angle correction on a plurality of images to be processed by utilizing a computer vision technology; and carrying out filtering and denoising treatment on the corrected image.
For example, when a plurality of captured images to be processed are images captured in the form of a video, the images may be subjected to angle offset and translation offset correction by a correction algorithm according to the correlation between the frames before and after the captured images; and carrying out gray level processing and image filtering on the shot image to eliminate the noise of the image recognition effect caused by the influence of the shot image caused by the shaking of the ship body.
And step S122, inputting the preprocessed images into a plurality of pre-trained deep learning network models for target recognition to obtain targets in a plurality of images to be processed recognized by the models and recognition distances between the targets recognized by the models and a camera for shooting the targets.
And step S123, carrying out weighted summation on the identification distance between the target identified by each model and the camera shooting the target through a preset balance function to obtain the distance between the target and the camera shooting the target.
For example, according to the pixel position of the target in the image, the distance 1 between the target and the camera is calculated by a triangulation technology; obtaining the distance 2 between the target and the camera through a monocular depth estimation technology according to the pixel position of the target in the image; and carrying out weighted sum on the distance 1 and the distance 2 to obtain more accurate distance between the target object and the camera. The weighting and the used weighting parameters can be calculated according to the height of the camera during shooting, shooting angles, distortion parameters, focal lengths, height differences and angle differences of multiple images during shooting to obtain each image and the weighting parameters; and carrying out weighted operation on the distances between the multiple groups of targets and the camera through the weight parameters to obtain a high-precision positioning result.
Therefore, by the method of the embodiment of the application, the recognition distances between the target recognized by each model and the camera shooting the target can be weighted and summed through the preset balance function, and the distance between the target and the camera shooting the target is obtained, so that the precision of distance recognition can be improved, and the synthesis of the three-dimensional graph can be conveniently carried out according to the recognized distances.
Referring to fig. 4, step S122 inputs the preprocessed image into a plurality of pre-trained deep learning network models for target recognition, so as to obtain targets in a plurality of images to be processed recognized by each model and recognition distances between the targets recognized by each model and a camera for shooting the targets, where the steps include:
and step S1221, inputting the preprocessed images into a plurality of pre-trained deep learning network models for target recognition to obtain targets in a plurality of images to be processed, which are recognized by the models.
Step S1222 obtains pixel positions of the target in each of the multiple to-be-processed images according to the target in the multiple to-be-processed images identified by each model.
Acquiring pixel positions of a target in each graph to be processed, and performing data enhancement on a ship image and a river bank image; and inputting the image into a pre-trained deep learning neural network for target recognition to obtain the pixel position of the target object in the image.
And step S1223, obtaining a first identification distance between the target and a camera for shooting the target by a triangulation algorithm according to the pixel position.
Step S1224, obtaining a second recognition distance between the target and the camera shooting the target through a monocular depth estimation algorithm according to the pixel position.
The triangulation positioning technology is based on coordinate transformation and a geometric principle, utilizes imaging equipment to shoot a target object, establishes an imaging equipment coordinate system, calculates the target object through a similar triangle to obtain a target distance in a camera coordinate system, and obtains the target distance in a real world coordinate system through coordinate transformation.
The monocular depth estimation technology is based on a deep learning recognition principle, estimates the depth of an image where a target is located, and calculates to obtain the distance of the target in a real world target system.
Step S123 performs weighted summation on the recognition distance between the target recognized by each model and the camera shooting the target through a preset balance function, to obtain the distance between the target and the camera shooting the target, including:
and step S1231, carrying out weighted summation on the first identification distance and the second identification distance through a preset balance function to obtain the distance between the target and a camera for shooting the target.
For example, the step of acquiring the distance between the object and the camera that photographs the object includes:
1. and carrying out external parameter measurement and internal parameter acquisition on the plurality of river bank cameras and the shipborne camera to obtain key information required by calculation such as camera distortion parameters, focal length and the like.
2. Establishing conversion and mapping of a camera coordinate system to a real coordinate system according to the camera parameter information;
3. acquiring a target identification output result, converting coordinates of a target in an image into real world coordinates, predicting by adopting a triangulation technology to obtain a real distance 1 between a target object and a camera, and predicting by adopting a monocular depth estimation technology to obtain a real distance 2;
4. and comprehensively calculating the calculated real distance 1 and real distance 2 through the weight parameters to obtain the final predicted target distance.
Referring to fig. 5, after the step S12 analyzes the multiple images to be processed by using the computer vision technology to obtain the targets in the multiple images to be processed and the distances between the targets and the cameras for capturing the targets, the method further includes:
and S14, acquiring a two-dimensional image of the current channel scene.
The current channel scene two-dimensional image is an image obtained by mapping calculation through the current channel scene image in advance.
And S15, performing distance matching on the targets according to the targets in the multiple to-be-processed images and the distances between the targets and the camera for shooting the targets to obtain a proportional scale.
And S16, marking a target in the two-dimensional image of the current channel scene according to the scale to obtain a two-dimensional scene image of the ship.
For example, target identification and detection results of ships and river banks are obtained; the edge calculation center performs scene mapping calculation on the channel by using large-scale calculation resources, maps the whole scene information of the channel into a two-dimensional plane image and constructs a channel scene two-dimensional image; according to the target identification and detection results, distance matching is carried out on the target ship, the position of the target ship in the channel scene is obtained through calculation of a calculation center, and the ship position is drawn in the channel scene two-dimensional image according to a scale; and the edge calculation center performs target extraction and mapping on the synthesized three-dimensional image, and draws a target image in the two-dimensional image of the airway scene according to a scale.
Therefore, by the method, the target can be calculated and marked in the two-dimensional image of the current channel scene according to the scale, the two-dimensional scene image of the ship is obtained, and a driver can drive the ship according to the two-dimensional scene image of the ship.
Referring to fig. 6, fig. 6 is an example diagram of a ship scene restoration and positioning method based on camera and edge calculation provided by this application example, including:
s61, acquiring ship images, river bank images, water surfaces around the ship and ship body images which are shot by a plurality of groups of cameras at different positions;
step S62, correcting the picture of each group of images to obtain corrected images;
s63, carrying out target identification on the corrected multiple groups of ship images and river bank images to obtain pixel positions of the multiple groups of targets in the images;
step S64, calculating the distance between the target of each group of images and the camera according to the pixel positions of the multiple groups of targets in the images;
step S65, carrying out comprehensive calculation according to the distances between the multiple groups of image targets and the camera to carry out high-precision positioning;
s66, synthesizing a plurality of groups of images of the water surface and the hull around the ship to establish a ship running three-dimensional image;
s67, carrying out target extraction and mapping on the ship running three-dimensional image and the positioning result to construct a ship running two-dimensional image;
and step S68, displaying the ship running three-dimensional image and the two-dimensional image.
Referring to fig. 7, fig. 7 is a diagram of an example of a ship scene restoration and positioning system based on camera and edge calculation provided by this application example, including:
and the river bank camera 101 is used for shooting a plurality of ship images.
Specifically, the camera 101 may be configured to capture images of the water surface and the hull around the ship at different angles and directions to obtain at least four sets of images, where the images include the water area around the ship and the hull target, and then send the images to the video synthesis device 104 of the vision processing device.
The onboard camera 102 is used for shooting a plurality of river bank images, water surface around the ship and hull images.
Specifically, the camera 102 may be configured to capture channel bank images at different heights, angles, and directions to obtain at least two sets of images, where the images include channel bank targets and are then sent to the vision processing device 103.
And the vision processing equipment 103 is used for carrying out image correction on the images shot by the ship-mounted camera, calculating the distances between targets in the images and the camera, and carrying out high-precision positioning on the distance between the ship and the river bank by comprehensively calculating a weight function.
Specifically, the vision processing device 103 may be configured to perform image correction on an image captured by the camera 102, perform target recognition on the image captured by the camera 101 and the corrected image, calculate a target distance according to a position of a target in the image, perform comprehensive calculation on the target distances calculated by the multiple cameras to obtain a distance between the ship and a river bank, and then send the distance to the scene reconstruction device 105.
And the video synthesis device 104 is used for synthesizing the ship running three-dimensional images by using the plurality of ship surrounding water surface and ship body images.
Specifically, the video synthesis device 104 may be configured to perform video synthesis on the images of the water area and the hull around the ship captured by the camera 102, create a three-dimensional stereo image of the ship, and then send the three-dimensional stereo image to the scene reconstruction device 105 and the display device 106.
And the scene reconstruction equipment 105 is used for carrying out scene mapping on the ship driving three-dimensional image and positioning to construct a ship driving two-dimensional image.
Specifically, the scene reconstruction device 105 may be configured to receive the ship positioning distance calculated by the video synthesis device 103 and the ship-driving three-dimensional stereo image synthesized by the video synthesis device 104, extract detailed target information of the three-dimensional stereo image, draw a two-dimensional image of the ship driving in the channel by combining the ship positioning distance and the channel scene, and then send the two-dimensional image to the display device 106.
And the display device 106 is used for displaying the three-dimensional stereoscopic image and the two-dimensional image.
Specifically, the display device 106 may be configured to receive the three-dimensional stereo image synthesized by the video synthesis device 104 and the ship-driving two-dimensional image drawn by the scene reconstruction device 105 for display.
In a second aspect of the embodiments of the present application, there is provided a system for restoring and positioning a ship scene based on camera and edge calculation, referring to fig. 8, the system includes:
a shore camera 801 for shooting an image of the ship;
a ship camera 802 for shooting images on shore;
the edge calculation device 803 is configured to obtain a plurality of images to be processed, where the plurality of images to be processed include images of a bank captured by a camera on the ship and images of the ship captured by the camera on the bank; analyzing the plurality of images to be processed by utilizing a computer vision technology to obtain targets in the plurality of images to be processed and distances between the targets and a camera for shooting the targets, wherein the targets comprise ships and shores; and synthesizing the plurality of images to be processed according to the target and the distance between the target and a camera for shooting the target to obtain a three-dimensional scene image of the ship.
Optionally, the system further includes a display device, configured to display the obtained three-dimensional scene image of the ship.
Optionally, the edge computing device is further configured to perform preprocessing on the multiple images to be processed by using a computer vision technology; inputting the preprocessed images into a plurality of pre-trained deep learning network models for target recognition to obtain targets in a plurality of images to be processed recognized by the models and recognition distances between the targets recognized by the models and a camera for shooting the targets; and carrying out weighted summation on the recognition distance between the target recognized by each model and the camera shooting the target through a preset balance function to obtain the distance between the target and the camera shooting the target.
Optionally, the edge computing device is further configured to perform translation correction and angle correction on the multiple images to be processed by using a computer vision technology; and carrying out filtering and denoising treatment on the corrected image.
Optionally, the edge computing device is further configured to input the preprocessed image into a plurality of pre-trained deep learning network models to perform target recognition, so as to obtain targets in a plurality of to-be-processed images recognized by the models; acquiring pixel positions of the target in each image to be processed according to the target in the images to be processed identified by each model; according to the pixel position, obtaining a first identification distance between a target and a camera for shooting the target through a triangulation algorithm; according to the pixel position, obtaining a second identification distance between the target and a camera for shooting the target through a monocular depth estimation algorithm; and carrying out weighted summation on the first identification distance and the second identification distance through a preset balance function to obtain the distance between the target and a camera for shooting the target.
Optionally, the edge computing device is further configured to obtain a current channel scene two-dimensional image, where the current channel scene two-dimensional image is an image obtained by performing mapping calculation on the current channel scene image in advance; according to the targets in the multiple images to be processed and the distances between the targets and a camera for shooting the targets, performing distance matching on the targets to obtain a proportional scale; and marking a target in the two-dimensional image of the current channel scene according to the scale of the proportion to obtain a two-dimensional scene image of the ship.
It can be seen that through the ship scene restoration and positioning system based on camera and edge calculation of this application embodiment, can be through according to the target and the distance between the target and the camera of shooting this target, it is right many images to be handled are synthesized, obtain the three-dimensional scene image of boats and ships, because the three-dimensional scene image of this boats and ships is more directly perceived, the judgement of the channel and the surrounding environment of perception boats and ships that can be convenient for more that the ship pilot is convenient to improve pilot's experience.
An embodiment of the present application further provides an electronic device, as shown in fig. 9, including a processor 901, a communication interface 902, a memory 903 and a communication bus 904, where the processor 901, the communication interface 902, and the memory 903 complete mutual communication through the communication bus 904, and the memory 903 is used for storing a computer program;
the processor 901 is configured to implement the following steps when executing the program stored in the memory 903:
acquiring a plurality of images to be processed, wherein the images to be processed comprise images of a shore shot by a camera on a ship and images of the ship shot by the camera on the shore;
analyzing the plurality of images to be processed by utilizing a computer vision technology to obtain targets in the plurality of images to be processed and distances between the targets and a camera for shooting the targets, wherein the targets comprise the ship and the shore;
and synthesizing the plurality of images to be processed according to the target and the distance between the target and a camera for shooting the target to obtain a three-dimensional scene image of the ship.
The communication bus mentioned in the electronic device may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown, but this does not mean that there is only one bus or one type of bus.
The communication interface is used for communication between the electronic equipment and other equipment.
The Memory may include a Random Access Memory (RAM) or a Non-Volatile Memory (NVM), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the processor.
The Processor may be a general-purpose Processor, including a Central Processing Unit (CPU), a Network Processor (NP), and the like; but also Digital Signal Processors (DSPs), application Specific Integrated Circuits (ASICs), field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components.
In yet another embodiment provided by the present application, a computer-readable storage medium is further provided, in which a computer program is stored, and the computer program, when executed by a processor, implements the steps of any of the above-mentioned camera and edge calculation-based ship scene restoration and positioning methods.
In yet another embodiment provided by the present application, there is also provided a computer program product containing instructions which, when run on a computer, cause the computer to perform any of the above-described camera and edge calculation based methods for vessel scene restoration and positioning.
In the above embodiments, all or part of the implementation may be realized by software, hardware, firmware, or any combination thereof. When implemented in software, it may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the application to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid State Disk (SSD)), among others.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrases "comprising a," "8230," "8230," or "comprising" does not exclude the presence of additional like elements in a process, method, article, or apparatus that comprises the element.
All the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on differences from other embodiments. In particular, for the system, the electronic device, the storage medium, and the computer program product embodiment, since they are substantially similar to the method embodiment, the description is simple, and the relevant points can be referred to the partial description of the method embodiment.
The above description is only for the preferred embodiment of the present application and is not intended to limit the scope of the present application. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application are included in the scope of protection of the present application.
Claims (7)
1. The ship scene restoration and positioning method based on the camera and the edge calculation is characterized by comprising the following steps of:
acquiring a plurality of images to be processed, wherein the images to be processed comprise images of a shore shot by a camera on a ship and images of the ship shot by the camera on the shore;
analyzing the multiple images to be processed by utilizing a computer vision technology to obtain targets in the multiple images to be processed and distances between the targets and a camera for shooting the targets, wherein the targets comprise the ship and the shore;
synthesizing the plurality of images to be processed according to the target and the distance between the target and a camera for shooting the target to obtain a three-dimensional scene image of the ship;
the analyzing the multiple images to be processed by using the computer vision technology to obtain the targets in the multiple images to be processed and the distances between the targets and the camera shooting the targets comprises the following steps: preprocessing the plurality of images to be processed by utilizing a computer vision technology; inputting the preprocessed images into a plurality of pre-trained deep learning network models for target recognition to obtain targets in the images to be processed recognized by the models and recognition distances between the targets recognized by the models and a camera for shooting the targets; weighting and summing the recognition distance between the target recognized by each model and the camera shooting the target through a preset balance function to obtain the distance between the target and the camera shooting the target;
the method for inputting the preprocessed image into a plurality of pre-trained deep learning network models for target recognition to obtain targets in the plurality of images to be processed recognized by the models and recognition distances between the targets recognized by the models and a camera for shooting the targets comprises the following steps: inputting the preprocessed images into a plurality of pre-trained deep learning network models for target recognition to obtain targets in the plurality of images to be processed, which are recognized by the models; acquiring pixel positions of the target in the multiple images to be processed, which are identified by the models, according to the target in the multiple images to be processed; according to the pixel position, obtaining a first identification distance between the target and a camera for shooting the target through a triangulation algorithm; according to the pixel position, obtaining a second identification distance between the target and a camera for shooting the target through a monocular depth estimation algorithm; the obtaining the distance between the target and the camera shooting the target by performing weighted summation on the recognition distance between the target recognized by each model and the camera shooting the target through a preset balance function includes: and carrying out weighted summation on the first identification distance and the second identification distance through a preset balance function to obtain the distance between the target and a camera for shooting the target.
2. The method of claim 1, wherein the pre-processing the plurality of images to be processed using computer vision techniques comprises:
carrying out translation correction and angle correction on the multiple images to be processed by utilizing a computer vision technology;
and carrying out filtering and denoising treatment on the corrected image.
3. The method of claim 1, wherein after analyzing the plurality of images to be processed by using the computer vision technique to obtain the target in the plurality of images to be processed and the distance between the target and the camera capturing the target, the method further comprises:
acquiring a current channel scene two-dimensional image, wherein the current channel scene two-dimensional image is an image obtained by mapping calculation through the current channel scene image in advance;
according to the targets in the multiple images to be processed and the distance between the target and a camera for shooting the target, performing distance matching on the target to obtain a proportional scale;
and marking the target in the two-dimensional image of the current channel scene according to the proportional scale to obtain a two-dimensional scene image of the ship.
4. Ship scene restoration and positioning system based on camera and edge calculation is characterized in that the system comprises:
the bank camera is used for shooting an image of the ship;
the ship camera is used for shooting images of the shore;
the system comprises edge computing equipment, a processing unit and a processing unit, wherein the edge computing equipment is used for acquiring a plurality of images to be processed, and the images to be processed comprise images of a shore shot by a camera on a ship and images of the ship shot by the camera on the shore; analyzing the plurality of images to be processed by utilizing a computer vision technology to obtain targets in the plurality of images to be processed and distances between the targets and a camera for shooting the targets, wherein the targets comprise the ship and the shore; synthesizing the plurality of images to be processed according to the target and the distance between the target and a camera for shooting the target to obtain a three-dimensional scene image of the ship;
the edge computing device is also used for preprocessing the images to be processed by utilizing a computer vision technology; inputting the preprocessed images into a plurality of pre-trained deep learning network models for target recognition to obtain targets in the images to be processed recognized by the models and recognition distances between the targets recognized by the models and a camera for shooting the targets; weighting and summing the recognition distance between the target recognized by each model and the camera shooting the target through a preset balance function to obtain the distance between the target and the camera shooting the target;
the edge computing device is further configured to input the preprocessed images into a plurality of pre-trained deep learning network models for target recognition, so as to obtain targets in the plurality of to-be-processed images recognized by the models; acquiring pixel positions of the target in the multiple images to be processed, which are identified by the models, according to the target in the multiple images to be processed; according to the pixel position, obtaining a first identification distance between the target and a camera for shooting the target through a triangulation algorithm; according to the pixel position, obtaining a second identification distance between the target and a camera for shooting the target through a monocular depth estimation algorithm; and carrying out weighted summation on the first identification distance and the second identification distance through a preset balance function to obtain the distance between the target and a camera for shooting the target.
5. The system of claim 4,
the edge computing equipment is also used for carrying out translation correction and angle correction on the multiple images to be processed by utilizing a computer vision technology; and carrying out filtering and denoising treatment on the corrected image.
6. An electronic device is characterized by comprising a processor, a communication interface, a memory and a communication bus, wherein the processor and the communication interface are used for realizing mutual communication by the memory through the communication bus;
a memory for storing a computer program;
a processor for implementing the method steps of any of claims 1 to 3 when executing a program stored in the memory.
7. A computer-readable storage medium, characterized in that a computer program is stored in the computer-readable storage medium, which computer program, when being executed by a processor, carries out the method steps of any one of the claims 1-3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110111326.1A CN112907728B (en) | 2021-01-27 | 2021-01-27 | Ship scene restoration and positioning method and system based on camera and edge calculation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110111326.1A CN112907728B (en) | 2021-01-27 | 2021-01-27 | Ship scene restoration and positioning method and system based on camera and edge calculation |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112907728A CN112907728A (en) | 2021-06-04 |
| CN112907728B true CN112907728B (en) | 2023-04-11 |
Family
ID=76120646
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110111326.1A Active CN112907728B (en) | 2021-01-27 | 2021-01-27 | Ship scene restoration and positioning method and system based on camera and edge calculation |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112907728B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114693763B (en) * | 2022-03-15 | 2023-04-07 | 武汉理工大学 | Method, system and device for constructing three-dimensional model of channel ship and storage medium |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2020125229A1 (en) * | 2018-12-20 | 2020-06-25 | 深圳云天励飞技术有限公司 | Feature fusion method and apparatus, and electronic device and storage medium |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107818326B (en) * | 2017-12-11 | 2018-07-20 | 珠海大横琴科技发展有限公司 | A kind of ship detection method and system based on scene multidimensional characteristic |
| CN109285389B (en) * | 2018-09-18 | 2020-08-14 | 广州中国科学院沈阳自动化研究所分所 | Automatic berthing system and method for unmanned ship |
| CN111524392B (en) * | 2020-04-22 | 2022-05-06 | 智慧航海（青岛）科技有限公司 | Comprehensive system for assisting intelligent ship remote driving |
| CN111986312B (en) * | 2020-08-14 | 2022-04-22 | 连云港引航站 | Ship track drawing method, terminal device and storage medium |
| CN112001971A (en) * | 2020-09-22 | 2020-11-27 | 北京神州泰岳软件股份有限公司 | Object identification method and system |
-
2021
- 2021-01-27 CN CN202110111326.1A patent/CN112907728B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2020125229A1 (en) * | 2018-12-20 | 2020-06-25 | 深圳云天励飞技术有限公司 | Feature fusion method and apparatus, and electronic device and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 李敏 ; 金晶 ; .基于VR技术的舰船航行环境快速重构研究.舰船科学技术.2020,(14),全文. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112907728A (en) | 2021-06-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108961327B (en) | Monocular depth estimation method and device, equipment and storage medium thereof | |
| CN107330439B (en) | Method for determining posture of object in image, client and server | |
| CN107516322B (en) | Image object size and rotation estimation calculation method based on log polar space | |
| CN111723691B (en) | Three-dimensional face recognition method and device, electronic equipment and storage medium | |
| CN109300151B (en) | Image processing method and device and electronic equipment | |
| EP3346445A1 (en) | Methods and devices for extracting an object from a video sequence | |
| CN112509036B (en) | Pose estimation network training and positioning method, device, equipment and storage medium | |
| CN111047634B (en) | Scene depth determination method, device, equipment and storage medium | |
| Fabio et al. | Performance evaluation of a low-cost stereo vision system for underwater object detection | |
| CN108229494B (en) | Network training method, processing method, device, storage medium and electronic equipment | |
| CN112907728B (en) | Ship scene restoration and positioning method and system based on camera and edge calculation | |
| CN110188179B (en) | Voice directional recognition interaction method, device, equipment and medium | |
| CN115482556A (en) | Method for key point detection model training and virtual character driving and corresponding device | |
| CN113256697B (en) | Three-dimensional reconstruction method, system, device and storage medium for underwater scene | |
| US11948344B2 (en) | Method, system, medium, equipment and terminal for inland vessel identification and depth estimation for smart maritime | |
| Olson | Adaptive-scale filtering and feature detection using range data | |
| CN117132649A (en) | Ship video positioning method and device for artificial intelligent Beidou satellite navigation fusion | |
| CN114463430B (en) | Ocean search and rescue system and method based on image processing | |
| CN115164900A (en) | Omnidirectional camera based visual aided navigation method and system in urban environment | |
| CN111311652B (en) | Regression correction network-based sonar image registration method and system | |
| CN114782529A (en) | High-precision positioning method and system for line grabbing point of live working robot and storage medium | |
| CN114648686A (en) | Method, system and device for identifying backlight water surface target by fusing laser point cloud and RGB image data | |
| Kallasi et al. | Object detection and pose estimation algorithms for underwater manipulation | |
| CN107802468B (en) | Blind guiding method and blind guiding system | |
| CN114761998A (en) | Method for determining depth from image through adaptive learning of neural network and system thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |