JP4482623B2 - Extraction method of ships from time series voyage images - Google Patents

Description

  The present invention relates to a method for extracting a ship from a time-series voyage image, and in particular, processes an image photographed during voyage to extract other sailing ships and directly confirms their presence.

  International efforts are being made for safe navigation of ships. At the 73rd IMO (International Maritime Organization) Maritime Safety Committee held in December 2000, AIS (Automatic Identification System): An amendment to the Maritime Life Safety Convention (SOLAS Convention), which included the mandatory installation of the automatic identification device, etc., was adopted and entered into force in July 2002. AIS is a wireless facility that automatically sends and receives information such as the ship's position, speed, and direction of travel. By communicating this information between ships and between land, ship collision prevention, navigation management, etc. It is expected to exert an excellent effect.

  However, since the AIS will be installed in stages over the next few years, the effect cannot be expected immediately. Basically, domestic cargo ships of less than 300 GT (Gross Tonage) are not obligated to install AIS. Therefore, in the sea area where ships of less than 300GT without AIS frequently come and go, other safe navigation support methods must be used in the future. Furthermore, it is indispensable to confirm the safety of the navigator visually as before, regardless of whether or not the AIS is operated when passing through a traffic route, bay, or narrow channel. Based on this situation, a system for supporting the lookout has been studied (see Non-Patent Documents 1, 2 and 3 below). However, there are few proposals for navigation support systems that are installed on actual ships and have been studied from the viewpoint of visual support.

The present invention processes images captured during navigation, extracts other navigational ships, confirms their presence directly, and constructs complex information through cooperation with AIS and radar to provide a higher level of safety. Further, the present invention relates to a method for extracting a ship from a time-series voyage image that contributes to efficient navigation.
Conventionally, research for extracting sailing ships based on time-series images has been performed (see Non-Patent Document 4 below). According to this method, the camera is fixed on the land, and the movement of the ship is tracked using both the optical flow and the template. Since this method uses a fixed camera as an image capturing environment, the ship can be extracted relatively stably on the assumption that the background is fixed. However, it is difficult to apply as a technique for extracting other ships from a sailing ship whose background is variable.

In addition, there is a method (see Non-Patent Document 5 below) in which a camera is installed on a berthed ship and an image is acquired. According to this method, after obtaining the optical flow of the entire image, the camera motion was estimated from the motion vector histogram and the background was removed. Furthermore, the position of the ship is estimated by projecting a time difference image. Since this method handles the optical flow of the entire screen, the calculation cost is high and there is room for examination in terms of the accuracy of extracting a ship. Therefore, research on a simpler and more robust method for extracting a ship is desired.
Saima Imazu, “Fundamental research on lookout automation,” Tokyo MOL Research Report (Natural Sciences) No. 42 (December 1991) Satoshi Kurokawa, Kurama Imazu, “Study on Automated Collection of Visual Information on Other Ships,” Japanese Society of Navigation Research, 82, pp. 85-91, 1990 Takuya Ono, Sakuma Imazu, “On the lookout image segmentation method”, Transactions of the Japan Navigation Society 89, pp. 169-177, 1993 Akio Satori, Masami Oshima, Experiments on Detection of Navigation Ships from Navigation Environment Images, Journal of the Japan Navigation Society 97, pp. 213-220, 1997 Mitsuo Suzuki, Masami Oshima, Detection Experiment of Sailing Ships Using Time Series Images from Shipboard, Proceedings of Japan Society for Navigation Studies, No.100, pp. 13-19, 1999 C. K. Chui, An Introduction to Wavelets, Academic Press, Boston, 1992. D. L. Donoho: “De-noising by Soft-thresholding,” IEEE Trans. Information Theory, Vol. 41, no. 3, pp. 613-627, 1995 Yu Tsuchiya, “Image Processing”, Corona, 4.2.2 Labeling Equivalent Table Method, pp. 92-93, 1999 Morio Onoe, Norihiko Maeda, Yuu Saito, “Superposition of Images Using Residual Sequential Test”, Information Processing, vol. 17, no. 7, pp. 634-640, 1976

  In general, if a background is constant, a moving object can be extracted by a background subtraction method. However, the background subtraction method cannot be used when the background is variable. For this reason, an active background subtraction method has been proposed. However, in the active background subtraction method, it is necessary to attach a sensor to the camera in order to measure the movement of the camera, and it is necessary to follow the procedure of applying the background subtraction method after compensating for the background motion. In addition, there is a problem that it is difficult to completely correct error distortion caused by a shift between the rotation center of the camera and the optical center.

  In view of the above situation, an object of the present invention is to provide a method for extracting a ship from a time-series voyage image that can more easily and robustly extract a ship.

In order to achieve the above object, the present invention provides
[1] A method for extracting a ship from a time-series voyage image, in which a ship region is estimated on an arbitrary image of a time-series voyage image by a spatial differentiation process and a grouping process, and a frame is formed from the image subjected to the grouping process. A ship is extracted by selecting an image on a time-series image after determining the image and comparing the selected image with the ship region .

[2] In the method for extracting a ship from the time-series voyage image as described in [1] above, the degree of coincidence is compared at the time of collation, and a region with a higher degree of coincidence is determined as a ship.
[3] In the method for extracting a ship from the time-series voyage image described in [2], the movement vectors of the estimated areas are further compared, and an area that moves more linearly is determined as a ship.

According to the present invention, the following effects can be achieved.
(A) No calibration is required at a complicated sensor or installation location. Therefore, the camera can be installed at any position on the bridge.
(B) Not only the extraction of the ship but also the buoy floating on the sea, the recognition of the anchored ship, and the land included in the background can be separated.

  (C) By exchanging information with radar and other nautical instruments (AIS, etc.), composite information can be constructed and ships that contribute to more advanced safe and efficient operations can be confirmed.

In the method of extracting a ship from a time-series voyage image, a ship region is estimated by spatial differentiation processing and grouping processing on an arbitrary image of the time-series voyage image, and a frame is determined from the image subjected to the grouping processing. An image on the time series image is selected, and a ship is extracted by comparing the selected image with the ship region . Therefore, a ship can be extracted more simply and robustly.

Hereinafter, embodiments of the present invention will be described in detail.
FIG. 1 is a flowchart of a method for extracting a ship from a time-series voyage image showing an embodiment of the present invention.
First, a region of a ship is estimated in an arbitrary image on a time series image. Spatial differentiation (step S1) is employed for this estimation. The results of spatial differentiation are grouped (step S2), and small groups with only one pixel are deleted as noise. Thereafter, the grouped area is collated with another image with an appropriate time interval (step S3).

At this time, since there is an influence due to the movement of the own ship and the movement of the other ship, collation by proximity scanning is performed within a certain range based on the coordinates of the original image. The matching degree for each group is determined by this collation, and a group having a high matching degree is determined as a ship candidate (step S4). A group having a low degree of coincidence is regarded as a wave and excluded (step S5).
Further, the result of collation between the original image indicating the grayscale image corresponding to the grouped region and the image to be collated is tabulated as a vector, a movement vector is obtained (step S6), and if the distance from the collation position is small as a feature thereof If it is larger than the ship (step S8), it is determined as a wave (step S7). By sequentially performing this process on the time-series image, it is possible to remove the wave and extract only the ship. In preparation for the emergence of new ships, the ship area estimation is reset every fixed period.

Hereinafter, a method for extracting a ship from a time-series voyage image will be described in detail.
[1] Ship contour extraction [Step S1]
The outline of the ship estimation area is extracted by the spatial differential method. This time, a Kirsch operator was used as the differential operator.
[2] Grouping [Step S2]
Since the result of the spatial differentiation is just a group of pixels as it is, the adjacent pixel groups are grouped and individually labeled (Non-Patent Document 6). The grouped area is estimated to be a ship area. All subsequent processing is performed in units of this group.

It can be assumed that the ship region has a certain number of pixels. Therefore, the minimal region of only one pixel can be removed because it is considered to be noise other than the ship or generated during differentiation. After removing the noise, a group having a large number of pixels is estimated as a ship or a fixed object.
Small waves can be eliminated by setting an appropriate threshold and deleting groups with a small number of pixels. At this time, since there is a possibility that a small floating object floating in the waves may be overlooked, the threshold value is set low (100 or less with respect to the maximum luminance 256). At this stage, the group is not deleted so much, and it is determined that the next collation processing is focused on.

[3] Verification (matching) [Step S3]
A time-series image after properly determining a frame is selected from the images subjected to the grouping process, and the ship region matching (matching) process is performed. Since the navigation environment time-series images are characterized in that there is little change between adjacent images, it is necessary to select a time interval between images when selecting images to be verified. However, if the interval is too long, a demerit that the scanning range of collation extends over a wide range appears. In this method, as a result of processing the experimental data while limiting the scanning range to within 100 pixels in the xy direction, an appropriate image interval was set to 10 frames (after 0.33 seconds).

  When the grouped estimation regions are collated between arbitrary frames, matching is performed at a relatively high rate as expected if the estimation region is a ship region. However, even when the estimation region is a wave, the degree of coincidence may be high if a wave having a shape similar to the sea surface in the vicinity exists. In other words, it will be mistaken for a ship. This is presumably because, when the contour is extracted, the differential value is sorted by the threshold value and the entire screen is binarized, so that the shade information is lost and only the position information is obtained.

Therefore, in order to improve accuracy, a region of the original image (grayscale image) corresponding to the coordinates of the grouped ship estimation regions is cut out and collated with the collated image. In this case, since the grayscale data of the image is collated instead of the binary data, the degree of coincidence of the ship region is further improved and the degree of coincidence of the wave region is lowered. For the verification, a residual sequential detection algorithm (SSDA) was used that speeds up the process by canceling the computation of a candidate that is not expected to correspond (Non-Patent Document 7). In the SSDA method, the residual R is calculated by the following equation (1). (U, v) indicates the position of the estimated area image in the image to be verified. I _{(u, v)} (m, n) is a partial image of the image to be verified, and T (m, n) is an estimated region image of size (TX, TY).

[4] Group Vector Movement Vector Evaluation [Step S6]
Even if the image is processed by the above-described method, the matching degree becomes high depending on the state of the wave at sea, and the wave cannot be completely discriminated. For example, this phenomenon is conspicuous on the sea surface where there is no white wave and there is a gentle swell. Therefore, a method of vectorizing the result of matching and judging by the group movement (flow) tendency was also used.

  A vector from the position O (Xo, Yo) of the group on the original image to the collation position M (u, v) of the image to be collated is converted into a movement vector MV. That is, it is as shown in the following formula (2).

Next, the coordinate position is tabulated for each group, and the movement vector MV is obtained by taking a history of collation positions. However, since the background vibrates due to the shaking of the ship, this process merely observes the relative change on the image. However, when this vector change is observed, in the case of a ship, it is the sum of shaking and movement in the traveling direction, whereas in the case of a wave, the change clearly diverges randomly. The wave can be discriminated with higher accuracy by the difference between the changes.

[5] Time-series continuous verification When the above-described verification is performed continuously between time-series images, waves are removed one after another and only the ship can be extracted. However, the process described in [3] above cannot cope with a case where a ship is off the screen (out of frame) or a new ship enters the screen (in frame). Furthermore, this determination cannot be overturned once a small vessel has been treated as a wave. Therefore, in the present invention, a contour extraction process (step S1) is periodically started, a new estimation area is defined, and matching is continued. Thereby, it can respond to the appearance of a new ship that is easy to overlook.

Next, an experimental environment and a processing environment will be described.
The experimental environment for acquiring the navigation environment time-series images and the processing environment for the captured images are summarized below.
(1) Experimental environment: A CCD camera and a video camera were set up on the Funabashi and Wings of the Tokyo MOL University training ship Kushiro Maru, and the captured images were recorded on a video recorder.

(2) Location of the experiment: Pictures were taken during the voyage of Tokyo Bay, Uraga Channel, Tateyama Bay, and southwest Boso Peninsula.
(3) Experimental equipment Video camera (DCR-TRV7: manufactured by SONY)
CCD camera (CCD-IRIS: made by SONY)
Video recorder (DSR-V10: made by SONY)
Image editor (WSD / 2XTreme: manufactured by ACCOM)
(4) Experiment period: Conducted during the Kushiromaru research voyage from 2000 to 2002.

(5) Situation of experiment: The photographing direction was an arbitrary direction centering on the starboard front of the ship. The shooting direction was changed as appropriate, but the camera was fixed during a series of shootings. Although the magnification was changed depending on the situation, it was fixed during a series of shootings. The focus was fixed at infinity. For the exposure, an automatic selection function of the photographing apparatus was used.
(6) Processing environment: Downloaded from a videotape taken at sea to a UNIX (registered trademark) environment (manufactured by SUN) with an image editor, and processed the image data.

Next, experimental results will be described.
The result of processing the photographed data by the method of the present invention is shown below.
FIG. 2 shows an arbitrary frame of a time-series image taken off Sagami.
In FIG. 2, the ship ahead of the ship's starboard is caught, and at this time, the ship is rolling in the starboard direction.

FIG. 3 shows the result of spatial differentiation of the screen of FIG. 2 in order to extract the contour of the ship. As can be seen from this figure, many contours of waves on the sea surface as well as ships are extracted.
FIG. 4 shows a result of grouping the contour pixels obtained in FIG. 3 and extracting only groups of 100 pixels or more. There are some big waves in addition to ships.
FIG. 5 shows the result of collating this group with the screen after 10 frames (about 0.33 seconds later) from the screen of FIG. A grouped area is surrounded by a rectangular white frame. A white line extending from the center of the frame is a vector to a region that matches as a result of matching on the screen after 10 frames. The ship area vector corresponds to the motion between 10 frames (roll + relative movement), whereas the wave group shows a divergence trend.

Further, FIG. 6 shows the result of collation with the screen 20 frames after the screen of FIG. It can be seen that the ship region is on the extension of the vector of FIG. 5 but the wave group vectors do not match.
Further, FIG. 7 is a processed image obtained by photographing the port side direction in the Uraga Channel. Many waves are grouped together, but their vectors are all diverging. Ships are partly divided and recognized, but can be understood as a group with the same direction of travel. Next, evaluation and consideration of the above test are performed as follows.

[1] Contour Extraction The Kirsch operator used to extract the contour performs differentiation from eight directions, so that the feature of the image can be extracted with high sensitivity.
However, in the present invention, the main purpose is to extract ships navigating the sea, and even if fine sea waves and cloud features are extracted, they become noise and become obstacles when grouping contours. Therefore, the number of extractions must be limited by setting a threshold according to the image. This was the same even when the operator was changed to a less sensitive operator (Sobel, Prewitt, etc.). For this reason, in the present invention, as a result of performing spatial differentiation of the image, it is determined that it is necessary to incorporate a feedback process in which the threshold value is changed according to the amount of extraction and redifferentiation is performed. This process makes it possible to respond flexibly to changes in shooting conditions.

[2] Collation As described above, as a characteristic of the navigation image, there is no significant change in the degree of coincidence in the sea surface collation. In the case where the search range is wide, there was even a case where the degree of coincidence was temporarily higher than that of the ship. Therefore, the wave cannot be clearly removed only by the judgment based on the difference in coincidence. As a result of the experiment, the wave tends to move at random. In other words, the ship takes a fixed orbit in time series, but in the case of waves, the movement tends to diverge randomly. This is considered to be due to the fact that the waves do not take the same shape twice on the time-series image. By verifying this flow difference, a ship and a wave can be distinguished.

In the present invention, the log (history) in which the ship estimation area has moved is tabulated and the determination is made based on the flow divergence.
Table 1 shows the coordinates (xo, yo) of each group in FIG. 4 and the coordinates (x1, y1) of each group after 10 frames (see FIG. 5), and after 20 frames (see FIG. 6). The coordinates (x2, y2) obtained are tabulated and the result of ship determination is shown.

The movement vectors MV1 and MV2 are expressed by the following equations (3) and (4). The distance is calculated from the movement vector 2 * MV1 estimated from FIG. 5 and the vector MV2 to the collation position in FIG.

By performing threshold processing on the distance, most waves can be removed by a single determination. In Table 1, Label5 is a group of ships in the image. Label 724 is determined to be a ship in this determination, but all the waves other than the ship are removed by continuing the process.

[3] Grouping There are cases where ships must be recognized on the background of land on the route and bay. In this case, if the estimated areas are grouped, the ship and the background may be combined. If the ship and the background are integrated, it is not possible to extract only the ship when collation is performed in units of groups. This countermeasure may be solved by subdividing the group into blocks and taking individual flows when collating. In this case, it is necessary to consider the balance between calculation cost and accuracy.

[4] Problem of image magnification The magnification of an image taken in the present invention was arbitrarily set. In actual Funabashi duty work, first of all, a wide range is monitored, and if necessary, it is confirmed with binoculars. An image captured over a wide range at a low magnification may not be extracted by mistake as a wave because the target ship is small. Therefore, in order to confirm a small or distant object, a zoom (enlargement) operation of the camera is necessary. For this reason, in order to apply the present invention to a work on duty for a bridge bridge, an apparatus capable of actively performing camera pan (horizontal rotation) and camera zoom operation is ideal.

As described above, according to the present invention, as a result of processing a real image, it was confirmed that it is effective for ship extraction.
The present invention finally aims at navigation support in cooperation with AIS and radar. For this purpose, necessary information includes the absolute azimuth of the camera optical axis, AIS and radar distance, azimuth information, and the like. As for the optical axis of the camera, there is a method of obtaining an absolute azimuth using the output of GPS or a gyrocompass. In addition, in order to acquire AIS and radar information, it is possible to acquire relative ship position and orientation information by constructing an information exchange interface with these devices.

  In addition, this invention is not limited to the said Example, Based on the meaning of this invention, a various deformation | transformation is possible and these are not excluded from the scope of the present invention.

  The present invention is suitable for extracting relative ship position and direction information more simply and robustly.

It is a flowchart of the extraction method of the ship from the time series voyage image which shows the Example of this invention. It is a figure which shows the arbitrary frames of the time series image concerning this invention. It is a figure which shows the result of carrying out the spatial differentiation of the screen of FIG. 2 concerning this invention. It is a figure which shows the result of having grouped the pixel of the outline obtained in FIG. 3 concerning this invention, and extracting only the group of 100 pixels or more. It is a figure which shows the result collated with the image after 10 frames from the screen of FIG. 2 concerning this invention. It is a figure which shows the result collated with the image after 20 frames from the screen of FIG. 2 concerning this invention. It is a figure which shows the process result in the Uraga waterway concerning this invention.

Claims (3)

(A) Estimate a ship region by spatial differentiation processing and grouping processing on an arbitrary image of time-series voyage images,
(B) selecting an image on a time-series image after defining a frame from the image subjected to the grouping process, and extracting the ship by comparing the selected image with the ship region. A method for extracting a ship from a time-series cruise image.   2. A method for extracting a ship from a time-series voyage image according to claim 1, wherein the degree of coincidence is compared at the time of collation, and a region having a higher degree of coincidence is determined as a ship. Extraction method.   3. The method for extracting a ship from a time-series voyage image according to claim 2, further comprising comparing the movement vectors of the estimated areas, and determining a more linearly moving area as a ship. Ship extraction method.