JP6915812B2 - Warning devices, warning methods, and warning programs for safe navigation - Google Patents

Description

The present invention relates to an alarm device, an alarm method, and an alarm program for safe navigation of a ship.

When constructing works at harbors, coasts, etc., maritime signs, such as buoys and buoys, are installed near the construction area so that vessels navigating the sea can sail safely. However, it is not easy to visually recognize the marine sign from a ship, and the construction period of the marine work is often short, and the marine sign is often not reflected in the nautical chart.

Therefore, it is desirable that the construction management manager monitors the sea near the construction area and issues an alarm to the vessels approaching the construction area. For example, there is a method of installing a radar in a construction area to detect an approaching ship and issue an alarm. There is a method of installing an automatic identification system (AIS) using an international VHF to detect an approaching ship and issue an alarm. However, both methods require wireless technology qualifications to install and operate antennas in order to transmit radio waves.

Therefore, there is a method in which a digital camera is installed in the construction area and an image recognition technology is used to detect a ship approaching the construction area and issue an alarm. For example, Patent Document 1 includes a monitor TV and a surveillance camera, sets a plurality of sensor points for displaying an intrusion detection area on the screen of the TV monitor with a mouse, and detects an intruder in contact with the set sensor points. Surveillance sensors are disclosed.

Japanese Unexamined Patent Publication No. 2009-18896

However, the invention described in Patent Document 1 can only detect that the ship has touched the set sensor point, and issues an alarm until the ship only touches the sensor point and does not approach the construction area. Further, with the conventional image recognition technology, it is not possible to eliminate the generation of white waves on the sea surface and the reflection of sunlight.

The present invention has been made to solve the above-mentioned problems, and it is an object of the present invention to propose a device for issuing an alarm to a ship approaching a construction area so that a ship sailing on the sea can sail safely. ..

In order to achieve the above object, the alarm device according to the present invention includes an image acquisition unit that acquires an image signal of an image including the sea surface from a video camera installed in a construction area, and a predetermined image for each frame from the acquired image signal. An image cutout portion that cuts out rectangular image data having a long side in the horizontal direction as a plurality of monitoring area image data, and binarization that generates image binarization data using a predetermined binarization threshold for the monitoring area image data. A data processing unit, a short-term summary unit that generates short-term summary image data by adding image binarization data over a predetermined addition time, and an image analysis of each short-term summary image data are performed in sequence to correspond to a ship candidate. A long-term summary section that detects the image area and tracks the image of the ship candidate to determine whether or not the ship candidate is likely to be heading for the construction area, and the ship candidate heading for the construction area. It is characterized by having an alarm signal output unit that outputs an alarm signal to an alarm when it is determined that the ship is likely to be a ship.

Further, in the alarm device according to the present invention, the binarization data processing unit includes a background image generation unit that generates background image data from a part of a plurality of monitoring area image data over a first time, and a background image data. Background difference 2 that takes the background difference from the predetermined monitoring area image data after the first time to the second time of the plurality of monitoring area image data and generates the image binarization data using the binarization threshold. It is preferable to have a binarization unit.

Further, in the alarm device according to the present invention, the binarization data processing unit calculates the average value of each of the R, G, and B color data included in the monitoring area image data, and the hue averaging unit and each of the monitoring area image data. It has a color difference binarization unit that generates color difference image data related to the difference between the hue value of the pixel and each average value of the color data, and generates color difference image binarization data using the color difference threshold. Is preferable.

Further, in the alarm device according to the present invention, it is preferable that the image including the sea surface is an image including the boundary between the sea surface and the sky or the sea surface and the land, and the monitoring area image data is the image data including the boundary.

The alarm method according to the present invention is to acquire an image signal of an image including the sea surface from a video camera installed in a construction area, and to obtain a rectangular image having a predetermined image horizontal direction as a long side for each frame from the acquired image signal. The data is cut out as a plurality of monitoring area image data, the image binarization data is generated from the monitoring area image data using a predetermined binarization threshold, and the image binarization data is used for a predetermined addition time. The short-term summary image data is generated by adding the data, and the image analysis of each short-term summary image data is performed in sequence to detect the image area corresponding to the ship candidate, and the image tracking of the ship candidate is performed so that the ship candidate is in the construction area. It determines whether or not the ship is heading toward the construction area, and outputs an alarm signal to the alarm when it is determined that the ship candidate is likely to be heading to the construction area. It is characterized by having.

The alarm program according to the present invention is a program for operating a computer, which acquires an image signal of an image including the sea surface from a video camera installed in a construction area, and a predetermined image for each frame from the acquired image signal. Cutting out rectangular image data having a long side in the horizontal direction as a plurality of monitoring area image data, generating image binarization data using the monitoring area image data using a predetermined binarization threshold, and image binar value. The short-term summary image data is generated by adding the conversion data over a predetermined addition time, and the image analysis of each short-term summary image data is sequentially performed to detect the image area corresponding to the ship candidate and the ship candidate. When it is determined by performing image tracking whether or not the ship candidate is likely to be a ship heading for the construction area, and when it is determined that the ship candidate is likely to be a ship heading for the construction area. It is characterized by outputting an alarm signal to an alarm device and causing a computer to execute the data.

According to the alarm device according to the present invention, it is possible to issue an alarm to a ship approaching the vicinity of the construction area so that the ship sailing on the sea can sail safely.

It is a figure explaining the outline of an example of an alarm device. It is a figure which shows an example of the block composition of an alarm device. It is a figure which shows an example of the processing flow performed by an alarm device. It is a figure explaining an example of statistical background subtraction. It is a figure explaining the effect of taking background subtraction after a sufficient time elapses from the generation of background image data, and (a) takes background subtraction with the previous image as the background image and the image after one frame as the foreground image. It is a figure which shows the example which digitized, and (b) is the figure which shows the example which took the background subtraction and binarized after a lapse of sufficient time from the generation of the background image data. It is a flowchart which shows an example of a long-term summary processing. It is a figure which shows an example of the block composition of an alarm device. It is a figure which shows an example of the processing flow performed by an alarm device.

Hereinafter, the alarm device, the alarm method, and the alarm program according to one aspect of the present disclosure will be described with reference to the drawings. However, it should be noted that the technical scope of the present disclosure is not limited to those embodiments, but extends to the inventions described in the claims and their equivalents. In the following description and figures, components having the same functional configuration are designated by the same reference numerals, so that duplicate description will be omitted.

[Overview of the alarm device according to the first embodiment]
FIG. 1 is a diagram illustrating an outline of an example of an alarm device according to the first embodiment.

The alarm device 1 is installed in a construction area in a harbor, a coast, or the like, and is connected to a video camera 2 and an alarm 3 also installed in the construction area. The video camera 2 photographs the sea surface near the construction area. The video camera 2 is, for example, a color video camera, a high-resolution color camera, a high-speed shooting video camera, and a high-resolution high-speed color video camera. The image taken by the video camera 2 is, for example, the image 4 shown by the broken line. The image 4 is transmitted to the alarm device 1 as an image signal. This is because the alarm device 1 determines whether or not there is a ship approaching the construction area based on the image signal.

When the alarm device 1 determines that there is a ship approaching the construction area, it outputs an alarm signal to the alarm device 3. The alarm device 3 is controlled by the alarm device 1 and, when receiving an alarm signal, issues an alarm to a ship approaching the construction area, for example, a patrol light (registered trademark), a flash light emitter, a siren, and a speaker. The alarm device 1 determines whether or not there is a ship approaching the construction section based on the image signal from the video camera 2, and if there is a ship, the alarm device 3 is used for the approaching ship. It is possible to ensure the safety of the ship by issuing an alarm notifying that the construction section is near.

The range of the image 4 captured by the video camera 2 includes the horizon, that is, the boundary line 41 seen as the boundary between the sea surface and the sky in the vertical direction to the boundary of the construction area, and includes the left and right ends of the construction area in the horizontal direction. It is preferable that it is set as follows. Vessels sailing on the sea can be seen farthest from the construction area near the horizon, because the sea level from near the horizon to the construction area can be monitored. The range of the image 4 may be set to include the vicinity of the left and right ends of the construction area in the lateral direction. Therefore, the video camera 2 is preferably a wide-angle camera. The alarm device 1 determines a ship approaching the construction section based on the image taken by the video camera 2.

When there is land far away from the construction area, the range of image 4 taken by the video camera 2 includes the boundary line visible as the boundary between the sea surface and the land in the vertical direction to the boundary of the construction area, and the construction area. It is preferable to set so as to include the left and right ends of the construction area in the lateral direction. Vessels sailing on the sea can be seen farthest from the land near the opposite shore, because the sea level from the opposite shore to the construction area can be monitored.

For example, the operator of the alarm device 1 confirms the display screen (not shown) of the alarm device 1 to set the shooting range of the video camera 2. For example, the position of the boundary line between the sea surface and the sky that can be visually recognized from the image captured by the video camera 2 changes depending on the weather. Further, the alarm device 1 may automatically identify, for example, the horizon and the boundary of the construction area from the image of the video camera 2 to control the shooting range of the video camera 2. For example, the alarm device 1 can set the horizon and the boundary of the construction area for each boundary where the brightness in the horizontal direction of the image changes significantly.

In determining a ship approaching the construction section, for example, there is a method in which the alarm device 1 analyzes the entire area of the captured image of the video camera 2 to determine whether the ship is approaching the construction section. However, the load of image processing is large, and the processing speed of the alarm device 1 becomes slow.

Therefore, it is preferable that the image area for monitoring whether the ship is approaching the construction section is small. For example, the operator of the alarm device 1 sets the horizontal line, that is, the boundary line 41 seen as the boundary between the sea surface and the sky as the first approach line 41, and sets the rectangular area including the first approach line in the first monitoring area image 42. Next, the operator of the alarm device 1 provides a virtual horizontal second approach line 43 on the sea 2.0 km away from the construction area, and sets a rectangular area including the second approach line in the second monitoring area image 44. do. Further, the operator of the alarm device 1 provides a virtual horizontal third approach line 45 on the sea 1 km away from the construction area, and sets a rectangular area including the third approach line in the third monitoring area image 46. The alarm device 1 stores each set monitoring area image.

The alarm device 1 detects whether or not the ship image is included in each monitoring area image, further determines whether or not the detected ship image is close to the construction section, and the ship is placed in the construction area. When they are approaching, the alarm device 1 operates the alarm device 3 to issue an alarm. Since the load of image processing is reduced, the processing speed of the alarm device 1 is increased. In this embodiment, the number of virtual approach lines is three, but it may be set to two or less or four or more.

In addition, there are static structures such as embankments, caissons, berthed vessels, and buoys on the sea of the harbor as well as vessels in transit. In addition, stationary natural objects, such as reefs, may be exposed. Hereinafter, static structures and stationary natural objects will be referred to as stationary objects. For example, in the first monitoring area image 42 in the image 4 of FIG. 1, a container ship 51 navigating in the lateral direction, a pleasure boat 52 navigating toward the construction area, a buoy 53, and an arm 54 of an anchored crane vessel are shown. Each contains a corresponding image. Further included are images of waves 55 on the sea surface and reflected light 56 of the sun. The alarm device 1 determines whether or not there is a ship approaching the construction section based on the first monitoring area image 42 including various objects, and if there is a ship, the alarm device 3 warns the approaching ship. Is used to output an alarm signal indicating that the construction area is near.

[About video camera]
The video camera 2 is, for example, a high-resolution color video camera capable of capturing an image having a resolution of 3840 × 2160 pixels and 60 frames per second. The coordinates of each pixel of the video camera 2 are represented by (x, y): 1 ≦ x ≦ 3840, 1 ≦ y ≦ 2160. The image data from the video camera 2 is for each pixel, for example, red (R) color data: R (x, y), green (G) color data: G (x, y), and blue (B). It has three color data: B (x, y). Some cameras do not output three color data for all pixels and output every other pixel, but for the sake of simplicity, it is assumed that three color data are output for all pixels in this embodiment. ..

[Alarm device block configuration]
FIG. 2 is a diagram showing an example of a block configuration of the alarm device according to the first embodiment.

The alarm device 1 has a display unit 11, an operation unit 12, and a control unit 13, and each unit is connected by a bus line. The display unit 11 is, for example, a display device. The operation unit 12 is, for example, a keyboard and a mouse. The display unit 11 and the operation unit 12 may be integrated, for example, as in a touch panel display device. Further, the video camera 2 and the alarm device 3 are connected to the alarm device 1 via the interface 14.

The control unit 13 has a control storage unit 131 and a control processing unit 132. The control storage unit 131 is composed of one or a plurality of semiconductor memories. For example, it has at least one of RAM, flash memory, EPROM, EEPROM, and other non-volatile memories. The control storage unit 131 stores driver programs, operating system programs, application programs, data, and the like used for processing by the control processing unit 132.

The control storage unit 131 stores as a driver program a device driver program that controls the video camera 2, the alarm device 3, the display unit 11, the operation unit 12, and the like. The computer program may be installed in the control storage unit 131 from a computer-readable portable recording medium such as a CD-ROM or a DVD-ROM using a known setup program or the like. Alternatively, it may be downloaded from a program server or the like and installed.

Further, the control storage unit 131 may temporarily store temporary data related to a predetermined process. The control storage unit 131 stores the monitoring area setting file 1311, the difference time setting file 1312, the binarization threshold setting file 1313, the addition time file 1314, the cluster determination file 1315, the tracking determination file 1316, and other processing parameter files 1317. Remember.

The control processing unit 132 includes one or more processors and peripheral circuits thereof. The control processing unit 132 comprehensively controls the operation of the alarm device 1, and is, for example, a processor such as an MPU (Micro Processor Unit). The control processing unit 132 may be composed of a plurality of MPUs.

The control processing unit 132 executes processing based on a program (operating system program, driver program, application program, etc.) stored in the control storage unit 131. Further, the control processing unit 132 may execute a plurality of programs (application programs and the like) in parallel. The control processing unit 132 includes an image acquisition unit 1321, an image cutting unit 1322, a binarized data processing unit, etc. 1323, a background image generation unit 1324, a background difference binarization unit 1325, a short-term summary unit 1326, a long-term summary unit 1327, etc. Has.

Each of these units included in the control processing unit 132 may be mounted on the alarm device 1 as an independent integrated circuit, circuit module, microprocessor, or firmware.

[Outline of processing flow performed by the alarm device according to the first embodiment]
FIG. 3 is a diagram showing an example of a processing flow performed by the alarm device according to the first embodiment.

The image acquisition unit 1321 of the alarm device 1 acquires an image signal of an image including the monitored sea surface taken by the video camera 2 (ST301). The image cutting unit 1322 generates image data from the acquired image signal, and cuts out the set monitoring area image data in order to detect a ship approaching the construction section (ST302).

The background image generation unit 1324 performs statistical processing on the three color data included in the monitoring area image data for the frames corresponding to the first time, for example, the three color data corresponding to the five frames of n = 1 to 5. , Generate background image data (ST303). The statistical processing is, for example, a processing of adding each of three color data of 5 frames and dividing each by 5 to obtain an average value as background image data. The background image data of the pixels (x, y) is vector data whose component is the average value of each color data.

The background subtraction binarization unit 1325 uses the monitoring area image data for which the second time has passed since the first time has elapsed as the foreground image data, and takes the background difference from the background image data of each color (ST304). The background difference binarization unit 1325 compares the background difference data of each color of each pixel with the binarization threshold of each color, and when the background difference data of each pixel is equal to or more than the binarization threshold for all three color RGB 1 , In other cases, it is set to 0 to generate background difference image binarization data (ST305). The background subtraction image binarization data is data in which image noise related to a stationary object on the sea surface is removed.

The background subtraction image binarization data is sequentially created and stored in the control storage unit 131 of the alarm device 1 by the binarization data processing unit 1323 (ST306). That is, after the lapse of the first time and after the lapse of the second time, the background subtraction image binarization data corresponding to each frame is generated and stored.

The short-term summary unit 1326 performs a short-term summary process for creating short-term summary image data by adding background subtraction image binarization data over a third time (ST307). The short-term summary image data removes noise that temporarily appears on the surface of the sea, such as waves, reflections of light on the surface of the sea, and minute shaking of stationary objects.

The long-term summary unit 1327 sequentially performs image analysis of the short-term summary image data and detects an image region corresponding to the ship candidate. Further, image tracking of the ship candidate is performed, and a long-term summarization process for determining that the ship candidate is a ship heading for the construction area is performed (ST308).

The alarm signal output unit 1328 outputs an alarm signal to the alarm 3 when it is determined by the long-term summarization process that the ship candidate is a ship heading for the construction area (ST309). The alarm 3, for example Patrite®, flashes an alarm light.

[About background subtraction]
The background subtraction enables a process of extracting an object that does not exist in the image acquired in advance by comparing the captured image with the image acquired in advance. The image acquired in advance is called a background image, and the captured image to be compared with the background image is called a foreground image. In the simple background subtraction method in which the previous image is used as the background image and the image after one frame is used as the foreground image, the images related to the stationary objects existing in both frames disappear, but the moving object to be detected, for example, a ship, is used. Part of the image also disappears. In addition, when sunshine fluctuations occur on the sea, the camera moves, or stationary objects disappear, appropriate background subtraction cannot be obtained. In the present embodiment, a plurality of images are statistically processed to generate a background image.

FIG. 4 is a diagram illustrating an example of statistical background subtraction in the first embodiment.

と表わされる。統計的処理を行うことにより、静止物の揺らぎが少ない安定した背景画像データが得られる。又、日照変動による画像内の微小な輝度変化に対応することができる。 The background image generation unit 1324 of the alarm device 1 statistically processes the series M1, M2, M3, M4, and M5 of the monitoring area image data corresponding to the frames for the first time, for example, 5 frames of n = 1 to 5. To generate background image data MS1. The statistical processing is, for example, a processing in which the sum of the data values of 5 frames for each of the 3 color data is taken and the average value obtained by dividing by 5 of the number of frames is used as the background image data MS1. Background image data MS1 to against the pixel coordinates (x, y) is the average value of the red color data _R B1 (x, y), the average value _G B1 green color data (x, y), the blue color data Assuming that the average value is B _B1 (x, y), it is represented by a vector [ _RB1 (x, y), _GB1 (x, y), _BB1 (x, y)]. For example, the average value R _B1 (x, y) of the red color data is

It is expressed as. By performing statistical processing, stable background image data with little fluctuation of stationary objects can be obtained. In addition, it is possible to deal with minute changes in brightness in the image due to fluctuations in sunshine.

The background subtraction binarization unit 1325 describes the three color data included in the monitoring area image data P1 when the second time elapses after the first time elapses corresponding to the number of frames used to create the background image data. Using the vector [ _RP1 (x, y), _GP1 (x, y), _BP1 (x, y)] as the foreground image data, the vector difference between the background image data MS1 and the foreground image data P is taken. The second time is preferably sufficiently longer than the first time. For example, when the frame rate of the video camera 2 is 60 fps and the first time is 5 frames, the second time is preferably about 2 minutes. By making the second time sufficiently longer than the first time, it is possible to prevent the image data related to the ship from disappearing.

After the lapse of the first time 1/12 second, the red color data corresponding to the pixel coordinates (x, y) of the foreground image data P1 at the time of the second time 2 minutes is R _P1 (x, y; (. It is expressed as 1/12) +120). Pixel coordinates (x, y) of the difference image data D1 red color data _{of, R D1 (x, y)} = R P1 (x, y; (1/12) +120) -R B1 (x, y) and Represented. Green color _{data, G D1 (x, y)} = G P1 (x, y; (1/12) +120) represented -G B (x, y) and. Color data of _{blue, B D1 (x, y)} = B P1 (x, y; (1/12) +120) -B B (x, y) is expressed as. With the passage of time, the difference image data D1, D2, D3, ... Are sequentially created. The background subtraction binarization unit 1325 stores the difference image data D1, D2, D3, ... In the control storage unit 131.

The background difference binarization unit 1325 uses the three binarization thresholds DTH _R , DTH _G , and DTH _B to convert the difference image data D1, D2, D3, ... Convert to DB2, DB3, .... DTH _R is the red binarization threshold, DTH _G is the green binarization threshold, and DTH _B is the blue binarization threshold. _{For example, by adjusting the three binarization thresholds DTH R} , DTH _G , and DTH _B according to the weather and the time zone, the detection accuracy of the ship may be improved.

For example, if the difference image data D1 at the pixel coordinates (x, y) is [ _RD1 (x, y), G _D1 (x, y), _BD1 (x, y)], the binarized data The value F _DB1 (x, y) is
_{R D1 (x, y) ≧} DTH R, _{and, G D1 (x, y)} ≧ DTH G, _and, B D1 _(x, y) when _{≧ DTH B, F DB1 (x} , y) = 1
_{R D1 (x, y) <} DTH R, _{or, G D1 (x, y)} <DTH G, _{or, B D1 (x, y)} < When _{DTH B, F DB1 (x,} y) = 0
Is. Since the amount of calculation is reduced by binarizing, the processing speed of the alarm device 1 is increased, and the processing algorithm is also simplified. In this example, the _{values that F DB1} (x, y) can take are 0 and 1, but a combination of 0 and 255 may be used, for example.

As another example, in background subtraction binarization, the sum of the differences of each color data may be taken and compared with one threshold value DTH. For example, if the difference image data D1 at the pixel coordinates (x, y) is [ _RD1 (x, y), G _D1 (x, y), _BD1 (x, y)], the binarized data The value F _DB1 (x, y) is
| R _D1 (x, y) | + | G _D1 (x, y) | + | _BD1 (x, y) |
F _DB1 (x, y) = 1
| R _D1 (x, y) | + | G _D1 (x, y) | + | _BD1 (x, y) |
F _DB1 (x, y) = 0
Is.

FIG. 5 is a diagram for explaining the effect of taking background subtraction after a sufficient time has elapsed from the generation of background image data, and FIG. 5A shows the background image with the previous image as the background image and the image after one frame as the foreground image. It is a figure which shows the example which took subtraction and binarized, and (b) is the figure which shows the example which took the background subtraction and binarized after a sufficient time has elapsed from the generation of background image data.

The distance from the position of the video camera 2 to the horizon is large between the position when the ship approaching the construction area in the background image is on the horizon and the position of the ship in the foreground image one frame later, that is, 1/12 second later. , It is almost the same because the moving distance of the ship is short. In the binarized data, the image of the ship in the background image and the image of the ship in the foreground image overlap, and the ship image is hardly recognized. If sufficient time elapses from the generation of the background image data in (b), for example, 2 minutes later, the background subtraction is taken and binarized, the ship approaches the direction of the video camera by about 200 meters in 2 minutes. The image is recognized.

[Modification example of background subtraction processing]
In the embodiment, the background subtraction processing is performed as a vector corresponding to the three color data included in the monitoring area image data, but the vector amount may be converted into a scalar amount and processed.

As one example, the luminance data Y (x, y) for each pixel (x, y) may be used for background subtraction processing. Luminance data Y (x, y) can be converted from RGB color data,
Y (x, y) = k _R R (x, y) + (1-k _R −k _B ) G (x, y) + k _B B (x, y)
Is. k _R and k _B are conversion coefficients, for example, ITU-RB. In the 709 method, (k _R , k _B ) = (0.2126, 0.0722), ITU-RB. In the 601 method, (k _R , k _B ) = (0.299, 0.114). For example, in the case of a video camera that directly outputs luminance (Y) data and color difference (U, V) data, conversion is not necessary.

と表わされる。統計的処理を行うことにより、静止物の揺らぎが少ない安定した背景画像データが得られる。又、日照変動による画像内の微小な輝度変化に対応することができる。 The background subtraction processing using the luminance data Y will be described. The background image generation unit 1324 of the alarm device 1 determines the brightness data for the series M1, M2, M3, M4, and M5 of the monitoring area image data corresponding to the frames for the first time, for example, 5 frames of n = 1 to 5. Statistical processing is performed on Y to generate background image data. In the statistical processing, the sum of the luminance data of 5 frames for the pixel coordinates (x, y) is taken and the average value divided by 5 of the number of frames is used as the luminance data for the pixel coordinates (x, y) of the background image data MS1. It is a process. _{The brightness Y B1} of the background image data with respect to the pixel coordinates (x, y) is

It is expressed as. By performing statistical processing, stable background image data with little fluctuation of stationary objects can be obtained. In addition, it is possible to deal with minute changes in brightness in the image due to fluctuations in sunshine.

The background subtraction binarization unit 1325 uses the brightness data P1 included in the monitoring area image data when the second time elapses after the first time elapses corresponding to the number of frames used to create the background image data as the foreground image. As the data, the difference between the background image data MS1 and the foreground image data P1 is taken. The second time is preferably sufficiently longer than the first time. For example, when the frame rate of the video camera 2 is 60 fps and the first time is 5 frames, the second time is preferably about 2 minutes. By making the second time sufficiently longer than the first time, it is possible to prevent the image data related to the ship from disappearing.

After the lapse of the first time 1/12 second, the brightness corresponding to the pixel coordinates (x, y) of the foreground image data P1 at the time of the second time 2 minutes is Y _P1 (x, y; (1/12). ) + 120). Luminance _{Y D1} of the pixel coordinates (x, y) of the difference image data D1 _{is, Y D1 (x, y)} = Y P1 (x, y; (1/12) +120) represented _-Y B1 (x, y) and Is done. With the passage of time, the difference image data D1, D2, D3, ... Are sequentially created. The background subtraction binarization unit 1325 stores the difference image data D1, D2, D3, ... In the control storage unit 131.

The background difference binarization unit 1325 converts the difference image data D1, D2, D3, ... To the background difference image binarization data DB1, DB2, DB3, ... Using the binarization threshold YTH. .. _{For example, assuming that the brightness Y D1} (x, y) of the pixel coordinates (x, y) of the difference image data D1, _{the value Y D1B} (x, y) of the binarized data is
When Y _D1 (x, y) ≥ YTH, Y _D1B (x, y) = 1
When Y _D1 (x, y) <YTH, Y _D1B (x, y) = 0
Is.

As another example, for background subtraction processing, a function C (x, y) = R (x, y) + G (x, y) + B (x,) of the sum of three color data for each pixel (x, y). y)
May be used. Arithmetic processing can be simplified.

As another example, weighting factor _{W R,} W G, the function _C W (x, y) of the sum of the three color data having a _{W B = W R × R (} x, y) + W G × G (x, y _{) + W B × B (x} , y)
May be used. For example, the color of the sea blue (B) system is to be the main color, the weighting coefficient W _B of blue, the other weighting coefficients W _R, if less than W _G, colors other than blue (B) primary The object held in is emphasized.

[About short-term summary processing]
The short-term summary unit 1326 adds background subtraction image binarization data DB1, DB2, DB3, ... Over a third time, and short-term summary image data SS1, SS2. Perform short-term summary processing to create SS3 ... By the short-term summarization process, the short-term summarization image data is removed from noises temporarily appearing on the sea surface, such as waves, fluctuations of light on the sea surface, and minute fluctuations of stationary objects. It is preferable that the third time is appropriately changed according to the wave generation cycle and the vertical movement cycle of the sea surface so that the effect of noise removal becomes large. For example, the third time is preferably 1 second or more and 30 seconds or less. It should be noted that the data value corresponding to the pixel coordinates (x, y) of the short-term summary data is not limited to 0 or 1, but is a positive number of 0 or 0 or more.

[Long-term summary processing]
By background subtraction processing and short-term summarization processing, stationary objects, vibrating stationary objects, and noise temporarily appearing on the sea surface are removed, and short-term summarization image data SS1, SS2. SS3 ... Is image data including an image relating to a moving object. The long-term summary unit 1327 has the short-term summary image data SS1, SS2. Image analysis of SS3 ... Is performed sequentially to detect an image area corresponding to the ship candidate, and image tracking of the ship candidate is performed to determine that the ship candidate is a ship heading for the construction area.

FIG. 6 is a flowchart showing an example of long-term summarization processing.

The long-term summary unit 1327 sets the tracking counter TRC to 0 (ST601). The short-term summary image data SS1 is read and cluster analysis is performed (ST602). The long-term summary unit 1327 detects whether or not it has a cluster area for an image corresponding to a ship candidate (ST603). Conditions for detecting the cluster area for an image corresponding to the ship candidate value F _{SS (x,} y) of the short summary image data in, for example, pixels in the region coordinates (x, y) is a predetermined determination threshold Fth or more, Moreover, the area S of the region is equal to or larger than the predetermined area Sth. If the cluster area for the image corresponding to the ship candidate is not detected (ST603: NO), the process returns to step ST601.

When the cluster area for the image corresponding to the ship candidate is detected (ST603: YES), the long-term summary unit 1327 calculates and controls the center coordinates O (x, y) of the cluster area and the lowermost Y coordinate of the area. It is stored in the storage unit 131 (ST604). The long-term summarization unit 1327 adds 1 to the tracking counter TRC (ST605). The long-term summary unit 1327 determines whether or not the value of the tracking counter TRC exceeds 1 (ST606).

When the value of the tracking counter TRC is 1 (ST606: NO), the long-term summary unit 1327 repeats steps ST602 to ST606 for the next short-term summary image data. When the value of the tracking counter TRC exceeds 1 (ST606: YES), it means that the cluster area for the image corresponding to the ship candidate is detected in both the short-term summary image data that are continuous in time series.

When the value of the tracking counter TRC exceeds 1 (ST606: YES), the long-term summary unit 1327 determines whether or not the detected vessel candidate is likely to be a vessel heading for the construction area (ST607). .. The long-term summary unit 1327 compares the short-term summary image data before and after. For example, when the center coordinates O (x, y) of the cluster area move to the lower part of the image, the area S increases, and the lowermost Y coordinate of the area decreases, the detected ship candidate is in the construction area. It is determined that the vessel is likely to be heading.

If it is determined that the detected vessel candidate is not likely to be a vessel heading for the construction area (ST607: NO), the tracking continuity is considered to be interrupted and the process returns to step ST601. When it is determined that the detected ship candidate is likely to be a ship heading for the construction area (ST607: YES), the long-term summary unit 1327 adds 1 to the tracking counter TRC (ST608). The long-term summary unit 1327 determines whether or not the tracking counter is equal to or higher than a predetermined threshold value TRCth (ST609). If the tracking counter is less than the predetermined threshold value TRCth (ST609: NO), the process returns to step ST602. When the tracking counter is equal to or higher than the predetermined threshold value TRCth (ST609: YES), the alarm signal output unit 1328 outputs an alarm signal to the alarm device 3 (ST610). The process ends.

[Overview of the alarm device according to the second embodiment]
In the first embodiment, the ship image is detected by the background subtraction method, but in the second embodiment, the ship image is detected by the color difference method.

[Alarm device block configuration]
FIG. 7 is a diagram showing an example of the block configuration of the alarm device according to the second embodiment.

The alarm device 7 has a display unit 71, an operation unit 72, and a control unit 73, and each unit is connected by a bus line. The display unit 71 is, for example, a display device. The operation unit 72 is, for example, a keyboard and a mouse. The display unit 71 and the operation unit 72 may be integrated, for example, as in a touch panel display device. Further, the video camera 2 and the alarm device 3 are connected to the alarm device 7 via the interface 74.

The control unit 73 includes a control storage unit 731 and a control processing unit 732. The control storage unit 731 is composed of one or a plurality of semiconductor memories. For example, it has at least one of RAM, flash memory, EPROM, EEPROM, and other non-volatile memories. The control storage unit 731 stores driver programs, operating system programs, application programs, data, and the like used for processing by the control processing unit 732.

The control storage unit 731 stores as a driver program a device driver program that controls the video camera 2, the alarm device 3, the display unit 71, the operation unit 72, and the like. The computer program may be installed in the control storage unit 731 from a computer-readable portable recording medium such as a CD-ROM or a DVD-ROM using a known setup program or the like. Alternatively, it may be downloaded from a program server or the like and installed.

Further, the control storage unit 731 may temporarily store temporary data related to a predetermined process. The control storage unit 731 stores the monitoring area setting file 7311, the hue weight setting file 7312, the binarization threshold setting file 7313, the addition time file 7314, the cluster determination file 7315, the tracking determination file 7316, and other processing parameter files 7317. do.

The control processing unit 732 includes one or more processors and peripheral circuits thereof. The control processing unit 732 comprehensively controls the operation of the alarm device 7, and is, for example, a processor such as an MPU (Micro Processor Unit). The control processing unit 732 may be composed of a plurality of MPUs.

The control processing unit 732 executes processing based on a program (operating system program, driver program, application program, etc.) stored in the control storage unit 731. Further, the control processing unit 732 may execute a plurality of programs (application programs and the like) in parallel. The control processing unit 732 includes an image acquisition unit 7321, an image cutout unit 7322, a binarization data processing unit 7323, a hue averaging unit 7324, a color difference binarization unit 7325, a short-term summary unit 1326, a long-term summary unit 1327, and the like. ..

Each of these parts included in the control processing unit 732 may be mounted on the alarm device 7 as an independent integrated circuit, circuit module, microprocessor, or firmware.
[Generation of color difference image binarization data]
The video camera 2 has a performance of, for example, a resolution of M × N pixels. The coordinates of each pixel are represented by (x, y): 1 ≦ x ≦ M, 1 ≦ y ≦ N. The image data from the video camera 2 is red (R) color data: R (x, y), green (G) color data: G (x, y), and blue (B) for each pixel. Color data: B (x, y). Some cameras do not output three color data for all pixels, but output every other pixel, but for the sake of simplicity, it is assumed that color data is output for all pixels in this embodiment. There are also cameras that output, for example, brightness (Y) data and color difference (U, V) data other than RGB color data, but since conversion between RGB data and YUV is possible, the RGB color data will be described below. Will be described as an example.

The hue averaging unit 7324 calculates, for example, the average values av (R), av (G), av (B) of the R, G, and B color data of all pixel coordinates (x, y) included in the monitoring area image data. do. The average value of the R, G, and B color data may be divided into a sea surface region and an empty region of the monitoring region image data. As another example, for example, the image data may be divided into 64 × 64 pixel blocks, and the difference image data may be generated for each block.

Color difference binarization unit 7325, the function _{F C (x, y) =} (R (x, y) -av (R)) 2 + (G (x, y) -av (G)) 2 + (B (X, y) -av (B)) ²
Is used to generate color difference image data obtained by converting vector values having R, G, and B color data as components into scalar values.

As another example, the color difference binarization unit 7325, R of each pixel coordinates included in the monitoring image data (x, y), G, hue weighting coefficient _W R of _B, W G, the function has a _{W B} F _{W (x, y) = W} R (R (x, y) -av (R)) + W G (G (x, y) -av (G)) + W B (B (x, y) -av (B )))
May be used to generate color difference image data. For example, the color of the sea blue (B) system is to be the main color, the weighting coefficient W _B of blue, the other weighting coefficients W _R, if less than W _G, the hue other than blue (B) primary The object held in is emphasized.

Color difference binarization unit 7325 converts the function _F C (x, y) to a two-valued function _F CB (x, y) by using the color difference threshold Cbth.
F _C (x, y) when _{≧ CBth, F CB (x,} y) = 1
F _C (x, y) <When _{CBth, F CB (x, y} ) = 0
Is. _FCB (x, y) indicates the color difference binarized data of the pixel (x, y). In this example, the _{values that FCB} (x, y) can take are 0 and 1, but a combination of 0 and 255 may be used, for example.

The color difference threshold CBth is not a fixed value but may be variable. For example, with the horizontal line as the first approach line, a rectangular area including the first approach line is set as the first monitoring area, and a rectangular area including a virtual horizontal second approach line on the sea 2.0 km away from the construction area is set. It is assumed that it is set in the second monitoring area. Since the images corresponding to each monitoring area are not the same, it is preferable to set CBth according to each monitoring area. The CBth is preferably set according to the time of day, for example, morning, noon, evening, and night. Further, it is preferable to set according to the weather and the sea condition, for example, sunny, cloudy, rain, ocean current, wave, tide and the like. This is because the sea level conditions change depending on the time of day and the weather.

The color difference threshold CBth may be set statistically using many sample data. Since there are many variable factors, it is preferable to use, for example, a statistical method of regression analysis or Bayesian estimation.

Color difference binarization unit 7325, the weighting factor _{W R,} W G, the function _F W _(x, y) with _{W B = W R (R (} x, y) -av (R)) + W G (G ( _{x, y) -av (G)} ) + W B (B (x, y) -av (B))
When the color difference image data is generated by using the above, the color difference binarization processing can be performed in the same manner. Weighting factor _{W R, W G, W B can} be statistically set using a number of sample data. Since there are many variable factors, it is preferable to use, for example, a statistical method of regression analysis or Bayesian estimation.

[Outline of processing flow performed by the alarm device according to the second embodiment]
FIG. 8 is a diagram showing an example of a processing flow performed by the alarm device according to the second embodiment.

The image acquisition unit 7321 of the alarm device 7 acquires an image signal of an image including the monitored sea surface taken by the video camera 2 (ST801). The image cutting unit 7322 generates image data from the acquired image signal, and cuts out the set monitoring area image data in order to detect a ship approaching the construction section {ST802).

The hue averaging unit 7324 calculates the average values av (R), av (G), av (B) of the R, G, and B color data of all pixel coordinates (x, y) included in the monitoring area image data ( ST803).

Color difference binarization unit 7325, the function _{F C (x, y) =} (R (x, y) -av (R)) 2 + (G (x, y) -av (G)) 2 + (B (X, y) -av (B)) ²
Is used to generate color difference image data obtained by converting vector values having R, G, and B color data as components into scalar values (ST804).

Color difference binarization unit 7325, the function _F C (x, y) and converted into binary function _F CB (x, y) to generate a color difference image binarized data by using the color difference threshold Cbth ( ST805). The color difference image binarization data is sequentially created and stored in the storage unit 731 of the alarm device 7 by the binarization data processing unit 7323 (ST806). That is, the color difference image binarization data corresponding to each frame is generated and stored.

The short-term summarization unit 7326 performs a short-term summarization process for creating short-term summarization image data by adding color difference image binarization data over a predetermined addition time (ST807). The short-term summary image data removes noise that temporarily appears on the surface of the sea, such as waves, reflections of light on the surface of the sea, and minute shaking of stationary objects.

The long-term summary unit 1327 sequentially performs image analysis of the short-term summary image data, detects an image area corresponding to the ship candidate, and tracks the image of the ship candidate to determine that the ship candidate is heading for the construction area. Perform a long-term summarization process for determination (ST808).

The alarm signal output unit 7328 outputs an alarm signal to the alarm 3 when it is determined by the long-term summarization process that the ship candidate is a ship heading for the construction area (ST809). The alarm 3, for example Patrite®, flashes an alarm light.

It will be appreciated by those skilled in the art that various changes, substitutions and modifications can be made to this without departing from the spirit and scope of the invention.

1,7 Alarm device 11,71 Display unit 12,72 Operation unit 13,73 Control unit 131,731 Control storage unit 132,732 Control processing unit 14,74 Interface 2 Video camera 3 Alarm

Claims (6)

An image acquisition unit that acquires image signals of images including the sea surface from a video camera installed in the construction area,
An image cutout portion that cuts out rectangular image data having a long side in a direction parallel to the boundary between the sea surface and the sky or the sea surface and the land as a plurality of monitoring area image data for each frame from the acquired image signal.
A binarization data processing unit that generates image binarization data using the monitoring area image data using a predetermined binarization threshold, and a binarization data processing unit.
A short-term summary unit that generates short-term summary image data by adding the image binarization data over a predetermined addition time, and
Image analysis of each short-term summary image data is sequentially performed to detect an image area corresponding to the ship candidate, image tracking of the ship candidate is performed, and the ship candidate is the length of the short-term summary image data on the construction area side. When heading to the side , the long-term summary section, which determines that the ship is likely to be heading to the construction area,
When the marine vessel candidate is determined to be likely a ship headed in the construction zone, a warning signal output unit for outputting an alarm signal to an alarm device,
Alarm device with. The binarized data processing unit
A background image generation unit that generates background image data from a part of the plurality of monitoring area image data over the first time, and a background image generation unit.
A background difference between the background image data and a predetermined monitoring area image data in which a second time has passed from the first time among the plurality of monitoring area image data is taken, and the binarization threshold is used to obtain the image 2. Background subtraction binarization unit that generates valued data,
The alarm device according to claim 1. The binarized data processing unit
A hue averaging unit for calculating the average value of each of the R, G, and B color data included in the monitoring area image data, and a hue averaging unit.
Color difference 2 that generates color difference image data related to the difference between the hue value of each pixel of the monitoring area image data and each average value of the color data, and generates the image binarization data using the color difference threshold. The binarization department and
The alarm device according to claim 1. Wherein the plurality of monitoring areas image data includes image data including the boundary, alarm device according to any one of claims 1 to 3. Obtaining image signals of images including the sea surface from a video camera installed in the construction area,
From the acquired image signal, rectangular image data having a long side in a direction parallel to the boundary between the sea surface and the sky or the sea surface and the land is cut out as a plurality of monitoring area image data for each frame.
To generate image binarization data using a predetermined binarization threshold value for the monitoring area image data,
To generate short-term summary image data by adding the image binarization data over a predetermined addition time.
Image analysis of each short-term summary image data is sequentially performed to detect an image area corresponding to the ship candidate, image tracking of the ship candidate is performed, and the ship candidate is the length of the short-term summary image data on the construction area side. When heading to the side , it is judged that there is a high possibility that the ship is heading to the construction area.
When the marine vessel candidate is determined to be likely a ship headed in the construction zone, and outputting an alarm signal to an alarm device,
Alarm method with. A program that runs a computer
Obtaining image signals of images including the sea surface from a video camera installed in the construction area,
From the acquired image signal, rectangular image data having a long side in a direction parallel to the boundary between the sea surface and the sky or the sea surface and the land is cut out as a plurality of monitoring area image data for each frame.
To generate image binarization data using a predetermined binarization threshold value for the monitoring area image data,
To generate short-term summary image data by adding the image binarization data over a predetermined addition time.
Image analysis of each short-term summary image data is sequentially performed to detect an image area corresponding to the ship candidate, image tracking of the ship candidate is performed, and the ship candidate is the length of the short-term summary image data on the construction area side. When heading to the side , it is judged that there is a high possibility that the ship is heading to the construction area.
When the marine vessel candidate is determined to be likely a ship headed in the construction zone, and outputting an alarm signal to an alarm device,
An alarm program that causes the computer to run.

Images