JP7270537B2 - Object detection device, ship, and object detection method - Google Patents

Description

The present invention relates to an object detection device that detects an object from an image, a ship equipped with the object detection device, and an object detection method.

In order to prevent the ship from colliding with obstacles such as floating objects such as buoys floating on the surface of the water, reefs, and other ships, seafarers should monitor the surroundings of the ship during navigation. There is a need to do. Monitoring work places a heavy burden on sailors, and if an obstacle is overlooked, a collision accident may occur. For the purpose of reducing the burden on sailors and preventing collision accidents, object detection devices that automatically detect surrounding objects from images taken by cameras are being developed.

As an example of such an object detection device based on image recognition, there is a device described in Non-Patent Document 1, for example. The device disclosed in Non-Patent Document 1 applies an object detection technique that applies a convolutional neural network to a landscape image in which the Uraga Channel is captured, and performs detection processing for detecting ships sailing on the Uraga Channel. In the future, autonomous ships, which are expected to spread, are expected to put into practical use object detection devices based on image recognition in order to monitor their surroundings.

Mitsuru Kobayashi, et al., ``Detection of Ship Shadows from Landscape Images Using Deep Learning,'' Proceedings of the Japan Society of Naval Architects and Ocean Engineers, 2018, No. 26, pp. 121-124

In the detection process by image recognition described above, a learning model obtained by pre-learning a plurality of images of the object to be detected is used, and the presence or absence of the object included in the image taken by the camera is used. Determination, identification processing of the type of the object, and the like are performed.

Here, for example, when an object such as another ship is detected using an image captured by a camera installed on a ship, the captured image is tilted compared to an image captured by a camera fixed on the ground. It often becomes a thing. Furthermore, since the inclination is not always constant, the inclination of the object included in the captured image also varies.

When performing detection processing on captured images of objects with various tilts, there are many cases where images of objects with tilts that differ greatly from the images of objects used for learning are targeted. There is a problem that the detection accuracy of is lowered. In such a case, in order to improve the detection accuracy of the object, images of the object taken at all tilts may be required during learning. In many cases, an image of the object photographed at all tilts, for example, an image of the object rotated at random, is added as a learning target. However, in this case, there is a possibility that the calculation time during learning becomes long.

The present invention has been made to solve the above problems, and an object detection device that improves the detection accuracy of an object without increasing the calculation time during learning, a ship equipped with the object detection device, and an object The object is to provide a detection method.

An object detection apparatus according to the present invention is an object detection apparatus for detecting an object using a first learning model generated using a plurality of first training data representing an image including the object, wherein the object detection a horizontal line detection unit for detecting a horizontal line included in an image represented by the detection image data used in the a correction unit that corrects detection image data; an object detection unit that performs detection processing of the object using the first learning model in an image represented by the detection image data corrected by the correction unit; Prepare.

A ship according to the present invention includes the object detection device described above.

An object detection method according to the present invention is an object detection method executed by an object detection device that detects an object using a first learning model generated using a plurality of first teacher data representing images including the object. a horizontal line detection step of detecting a horizontal line included in an image represented by the detection image data used for detecting the object; a correction step of correcting the detection image data so as to be less than or equal to an angle; and detection processing of the object using the first learning model in an image represented by the detection image data corrected in the correction step. and an object detection step of performing

According to the object detection device, the ship equipped with the object detection device, and the object detection method according to the present invention, the horizon detection unit detects the horizon included in the image represented by the detection image data used for object detection. The correction unit corrects the detection image data so that the first angle between the horizontal direction and the horizontal line is equal to or less than a predetermined angle. Then, the object detection unit performs object detection processing on the corrected image data for detection using the first learning model. As a result, the inclination from the horizontal direction of the image of the object represented by the detection image data, which is the target of the detection process, is suppressed within a predetermined range (within a predetermined angle). Therefore, for example, when the first learning model is generated, if the inclination of the image of the object represented by the first teacher data from the horizontal direction is suppressed within a predetermined range, the image of the object represented by the first teacher data , the difference in inclination from the image of the object represented by the detection image data is suppressed. That is, since the matching between the first teacher data to be learned and the detection target data is increased for the image of the target object, the accuracy of object detection by the detection process is improved. In addition, in order to detect an object tilted beyond a predetermined range from the horizontal direction, it is necessary to randomly rotate the learning target image to generate images of the object tilted at various angles during learning. Since it is no longer necessary to learn a huge number of images, it is possible to suppress an increase in calculation time during learning.

1 is a block diagram showing a configuration example of an object detection device according to Embodiment 1; FIG. It is an example of the image represented by the image data for detection before correction|amendment. 3 is a diagram showing an image after correction of the image illustrated in FIG. 2; FIG. 5 is a flowchart showing an example of object detection processing by the object detection device according to Embodiment 1; 9 is a flowchart showing an example of object detection processing by the object detection device according to Embodiment 2; 9 is a flowchart showing an example of object detection processing by the object detection device according to Embodiment 2;

An object detection device, a ship, and an object detection method according to embodiments will be described below with reference to the drawings. In the drawings below, the size relationship of each component may differ from the actual size.

Embodiment 1.
FIG. 1 is a block diagram showing a configuration example of an object detection device according to Embodiment 1. FIG. The object detection device 1 uses a convolutional neural network to detect an object from an image represented by input image data. Prior to detecting an object on an image, the object detection device 1 generates a learning model to be used for detection processing of the object, and stores the learning model in the object detection device 1 or another storage device. The object detection device 1 generates a learning model by performing learning using a plurality of input image data as teacher data. Hereinafter, a learning model for use in object detection processing will be referred to as a first learning model, and teacher data used to generate the first learning model will be referred to as first teacher data.

The object detection device 1 includes a learning processing unit 2 , a storage unit 3 that stores calculation results by the learning processing unit 2 , and a detection processing unit 4 . The learning processing unit 2 generates a first learning model using each of a plurality of image data captured by a camera or the like or a plurality of image data read by a scanner or the like as first teacher data. Below, the plurality of image data used in generating the first learning model may also be referred to as a data set.

The storage unit 3 stores the first learning model generated by the learning processing unit 2 . A detection processing unit 4 detects an object to be detected from image data obtained by the camera 5 . In the first embodiment, it is assumed that the detection target of the detection processing unit 4 is a ship appearing on the water. Therefore, it is assumed that the image data as the first teacher data included in the data set used by the learning processing unit 2 is image data including ships sailing or anchoring in various seas or rivers. . However, the data set may include image data including ships on the ground. The image used by the detection processing unit 4 to detect a ship is an image captured by the camera 5, and the image may include the sea, a river, or the like. The image may include at least one of land areas such as islands or beaches, wharves, quay walls, bridges, and the like. In the following description, image data used by the detection processing unit 4 to detect a detection target object such as a ship may be referred to as detection image data.

The camera 5 in Embodiment 1 may be installed on a means of transportation such as a ship, or may be installed on land. The camera 5 includes a lens 50 , an imaging section 51 , an A/D conversion circuit 52 (Analog-to-Digital Converter), and an output section 53 . Reflected light from the subject enters the lens 50 and is condensed by the lens 50 onto the imaging section 51 . The imaging unit 51 is, for example, a CCD (Charge Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor) image sensor, or the like, and has a plurality of light receiving elements arranged in a matrix. The imaging unit 51 converts light into electrical signals using a plurality of light receiving elements.

The A/D conversion circuit 52 acquires an electric signal from the imaging unit 51 and converts the electric signal from an analog signal to a digital signal. The output unit 53 outputs the electrical signal converted into a digital signal by the A/D conversion circuit 52 to the object detection device 1 .

The detection processing unit 4 generates image data for representing a two-dimensional image from the electrical signal acquired from the camera 5 . The image data corresponds to detection image data. It should be noted that the electrical signal that is the source of the image data for detection that the detection processing unit 4 acquires from the camera 5 may also be referred to as image data for detection below. The detection processing unit 4 uses the first learning model stored in the storage unit 3 to perform ship detection processing on the image represented by the detection image data.

Here, the camera 5 may be installed on a ship instead of on a flat ground in order to prevent ships from colliding with each other. Since a ship is swayed by waves and swayed by the flow of water in the sea or river, the image captured by the camera 5 installed on the ship is tilted from the actual horizontal direction. can be. In this case, when another ship is photographed by the camera 5, the image showing the other ship may also be tilted.

In order for a ship on an inclined image to be detected as a ship by detection processing, it is preferable to use a first learning model generated using image data representing a similarly inclined ship as first teacher data. . If the image data representing the tilted ship is not used when generating the first learning model, there is a possibility that the ship cannot be detected on the greatly tilted image.

In order to detect objects such as ships in tilted images, conventional object detection devices use data augmentation during learning to randomly rotate the images in the learning target data set. As a target, it was increasing robustness against angles.

However, when adding tilted images to a data set during learning, the number of image data included in the data set significantly increases, resulting in a problem of longer learning time.

Here, when the camera is fixed on the ship, it is assumed that the frequency with which the image obtained by the camera is greatly tilted is not so high. Therefore, it is assumed that the frequency in which the ship (ship image) is greatly tilted in the image is not so high. For example, when the sea is very rough, such as when it is stormy, it is estimated that there are not many ships sailing, and other ships (ship images) on the image obtained by photographing with a camera fixed to the ship ) is also assumed to remain within a predetermined range.

Therefore, in order to shorten the learning time, the object detection device 1 according to the first embodiment does not rotate the image data included in the data set during learning, and the object detection device 1 sails or berths in the sea, rivers, or the like. The image data representing the photographed image of the ship as the subject is used as it is as the first teacher data. In addition, the object detection device 1 does not reduce the detection accuracy of the ship even when the ship is tilted greatly on the image. is corrected so as to match the inclination of the image represented by the first teacher data. It should be noted that the image data used as the first teacher data is obtained by photographing with a camera fixed on the ground or on a ship, and the inclination of the image represented by the image data is predetermined, such as 5°. It is assumed that the angle is less than or equal to the predetermined angle. Note that the image data used as the first training data includes image data representing a slightly tilted ship. Guaranteed.

When the image represented by the detection image data is tilted, the detection processing unit 4 adjusts the detection image data so that the tilt is 0° or a predetermined angle of 5° or less, for example. Correction processing such as rotation processing is performed. Specifically, the detection processing unit 4 corrects the horizontal line in the image so that the inclination of the horizontal line in the image is 0° or less than the predetermined angle. The horizontal direction in the image corresponds to the X direction in the two-dimensional coordinate system (XY coordinate system) of the image. It is the same direction as the horizontal direction of the screen when it is installed on the Note that the object detection device 1 may not have a display device for screen display, and an image may not be a display target.

After correcting the inclination of the image, the detection processing unit 4 uses the first learning model to perform detection processing of the vessel on the image.

The detection processing unit 4 includes a detection image acquisition unit 40 , a horizon detection unit 41 , a correction unit 42 , and an object detection unit 43 in order to execute the processing before the ship detection processing as described above. 1, the connection relationship between the detection image acquisition unit 40 and the horizon detection unit 41, the connection relationship between the horizon detection unit 41 and the correction unit 42, and the connection relationship between the correction unit 42 and the object detection unit 43 Although the connection relationship between is shown, the connection relationship is not limited to this, and for example, the object detection unit 43 and the horizon detection unit 41 may be connected.

The detection image acquisition unit 40 communicates with the camera 5 by wire or wirelessly, acquires an electrical signal from the camera that is the basis of detection image data, and generates detection image data from the electrical signal. The horizontal line detection unit 41 detects a horizontal line by edge detection processing, processing using RGB differences, or the like. In addition, the horizon detection part 41 may detect not only a horizon but the boundary line of a quay or land and the sky which extends in parallel with a horizon. In addition, the said boundary line is also described as a horizontal line below.

The horizontal line detection process will be described in detail below. The detection image data includes values of R (Red), G (Green), and B (Blue) in the RGB color space for each pixel. It should be noted that a pixel refers to the smallest element in image data associated with color information such as RGB values. direction) with the Y coordinate are associated. The image data for detection is Y, U (Cb), and V (Cr) values in the YUV color space (YCbCr color space), or H (Hue) in the HSV color space, instead of the respective values in the RGB color space. , S (Saturation), and V (Value) may be provided for each pixel. Note that Y in the YUV color space is luminance, and U(Cb) and V(Cr) are color differences. Also, in the HSV color space, H is hue, S is saturation, and V is brightness. Each value in the RGB color space, each value in the YUV color space (YCbCr color space), and each value in the HSV color space can be converted to each other.

The horizontal line detection unit 41 calculates, for example, the brightness or brightness of each pixel from each value in the RGB color space of each pixel, and determines the degree of change in brightness or brightness in the image represented by the detection image data. Extract steep parts. It should be noted that the portion where the degree of change in brightness or lightness is steep refers to, for example, a portion where the degree of change in brightness or lightness is greater than or equal to a predetermined threshold. Here, for example, pixels representing the sky and pixels representing the sea have different brightness (brightness). ), the degree of change in luminance (brightness) at the boundary between the sky and the sea is large. For this reason, the threshold is the degree of change in luminance (brightness) at the boundary between the sky and the sea, the degree of change in luminance (brightness) at the boundary between land and sea, and the sea (or It is determined in advance based on the degree of change in brightness (brightness) between pixels representing the sky or land.

Here, it is considered that the horizontal line forms a substantially straight line in the portion where the degree of luminance (brightness) change is steep. Similarly, among areas where the degree of change in brightness (brightness) is steep, the boundary between two buildings with greatly different colors that are adjacent on land, or the boundary between the sky and clouds or a flying object, etc. In the process of detecting a horizontal line, it must be omitted as a candidate for the part indicating the horizontal line. For this reason, the horizontal line detection unit 41 selects a portion where the degree of change in brightness (brightness) is steep and extends linearly as a candidate for the horizontal line. Hereinafter, such a portion where the degree of luminance (brightness) change is steep and which extends linearly will be referred to as a straight portion.

However, in this state, the horizon detection unit 41 may still extract the boundary between the sloped land or quay and the sky as the horizon. For this reason, the horizontal line detection unit 41 extracts the longest straight portion as the horizontal line from among the portions that can be candidates for the portion indicating the horizontal line. However, when the image data for detection represents a narrow bay, and the boundary between the water surface and the land forms a curved line, or the boundary between the land and the sky becomes a straight line, the straight line is not parallel to the horizon, and the straight line is longer than the boundary line between the land or the sky and the sea, the horizon detection unit 41 prevents erroneous detection such as the boundary between the land and the sky being the horizon line. may do.

In order for the horizon detection unit 41 to accurately detect the horizon even in such a case, the learning processing unit 2 of the object detection device 1 learns, for example, image data representing the sea, rivers, or the like as teacher data. Alternatively, a learning model related to the river or the like is generated in advance, and the learning model related to the sea or the river is stored in the storage unit 3, and the horizon detection unit 41 detects the Horizon detection may be performed using a learning model. Hereinafter, the learning model relating to the sea, rivers, etc. will be referred to as a second learning model, and the teacher data used to generate the second learning model will be referred to as a second teacher data. Note that the second learning model may include information regarding the color of the sea, rivers, or the like. The information about the color of the sea or river may include RGB values, YUV (YCbCr) values, HSV values, or the like.

Edge detection processing based on luminance (brightness) is used for the above-described horizontal line detection processing. A degree of change or the like may be used. Note that when edge detection processing based on hue or saturation is used, a portion where the degree of change in hue or saturation is greater than or equal to a predetermined threshold value and which extends linearly is defined as a straight portion. good too.

The correction unit 42 corrects the detection image data when the inclination of the detected straight line portion indicating the horizontal line from the horizontal direction of the image exceeds the predetermined angle. For example, the correction unit 42 corrects the detection image data as follows. The correction unit 42 calculates the inclination of the horizontal line from the horizontal direction in the image or the angle between the horizontal direction and the horizontal line in the image. Below, the said angle is described as a 1st angle. When the calculated first angle is larger than the above-described predetermined angle, the correction unit 42 adjusts the detection image so that the first angle is equal to or less than the predetermined angle, such as setting the inclination to 0 or setting the first angle to 0°. Correct the data. Note that the correction unit 42 may perform rotation processing on all detection image data, not only when the first angle is greater than the above-described predetermined angle.

Here, when the image is rotated, if it is output in the original image size, there is a risk that part of the detection image data, such as data at the edge of the image, will be lost. Therefore, in order to output the entire detection image data to the object detection unit 43, the correction unit 42 increases the amount of data allocated to the detection image data according to the tilt to be corrected and the first angle. For example, the correction unit 42 may increase the number of pixels. Note that the amount of data corresponds to the area of the image when the image based on the image data for detection is displayed on the screen.

The object detection unit 43 uses the first learning model stored in the storage unit 3 with a convolutional neural network to perform ship detection processing in the image represented by the detection image data.

Specific processing contents by the horizon detection unit 41, the correction unit 42, and the object detection unit 43 will be described below with reference to FIGS. 2 and 3. FIG. FIG. 2 is an example of an image represented by detection image data before correction. FIG. 3 is a diagram showing an image after the image illustrated in FIG. 2 has been corrected. The X direction of the XY coordinate system of the images shown in FIGS. 2 and 3 is the horizontal direction in the image, and the Y direction corresponds to the vertical direction in the image. 2 and 3, the hatched area A represents the sky and the other hatched area B represents the sea. A straight portion C represents a horizontal line. An artifact D represents a building, a ship, or the like. A frame F1 in FIG. 2 shows a display range of an image based on the detection image data corresponding to the amount of data assigned to the detection image data before correction by the correction unit 42 . In the image shown in FIG. 2, the horizontal line detection unit 41 detects that the degree of change in brightness (brightness) between region A and artifact D and region B is steep, or that the degree of change in RGB is large. For this reason, the straight line portion C is set as a candidate for the straight line portion indicating the horizontal line. Further, the horizontal line detection unit 41 extracts the straight line portion C as the horizontal line because the straight line portion C is the longest among the straight line portion candidates representing the horizontal line.

The correction unit 42 calculates a first angle between the horizontal direction and the horizontal line C in the image. When the first angle is greater than the predetermined angle, the correction unit 42 performs correction processing by rotating the detection image data so that the first angle is equal to or less than the predetermined angle. In the case shown in FIG. 3, the correction unit 42 performs rotation processing with the first angle set to 0°. Therefore, the horizontal line C is parallel to the horizontal direction in the image.

A frame F2 in FIG. 3 shows an image display range corresponding to the amount of data assigned to the corrected detection image data. Note that the image in this frame F2 may be displayed in its entirety by scrolling. The length L1 of the long side of the frame F2 in FIG. 3 corresponds to the length of the long side of the frame F1 shown in FIG. 2, and the length L2 of the short side of the frame F2 is the length of the short side of the frame F1. corresponds to As shown in FIG. 3, when a data amount corresponding to the frame F1 is assigned to the corrected image data for detection, a rectangle formed by using a long side of length L1 and a short side of length L2 is obtained. Information in other parts may be missing. In order to prevent such omissions, the correcting unit 42 increases the range in the X direction and the range in the Y direction of the detection image data after correction from the detection image data before correction, so that all of the detection image data The amount of data that can include all the color information of each pixel is assigned as the amount of data for the detection image data after the correction.

The object detection unit 43 detects the ship E from the image represented by the corrected detection image data based on the first learning model.

The object detection device 1 according to Embodiment 1 includes, for example, a processor such as a GPU (Graphics Processing Unit) or a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), or a memory such as a cache memory. , a storage device such as a HDD (Hard Disk Drive), an interface circuit for communication, and an input device such as a scanner or a camera. The function of the storage unit 3 can be realized by a storage device. The function of generating a learning model such as a first learning model and a second learning model by the learning processing unit 2 is such that a processor reads a learning program stored in a memory using a data set input to an input device. It can be realized by executing The function of the detection image acquisition section 40 in the detection processing section 4 can be realized by a communication interface circuit. The functions of the horizontal line detection unit 41, the correction unit 42, and the object detection unit 43 in the detection processing unit 4 are such that the processor reads out the object detection program stored in the memory using the detection image data acquired from the camera 5. It can be realized by executing All or part of the functions of the object detection device 1 described above may be realized by dedicated hardware.

4 is a flowchart illustrating an example of object detection processing by the object detection device according to Embodiment 1. FIG. In step S<b>1 , the learning processing unit 2 of the object detection device 1 generates a first learning model using a data set of images including ships, and stores the first learning model in the storage unit 3 .

The object detection device 1 waits until it acquires an electrical signal, which is the basis of detection image data, from the camera 5 in step S2 (step S2: NO). If the detection image acquiring unit 40 in the detection processing unit 4 acquires the electric signal that is the source of the detection image data from the camera 5 in step S2 (step S2: YES), the detection image acquiring unit 40 in step S3 generates image data for detection from the electrical signal.

In step S4, the horizontal line detection unit 41 performs horizontal line detection processing in the image represented by the detection image data by the above-described edge detection processing or by processing using differences in RGB values. In the first embodiment, the horizontal line detection unit 41 extracts the longest straight line from among the straight lines on the image as the horizontal line. which includes RGB values, YUV (YCbCr) values, HSV values, or the like may be used in the horizontal line detection process.

When the horizontal line detection unit 41 detects a horizontal line through the horizontal line detection processing in step S4 (step S5: YES), in step S6, the correction unit 42 calculates the first angle between the horizontal direction and the horizontal line in the image. calculate. The first angle can be calculated by obtaining the inclination of the horizontal line from the horizontal direction.

In step S7, the correction unit 42 determines whether or not the first angle calculated in step S6 is greater than the predetermined angle. The predetermined angle is, as described above, the upper limit of the angle between the horizontal line and the horizontal direction in the first teacher data used in generating the first learning model, and is, for example, 5°. If the first angle is greater than the predetermined angle (step S7: YES), the correction unit 42 corrects the detection image data in order to make the first angle less than or equal to the predetermined angle in step S8. This corrects the tilt of the image. For example, the correction unit 42 may correct the detection image data so that the first angle becomes 0°, like the image shown in FIG. 2 is corrected to the image shown in FIG. Alternatively, the correction unit 42 may correct the detection image data so that the first angle becomes a specific angle equal to or less than a predetermined angle such as 1° or 2°. Alternatively, the correction unit 42 determines that the magnitude of the first angle is the mode or average value of the magnitudes of the angles between the horizontal line and the horizontal direction in the image represented by each of the plurality of first teacher data. The detection image data may be corrected so that Note that the mode value or the average value is equal to or smaller than the predetermined angle and may be stored in the storage unit 3 in advance.

In step S8, in order to prevent loss of the detection image data after correction, the correction unit 42 changes the range in the X direction and the range in the Y direction of the detection image data after correction to the detection image data before correction. The amount of data that can include the X coordinate, the Y coordinate, and the color information of each pixel in the corrected detection image data is added to the data amount of the corrected detection image data. and Following step S8, in step S9, the object detection unit 43 performs ship detection processing in the image represented by the corrected detection image data. If no horizontal line is detected in step S5 (step S5: NO), the correction unit 42 does not execute processing, and in step S9 the object detection unit 43 detects the detection image generated in step S3. Perform a vessel detection process in the image represented by the data. Further, when the first angle is equal to or less than the predetermined angle in step S7 (step S7: NO), the correction unit 42 does not execute the correction process, and the object detection unit 43 detects the angle generated in step S3 in step S9. Then, ship detection processing is executed in the image represented by the detection image data. After the processing in step S9, the object detection device 1 ends the object detection processing. In one example of the object detection process, the correction unit 42 determines whether or not the first angle is greater than a predetermined angle in step S7, and corrects the tilt of the image when the first angle is greater than the predetermined angle. However, the correction unit 42 may perform the image correction process in step S8 after the process in step S6 without performing the process in step S7.

The object detection apparatus 1 according to Embodiment 1 is an object detection apparatus that detects an object using a first learning model generated using a plurality of first training data representing an image including an object such as a ship. A horizontal line detection unit 41 , a correction unit 42 and an object detection unit 43 are provided. The horizontal line detection unit 41 performs detection processing of a horizontal line included in an image represented by detection image data used for object detection. The correction unit 42 corrects the detection image data so that the first angle between the horizontal direction of the image and the horizontal line is equal to or less than a predetermined angle. The object detection unit 43 performs object detection processing using the first learning model in the image represented by the detection image data corrected by the correction unit 42 .

The horizontal line detection unit 41 detects a horizontal line in an image to be used for object detection, and the correction unit 42 corrects the first angle between the horizontal direction in the image and the horizontal line to be a predetermined angle or less. , the inclination of the image represented by the detection image data is suppressed within a predetermined range (within a predetermined angle). As a result, when an object to be detected is present in the image represented by the detection image data, the inclination of the object from the horizontal direction is also suppressed within a predetermined range. Therefore, if the inclination from the horizontal direction of the object image represented by the first teacher data at the time of learning is suppressed within a predetermined range, the object image represented by the first teacher data and the detection image Differences in tilt with the image of the object represented by the data are suppressed. That is, since the matching between the first teacher data to be learned and the detection target data is increased for the image of the target object, the accuracy of object detection by the detection process is improved. In addition, in order to detect an object tilted beyond a predetermined range from the horizontal direction, it is necessary to randomly rotate the learning target image to generate images of the object tilted at various angles during learning. Since it is no longer necessary to learn a huge number of images, it is possible to suppress an increase in calculation time during learning.

The correcting unit 42 in Embodiment 1 corrects the detection image data of the image in which the horizontal line is detected so that the first angle is equal to or less than the predetermined angle when the first angle is greater than the predetermined angle. That is, the correction unit 42 performs correction processing only when the first angle is greater than the predetermined angle, and does not perform correction processing when the first angle is less than or equal to the predetermined angle. As a result, unnecessary processing can be suppressed, and the computational processing load can be reduced.

When the first angle is larger than a predetermined angle, the correction unit 42 according to the first embodiment adjusts the detection image data of the image in which the horizontal line is detected such that the first angle is equal to or less than the predetermined angle. Correction processing corresponding to rotation processing of the image in which is detected is performed. Further, the correction unit 42 increases the amount of data allocated to the corrected detection image data according to the first angle. As a result, the information included in the corrected image data for detection is not lost, and detection failures are reduced.

When the image represented by the first training data in Embodiment 1 includes a horizontal line, the angle of the horizontal line from the horizontal direction may be less than or equal to a predetermined angle. This eliminates the need to use, as the first training data, a large number of images obtained by randomly rotating the object included in the first training data image so that it has all the tilts. It is possible to suppress the increase in time. Then, a first learning model that uses only image data in which the angle between the horizontal line and the horizontal direction is equal to or less than a predetermined angle as the first training data detects an object included in the image of the detection image data after correction. Since it is used for processing, it is possible to suppress an increase in calculation time during learning and to improve the object detection accuracy.

The object to be detected in the first embodiment is a ship. By using the above-described detection process for detecting ships, ships can accurately detect other ships, which leads to suppression of collision accidents between ships.

The horizontal line detection unit 41 according to the first embodiment performs edge detection processing on the image represented by the detection image data, and detects a straight line that is a portion where the degree of change in brightness or brightness is steep and extends linearly. Detect the part as a horizontal line. This allows horizontal lines to be detected in the image.

The horizontal line detection unit 41 according to Embodiment 1 detects the longest straight line portion as the horizontal line when there are a plurality of straight line portions in the image represented by the detection image data. This improves the detection accuracy of the horizontal line.

The horizontal line detection unit 41 according to Embodiment 1 detects a horizontal line from the degree of change of RGB in an image represented by detection image data. This allows horizontal lines to be detected in the image.

The horizon detection unit 41 according to Embodiment 1 uses the second learning model related to the sea, rivers, or the like in the horizon detection process. As a result, for example, the possibility that the boundary line between the sky and land or the boundary line between the sky and a building is detected as a horizontal line is reduced, and the horizon detection unit 41 can detect the horizon more accurately. Become.

Although the object detection device 1 includes the learning processing unit 2 in the first embodiment, the learning processing unit 2 does not necessarily have to be provided. For example, the learning processing unit 2 may be provided outside.

In the first embodiment, the detection image data is input from the camera 5. However, the detection image data is input to the object detection apparatus 1 from an external device (not shown) such as a data server via a network. It may be something that is done.

Furthermore, a ship may be provided with the object detection device 1 described in the first embodiment. Ships equipped with the object detection device 1 are, for example, container ships, oil tankers, car carriers, passenger ships, LNG (Liquefied Natural Gas) carriers, and the like. The type of ship equipped with the object detection device 1 is not limited.

Embodiment 2.
In the embodiment described above, when the horizon detection unit 41 could not detect the horizon, the ship detection processing was performed without the processing by the correction unit 42 or the like. However, it is unlikely that the ship will be detected properly while the image is tilted greatly. In addition, the horizontal line detection unit 41 may be unable to detect the horizontal line after extracting a plurality of candidate straight line portions as the horizontal line. In some cases. The object detection device 1 according to Embodiment 2 performs feedback processing when the horizon line is not detected, and performs the horizon detection processing again, thereby improving the ship detection accuracy. Each component of the object detection device 1 according to the second embodiment is the same as that of the first embodiment shown in FIG. However, in the second embodiment, it is assumed that the object detection unit 43 and the horizon detection unit 41 are connected in addition to the connection relationship between the constituent elements shown in FIG. In the following description, unless otherwise specified, only the functions and operations of the constituent elements of the object detection apparatus 1 according to the second embodiment that are different from those of the first embodiment will be described.

When the horizon detection unit 41 in Embodiment 2 does not detect the horizon and the object detection unit 43 detects at least one ship, the detected waterline of the ship is detected. . In addition, the waterline shall mean the line along which a ship floating on the water touches the surface of the water.

When the waterline is detected, the horizon detector 41 estimates the horizon from the waterline. The direction of the horizon is often equal to the direction of the waterline. So, for example, if one waterline is detected, the direction of the waterline can be the direction of the horizon. Since the angle between the horizontal line and the horizontal direction is used for image rotation, it is sufficient if the direction of the horizontal line can be estimated. Any straight line parallel to the waterline may be assumed as the horizon. On the other hand, when a plurality of waterlines are detected, the horizontal line detection unit 41 selects an arbitrary straight line whose slope is the average value or the mode of the slopes of these waterlines from the horizontal direction, or one of these waterlines. Any straight line parallel to one of may be assumed as a horizontal line.

The correction unit 42 performs the same correction as described above based on the second angle, which is the angle from the horizontal direction of the horizontal line estimated by the horizontal line detection unit 41 .

5 and 6 are flowcharts showing an example of object detection processing by the object detection device according to the second embodiment. Since the processing in each step in the series of processing from step S11 to step S19 shown in FIG. 5 is the same as the processing in each step from step S1 to step S9 in the first embodiment, description thereof will be omitted. After the ship detection processing by the object detection unit 43 in step S19, the object detection unit 43 determines in step S20 whether or not the horizon has already been detected by the processing in step S14. Note that the determination may be performed by the horizontal line detection unit 41 . If the horizon has already been detected (step S20: YES), the object detection device 1 terminates the object detection process.

When the horizontal line is not detected by the processing in step S14 (step S20: NO), in step S21, the object detection unit 43 determines whether or not at least one vessel has been detected. Note that the determination may be performed by the horizontal line detection unit 41 . If at least one ship has not been detected (step S21: NO), the object detection device 1 terminates the object detection process. When at least one vessel is detected (step S21: YES), the horizon detector 41 performs detection processing of the waterline of the detected vessel in step S22 in FIG.

In step S22, the horizontal line detection unit 41 performs, for example, at least one of edge detection processing, processing using differences in RGB values, and processing using the second learning model. A detection process for the waterline in the represented image is performed. For example, the horizon detection unit 41 focuses on the detected ship and performs edge detection processing or processing using differences in RGB values in the downward direction along the vertical direction in the image, thereby detecting the waterline. processing may be performed. The horizontal line detection unit 41 may also detect the waterline by performing edge detection processing along the horizontal direction of the image, processing using differences in RGB values, or the like. .

When the horizon detection unit 41 detects the waterline of at least one vessel through the waterline detection process in step S22 (step S23: YES), the horizon detection unit 41 detects the horizon based on the detected waterline in step S24. presume. The horizontal line may be estimated as a straight line parallel to one of the detected one or more waterlines, or may be estimated as a straight line whose slope is the average value of the slopes of the plurality of detected waterlines. .

After the processing in step S24, in step S25, the correction unit 42 calculates a second angle between the horizontal direction and the horizontal line estimated in step S24. In step S26, the correction unit 42 determines whether or not the second angle calculated in step S25 is greater than the predetermined angle described above. If the second angle is greater than the predetermined angle (step S26: YES), the correction unit 42 corrects the detection image data in step S27 so that the second angle is equal to or less than the predetermined angle. This corrects the tilt of the image. For example, the correction unit 42 may correct the detection image data so that the second angle becomes 0°, like the image shown in FIG. 2 is corrected to the image shown in FIG. Alternatively, the correction unit 42 may correct the detection image data so that the second angle becomes a specific angle equal to or less than a predetermined angle such as 1° or 2°. Alternatively, the correction unit 42 determines that the magnitude of the second angle is the mode or average value of the magnitudes of the angles between the horizontal line and the horizontal direction in the image represented by each of the plurality of first teacher data. The detection image data may be corrected so that Note that the mode value or the average value is equal to or smaller than the predetermined angle and may be stored in the storage unit 3 in advance.

In step S27, the correction unit 42 increases the range in the X direction and the range in the Y direction of the detection image data after correction from the detection image data before correction, as in step S8. The amount of data that can include the X coordinate, the Y coordinate, and the color information of each pixel in the image data for detection is set as the amount of data for the image data for detection after correction. Subsequent to step S27, in step S28, the object detection unit 43 executes the process of detecting the ship in the image represented by the corrected detection image data and not detected in step S19.

If the waterline is not detected in step S23 (step S23: NO), the correction unit 42 does not execute the process, and in step S28 the object detection unit 43 detects the detection image data generated in step S13. A detection process is executed for any vessel in the displayed image that was not detected in step S19. Note that when the waterline is not detected in step S23, the object detection device 1 may end the object detection process instead of the process in step S28.

If the second angle is equal to or less than the predetermined angle in step S26 (step S26: NO), the correction unit 42 does not perform correction processing, and in step S28 the object detection unit 43 detects the detection generated in step S13. A detection process for a ship that is not detected in step S19 in the image represented by the image data is executed. Note that when the second angle is equal to or less than the predetermined angle in step S26, the object detection device 1 may end the object detection process instead of the process in step S28. In addition, the correction|amendment part 42 does not necessarily need to perform step S26. That is, the correction unit 42 may perform rotation processing on all detection image data, not only when the second angle is greater than the above-described predetermined angle. After the processing in step S28, the object detection device 1 ends the object detection processing.

The horizon detection unit 41 according to the second embodiment detects the waterline of the detected ship when the object detection unit 43 detects at least one ship when the horizon is not detected. Then, when at least one waterline is detected, the horizon detector 41 estimates the horizon from the at least one detected waterline. The correction unit 42 corrects the detection image data of the image in which the horizontal line is estimated so that the second angle of the estimated horizontal line from the horizontal direction is equal to or less than a predetermined angle. The object detection unit 43 uses the first learning model in the image represented by the detection image data after the second angle is corrected to be equal to or less than the predetermined angle, and detects a ship other than the already detected ship. Perform detection processing. As a result, even if the horizon is not detected, the object detection unit 43 estimates the horizon through feedback processing and corrects the inclination of the image to detect other ships that could not have been detected without correcting the image. It can be detected, and the detection accuracy of the ship is improved.

The correction unit 42 according to the second embodiment corrects the detection image data of the image in which the horizontal line is estimated so that the second angle is equal to or less than the predetermined angle when the second angle is greater than the predetermined angle. That is, the correction unit 42 performs correction processing only when the second angle is greater than the predetermined angle, and does not perform correction processing when the second angle is less than or equal to the predetermined angle. As a result, unnecessary processing can be suppressed, and the computational processing load can be reduced.

When the image represented by the first training data in Embodiment 2 includes a horizontal line or a waterline, the angle of the horizontal line or the waterline from the horizontal direction may be less than or equal to a predetermined angle. This eliminates the need to use the image data of the randomly rotated image as the first teacher data, thereby eliminating the need to rotate the image data to be learned. Then, a first learning model that uses only image data in which the angle between the horizontal line or the waterline and the horizontal direction is equal to or less than a predetermined angle as the first training data is used as the first learning model to determine the object using the corrected detection image data. Since it is used for detection processing, the accuracy of object detection can be improved without increasing the learning time.

A program for causing a computer to execute the object detection method described with reference to FIGS. , or may be installed in the object detection device 1 via a network. Note that the recording medium is, for example, a magnetic disk or an optical disk. The optical disc is, for example, a CD (Compact Disc) or a DVD (Digital Versatile Disc). Alternatively, the recording medium may be a USB (Universal Serial Bus) memory equipped with a semiconductor memory such as a flash memory.

1 object detection device, 2 learning processing unit, 3 storage unit, 4 detection processing unit, 5 camera, 40 detection image acquisition unit, 41 horizon detection unit, 42 correction unit, 43 object detection unit, 50 lens, 51 imaging unit, 52 A/D conversion circuit, 53 output section, A, B area, C horizon, D artifact, E ship, F1, F2 frame, L1, L2 length.

Claims (10)

An object detection device that detects an object using a first learning model generated using a plurality of first training data representing an image containing the object,
a horizontal line detection unit that detects a horizontal line included in an image represented by the detection image data used to detect the object;
a correction unit that corrects the detection image data so that a first angle between the horizontal direction in the image and the horizontal line is equal to or less than a predetermined angle;
an object detection unit that performs detection processing of the object using the first learning model in an image represented by the detection image data corrected by the correction unit;
An object detection device comprising: wherein, when the first angle is greater than the predetermined angle, the correction unit corrects the detection image data of the image in which the horizontal line is detected so that the first angle is equal to or less than the predetermined angle. Item 1. The object detection device according to item 1. The correction unit is
When the first angle is greater than the predetermined angle, the horizontal line is detected such that the first angle is less than or equal to the predetermined angle with respect to the detection image data of the image in which the horizontal line is detected. 3. The object according to claim 1, wherein correction processing corresponding to rotation processing of the image is performed, and a data amount assigned to the detection image data after correction is increased according to the first angle. detection device. 4. The object detection device according to claim 1, wherein the object is a ship. The horizontal line detection unit
Edge detection processing is performed on the image represented by the detection image data, and a linear portion, which is a portion where the degree of change in luminance or brightness is steep and extends linearly, is detected as the horizontal line. The object detection device according to any one of claims 1 to 4. The horizontal line detection unit
6. The object detection device according to claim 5, wherein when the image represented by the detection image data includes a plurality of the straight line portions, the longest straight line portion is detected as the horizontal line. The horizontal line detection unit
7. The object detection apparatus according to claim 5, wherein said horizon detection process uses a second learning model generated using second teacher data representing an image including the sea or a river. The horizontal line detection unit
When the horizontal line is not detected and the object detection unit detects at least one of the objects from the image represented by the detection image data, detection processing of the waterline of the detected object is performed. and if at least one of said waterlines is detected, estimating a horizon from said at least one detected waterline;
The correction unit is
correcting the detection image data of the image in which the horizontal line is estimated such that a second angle of the estimated horizontal line from the horizontal direction is equal to or less than the predetermined angle;
The object detection unit is
In the image represented by the detection image data after the second angle has been corrected to be equal to or less than the predetermined angle, using the first learning model, an object other than the at least one object that has already been detected , the object detection device according to any one of claims 1 to 7, wherein the object detection process is performed. A ship equipped with the object detection device according to any one of claims 1 to 8. An object detection method executed by an object detection device that detects an object using a first learning model generated using a plurality of first training data representing an image containing the object, comprising:
a horizontal line detection step of performing detection processing of a horizontal line included in an image represented by the detection image data used for detecting the object;
a correction step of correcting the detection image data such that a first angle between the horizontal direction in the image and the horizontal line is equal to or less than the predetermined angle;
an object detection step of performing detection processing of the object using the first learning model in an image represented by the detection image data after correction in the correction step;
An object detection method comprising:

Images