JP7030732B2 - Same structure detection device, same structure detection method and same structure detection program - Google Patents

Description

The present invention relates to a technique for determining the identity of a structure captured in a plurality of captured images.

Public infrastructure is the foundation of industry and livelihoods. Examples of public infrastructure installed along roads include structures such as traffic lights, lamps, and utility poles. Inspection and maintenance of public infrastructure is important. For example, if the traffic light of the traffic light is dirty or a tree hides the traffic light, the traffic light becomes difficult to see, which may lead to a serious accident.

"Digital Transformation that Innovates the Maintenance Cycle of Communication Infrastructure", NTT Technology Journal, Nippon Telegraph and Telephone Corporation, January 2019, Volume 31, Issue 1, pp. 32-35

Conventionally, abnormalities in structures along the road have been visually confirmed and inspected and maintained by walking or by bicycle. However, the structures along the road are scattered over a long distance, and the amount is very large, so that an efficient inspection method has been desired.

A possible method is to drive a car along the road, take a picture of the structure along the road with an in-vehicle camera, and watch the video taken to detect an abnormality in the structure. When a long structure such as a traffic light, a light, and a utility pole is photographed with one camera, the entire structure may not be captured when the distance between the camera and the structure is short. If you take a picture of a structure with multiple cameras with different tilt angles, you can take a picture of the entire structure. Furthermore, by changing not only the tilt angle but also the pan angle for each camera, a more suitable image can be obtained for inspection.

However, if the pan angle is changed for each camera, there is a problem that the time when the structure is photographed shifts. It is also difficult to determine the same structure by comparing a plurality of captured moving images.

The present invention has been made in view of the above, and an object of the present invention is to detect the same structure among a plurality of captured images.

The same structure detection device according to the present invention is the same structure detection device that extracts a photographed image showing the same structure from a plurality of photographed image groups photographed by a plurality of cameras at regular intervals. The same structure in the plurality of captured image groups based on the difference in the imaging timing of the structure considering the distance and the imaging direction of each of the plurality of cameras or the distance from the imaging position of each of the plurality of cameras to the structure. It is characterized by comprising the same structure determination unit that estimates a reference image to be captured and determines that the structure captured in the captured image within a predetermined range from the reference image is the same structure.

According to the present invention, the same structure can be detected among a plurality of captured images.

It is a functional block diagram which shows the structure of the structure detection apparatus in this embodiment. It is a figure for demonstrating the installation situation of a plurality of cameras installed in a vehicle. It is a figure for demonstrating the installation situation of a plurality of cameras installed in a vehicle. It is a figure which shows the example of the frame cut out at a fixed distance from a photographed moving image. It is a figure which shows the state which the structure to be detected was detected in the frame. It is a figure which shows how the position of a structure moves in a frame. It is a figure which shows the appearance of the same structure in the frame cut out from the moving image taken by a plurality of cameras. It is a figure which shows the output screen example. It is a flowchart of the main routine of a structure detection device. It is a flowchart which shows the flow of the same determination process of a structure in a single camera. It is a figure which shows the example of the detection frame which was erroneously detected. It is a figure which shows the example of the overlapping detection frame which detected one structure a plurality of times. It is a figure which shows the example of the overlap detection frame when a plurality of structures overlap. It is a flowchart which shows the flow of the determination process of the same structure in the moving image taken by a single camera. It is a figure which shows the example which a new structure appears in a frame. It is a figure which shows another example in which a new structure appears in a frame. It is a figure which shows the change of the detection position of a structure in a frame. It is a figure which shows the example which the detection position of a structure moves out of a frame. It is a flowchart which shows the flow of the allocation process of the remaining structure. It is a figure which shows the example which increased the number of structures detected in the current frame. It is a figure which shows the example which the number of structures detected in the current frame decreased. It is a flowchart which shows the flow of the determination process of the same structure among the moving images taken by a plurality of cameras. It is a figure which shows the example which the representative frame exists in a reference position. It is a figure which shows the example which the representative frame exists in the position deviated from the reference position. It is a figure which shows the example which there are a plurality of representative frames within an allowable range. It is a figure which shows the measurement example of LIDAR. It is a figure for demonstrating the calculation of the deviation of the position where a structure is photographed. It is a figure for demonstrating the calculation of the time difference between GPS and a camera.

One of the embodiments of the present invention will be described below.

[Structure of structure detection device]
The structure detection device 1 of the present embodiment extracts a frame showing the structure to be detected from a plurality of shot videos taken by a plurality of cameras installed on the vehicle for the structure along the road, and takes a plurality of shots. It is a device that links and displays frames in which the same structure appears in a moving image in an integrated manner. In this embodiment, a traffic light will be described as an example of the structure to be detected. The structure to be detected may be a lamp, a utility pole, or a roadside tree as long as it is installed along the road, or it may be a low-height structure such as a signboard, a sign, a vending machine, or a post. There may be. The detection target may be a road marking or the like taken by pointing the camera downward.

The structure detection device 1 shown in FIG. 1 includes a still image acquisition unit 11, a storage unit 12, a structure detection unit 13, the same structure determination unit 14 for each camera, the same structure determination unit 15 between cameras, and an output unit 16. , And an abnormality detection unit 17. Each part included in the structure detection device 1 may be configured by a computer provided with an arithmetic processing unit, a storage device, and the like, and the processing of each part may be executed by a program. This program is stored in a storage device included in the structure detection device 1, and can be recorded on a recording medium such as a magnetic disk, an optical disk, or a semiconductor memory, or can be provided through a network.

The still image acquisition unit 11 acquires a still image used for detecting a structure. In the present embodiment, a commercially available camera and a commercially available GPS logger are mounted on a vehicle to photograph a structure along a road while traveling, and to acquire position information of the vehicle. The captured moving image is stored in the recording medium 101, and the position information of the vehicle is stored in the recording medium 102. The still image acquisition unit 11 reads the captured moving image from a recording medium 101 such as a memory card in which the captured moving image is stored, acquires the position information from the recording medium 102 in which the file in which the position information is recorded is stored, and obtains the position information at regular intervals. Get a still image. The still image acquisition unit 11 stores the still image acquired at regular intervals in the storage unit 12 in association with the position information acquired from the recording medium 102. In the present embodiment, since the structure is photographed by using a plurality of cameras, still images are acquired from each of the plurality of photographed moving images, and the still image group is stored in the storage unit 12 for each photographed moving image. The still image acquisition unit 11 may acquire the captured moving image and the position information via the network. The still image acquisition unit 11 is not limited to moving images, and may read still images taken at predetermined intervals.

As shown in FIGS. 2 and 3, in the present embodiment, a plurality of cameras having different pan angles and tilt angles with respect to the traveling direction of the vehicle are mounted on the vehicle and simultaneously photographed. For example, the cameras 1 in FIGS. 2 and 3 are installed so that the pan angle is 90 degrees and the tilt angle is 0 degrees, and photographs the lateral direction of the vehicle. The cameras 2 to 4 are installed in order so that the pan angle is small and the tilt angle is large. The pan angle and tilt angle are adjusted so that the entire long structure such as a traffic light can be photographed by the four cameras 1 to 4. Specifically, the camera 1 photographs the lower part of the structure, and the camera 4 photographs the crown of the structure. Cameras 2 and 3 take pictures of the middle part of the structure. Either the pan angle or the tilt angle of a plurality of cameras may be changed.

The camera 1 for photographing the lower part of the structure may be installed so as to be oriented sideways with respect to the traveling direction of the vehicle. This makes it possible to take a clearer picture of a name plate or a signboard attached to the bottom of a structure. The camera 4 for photographing the top of the head of the structure may be installed so as to photograph diagonally forward with a small pan angle. If you look up from the side of the vehicle and take a picture of the top of the structure, you may not be able to take a picture of the top of the head due to roadside trees. It is advisable to reduce the pan angle and shoot the top of the head from a distance.

Not only a camera that shoots diagonally forward, but also a camera that shoots diagonally backward may be installed. By installing a camera that shoots in the front-back direction, even if the front camera is backlit, the rear camera will be in front light. A camera that shoots the front (direction of travel) or a camera that shoots the rear may be installed.

The still image acquisition unit 11 acquires a still image (hereinafter, also referred to as a “frame”) from the captured moving image at regular intervals based on the position information acquired by the GPS logger. When acquiring the frame, as shown in FIG. 4, the cutting distance is adjusted so that one structure is reflected in the frames of 3 to 5. In the example of FIG. 4, frames are cut out from the captured moving image every 10 cm. The number assigned below the frame in FIG. 4 is a frame number. The still image acquisition unit 11 assigns frame numbers in the order in which the frames are cut out. The cutting distance may be changed depending on the pan angle of the camera of the captured moving image, that is, the distance from the camera to the structure. Specifically, the video shot with a camera with a small pan angle with respect to the traveling direction of the vehicle shows the structure for a longer time than the video shot with a camera installed sideways (with a large pan angle) of the vehicle. Therefore, the cutting distance is lengthened. For example, assuming that the cut-out distance of the moving image taken by the lateral camera 1 is 10 cm, the cut-out distance of the camera 2 is 20 cm, the cut-out distance of the camera 3 is 30 cm, and the cut-out distance of the camera 4 is 50 cm. The appropriate value for the cutting distance depends on the distance between the vehicle and the structure. Therefore, the structure is set so as to be within an appropriate number of images according to the shooting conditions. On a road with a wide shoulder, the distance between the vehicle and the structure becomes long, so the cutting distance is lengthened.

The storage unit 12 stores the frames acquired by the still image acquisition unit 11 for each captured moving image. For example, when the still image acquisition unit 11 acquires a frame from four captured moving images, the storage unit 12 stores four still image groups cut out from each of the captured moving images. The storage unit 12 may store the captured moving image read from the recording medium 101.

The structure detection unit 13 detects a frame in which the structure to be detected is reflected from the frames included in each of the still image groups stored in the storage unit 12, and stores the position of the structure in the frame in the storage unit 12. .. The structure detection unit 13 detects, for example, a frame in which the structure is captured by using artificial intelligence trained in the image of the structure to be detected. In the example of FIG. 5, the structure detection unit 13 detected the lower part of the traffic light TL. The structure detection unit 13 stores the position information in the frame of the detection frame D that has detected the traffic light TL in the storage unit 12. For example, the coordinates, width, height, and certainty of the detection result on the frame of the detection frame D are stored in the storage unit 12.

The same structure determination unit 14 for each camera (hereinafter, simply referred to as “same structure determination unit 14”) is between frames based on the change in the position of the structure within the frame detected in a plurality of consecutive frames. Judge the same structure in. Specifically, the same structure determination unit 14 determines that the structures that move in the same direction as the movement direction of the landscape in the frame in the frame are the same structure. The plurality of continuous frames are a series of frames cut out from the captured moving image by the still image acquisition unit 11 at regular intervals. When the left side with respect to the traveling direction of the vehicle is photographed, the structure moves from right to left in the frame as shown in FIG. The same structure determination unit 14 determines that a structure whose detection position (detection frame) moves from right to left in a plurality of consecutive frames is the same structure. When the right side of the vehicle's direction of travel is photographed, the structure moves from left to right within the frame. In a frame taken with a camera installed in front of or behind the vehicle, the structure moves from top to bottom or from bottom to top within the frame. That is, the same structure determination unit 14 determines a structure that moves in the same direction as the movement direction of the landscape in the frame as the same structure.

The same structure determination unit 14 extracts a plurality of frames in which the same structure is determined to be captured, and then selects a frame in which the structure is most captured from among the plurality of frames as a representative frame. When a camera that shoots the front and the back is installed, a normal light frame may be automatically selected as a representative frame.

The same structure determination unit 14 performs the above-mentioned determination process of the same structure for each still image group, that is, a captured moving image.

By the determination process of the same structure determination unit 14, it is possible to extract a frame in which the same structure is captured from a group of still images cut out at regular intervals from the captured moving image.

The same structure determination unit 15 between cameras (hereinafter, simply referred to as “same structure determination unit 15”) captures the same structure based on the shooting direction of each camera and the cutout interval (distance) of the still image group. The representative frame is associated with the still image group.

By the determination process of the same structure determination unit 15, as shown in FIG. 7, it is possible to extract a frame in which the same structure is captured from a group of still images cut out from moving images taken by different cameras.

The same structure determination unit 15 may register the position information of the structure detected from a plurality of still image groups and determined as the same structure in the storage unit 12. For example, an ID is assigned to the structure, the frame in which the structure is shown is associated with the ID, and the position information of the structure is associated with the ID. As the position information of the structure, the position information of the vehicle obtained from the GPS logger can be used. For example, the position information of the representative frame taken by the camera installed sideways with respect to the traveling direction of the vehicle is used as the position information of the structure. At this time, the shooting direction of the camera may be specified based on the traveling direction of the vehicle and the installation direction of the camera, and the position where the position information is moved in the shooting direction may be used as the position information of the structure. The distance from the vehicle to the structure may be measured by LIDAR (Light Detection and Ringing), and the position of the structure may be obtained more accurately. Further, when a camera for photographing the front and the rear is installed, the same structure determination unit 15 is one of the frames for photographing the same structure obtained from the front and rear cameras for photographing the same position of the structure. One frame may be selected, and in particular, a normal light frame may be selected from the captured frames. For example, it may be determined to be forward light or backlight based on the brightness in the detection frame of the frame, or the direction of the vehicle is obtained based on the information obtained from the GPS logger, and the camera is installed based on the camera installation angle. The shooting direction may be determined and checked against the direction of the sun according to the time to determine whether or not the subject is backlit.

The abnormality detection unit 17 detects an abnormality (for example, crack, chipping, rust, etc.) of the structure from the frame in which the structure is shown. The abnormality detection unit 17 may detect an abnormality in the structure not only in the representative frame but also in all the frames in which the structure is shown. The abnormality detection unit 17 detects the abnormality of the structure from the frame in which the structure is shown by using artificial intelligence that has learned the abnormality of the structure to be detected, for example. As the artificial intelligence, for example, by using an image deep learning technique, it is possible to detect an image portion of a crack, a chip, or a rust in an input image by learning an image of a crack, a chip, or a rust of a structure in advance. By using the learned artificial intelligence as the abnormality detection unit 17, the abnormality of cracks, chips, or rust generated in the structure may be detected.

The output unit 16 outputs a plurality of still images of the same structure taken by different cameras side by side. By the processing of the structure detection unit 13 and the same structure determination units 14 and 15, the identity of the structure reflected in the frame cut out from a plurality of captured moving images is determined, and the position of the structure is also specified. For example, the output unit 16 plots the position where the structure is detected on a map, accepts the selection of the structure on the map, and displays a still image showing the selected structure.

FIG. 8 shows an example of an output screen. The output screen 100 shown in the figure is provided with a map area 110, an information area 120, and a moving image area 130 in the upper row, and still images 141 to 144 showing a structure are displayed in the lower row.

In the map area 110, the structure detected from the captured moving image is plotted on the map. The color and shape of the plot may be changed depending on the presence and degree of defects in the structure. For example, a structure without defects is plotted in blue. Plot in yellow or red depending on the degree of defect. When the mark indicating the structure is selected on the map, the still images 141 to 144 showing the selected structure are displayed in the lower row. The displayed still images 141 to 144 are representative frames selected by the same structure determination unit 14.

In the map area 110 of FIG. 8, the mark of the structure is displayed on the road, but the mark of the structure may be displayed closer to the right side or the left side of the road. For example, when the left side of the traveling direction of the vehicle is photographed, the mark of the structure is displayed on the left side of the traveling direction. Alternatively, the traveling direction of the vehicle may be illustrated on the road with an arrow or the like.

In the information area 120, information such as the selected structure is displayed. For example, the shooting date and time and the position information of the structure are displayed. Further, in the example of FIG. 8, it is possible to select whether or not to display the cracks and chips detected by the abnormality detection unit 17 described later. When the checked abnormality exists in the still image, the position where the abnormality is detected may be indicated as shown in the still image 142 of FIG. If an abnormality is detected in an undisplayed still image (one showing the same structure) other than the representative frame, that fact may be displayed on the information area 120 or the still images 141 to 144.

If there is a still image of the structure taken at different times, the shooting date and time may be selectable and the still image of the structure taken at the selected shooting date and time may be displayed. For example, if there are still images taken in 2014 and still images taken in 2018 for the selected structure, the shooting date and time of the information area 120 can be selected to be either 2014 or 2018. do. A still image taken at the selected time is displayed in the lower part of the output screen 100. Further, only the structure for which an abnormality is newly detected may be displayed in the map area 110, or different marks may be displayed so as to be distinguishable.

In the moving image area 130, the captured moving image taken by any of the cameras is reproduced. The shooting video to be displayed can be selected. In the example of FIG. 8, the captured moving image to be played back can be selected from the four captured moving images. For example, when any of the still images 141 to 144 is selected, the corresponding captured moving image is displayed in the moving image area 130. When playing back the captured video, the position of the vehicle may be illustrated in the map area 110 according to the time stamp of the captured video. While the detected structure is shown in the captured video, the captured video may be played back slowly.

In the lower part of the output screen 100, still images 141 to 144 of the structure taken by a plurality of cameras are displayed. When any of the still images 141 to 144 is selected, a plurality of still images determined by the same structure determination unit 14 to show the same structure may be displayed side by side. For example, when the still image 142 is selected, a still image (taken with the same camera) showing several same structures before and after the still image 142 is displayed.

In addition, an image of the surroundings of the structure may be displayed to determine if an aerial work platform is available during maintenance of the structure. For example, a panoramic image is generated and displayed by connecting a still image of the lower part of the structure to a still image around the structure.

[Operation of structure detection device]
Next, the operation of the structure detection device 1 will be described. FIG. 9 is a flowchart of the main routine of the structure detection device 1.

When the structure detection device 1 starts the process, in step S1, the still image acquisition unit 11 reads the captured moving image from the recording medium 101 and reads the file in which the position information is recorded from the recording medium 102. The still image acquisition unit 11 acquires a frame from the captured moving image at regular intervals, associates the position information with the frame, and stores the frame in the storage unit 12. The still image acquisition unit 11 acquires a photographed moving image from each of the plurality of recording media 101, and performs the process of step S1 for each of the acquired photographed moving images.

In step S2, the structure detection unit 13 detects a frame in which the structure to be detected is reflected from the frame acquired by the still image acquisition unit 11, and stores the position of the structure in the frame in the storage unit 12.

In step S3, the same structure determination unit 14 determines the identity of the structure based on the change in the detection position of the structure in the frame for each still image group acquired from the same captured moving image. Select a representative frame from the frames that show the same structure. The same structure determination unit 14 performs the process of step S3 for each of the still image groups acquired from each of the captured moving images. The details of the structure identity determination in the single camera in step S3 will be described later.

In step S4, the same structure determination unit 15 associates the representative frames in which the same structure is captured between the still image groups based on the shooting direction and the cutting interval of the shot moving image. The details of the structure identity determination among the plurality of cameras in step S4 will be described later.

In step S5, the output unit 16 displays a plurality of representative frames of the same structure taken by different cameras side by side.

[Identity determination of structures in a single camera]
Subsequently, the identity determination process of the structure in a single camera will be described. FIG. 10 is a flowchart showing the flow of the same determination process of the structure in a single camera.

In step S31, the same structure determination unit 14 deletes the detection frame erroneously detected by the structure detection unit 13. For example, as shown in FIG. 11, when the structure detection unit 13 detects the structure in the frame n, but the structure is not detected in the previous and next several frames n-1, n + 1, the detection frame D0 of the frame n Is determined to be an erroneous detection, and the detection frame D0 is deleted.

In step S32, the same structure determination unit 14 merges detection frames having overlapping detection areas in one frame. For example, as shown in FIG. 12, only one structure is shown in the frame, but when the structure detection unit 13 outputs a plurality of detection frames D1 and D2 having overlapping regions for the structure, they are the same. The structure determination unit 14 merges the detection frames D1 and D2. Either of the detection frames D1 and D2 may be deleted.

Further, as shown in FIG. 13, it is conceivable that a plurality of structures are overlapped in the frame and the plurality of detection frames D3 and D4 are overlapped. It may be determined whether or not to process the overlapping detection frames according to the degree of overlap of the detection frames. For example, when the degree of overlap of the detection frames is equal to or higher than a predetermined threshold value, the overlap processing of the detection frames is performed. If the structures to be detected are duplicated and do not appear in the frame, it may be determined whether to merge or delete the detection frames according to the detection target.

The same structure determination unit 14 performs the process of step S33 for each frame of the still image group in the order of frame numbers, and obtains a set of frames showing the same structure (hereinafter referred to as “set of same structure”). Ask. In step S33, the same structure determination unit 14 determines the same structure between frames based on the change in the position of the structure within the frame detected in a plurality of consecutive frames. Details of the process in step S33 will be described later.

In step S34, the same structure determination unit 14 deletes a set in which the number of elements (number of frames) of the set of the same structure is less than a certain value as a false detection. Since the still image acquisition unit 11 cuts out frames so that the same structure appears in a predetermined number of frames or more, the same structure determination unit 14 determines that the number of frames in which the same structure appears is the number of frames. A set below a certain level (for example, 1 to 2 frames when the same structure is cut out so as to appear in about 5 frames) is presumed to be a false detection of the structure. For example, in a video shot taken by a camera that shoots the front left side of a vehicle, a structure on the right side of the direction in which the vehicle turns left when the vehicle turns left moves from the left side to the right side in the shot video. Since the same structure determination unit 14 determines the structure moving from the right side to the left side in the captured moving image as the same structure, the structure moving from the left side to the right side in the captured moving image is the frame cut out from the captured moving image. Each is treated as a newly detected structure. Therefore, the number of elements in the set of frames of the structure is about 1 or 2, and it can be deleted as a false positive.

In step S35, the same structure determination unit 14 selects a representative frame from the set of the same structure. Among the frames included in the set of the same structure, the frame in which the structure is best captured is taken as the representative frame. As the frame that appears best, for example, the middle frame when the frames included in the set of the same structure are arranged in the order of frame numbers may be used as the representative frame. Alternatively, one frame in which the structure is shown in the center may be selected as the representative frame of the structure. Alternatively, when a camera that shoots the front and the rear is installed, a frame of forward light may be automatically selected as a representative frame. For example, it is determined to be forward light or backlight based on the brightness in the detection frame of the frame. You may. Alternatively, a representative frame may be selected from frames other than such frames by determining shooting defects such as out-of-focus and blurring.

By the above processing, a set of frames in which the same structure is shown and a representative frame of the structure are obtained for each still image group cut out from the captured moving image.

Subsequently, the determination process of the same structure in step S33 of FIG. 10 will be described. FIG. 14 is a flowchart showing the flow of determination processing of the same structure in a moving image taken by a single camera.

In step S331, the same structure determination unit 14 determines whether or not the structure to be detected is detected from the current frame to be processed.

When a structure is detected from the current frame, in step S333, the same structure determination unit 14 determines whether or not the same candidate structure exists. The same candidate structure is a structure (detection frame) detected up to the previous frame n-1 and is presumed to be the same structure. The set of the same candidate structure is a set of structures presumed to be the same structure. There may be multiple sets of the same candidate structure. Structures detected after the current frame n may be added to the set of the same candidate structures.

If the same candidate structure does not exist, a new set of the same candidate structure is created in step S334, and the structure detected from the current frame n is registered in the set of the same candidate structure. When a plurality of structures are detected from the current frame n, a new set of the same candidate structures is created by the number of the detected structures.

When the same candidate structure exists, in step S335, the same structure determination unit 14 determines the structure detected on the right side of the coordinates of the rightmost structure among the same candidate structures in the current frame n. Register in a new set of identical candidate structures. Since the position of the same structure reflected in the frame moves from the right side to the left side, it is estimated that the structure detected on the right side of the structure detected up to the previous frame n-1 is a new structure. do.

The same structure determination unit 14 determines the right side or the left side by comparing the positions of the detection frames that have detected the structure in the frame. For example, the x-coordinate within the frame of the vertical center line of the detection frame is used as a reference.

In the example of FIG. 15, the structure of the detection frame D5 was detected in the front frame n-1. In the process of the previous frame n-1, the detection frame D5 is registered in the set of any of the same candidate structures. It is assumed that the detection frame D5 is on the rightmost side of the same candidate structure. In the current frame n, the structure of the detection frame D6 and the structure of the detection frame D7 were detected. The position of the detection frame D6 in the frame is on the left side of the position of the detection frame D5 in the frame, and the position of the detection frame D7 in the frame is on the right side of the position of the detection frame D5 in the frame. Since the detection frame D7 appears on the right side of the rightmost detection frame D5 in the same candidate structure, the detection frame D7 is registered in a new set of the same candidate structures.

In the example of FIG. 16, in the current frame n, the structure of the detection frame D7 and the structure of the detection frame D8 were detected. The positions of the detection frames D7 and D8 in the frame are both to the right of the positions in the frame of the detection frame D5. Since the detection frames D7 and D8 are detected on the right side of the rightmost detection frame D5 in the same candidate structure, each of the detection frames D7 and D8 is registered in each of the new sets of the same candidate structures. That is, two new sets of the same candidate structure are created.

If the fixed distance to be cut out from the captured video is short, the frame in which the structure is shown may be cut out at the same position due to the error of the GPS logger, such as when the vehicle is temporarily stopped. For example, frame n-1 and frame n in FIG. 17 have the same position in the frame of the structure. At this time, the structure detection unit 13 may detect the structure at a different position. In the example of FIG. 17, in the frame n-1, the structure is detected by the detection frame D9 on the left side, and in the frame n, the structure is detected by the detection frame D10 on the right side. Although the detection frames D9 and D10 detect the same structure, the detection frame D10 of the current frame n exists on the right side of the detection frame D9, so that the same structure determination unit 14 uses the detection frame D10 as a new same candidate. It will be registered as a structure. Therefore, the same structure determination unit 14 sets the deviation allowable range in pixel units, and even if the detection frame D10 is on the right side of the detection frame D9, if it is within the deviation allowable range, the detection frame D10 is set to the detection frame D9. Same candidate structure.

Among the structures detected in the current frame n in step S335, the structures registered in the set of the same candidate structures are not subject to the subsequent processing in steps S336 and S337. Specifically, in the example of FIG. 15, since the detection frame D7 detected in the current frame n is registered in the new set of the same candidate structures, it is not subject to the processing in steps S336 and S337. Since the detection frame D6 detected in the current frame n has not yet been registered in the set of the same candidate structures, it is the processing target of steps S336 and S337.

In step S335, when all the structures detected in the current frame n are registered in the new set of the same candidate structures (example of FIG. 16), the same candidate structures created before the previous frame n-1 Confirm the confirmation of the set. Confirmation is to presume that the same structure as the same candidate structure does not appear after the current frame n, and to confirm the structure included in the set of the same candidate structure as the same structure. At the time of confirmation of confirmation, the allowable number of detection omissions may be set, and if the set of the same candidate structures does not change beyond the allowable number, the set of the same candidate structures may be confirmed as the same structure. .. The same structure should appear in consecutive frames. If no structure is detected in a certain frame and a structure moving from right to left is detected in the frames within the allowable number of frames before and after, it is estimated that the frame in which the structure is not detected is a detection omission. do.

In step S336, the same structure determination unit 14 determines the set of the same candidate structures on the left side of the structure detected on the leftmost side in the current frame n as the same structure.

In the example of FIG. 18, in the front frame n-1, the structures of the detection frames D12 and D13 are detected. That is, at the time of the previous frame n-1, there are two sets of the same candidate structure including the detection frame D12 and a set of the same candidate structures including the detection frame D13. In the current frame n, the structure of the detection frame D14 was detected. The detection frame D14 is on the right side of the detection frame D12 and on the left side of the detection frame D13. The structure of the detection frame D12 does not move to the position of the detection frame D14 in the next frame n, and the structure that can be estimated that the structure of the detection frame D12 has moved is not detected in the current frame n. Therefore, since it can be estimated that the same structure as the detection frame D12 does not appear after the current frame n, the set of the same candidate structures including the detection frame D12 is determined as the same structure.

Since the detection frame D13 of the front frame n-1 of FIG. 18 is on the left side of the detection frame D14 of the current frame n, the structure of the detection frame D13 may be the same as the structure of the detection frame D14. The detection frame D14 is processed in the next step S337.

In addition, also in step S336, the same structure determination unit 14 may apply the deviation allowable range to the determination of the position of the structure.

In step S337, the same structure determination unit 14 allocates the remaining structures detected in the current frame n to the same candidate structure. The remaining structures are those that are not registered in the set of the same candidate structures among the structures detected in the current frame n. Details of the process of step S337 will be described later.

By the processing up to step S337, all the structures detected in the current frame n are registered in one of the sets of the same candidate structures. Further, by processing the continuous frames in steps S331 to S337, as a result, the detection position of the structure moves in the same direction as the movement direction of the landscape reflected in the continuous frames (left direction in the present embodiment). In this case, the structures shown in the frame are determined to be the same structure.

On the other hand, when the structure is not detected from the current frame n in the process of step S331, in step S332, the same structure determination unit 14 confirms the determination of the same candidate structure created before the previous frame n-1. do.

If the size of the structure (the size of the detection frame) detected from the current frame n in the process of step S331 is extremely different from the size of the same candidate structure, the structure detected in the current frame n May be excluded from the target. For example, the detection frame for detecting a structure at a distant position that is not the target is extremely smaller than the detection frame for the structure along the road to be detected.

Subsequently, the allocation process of the remaining structures in step S337 of FIG. 14 will be described. FIG. 19 is a flowchart showing the flow of allocation processing of the remaining structures.

In step S3371, the same structure determination unit 14 determines whether or not the number of remaining structures detected in the current frame n is larger than the number of sets of the same candidate structures.

When the number of remaining structures is larger than the set of the same candidate structures, in step S3372, a new set of the same candidate structures is created by a number larger than the number of the sets of the same candidate structures, and the right side of the remaining structures is created. Register in each of the new sets of the same candidate structures in order from.

For example, as shown in FIG. 20, the set of the same candidate structures is only the set including the detection frame D15 detected in the previous frame n-1, and in the current frame n, the structures of the two detection frames D16 and D17. Consider the case where an object is detected. Since the positions of the detection frames D16 and D17 are both on the left side of the detection frame D15, it cannot be determined whether the structure of the detection frame D15 is the same as any of the structures of the detection frames D16 and D17. Therefore, the same structure determination unit 14 registers the rightmost detection frame D17 in the current frame n in a new set of the same candidate structures.

In step S3373, the same structure determination unit 14 determines whether or not the number of remaining structures detected in the current frame n is less than the number of sets of the same candidate structures.

When the number of remaining structures is smaller than the number of sets of the same candidate structures, in step S3374, the set of the same candidate structures is set in order from the set of the same candidate structures on the left side by the number larger than the number of the remaining structures. Is determined as the same structure.

For example, as shown in FIG. 21, there are two sets of the same candidate structures including the detection frames D18 and D19 detected in the previous frame n-1, and in the current frame n, the structure of the detection frame D20 is present. Consider the case where only one is detected. Since the positions of the detection frames D18 and D19 are both on the right side of the detection frame D20, it cannot be determined whether the structure of the detection frame D20 is the same as any of the structures of the detection frames D18 and D19. Therefore, the same structure determination unit 14 estimates that the structure of the leftmost detection frame D18 in the front frame n-1 does not appear after the current frame n, and sets the same candidate structures including the detection frame D18 to the same structure. Confirmed as a thing.

By the above steps S3371 to S3374, the number of sets of the same candidate structure and the number of remaining structures are the same.

In step S3375, the same structure determination unit 14 allocates the remaining structures to the set of the same candidate structures in order.

By the above processing, all the structures detected in the current frame n are assigned to the same candidate structure.

If the remaining structures have the same candidate structure having the same structure characteristics before the start of the process of FIG. 19, the remaining structures may be registered in the set of the same candidate structures. .. For example, when deep learning is used, it may be determined that the structure is the same even if the position where the structure is reflected is different depending on the characteristics of the structure. For example, if the structure is the same, the abnormality detection unit 17 is considered to detect the same abnormality. Therefore, information on the abnormality of the structure detected by the abnormality detection unit 17 is stored in the storage unit 12, and the abnormality of the structure is stored. The same candidate structure of the structure may be determined by using the information regarding. Further, in the case of the same structure, the certainty of the detection frame detected by the structure detection unit 13 tends to be close to each other. Therefore, the same candidate structure of the structure is determined by using the certainty of the detection frame. May be good. When all the remaining structures are assigned to the same candidate structure in this preprocessing, the confirmation of the set of the same candidate structure is confirmed, and the process of FIG. 19 is not performed.

[Identity judgment of structure among multiple cameras]
Subsequently, the identity determination process of the structure among a plurality of cameras will be described.

The identity of the structure between multiple cameras can be determined based on the shooting time and shooting direction.

The frame cutting interval (distance) of the camera 1 is s1, and the frame cutting interval of the camera 2 is s2. A serial number (0, 1, 2, ...) Is assigned to the frame cut out from the video shot by the camera 1, and a serial number (0, 1, 2, ...) Is also given to the frame cut out from the video shot by the camera 2. Is given. Assuming that the cut-out time of the frame 0 of the cameras 1 and 2 is the same, the frame j0 of the camera 2 at the same time as the frame i of the camera 1 is obtained by j0 = i * (s1 / s2).

Further, it is assumed that the camera 1 photographs the side of the vehicle and the camera 2 photographs the diagonally front of the vehicle. Assuming that the structure is shown in the frame i of the camera 1, the structure is shown in the camera 2 before the frame j0 described above by the f2 frame. The deviation (f2) in the shooting timing of the structure between the cameras 1 and 2 is determined by the pan angle and the distance from the shooting position of the camera to the structure. The structure shown in the frame i of the camera 1 is the same as the structure shown in the frame j = i * (s1 / s2) + f2 of the camera 2.

In this way, a frame in which the same structure is captured among a plurality of cameras is uniquely defined. However, the distance between the vehicle equipped with the camera and the structure may not be constant, it may not be possible to cut out accurately due to GPS error or camera time setting error, or there may be erroneous detection or detection omission of the structure detection unit 13. Therefore, the corresponding frame between the plurality of cameras may deviate from the reference position (frame j) obtained by the calculation.

Therefore, the same structure determination unit 15 of the present embodiment determines the identity of the structure among a plurality of cameras while considering errors, false positives, and omissions in detection.

FIG. 22 is a flowchart showing the flow of the determination process of the same structure between the moving images taken by a plurality of cameras.

The same structure determination unit 15 repeats the processes of steps S41 to S43 for all the representative frames included in the still image group for each still image group cut out from the moving image taken by each camera.

In step S41, the same structure determination unit 15 extracts representative frames in other still image groups corresponding to the representative frames to be processed.

For example, it is assumed that the frame i cut out from the moving image taken by the camera 1 is a representative frame to be processed. Based on the difference in the shooting timing of the structure considering the cutting distance of the frame from the video shot by the cameras 1 to 4 and the orientation of the cameras 1 to 4, the frames j and k of the cameras 2 to 4 corresponding to the frame i of the camera 1 , L (reference position) can be uniquely obtained.

FIG. 23 shows the most ideal case where the representative frame of each still image group is the most ideal. In the example of FIG. 23, the frames j, k, and l corresponding to the frame i obtained by calculation are representative frames. In this case, the same structure determination unit 15 extracts frames j, k, and l from each of the still image groups as candidates corresponding to the frame i.

However, the calculated frames j, k, and l may not be representative frames. In the example of FIG. 24, the frame k and the frame l are not representative frames. Therefore, the same structure determination unit 15 sets the allowable range m1 to m4 of the front-back deviation from the reference position for each of the cameras 1 to 4, and sets the representative frame within the allowable range as a candidate corresponding to the frame i to be processed. To extract. In the example of FIG. 24, the representative frame k-3 exists in the allowable range m3 from the reference position for the camera 3, and the representative frame l + 2 exists in the allowable range m4 from the reference position for the camera 4. The same structure determination unit 15 extracts frames j, k-3, and l + 2 from each of the still image groups as candidates corresponding to the frame i.

In addition, there may be a plurality of representative frames within the permissible range. The same structure determination unit 15 extracts all representative frames within the permissible range as candidates corresponding to the frames to be processed. In the example of FIG. 25, the camera 3 has two representative frames, the frame k and the frame k-3, within the allowable range m3, and the camera 4 has two representative frames, the frame l and the frame l + 2, within the allowable range m4. There is a representative frame. The same structure determination unit 15 extracts frames j, k, k-3, l, and l + 2 from each of the still image groups as candidates corresponding to the frame i.

If the representative frame does not exist within the permissible range, the frame at the reference position is extracted. In the reliability calculation described later, the reliability is set to 0 for a camera whose representative frame does not exist within the allowable range.

In step S42, the same structure determination unit 15 creates all combinations for the extracted representative frames.

In the example of FIG. 23, since the frames j, k, l are extracted as candidates corresponding to the frame i, a combination of the frames i, j, k, l is created as a candidate of the same structure. In the following, the combination of frames i, j, k, l created by the same structure determination unit 15 is represented as (i, j, k, l).

In the example of FIG. 24, the combination of (i, j, k-3, l + 2) is obtained.

In the example of FIG. 25, since the frames k, k-3 for the camera 3 and the frames l, l + 2 for the camera 4 are extracted, (i, j, k-3, l), (i, j, k-3, l + 2). ), (I, j, k, l), (i, j, k, l + 2) can be obtained.

In step S43, the same structure determination unit 15 calculates the reliability for all combinations. The same structure determination unit 15 determines the reference reliability d1 to d4 for each of the cameras 1 to 4, and lowers the reliability according to the distance (number of frames) away from the reference position. Specifically, the reliability of each combination is calculated using the following equation (1).

Reliability = d1 + d2 * x ^ (v2a) + d3 * x ^ (v3a) + d4 * x ^ (v4a)
x is a coefficient smaller than 1 for lowering the reliability, and for example, x = 0.99. You may set x for each camera. v2a, v3a, v4a are the number of frames away from the reference position. Regarding the reference reliability d1 to d4, for example, d1 = 0.35, d2 = 0.4, d3 = 0.45, d4 = 0.5. If the structure to be detected is a traffic light, the upper camera can easily capture the characteristics of the traffic light, so that the reliability of the upper camera is increased. If the position of the structure can be estimated by distance measurement by LIDAR, which will be described later, or if the position information is known in advance, the reliability d1 to d4 may be corrected based on the information.

For cameras that do not have a representative frame within the permissible range, the reliability is calculated as 0. For example, in the camera 2, if the representative frame does not exist within the permissible range, it is calculated as d2 = 0. If the total value of the reliability of the cameras 1, 3 and 4 is equal to or higher than the threshold value c described later, it is possible to prevent the detection omission of the camera 2.

The above steps S41 to S43 are performed for all the representative frames of the still image group, and further, each of the still image groups is performed. That is, not only the representative frame of the camera 1 is used as a reference, but also the combination is created with the representative frames of the cameras 2 to 4 as the reference, and the reliability of the combination is calculated.

In step S44, the same structure determination unit 15 determines the same structure among a plurality of cameras in descending order of reliability from all combinations.

For example, for the combination of (i, j, k-3, l), (i, j, k-3, l + 2), (i, j, k, l), (i, j, k, l + 2). The combination of (i, j, k, l) has the highest reliability. Therefore, the structures shown in the frame i, the frame j, the frame k, and the frame l are determined as the same structure. The same structure determination unit 15 assigns an ID to the structure, and associates the frame i, the frame j, the frame k, and the frame l with the ID. When the ID of the structure is specified in the output unit 16, the frame i, the frame j, the frame k, and the frame l associated with the ID are displayed.

Frames included in a combination confirmed as the same structure are excluded from other combinations. For combinations including excluded frames, the reliability of the excluded frames is set to 0, and the reliability of the combination is recalculated. For example, the new reliability of the combination of (i, j, k-3, l + 2) is 0 + 0 + d3 * x ^ 3 + d4 * x ^ 2.

The same structure determination unit 15 determines the same structure until there are no combinations having a reliability equal to or higher than the threshold value c. For example, the threshold value c = 1.

Instead of excluding the frames included in the combination determined as the same structure, the reliability of the determined frames may be lowered. For example, when the reliability of the determined frame is 1/10, the new reliability of the combination of (i, j, k-3, l + 2) is d1 * 1/10 + d2 * 1/10 + d3 * x ^ 3 + d4 * x. It becomes ^ 2.

It should be noted that the deviation in the timing at which the structure is captured between the cameras (the number of frames f2 corrected when the reference position is obtained) may be automatically calculated. For example, when calculating the number of frames f2 of the cameras 1 and 2, the reference position of the camera 2 corresponding to the representative frame of the camera 1 is obtained while changing the number of frames f2 little by little, and the representative frame of the camera 2 is the most at the reference position. Find the number of frames f2 that increases.

[Use of LIDAR]
By using LIDAR or a simple laser rangefinder, the distance to the object can be measured accurately.

FIG. 26 shows a measurement example of LIDAR. In the graph of FIG. 26, the horizontal axis represents the mileage and the vertical axis represents the distance to the structure. When the LIDAR is placed on the vehicle and the distance to the object in the shooting direction is measured, the distance to the cylindrical structure along the road can be accurately measured as shown in FIG. 26.

When the GPS logger mounted on the vehicle is used, the position information obtained by the GPS logger is the position information of the vehicle, which is a little away from the position of the structure. By measuring the distance from the vehicle to the structure with LIDAR, more accurate position information of the structure can be obtained.

In addition, LIDAR can be used to calculate and correct a deviation in the shooting timing of the structure used in the structure identity determination process between a plurality of cameras. In the structure identity determination process between a plurality of cameras, the difference in the shooting timing of the structure between the cameras is set in advance to determine the corresponding frame (reference position). A specific description will be given with reference to FIG. 27. Assuming that a structure exists at position TL0, a camera with a pan angle of θ photographs a structure at position TL0 at position P2, and a camera with a pan angle of 90 degrees captures a structure at position TL0 at position P1. It is assumed that you will be shooting things. Based on the distance L0 between the positions P1 and P2, the deviation of the number of frames in which the structure is photographed is determined between the cameras.

However, when the structure actually exists at the position TL, the camera having the pan angle θ will take a picture of the structure at the position TL at the position P3. Originally, it is necessary to determine the deviation of the number of frames in which the structure is photographed between the cameras based on the distance L1 between the positions P1 and the position P3.

When the position from the vehicle at position P1 to the structure at position TL is accurately measured using LIDAR, the position P3 at which the structure can be photographed by a camera having a pan angle of θ and the distance L1 between the position P1 and the position P3 are obtained. It can be calculated, and the deviation of the number of frames in which the structure is photographed between the cameras can be accurately determined based on the distance L1. Each time the position of the structure can be measured, the distance from the photographing position to the structure may be calculated to estimate the frame as the reference position.

[Correction of error between GPS and recorded video]
If the time managed by the camera and the time managed by the GPS logger are different, an error will occur in the position information of the frame cut out from the captured video. In the present embodiment, the error between the GPS and the captured moving image is corrected before the start of processing.

It is assumed that the position of the structure TL in FIG. 28 is known. The structure TL is photographed with a camera installed at an angle of 90 degrees with respect to the traveling direction of the vehicle, and position information is acquired with a GPS logger.

A frame is cut out from the video shot by the camera, and the frame in which the structure TL is shown in the center is selected. In the example of FIG. 28, the frame taken at the position P2 is selected. The time unit for cutting out the frame is the time unit of the captured moving image (for example, 30 images per second for 30 fps) or the measurement unit of the GPS logger (for example, 0.2 seconds).

From the position information measured by the GPS logger, the position information closest to the structure TL whose position is known is selected. In the example of FIG. 28, the position P2 is closest to the structure TL.

Set the GPS and the time of the captured video based on the time of the selected frame and the time of the selected position information. For example, if the time of the frame is 12:00:00:400 and the time of the position information is 12:00:00: 000, the processing is performed in consideration of the time difference of 0.4 seconds.

As described above, according to the present embodiment, the still image acquisition unit 11 acquires still images (frames) from each of the plurality of captured moving images at regular intervals, and the structure detection unit 13 is the detection target. The same structure determination unit 14 for each camera detects the frame in which the structure is captured, determines the same structure between the frames for each shot video, and the same structure determination unit 15 between the cameras determines between the shot videos. The same structure is determined, and the output unit 16 displays the frames acquired from a plurality of captured moving images of the same structure side by side, so that the structure can be inspected efficiently.

Shooting with multiple cameras at different tilt and pan angles can handle tall structures and backlight. By cutting out frames at short intervals from the recorded video, it is possible to prevent omission of cutting out of the target structure. However, if you want to see the frame in which a structure is shown, you have to search from a huge number of frames. Also, if you change the pan angle, the timing at which the structure is captured will change between multiple shot videos, so the same structure may not be captured at the same time with multiple cameras, and you must search for the same structure from each. .. By using the structure detection device 1 of the present embodiment, images showing the same structure are collected, and images of the structure taken from various angles can be obtained by simply specifying the structure to be viewed on the map. Can be listed. Further, the structure detection device 1 displays the frame in which the structure is best captured side by side as a representative frame. As a result, the structure can be inspected efficiently.

According to the present embodiment, the structure detection unit 13 detects a frame in which the structure to be detected is reflected, and the same structure determination unit 14 for each camera is a structure between continuous frames n-1 and n. By determining whether or not the structure reflected in frame n-1 and frame n is the same structure based on the change in the position of the detection frame of, the same structure is determined between consecutive frames. can. Specifically, it is determined that the structure of the detection frame that moves in the same direction as the movement direction of the landscape in the frame is the same structure.

According to the present embodiment, the same structure determination unit 15 between the cameras corresponds to the representative frame i of the camera 1 based on the difference in the shooting timing of the structure considering the cutting distance of the frame and the orientations of the cameras 1 to 4. By estimating the frames j, k, l (reference position) of the cameras 2 to 4 to be used and extracting representative frames within the allowable range from the reference positions of the cameras 2 to 4, it is possible to interpose a plurality of captured images having different pan angles. Can determine the same structure with.

1 ... Structure detection device 11 ... Still image acquisition unit 12 ... Storage unit 13 ... Structure detection unit 14 ... Same structure determination unit for each camera 15 ... Same structure determination unit between cameras 16 ... Output unit 17 ... Abnormality detection Parts 101, 102 ... Recording medium

Claims (6)

It is the same structure detection device that extracts the captured images of the same structure from a plurality of captured images taken at regular intervals by multiple cameras.
The same structure in the plurality of captured image groups based on the difference in the imaging timing of the structure in consideration of the fixed distance and the imaging direction of each of the plurality of cameras or the distance from the imaging position of each of the plurality of cameras to the structure. An identical structure detection device comprising an same structure determination unit that estimates a reference image in which an object is captured and determines that the structure captured in a captured image within a predetermined range from the reference image is the same structure. The same structure detection device according to claim 1, wherein the same structure detection device is used.
The same structure determination unit is characterized in that it estimates the reference image of another photographed image group for each of the plurality of photographed image groups with reference to each of the plurality of photographed image groups. .. The same structure detection device according to claim 1 or 2.
When a plurality of structures are captured in the captured image within the predetermined range, the same structure determination unit selects the captured image in which the structure is captured from each of the plurality of captured image groups. A combination is created, and for each of the combinations, the reliability is calculated based on the degree of separation between the reference image and the captured image in which the structure is shown, and the same structure is calculated according to the high degree of reliability. The same structure detection device, characterized in that it is a combination of the photographed images in which the image is captured. The same structure detection device according to any one of claims 1 to 3.
The same structure determination unit determines the deviation of the photographing timing while changing the deviation of the photographing timing so that the photographed image in which the structure is reflected in the photographed image estimated as the reference image is the largest. Or, when the position of the structure can be estimated, the same structure detection device is characterized in that the distance from the photographing position to the structure is calculated based on the position of the structure. It is the same structure detection method that detects the target structure from a group of captured images taken at regular intervals by multiple cameras.
By computer
The same in the plurality of captured image groups based on the difference in the imaging timing of the structure in consideration of the fixed distance and the imaging direction of each of the plurality of cameras, or the distance from the imaging position of each of the plurality of cameras to the structure. Steps to estimate the reference image of the structure and
A method for detecting the same structure, which comprises a step of determining the structure appearing in a captured image within a predetermined range from the reference image as the same structure. The same structure detection program, characterized in that a computer is operated as each part of the same structure detection device according to any one of claims 1 to 4.

Images