JP7356383B2 - Image judgment device, image judgment method, and program - Google Patents

Description

The present invention relates to an apparatus that processes images taken from a moving viewpoint and makes decisions in real time.

Conventionally, there has been a technology in which a camera is installed in a car to take images such as videos or continuous still images, and the captured image data is analyzed using AI using neural networks to determine various malfunctions. Ta.

For example, Patent Document 1 discloses a device that uses AI to detect cracks in road pavement, calculates the degree of damage, and draws the calculation results on a map.

Patent No. 6516384

By installing a camera on a moving car and having the device perform judgment processing in real time, objects can be detected while the car is moving, and the person driving the car can be notified on the spot. , there are demands to encourage response.

Specifically, for example, we use cars to patrol telephone poles to check for bird nests.If a nest is discovered, a serious disaster such as an electrical leak can occur immediately. Because of the danger of nesting, there is a demand for patrollers to stop their vehicles and remove nests on the spot.

For example, we use cars to patrol road pavements to check for potholes, and if a pothole is found, a serious accident such as a fall of a passerby can occur immediately. Because of the danger of this occurring, there is a demand for patrollers to stop their vehicles and fill in the holes on the spot.

In addition, while patrols may involve immediate response work on the spot, if the patroller determines that the risk of a serious disaster occurring immediately is not relatively high, There is also a task of bringing back the results discovered during patrols, compiling the results discovered after the fact, and requesting a specialized department to deal with construction etc., and for use in this purpose. There is a demand to collect and confirm the results after the test is taken home. Further, it is desirable that the results be confirmed within as short a time as possible after the end of the patrol.

Conventionally, technology for analyzing photographed image data using AI has had a problem in that analysis and notification cannot be performed in real time.

One of the reasons why analysis cannot be performed in real time is due to the equipment configuration, in which the captured image data is taken home as a recording medium, and is input into the analysis equipment along with the media containing the GPS location information for analysis. Ta. For this reason, there is a separation between the business stage of driving the car and photographing it, and the business stage of performing analysis using the device after driving, making it impossible to perform analysis in real time.

Another reason why analysis cannot be performed in real time is that the neural network used internally by the analyzer requires a large amount of calculation, and the processing speed cannot keep up with the large number of input images. Analysis could not be performed in real time.

In addition, conventional technology that uses AI to analyze captured image data requires a large amount of calculations from the time the data is brought back and analysis is started until the analysis is completed, which takes time, resulting in the end of the patrol. There was a problem that the results could not be confirmed within a short period of time.

The present invention has been developed in consideration of the above-mentioned problems, and an object of the present invention is to use an image taken from a moving viewpoint as input, judge the object in real time, notify the user, and The purpose of the present invention is to provide a technology for quickly displaying a list of results.

An image determination device according to one aspect of the present invention includes a detection unit that inputs images in chronological order taken from a moving viewpoint and detects a target object from the images, and an image of the images in the chronological order and the image of the target object. an image storage unit that temporarily stores the detected position, an object selection unit that selects the same object only once from the images in the chronological order, and a notification or detection result regarding the selected object. The image in the chronological order is an image taken horizontally or diagonally in front of the traveling direction, and the target selection unit is configured to store the images in the chronological order. The identity of the object is determined based on the moving direction of the detected position of the object in the image, and the storage contents of the image storage unit are determined for the object whose detected position moves in a specific direction. The object is selected from among the images in which the detected position of the object is located closest to the center of the image.

An image determination method according to one aspect of the present invention is an image determination method executed by a computer, and includes the steps of inputting images taken in chronological order from a moving viewpoint, detecting a target object from the images, and storing images. a step of temporarily holding the images in the chronological order and the detected position of the object on the image; a step of selecting the same object only once from the images in the chronological order; the selected target object, and the chronologically ordered images are images taken horizontally or diagonally in front of the traveling direction; In the step of determining the identity of the object based on the moving direction of the detected position on the image of the object in the chronological order images, for the object whose detected position moves in a specific direction. , the object is selected from the image in which the detected position of the object is located closest to the center of the image among the contents stored in the image storage section.

A program according to one aspect of the present invention includes a process of inputting chronologically ordered images taken from a moving viewpoint into a computer, detecting a target object from the images, and storing the chronologically ordered images and the A process of temporarily retaining the detected position of the object on the image, a process of selecting the same object only once from the images in the chronological order, and a notification or detection result about the selected object. The images in chronological order are images taken horizontally or diagonally in front of the traveling direction, and in the selecting process, the images in chronological order are The identity of the object is determined based on the moving direction of the detected position of the object on the image of the object, and for the object whose detected position moves in a specific direction, among the contents stored in the image storage unit, The object is selected from the image in which the detected position of the object is located closest to the center of the image.

According to the present invention, it is possible to provide a device that receives an image taken from a moving viewpoint as input, determines an object in real time, notifies the user, and promptly displays a list of the determined results. I can do it.

FIG. 1 is a configuration diagram showing the configuration of a real-time image determination device. This is an example of an image used for learning the first parameter used by the first detection unit. This is an example of an image used for learning the second parameter used by the second detection unit. 3 is a flowchart showing the operation of the real-time image determination device. It is a flowchart which shows operation of an object selection part. This is an example of an image taken by a camera. It is a figure showing the result of the 1st detection processing. It is a figure which shows the result of image extraction processing. It is a figure which shows the result of a frame drawing process. It is a figure which shows the example of a notification. It is a figure which shows the example of a result display screen. FIG. 2 is a configuration diagram showing the configuration of a second real-time image determination device.

One embodiment of the present invention is shown below.

<Configuration of real-time image judgment device>
FIG. 1 shows the configuration of a real-time image determination device in this embodiment.

The real-time image determination device 100 includes an image reduction section 112, a first detection section 113, a target selection section 114, an image extraction section 115, a second detection section 116, a frame drawing section 117, a notification section 118, a result storage section 119, and , a result display section 120.

The camera 111 is attached to a running car and is installed to take pictures of the front. The real-time image determination device 100 receives an image captured by the camera 111 as input.

Further, a GPS device 101 is prepared, and recorded GPS logs are input to the real-time image determination device 100. The GPS log is information that describes latitude and longitude at each time. GPS devices that record such logs are commercially available, or a drive recorder with a built-in GPS may be used. Alternatively, the real-time image determination device 100 may include a GPS positioning function.

The image reduction unit 112 reduces the amount of information by reducing the input image.

The first detection unit 113 is configured by providing a first parameter 132 obtained as a result of pre-learning to a neural network 131, and detects a target object from an image whose information amount has been reduced by the image reduction unit 112.

The neural network 131 uses an object detection neural network. An object detection neural network uses deep learning technology to detect the presence of one or more pre-trained objects in a given input image, and to detect the presence of a rectangle surrounding that presence. This is a method that allows you to obtain location coordinates. R-CNN, Faster R-CNN, Single Shot MultiBox Detector, etc. have been published as such methods, and implementations are also available as open sources.

In this embodiment, at the manufacturing stage of the real-time image determination device 100, various images related to the detection target are input to the learning function of the neural network 131 to perform learning, and the learning result data obtained as a result of the learning is is incorporated into the real-time image determination device 100 as the first parameter 132 and used.

FIG. 2 is an example of an image used for learning the first parameter 132 used by the first detection unit 113.

Image 200 is an example of an image prepared for learning, but during learning, an image that looks similar to that captured by camera 111 is prepared, like image 200, and learning is performed based on this. The image 200 shows a white line 201 on the left, a white line 202 on the right, and a lane 203 in which a vehicle travels. A utility pole 210 stands on the left side of lane 203, and multiple electric wires are strung up on it. Since we want the program to learn to find the telephone pole 210 in particular in the image 200, a rectangle 211 surrounding the telephone pole 210 is annotated (tagged) and the annotated image 200 is used for learning. Similarly, by annotating and learning images of various utility poles, the first detection unit 113 functions as a neural network that detects utility pole parts from images. 100 and use it.

The target selection unit 114 selects the image and target object to be processed by the second detection unit 116 at the subsequent stage, based on the detected position on the image or the size of the target object on the image.

The image extraction unit 115 obtains a partial image of the object selected by the object selection unit 114 from the original image whose information amount has not been reduced by the image reduction unit 112.

The second detection unit 116 is configured by giving a second parameter 142 obtained as a result of pre-learning to a neural network 141, inputs a partial image in which only the target object is extracted, and classifies the target object.

The neural network 141 uses an image judgment neural network. An image determination neural network is a method that uses deep learning technology to determine which class a given input image is closest to among a plurality of pre-trained image classes. As such methods, resnet, vgg16, etc. have been published, and implementations are also available as open sources.

In this embodiment, at the manufacturing stage of the real-time image determination device 100, various images related to the detection target are input to the learning function of the neural network 141 to perform learning, and the learning result data obtained as a result of the learning is is incorporated into the real-time image determination apparatus 100 as the second parameter 142 and used.

FIG. 3 is an example of an image used for learning the second parameter used by the second detection unit.

Learning is performed by dividing into two classes: group A, which indicates successful detection, and group B, which indicates failed detection. Image 300 is an example of Group A, and has the feature that the entire image is a rectangular area surrounding a utility pole 301, and that a nest 302 (bird's nest) has been formed on the utility pole. On the other hand, image 310 is an example of group B, and the entire image is a rectangular area surrounding utility pole 311, and there is no nest (bird's nest) on the utility pole.

Similarly, by classifying images of various utility poles into group A or group B for learning, the second detection unit 116 is able to nest when inputted with a partial image of only the utility pole portion of the camera-captured image. Since it functions as a neural network that determines whether something is present (group A) or not (group B), this is incorporated into the real-time image determination device 100 and utilized.

The frame drawing unit 117 generates an image in which the object classified by the second detection unit is surrounded by a frame. For example, the frame drawing unit 117 generates an image in which a telephone pole containing a nest is surrounded by a frame.

The notification unit 118 has a function of notifying the driver of the vehicle of the determination result in real time, and performs notification by sound, vibration, and screen display.

The result storage unit 119 stores the image of the frame drawing result and the time at which it was processed in a storage medium. The result storage unit 119 may add position information of the object to the frame drawing result and store the result.

The result display unit 120 has a function of displaying the determination results on a map, so that the user can check the series of determination results later after the user has finished driving the car.

<Operation of real-time image judgment device>
The real-time image determination device 100 receives data of an image photographed by the camera 111 as input, and performs the operations described below.

FIG. 4 is a flowchart showing the operation of the real-time image determination device 100. FIG. 5 is a flowchart showing the operation of the target selection unit 114. In the following, an example will be described in which the target object is a utility pole and a utility pole in which a nest exists is determined. Note that the object is not limited to utility poles, nor is it limited to determining whether or not a nest exists.

The camera 111 captures an image at that time, and the real-time image determination device 100 receives this image as input (step S1). Note that the camera 111 and the real-time image determination device 100 may be installed adjacent to each other, or may be installed at a remote location via a wireless network.

FIG. 6 is an example of an image taken by the camera 111. The image 600 was taken while the vehicle was driving on a two-lane road (one lane on each side) with a center line, and a utility pole 610 is visible. A nest 612 exists on the utility pole 610.

The image reduction unit 112 resizes the input image to reduce the amount of information by reducing the size of the image (step S2).

The first detection unit 113 performs utility pole detection processing using the reduced image as input, and if detected, obtains the coordinates of a rectangle indicating the position of the utility pole (step S3). Note that the object detection neural network can detect the presence of one or more objects that have been trained in advance, so if a plurality of telephone poles are detected, a rectangular coordinate arrangement is obtained.

FIG. 7 is a diagram showing the results of the first detection process. A telephone pole 610 is detected in the image 700 obtained as a result of the image reduction process, and a rectangle 711 is obtained.

The object selection unit 114 determines the identity of the object from the coordinates of the obtained rectangle according to the procedure of the flowchart in FIG. 5, and selects only the object to be processed in the subsequent stage.

That is, the target selection unit 114 selects only rectangles whose X coordinates are wholly or partially within a certain range of the entire image from the obtained rectangular coordinate arrangement (step S41). If the list is an empty set, there is no selection target and the subroutine ends (step S42).

The above-mentioned certain range may be a constant value predetermined according to the angle of view (the composition of the photograph taken by the camera), and in particular, the point P where the object first appears as the car moves, and It may also be a fixed range corresponding to a section consisting of a point Q where the object is finally outside the photographing range. In the example of the image 700, the first range indicated by the X coordinate corresponding to the section from point P to point Q is set as a fixed range.

Next, the target selection unit 114 determines that, for the coordinates of the rightmost rectangle among the obtained rectangular coordinates, the X coordinates are wholly or partially within a second fixed range of the entire image. If so, the "processed" flag is cleared to 0, and the subroutine is ended as there is no selection target (step S43).

The second certain range is a part of the certain range, and in particular, a certain range set at or near the point P where the object first appears as the car travels. For example, for a point R that divides the point P and the point Q at a ratio of 1:2, it may be a fixed range that corresponds to the section from point P to point R. It's okay. In the example of the image 700, the second range indicated by the X coordinate corresponding to the section from point P to point R is set as the second fixed range.

Next, if the "processed" flag is 1 for the coordinates of the rightmost rectangle among the array of coordinates of the obtained rectangles, the target selection unit 114 determines that there is no selection target and executes a subroutine. The process ends (step S44).

Next, the target selection unit 114 sets the "processed" flag to 1, sets the coordinates of the rightmost rectangle among the obtained rectangular coordinates as the selection target, and ends the subroutine ( Step S45).

Note that the “processed” flag used by the target selection unit 114 is cleared to 0 when the real-time image determination device 100 is started.

Here, the meaning of the above series of steps in the target selection process will be described. In the target selection process, when images taken from a moving car are continuously and repeatedly processed in real time, the notification that a target has been found is sent to the location where the camera can capture the clearest image, i.e. near the center of the effective shooting range. Since it is desired that the notification is performed when an object arrives at , and the notification is to be made only once for the discovery of the same object, the request described on the left can be realized by steps S41 to S45.

In steps S41 and S42, the only utility poles to be detected are those appearing on the left side of the road, and those that are initially seen near the vanishing point in the distance in the photographed image based on perspective gradually approach. It is intended to be used for processing objects that become large images.

In step S43, the subsequent processing is intended to be performed only when the object comes near the center of the image. Near the center means that the image is roughly divided into three parts: left, center, and right, and the captured images can be processed continuously in real time by processing only once when the object crosses the right section and enters the center section. A certain amount of calculation time is required for each series of processing that occurs when processing repeatedly, and due to this, it is not possible to process the camera image taken immediately after the object approaches the center section. Even if a camera image taken after a certain amount of time has elapsed since the object entered the center section is to be processed, the camera image taken while the object was located somewhere within the center section must be processed. Therefore, it is possible to realize the above-mentioned intention of performing subsequent processing only when the image reaches the vicinity of the center of the image.

In step S44, the subsequent processing is intended to be performed only once for one object. As the car moves, objects on the left side of the road move further to the left in the image, so while they are in the first range in Figure 7, they only need to be processed once, and the "processed" flag allows them to be processed a second time. This is achieved by not doing the following. On the other hand, if a new object appears in the image, it will appear at a position close to the vanishing point in the angle of view, that is, in the second range in Fig. 7, so by clearing the "processed" flag to 0, the corresponding The next time the object falls within the range of the central section, it can be targeted for subsequent processing. Note that subsequent processing may be performed based on the size of the target object on the image. Specifically, after the "processed" flag is cleared to 0, when the size of the object on the image becomes larger than a predetermined size, the object is selected and the "processed" flag is cleared to 0. ” flag to 1.

In this way, the target selection process has the effect that notification can be performed only once in the most appropriate manner, and furthermore, the neural network in the second detection unit 116, which requires processing time due to the large size of the image to be processed, can be obtained. Since the number of times of network processing can be kept low, the effect of reducing the overall processing time can also be obtained.

In the neural network processing in the first detection unit 113, the image size is reduced by the image reduction unit 112 in advance, so the processing time per time is significantly reduced compared to the processing in the second detection unit 116. It's short enough. In this way, the process from step S1 to step S4 is repeated many times, so it takes a short processing time, and the process after that is executed only a relatively small number of times, so it takes a relatively long time. By doing so, it becomes possible to increase the total number of times of processing for the entire apparatus, and the purpose of this apparatus, which is image determination in real time, can be fulfilled.

On the other hand, in the present embodiment, utility poles are objects that are easy to distinguish, and even if the image size is reduced to a smaller size by the image reduction unit 112, it is possible to identify the poles with sufficiently high accuracy. Also from this point of view, this embodiment can sufficiently achieve the purpose of the present device.

Although the present embodiment includes the image reduction section 112, the scope of the present invention is not limited to this, and the first detection section 113 does not include the image reduction section 112, and the first detection section 113 It may be configured to process it as is.

The image extraction unit 115 extracts pixels at positions corresponding to the coordinates of the rectangle selected in the target selection process from the image input in step S1 to obtain a partial image (step S5).

FIG. 8 is a diagram showing the results of the image extraction process. Image 800 is a partial image obtained by cutting out an equivalent rectangular range from the original image 600 before resizing to the range of rectangle 711 in image 700 that has been resized to reduce the image size in step S2.

The second detection unit 116 performs a nest detection process using the obtained partial image as input, and determines the presence or absence of a nest (step S6).

For example, in the case of image 800, since a nest exists, the second detection unit 116 determines that the image is of the group A class, that is, a nest exists, according to the content learned in advance.

When the real-time image determination device 100 determines that nesting exists, the real-time image determination device 100 performs the following processing.

The frame drawing unit 117 generates an image in which the determined nesting range is surrounded by a frame (step S7).

FIG. 9 is a diagram showing the results of the frame drawing process.

The image 600 before resizing is used to generate the image 900. For example, for the original image 600 before resizing, a rectangle equivalent to the range of the rectangle 711 in the image 700 resized and reduced in size in step S2 is drawn with a thick colored frame line, and the image 900 is created. obtain. A symbol 911 indicates the drawn frame.

The notification unit 118 notifies the driver of the vehicle of the determined result using sound, vibration, and screen display (step S8), and the result storage unit 119 stores the image of the frame drawing result and the time at the time of processing. is stored in the storage medium (step S9).

FIG. 10 is a diagram showing an example of notification. A partial image 1010 of the object and a message 1020 indicating that the object has been found are displayed on the screen 1000. The notification screen may be a screen in which a message is superimposed on the image 900 of the frame drawing result.

The notification unit 118 notifies the driver of the vehicle using sound and vibration, so that the driver can safely know that an object has been discovered even while driving the vehicle on patrol duty. You have the advantage of being able to. In addition, by notifying the driver of the vehicle through a screen display, the driver can use the screen display to inform the driver of which utility poles and how the nests were located after stopping the vehicle for patrol duties. Since it can be confirmed above, there is an advantage that subsequent removal work can be performed easily.

The real-time image determination device 100 returns to step S1 and repeats the process described above.

Furthermore, the result display unit 120 reads the GPS log recorded by the GPS device 101 using an SD memory card, USB connection, network communication, or other means, and displays the latitude and longitude for each time recorded in the log. Using the information described, the correspondence between the image and the position is determined by comparing the time stored in the result storage unit 119 at the time when the object was determined, and the latitude and longitude where the object is located are obtained. , display the judgment results on the map. Note that the above processing procedure performed by the result display unit 120 of the real-time image determination device 100 does not involve complicated processing using a neural network that requires a large amount of calculation, so the results can be displayed quickly.

FIG. 11 is a diagram showing an example of a result display screen. The result display screen 1100 includes a map display area 1110, a determination image display area 1120, and a panoramic image display area 1130. In the map display area 1110, a plurality of locations where objects have been determined along the travel route of the car are displayed as pins 1111A to 1111C on the map. The display mode of the pins 1111A to 1111C may be changed depending on the determination result of the target object. Furthermore, when the user indicates any of the pins 1111A to 1111C by operating a mouse, tablet, touch panel, etc., the image of the frame drawing result stored in the result display unit 120 corresponding to the pin 1111A to 1111C is displayed in the judgment image display area. 1120 and a panoramic image display area 1130. In the determination image display area 1120, a partial image obtained by cutting out the object is displayed. A camera-captured image is displayed in the panoramic image display area 1130.

In this embodiment, the real-time image determination device 100 includes a notification unit 118. The scope of the present invention is not limited to this, and part or all of the functions of the notification unit 118 may be provided as a separate device. Further, at this time, the processing results of the real-time image determination device 100 may be input to the notification device via the network. For example, only an audio notification may be provided at the shooting site, the information may be transmitted to a remote inspection center via a network, and the image may be displayed on the notification device installed at the inspection center.

In this embodiment, the real-time image determination device 100 includes a result display section 120. The scope of the present invention is not limited to this, and the result display section 120 may be a separate device. Further, at this time, the data stored in the result storage unit 119 of the real-time image determination device 100 may be input to the display device via a network or a recording medium.

In the present embodiment, the real-time image determination device 100 performs the process of determining the correspondence between the latitude and longitude information measured by the GPS device 101 and the image captured by the camera not at the time of capturing but after the image is captured on the result display screen. 120 to display the results. The scope of the present invention is not limited to this, and a GPS positioning function may be provided in the real-time image determination device 100 and performed at the time of photographing. Further, at this time, the result storage unit 119 may store at least the image and the latitude and longitude information. Further, the notification unit 118 may display not only the image of the frame drawing result but also a map of the point on the screen.

In the above series of explanations, the GPS device 101 and the real-time image determination device 100 handle latitude and longitude information, but other formats of information indicating geographic locations may be used.

In this embodiment, the real-time image determination device 100 repeats the processing from step S1 to step S9 without interruption, but may include a pause in the processing as appropriate. For example, a GPS positioning function may be provided in the real-time image determination device 100, and the processing may be suspended until the vehicle has traveled a certain distance (for example, 50 cm).

In this embodiment, the neural network 141 uses an image judgment neural network. The scope of the present invention is not limited to this, and other neural networks that implement the necessary functions may be used, for example, an object detection neural network may be used.

In that case, for learning to obtain the second parameter 142, only images in which nests are present may be used, without using images in which nests do not exist (corresponding to group B). Furthermore, instead of an image that only extracts the rectangular area surrounding the utility pole, we annotate only the nesting position in the image by using an image of the angle of view taken with a camera, and prepare a large number of similar annotated images for learning. The second parameter 142 may be obtained by doing so. Furthermore, step S5 in which the image extraction unit 115 obtains a partial image may be omitted. Further, in step S7 in which the frame drawing unit 117 draws the frame, the frame may be drawn in the rectangular range in which the neural network 141 has detected the object. Further, if both the neural network 131 and the neural network 141 are object detection neural networks, the computer program that implements the neural networks uses the same subroutine and provides the first parameter 132 to configure the first detection unit 113. , the second parameter 142 may be given to configure the second detection unit 116.

In this embodiment, the neural network 131 uses an object detection neural network. The scope of the present invention is not limited to this, and other neural networks that implement the necessary functions may be used.

In the present embodiment, the camera 111 is installed so as to photograph the front of a traveling car, and the real-time image determination device 100 is operated by inputting an image having such an angle of view. In addition, only utility poles on the left side of the road in the image were targeted for determination. The scope of the present invention is not limited to this, but for example, the camera may be placed directly to the side, diagonally in front of you, or diagonally in front and slightly upward, and the zoom magnification may be set slightly higher than the standard to telephoto. The operation may be performed using an image taken at such an angle of view as input. In that case, for example, the camera is rotated 90 degrees to the right and installed to take a portrait image, the camera is tilted 30 degrees to the left and 30 degrees upward, and the zoom magnification is set to 1. It may be increased by 5 times to 2.5 times. When the angle of view is changed, the range used in step S42 and step S43 may be changed according to the angle of view. The range may also be set based on the point Q at which the object finally falls outside the photographing range. In addition, especially when shooting at an angle of view where there is no vanishing point in perspective, point P may become the edge of the image rather than the vanishing point, so as a result of the above settings, Point P may be the right edge of the screen, and similarly, point Q may be the left edge of the screen. At this time, if point P is the right edge of the screen and point Q is the left edge of the screen, all candidates will always be selected in step S41, which is redundant, so steps S41 and S42 are omitted. It's okay. Furthermore, if the camera is installed so as to rotate the camera 90 degrees to the right and capture a vertically elongated image, the real-time image determination device 100 inputs an image with such a view angle, and in step S1 rotates the input image by 90 degrees. The subsequent processing may be performed after rotation. Furthermore, objects on the right side of the road may be processed, or the camera may be installed to photograph the rear of the vehicle. The present invention may be implemented by reading "right" in reverse. In addition, real-time image judgment can process left and right objects at once by providing some or all of the functional units included in the real-time image judgment device 100 in a multiplex manner, or by performing multiple processes through time multiplexing. It may also be implemented as a device.

In this embodiment, the target selection process executed by the target selection unit 114 of the real-time image determination device 100 is realized by steps S41 to S45 described above. The scope of the present invention is not limited to this, and the real-time image determination device may be configured using other processing procedures that achieve the same purpose.

For example, the target selection unit 114 includes a temporary storage area that can store a plurality of images and coordinate position information, and selects the coordinates of the rightmost rectangle among the coordinates of the rectangle selected in step S41. and the corresponding image in the temporary storage area, and determine from the storage contents of the temporary storage area that the coordinates change from right to left by executing the target selection process multiple times, and store the image in the temporary storage area. The subroutine may be configured to terminate the subroutine by selecting the image whose coordinates are most centrally located among the contents as the selection target in the target selection process. When such a configuration is adopted, since a series of images are stored in the temporary storage area and the images are selected later as targets, there is an advantage that the image with the most appropriate coordinate position can be selected. On the other hand, due to the mechanism of memorizing and selecting later, there is a problem that subsequent processing is delayed for a moment, and the timing of notification to the user is also delayed for a moment. Since the notification delay is considered to be on the order of seconds, it can be considered to be sufficiently acceptable in light of the purpose of this device.

In this embodiment, the real-time image determination device 100 first performs the following processing in step S1 by inputting the image that the camera is shooting at that time, performs a series of processing up to step S9, and then repeats the processing. Returning to step S1, the process is again performed using the image taken by the camera at that time as input. The present embodiment is not limited to this, and when performing step S1 again, the image that was taken between the time when step S1 is first performed and the time when step S1 is performed again is It may be input instead of the image taken by the camera.

For example, the real-time image determination device 100 includes a ring buffer of a fixed size for holding images, and continuously stores the image taken by the camera 111 at the time in the ring buffer together with information about the time at which the image was taken. When the real-time image determination device 100 first performs step S1, it acquires the latest image held by the ring buffer, uses it as input, and performs subsequent processing. Also, the position of the ring buffer from which the image was acquired at this time is stored. Furthermore, when the real-time image determination apparatus 100 performs a series of processes up to step S9 and then returns to step S1 to perform the process, the real-time image determination apparatus 100 uses the stored position of the ring buffer at the time of the previous process and its position. Compare the latest ring buffer positions at the moment. When the positions of both are within a certain value, that is, when less than a certain amount of time has passed since the previous image input process, the latest image held in the ring buffer is acquired in step S1 and input. and perform the following processing. On the other hand, when the positions of both are above a certain value, that is, when more than a certain amount of time has passed since the previous image input process, in step S1, the previous image is selected from among the information held by the ring buffer. An image held in a ring buffer at a position that is forward by the predetermined value from the position at the time of input processing is acquired, used as input, and subsequent processing is performed.

By performing the above series of steps, if the series of processes from Steps S1 to S9 accidentally takes a long processing time, there will be a time interval of more than a certain amount between the previous input and the current input. Therefore, there is an advantage that it is possible to prevent the problem of a target object passing by and missing detection in an image taken from a moving vehicle or the like.

<Second real-time image determination device>
By partially changing or simplifying the configuration of the device described above, it is also possible to configure a second real-time image determining device that determines other objects. Another embodiment of the present invention will be shown below.

FIG. 12 shows the configuration of the second real-time image determination device.

The second real-time image determination device 1200 differs from the real-time image determination device 100 in that it does not have the image extraction unit 115 and does not have the second detection unit 116. For a configuration that does not include the image extraction unit 115 and the second detection unit 116, the second real-time image determination device 1200 may be configured by replacing the above description with the assumption that the second detection unit 116 always succeeds in detection. .

Furthermore, in the present embodiment, the second real-time image determination device 1200 uses a learning data group annotated with potholes on the road when learning the first parameters 132 constituting the first detection unit 113. This allows the first detection unit 113 to function as a neural network for detecting potholes from images, so this is incorporated into the second real-time image determination device 1200 and utilized.

The operation of the second real-time image determination device 1200 will be briefly described below with reference to FIG. The second real-time image determination device 1200 inputs an image taken of a road with the camera 111 facing the direction of travel (step S1), and the image reduction unit 112 reduces the amount of information by reducing the input image. (Step S2), the first detection unit 113 detects potholes from the image (Step S3).

The target selection unit 114 selects a pothole detected at an appropriate position as a target for subsequent processing (step S4). Specifically, in steps S42 and S43 of FIG. 5, the fixed range and the second fixed range are specified using a Y coordinate range instead of an X coordinate. In an image taken of the road with the camera 111 facing the direction of travel, the pothole moves from far away to nearer side. In other words, the pothole moves from top to bottom on the image. The potholes at the bottom of the image are close to the driving position and appear larger. A fixed range is specified using the Y coordinate, and a range above the fixed range is specified as a second fixed range. After the first detection unit 113 detects a pothole in the second fixed range, when the detected pothole leaves the second fixed range and moves downward, the pothole is selected as a target for subsequent processing. .

Since the second real-time image determination device 1200 does not have the image extraction unit 115 and the second detection unit 116, when it is determined that there is a selection target in step S4, the second real-time image determination device 1200 always Assuming that the detection is successful, frame drawing processing (step S7), result notification processing (step S8), and result storage processing (step S9) are executed.

The intention of the operation of the second real-time image determination device 1200 described above will be described below. When patrolling roads, potholes can cause serious accidents such as bicycles falling over, so potholes are discovered on the spot so that patrollers can take appropriate measures, and potholes can be detected on the spot so that patrollers can take appropriate measures. There is a request to compile a report including potholes at a later date. The second real-time image determination device 1200 can be installed in such a patrol vehicle to present appropriate information to the patroller. When you are driving on a road, if there is a pothole, it will initially appear in the distance, and as you drive, it will gradually appear closer to you, so if you express it in terms of image coordinates, the Y coordinate will move from the top to the bottom. I can express it. Potholes that are so large and damaged as to be dangerous to citizens are also greatly damaged in appearance, and can be detected with sufficient accuracy using a neural network even in a resized image that has been reduced in size by the image reduction unit 112. I can do it. Further, since there is no special request to classify potholes by type, and there is no special request to receive notification only in the case of a special type of pothole, the second detection unit 116 detects potholes. There is no need to determine which class it belongs to in more detail, and therefore the second real-time image determination device 1200 can omit the second detection unit 116 and the image extraction unit 115, which is a functional unit that generates its input. .

In the above, the real-time image determining device 100 is an implementation of a device that determines nesting of utility poles, and the second real-time image determining device 1200 is an implementation of a device that determines potholes on road pavement. The scope of the present invention is not limited thereto, and can be used in various applications in which images taken from moving viewpoints are continuously determined in real time and notifications are made.

For example, the real-time image determination device 100 detects, as a target object, a power line, a communication line, a traffic signal, a utility pole on which at least one of a traffic sign is installed, or a structure around a road similar to a utility pole, and detects the surface of the structure. It can detect cracks in concrete, peeling of concrete, exposed reinforcing bars, rust on metal, and adhesion of floating objects (ropes, kites, etc.). Further, the second real-time image determination device 1200 can determine cracks in road pavement, fallen objects, rocks (falling rocks, etc.), earth and sand (landslides, etc.), and animals (including carcasses, etc.). Further, the camera may not only be mounted on a passenger car, but may also be mounted on a bicycle, a motorcycle, a vehicle running on a fixed track (such as a train), a pedestrian, or other moving objects to take pictures. Furthermore, without being limited to these, the device of the present invention can be used for various purposes by appropriately training the neural network.

The apparatus of the present invention can also be realized by a computer and a program, and the program can be recorded on a recording medium or provided through a network. Specifically, the real-time image determination apparatuses 100 and 1200 described above include, for example, a CPU (Central Processing Unit, processor), a memory, a storage (HDD: Hard Disk Drive, SSD: Solid State Drive), A general-purpose computer system can be used that includes a communication device, an input device, and an output device. In this computer system, each function of the real-time image determining apparatuses 100 and 1200 is realized by the CPU executing a program for the real-time image determining apparatus 100 loaded onto the memory. Furthermore, the program for the real-time image judgment apparatus 100, 1200 can be stored in a computer-readable recording medium such as an HDD, SSD, USB memory, CD-ROM, DVD-ROM, or MO, or can be distributed via a network. You can also.

Further, the present invention is not limited to the above-described embodiments, and many modifications can be made within the scope of the invention.

100, 1200...Real-time image determination device 101...GPS device 111...Camera 112...Image reduction section 113...First detection section 114...Target selection section 115...Image extraction section 116...Second detection section 117...Frame drawing section 118... Notification section 119...Result storage section 120...Result display section 131...Neural network 132...First parameter 141...Neural network 142...Second parameter

Claims (6)

a detection unit that inputs images taken in chronological order from a moving viewpoint and detects a target object from the images;
an image storage unit that temporarily stores the chronologically ordered images and the detected position of the object on the image;
an object selection unit that selects the same object only once from the images in the chronological order;
a processing unit configured to perform at least either notification or storage of detection results regarding the selected target object;
The images in chronological order are images taken horizontally or diagonally in front of the traveling direction,
The object selection unit determines the identity of the object based on the moving direction of the detected position of the object on the image in the chronological order, and selects the object whose detected position moves in a specific direction. An image determination device for selecting an object from among the images in which the detected position of the object is located closest to the center of the image among the contents stored in the image storage unit. The image determination device according to claim 1,
The image determination device is characterized in that the target selection unit selects the target object from images in which the detected position on the image or the size of the target object on the image is appropriate. The image determination device according to claim 1 or 2,
When the interval between the previously processed image and the latest image input for the next processing is long, the detection unit stores the image input between the previously processed image and the latest image in the image storage unit. An image determination device characterized in that an image is acquired from. The image determination device according to any one of claims 1 to 3,
The image determination device is characterized in that the object is a defect on the road including at least one of potholes in the pavement, cracks in the pavement, fallen objects, rocks, earth and sand, and animals on the road. An image determination method executed by a computer, the method comprising:
inputting images taken in chronological order from a moving viewpoint and detecting an object from the images;
temporarily retaining the chronologically ordered images and the detected position of the object on the image in an image storage unit;
selecting the same object only once from the chronologically ordered images;
carrying out at least one of notification and storage of detection results regarding the selected object;
The images in chronological order are images taken horizontally or diagonally in front of the traveling direction,
In the selecting step, the identity of the object is determined based on the moving direction of the detected position of the object on the image in the chronological order, and the object whose detected position moves in a specific direction An image determination method comprising: selecting an object from an image in which a detected position of the object is located closest to the center of the image among the contents stored in the image storage unit. to the computer,
A process of inputting images taken in chronological order from a moving viewpoint and detecting a target object from the images;
a process of temporarily retaining the chronologically ordered images and the detected position of the object on the image in an image storage unit;
a process of selecting the same object only once from the images in the chronological order;
causing a process to execute at least one of notification and storage of detection results regarding the selected target object;
The images in chronological order are images taken horizontally or diagonally in front of the traveling direction,
In the selection process, the identity of the object is determined based on the moving direction of the detected position of the object on the image in the chronological order, and the object whose detected position moves in a specific direction is determined. A program for selecting an object from among the images in which the detected position of the object is located closest to the center of the image among the contents stored in the image storage unit.

Images