JP7014000B2 - Image processing programs, equipment, and methods - Google Patents

Description

The present invention relates to an image processing program, an image processing apparatus, and an image processing method.

Conventionally, there is a license plate recognition system that identifies a vehicle traveling on a road by reading the license plate of the vehicle included in the image from an image captured by a camera installed on the road. For example, in a police investigation or the like, by using this license plate recognition system and searching for an image using the license plate information as a search key, it is possible to know the traffic status of a specific vehicle.

However, for example, in the sighting information of a crime scene, the license plate information is often incomplete, so a specific vehicle may be narrowed down based on the shape of the vehicle or the like. Therefore, there is a need for a vehicle type discrimination method that searches for an image including a vehicle of a specified vehicle type from an image captured by a camera installed on the road.

In addition, the appearance of the vehicle in the image captured by the camera varies depending on the installation position and angle of the camera, the width of the lane, and the like. Therefore, in order to discriminate the vehicle type, the difference in appearance is determined by using a feature amount that does not depend on the difference in the appearance of the vehicle in the image, or by converting it so that it looks the same in advance. A process to absorb is required.

As an example of a method that uses features that do not depend on the difference in appearance, there is a method that uses machine learning such as deep learning. In this method, by learning various data with different appearances, it is possible to select features that do not depend on the appearance.

Further, as an example of a method of converting so as to have the same appearance in advance, there is a method of performing matching after geometrically converting a template or a target image to match the appearance when performing pattern matching.

Japanese Unexamined Patent Publication No. 5-314390 Japanese Unexamined Patent Publication No. 2014-2534 Japanese Unexamined Patent Publication No. 2004-303164 Japanese Unexamined Patent Publication No. 2016-62369

However, in the method using machine learning, in order to make a correct discrimination, it is necessary to collect a large amount of image data captured by changing the imaging environment for vehicles of the same vehicle type as correct answer data used for learning. Furthermore, if the types of vehicles to be identified are newly increased, it is necessary to re-learn, but since new vehicles are being introduced to the market one after another from automobile manufacturers, new vehicles are regularly introduced. It is necessary to collect data and relearn.

Further, in the pattern matching method for performing geometric transformation, it is necessary to accurately specify the corresponding feature when performing geometric transformation between the template and the target image. However, in the prior art, for example, it is premised that all the features to be compared are on the same plane as the license plate, or it is necessary to specify the three-dimensional position and shape of the target vehicle from the moving image of each vehicle. It's not realistic.

As one aspect, it is an object of the present invention to easily extract a captured image including a vehicle of a predetermined vehicle type even if the appearance of the vehicle in the captured image is different.

As one embodiment, among the captured images captured by the first camera, the captured image including the vehicle classified into the first vehicle type is specified, and the number of the vehicle included in the identified captured image is extracted. Then, among the captured images captured by the second camera, the captured image including the vehicle with the number of the extracted vehicle is extracted, and the extracted captured image is used as the captured image of the vehicle corresponding to the first vehicle type. Learn as.

As one aspect, even if the appearance of the vehicle in the captured image is different, the captured image including the vehicle of a predetermined vehicle type can be easily extracted.

It is a block diagram which shows the schematic structure of the monitoring system which concerns on 1st to 3rd Embodiment. It is a schematic diagram for demonstrating the installation situation of the road device. It is a functional block diagram of a road device. It is a figure which shows an example of a monitoring information database. It is a functional block diagram of the image processing apparatus which concerns on 1st to 3rd Embodiment. It is a figure for demonstrating the difference of appearance of a vehicle in a captured image. It is a figure which shows an example of a search result list. It is a block diagram which shows the schematic structure of the computer which functions as the image processing apparatus which concerns on 1st to 3rd Embodiment. It is a flowchart which shows an example of image processing in 1st to 3rd Embodiment. It is a flowchart which shows an example of the 1st learning search processing in 1st Embodiment. It is a flowchart which shows an example of the 2nd learning search processing in 1st Embodiment. It is a figure for demonstrating the outline of the search for the captured image taken at the same point and different points. It is a flowchart which shows an example of the 1st learning search processing in 2nd Embodiment. It is a figure for demonstrating the case of extracting a plurality of templates while shifting the area. It is a flowchart which shows an example of the 1st learning search processing in 3rd Embodiment. It is a flowchart which shows an example of the 2nd learning search processing in 3rd Embodiment.

Hereinafter, an example of an embodiment of the present invention will be described in detail with reference to the drawings. In each of the following embodiments, a case where the image processing apparatus of the present invention is applied to the vehicle monitoring system will be described.

<First Embodiment>
As shown in FIG. 1, the monitoring system 100 according to the first embodiment includes one or more image processing devices 10, a plurality of road devices 20, and a monitoring information database (DB) 30. Each of the road devices 20 and each of the image processing devices 10 and the monitoring information DB 30 are connected to each other via a network such as the Internet.

As shown in FIG. 2, the road device 20 includes a camera 22 and an image recognition device 24, and is provided on the road on which the vehicle travels. The camera 22 captures a predetermined range of the road on which the vehicle travels, and outputs the captured image.

The image recognition device 24 functionally includes an acquisition unit 25, a recognition unit 26, and a transmission unit 27, as shown in FIG.

The acquisition unit 25 acquires the captured image output from the camera 22 and delivers it to the recognition unit 26.

The recognition unit 26 identifies a region including a license plate from the captured image delivered from the acquisition unit 25, and acquires vehicle number information from the identified region by text recognition. Since a conventionally known method can be used for specifying the area of the license plate and recognizing the text of the number information, detailed description thereof will be omitted here. The recognition unit 26 passes the recognized number information to the transmission unit 27.

The transmission unit 27 includes the road device ID which is the identification information of the own device (road device 20), the capture date and time of the captured image acquired by the acquisition unit 25, the number information recognized by the recognition unit 26, and the captured image. It is associated with the image data and transmitted to the monitoring information DB 30 as monitoring information.

FIG. 4 shows an example of the monitoring information DB 30. In the example of FIG. 4, the monitoring information DB 30 includes a road device table 32 and a monitoring information table 34.

Information about each road device 20 is stored in the road device table 32. In the example of FIG. 4, the road device table 32 includes information on the “road device ID”, the “point”, and the “imaging direction”. The "point" is information on the place where the road device 20 identified by the road device ID is installed. The "imaging direction" is information indicating the imaging direction of the camera 22 of the road device 20, and in the example of FIG. 4, the imaging direction is represented by which direction the vehicle is installed so that the vehicle can be imaged from the front. ing.

Further, in the example of FIG. 4, the monitoring information table 34 contains information of "road device ID", "imaging date and time", "number information", and "image data", and is transmitted from each road device 20. Monitoring information is stored.

Functionally, as shown in FIG. 5, the image processing device 10 includes a first learning unit 12, a first search unit 14, a second learning unit 16, and a second search unit 18.

The first learning unit 12 identifies one captured image including a vehicle classified into the vehicle type to be searched among the captured images captured by the camera 22 of the road device 20 at the point X. The "vehicle type" in the present embodiment is a vehicle model (product name).

Specifically, the first learning unit 12 obtains an captured image indicated by "image data" included in the monitoring information in which the "point" is the point X from the monitoring information stored in the monitoring information table 34 of the monitoring information DB 30. Present to the user. Then, the first learning unit 12 identifies one captured image by accepting the designation of the captured image captured by the vehicle of the vehicle type to be searched from the user. The first learning unit 12 sets the specified captured image as the template X for the point X. Setting the specified captured image in the template X is an example of the first learning process of the present invention.

Here, as a learning process for searching a captured image of a vehicle of a vehicle type to be searched, a method of learning a characteristic part of each vehicle type and determining a parameter of a vehicle type discriminating device can be considered. However, in this method, it is necessary to prepare a large amount of image data for learning in advance for each vehicle type, and re-learning is required every time a new vehicle is put on the market. In the present embodiment, simple learning is realized by selecting a template from the captured image as in the first learning unit 12.

When the first search unit 14 receives the search instruction for the vehicle of the vehicle type to be searched, the first search unit 14 applies the template X to the captured image captured by the camera 22 of the road device 20 at the point X, and applies the template X to the vehicle of the vehicle type to be searched. Search for the captured image captured by. Specifically, the first search unit 14 includes each of the captured images indicated by the "image data" included in the monitoring information in which the "point" is the point X among the monitoring information stored in the monitoring information table 34. Perform template matching with template X. The first search unit 14 extracts the monitoring information of the captured image as the template X and the monitoring information of the captured image whose matching degree with the template X is equal to or higher than a predetermined threshold value as the search result for the point X. Add to the search result list (details will be described later).

The second learning unit 16 extracts the vehicle number information included in the captured image set in the template X by the first learning unit 12. Specifically, the second learning unit 16 extracts "number information" of the monitoring information of the captured image set in the template X from the monitoring information table 34.

Further, the second learning unit 16 extracts the captured image including the vehicle having the license plate showing the extracted number information from the captured images captured by the camera 22 of the road device 20 at the point Y different from the point X. .. Specifically, in the second learning unit 16, from the monitoring information table 34, "image data" included in the monitoring information in which the "point" is the point Y and the "number information" matches the extracted number information is The captured image shown is extracted.

The second learning unit 16 sets the extracted captured image as a template Y for searching the captured image captured by the vehicle of the vehicle type to be searched from the captured image at the point Y. Setting the extracted captured image in the template Y is an example of the second learning process of the present invention.

Here, when searching for an image captured by a vehicle of the vehicle type to be searched from the image captured by the camera 22 of the road device 20 at the point Y different from the point X, the template X at the point X is used. Consider the case of application. In this case, as shown in FIG. 6, the positional relationship between the camera 22 and the passing vehicle differs for each camera due to differences in the installation position and angle of the camera 22, the width of the lane, and the like. Therefore, even if the captured images of the same vehicle are captured, if the points are different, the appearance of the vehicle on the captured images will be different.

Therefore, it is difficult to search the captured image to be searched from the captured image of the point Y by applying the template X selected and set from the captured image of the point X. In addition, since each part of the front surface of the vehicle, which is a feature of template matching, is not on the same plane, appropriate matching is performed even if geometric transformation such as rotation or enlargement / reduction of the template or captured image is performed. May not be possible.

Further, in the above-mentioned method of learning the characteristics of each vehicle type and determining the parameters of the vehicle type discriminating device, it is conceivable to absorb the difference in the appearance of the vehicle in the captured image for each point. However, in this case, in addition to the learning data for each vehicle type, it is necessary to prepare learning data in which the positional relationship between the vehicle and the camera 22 is changed according to the situation at each point.

In the present embodiment, the number information of the template X learned by the first learning unit 12 is propagated to the point Y as in the second learning unit 16. As a result, at the point Y, the captured image of the vehicle of the same vehicle type as the vehicle type indicated by the template X of the point X, and the captured image corresponding to the appearance at the point Y, is the template Y at the point Y. Can be set to.

When the second search unit 18 receives the search instruction for the vehicle of the vehicle type to be searched, the second search unit 18 applies the template Y to the captured image captured by the camera 22 of the road device 20 at the point Y, and applies the template Y to the vehicle of the vehicle type to be searched. Search for the captured image captured by. Specifically, the second search unit 18 includes each of the captured images indicated by the "image data" included in the monitoring information in which the "point" is the point Y among the monitoring information stored in the monitoring information table 34. Perform template matching with template Y. The second search unit 18 extracts the monitoring information of the captured image as the template Y and the monitoring information of the captured image whose matching degree with the template Y is equal to or higher than a predetermined threshold value as the search result for the point Y. Add to the search result list.

FIG. 7 shows an example of the search result list. In the example of FIG. 7, the search result list acquires the "point" and the "imaging direction" corresponding to the "road device ID" included in the monitoring information from the road device table 32 in the monitoring information extracted as the search result. And associated. The second search unit 18 sorts and outputs the search result list by using the designated items as keys. For example, it is possible to sort by the road device ID in order of the imaging date and time. In this case, it is possible to check the vehicles of the vehicle type to be searched that have passed a certain point in chronological order. Further, for example, it is possible to sort by number information in order of imaging date and time. In this case, the points where a certain vehicle has passed can be confirmed in chronological order.

Further, when the second search unit 18 includes a plurality of search results of captured images of vehicles having the same number information at the same point or all points, the second search unit 18 searches for captured images of vehicles having the same number information. You may create a search result list so that the results do not overlap. This makes it possible to reduce the redundancy of the search result list.

The first search unit 14 or the second search unit 18 outputs the final search result list.

The image processing device 10 can be realized by, for example, the computer 40 shown in FIG. The computer 40 includes a CPU (Central Processing Unit) 41, a memory 42 as a temporary storage area, and a non-volatile storage unit 43. Further, the computer 40 is connected to an input / output device 44 such as an input device and a display device, an R / W (Read / Write) unit 45 that controls reading and writing of data to the storage medium 49, and a network such as the Internet. It is provided with a communication I / F (Interface) 46. The CPU 41, the memory 42, the storage unit 43, the input / output device 44, the R / W unit 45, and the communication I / F 46 are connected to each other via the bus 47.

The storage unit 43 can be realized by an HDD (Hard Disk Drive), an SSD (Solid State Drive), a flash memory, or the like. The storage unit 43 as a storage medium stores an image processing program 50 for causing the computer 40 to function as the image processing device 10. The image processing program 50 includes a first learning process 52, a first search process 54, a second learning process 56, and a second search process 58.

The CPU 41 reads the image processing program 50 from the storage unit 43, expands the image processing program 50 into the memory 42, and sequentially executes the processes included in the image processing program 50. The CPU 41 operates as the first learning unit 12 shown in FIG. 5 by executing the first learning process 52. Further, the CPU 41 operates as the first search unit 14 shown in FIG. 5 by executing the first search process 54. Further, the CPU 41 operates as the second learning unit 16 shown in FIG. 5 by executing the second learning process 56. Further, the CPU 41 operates as the second search unit 18 shown in FIG. 5 by executing the second search process 58. As a result, the computer 40 that has executed the image processing program 50 functions as the image processing device 10. The CPU 41 that executes the program is hardware.

The function realized by the image processing program 50 can also be realized by, for example, a semiconductor integrated circuit, more specifically, an ASIC (Application Specific Integrated Circuit) or the like.

Next, the operation of the monitoring system 100 according to the first embodiment will be described. Each time the vehicle passes through the imaging range, the camera 22 of each road device 20 captures the passing vehicle and outputs the captured image. The image recognition device 24 of the road device 20 acquires the captured image output from the camera 22 and recognizes the number information from the captured image. Then, the image recognition device 24 associates the street device ID of its own device, the capture date and time of the captured image, the recognized number information, and the image data of the captured image, and transmits the monitoring information to the monitoring information DB 30.

In the monitoring information DB 30, the monitoring information transmitted from each road device 20 is stored in, for example, the monitoring information table 34 as shown in FIG. Then, when a search instruction is input to the image processing device 10, the image processing device 10 executes the image processing shown in FIG.

In step S10 of FIG. 9, the first search unit 14 accepts the designation of the range of the place where the captured image was captured and the range of the date and time as the search range.

Next, in step S20, the first search unit 14 includes the "point" in the range of the place designated as the search range from the monitoring information table 34 of the monitoring information DB 30, and the "imaging date and time" is designated as the search range. Extract the monitoring information included in the range of the date and time. In the following, the monitoring information group extracted in this step is referred to as a "search target list".

Next, in step S30, the first learning search process shown in FIG. 10 is executed.

In step S31 of FIG. 10, the first learning unit 12 determines one point from the “point” of the monitoring information included in the search target list. This point is referred to as "point X".

Next, in step S32, the first learning unit 12 presents to the user the captured image indicated by the "image data" included in the monitoring information in which the "point" is the point X from the search target list. For example, the first learning unit 12 can present the thumbnail image of the captured image to the user by displaying the thumbnail image in a list or displaying the captured image in a state in which the captured images can be sequentially switched.

Next, in step S33, the first learning unit 12 determines whether or not the designation of the captured image captured by the vehicle of the vehicle type to be searched is accepted from the captured image presented to the user. When the designation of the captured image is accepted, the process proceeds to step S34. On the other hand, if the designation of the captured image is not accepted, the process returns to step S31, and the first learning unit 12 determines a point different from the previous time as the point X.

In step S34, the first learning unit 12 sets the captured image designated in step S33 as the template X for the point X.

Next, in step S35, the first search unit 14 obtains each of the captured images indicated by the "image data" included in the monitoring information in which the "point" is the point X among the monitoring information included in the search target list. Perform template matching with template X. The first search unit 14 searches for captured images whose matching degree with the template X is equal to or higher than a predetermined threshold value.

Next, in step S36, the first search unit 14 extracts the monitoring information of the captured image as the template X and the monitoring information of the searched captured image as search results for the point X, and adds them to the search result list. do. Then, the processing returns to the image processing (FIG. 9).

Next, in step S40 of FIG. 9, the second learning search process shown in FIG. 11 is executed.

In step S41 of FIG. 11, the second learning unit 16 extracts the “number information” of the monitoring information of the captured image set in the template X in the step S34 of the first learning search process from the monitoring information table 34.

Next, in step S42, the second learning unit 16 sets the variable i for specifying the point Y different from the point X to 1. In the following, among the types of "points" included in the search target list, the point processed i-th in the second learning search process is referred to as "point i".

Next, in step S43, the second learning unit 16 determines whether or not the variable i exceeds the number N of types of “points” included in the search target list. If i is N or less, the process proceeds to step S44.

In step S44, the second learning unit 16 determines whether or not the point i is different from the point X. If point i ≠ point X, the process proceeds to step S45, and if point i = point X, the process proceeds to step S50.

In step S45, the second learning unit 16 includes "point" in the monitoring information included in the search target list, which is included in the monitoring information whose "point" is the point i and whose "number information" matches the extracted number information. The captured image indicated by "image data" is extracted.

In step S46, the second learning unit 16 determines whether or not the corresponding captured image could be extracted in step S45. If it can be extracted, the process proceeds to step S47. On the other hand, the corresponding captured image cannot be extracted due to reasons such as the vehicle included in the template X not passing through the point i or the license plate not being properly recognized in the captured image at the point i. If so, the process proceeds to step S50.

In step S47, the second learning unit 16 sets the extracted captured image as the template i for searching the captured image captured by the vehicle of the vehicle type to be searched from the captured image at the point i.

Next, in step S48, the second search unit 18 obtains each of the captured images indicated by the "image data" included in the monitoring information in which the "point" is the point i among the monitoring information included in the search target list. Perform template matching with template i. The second search unit 18 searches for captured images whose matching degree with the template i is equal to or higher than a predetermined threshold value.

Next, in step S49, the second search unit 18 extracts the monitoring information of the captured image as the template i and the monitoring information of the searched captured image as search results for the point i, and adds them to the search result list. Then, the process proceeds to step S50.

In step S50, the second search unit 18 increments the variable i by 1, and the process returns to step S43. Then, if it is determined in step S43 that i> N, the processing returns to the image processing (FIG. 9).

Next, in step S60 of FIG. 9, the second search unit 18 outputs, for example, a search result list as shown in FIG. 7, and the image processing ends.

As described above, according to the monitoring system 100 according to the first embodiment, as shown in FIG. 12, the image processing device 10 with respect to the captured image captured at the same point as the captured image specified in the template. Then, apply template matching by template. On the other hand, for the captured images at different points, the number information extracted from the captured images specified in the template is propagated to select the template at different points.

This makes it possible to set a template corresponding to the appearance of the vehicle in the captured image for each point, prepare a vehicle type discrimination device in advance for each point and each vehicle type, or manually set the template. There is no need to do it. Therefore, even if the appearance of the vehicle in the captured image is different, the captured image including the vehicle of a predetermined vehicle type can be easily extracted.

<Second Embodiment>
Next, the second embodiment will be described. In the monitoring system according to the second embodiment, the same parts as those of the monitoring system 100 according to the first embodiment are designated by the same reference numerals and detailed description thereof will be omitted.

As shown in FIG. 1, the monitoring system 200 according to the second embodiment includes an image processing device 210 in place of the image processing device 10 of the monitoring system 100 according to the first embodiment.

Functionally, as shown in FIG. 5, the image processing device 210 includes a first learning unit 212, a first search unit 214, a second learning unit 216, and a second search unit 218.

The first learning unit 212 identifies a plurality of captured images including vehicles classified into the vehicle type to be searched among the captured images captured by the camera 22 of the road device 20 at the point X. Specifically, the first learning unit 212 sets the captured image designated by the user as the captured image captured by the vehicle of the vehicle type to be searched from the captured image at the point X as the initial template. Further, the first learning unit 212 specifies a predetermined number of captured images whose matching degree with the initial template is equal to or higher than a predetermined threshold value among the captured images at the point X. Then, the first learning unit 212 sets the initial template and the specified number of captured images as the template X of the point X.

The first search unit 214 performs template matching with each of the plurality of templates X set by the first learning unit 212 and each of the captured images not set in the template among the captured images at the point X. The first search unit 214 uses the monitoring information of the captured image of each template X and the monitoring information of the captured image whose matching degree with any of the templates X is equal to or higher than a predetermined threshold value as the search result for the point X. Extract and add to the search result list.

The second learning unit 216 extracts "number information" of the monitoring information of the captured image set in each template X from the monitoring information table 34, respectively.

Further, in the second learning unit 216, "image data" indicated by the monitoring information included in the monitoring information in which the "point" is the point Y and the "number information" matches any of the extracted number information from the monitoring information table 34. Extract the captured image. The second learning unit 216 sets each of the extracted captured images as a template Y at the point Y.

The second search unit 218 includes each of the captured images indicated by the "image data" included in the monitoring information in which the "point" is the point Y, and each of the template Y, among the monitoring information stored in the monitoring information table 34. Template matching is performed. The second search unit 218 searches for the monitoring information of the captured image, which is each template Y, and the monitoring information of the captured image whose matching degree with any of the templates Y is equal to or higher than a predetermined threshold value, as a search result for the point Y. And add it to the search result list.

The image processing device 210 can be realized by, for example, the computer 40 shown in FIG. An image processing program 250 for making the computer 40 function as an image processing device 210 is stored in the storage unit 43 of the computer 40. The image processing program 250 has a first learning process 252, a first search process 254, a second learning process 256, and a second search process 258.

The CPU 41 reads the image processing program 250 from the storage unit 43, expands the image processing program 250 into the memory 42, and sequentially executes the processes included in the image processing program 250. The CPU 41 operates as the first learning unit 212 shown in FIG. 5 by executing the first learning process 252. Further, the CPU 41 operates as the first search unit 214 shown in FIG. 5 by executing the first search process 254. Further, the CPU 41 operates as the second learning unit 216 shown in FIG. 5 by executing the second learning process 256. Further, the CPU 41 operates as the second search unit 218 shown in FIG. 5 by executing the second search process 258. As a result, the computer 40 that has executed the image processing program 250 functions as the image processing device 210.

The function realized by the image processing program 250 can also be realized by, for example, a semiconductor integrated circuit, more specifically, an ASIC or the like.

Next, the operation of the monitoring system 200 according to the second embodiment will be described. Similar to the first embodiment, the image processing apparatus 210 performs the image processing shown in FIG. In the second embodiment, the first learning search process shown in FIG. 13 is executed in step S30, and the second learning search process shown in FIG. 11 is executed in step S40. Hereinafter, the first learning search process and the second learning search process in the second embodiment will be described. In FIG. 13, the same processing as that of the first learning search processing in the first embodiment is designated by the same reference numerals and detailed description thereof will be omitted.

After passing through steps S31 to S33 of FIG. 13, in the next step S231, the first learning unit 212 sets the captured image designated in the above step S33 as the initial template.

Next, in step S232, the first learning unit 212 specifies a predetermined number of captured images whose matching degree with the initial template is equal to or higher than a predetermined threshold value among the captured images at the point X.

Next, in step S233, the first learning unit 212 sets the initial template and the specified number of captured images as the template X at the point X.

In step S35, the only difference from the first embodiment is that there are a plurality of templates X.

Further, in the second learning search process shown in FIG. 11, in steps S41, S45, S47, and S48, only a plurality of templates i are different from the first embodiment, and therefore detailed description thereof will be omitted.

According to the monitoring system 200 according to the second embodiment, the image processing device 210 sets a plurality of templates for searching the captured image to be searched from the captured images at each point. As a result, it is possible to reduce search omissions as compared with the case where one template is used for each point.

In the second embodiment, the case of searching the captured image captured by the vehicle of the vehicle type to be searched by template matching between each of the set plurality of templates and the captured image has been described, but the present invention is not limited to this. .. For example, even if the characteristics of the captured image set in the template for each point are learned to generate one discriminative model, and this discriminative model is used to search the captured image to be searched from the captured images at each point. good.

<Third Embodiment>
Next, the third embodiment will be described. In the monitoring system according to the third embodiment, the same parts as those of the monitoring system 100 according to the first embodiment are designated by the same reference numerals and detailed description thereof will be omitted.

As shown in FIG. 1, the monitoring system 300 according to the third embodiment includes an image processing device 310 in place of the image processing device 10 of the monitoring system 100 according to the first embodiment.

Functionally, as shown in FIG. 5, the image processing device 310 includes a first learning unit 312, a first search unit 314, a second learning unit 316, and a second search unit 318.

Similar to the first learning unit 212 in the second embodiment, the first learning unit 312 sets a predetermined number of captured images as the template X for the point X. At this time, the first learning unit 312 sets the region on the captured image for template matching for extracting a predetermined number of captured images within a predetermined range based on the position where the template is extracted from the captured image. Further, within a predetermined range, a plurality of specified templates are set so as to be extracted from different positions of the captured image.

Specifically, the first learning unit 312 presents the captured image of the point X to the user, and accepts and accepts the designation of the image portion of the vehicle in the captured image captured by the vehicle of the vehicle type to be searched. Set the image part as the initial template. Further, the first learning unit 312 performs template matching in an area within a predetermined range from the position where the initial template is extracted and different from the area where the initial template is extracted. The first learning unit 312 sets the captured images extracted from the different regions in the next template. Further, the first learning unit 312 performs template matching in an area within a predetermined range from the position where the newly set template is extracted and different from the area where the template is extracted. The first learning unit 312 sets the captured images extracted from the different regions in the next template. The first learning unit 312 repeats this process and extracts a predetermined number of templates.

For example, as shown in FIG. 14, when the initial template A is specified in the area A, the first learning unit 312 performs template matching by the initial template A in the area B and extracts the template B. Further, the first learning unit 312 performs template matching by the template B in the area C and extracts the template C.

The first search unit 314 applies each of the templates X to the captured image of the point X to perform template matching, and searches for the captured image of the vehicle of the vehicle type to be searched at the point X. At this time, the first search unit 314 performs template matching within a predetermined range including the positions where each of the templates X is extracted.

In this way, while shifting the area for template matching little by little, a predetermined number of templates are extracted from various positions in the captured image, and template matching is performed within a predetermined range including the positions where each template is extracted. As a result, it is possible to accurately search for the captured image of the vehicle to be searched by absorbing the slight difference in the appearance of the vehicle depending on the position in the captured image, which occurs even in the captured image at the same point. Can be done.

The second learning unit 316 extracts "number information" of the monitoring information of the captured image including the image of the vehicle set in each template X from the monitoring information table 34, respectively.

Further, the second learning unit 316 shows "image data" included in the monitoring information from the monitoring information table 34, in which the "point" is the point Y and the "number information" matches any of the extracted number information. Extract the captured image. The second learning unit 316 identifies an image portion of the vehicle included in each of the extracted captured images, and sets the identified image portion as the template Y of the point Y. Since a conventionally known method can be used as a method for specifying the image portion of the vehicle, detailed description thereof will be omitted here.

Further, the second learning unit 316 extracts a predetermined number of template Ys from various positions in the captured image while gradually shifting the template matching area as in the first learning unit 312.

The second search unit 318 includes each of the captured images indicated by the "image data" included in the monitoring information in which the "point" is the point Y, and each of the template Y, among the monitoring information stored in the monitoring information table 34. Template matching is performed. At this time, the second search unit 318 performs template matching within a predetermined range including the positions where each of the templates Y is extracted.

The image processing device 310 can be realized by, for example, the computer 40 shown in FIG. An image processing program 350 for causing the computer 40 to function as an image processing device 310 is stored in the storage unit 43 of the computer 40. The image processing program 350 has a first learning process 352, a first search process 354, a second learning process 356, and a second search process 358.

The CPU 41 reads the image processing program 350 from the storage unit 43, expands the image processing program 350 into the memory 42, and sequentially executes the processes included in the image processing program 350. The CPU 41 operates as the first learning unit 312 shown in FIG. 5 by executing the first learning process 352. Further, the CPU 41 operates as the first search unit 314 shown in FIG. 5 by executing the first search process 354. Further, the CPU 41 operates as the second learning unit 316 shown in FIG. 5 by executing the second learning process 356. Further, the CPU 41 operates as the second search unit 318 shown in FIG. 5 by executing the second search process 358. As a result, the computer 40 that has executed the image processing program 350 functions as the image processing device 310.

The function realized by the image processing program 350 can also be realized by, for example, a semiconductor integrated circuit, more specifically, an ASIC or the like.

Next, the operation of the monitoring system 300 according to the third embodiment will be described. Similar to the first embodiment, the image processing device 310 performs the image processing shown in FIG. In the third embodiment, the first learning search process shown in FIG. 15 is executed in step S30, and the second learning search process shown in FIG. 16 is executed in step S40. In FIG. 15, the same processing as the first learning search processing in the first embodiment is designated by the same reference numeral, and in FIG. 16, the same processing as the second learning search processing in the first embodiment is designated by the same reference numeral. The same reference numerals are given and detailed description thereof will be omitted.

First, the first learning search process will be described.

After passing through steps S31 and S32 of FIG. 15, in the next step S331, the image portion of the vehicle in the captured image in which the vehicle of the vehicle type to be searched is captured from the captured image presented to the user by the first learning unit 312. Determine if the designation has been accepted. When the designation of the image portion is accepted, the process proceeds to step S332. On the other hand, if the designation of the image portion is not accepted, the process returns to step S31, and the first learning unit 312 determines a point different from the previous time as the point X.

In step S332, the first learning unit 312 sets the image portion designated in step S331 as the template X for the point X.

Next, in step S333, the first learning unit 312 captures one image from the captured image indicated by the "image data" included in the monitoring information in which the "point" is the point X among the monitoring information included in the search target list. Select an image.

Next, in step S334, the first learning unit 312 matches the template with the template X within a predetermined range based on the position where the template X is extracted from the captured image in the captured image selected in step S333. I do. The first learning unit 312 specifies an image portion whose matching degree with the template X is equal to or higher than a predetermined threshold value.

Next, in step S335, the first learning unit 312 determines whether or not the image portion matching the template X is specified in step S334. If the image portion is specified, the process proceeds to step S336, and if not specified, the process returns to step S333 and selects the next captured image.

In step S336, the first learning unit 312 sets the image portion specified in step S335 as a new template X for the point X.

Next, in step S337, the first learning unit 312 determines whether or not the total number of image portions (number of templates) set in the template X has reached a predetermined number in steps S332 and S336. When the predetermined number is reached, the process proceeds to step S338. If the number of templates has not reached a predetermined number, the process returns to step S333, and the first learning unit 312 repeats the processes of steps S333 to S336 using the newly set template X.

In step S338, the first search unit 314 applies each of the templates X to the captured image of the point X to perform template matching, and the captured image of the vehicle of the vehicle type to be searched at the point X is captured. To search for. At this time, the first search unit 314 performs template matching within a predetermined range including the positions where each of the templates X is extracted.

Next, the second learning search process will be described.

Through steps S41 to S46 of FIG. 16, in the next step S341, the second learning unit 316 identifies and identifies the image portion of the vehicle included in each of the captured images extracted based on the number information of the template X. The image part is set as the template i of the point i.

Next, in step S342, the second learning unit 316 performs the same processing as in steps S333 to S337 of the first learning search process (FIG. 15), gradually shifting the template matching area, and variously in the captured image. A predetermined number of templates i are extracted from various positions.

Next, in step S343, the second search unit 318 uses each of the captured images indicated by the "image data" included in the monitoring information in which the "point" is the point i among the monitoring information stored in the monitoring information table 34. And template matching with each of the templates i. At this time, the second search unit 318 performs template matching within a predetermined range including the positions where each of the templates i is extracted.

As described above, according to the monitoring system 300 according to the third embodiment, the image processing apparatus 310 extracts a predetermined number of templates from various positions in the captured image while shifting the area little by little. Then, template matching is performed within a predetermined range including the position where this template is extracted. As a result, it is possible to accurately search for the captured image of the vehicle to be searched by absorbing the slight difference in the appearance of the vehicle depending on the position in the captured image, which occurs even in the captured image at the same point. Can be done.

In each of the above embodiments, the item "vehicle type" may be provided in the monitoring information table 34, and the information of the vehicle type that is the key to the search may be stored in the monitoring information of the searched captured image. In this case, when accepting the designation of the first template X from the user, the specification of the vehicle type may be accepted by text.

Further, in the above embodiment, the embodiment in which the image processing program is stored (installed) in the storage unit in advance has been described, but the present invention is not limited to this. The program according to the disclosed technique can also be provided in a form stored in a storage medium such as a CD-ROM, a DVD-ROM, or a USB memory.

The following additional notes will be further disclosed with respect to each of the above embodiments.

(Appendix 1)
Among the captured images captured by the first camera, the captured images including the vehicles classified into the first vehicle type are identified.
The number of the vehicle included in the specified captured image is extracted and
From the captured images captured by the second camera, the captured image including the vehicle with the number of the extracted vehicle is extracted.
The extracted image is learned as an image of a vehicle corresponding to the first vehicle type.
An image processing program characterized by having a computer perform processing.

(Appendix 2)
Among the captured images captured by the first camera, the captured images including the vehicles classified into the first vehicle type are identified.
The first learning process of learning the identified captured image as a captured image of a vehicle corresponding to the first vehicle type is executed.
The number of the vehicle included in the specified captured image is extracted and
From the captured images captured by the second camera, the captured image including the vehicle with the number of the extracted vehicle is extracted.
A second learning process of learning the extracted captured image as a captured image of a vehicle corresponding to the first vehicle type is executed.
When the search instruction for the vehicle of the first vehicle type is received, the learning result of the first learning process is applied to the captured image captured by the first camera to search for the vehicle of the first vehicle type. , The learning result by the second learning process is applied to the captured image captured by the second camera to search for a vehicle of the first vehicle type.
An image processing program characterized by having a computer perform processing.

(Appendix 3)
The search for the vehicle of the first vehicle type performed on the captured image captured by the first or second camera includes the captured image including the captured image of the vehicle not used for the first or second learning process. Let the target do it,
The image processing program according to Appendix 2, characterized by the above.

(Appendix 4)
In the first learning process, one or a plurality of captured images selected from the captured images captured by the first camera are used as the first template, or the captured images captured by the first camera. From the above, it is to learn the characteristics of a plurality of captured images identified based on the first template.
In the second learning process, among the captured images captured by the second camera, one or a plurality of captured images including the vehicle number indicated by the one or a plurality of the first templates are referred to as the second template. The image processing according to Appendix 2 or Appendix 3, which is to learn the characteristics of a plurality of captured images specified based on the second template from the captured image captured by the second camera. program.

(Appendix 5)
The first template or the second template is an image portion showing a vehicle in a captured image.
When the captured image is specified based on the first template or the second template, the area for matching with the first template or the second template is defined as the first template or the second template. The template is set within a predetermined range based on the extracted position, and the image portion matching the first template or the second template is set in the new first template or the second template. The image processing program according to Appendix 4.

(Appendix 6)
When a plurality of the first template or the second template is set, the matching area is performed so that the positions of each of the first templates or the second templates in the captured image are different. The image processing program according to Appendix 5 for setting.

(Appendix 7)
When searching for a captured image including a vehicle of the first vehicle type, the additional feature is to search from the captured image captured by the first camera or the second camera existing in a predetermined range. The image processing program according to any one of Items 2 to 6.

(Appendix 8)
The image processing according to any one of Supplementary note 2 to Supplementary note 7, wherein when searching for a captured image including a vehicle of the first vehicle type, the image is searched from the captured images captured within a predetermined period. program.

(Appendix 9)
Item 1. Image processing program described in.

(Appendix 10)
Among the captured images captured by the first camera, the first learning unit for specifying the captured image including the vehicle classified into the first vehicle type, and
The number of the vehicle included in the specified captured image is extracted, and the captured image including the vehicle with the extracted number of the vehicle is extracted from the captured images captured by the second camera, and the extracted captured image is used. , A second learning unit that learns as a captured image of a vehicle corresponding to the first vehicle type,
An image processing device characterized by including.

(Appendix 11)
Among the captured images captured by the first camera, the captured image including the vehicle classified into the first vehicle type is specified, and the identified captured image is used as the captured image of the vehicle corresponding to the first vehicle type. The first learning unit that executes the first learning process to be learned,
The number of the vehicle included in the specified captured image is extracted, and the captured image including the vehicle with the extracted number of the vehicle is extracted from the captured images captured by the second camera, and the extracted captured image is used. , A second learning unit that executes a second learning process for learning as a captured image of a vehicle corresponding to the first vehicle type.
When the search instruction for the vehicle of the first vehicle type is received, the learning result of the first learning process is applied to the captured image captured by the first camera to search for the vehicle of the first vehicle type. The first search unit and
A second search unit that searches for a vehicle of the first vehicle type by applying the learning result of the second learning process to the captured image captured by the second camera.
An image processing device characterized by including.

(Appendix 12)
The first search unit searches for a vehicle of the first vehicle type performed on the captured image captured by the first camera, and includes a captured image of a vehicle that is not used in the first learning process. To be executed by the target,
The second search unit searches for a vehicle of the first vehicle type performed on the captured image captured by the second camera, and includes an captured image of a vehicle that is not used in the second learning process. The image processing apparatus according to Appendix 11, wherein the image processing apparatus is to be executed.

(Appendix 13)
In the first learning process, one or a plurality of captured images selected from the captured images captured by the first camera are used as the first template, or the captured images captured by the first camera. From the above, it is to learn the characteristics of a plurality of captured images identified based on the first template.
In the second learning process, among the captured images captured by the second camera, one or a plurality of captured images including the vehicle number indicated by the one or a plurality of the first templates are referred to as the second template. 11 or 12 characterized by learning the characteristics of a plurality of captured images specified based on the second template from the captured images captured by the second camera. The image processing apparatus according to.

(Appendix 14)
The first template and the second template are image portions showing a vehicle in a captured image.
When the captured image is specified based on the first template, the first learning unit uses the region for matching with the first template as a reference from the position where the first template is extracted. The image portion within a predetermined range and matched with the first template is set in the new first template.
In the second learning unit, when the captured image is specified based on the second template, the region for matching with the second template is set with reference to the position where the second template is extracted. The image processing apparatus according to Appendix 13, wherein an image portion within a predetermined range and matched with the second template is set in a new second template.

(Appendix 15)
When a plurality of the first templates are set, the first learning unit sets a region for performing the matching so that the positions of the first templates are different in the captured image.
The image processing apparatus according to Appendix 14, wherein the second learning unit sets a region for performing the matching so that the positions of the second templates in the captured image are different from each other.

(Appendix 16)
When searching for a captured image including a vehicle of the first vehicle type,
The first search unit searches from the captured images captured by the first camera existing within a predetermined range.
The image processing apparatus according to any one of Supplementary note 11 to Supplementary note 15, wherein the second search unit searches from an image captured by the second camera existing in a predetermined range.

(Appendix 17)
The first search unit and the second search unit are characterized in that, when searching for a captured image including a vehicle of the first vehicle type, the first search unit and the second search unit search from the captured images captured within a predetermined period. The image processing apparatus according to any one of Supplementary note 16.

(Appendix 18)
When the first search unit or the second search unit outputs a captured image including the searched vehicle of the first vehicle type, the first search unit or the second search unit does not duplicately output the captured image of the vehicle having the same number. The image processing apparatus according to any one of Supplementary note 11 to Supplementary note 17, which is a feature.

(Appendix 19)
Among the captured images captured by the first camera, the captured images including the vehicles classified into the first vehicle type are identified.
The number of the vehicle included in the specified captured image is extracted and
From the captured images captured by the second camera, the captured image including the vehicle with the number of the extracted vehicle is extracted.
The extracted image is learned as an image of a vehicle corresponding to the first vehicle type.
An image processing method characterized by a computer performing the processing.

(Appendix 20)
Among the captured images captured by the first camera, the captured images including the vehicles classified into the first vehicle type are identified.
The first learning process of learning the identified captured image as a captured image of a vehicle corresponding to the first vehicle type is executed.
The number of the vehicle included in the specified captured image is extracted and
From the captured images captured by the second camera, the captured image including the vehicle with the number of the extracted vehicle is extracted.
A second learning process of learning the extracted captured image as a captured image of a vehicle corresponding to the first vehicle type is executed.
When the search instruction for the vehicle of the first vehicle type is received, the learning result of the first learning process is applied to the captured image captured by the first camera to search for the vehicle of the first vehicle type. , The learning result by the second learning process is applied to the captured image captured by the second camera to search for a vehicle of the first vehicle type.
An image processing method characterized by a computer performing the processing.

10, 210, 310 Image processing device 12, 212, 312 First learning unit 14, 214, 314 First search unit 16, 216, 316 Second learning unit 18, 218, 318 Second search unit 20 Road device 22 Camera 24 Image recognition device 30 Monitoring information database 32 Road device table 34 Monitoring information table 40 Computer 41 CPU
42 Memory 43 Storage unit 49 Storage media 50, 250, 350 Image processing program 100, 200, 300 Monitoring system

Claims (13)

Among the captured images captured by the first camera, the captured images including the vehicles classified into the first vehicle type are identified.
The number of the vehicle included in the specified captured image is extracted and
From the captured images captured by the second camera, the captured image including the vehicle with the number of the extracted vehicle is extracted.
The extracted image is learned as an image of a vehicle corresponding to the first vehicle type.
An image processing program characterized by having a computer perform processing. Among the captured images captured by the first camera, the captured images including the vehicles classified into the first vehicle type are identified.
The first learning process of learning the identified captured image as a captured image of a vehicle corresponding to the first vehicle type is executed.
The number of the vehicle included in the specified captured image is extracted and
From the captured images captured by the second camera, the captured image including the vehicle with the number of the extracted vehicle is extracted.
A second learning process of learning the extracted captured image as a captured image of a vehicle corresponding to the first vehicle type is executed.
When the search instruction for the vehicle of the first vehicle type is received, the learning result of the first learning process is applied to the captured image captured by the first camera to search for the vehicle of the first vehicle type. , The learning result by the second learning process is applied to the captured image captured by the second camera to search for a vehicle of the first vehicle type.
An image processing program characterized by having a computer perform processing. The search for the vehicle of the first vehicle type performed on the captured image captured by the first or second camera includes the captured image including the captured image of the vehicle not used for the first or second learning process. Let the target do it,
The image processing program according to claim 2, wherein the image processing program is characterized by the above. In the first learning process, one or a plurality of captured images selected from the captured images captured by the first camera are used as the first template, or the captured images captured by the first camera. From the above, it is to learn the characteristics of a plurality of captured images identified based on the first template.
In the second learning process, among the captured images captured by the second camera, one or more captured images including the vehicle numbers indicated by the one or more first templates are referred to as the second template. 2. The method according to claim 2 or 3, wherein the feature of a plurality of captured images specified based on the second template is learned from the captured images captured by the second camera. Image processing program. The first template or the second template is an image portion showing a vehicle in a captured image.
When the captured image is specified based on the first template or the second template, the area for matching with the first template or the second template is defined as the first template or the second template. The template is set within a predetermined range based on the extracted position, and the image portion matching the first template or the second template is set in the new first template or the second template. The image processing program according to claim 4. When a plurality of the first template or the second template is set, the matching area is performed so that the positions of each of the first template or each of the second template in the captured image are different. The image processing program according to claim 5. When searching for a captured image including a vehicle of the first vehicle type, the claim is characterized by searching from the captured image captured by the first camera or the second camera existing in a predetermined range. The image processing program according to any one of items 2 to 6. The invention according to any one of claims 2 to 7, wherein when searching for a captured image including a vehicle of the first vehicle type, the search is performed from the captured images captured within a predetermined period. Image processing program. Any of claims 2 to 8, wherein when the captured image including the searched vehicle of the first vehicle type is output, the captured image of the vehicle having the same number is not output in duplicate. The image processing program according to item 1. Among the captured images captured by the first camera, the first learning unit for specifying the captured image including the vehicle classified into the first vehicle type, and
The number of the vehicle included in the specified captured image is extracted, and the captured image including the vehicle with the extracted number of the vehicle is extracted from the captured images captured by the second camera, and the extracted captured image is used. , A second learning unit that learns as a captured image of a vehicle corresponding to the first vehicle type,
Image processing equipment including. Among the captured images captured by the first camera, the captured image including the vehicle classified into the first vehicle type is specified, and the identified captured image is used as the captured image of the vehicle corresponding to the first vehicle type. The first learning unit that executes the first learning process to be learned,
The number of the vehicle included in the specified captured image is extracted, and the captured image including the vehicle with the extracted number of the vehicle is extracted from the captured images captured by the second camera, and the extracted captured image is used. , A second learning unit that executes a second learning process for learning as a captured image of a vehicle corresponding to the first vehicle type.
When the search instruction for the vehicle of the first vehicle type is received, the learning result of the first learning process is applied to the captured image captured by the first camera to search for the vehicle of the first vehicle type. The first search unit and
A second search unit that searches for a vehicle of the first vehicle type by applying the learning result of the second learning process to the captured image captured by the second camera.
An image processing device characterized by including. Among the captured images captured by the first camera, the captured images including the vehicles classified into the first vehicle type are identified.
The number of the vehicle included in the specified captured image is extracted and
From the captured images captured by the second camera, the captured image including the vehicle with the number of the extracted vehicle is extracted.
The extracted image is learned as an image of a vehicle corresponding to the first vehicle type.
An image processing method characterized by a computer performing the processing. Among the captured images captured by the first camera, the captured images including the vehicles classified into the first vehicle type are identified.
The first learning process of learning the identified captured image as a captured image of a vehicle corresponding to the first vehicle type is executed.
The number of the vehicle included in the specified captured image is extracted and
From the captured images captured by the second camera, the captured image including the vehicle with the number of the extracted vehicle is extracted.
A second learning process of learning the extracted captured image as a captured image of a vehicle corresponding to the first vehicle type is executed.
When the search instruction for the vehicle of the first vehicle type is received, the learning result of the first learning process is applied to the captured image captured by the first camera to search for the vehicle of the first vehicle type. , The learning result by the second learning process is applied to the captured image captured by the second camera to search for a vehicle of the first vehicle type.
An image processing method characterized by a computer performing the processing.

Images