JP7199645B2 - Object recognition system and object recognition method - Google Patents

Description

The present invention relates to an object recognition system and an object recognition method that perform processing such as object recognition.

Conventionally, object recognition methods for recognizing objects from images have been proposed. The method of object recognition can be suitably used mainly in a system that handles captured images, and can be used for automating and improving the efficiency of various processes.

As an example, Patent Literature 1 discloses an object recognition device that detects a target around a vehicle and recognizes the object based on the detected target. Further, according to Patent Document 2, a target region in which the shape of a recognition target exists is extracted from an image captured by an imaging device provided in a vehicle, and the extracted target region out of the entire region of the image is An object recognition device is disclosed that selectively executes recognition processing for a recognition target object.

As specific means for object recognition, YOLO suitable for object recognition from moving images, TENSORFLOW (registered trademark, hereinafter the same) suitable for object recognition from still images, etc. have been developed. Artificial intelligence [AI] technology is applied to these means, and it is possible to improve the accuracy of object recognition through machine learning. In machine learning, advanced deep learning tends to be adopted, and the accuracy and speed of object recognition are improving.

JP 2018-041396 A JP 2017-130155 A

By the way, when performing object recognition of another object (second object) accompanying a certain object (first object) in a moving image, it is necessary to recognize the second object directly from the moving image with high accuracy. is not easy. Note that the “object” in the present application is a concept corresponding to annotation data in machine learning, object recognition, and the like. For example, when trying to recognize a number (an example of a second object) displayed on a license plate from a video captured by a camera that monitors a vehicle (an example of a first object), the vehicle A more precise recognition process is required compared to the object recognition of . Therefore, when trying to recognize the number directly from the moving image, there is a possibility that the recognition accuracy may be lowered.

In order to solve this problem, a method of recognizing a number by extracting a still image from a moving image and recognizing an object from the extracted still image is conceivable. However, for example, if a still image of each frame is uniformly extracted from a moving image, unnecessary still images that do not include a vehicle will also be extracted, and there is a risk that the operation load of the system will become excessive. In addition, if the still images to be extracted contain many unnecessary still images, there is a possibility that the speed and accuracy of object recognition may be lowered.

In view of the above-described problems, the present invention provides an object recognition system and an object recognition method that facilitate accurate recognition of a second object that accompanies a first object in a moving image while suppressing the operation load of the system. for the purpose of providing

An object recognition system according to the present invention comprises: a moving image object recognition unit that recognizes a first object from a moving image; a still image extracting unit that extracts still images of a plurality of frames including the first object from the moving image; a still image object recognition unit for recognizing a second object attached to the first object from the still image. According to this configuration, it becomes easy to accurately recognize the second object that accompanies the first object in the moving image while suppressing the operation load of the system. Note that "accompanying" here is not limited to the form in which the second object is included in the first object, but also includes other forms in which the time appearing in the moving image is closely related.

More specifically, the above configuration is an extracting unit that extracts, among the extracted still images, still images that satisfy a predetermined condition regarding the degree of exposure of at least one of the first object and the second object. and the still image object recognition unit may recognize a second object included in the first object from the extracted still image. According to this configuration, it is possible to more efficiently recognize the second object included in the first object from the still image.

Further, more specifically, as the above configuration, a first process of extracting, from among the extracted still images, one having the highest degree of exposure of a specific object that is a second object; a second process for recognizing display information in the second object from the image; A third process in which the second process is repeated may be executed. The "display information" here is characters, graphics, symbols, or a combination thereof, and may correspond to, for example, a number (automobile registration number) in the present application.

Further, more specifically, as the above configuration, when the recognition in the second process is not successful even if the number of repetitions reaches a predetermined number, the first process to the third process are performed with the specific object as the first object. may be configured to execute

Further, more specifically as the above configuration, the still image extracting unit selects at least one of the resolution of the still image to be extracted and the time interval for extracting the still image according to the degree of exposure of the first object in the moving image. It is good also as a structure which adjusts.

According to this configuration, it is possible to efficiently extract still images of high importance while minimizing the burden on the system. Further, more specifically, the above configuration may be configured such that a correction process suitable for deep learning is performed on the one having the highest degree of exposure of the second object among the plurality of extracted still images.

Further, more specifically as the above configuration, the moving image is an image captured using a camera, and a distance detection unit for detecting a distance between the camera and a first object is provided, and the still image extraction unit is , the extraction of the still image may be started when the distance becomes equal to or less than a predetermined value. According to this configuration, it is possible to minimize the extraction of an unclear still image in which the first target image is small.

Further, more specifically, the above configuration includes a direction detection unit that detects the orientation of the first object, and the still image extracting unit extracts the still image when the orientation satisfies a predetermined condition. It is good also as a structure which starts. According to this configuration, it is possible to minimize extraction of a still image having a problem with the orientation of the first object.

More specifically, the above configuration may be configured such that the information on the first object and the information on the second object are held in association with each other. According to this configuration, it is possible to collectively manage the first object and the second object. Furthermore, in the above configuration, different object recognition techniques may be used in the moving image object recognition section and the still image object recognition section. According to this configuration, it is possible to efficiently perform object recognition by using the optimum method for object recognition from moving images and object recognition from still images.

Further, an object recognition method according to the present invention includes a moving image object recognition step of recognizing a first object from a moving image, a still image extracting step of extracting a still image including the first object from the moving image, and and a still image object recognition step of recognizing a second object attached to the first object.

According to the object recognition system and the object recognition method according to the present invention, it becomes easy to accurately recognize the second object accompanying the first object in the moving image while suppressing the operation load of the system.

1 is a block diagram relating to the configuration of a vehicle management system 1 according to this embodiment; FIG. It is explanatory drawing of a mode that the smart phone 11 was installed in the site. 2 is a block diagram of a functional configuration of an information processing device 14; FIG. FIG. 3 is an explanatory diagram relating to an image of a smartphone 11; It is an explanatory diagram of a local license plate. 10 is a flowchart relating to the flow of pre-stage processing; FIG. 5 is an explanatory diagram of how the distance D changes with the movement of the vehicle; 4 is a timing chart for extracting a still image from a moving image; 10 is a flowchart regarding the flow of post-processing. FIG. 10 is an explanatory diagram related to information such as check results;

A vehicle management system according to an embodiment of the present invention (one form of an object recognition system according to the present invention) will be described below with reference to each drawing.

1. Configuration of Vehicle Management System FIG. 1 is a block diagram showing a schematic configuration of a vehicle management system 1 according to the present embodiment. As shown in this figure, the vehicle management system 1 includes an approach road front side photographing smartphone 11a, an approach road rear side photographing smartphone 11b, an exit road front side photographing smartphone 11c, an exit road rear side photographing smartphone 11d, a communication network 12, An edge server 13 , an information processing device 14 and a management server 15 are provided. In the following description, the smartphones 11a to 11d may be collectively referred to as "smartphone 11". Moreover, in each drawing, a smart phone (SmartPhone) may be abbreviated as "SP."

In this embodiment, as an example, a plurality of smartphones 11 are installed in each of a plurality of sites (sites 1 to 3 in the example shown in FIG. 1). The "site" in this embodiment is a place where vehicles can enter and exit with the permission of a business operator who manages the site (hereinafter referred to as "administrator"), for example, a parking lot owned by the administrator. is applicable. When a site has multiple entrances and exits for vehicles, by installing the smartphones 11 near all the entrances and exits, it is possible to monitor vehicles entering or leaving the site without omission. It should be noted that the "vehicle" in this embodiment refers to a vehicle equipped with a license plate, and the "number" refers to the vehicle registration number indicated on the license plate. Generally, license plates are provided on both the front and rear sides of a vehicle.

On the other hand, the edge server 13, the information processing device 14, and the management server 15 are collectively installed in the management center. The "management center" in this embodiment is a place where vehicles entering and exiting the site are managed, and corresponds to, for example, a room in a building owned by a manager. The vehicle management system 1 automatically monitors vehicles entering each site and collectively manages them.

The management server 15 is a server used for managing vehicles and the like that enter each site. In the management server 15, the numbers of all vehicles permitted to enter the premises (hereinafter referred to as "permitted vehicles" for convenience) are registered as a database. The management server 15 accumulates this information in the database each time a new permit vehicle number is entered by an administrator or the like. Note that the management server 15 may be provided on a data center via the Internet network.

The smartphone 11 has a camera function of capturing a subject and obtaining a moving image (image), and also has a function of measuring the distance from the smartphone to the subject (ranging function). This ranging function is realized by a “stereo camera” in which the smartphone 11 is equipped with a plurality of lenses. The distance measurement function may be realized by providing a distance measurement sensor or the like instead of the stereo camera. The smartphone for photographing the front side of the approach road 11a has a role of photographing the front side of the vehicle entering the site, and the smartphone for photographing the rear side of the approach road 11b has a role of photographing the rear side of the vehicle entering the site. The smart phone 11c for taking pictures of the front side of the vehicle leaving the site, and the smart phone 11d for taking pictures of the rear side of the exit road take pictures of the rear side of the vehicle leaving the site. Note that a camera alone or a device having another camera may be applied instead of the smart phone 11 .

FIG. 2 illustrates how the smartphone 11 is installed within the site. As shown in the figure, the smartphone 11 is installed so that the subject is a passageway for vehicles entering and leaving the site. As a result, when a vehicle passes through an approach road (traffic road for entering the site) and an exit road (traffic road for exiting the site), the appearance of the front and rear sides of the vehicle can be viewed as the subject of the smartphone 11. can be accommodated. The smartphone 11 can store the license plate of the vehicle (or the number information displayed on the license plate), the driver, the seat belt worn by the driver, dirt and scratches (including dents) on the vehicle, and predetermined equipment (hereinafter , these are sometimes collectively referred to as “license plates, etc.”) to be placed in an appropriate position so that the subject can be captured. Note that the “equipment” here is, for example, an item that a permitted vehicle is obligated to equip, and can be a subject of the smartphone 11 .

For example, as shown in FIG. 2, each smartphone 11 is provided near the security guard's room, and is desirably installed at a position (support in the example of this figure) where the entire vehicle can be photographed obliquely from above. In the example of this figure, the approach road front side photographing smartphone 11a and the approach road rear side photographing smartphone 11b are installed so that their rear portions face each other almost directly above the approach road. The smartphone 11b for photographing the rear side of the approach road is appropriately arranged so as to photograph the entire front side of the vehicle traveling on the road from diagonally above the front and the entire rear side of the vehicle traveling on the approach road from the diagonally upper rear. ing.

On the other hand, the smartphone for photographing the front side of the exit road 11c and the smartphone for photographing the rear side of the exit road 11d are installed so that their rear portions face each other almost directly above the road to exit, and the smartphone for photographing the front side of the exit road 11c is installed on the vehicle traveling on the exit road. The smartphone 11d for photographing the rear side of the exit road is appropriately arranged so as to photograph the entire front side of the vehicle from diagonally above the front, and to photograph the entire rear side of the vehicle traveling on the exit road from the diagonally upper rear.

In this way, each smartphone 11 is desirably arranged in a position overlapping the vehicle when viewed from above, and this makes it possible to simplify or omit various corrections in the vehicle width direction for image data obtained by the smartphone 11. . Note that in the following description, a combination of smartphones 11 that capture images of the same vehicle from the front and rear sides, that is, a combination of a smartphone 11a for photographing the front side of an approach road and a corresponding smartphone 11b for photographing the rear side of an approach road, and a combination of a smartphone 11b for photographing the rear side of an approach road Each combination of the front side photographing smartphone 11c and the corresponding exit road rear side photographing smartphone 11d may be expressed as "a pair of smartphones 11".

For example, in the image of the smartphone 11c for photographing the front side of the exit road, as illustrated in FIG. 4, the license plate on the front side of the vehicle C1 is directly reflected, and the driver and the seat belt worn by the driver are reflected through the windshield. become. The above license plate and the like are included in the vehicle in both moving images and still images showing the vehicle, and are attached to the vehicle. Further, since the smartphone 11 has a distance measurement function, it is possible to obtain information on the distance D from the smartphone 11 to the vehicle if the position of the vehicle in the subject is specified. This distance D is detected by a distance detection unit 43, which will be described later. In addition, the smartphone 11 may be provided in the security guard's room in order to prevent the temperature of the smartphone 11 from rising, lowering, or deteriorating over time, and to more accurately photograph the sides of the vehicle. Entry and exit can be managed by collating data obtained from smartphones (11a, 11b) that photograph incoming vehicles and smartphones (11c, 11d) that photograph exiting vehicles.

The communication network 12 is a network used for communication between each smartphone 11 and the information processing device 14 . As a specific form of the communication network 12, either a wired network or a wireless network may be applied. It is also possible to use the Internet or the like as the communication network 12 .

The edge server 13 is interposed between the communication network 12 and the information processing device 14, for example, an environment in which deep learning can be executed, various values used in deep learning (learned hyperparameters of artificial intelligence, model structural information (hyperparameters, weight data given when the learning data is learned, reward function in the reinforcement learning model) are stored. The edge server 13 is installed with environment software (Python, anaconda, jupyter, opencv, TENSORFLOW, YOLO, etc.) capable of executing deep learning.

The information processing device 14 is composed of a server with higher performance than the edge server 13, and in addition to object recognition from moving images and still images, new learning (reinforcement learning, additional learning) of deep learning related to vehicle monitoring and management. It is a device that executes related processing and the like. The information processing device 14 also includes a moving image processing engine 14a that performs processing such as image recognition on moving images, and a still image processing engine 14b that performs processing such as image recognition on still images.

The moving image processing engine 14a employs algorithms such as YOLO (You Only Look Once) and OPEN CV (Open Source Computer Vision Library), and is excellent in the function of recognizing objects from moving images in real time. The moving image processing engine 14a is capable of accurately and quickly object-recognizing all vehicles with different appearances (inclination, size, direction) as "vehicles" by machine learning. As a result, it is possible to minimize omissions in recognizing vehicles in moving images. "Machine learning" is a method of autonomously discovering laws and rules by repeatedly learning based on given information. However, the specific configuration of the moving image processing engine 14a is not limited to the above example, and instead of YOLO or the like, other means suitable for object recognition from moving images may be employed.

On the other hand, the still image processing engine 14b employs TENSORFLOW, which is a machine learning library, and is excellent in the function of quickly and accurately recognizing objects from still images. In particular, TENSORFLOW is a library capable of deep learning, and can smoothly process multidimensional data structures. “Deep learning” is machine learning using a multi-layered neural network (an information processing model imitating the mechanism of the human brain and nervous system).

According to the still image processing engine 14b, it is possible to recognize objects such as a license plate from a still image including a vehicle with high accuracy, and also to recognize the number displayed on the license plate. However, the specific configuration of the still image processing engine 14b is not limited to the above example, and other means suitable for object recognition from still images may be employed instead of TENSORFLOW. Also, the edge server 13 and the information processing device 14 may be realized by the same server.

FIG. 3 shows a block diagram of the main functional configuration of the information processing device 14. As shown in FIG. As shown in the figure, the information processing device 14 includes a control unit 40, a communication unit 41, a moving object recognition unit 42, a distance detection unit 43, a speed detection unit 44, a still image extraction unit 45, an exposure detection unit 46, an extraction It has a section 47 , an image processing section 48 , a still image object recognition section 49 , a check execution section 50 and an abnormality signal output section 51 .

The control unit 40 appropriately controls the functional units 41 to 51 so that the information processing device 14 operates normally. The main operations of the information processing device 14 will be explained in detail later. The communication unit 41 communicates with external devices including each smartphone 11 and the management server 15 .

The moving image object recognition unit 42 recognizes the vehicle from the moving image using the object recognition function from the moving image by the moving image processing engine 14a. It should be noted that the moving image object recognition unit 42 can recognize objects separately for moving images in which a plurality of vehicles appear at the same time. For example, when two vehicles enter the subject of one smartphone 11, it is possible to recognize the objects of these two vehicles separately, and the information processing device 14 advances processing focusing on each in parallel. It is possible. The object recognition is mainly performed in the process of step S10, which will be described later.

The distance detection unit 43 detects a distance D (see FIG. 2 ) between the vehicle whose object has been recognized and the smartphone 11 using a distance measurement function provided in the smartphone 11 . Detection of the distance is mainly performed in the process of step S12, which will be described later.

The speed detection unit 44 detects the speed of the vehicle whose object has been recognized. As a method of detecting the speed, a method of using a speed sensor installed near the smartphone 11, a method of detecting the speed from the motion of the vehicle in the moving image, or the like can be adopted. The detection of the speed is mainly carried out in the process of step S11, which will be described later.

The still image extraction unit 45 extracts a plurality of still images (images of each frame, for example, 30 images at intervals of 0.1 seconds) from the moving image and temporarily stores them in a storage area. Note that the still image extraction unit 45 can appropriately change the resolution of the still images extracted from the moving image and the time interval for extracting the still images from the moving image. Extraction of the still image is mainly performed in the process of step S13, which will be described later.

The exposure detection unit 46 detects the exposure of a vehicle whose object has been recognized in one frame of a moving image (corresponding to one still image) (hereinafter referred to as "first exposure"). The first degree of exposure may be the ratio between the size (area) of the vehicle in one frame of the moving image and the size of the frame (exposure ratio to the frame), or may be the size of the vehicle itself. As information on the size of the vehicle, the area inside the contour of the vehicle may be used, and other information that serves as an indicator of the size (for example, a rectangle indicated by broken lines in FIG. 4 (a rectangle whose four sides are in contact with the vehicle) ) may be adopted. In addition, another value that allows comparison of vehicle exposure levels between frames may be considered as the first exposure level. The detection of the first exposure is mainly carried out in the process of step S14, which will be described later. Further, the exposure detection unit 46 detects the exposure of the license plate recognized as an object in one frame of the moving image (corresponding to one still image) (hereinafter referred to as "second exposure"). The second degree of exposure may be the ratio of the size of the license plate to the size of the vehicle in one frame of the video (exposure ratio to the vehicle), and the ratio of the size of the license plate to the size of the frame (exposure ratio to the frame). Alternatively, it may be the size of the license plate itself. Also, as in the case of the first degree of exposure, various values that enable comparison of the degree of exposure of the license plate between frames can be regarded as the second degree of exposure. The detection of the second exposure is mainly performed in the process of step S20, which will be described later.

The extracting unit 47 extracts a predetermined condition (hereinafter referred to as “exposure condition” for convenience) regarding at least one of the first exposure and the second exposure among the plurality of still images extracted by the still image extracting unit 45. ) are extracted. The degree of exposure condition may be, for example, that the value of the degree of exposure is a predetermined value or more, or that the degree of exposure is the highest, and that the value of the degree of exposure is counted from the highest degree of exposure and falls within a predetermined number. It is good as Specific contents of the exposure condition according to the present embodiment will be described in detail again. The higher the degree of exposure, the more likely it is that the still image object recognizing unit 49 will be able to recognize the numbered object more easily. . In addition, instead of the exposure level condition, another condition regarding ease of object recognition by the still image object recognition unit 49 may be set. The extraction is mainly performed in the process of step S20, which will be described later.

The image processing unit 48 sequentially performs image processing such as threshold processing, edge processing (edge detection processing), and tilt correction processing on the still image. Note that the image processing unit 48 may perform only one or two of these image processes, and further performs other image processes for making object recognition from still images more advantageous. You may implement it. The image processing is mainly performed in the processing of step S21, which will be described later. These image processes can also be regarded as correction processes suitable for deep learning.

Here, "threshold processing" is processing for converting an image into a binary image (single-channel image). According to threshold processing, for example, when converting to a black-and-white binary image, pixels whose channel values exceed a predetermined threshold are converted to white pixels, and pixels whose channel values do not exceed the threshold are converted to black pixels. pixels, respectively. An image that has undergone threshold processing makes it easier to select portions of the image that differ in brightness.

"Edge processing" is processing for detecting portions (edges) where brightness (shading) or color suddenly changes in an image. Since the contours and lines of an object in an image generally have abrupt changes in shading, it is possible to detect these contours and lines by edge processing. Edges are important information that reflects the structure of objects, and edge processing is extremely useful when recognizing objects from still images. In order to perform edge processing more effectively, it is usually useful to perform threshold processing on the image in advance.

A Canny edge detector may be employed as an algorithm for performing edge processing. According to the edge processing (Canny processing) when this algorithm is adopted, compared to the case where other algorithms (Sobel filter, Laplacian filter, etc.) are adopted, there are fewer contour detection omissions and false detections, and each point It is characterized in that it is easy to detect a single contour and to detect a portion that is truly an edge. Canny processing is achieved by smoothing an image with a Gaussian filter, calculating the magnitude and direction of the gradient from the differential calculation result of this smoothed image, and performing Non maximum Suppression processing and Hysteresis Threshold processing. .

Further, the "tilt correction process" is a process of rotating the image so as to eliminate the tilt when a straight line or the like detected in the image is tilted from the horizontal direction (or the vertical direction). For example, by performing tilt correction processing so that the horizontal edge of the license plate in the image is aligned with the horizontal direction, the character strings of the number are lined up in the horizontal direction, making it easier to recognize the number. It becomes possible to In order to facilitate detection of straight lines and the like in an image, it is usually useful to perform edge processing in advance.

The still image object recognition unit 49 uses the object recognition function from the still image by the still image processing engine 14b to extract the license plate of the vehicle from the still image, the license plate number (information indicated on the license plate), and the seat worn by the driver. Object recognition of belts, vehicle dirt/scratches, and accessories. Object recognition from a still image is a concept that includes recognition of display information represented in a still image, such as the case of recognizing a vehicle license plate. The object recognition is mainly performed in the process of step S22 described later, but the object recognition of the license plate is performed in the process of step S20 in order to detect the second exposure degree described above. As described above, in this embodiment, the moving image object recognition unit 42 and the still image object recognition unit 49 use different object recognition methods. Therefore, compared to the case where the same object recognition method is used for both, it is possible to perform object recognition efficiently by using the optimal method for object recognition from moving images and object recognition from still images. .

Note that the still image object recognition unit 49 can recognize a number not only from a general license plate without a pattern, but also from a license plate showing a so-called local number. FIG. 5 shows an example of a recognizable license plate showing a local number (local license plate). On the license plate, a pattern (here, a wave pattern) is depicted in addition to numbers and symbols that identify the vehicle. The still image object recognition unit 49 can extract only the characters and symbols registered as the number from the primitive shape judgment for such a license plate and use them as the identification number of the vehicle. Therefore, in the example shown in FIG. 5, the number of "Sumida-ku s1234" is extracted.

The check execution unit 50 performs a predetermined check on the object-recognized number and the like. This check is carried out in the processing of step S24, which will be described later, and the details thereof will be explained again in detail.

The abnormality signal output unit 51 outputs an abnormality signal for notifying an administrator or the like of an abnormality in the check result. This abnormality signal notifies the person in charge of management or the like of the abnormality in the check result, and can be an alert sound (auditory signal) or a warning lamp (visual signal). The output of the abnormality signal is mainly carried out in the process of step S26, which will be described later.

2. Operation of Vehicle Management System Next, an outline of the operation of the vehicle management system 1 will be described. First, the vehicle management system 1 executes a series of processes (hereinafter referred to as "pre-process" for convenience) mainly for extracting still images from moving images. The flow of this pre-processing will be described below with reference to the flowchart shown in FIG.

(1) Pre-stage processing Each smartphone 11 installed on the premises continuously captures an image of a subject, and the moving image is sent to the information processing device 14 in real time. On the other hand, the information processing device 14 continuously performs vehicle object recognition processing on this moving image. As a result, when a vehicle appears in the subject of any smartphone 11, in other words, when the vehicle enters the site and passes through the road in the subject, the information processing device 14 recognizes the vehicle as an object. (step S10). In this manner, the information processing device 14 monitors vehicles entering the site.

The shooting mode of the subject in each smartphone 11 may be variable according to conditions such as weather, time, and season. For example, in the case of backlight or a dark place, the HDR (High Dynamic Range) function of each smartphone 11 may be automatically enabled. As a result, it is possible to acquire a moving image that is as clear as possible according to the situation at that time.

When the vehicle is recognized as an object (Yes in step S10), the information processing device 14 performs subsequent processes (steps S11 to S18) on the vehicle. Note that the information processing device 14 detects when a plurality of vehicles are recognized as objects at the same time, that is, when a plurality of vehicles appear in the subject of the same smartphone 11 at the same time, or when vehicles appear in the subjects of a plurality of smartphones 11 at the same time. , all of these vehicles are recognized as objects, and the subsequent processing is performed individually for each vehicle.

First, the information processing device 14 detects the speed of the vehicle whose object has been recognized (step S11). Information on the detected vehicle speed is recorded in the management server 15 by the process of step S24, which will be described later. Furthermore, the information processing device 14 monitors the timing when the distance D between the vehicle and the smartphone 11 becomes equal to or less than a predetermined threshold (step S12).

Here, FIG. 7 illustrates how the distance D changes as the vehicle moves. As shown in this figure, the distance D is smaller when the vehicle moves to a position where the vehicle is larger and clearer than when the vehicle starts to appear on the smartphone 11 . Note that the threshold for distance is set according to the distance D when the vehicle is expected to appear moderately large. Therefore, the information processing device 14 can detect the timing at which the vehicle starts appearing appropriately large by performing the process of step S12.

When the distance D becomes equal to or less than the threshold (Yes in step S12), the information processing device 14 starts still image extraction processing (step S13). Thereafter, the information processing device 14 sequentially extracts still images from the moving image until the still image extracting process ends. By suspending the execution of the still image extraction process until the distance D becomes equal to or less than the threshold, it is possible to minimize the extraction of unclear still images in which the reflection of the vehicle is small.

Note that the condition for starting the still image extraction process may be, for example, when the direction of the vehicle satisfies a predetermined condition instead of when the distance D becomes equal to or less than the threshold as in the present embodiment. In this case, the information processing device 14 is provided with a function unit (direction detection unit) for detecting the orientation of the vehicle, and the still image extracting process is started when the detected orientation satisfies a predetermined condition. You should do it like this. The orientation of the vehicle can be recognized from the state of the vehicle in the moving image. The predetermined condition regarding the direction of the vehicle is, for example, a state in which the vehicle faces straight ahead with respect to the smartphone 11, that is, a state in which the front surface of the license plate on the front side of the vehicle faces straight toward the smartphone 11 as a reference direction. The difference from the direction should be a predetermined value or less (for example, 30° or less). In this way, the execution of the still image extraction process is suspended until the orientation of the vehicle satisfies the predetermined condition, thereby extracting a still image that has a problem with the orientation of the vehicle (which easily interferes with license plate recognition). can be minimized.

After the still image extracting process is started, the information processing device 14 detects the first exposure degree of the vehicle for the latest one frame of the moving image (step S14). The higher the first degree of exposure, the more likely that the vehicle will appear more clearly in the still image, and the more likely that the still image will be useful for recognizing the license plate, etc. included in the vehicle, the still image is more important. It can be said that the degree is high. Note that the detection of the first exposure may be performed directly from the moving image, or may be performed from the most recent extracted still image.

After that, the information processing device 14 adjusts the resolution of the extracted still image according to the detected first exposure (step S15). More specifically, the information processing device 14 increases the resolution of the extracted still image as the first exposure level increases. This makes it possible to preferentially obtain a still image with a high degree of importance, making it easier to recognize a license plate or the like from the still image. If high-resolution still images are to be constantly obtained, it is necessary to handle a large number of still images with a large data size, which tends to increase the load on the system. By adjusting the resolution, it is possible to solve such problems as much as possible.

Further, the information processing device 14 adjusts the time interval for extracting still images according to the detected first exposure (step S16). More specifically, the information processing device 14 shortens the time interval for extracting still images as the first exposure is higher, and increases the number of still images extracted per unit time. This makes it possible to preferentially obtain a still image with a high degree of importance, making it easier to recognize a license plate or the like from the still image. If the time interval for extracting still images is always shortened, it is necessary to handle a large number of still images, and the load on the system tends to increase. By adjusting the time interval, such problems can be eliminated as much as possible.

The series of processes of steps S14 to S16 described above is repeated until the vehicle is no longer recognized (that is, until the vehicle passes through the range displayed on the smartphone 11) (step S17). In this way, the resolution of the still image and the time interval for extracting the still image are feedback-controlled according to the first degree of exposure, and it is possible to efficiently extract the still image of high importance.

The timing chart shown in FIG. 8 illustrates timings at which still images are extracted from moving images obtained by a pair of smartphones 11 (smartphones that photograph the front and rear sides of the same vehicle). The colored portions in this figure indicate the timings at which still images are extracted. As shown in this figure, a still image of the front side of the vehicle is acquired during a period in which the vehicle is present in the subject of the front imaging smartphone (11a or 11c). After that, the vehicle moves, and a still image of the rear side of the vehicle is acquired during a period in which the vehicle exists within the subject of the rear-side photographing smartphone (11b or 11d). Also, as shown in FIG. 8, the higher the first exposure, the more high-resolution still images are acquired.

In this embodiment, one of the processes of steps S14 and S15 may be omitted, and the processes of steps S14 to S16 may be omitted. On the other hand, when the vehicle is no longer recognized (Yes in step S17), the still image extracting process for that vehicle ends (step S18). When the pre-processing corresponding to each of the pair of smartphones 11 is executed, still images of a plurality of frames for one vehicle are obtained. A group of still images obtained in this manner is used in a series of processes in steps S20 to S26 (hereinafter referred to as "post-stage processing" for convenience).

(2) Post-stage processing Next, the flow of post-stage processing will be described with reference to the flowchart shown in FIG. This post-stage processing is executed each time the pre-stage processing for one vehicle is completed. The information processing device 14 extracts still images satisfying the above-described exposure condition from among the plurality of still images obtained by the pre-processing in order to obtain still images used for number recognition and the like (step S20).

Here, in the present embodiment, as exposure conditions, the first condition, second condition, third condition, fourth condition, and fifth condition are set in descending order of priority as follows.
1st condition: The 2nd degree of exposure is the highest 2nd condition: The 2nd degree of exposure is the 2nd highest 3rd condition: The 2nd degree of exposure is the 3rd highest 4th condition: The 1st degree of exposure is the highest High Fifth condition: The first exposure level is the second highest. However, for the fourth and fifth conditions, a still image that satisfies any one of the first to third conditions is excluded. Further, each condition after the fifth condition may be appropriately set.

That is, when the process of step S20 is performed for the first time, the first condition is valid as the exposure condition. However, when the recognition of the number is not successful in the subsequent processing of step S23 and the processing of step S20 is performed next, the second condition is valid as the exposure condition. Similarly, the third condition is validated when the process of step S20 is performed next, the fourth condition is validated when the process of step S20 is performed next, and the next condition is validated. When the process of step S20 is performed in , the fifth condition is valid.

As described above, in this embodiment, first, a predetermined number (three in the example of this embodiment) of still images with the highest second exposure are extracted with the highest priority, and then the images with the first exposure are extracted. A predetermined number (two in the example of the present embodiment) of still images are preferentially extracted from the highest number. It should be noted that the values of these predetermined numbers are examples, and other values may be adopted. In this embodiment, since a moving vehicle is photographed in particular, a situation may occur in which the license plate is missing or not properly visible in the photographed moving image of the vehicle. Such a situation is highly likely to have a fatal adverse effect on number recognition, but by extracting the still image with the second high degree of exposure as the highest priority as in the present embodiment, such a situation can be avoided as much as possible. can be suppressed. In this embodiment, since the high first exposure also greatly affects the number recognition success rate, the still image to be extracted is determined by placing importance on the first exposure next to the second exposure. I have to. If the vehicle is photographed from only one direction, there is a risk that the license plate may not be properly captured due to backlight. Since it is possible to obtain both the video showing the image and the video showing the license plate on the rear side, such a problem can be avoided.

After performing the process of step S20, the information processing device 14 performs the above-described image processing on the extracted still image (step S21), and recognizes an object such as a license plate on the image-processed still image. is executed (step S22). As for the license plate, recognition of the number (display information) indicated on the plate is executed. Here, if the number is successfully recognized (Yes in step S23), the next step S24 is processed. If the number is not successfully recognized (No in step S23), step S20 is performed. is processed again.

When the process of step S20 is performed again, the second condition is applied instead of the first condition as described above, and the third condition is applied when the process of step S20 is performed again. . As described above, in the present embodiment, a first process of extracting the extracted still image with the highest second exposure level, a second process of recognizing the number from the extracted still image, If the recognition is not successful, a third process of extracting the still image with the next highest second exposure level and recognizing the number again is executed, until the recognition is successful. 3 processing is repeated.

Furthermore, in the present embodiment, when the recognition is not successful even after repeating the third processing a predetermined number of times, a fourth processing for extracting the still image having the highest first exposure level among the extracted still images; A fifth process of recognizing the number from the extracted still image, and if the recognition is unsuccessful, extracting the still image with the next highest first exposure level and recognizing the number again. A sixth process is executed, and the sixth process is repeated until the recognition is successful.

Next, the information processing device 14 checks the number and the like based on the object recognition result, and records the result in the management server 15 (step S24). More specifically, the information processing device 14 checks the object-recognized number against the database (numbers of all permitted vehicles) stored in the management server 15 . As a result, if it matches the number of any permitted vehicle, it is determined to be normal, otherwise it is determined to be abnormal. The information processing device 14 determines that the seat belt and accessories worn by the driver are normal if they are correctly recognized as objects (that is, if they are correctly worn or equipped), and are otherwise abnormal. . The information processing device 14 determines that the vehicle speed detected in step S11 is normal if it does not exceed a predetermined allowable upper limit speed (for example, 30 km/h), and that it is abnormal otherwise. Further, the information processing device 14 determines that the vehicle is abnormal when dirt or damage exceeding a predetermined standard is recognized as an object, and that the vehicle is normal otherwise.

Furthermore, the information processing device 14 associates these check results (discrimination results), the recognized number, the date and time of photography (current date and time), the identification number of the smartphone 11 used for photography, and the vehicle speed with each vehicle. is recorded in the management server 15. FIG. 10 exemplifies information such as check results recorded in the management server 15 . In the example shown in this figure, a management number is assigned to each vehicle, and information of each item is recorded. The management server 15 also stores moving images and still images used for object recognition, and associates vehicle information with information related to the vehicle, such as a license plate (including not only the number but also the driver and equipment). It is designed to hold This makes it possible to collectively manage vehicles, license plates, and the like. As for the driver (person), the facial expression and the like are also stored in association with each other.

After that, if there is no abnormality in any of the above check results (No in step S25), the information processing device 14 ends the post-processing. However, if there is an abnormality in any of the check results (Yes in step S25), the information processing device 14 outputs an abnormality signal (step S26), after which the post-processing ends. In the example shown in FIG. 10, the seat belt check result for control number No. 00001 is abnormal, and the vehicle speed check result for control number No. 00002 is abnormal. will be output.

When the abnormal signal is output, the person in charge of management who notices it can take appropriate measures after confirming the check result. The user of the vehicle may be informed that the check result indicates an abnormality. Further, the abnormal signal may be continuously output until the person in charge of management performs a predetermined operation. Furthermore, in this case, if the output of the abnormality signal continues for a certain period of time or more, a message (e.g., electronic mail) may be sent to notify the terminal or the like owned by the person in charge of management or the user of the abnormality.

3. Others As described above, the vehicle management system 1 includes a moving image object recognition unit 42 that recognizes a vehicle (an example of a first object) from a moving image, and a still image extracting unit that extracts a still image including the vehicle from the moving image. 45, and a still image object recognition unit 49 that recognizes a license plate or the like attached to the vehicle (an example of a second object) from the still image.

Therefore, according to the vehicle management system 1, it is easy to accurately recognize the license plate and the like while suppressing the operation load of the system. That is, although there is a concern that recognition accuracy may be lowered if an attempt is made to directly recognize a license plate or the like from a moving image, highly accurate object recognition is facilitated by recognizing from a still image. Furthermore, it is possible to prevent the still image extracting process from being performed when the moving image does not include a vehicle, thereby omitting unnecessary processing and reducing the operation load of the system.

Furthermore, the vehicle management system 1 includes an extracting unit 47 that extracts still images that satisfy an exposure condition from among the plurality of extracted still images. Recognize objects such as license plates included. Therefore, it is possible to more efficiently recognize the license plate or the like included in the vehicle from the still image.

Although the embodiments of the present invention have been described above, the configuration of the present invention is not limited to the above embodiments, and various modifications can be made without departing from the gist of the invention. The technical scope of the present invention is defined by the scope of claims rather than the description of the above embodiments, and is understood to include all modifications within the scope and meaning equivalent to the scope of claims. should.

INDUSTRIAL APPLICABILITY The present invention can be used for systems that recognize objects from moving images.

1 vehicle management system 11 smartphone 11a smartphone for photographing the front side of the approach road 11b smartphone for photographing the rear side of the approach road 11c smartphone for photographing the front side of the exit road 11d smartphone for photographing the rear side of the exit road 12 communication network 13 edge server 14 information processing device 14a video processing engine 14b still image processing engine 15 management server 40 control unit 41 communication unit 42 moving image object recognition unit 43 distance detection unit 44 speed detection unit 45 still image extraction unit 46 exposure detection unit 47 extraction unit 48 image processing unit 49 still image object recognition Unit 50 Check execution unit 51 Abnormal signal output unit

Claims (4)

an image object recognition unit that recognizes a vehicle as an object from image information including a plurality of frames;
a still image extraction unit for extracting still images of a plurality of frames including the vehicle from the image information, wherein the higher the degree of exposure of the vehicle in the image information, the shorter the time interval for extracting the still images; ,
a still image object recognition unit that recognizes a license plate associated with the vehicle from the still image,
A front side photographing camera for photographing the front side of the same vehicle and a rear side photographing camera for photographing the rear side are provided,
The image information includes both a front image taken using the front camera and a rear image taken using the rear camera,
The front side photographing camera and the rear side photographing camera are arranged at positions overlapping with the vehicle in an upward view, and the front side photographing camera is arranged so as to photograph the entire front side of the vehicle from an obliquely upper forward direction, The image recognition system , wherein the camera for photographing the rear side is arranged so as to photograph the entire rear side of the vehicle from an obliquely upper rear direction . The still image object recognition unit
The license plate on the front side of the vehicle is object-recognized from the still image extracted from the front-side image, and the license plate on the rear side of the vehicle is object-recognized from the still image extracted from the rear-side image. 2. The image recognition system according to claim 1, characterized by: an extraction unit that extracts, from among the extracted still images, those that satisfy a predetermined condition regarding the degree of exposure of at least one of the vehicle and the license plate;
3. The image recognition system according to claim 1, wherein the still image object recognition unit recognizes the license plate included in the vehicle from the extracted still image. a first process of extracting the one with the highest degree of exposure of the specific object, which is the license plate, from among the extracted still images;
a second process for recognizing display information in the license plate from the extracted still image;
a third process of extracting the still image with the next highest degree of exposure of the specific object and repeating the second process until the recognition is successful;
4. The method according to claim 3, wherein when the recognition in the second process is not successful even if the number of repetitions reaches a predetermined number, the first process to the third process are executed with the vehicle as the specific object. The object recognition system described.

Images