JP2021128796A - Object recognition system and object recognition method - Google Patents

Abstract

To provide an object recognition system that facilitates recognition of a second object accompanied with a first object in a moving image with high accuracy while suppressing a working burden of a system.SOLUTION: An object recognition system comprises a moving image object recognition part conducting object recognition of a first object from a moving image, a still picture extraction part extracting a still picture having multiple frames including the first object from the moving image, and a still picture object recognition part conducting object recognition of a second object accompanied with the first object from the still picture.SELECTED DRAWING: Figure 6

Description

The present invention relates to an object recognition system and an object recognition method that perform processing such as object recognition.

Conventionally, an object recognition method for recognizing an object from an image has been proposed. The object recognition method can be suitably used in a system that mainly handles captured images, and can be useful for automation and efficiency improvement of various processes.

As an example, according to Patent Document 1, an object recognition device that detects a target around a vehicle and recognizes an object based on the detected target is disclosed. Further, according to Patent Document 2, a target area in which the shape of the recognition object exists is extracted from the image captured by the image pickup device provided in the vehicle, and the extracted target area is obtained from the entire area of the image. An object recognition device that selectively executes recognition processing of a recognition object is disclosed.

As specific means for performing object recognition, YOLO, which is suitable for object recognition from moving images, TENSORFLOW (registered trademark, the same applies hereinafter) and the like, which are suitable for object recognition from still images, have been developed. Artificial Intelligence (AI) technology is applied to these means, and it is possible to improve the accuracy of object recognition by machine learning. Further, in machine learning, there is a tendency that advanced deep learning is adopted, and the accuracy and speed of object recognition are improving.

JP-A-2018-041396 JP-A-2017-130155

By the way, when recognizing another object (second object) attached to one object (first object) in the moving image, the second object is directly and accurately recognized from the moving image. Is not easy in some cases. The "object" in the present application is a concept corresponding to annotation data in machine learning, object recognition, and the like. For example, when trying to recognize the number displayed on the license plate (an example of the second object) from the video shot by the camera that monitors the vehicle (an example of the first object), the vehicle must be recognized in order to recognize the number. More precise recognition processing is required than object recognition. Therefore, if an attempt is made to recognize the number directly from the moving image, the recognition accuracy may be lowered.

In order to solve this problem, a method of extracting a still image from a moving image and recognizing a number by recognizing an object from the extracted still image can be considered. However, for example, if the still images of each frame are uniformly extracted from the moving image, unnecessary still images that do not include the vehicle are also extracted, and the operating load of the system may become excessive. Further, if the extracted still image contains a large number of such unnecessary still images, the speed and accuracy of object recognition may decrease.

In view of the above-mentioned problems, the present invention is an object recognition system and an object recognition method that facilitates accurate recognition of a second object associated with a first object in a moving image while suppressing the operating load of the system. The purpose is to provide.

The object recognition system according to the present invention includes a moving object recognition unit that recognizes a first object from a moving image, a still image extraction unit that extracts a still image of a plurality of frames including the first object from the moving image, and the above. The configuration includes a still image object recognition unit that recognizes a second object attached to the first object from a still image. According to this configuration, it becomes easy to accurately recognize the second object attached to the first object in the moving image while suppressing the operation load of the system. The term "accompanying" here is not limited to the form in which the first object includes the second object, but also includes other forms closely related to the time when it appears in the moving image.

More specifically, as the above configuration, an extraction unit that extracts, among the extracted still images, those that satisfy a predetermined condition regarding the high degree of exposure of at least one of the first object and the second object. The still image object recognition unit may be configured to recognize the second object included in the first object from the extracted still image. According to this configuration, the second object included in the first object can be recognized more efficiently from the still image.

More specifically, as the above configuration, the first process of extracting the extracted still image having the highest degree of exposure of the specific object, which is the second object, and the extracted still image. The second process of recognizing the display information in the second object from the image and the extraction of the still image having the next highest degree of exposure of the specific object from the still images until the recognition is successful are extracted. The configuration may be such that the third process, in which the two processes are repeated, and the third process are executed. The "display information" here is a character, a figure, a symbol, or a combination thereof, and may correspond to, for example, a number (automobile registration number) in the present application.

More specifically, as the above configuration, when the recognition in the second process is not successful even if the number of repetitions reaches a predetermined number, the specific object is set as the first object and the first to third processes are performed. May be configured to execute.

More specifically, as the above configuration, the still image extraction unit sets at least one of the resolution of the still image to be extracted and the time interval for extracting the still image according to the degree of exposure of the first object in the moving image. It may be configured to be adjusted.

According to this configuration, it is possible to efficiently extract highly important still images while suppressing the burden on the system as much as possible. Further, more specifically, the configuration may be such that a correction process suitable for deep learning is applied to the one having the highest degree of exposure of the second object among the plurality of extracted still images.

More specifically, as the above configuration, the moving image is an image taken by a camera, and includes a distance detecting unit for detecting the distance between the camera and the first object, and the still image extraction unit is provided. The still image may be extracted when the distance becomes equal to or less than a predetermined value. According to this configuration, it is possible to suppress extraction of an unclear still image in which the reflection of the first object is small as much as possible.

More specifically, the configuration includes a direction detection unit that detects the orientation of the first object, and the still image extraction unit extracts the still image when the orientation satisfies a predetermined condition. It may be a configuration to start. According to this configuration, it is possible to suppress extraction of a still image having a problem in the orientation of the first object as much as possible.

Further, more specifically, the above configuration may be a configuration in which the information of the first object and the information of the second object are held in association with each other. According to this configuration, it is possible to collectively manage the first object and the second object. Further, in the above configuration, the moving object recognition unit and the still image object recognition unit may use different object recognition methods. According to this configuration, it is possible to efficiently perform object recognition by using the optimum methods for object recognition from moving images and object recognition from still images.

Further, the object recognition method according to the present invention includes a moving object recognition step of recognizing a first object from a moving image, a still image extraction step of extracting a still image including the first object from the moving image, and a still image. The method includes a still image object recognition step of recognizing a second object attached to the first object.

According to the object recognition system and the object recognition method according to the present invention, it becomes easy to accurately recognize the second object attached to the first object in the moving image while suppressing the operation load of the system.

It is a block diagram concerning the structure of the vehicle management system 1 which concerns on this embodiment. It is explanatory drawing of the appearance that the smartphone 11 was installed in the site. It is a block diagram concerning the functional configuration of the information processing apparatus 14. It is explanatory drawing about the image of the smartphone 11. It is explanatory drawing about a local license plate. It is a flowchart about the flow of the pre-stage processing. It is explanatory drawing of the appearance that the distance D changes with the movement of a vehicle. It is a timing chart regarding the extraction of a still image from a moving image. It is a flowchart about the flow of post-stage processing. It is explanatory drawing about information such as a check result.

The vehicle management system according to the embodiment of the present invention (one form of the object recognition system according to the present invention) will be described below with reference to each drawing.

1. 1. Configuration of Vehicle Management System FIG. 1 is a block diagram showing a schematic configuration of vehicle management system 1 according to the present embodiment. As shown in this figure, the vehicle management system 1 includes a smartphone 11a for shooting on the front side of the approach road, a smartphone 11b for shooting on the rear side of the approach road, a smartphone 11c for shooting on the front side of the exit road, a smartphone 11d for shooting on the rear side of the exit road, and a communication network 12. It includes an edge server 13, an information processing device 14, and a management server 15. In the following description, the above smartphones 11a to 11d may be collectively referred to as "smartphone 11". Further, in each drawing, a smartphone (SmartPhone) may be abbreviated as "SP".

In this embodiment, as an example, a plurality of smartphones 11 are installed in each of a plurality of sites (sites 1 to 3 in the example shown in FIG. 1). The "site" in the present embodiment is a place where vehicles obtained with the permission of the business operator or the like (hereinafter referred to as "manager") who manages the site can enter and exit, for example, a parking lot owned by the manager or the like. Is applicable. When there are a plurality of vehicle entrances and exits on the site, by installing smartphones 11 near all the entrances and exits, it is possible to monitor vehicles entering or leaving the site without omission. The "vehicle" in the present embodiment is a vehicle provided with a license plate, and the "number" is a vehicle registration number represented on the license plate. Generally, license plates are provided on both the front and rear sides of the vehicle.

On the other hand, the edge server 13, the information processing device 14, and the management server 15 are collectively installed in the management center. The "management center" in the present embodiment is a place where vehicles entering and exiting the site are managed, and corresponds to, for example, a room in a building owned by the manager. The vehicle management system 1 plays a role of automatically monitoring vehicles entering each site and collectively managing them.

The management server 15 is a server used for managing vehicles and the like entering each site. In the management server 15, the numbers of all vehicles permitted to enter the site (hereinafter, referred to as "permitted vehicles" for convenience) are registered as a database. The management server 15 stores this information in the database every time a new permitted vehicle number is input by the administrator or the like. The management server 15 may be provided on the data center via the Internet network.

The smartphone 11 has a camera function of capturing a subject and obtaining a moving image (video), and also has a function of measuring the distance from the own device to the subject (distance measuring function). This distance measuring function is realized by a "stereo camera" in which a plurality of lenses are mounted on the smartphone 11. The distance measuring function may be realized by providing a distance measuring sensor or the like instead of the stereo camera. In addition, the smartphone 11a for photographing the front side of the approach road has a role of photographing the front side of the vehicle entering the site, and the smartphone 11b for photographing the rear side of the approach road has a role of photographing the rear side of the vehicle entering the site. The smartphone 11c for shooting has a role of photographing the front side of the vehicle leaving the site, and the smartphone 11d for photographing the rear side of the exit road has a role of photographing the rear side of the vehicle leaving the site. A camera alone or other device having a camera may be applied instead of the smartphone 11.

FIG. 2 illustrates a state in which the smartphone 11 is installed on the premises. As shown in this figure, the smartphone 11 is installed so that the passage of vehicles entering and exiting the site is the subject. As a result, when the vehicle passes through the approach road (passage for entering the site) and the exit route (passage for exiting the site), the appearance of the front side and the rear side of the vehicle is used as the subject of the smartphone 11. It is possible to fit it. The smartphone 11 includes a vehicle license plate (or information on the number represented on the license plate), a driver, a seatbelt worn by the driver, dirt and scratches (including dents, etc.) on the vehicle, and predetermined equipment (hereinafter, dents, etc.). , These are sometimes collectively referred to as "license plates, etc."), and it is desirable that they be installed at appropriate positions so that they can be accommodated in the subject. The "equipment" here is, for example, one that is obliged to be equipped in a licensed vehicle, and can be a subject of the smartphone 11.

For example, as shown in FIG. 2, it is desirable that each smartphone 11 is provided near the guard room and is installed at a position where the entire vehicle can be photographed from diagonally above (post in the example of this figure). In the example of this figure, the smartphone 11a for shooting on the front side of the approach road and the smartphone 11b for shooting on the rear side of the approach road are installed so that the rear portions face each other almost directly above the approach road, and the smartphone 11a for shooting on the front side of the approach road enters. The smartphone 11b for shooting the rear side of the approach road is appropriately arranged so as to shoot the entire front side of the vehicle traveling on the road from diagonally above and front, and the entire rear side of the vehicle traveling on the approach road is photographed diagonally from above and behind. ing.

On the other hand, the smartphone 11c for shooting on the front side of the exit road and the smartphone 11d for shooting on the rear side of the exit road are installed so that the rear parts face each other almost directly above the exit road, and the smartphone 11c for shooting on the front side of the exit road is a vehicle traveling on the exit road. The smartphone 11d for photographing the rear side of the exit road is appropriately arranged so as to photograph the entire rear side of the vehicle traveling on the exit road from diagonally above and rear.

As described above, it is desirable that each smartphone 11 is arranged at a position overlapping the vehicle in upward view, which makes it possible to simplify or omit various corrections in the vehicle width direction for the image data obtained by the smartphone 11. .. In the following description, the combination of the smartphone 11 that shoots the same vehicle from the front side and the rear side, that is, the combination of the smartphone 11a for shooting the front side of the approach road and the corresponding smartphone 11b for shooting the rear side of the approach road, and the exit route. Each combination of the front side shooting smartphone 11c and the corresponding exit road rear side shooting smartphone 11d may be expressed as "a pair of smartphones 11".

For example, in the image of the smartphone 11c for shooting on the front side of the exit road, as illustrated in FIG. 4, the license plate on the front side of the vehicle C1 is directly reflected, and the driver and the seat belt worn by the driver are reflected through the windshield. become. The above-mentioned license plates and the like are included in the vehicle in both the moving image and the still image showing the vehicle, and are attached to the vehicle. Further, since the smartphone 11 has a distance measuring function, if the position of the vehicle in the subject is specified, it is possible to obtain information on the distance D from the smartphone 11 to the vehicle. This distance D is detected by the distance detection unit 43, which will be described later. In addition, the smartphone 11 may be provided in the guard room in order to suppress the temperature rise / fall and aging deterioration of the smartphone 11 and to photograph the side surface of the vehicle more accurately. By collating the data obtained from the smartphones (11a, 11b) that photograph the approaching vehicle and the smartphones (11c, 11d) that photograph the exiting vehicle, entry / exit management becomes possible.

The communication network 12 is a network used for communication between each smartphone 11 and the information processing device 14. As a specific form of the communication network 12, either a wired network or a wireless network may be applied. It is also possible to use the Internet or the like for the communication network 12.

The edge server 13 is interposed between the communication network 12 and the information processing device 14, and for example, an environment in which deep learning can be executed and various values used in deep learning (hyperparameters learned by artificial intelligence, structure information of a model). The hyper-parameters that become, the weight data given when the training data is trained, and the reward function in the reinforcement learning model) are stored. Software (Python, anaconda, jupyter, opencv, TENSORFLOW, YOLO, etc.) that can execute deep learning is installed on the edge server 13.

The information processing device 14 is composed of a server having higher performance than the edge server 13, and includes object recognition from moving images and still images, as well as new learning (reinforcement learning, additional learning) of deep learning related to vehicle monitoring and management. It is a device that executes related processing. Further, the information processing device 14 includes a moving image processing engine 14a that performs processing such as image recognition for moving images, and a still image processing engine 14b that performs processing such as image recognition for still images.

The video processing engine 14a employs algorithms such as YOLO (You Only Look Once) and OPEN CV (Open Source Computer Vision Library), and is excellent in the function of recognizing an object from a video in real time. The moving image processing engine 14a is capable of accurately and quickly recognizing a "vehicle" for any vehicle having a different appearance (tilt, size, orientation) by machine learning. As a result, it is possible to suppress the omission of recognition of the vehicle in the moving image as much as possible. "Machine learning" is a method of autonomously finding laws and rules by iteratively learning based on given information. However, the specific configuration of the moving image processing engine 14a is not limited to the above example, and other means suitable for object recognition from moving images may be adopted instead of YOLO or the like.

On the other hand, the still image processing engine 14b employs TENSORFLOW, which is a machine learning library, and is excellent in the function of quickly and accurately recognizing an object from a still image. In particular, TENSORFLOW is a library capable of deep learning, and can smoothly process multidimensional data structures. "Deep learning" is machine learning using a multi-layered neural network (information processing model that imitates the mechanism of the human cranial nerve system).

According to the still image processing engine 14b, it is possible to recognize an object such as a license plate with high accuracy from a still image including a vehicle, and it is also possible to recognize a number displayed on the license plate. However, the specific configuration of the still image processing engine 14b is not limited to the above example, and other means suitable for object recognition from a still image may be adopted instead of TENSORFLOW. Further, the edge server 13 and the information processing device 14 may be realized by the same server.

Here, a block diagram of a main functional configuration of the information processing apparatus 14 is shown in FIG. As shown in this figure, the information processing device 14 includes a control unit 40, a communication unit 41, a moving object recognition unit 42, a distance detection unit 43, a speed detection unit 44, a still image extraction unit 45, an exposure degree detection unit 46, and an extraction. It includes a unit 47, an image processing unit 48, a still image object recognition unit 49, a check execution unit 50, and an abnormal signal output unit 51.

The control unit 40 appropriately controls each of the functional units 41 to 51 so that the information processing device 14 operates normally. The main operation of the information processing device 14 will be described in detail again. The communication unit 41 executes communication with an external device including each smartphone 11 and the management server 15.

The moving object recognition unit 42 recognizes the vehicle from the moving image by using the object recognition function from the moving image by the moving image processing engine 14a. For moving images in which a plurality of vehicles appear at the same time, the moving image object recognition unit 42 can recognize the objects separately. For example, when two vehicles enter the subject of one smartphone 11, it is possible to recognize these two vehicles separately, and the information processing device 14 advances the processing focusing on each in parallel. It is possible. The object recognition is mainly carried out in the process of step S10 described later.

The distance detection unit 43 detects the distance D (see FIG. 2) between the object-recognized vehicle and the smartphone 11 by using the distance measurement function provided in the smartphone 11. The detection of the distance is mainly carried out in the process of step S12 described later.

The speed detection unit 44 detects the speed of the vehicle recognized as an object. As a method for detecting the speed, a method using a speed sensor installed in the vicinity of the smartphone 11, a method for detecting the speed from the movement of the vehicle in the moving image, and the like can be adopted. The detection of the speed is mainly carried out in the process of step S11 described later.

The still image extraction unit 45 extracts a plurality of still images (images of each frame, for example, 30 images at intervals of 0.1 seconds) from the moving image and temporarily holds them in the storage area. The still image extraction unit 45 can appropriately change the resolution of the still image extracted from the moving image and the time interval for extracting the still image from the moving image. Extraction of the still image is mainly carried out in the process of step S13 described later.

The exposure degree detection unit 46 detects the exposure degree (hereinafter, referred to as “first exposure degree”) of the vehicle recognized as an object in one frame (corresponding to one still image) of the moving image. The first degree of exposure may be the ratio of the size (area) of the vehicle to the size of the frame (exposure ratio to the frame) in one frame of the moving image, or may be the size of the vehicle itself. Further, as the information on the size of the vehicle, the area inside the contour of the vehicle may be adopted, and other information as an index of the size (for example, a rectangle shown by a broken line in FIG. 4 (a rectangle whose four sides are in contact with the vehicle)). ) Internal area) may be adopted. In addition, another value that enables comparison of the high degree of exposure of the vehicle between the frames may be regarded as the first degree of exposure. The detection of the first degree of exposure is mainly carried out in the process of step S14, which will be described later. Further, the exposure degree detection unit 46 detects the exposure degree (hereinafter, referred to as “second exposure degree”) of the license plate recognized as an object in one frame (corresponding to one still image) of the moving image. The second degree of exposure may be the ratio of the size of the license plate to the size of the vehicle (exposure ratio to the vehicle) in one frame of the moving image, or as the ratio of the size of the license plate to the size of the frame (exposure ratio to the frame). It may be the size of the license plate itself. Further, as in the case of the first exposure degree, various values that make it possible to compare the heights of the exposure degree of the license plates between the frames can be regarded as the second exposure degree. The detection of the second degree of exposure is mainly carried out in the process of step S20 described later.

The extraction unit 47 refers to a predetermined condition regarding at least one of the first exposure degree and the second exposure degree among the plurality of still images extracted by the still image extraction unit 45 (hereinafter, for convenience, “exposure degree condition”. Extract those that satisfy (referred to). The exposure degree condition may be, for example, that the value of the degree of exposure is equal to or higher than a predetermined value, that the degree of exposure may be the highest, and that the value falls within a predetermined number counting from the one with the highest degree of exposure. It may be. The specific contents of the exposure condition according to the present embodiment will be described in detail again. The higher the degree of exposure, the higher the possibility that the still image object recognition unit 49 can easily recognize the numbered object. Therefore, if a still image satisfying the degree of exposure is used, the object recognition can be performed more advantageously. .. In addition, instead of the exposure degree condition, another condition regarding the ease of performing object recognition by the still image object recognition unit 49 may be set. The extraction is mainly carried out in the process of step S20 described later.

The image processing unit 48 sequentially performs each image processing of the threshold value processing, the edge processing (edge detection processing), and the tilt correction processing on the still image. The image processing unit 48 may perform only one or two of these image processes, and further performs other image processes for making object recognition from a still image more advantageous. It may be carried out. The image processing is mainly carried out in the process of step S21 described later. Note that these image processes can also be regarded as correction processes suitable for deep learning.

Here, the "threshold processing" is a processing for converting an image into a binary image (single channel image). According to the threshold processing, for example, when converting to a black-and-white binary image, pixels whose channel value exceeds a predetermined threshold are white pixels, and pixels whose channel value does not exceed the threshold are black. It will be converted to each pixel. For images that have been subjected to threshold processing, it becomes easy to select parts having different brightness in the image.

Further, the "edge processing" is a processing for detecting a portion (edge) in the image where the brightness (shading) or the color suddenly changes. Since the contours and lines of an object in an image generally have abrupt changes in shading and the like, it is possible to detect the contours and lines by edge processing. Edges are important information that reflects the structure of an object, and edge processing is extremely useful when performing object recognition from a still image. In order to perform the edge processing more effectively, it is usually useful to perform the threshold processing on the image in advance.

A Canny edge detector may be adopted as an algorithm for performing edge processing. According to the edge processing (canny processing) when this algorithm is adopted, there are less omissions and false detections of contours compared to the case where other algorithms (Sobel filter, Laplacian filter, etc.) are adopted, and each point. It has the feature that it is easy to detect a single contour and to detect a part that is a true edge. Canny processing is achieved by smoothing the image with a Gaussian filter, calculating the magnitude and direction of the gradient from the calculation result of the derivative of the smoothed image, and performing Non maximum Suppression processing and Hysteresis Threshold processing. ..

Further, the "tilt correction process" is a process of rotating an image so as to eliminate the inclination when a straight line or the like detected in the image is inclined from the horizontal direction (or the vertical direction). For example, by performing tilt correction processing so that the laterally extending edges of the license plate in the image match in the horizontal direction, the character strings of the numbers are lined up in the horizontal direction, making it easier to recognize the numbers. It becomes possible to do. It is usually useful to perform edge processing in advance in order to facilitate detection of straight lines and the like in the image.

The still image object recognition unit 49 uses the object recognition function from the still image by the still image processing engine 14b to obtain the license plate, number (information displayed on the license plate) of the vehicle from the still image, and the seat worn by the driver. Object recognition of belts, vehicle dirt / scratches, and equipment. Object recognition from a still image is a concept that includes recognizing display information displayed on a still image, as in the case of recognizing a vehicle number. The object recognition is mainly carried out in the process of step S22 described later, but in order to detect the second exposure degree described above, the object recognition of the license plate is carried out in the process of step S20. As described above, in the present embodiment, different object recognition methods are used in the moving image object recognition unit 42 and the still image object recognition unit 49. Therefore, compared to the case where the same object recognition method is used in both cases, it is possible to efficiently perform object recognition by using the optimum methods for object recognition from moving images and object recognition from still images. ..

The still image object recognition unit 49 can recognize the number not only from a general license plate without a pattern but also from a license plate on which a so-called local number is displayed. FIG. 5 shows an example of a license plate that can be recognized and represents a local number (local license plate). On the license plate, a design (here, a wave design) is drawn in addition to the numbers and symbols that identify the vehicle. Even for such a license plate, the still image object recognition unit 49 can extract only the characters and symbols registered as numbers by the primitive shape determination and use them as the identification number of the vehicle. Therefore, in the example shown in FIG. 5, the number of "Sumida-ku s1234" is extracted.

The check execution unit 50 checks the predetermined contents of the object-recognized number and the like. The check is carried out in the process of step S24 described later, and the contents thereof will be described in detail again.

The abnormality signal output unit 51 outputs an abnormality signal for notifying the administrator or the like of the abnormality of the check result. This abnormal signal informs the person in charge of management or the like that the check result is abnormal, and can be used as an alert sound (auditory signal), a warning lamp (visual signal), or the like. The output of the abnormal signal is mainly carried out in the process of step S26 described later.

2. Operation of the vehicle management system Next, an outline of the operation of the vehicle management system 1 will be described. First, the vehicle management system 1 mainly executes a series of processes for extracting a still image from a moving image (hereinafter, referred to as "pre-stage processing" for convenience). Hereinafter, the flow of this pre-stage processing will be described with reference to the flowchart shown in FIG.

(1) Pre-stage processing Each smartphone 11 installed on the premises continuously shoots a subject, and the moving image is sent to the information processing device 14 in real time. On the other hand, the information processing device 14 continuously performs the object recognition process of the vehicle for this moving image. As a result, when a vehicle appears on the subject of any smartphone 11, in other words, when the vehicle enters the site and passes through the passage in the subject, the information processing device 14 recognizes the vehicle as an object. Can be done (step S10). In this way, the information processing device 14 monitors the vehicle entering the site.

The shooting mode of the subject in each smartphone 11 may be variable depending on the conditions such as weather, time, and season. For example, in the case of backlight or a dark place, the HDR (High Dynamic Range) function of each smartphone 11 may be automatically enabled. As a result, it is possible to acquire as clear a moving image as possible according to the situation at that time.

When the vehicle is recognized as an object (Yes in step S10), the information processing device 14 performs the subsequent processes (steps S11 to S18) on the vehicle. The information processing device 14 recognizes an object at the same time when a plurality of vehicles are recognized at the same time, that is, when a plurality of vehicles appear on the subject of the same smartphone 11 at the same time or when the vehicles appear on the subjects of the plurality of smartphones 11 at the same time. All of these vehicles are recognized as objects, and the subsequent processing is individually performed for each vehicle.

First, the information processing device 14 detects the speed of the vehicle recognized as an object (step S11). The detected vehicle speed information is recorded in the management server 15 by the process of step S24 described later. Further, the information processing device 14 monitors the timing when the distance D between the vehicle and the smartphone 11 becomes equal to or less than a predetermined threshold value (step S12).

Here, FIG. 7 illustrates how the distance D changes with the movement of the vehicle. As shown in this figure, the distance D is smaller when the vehicle advances to a position where the vehicle is larger and clearly reflected than when the vehicle starts to be reflected on the smartphone 11. The distance threshold is set according to the distance D when the vehicle is expected to appear reasonably large. Therefore, the information processing device 14 can detect the timing when the vehicle starts to appear appropriately large by performing the process of step S12.

When the distance D becomes equal to or less than the threshold value (Yes in step S12), the information processing apparatus 14 starts the still image extraction process (step S13). After that, the information processing device 14 sequentially extracts the still image from the moving image until the still image extraction process is completed. By suspending the execution of the still image extraction process until the distance D becomes equal to or less than the threshold value, it is possible to suppress the extraction of an unclear still image with a small image of the vehicle as much as possible.

The condition for starting the still image extraction process may be, for example, when the direction of the vehicle satisfies a predetermined condition, instead of the case where the distance D becomes equal to or less than the threshold value as in the present embodiment. In this case, the information processing device 14 is provided with a function unit (direction detection unit) for detecting the direction of the vehicle, and the still image extraction process is started when the detected direction satisfies a predetermined condition. You can do it. The orientation of the vehicle can be recognized from the state of the vehicle in the moving image. The predetermined condition regarding the orientation of the vehicle is, for example, the orientation and reference of the vehicle with the state in which the vehicle is facing directly in front of the smartphone 11, that is, the state in which the front of the license plate on the front side of the vehicle is directly facing the smartphone 11. It suffices that the difference from the direction is not more than a predetermined value (for example, 30 ° or less). In this way, the execution of the still image extraction process is suspended until the orientation of the vehicle satisfies the predetermined condition, so that there is a problem in the orientation of the vehicle (prone to hinder the recognition of the number). Can be suppressed as much as possible.

After the still image extraction process is started, the information processing device 14 detects the first exposure degree of the vehicle for the latest one frame of the moving image (step S14). The higher the first exposure, the more clearly the vehicle is likely to appear in the still image, and the more likely it is to be useful for recognizing the license plate etc. contained in the vehicle, so the still image is more important. It can be said that the degree is high. The first exposure may be detected directly from the moving image or from the latest extracted still image.

After that, the information processing device 14 adjusts the resolution of the extracted still image according to the detected first exposure degree (step S15). More specifically, the information processing apparatus 14 increases the resolution of the extracted still image as the first exposure degree increases. As a result, it is possible to preferentially obtain a still image having a high degree of importance, and it is possible to make it easier to recognize the license plate or the like from the still image. If a still image having a high resolution is to be obtained at all times, it is necessary to handle a large amount of still images having a large data size, which tends to increase the load on the system. By adjusting the resolution, it is possible to solve such a problem as much as possible.

Further, the information processing apparatus 14 adjusts the time interval for extracting the still image according to the detected first exposure degree (step S16). More specifically, the information processing apparatus 14 shortens the time interval for extracting still images as the first exposure degree increases, and increases the number of still images extracted per unit time. As a result, it is possible to preferentially obtain a still image having a high degree of importance, and it is possible to make it easier to recognize the license plate or the like from the still image. If the time interval for extracting still images is always shortened, it is necessary to handle a large number of still images, which tends to increase the load on the system. However, as in the present embodiment, depending on the degree of exposure of the vehicle. By adjusting the time interval, it is possible to solve such a problem as much as possible.

The series of processes of steps S14 to S16 described above are repeatedly performed until the vehicle is no longer recognized (that is, until the vehicle passes the range reflected on the smartphone 11) (step S17). In this way, the resolution of the still image and the time interval for extracting the still image are feedback-controlled according to the first exposure degree, and it is possible to efficiently extract the still image having high importance.

The timing chart shown in FIG. 8 exemplifies the timing at which a still image is extracted from a moving image obtained by a pair of smartphones 11 (each smartphone that captures the front side and the rear side of the same vehicle). The colored parts in this figure indicate the timing at which the still image is extracted. As shown in this figure, during the period when the vehicle is in the subject of the front side shooting smartphone (11a or 11c), the still image of the front side of the vehicle is acquired. After that, the vehicle moves, and during the period in which the vehicle exists in the subject of the rear side shooting smartphone (11b or 11d), a still image of the rear side of the vehicle is acquired. Further, as shown in FIG. 8, the higher the first exposure degree, the more high-resolution still images are acquired.

In this embodiment, one of the processes of steps S14 and S15 may be omitted, or the processes of steps S14 to S16 may be omitted. On the other hand, when the vehicle is no longer recognized (Yes in step S17), the still image extraction process for the vehicle ends (step S18). When the pre-stage processing corresponding to each of the pair of smartphones 11 is executed, still images of a plurality of frames for one vehicle can be obtained. The still image group thus obtained is used for a series of processes (hereinafter, referred to as "post-stage processing" for convenience) in steps S20 to S26 described later.

(2) Post-stage processing Next, the flow of post-stage processing will be described with reference to the flowchart shown in FIG. This post-stage processing is executed every time the pre-stage processing for one vehicle is completed. In order to obtain a still image used for number recognition or the like, the information processing device 14 extracts a plurality of still images obtained by the pre-stage processing that satisfy the above-mentioned exposure degree condition (step S20).

Here, in the present embodiment, as the exposure degree condition, the first condition, the second condition, the third condition, the fourth condition, and the fifth condition are set in order from the highest priority as follows.
1st condition: 2nd exposure is the highest 2nd condition: 2nd exposure is the 2nd highest 3rd condition: 2nd exposure is the 3rd highest 4th condition: 1st exposure is the highest High 5th condition: 1st exposure is the 2nd highest However, for the 4th and 5th conditions, still images satisfying any of the 1st to 3rd conditions are excluded. Further, each condition after the fifth condition may be set as appropriate.

That is, when the process of step S20 is performed for the first time, the first condition is valid as the exposure degree condition. However, when the number recognition is not successful in the subsequent processing in step S23 and the processing in step S20 is performed next time, the second condition is valid as the exposure degree condition. Similarly, the third condition is valid when the process of step S20 is performed next, and the fourth condition is valid when the process of step S20 is performed next. The fifth condition is valid when the process of step S20 is performed.

As described above, in the present embodiment, a predetermined number (three in the example of the present embodiment) of still images are first extracted from the one with the highest second exposure degree with the highest priority, and then the first exposure degree is obtained. A predetermined number (two in the example of this embodiment) of still images are preferentially extracted from the highest one. It should be noted that these predetermined numbers of values are examples, and other values may be adopted. In this embodiment, since a moving vehicle is photographed in particular, a situation may occur in which the license plate is missing or does not look appropriate in the photographed moving image of the vehicle. Such a situation is likely to have a fatal adverse effect on the recognition of the number, but by extracting the still image with a high second degree of exposure with the highest priority as in the present embodiment, such a situation can be eliminated as much as possible. It is possible to suppress it. Further, in the present embodiment, since the high degree of first exposure also greatly affects the success rate of number recognition, the high degree of first exposure should be emphasized next to the second degree of exposure to determine the still image to be extracted. I have to. If the vehicle is photographed from only one direction, the license plate may not be properly reflected due to backlight. However, in the present embodiment, the vehicle is photographed from the front side and the rear side using a pair of smartphones, and the license plate on the front side is photographed. Since it is possible to obtain both a video showing the image and a video showing the license plate on the rear side, such a problem can be avoided.

After performing the processing in step S20, the information processing apparatus 14 performs the image processing described above on the extracted still image (step S21), and recognizes an object such as a number plate on the image-processed still image. Is executed (step S22). Regarding the license plate, the recognition of the number (display information) represented on the license plate is executed. Here, if the number recognition is successful (Yes in step S23), the processing of the next step S24 is performed, but if the number recognition is not successful (No in step S23), step S20. Processing is performed again.

When the process of step S20 is performed again, the second condition is applied instead of the first condition as described above, and when the process of step S20 is performed again, the third condition is applied. .. As described above, in the present embodiment, the first process of extracting the second most exposed still image from the extracted still images and the second process of recognizing the number from the extracted still images are If the recognition is not successful, the third process of extracting the second highest degree of exposure from each of the still images and recognizing the number again is executed, and the second process is performed until the recognition is successful. 3 The processing is repeated.

Further, in the present embodiment, when the recognition is not successful even if the third process is repeated a predetermined number of times, the fourth process for extracting the one with the highest degree of first exposure from each of the extracted still images. The fifth process of recognizing the number from the extracted still image, and when the recognition is not successful, the one with the next highest degree of exposure is extracted from each of the still images and the number is recognized again. The sixth process to be performed and the sixth process are executed, and the sixth process is repeated until the recognition is successful.

Next, the information processing device 14 checks the number and the like based on the result of the object recognition, and records the result in the management server 15 (step S24). More specifically, the information processing apparatus 14 executes collation with the database (numbers of all permitted vehicles) stored in the management server 15 for the number recognized as an object. As a result, if it matches the number of any of the permitted vehicles, it is determined to be normal, and if not, it is determined to be abnormal. Further, the information processing device 14 determines that the seatbelt and the equipment worn by the driver are normal if the object is correctly recognized (that is, if the seatbelt and the equipment are correctly worn or equipped), and otherwise abnormal. .. Further, the information processing device 14 determines that the vehicle speed detected in the process of step S11 is normal if it does not exceed a predetermined allowable upper limit speed (for example, 30 km / h), and is abnormal otherwise. Further, the information processing device 14 determines that the dirt or scratches on the vehicle are abnormal when the dirt or scratches exceeding a predetermined standard are recognized as an object, and normal if not.

Further, the information processing device 14 associates these check results (results of determination), recognized numbers, shooting date / time (current date / time), identification number of the smartphone 11 used for shooting, and vehicle speed for each vehicle. Is recorded on the management server 15. FIG. 10 illustrates information such as a check result recorded on the management server 15. In the example shown in this figure, a control number is assigned to each vehicle, and information on each item is recorded. The management server 15 also stores videos and still images used for object recognition, and associates vehicle information with information such as license plates (including drivers and equipment in addition to numbers) related to the vehicle. It is designed to hold. This makes it possible to manage the vehicle, license plate, etc. collectively. In addition, the facial expressions of the driver (person) are also linked and held.

After that, if there is no abnormality in any of the above check results (No in step S25), the information processing apparatus 14 ends the post-stage processing. However, if any of the check results is abnormal (Yes in step S25), the information processing apparatus 14 outputs an abnormal signal (step S26), after which the subsequent processing ends. In the example shown in FIG. 10, since the seatbelt check result is abnormal for control number No.00001 and the vehicle speed check result is abnormal for control number No.00002, an abnormal signal is given to these results. Will be output.

When an abnormal signal is output, the person in charge of management who notices it can take appropriate measures after confirming the check result. If there is an abnormality in the check result, the user of the vehicle may be notified. Further, the abnormal signal may be continuously output until the person in charge of management performs a predetermined operation. Further, in this case, if the output of the abnormality signal continues for a certain period of time or more, a message (for example, e-mail) notifying the abnormality may be sent to the terminal or the like possessed by the person in charge of management or the user.

3. 3. Others As described above, the vehicle management system 1 includes a moving image object recognition unit 42 that recognizes a vehicle (an example of a first object) from a moving image, and a still image extracting unit that extracts a still image including the vehicle from the moving image. A still image object recognition unit 49 that recognizes a number plate or the like (an example of a second object) attached to a vehicle from the still image is provided.

Therefore, according to the vehicle management system 1, it is easy to recognize the license plate and the like with high accuracy while suppressing the operation load of the system. That is, if an attempt is made to directly recognize a license plate or the like from a moving image, there is a concern that the recognition accuracy may decrease, but by recognizing from a still image, highly accurate object recognition is easy. Furthermore, it is possible to prevent the still image extraction process from being performed when the moving image does not include the vehicle, eliminate unnecessary processing, and reduce the operating load of the system.

Further, the vehicle management system 1 includes an extraction unit 47 that extracts an extracted still image that satisfies the degree of exposure condition, and the still image object recognition unit 49 extracts the extracted still image into the vehicle. It recognizes the included license plate and the like as an object. Therefore, it is possible to more efficiently recognize the license plate and the like included in the vehicle from the still image.

Although the embodiments of the present invention have been described above, the configuration of the present invention is not limited to the above embodiments, and various modifications can be made without departing from the gist of the invention. The technical scope of the present invention is shown not by the description of the above embodiment but by the scope of claims, and is understood to include all modifications belonging to the meaning and scope equivalent to the scope of claims. Should be.

The present invention can be used in a system or the like that recognizes an object from a moving image.

1 Vehicle management system 11 Smartphone 11a Smartphone for front side shooting of approach road 11b Smartphone for shooting rear side of approach road 11c Smartphone for shooting front side of exit road 11d Smartphone for shooting rear side of exit road 12 Communication network 13 Edge server 14 Information processing device 14a Video processing engine 14b Still image processing engine 15 Management server 40 Control unit 41 Communication unit 42 Video object recognition unit 43 Distance detection unit 44 Speed detection unit 45 Still image extraction unit 46 Exposure detection unit 47 Extraction unit 48 Image processing unit 49 Still image object recognition Part 50 Check execution part 51 Abnormal signal output part

Claims (6)

An image object recognition unit that recognizes the first object from image information including a plurality of frames, and an image object recognition unit.
A still image extraction unit that extracts a still image of a plurality of frames including the first object from the image information, and a still image extraction unit.
A still image object recognition unit that recognizes a second object attached to the first object from the still image is provided.
The still image extraction part is
An image recognition system characterized in that the higher the degree of exposure of the first object in the image information, the shorter the time interval for extracting the still image. The first object is a vehicle, the second object is the license plate of the vehicle,
The image recognition system according to claim 1, wherein the image information is an image taken by a camera. As the camera, a front side shooting camera for shooting the front side of the same vehicle and a rear side shooting camera for shooting the rear side are provided.
The image information includes both a front image taken by the front camera and a rear image taken by the rear camera.
The still image object recognition unit is
The license plate on the front side of the vehicle is recognized as an object from the still image extracted from the front image, and the license plate on the rear side of the vehicle is recognized as an object from the still image extracted from the rear image. The image recognition system according to claim 2, wherein the image recognition system is characterized by the above. Each of the extracted still images is provided with an extraction unit for extracting an image that satisfies a predetermined condition regarding the high degree of exposure of at least one of the first object and the second object.
The image recognition according to any one of claims 1 to 3, wherein the still image object recognition unit recognizes a second object included in the first object from the extracted still image. system. Among the extracted still images, the first process of extracting the one with the highest degree of exposure of the specific object, which is the second object, and
The second process of recognizing the display information in the second object from the extracted still image, and
Until the recognition is successful, the third process of extracting the next highest exposure of the specific object from the still images and repeating the second process is executed.
The claim is characterized in that when the recognition in the second process is not successful even if the number of repetitions reaches a predetermined number, the first process to the third process are executed with the specific object as the first object. 4. The object recognition system according to 4. The still image extraction part is
The image recognition system according to claim 3, wherein the time interval for extracting the still image from the front image is shorter than the time interval for extracting the still image from the rear image.

Images