JP6943116B2 - Mobile recognition device, program and mobile recognition system - Google Patents

Description

The present invention relates to mobile recognition devices, programs and mobile recognition systems.

Conventionally, a technique for recognizing a moving body moving on a traveling path has been known. For example, a vehicle can be mentioned as an example of a moving body. As an example of the technique for recognizing a moving object, a technique for recognizing the license plate information of a vehicle is disclosed (see, for example, Patent Document 1).

In this technology, a license plate provided on the front side of the vehicle is photographed by a camera (front camera) installed directly above the vehicle's driving path (hereinafter, also referred to as "lane"), and directly above the lane. The license plate provided on the back side of the vehicle is photographed by the camera (rear camera) installed in. Then, by tracking the vehicle in the image taken by the camera (intermediate camera) provided between the front camera and the rear camera, the vehicle photographed by the front camera and the vehicle photographed by the rear camera are the same. The sex is guaranteed.

International Publication No. 2008/010308

However, there may be cases where the vehicle license plate information alone is not sufficient as the information necessary for discriminating the vehicle. For example, when the ETC (Electronic Toll Collection) tollhouse becomes free-flow, in addition to the license plate information of the vehicle, various types are used to determine the detailed information (vehicle type classification) of the vehicle used for the toll calculation. Vehicle information (eg, vehicle size, number of axles, etc.) may be required.

Further, when the intermediate camera is provided between the front camera and the rear camera, the intermediate camera is also provided directly above the lane. However, in such a case, the number of devices installed directly above the lane will increase, and there is a risk of falling objects on the lane due to some accident (for example, strong wind, deterioration of fixtures, etc.). Will increase.

Therefore, the present invention has been made in view of the above problems, and an object of the present invention is to provide a technique capable of improving the discrimination accuracy of a moving body. As an example, an ETC service that recognizes vehicle license plate information and vehicle information and determines detailed vehicle information (vehicle type classification) based on vehicle license plate information and vehicle information to create a free flow. It is hoped that applicable technology will be provided. Further, it is desired to provide a technology capable of more safely discriminating a vehicle by reducing the possibility that the camera falls into the lane.

In order to solve the above problem, according to a certain viewpoint of the present invention, a first image of a vehicle provided on the upper part of the traveling path and passing through the first photographing area of the traveling path is captured. A first image acquisition unit that acquires the first front image from the camera, and a second image acquisition area that is provided on the upper part of the travel path and does not overlap with the first photography area of the travel path. A second image acquisition unit that acquires the first rear image from a second camera that captures a first rear image of the passing vehicle , and a second image acquisition unit that is provided on the side portion of the travel path and that is the first. and a front-side common area and a part or the whole of the rear side common area of the second imaging area which is a part or the whole of the imaging region as the imaging region, a second front image of the vehicle, the vehicle the side image and a second rear image of the vehicle from the third camera that photographed continuously, and a third image acquisition unit for acquiring the second front image, the side image and said second rear image , The discriminating unit that discriminates the vehicle type classification of the vehicle based on the first front image, the second front image, the side image, the first rear image, and the second rear image. The discriminating unit is based on the license plate information of the vehicle recognized from at least one of the first front image and the first rear image, and the vehicle information recognized from the side image. When the vehicle type classification of the vehicle is determined and it is determined that the front license plate information recognized from the first front image and the rear license plate information recognized from the first rear image do not match. In addition, when it is determined that the vehicle has a towed vehicle and a towed vehicle, the vehicle type classification of the vehicle is determined based on at least the type of the towed vehicle and the number of axles of the towed vehicle. A recognition device is provided.

The vehicle information may include at least one of the size of the vehicle, the number of axles of the vehicle, and information indicating whether the vehicle has a towed vehicle and a towed vehicle.

The discriminating unit detects the moving body in the front side common area in each of the first front image and the second front image, and from the front side common area in the second front image, the second When the movement of the moving body to the back side common area in the back image is detected, and the moving body is detected in the back side common area in each of the second back image and the first back image, the said. The moving body may be discriminated.

The discriminating unit predicts the timing at which the moving body enters the front side common area based on the image captured by the third camera, starts recognizing the front side license plate information based on the timing, and the above-mentioned. The timing at which the moving body enters the rear side common area may be predicted based on the image captured by the third camera, and the recognition of the back side license plate information may be started based on the timing.

Further, according to another aspect of the present invention, a first camera is provided on the upper part of the traveling path and captures a first front image of the vehicle passing through the first photographing area in the traveling path. To pass through the first image acquisition unit for acquiring the first front image and the second image acquisition area provided in the upper part of the travel path and which does not overlap with the first photographing area in the travel path. A second image acquisition unit that acquires the first rear image from a second camera that captures the first rear image of the vehicle , and a second image acquisition unit that is provided on the side portion of the travel path and is provided in the first shooting area. some or and a front side common area and the part or the whole of the rear side common region of the second imaging region in whole as the imaging region, a second front image of the vehicle, side image of the vehicle A third image acquisition unit that acquires the second front image, the side image, and the second rear image from the third camera that continuously captures the second rear image of the vehicle, and the above. A discriminating unit that discriminates the vehicle type classification of the vehicle based on the first front image, the second front image, the side image, the first rear image, and the second rear image is provided. The discriminating unit is based on the license plate information of the vehicle recognized from at least one of the first front image and the first rear image and the vehicle information recognized from the side image. When it is determined that the vehicle type classification of the vehicle does not match the front license plate information recognized from the first front image and the rear license plate information recognized from the first rear image, and , A moving body recognition device that determines the vehicle type classification of the vehicle based on at least the type of the towed vehicle and the number of axles of the towed vehicle when it is determined that the vehicle has a towed vehicle and a towed vehicle. A program is provided to make it function as.

Further, according to another aspect of the present invention, the first camera provided at the upper part of the traveling path and capturing the first front image of the vehicle passing through the first photographing area in the traveling path, and the above. A second camera provided at the upper part of the travel path and capturing a first rear image of the vehicle passing through a second photographing area that does not overlap with the first photographing area of the traveling path, and the traveling path. The front side common area, which is a part or all of the first shooting area, and the back side common area, which is a part or all of the second shooting area, are included as a shooting area. A third camera that continuously captures a second front image of the vehicle, a side image of the vehicle, and a second rear image of the vehicle, the first front image, the second front image, and the side surface. image, on the basis of the first rear image and the second back image, seen including a mobile recognition device, the comprising a determination section which determines whether a vehicle type classification of the vehicle, wherein the determination unit, the first Based on the license plate information of the vehicle recognized from at least one of the front image and the first rear image of the vehicle and the vehicle information recognized from the side image, the vehicle type classification of the vehicle is determined. When it is determined that the front license plate information recognized from the first front image and the rear license plate information recognized from the first rear image do not match, and the vehicle is covered with the towing vehicle. When it is determined that the vehicle has a towed vehicle, a moving body recognition system is provided that determines the vehicle type classification of the vehicle based on at least the type of the towed vehicle and the number of axles of the towed vehicle.

As described above, according to the present invention, there is provided a technique capable of improving the discrimination accuracy of a moving body.

It is the first figure for demonstrating the outline of the mobile body recognition system which concerns on this embodiment. It is a 2nd figure for demonstrating the outline of the mobile body recognition system which concerns on this embodiment. It is a figure for demonstrating the outline of recognition of license plate information. It is a figure for demonstrating the outline at the time of recognizing the vehicle which has a towed vehicle and a towed vehicle by a moving body recognition system. It is a figure for demonstrating the outline of recognition of the license plate information when the vehicle which has a towed vehicle and a towed vehicle is recognized by a moving body recognition system. It is a figure for demonstrating an example of discriminating the vehicle type classification of the vehicle used for charge calculation. It is a figure for demonstrating an example of discriminating the vehicle type classification of the vehicle used for charge calculation. It is a figure for demonstrating an example of discriminating the vehicle type classification of the vehicle which has a towed vehicle and a towed vehicle. It is a block diagram which shows the functional structure of the mobile body recognition apparatus which concerns on this embodiment. It is a figure for demonstrating the calibration process between cameras. It is a figure which shows the case where the vehicle exists in the common area on the front side in the front image of an intermediate camera. It is a figure which shows the case where a vehicle exists in the intermediate camera side image. It is a figure which shows the case where the vehicle exists in the common area on the back side in the rear side image of an intermediate camera.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the present specification and the drawings, components having substantially the same functional configuration are designated by the same reference numerals, so that duplicate description will be omitted.

Further, in the present specification and the drawings, a plurality of components having substantially the same functional configuration are distinguished by adding different numbers after the same reference numerals. In addition, similar components of different embodiments are distinguished by adding different alphabets after the same reference numerals. However, if it is not necessary to distinguish each of a plurality of components having substantially the same functional configuration, only the same reference numerals are given.

[1. System overview]
Subsequently, the outline of the mobile body recognition system according to the present embodiment will be described. FIG. 1 is a first diagram for explaining an outline of the mobile body recognition system according to the present embodiment. As shown in FIG. 1, the mobile body recognition system 1 includes a mobile body recognition device 10, a first camera (hereinafter, also referred to as “forward camera”) 30, and a second camera (hereinafter, “rear camera”). It also includes a 40 and a third camera (hereinafter, also referred to as an “intermediate camera”) 20. The front camera 30, the rear camera 40, and the intermediate camera 20 are each connected to the moving object recognition device 10.

Further, referring to FIG. 1, a state in which the vehicle 50-1 as an example of the moving body travels in the lane is shown at an angle at which the side surface of the vehicle 50-1 can be seen. FIG. 1 shows the positions of vehicles 50-1 traveling in the lane when the time T is T10 and when the time T is T20. A front side license plate 51-1 is provided on the front surface of the vehicle 50-1. On the other hand, the back side license plate 52-1 is provided on the back surface of the vehicle 50-1.

In this embodiment, a vehicle will be mainly used as an example of the moving body. However, the moving body that can be applied to this embodiment is not limited to the vehicle. For example, the moving body may be a ship, a train, or another moving body.

The front camera 30 is provided in the upper part of the lane, and is a first front image (hereinafter, hereinafter) of the vehicle 50-1 passing through the first shooting area (hereinafter, also referred to as “front camera shooting area”) in the lane. , Also called "forward camera image"). The front camera 30 looks down on the front license plate 51-1 of the vehicle 50-1 from diagonally above when the vehicle 50-1 traveling in the lane from the back side to the front side enters the front camera shooting area. It is possible to shoot at various angles. However, the installation position of the front camera 30 is not limited. For example, the front camera 30 may be installed on the side of the road. The front camera image captured by the front camera 30 is provided to the moving object recognition device 10.

The rear camera 40 is provided in the upper part of the lane, and the vehicle 50-1 passes through a second shooting area (hereinafter, also referred to as “rear camera shooting area”) that does not overlap with the front camera shooting area in the lane. The first rear image (hereinafter, also referred to as "rear camera image") is taken. The rear camera 40 looks down on the rear license plate 52-1 of the vehicle 50-1 from diagonally above when the vehicle 50-1 traveling in the lane from the front side to the back side enters the rear camera shooting area. It is possible to shoot at various angles. However, the installation position of the rear camera 40 is not limited. For example, the rear camera 40 may be installed on the side of the road. The rear camera image taken by the rear camera 40 is provided to the moving object recognition device 10.

The intermediate camera 20 includes a front side common area that is a part or all of the front camera shooting area and a back side common area that is a part or all of the rear camera shooting area as a shooting area. This enables tracking processing of vehicle 50-1, as will be described later. In the following, a case where the front side common area is the entire front camera shooting area and a case where the back side common area is the entire rear camera shooting area will be mainly described. However, the front side common area may be a part of the front camera shooting area, and the back side common area may be a part of the rear camera shooting area.

Further, the intermediate camera 20 is provided on the side of the lane. As a result, the intermediate camera 20 has a second front image of the vehicle 50-1 (hereinafter, also referred to as "intermediate camera front image") and a side image of the vehicle 50-1 (hereinafter, also referred to as "intermediate camera side image"). ) And the second rear image of the vehicle 50-1 (hereinafter, also referred to as “intermediate camera rear image”) can be continuously captured. Further, this makes it possible to discriminate the vehicle more safely by reducing the possibility that the camera falls into the lane.

It should be noted that the entire vehicle 50-1 does not always have to be captured in each of the intermediate camera front image, the intermediate camera side image, and the intermediate camera rear image. That is, in the state where the intermediate camera front image, the intermediate camera side image, and the intermediate camera rear image can be continuously shot, a part or all of the intermediate camera front image, the intermediate camera side image, and the intermediate camera rear image is temporarily taken. It may include a case where the camera is shielded by an object other than the vehicle 50-1 (for example, an electric pole). Hereinafter, the side portion of the lane will be further described with reference to FIG.

FIG. 2 is a second diagram for explaining the outline of the mobile body recognition system 1 according to the present embodiment. With reference to FIG. 2, the lane R in which the vehicle 50-1 travels is shown, and the state in which the vehicle 50-1 travels in the lane R is shown at an angle at which the back surface of the vehicle 50-1 can be seen. Further, in FIG. 2, the space above the lane R is shown as the upper U of the lane R. The side portion of the lane R corresponds to a space other than the upper U (that is, spaces E-1 and E-2 diagonally above the lane R). In the example shown in FIG. 2, the intermediate camera 20 is provided in the space E-1 diagonally above the lane R.

In the present embodiment, the intermediate camera 20 has a fisheye lens 210 so that the shooting area of the intermediate camera 20 includes at least a part of the front camera shooting area and a part of the rear camera shooting area. Mainly assume the case of (fisheye camera). However, the intermediate camera 20 does not have to be a fisheye camera. For example, as the intermediate camera 20, a plurality of cameras having a normal lens may be used. At this time, it is sufficient that the entire plurality of cameras having the normal lens include at least a part of the front camera shooting area and a part of the rear camera shooting area.

[2. Recognition of license plate information]
Next, an outline of recognition of license plate information will be described. FIG. 3 is a diagram for explaining an outline of recognition of license plate information. With reference to FIG. 3, the front camera image IF-1 and the rear camera image IB-1 are shown. The front camera image IF-1 shows the front side license plate 51-1 of the vehicle 50-1. Further, the rear camera image IB-1 shows the license plate 52-1 on the back side of the vehicle 50-1.

The mobile body recognition device 10 cuts out a region including the front side license plate 51-1 from the front camera image IF-1 as a front side license plate cut-out image KF-1. Further, the moving body recognition device 10 cuts out a region including the back side license plate 52-1 from the rear camera image IB-1 as a back side license plate cut-out image KB-1.

The mobile body recognition device 10 attempts to recognize the front side license plate information based on the front side license plate cutout image KF-1. Further, the mobile body recognition device 10 attempts to recognize the back side license plate information based on the back side license plate cut-out image KB-1. Here, it is assumed that the mobile body recognition device 10 recognizes both the front side license plate information and the back side license plate information. In such a case, the mobile body recognition device 10 may recognize either the front side license plate information or the back side license plate information as the license plate information N-1.

At this time, the mobile body recognition device 10 may adopt either the front side license plate information or the back side license plate information as the license plate information N-1. For example, the mobile body recognition device 10 may adopt the license plate information of the predetermined side of the front side and the back side. Alternatively, the mobile body recognition device 10 may adopt the license plate information on the front side and the back side, which have higher recognition accuracy.

Alternatively, it is assumed that the mobile body recognition device 10 fails to recognize either the front side license plate information or the back side license plate information. In such a case, when the moving body recognition device 10 succeeds in recognizing the front side license plate information, the front side license plate information may be recognized as the license plate information, and the back side license plate information is successfully recognized. In that case, the license plate information on the back side may be recognized as the license plate information.

[3. Specific example of vehicle information]
In this embodiment, we mainly propose a technique capable of improving the discrimination accuracy of a vehicle. More specifically, by recognizing the vehicle license plate information and the vehicle information and determining the detailed vehicle information (vehicle type classification) based on the vehicle license plate information and the vehicle information, free flow is achieved. We propose technologies that can be applied to ETC services. Here, the vehicle information is not particularly limited. For example, vehicle information may include at least one of vehicle size, number of axles of the vehicle, and information indicating whether the vehicle has a towed vehicle and a towed vehicle.

Here, in order to show the importance of information indicating whether the vehicle has a towed vehicle and a towed vehicle, a case where the vehicle has a towed vehicle and a towed vehicle will be described. FIG. 4 is a diagram for explaining an outline when a vehicle having a towed vehicle and a towed vehicle is recognized by the moving body recognition system 1. Referring to FIG. 4, a vehicle having a towed vehicle 50-2 and a towed vehicle 50-3 traveling in a lane is shown at an angle at which the side surface of the vehicle can be seen. A front side license plate 51-2 is provided on the front surface of the towing vehicle 50-2. On the other hand, the back side license plate 52-3 is provided on the back surface of the towed vehicle 50-3.

FIG. 5 is a diagram for explaining an outline of recognition of license plate information when a vehicle having a towed vehicle 50-2 and a towed vehicle 50-3 is recognized by the moving body recognition system 1. With reference to FIG. 5, the front camera image IF-2 and the rear camera image IB-2 are shown. The front camera image IF-2 shows the front license plate 51-2 of the towing vehicle 50-2. Further, the rear camera image IB-2 shows the license plate 52-3 on the back side of the towed vehicle 50-3.

The mobile body recognition device 10 cuts out a region including the front side license plate 51-2 from the front camera image IF-2 as a front side license plate cut-out image KF-2. Further, the moving body recognition device 10 cuts out a region including the back side license plate 52-3 from the rear camera image IB-2 as a back side license plate cut-out image KB-2.

The mobile body recognition device 10 attempts to recognize the front side license plate information based on the front side license plate cutout image KF-2. Further, the mobile body recognition device 10 attempts to recognize the back side license plate information based on the back side license plate cut-out image KB-2. Here, it is assumed that the mobile body recognition device 10 recognizes both the front side license plate information and the back side license plate information. However, as shown in FIG. 5, the front side license plate information recognized from the front side license plate 51-2 of the towing vehicle 50-2 and the back side license plate 52-3 of the towed vehicle 50-3 are recognized. It is assumed that it may be different from the license plate information on the back side.

In this way, when the vehicle has a towing vehicle 50-2 and a towed vehicle 50-3, a situation arises in which the front license plate information recognized by the vehicle and the rear license plate information are different. obtain. In addition to that, there may be a situation in which the front side license plate information and the back side license plate information recognized by the vehicle are different due to erroneous recognition. However, if it is recognized from the side image of the intermediate camera whether the vehicle has the towed vehicle 50-2 and the towed vehicle 50-3, the front side license plate information and the back side license plate information are different for any reason. Can be determined.

[4. Example of vehicle classification]
Next, an example of determining the vehicle type classification of the vehicle used for the charge calculation will be described. It should be noted that the vehicle type classification discrimination example described below is merely an example of vehicle type classification discrimination. Therefore, the vehicle type classification may be discriminated by a method other than the discriminating example described below. 6 and 7 are diagrams for explaining an example of determining the vehicle type classification of the vehicle used for the charge calculation. With reference to FIGS. 6 and 7, the vehicle type classification, the vehicle type name, the license plate information, the vehicle size, and the axle are associated in advance.

Here, referring to FIGS. 6 and 7, the license plate information includes the vehicle model number, the size of the license plate, and the color of the license plate described on the license plate. The "*" included in the vehicle model number indicates an arbitrary one-digit number. In addition, the vehicle size includes the vehicle length, vehicle width, and vehicle height. Axles include the number of axles. In this way, the vehicle type classification can be determined based on the license plate information and the vehicle information (for example, the vehicle size and the number of axles).

FIG. 8 is a diagram for explaining an example of discriminating the vehicle type classification of a vehicle having a towed vehicle and a towed vehicle. With reference to FIG. 8, the vehicle type classification, the vehicle type name, the type of the towed vehicle, the number of axles of the towed vehicle, and the total axle are associated in advance. As shown in FIG. 8, the vehicle type classification of the towed vehicle and the vehicle having the towed vehicle can be determined based on the type of the towed vehicle and the number of axles of the towed vehicle.

[5. Example of functional configuration of mobile recognition device]
Subsequently, the details of the present embodiment will be described. First, a functional configuration example of the mobile body recognition device 10 according to the present embodiment will be described. FIG. 9 is a block diagram showing a functional configuration of the mobile body recognition device 10 according to the present embodiment. As shown in FIG. 9, the mobile body recognition device 10 according to the present embodiment has a first processing unit (hereinafter, also referred to as “front camera image processing unit”) 130 and a second processing unit (hereinafter, ““ front camera image processing unit ”) 130. It includes a rear camera image processing unit (also referred to as "rear camera image processing unit") 140, a third processing unit (hereinafter, also referred to as "intermediate camera image processing unit") 120, an inter-camera calibration unit 150, and a discrimination processing unit 160. ..

The front camera image processing unit 130 has a first image acquisition unit (hereinafter, also referred to as “front camera image acquisition unit”) 131 and a front number recognition unit 132. The rear camera image processing unit 140 has a second image acquisition unit (hereinafter, also referred to as “rear camera image acquisition unit”) 141 and a rear number recognition unit 142. The intermediate camera image processing unit 120 includes a third image acquisition unit (hereinafter, also referred to as “intermediate camera image acquisition unit”) 141, an intermediate camera calibration unit 122, and a vehicle information recognition unit 123.

The intermediate camera image processing unit 120, the front camera image processing unit 130, the rear camera image processing unit 140, the inter-camera calibration unit 150, and the discrimination processing unit 160 include a CPU (Central Processing Unit) and the like, and are provided by a non-volatile storage device. The function can be realized by expanding the stored program into a RAM (Random Access Memory) by the CPU and executing the program. At this time, a computer-readable recording medium on which the program is recorded may also be provided.

Alternatively, the intermediate camera image processing unit 120, the front camera image processing unit 130, the rear camera image processing unit 140, the inter-camera calibration unit 150, and the discrimination processing unit 160 may be configured by dedicated hardware, or a plurality of them. It may be configured by a combination of hardware.

In the example shown in FIG. 9, the intermediate camera image processing unit 120, the front camera image processing unit 130, and the rear camera image processing unit 140 are provided inside the moving body recognition device 10. However, the intermediate camera image processing unit 120, the front camera image processing unit 130, and the rear camera image processing unit 140 may be provided outside the moving body recognition device 10. For example, the intermediate camera image processing unit 120 may be provided inside the intermediate camera 20, the front camera image processing unit 130 may be provided inside the front camera 30, or the rear camera image processing unit 130 may be provided inside the rear camera 30. The unit 140 may be provided inside the rear camera 40.

(5.1. Calibration processing)
First, the calibration process executed by the mobile body recognition device 10 will be described. As an example of the first calibration process, the intermediate camera calibration performed by the intermediate camera calibration unit 122 will be described.

(5.1.1. Intermediate camera calibration process)
Here, as will be described later, the vehicle information recognition unit 123 recognizes the vehicle size as an example of vehicle information from the intermediate camera side image acquired by the intermediate camera image acquisition unit 121. However, when the intermediate camera 20 has a fisheye lens, the side image of the intermediate camera taken based on the light transmitted through the fisheye lens is distorted, and the recognition accuracy of the vehicle size based on the side image of the intermediate camera is improved. It is expected that it will not improve.

Therefore, when the intermediate camera 20 has a fisheye lens, the intermediate camera calibration unit 122 captures an image captured by the intermediate camera 20 (hereinafter, also referred to as “omnidirectional image”) by a central projection method (hereinafter, also referred to as “omnidirectional image”). , Also called "flat image"). As a method for converting an omnidirectional image into a flat image, the method disclosed in Japanese Patent Application Laid-Open No. 2017-146704 may be used. Further, the method for converting the omnidirectional image into the flat image may be determined according to the characteristics of the fisheye lens.

When the omnidirectional image is converted into a flat image in this way, the intermediate camera calibration unit 122 calculates the road plane formula based on the flat image. The road plane formula may be calculated based on the distance between two parallel straight lines on the road plane, as disclosed in Japanese Patent Application Laid-Open No. 2013-148971 and Japanese Patent Application Laid-Open No. 2014-164694. Alternatively, the road plane type may be calculated based on the length between the existing marks on the road plane, or the road plane type may be calculated based on the length of the pillar provided with the front camera 30 and the rear camera 40. May be calculated.

(5.1.2. Calibration process between cameras)
FIG. 10 is a diagram for explaining the inter-camera calibration process. Here, as will be described later, the vehicle tracking process ensures the identity of the vehicle photographed by the front camera 30 and the vehicle photographed by the rear camera 40. In order for such tracking processing to be executed accurately, the alignment between the front camera 30 and the intermediate camera 20 and the alignment between the rear camera 40 and the intermediate camera 20 need to be executed in advance. There is.

That is, it is necessary for the inter-camera calibration unit 150 to know in advance which position of the front camera front image corresponds to the front side common area (here, the entire front camera shooting area) in the front camera shooting area. be. Further, it is necessary for the inter-camera calibration unit 150 to know in advance which position of the rear camera front image corresponds to the rear side common area (here, the entire rear camera shooting area) in the rear camera shooting area. be. For example, as the inter-camera calibration process, a manual alignment method and an installation information utilization method can be used.

According to the manual alignment method, the camera-to-camera calibration process is executed with high accuracy. In the manual alignment method, the inter-camera calibration unit 150 aligns the front camera 30 and the intermediate camera 20 based on the manual operation by the user. For example, an object is placed on the lane so that a predetermined object (for example, a predetermined marker, a specific vehicle, a normal vehicle traveling under a predetermined condition, etc.) is captured in a common area on the front side in a front camera shooting area. When installed, the front side common area in the front camera shooting area and the position where the object is captured in the intermediate camera image IM-T0 may be associated with each other. With reference to FIG. 10, the intermediate camera image IM-T0 shows the position P-1 corresponding to the front side common area.

Here, the predetermined conditions are not particularly limited. For example, the predetermined condition may be a condition in which there is a low possibility that an erroneous association is made (for example, a condition that there is no other vehicle traveling in front of or behind the vehicle). Further, any vehicle may be used as a specific vehicle and a normal vehicle traveling under predetermined conditions. However, it is desirable that a small light vehicle be used as a specific vehicle and a normal vehicle that travels under predetermined conditions. Since it is assumed that the angles of view of the front camera 30 and the rear camera 40 are not very wide, if a small light vehicle having a relatively large license plate relative to the size of the vehicle is used, the recognition accuracy of the license plate information will be accurate. This is because it is assumed that the decrease in the number of plates can be suppressed.

Further, in the manual alignment method, it is not necessary to manually perform all the alignment processing. For example, in the case of passing a vehicle under the condition that there is a low possibility that an erroneous association is made as described above, the vehicle is automatically separated from the image of the front camera 30 and the image of the intermediate camera 20 by existing image processing methods such as background subtraction. If it is confirmed that the correspondence visually confirmed by the user is correct after the target detection and the association of the vehicle positions are automatically performed, the alignment of the positions can be automated only by the confirmation work. The rear camera 40 and the intermediate camera 20 can be associated with each other in the same manner.

Similarly, in the manual alignment method, the inter-camera calibration unit 150 aligns the rear camera 40 and the intermediate camera 20 based on the manual operation by the user. For example, an object is placed on the lane so that a predetermined object (for example, a predetermined marker, a specific vehicle, a normal vehicle traveling under a predetermined condition, etc.) is captured in a common area on the rear side in a rear camera shooting area. When installed, the common area on the back side in the rear camera shooting area and the position where the object is captured in the intermediate camera image IM-T0 may be associated with each other. With reference to FIG. 10, the intermediate camera image IM-T0 shows the position P-2 corresponding to the common area on the back side.

The installation information utilization method is an effective method when camera parameters and camera installation information can be obtained with high accuracy. For example, in the installation information utilization method, the inter-camera calibration unit 150 includes camera parameters (for example, resolution, angle of view, optical axis direction, etc.) of the front camera 30 and the intermediate camera 20, and the front camera 30 and the intermediate camera 20. Based on each installation information (for example, installation position and installation angle), the position P-1 corresponding to the front side common area in the intermediate camera image IM-T0 may be calculated.

Similarly, in the installation information utilization method, the inter-camera calibration unit 150 determines the camera parameters (for example, resolution, angle of view, optical axis direction, etc.) of the rear camera 40 and the intermediate camera 20, and the rear camera 40 and the intermediate. Based on the installation information of each of the cameras 20 (for example, the installation position and the installation angle), the position P-2 corresponding to the common area on the back side in the intermediate camera image IM-T0 may be calculated.

The manual alignment method and the installation information utilization method may be used alone or in combination. For example, when the installation information utilization method is used, there is a possibility that an error may occur in the position P-1 corresponding to the front side common area in the intermediate camera image IM-T0. Therefore, the inter-camera calibration unit 150 calculates the position P-1 corresponding to the front side common area in the intermediate camera image IM-T0 using the installation information utilization method, and then uses the manual alignment method to calculate the position P-1. 1 may be corrected.

Similarly, when the installation information utilization method is used, there is a possibility that an error may occur in the position P-2 corresponding to the common area on the back side in the intermediate camera image IM-T0. Therefore, the inter-camera calibration unit 150 calculates the position P-2 corresponding to the common area on the back side in the intermediate camera image IM-T0 using the installation information utilization method, and then uses the manual alignment method to calculate the position P-. 2 may be corrected.

(5.1.3. Vehicle discrimination)
Subsequently, the determination of the vehicle will be described. The front camera image acquisition unit 131 acquires the front camera image from the front camera 30. The rear camera image acquisition unit 141 acquires the rear camera image from the rear camera 40. The intermediate camera image acquisition unit 121 acquires the intermediate camera front image, the intermediate camera side image, and the intermediate camera rear image from the intermediate camera 20. Then, the discrimination unit discriminates the vehicle based on the front camera image, the intermediate camera front image, the intermediate camera side image, the rear camera image, and the intermediate camera rear image.

This makes it possible to improve the discrimination accuracy of the vehicle. As an example of vehicle discrimination, as described above, discrimination of vehicle type classification used for fare calculation can be mentioned. The discrimination unit may correspond to the front number recognition unit 132, the vehicle information recognition unit 123, the back number recognition unit 142, and the discrimination processing unit 160.

(Recognition of license plate information)
The front number recognition unit 132 recognizes the front side license plate information from the front camera image acquired by the front camera image acquisition unit 131. Further, the rear license plate recognition unit 142 recognizes the rear license plate information from the rear camera image acquired by the rear camera image acquisition unit 141. As a technique for recognizing license plate information, the techniques described in the following various documents can be used.

(Reference 1) July 2012 ・ SEI technical review, development of automatic license plate reader, Takehiko Kato, Kayo Tanaka, Masatoshi Asada, Hiroyuki Yasuhara, Kanaei Tachiki, Michihiro Kodama, Yusuke Yasuhara, Toshihiro Asai, Ogiuchi Yasuo (Reference 2) January 2014, SEI Technical Review, Automatic License Plate Detection / Recognition in Thailand, Yasuo Ogiuchi, Masakatsu Higashikubo, Sakda, Panwai
(Reference 3) "Development of license plate recognition for high-speed traveling vehicles", Yasuhiro Aoki, Toshio Saito (Toshiba), ITS Study Group 2014/9
(Reference 4) Mitsubishi Heavy Industries Technical Report, VOL45, NO. 3: 2008, Development of vehicle number recognition system for low-resolution cameras

(Recognition of vehicle information)
The vehicle information recognition unit 123 recognizes vehicle information from the side image of the intermediate camera. Since the side surface of the vehicle is relatively large in the side image of the intermediate camera, the characteristics of the vehicle are relatively easy to appear. Therefore, according to the present embodiment, the vehicle information can be recognized with higher accuracy than the case where the vehicle information is recognized from the image taken by the intermediate camera provided directly above the lane. Here, the vehicle information is not particularly limited. For example, vehicle information may include at least one of vehicle size, number of axles of the vehicle, and information indicating whether the vehicle has a towed vehicle and a towed vehicle.

Here, the method of recognizing the vehicle size is not particularly limited. For example, as a method for recognizing the vehicle size, the method disclosed in Japanese Patent Application Laid-Open No. 2013-148971 can be used. The vehicle information recognition unit 123 converts the intermediate camera side image into a flat image by the method of converting the omnidirectional image into a flat image as described above, and the flat image and the road plane type calculated by the intermediate camera calibration unit 122. Based on the above, the vehicle size may be recognized.

Further, the method of recognizing the number of axles of the vehicle is not particularly limited. For example, the vehicle information recognition unit 123 performs Hough transform on the side image of the intermediate camera by the method of converting the omnidirectional image into a flat image as described above, and performs ellipse detection on the image after the Hough transform. good. Then, the vehicle information recognition unit 123 can recognize the detected number of ellipses as the number of tires (that is, the number of axles).

Further, the method of recognizing information indicating whether the vehicle has a towed vehicle and a towed vehicle is not particularly limited. For example, the vehicle information recognition unit 123 pulls the vehicle when the characteristic pattern of the vehicle extracted from the side image of the intermediate camera matches the characteristic pattern of the vehicle having the towed vehicle and the towed vehicle prepared in advance. It may be determined that the vehicle has a vehicle and a towed vehicle. Alternatively, if the learning model for determining whether the vehicle has a towed vehicle and a towed vehicle is constructed in advance by machine learning, the vehicle information recognition unit 123 uses the learning model to determine whether the vehicle has a towed vehicle and a towed vehicle. You may decide to have it.

The discrimination processing unit 160 determines the vehicle type classification of the vehicle based on the vehicle license plate information recognized from at least one of the front camera image and the rear camera image and the vehicle information recognized from the intermediate camera side image. Determine. As a result, a vehicle recognition technology applicable to the free-flowing ETC service can be provided. The recognition of license plate information is as described above. Further, an example of the relationship between the license plate information and the vehicle information and the vehicle type classification has already been described with reference to FIGS. 6 to 8.

(When the vehicle has a towed vehicle and a towed vehicle)
As described above, it is assumed that the front side license plate information recognized from the vehicle and the back side license plate information are different from each other in the case where the vehicle has a towed vehicle and the towed vehicle and the case where the vehicle is misrecognized. Can be done. However, when it is determined from the side image of the intermediate camera that the vehicle has a towed vehicle and a towed vehicle, the reason why the front side license plate information and the rear side license plate information are different is that the vehicle is a towed vehicle and a towed vehicle. Can be identified as having.

Therefore, when the discrimination processing unit 160 determines that the front license plate information recognized from the front camera image and the rear license plate information recognized from the rear camera image do not match, and the vehicle is determined from the intermediate camera side image. If it is determined that the vehicle has a towed vehicle and a towed vehicle, the vehicle type classification may be determined at least based on the type of the towed vehicle and the number of axles of the towed vehicle. An example of the relationship between the type of towing vehicle and the number of axles of the towed vehicle and the vehicle type classification has already been described with reference to FIG.

(Vehicle tracking process)
In the present embodiment, the tracking process of the vehicle captured in the side image of the intermediate camera ensures the identity of the vehicle captured in the front camera image and the vehicle captured in the rear camera image. As described above, since the side surface of the vehicle is relatively large in the side image of the intermediate camera, the characteristics of the vehicle are relatively easy to appear. Therefore, according to the present embodiment, the vehicle tracking process can be executed with higher accuracy than the case where the vehicle tracking process captured in the image taken by the intermediate camera provided directly above the lane is executed.

FIG. 11 is a diagram showing a case where a vehicle exists in a common area on the front side in the front image of the intermediate camera. With reference to FIG. 11, the position P-1 corresponding to the front side common area in the intermediate camera front image IM-T10 is shown. The front side license plate 51-1 of the vehicle 50-1 is shown in the front side common area corresponding to the position P-1. That is, the front side license plate information can be recognized from the front side common area corresponding to the position P-1.

FIG. 12 is a diagram showing a case where a vehicle is present in the side image of the intermediate camera. Referring to FIG. 12, the side surface image IM-T15 of the intermediate camera shows the side surface of the vehicle 50-1. That is, the vehicle information of the vehicle 50-1 can be recognized from the intermediate camera side image IM-T15.

FIG. 13 is a diagram showing a case where a vehicle exists in a common area on the rear side in the rear image of the intermediate camera. With reference to FIG. 13, the position P-2 corresponding to the rear side common area in the intermediate camera rear image IM-T20 is shown. The back side license plate 52-1 of the vehicle 50-1 is shown in the back side common area corresponding to the position P-2. That is, the back side license plate information can be recognized from the back side common area corresponding to the position P-2.

Specifically, the discrimination processing unit 160 detects the vehicle in the front side common area in each of the front camera image and the intermediate camera front image, and changes from the front side common area in the intermediate camera front image to the back side common area in the intermediate camera rear image. It is assumed that the movement of the vehicle is detected (by tracking processing), and the vehicle is detected in the rear side common area in each of the intermediate camera rear image and the rear camera image. In such a case, it is presumed that the vehicle captured in the front camera image and the vehicle captured in the rear camera image are the same.

Therefore, in such a case, the discrimination processing unit 160 may discriminate the vehicle. By such vehicle tracking processing, the identity of the vehicle photographed by the front camera 30 and the vehicle photographed by the rear camera 40 is ensured. That is, even if the lane is congested or the vehicle changes lanes due to such vehicle tracking processing, the vehicle photographed by the front camera 30 and the vehicle photographed by the rear camera 40 It is possible to reduce the possibility of erroneous mapping.

(Timing of number recognition)
It is assumed that the front camera 30 and the rear camera 40 are set to have a relatively narrow angle of view in order to suppress a decrease in the recognition accuracy of the license plate information. Therefore, if the recognition of the front license plate information based on the front camera image and the recognition of the rear license plate information based on the rear camera image are not always executed, recognition omission may occur.

On the other hand, it is assumed that the angle of view of the intermediate camera 20 is set relatively wide. Therefore, it is considered that the timing at which the vehicle enters the front side corresponding region in the front camera image can be predicted based on the image captured by the intermediate camera 20. Similarly, it is considered that the timing at which the vehicle enters the front side corresponding region in the rear camera image can be predicted based on the image taken by the intermediate camera 20.

Therefore, the discrimination processing unit 160 may predict the timing when the vehicle enters the front side corresponding region based on the image captured by the intermediate camera 20, and start the recognition of the front side license plate information based on the timing. Then, the discrimination processing unit 160 may predict the timing when the vehicle enters the rear side corresponding region based on the image captured by the intermediate camera 20, and start the recognition of the back side license plate information based on the timing. This makes it possible to reduce the possibility of omission of recognition of license plate information and suppress power consumption and heat generation by the mobile body recognition device 10.

[Effect of this embodiment]
As described above, according to the present embodiment, from the first camera provided in the upper part of the traveling path and capturing the first front image of the moving body passing through the first photographing area in the traveling path. , A first image acquisition unit that acquires a first front image, and a first moving body that is provided at the upper part of the traveling path and passes through a second photographing area that does not overlap with the first photographing area of the traveling path. A second image acquisition unit that acquires the first rear image from the second camera that captures the rear image of the vehicle, and a front surface that is provided on the side of the traveling path and is a part or the whole of the first shooting area. The side common area and the back side common area which is a part or all of the second shooting area are included as a shooting area, and the second front image of the moving body, the side image of the moving body, and the second rear image of the moving body are included. A third image acquisition unit that acquires a second front image, a side image, and a second rear image from a third camera that continuously shoots a first front image, a second front image, and a side surface. Provided is a mobile body recognition device including a discriminating unit that discriminates a moving body based on an image, a first rear image, and a second rear image.

According to such a configuration, it is possible to improve the discrimination accuracy of the moving body. As an example, an ETC service that recognizes vehicle license plate information and vehicle information and determines detailed vehicle information (vehicle type classification) based on vehicle license plate information and vehicle information to create a free flow. Applicable technology is provided. Further, the third camera is provided on the side of the traveling path, and the possibility that the third camera falls on the traveling path is reduced, so that the vehicle can be discriminated more safely.

[Explanation of modified example]
Although the preferred embodiments of the present invention have been described in detail with reference to the accompanying drawings, the present invention is not limited to such examples. It is clear that a person having ordinary knowledge in the field of technology to which the present invention belongs can come up with various modifications or modifications within the scope of the technical ideas described in the claims. , These are also naturally understood to belong to the technical scope of the present invention.

1 Moving object recognition system 10 Moving object recognition device 120 Intermediate camera image processing unit 121 Intermediate camera image acquisition unit 122 Intermediate camera calibration unit 123 Vehicle information recognition unit 130 Front camera image processing unit 131 Front camera image acquisition unit 132 Front number recognition unit 140 Rear camera image processing unit 141 Rear camera image acquisition unit 142 Rear number recognition unit 150 Inter-camera calibration unit 160 Discrimination processing unit 20 Intermediate camera 210 Fish-eye lens 30 Front camera 40 Rear camera 50 Vehicle 51 Front side number plate 52 Rear side number plate

Claims (6)

A first image for acquiring the first front image from a first camera provided at the upper part of the traveling path and capturing a first front image of a vehicle passing through the first photographing area in the traveling path. Acquisition department and
From a second camera that captures a first rear image of the vehicle that passes through a second photographing area that is provided in the upper part of the traveling path and does not overlap with the first photographing area in the traveling path. The second image acquisition unit that acquires the rear image of 1 and
The front side common area, which is a part or all of the first shooting area, and the back side common area, which is a part or all of the second shooting area, provided on the side of the traveling path are used as a shooting area. wherein, a second front image of the vehicle, the second rear image side image and the vehicle of the vehicle from the third camera continuously photographing, the second front image, the side image and the second A third image acquisition unit that acquires the rear image of 2 and
A discriminating unit that discriminates the vehicle type classification of the vehicle based on the first front image, the second front image, the side image, the first rear image, and the second rear image.
Equipped with a,
The discriminating unit
Vehicle type classification of the vehicle based on the license plate information of the vehicle recognized from at least one of the first front image and the first rear image and the vehicle information recognized from the side image. To determine
When it is determined that the front license plate information recognized from the first front image and the rear license plate information recognized from the first rear image do not match, and the vehicle is towed and towed. When it is determined that the vehicle has a vehicle, the vehicle type classification of the vehicle is determined based on at least the type of the towing vehicle and the number of axles of the towed vehicle.
Mobile recognition device. The vehicle information includes at least one of the size of the vehicle, the number of axles of the vehicle, and information indicating whether the vehicle has a towed vehicle and a towed vehicle.
The mobile recognition device according to claim 1. The discriminating unit
The moving body is detected in the front side common area in each of the first front image and the second front image, and from the front side common area in the second front image to the back surface in the second back image. When the movement of the moving body to the side common area is detected and the moving body is detected in the back side common area in each of the second back image and the first back image, the moving body is determined. conduct,
The mobile recognition device according to claim 1. The discriminating unit predicts the timing at which the moving body enters the front side common area based on the image captured by the third camera, starts recognizing the front side license plate information based on the timing, and the above-mentioned. The timing at which the moving body enters the rear side common area is predicted based on the image captured by the third camera, and the recognition of the back side license plate information is started based on the timing.
The mobile recognition device according to claim 3. Computer,
A first image for acquiring the first front image from a first camera provided at the upper part of the traveling path and capturing a first front image of a vehicle passing through the first photographing area in the traveling path. Acquisition department and
From a second camera that captures a first rear image of the vehicle that passes through a second photographing area that is provided in the upper part of the traveling path and does not overlap with the first photographing area in the traveling path. The second image acquisition unit that acquires the rear image of 1 and
The front side common area, which is a part or all of the first shooting area, and the back side common area, which is a part or all of the second shooting area, provided on the side of the traveling path are used as a shooting area. wherein, a second front image of the vehicle, the second rear image side image and the vehicle of the vehicle from the third camera continuously photographing, the second front image, the side image and the second A third image acquisition unit that acquires the rear image of 2 and
A discriminating unit that discriminates the vehicle type classification of the vehicle based on the first front image, the second front image, the side image, the first rear image, and the second rear image.
Equipped with a,
The discriminating unit
Vehicle type classification of the vehicle based on the license plate information of the vehicle recognized from at least one of the first front image and the first rear image and the vehicle information recognized from the side image. To determine
When it is determined that the front license plate information recognized from the first front image and the rear license plate information recognized from the first rear image do not match, and the vehicle is towed and towed. When it is determined that the vehicle has a vehicle, the vehicle type classification of the vehicle is determined based on at least the type of the towing vehicle and the number of axles of the towed vehicle.
A program to function as a mobile recognition device. A first camera provided at the upper part of the roadway and capturing a first front image of a vehicle passing through the first shooting area of the roadway, and a first camera.
A second camera provided on the upper part of the traveling path and capturing a first rear image of the vehicle passing through a second photographing area of the traveling path that does not overlap with the first photographing area.
The front side common area, which is a part or all of the first shooting area, and the back side common area, which is a part or all of the second shooting area, provided on the side of the traveling path are used as a shooting area. wherein a third camera for capturing a second front image of the vehicle, the second rear image side image and said vehicle of said vehicle continuously,
A moving body including a discriminating unit that discriminates the vehicle type classification of the vehicle based on the first front image, the second front image, the side image, the first rear image, and the second rear image. Recognition device and
Only including,
The discriminating unit
Vehicle type classification of the vehicle based on the license plate information of the vehicle recognized from at least one of the first front image and the first rear image and the vehicle information recognized from the side image. To determine
When it is determined that the front license plate information recognized from the first front image and the rear license plate information recognized from the first rear image do not match, and the vehicle is towed and towed. When it is determined that the vehicle has a vehicle, the vehicle type classification of the vehicle is determined based on at least the type of the towing vehicle and the number of axles of the towed vehicle.
Mobile recognition system.

Images