JP7465901B2 - Information processing program, information processing device, and information processing system - Google Patents

Description

The present invention relates to an information processing program, an information processing device, and an information processing system.

Conventionally, there is known a technique for estimating speed information related to the traveling speed of a vehicle traveling on a road. For example, there is known a technique for estimating the traveling speed of a vehicle based on images captured by multiple imaging devices installed at a fixed distance along the traveling direction of the vehicle.

Japanese Patent Application Laid-Open No. 8-55293

However, the above conventional technology requires multiple imaging devices installed at regular intervals to determine the vehicle speed.

The present application has been made in consideration of the above, and aims to provide an information processing program, an information processing device, and an information processing system that can accurately estimate speed information related to the traveling speed of a vehicle traveling on a road, even with only one imaging device.

The information processing program of the present application causes a computer to execute a detection step of detecting a fixed object area, which is an area in an image acquired from an imaging device that captures an image of a road, in which a fixed object fixed on the road is captured, and an estimation step of estimating speed information related to the traveling speed of a vehicle traveling on the road, using a plurality of reference lines on the image based on the position of the fixed object area detected by the detection step.

The estimation step includes a step of detecting a vehicle in the image using a trained machine learning model based on training data including a combination of a vehicle image in which a specific vehicle is captured and information indicating the position and class of the specific vehicle captured in the vehicle image.

According to one aspect of the embodiment, even with a single imaging device, it is possible to accurately estimate speed information related to the traveling speed of a vehicle traveling on a road.

FIG. 1 is a diagram illustrating an example of the configuration of an information processing system according to an embodiment. FIG. 2 is a diagram for explaining an overview of information processing according to the embodiment. FIG. 3 is a diagram illustrating an example of a road area estimation process according to the embodiment. FIG. 4 is a diagram illustrating an example of a process for detecting the edge of a fixed object region according to the embodiment. FIG. 5 is a diagram illustrating an example of a road type estimation process according to the embodiment. FIG. 6 is a diagram illustrating an example of a process for detecting a vehicle in an image using a machine learning model according to an embodiment. FIG. 7 is a diagram illustrating an example of a measurement process of a travel distance of a vehicle according to the embodiment. FIG. 8 is a diagram illustrating an example of a process for determining whether a vehicle has passed through a reference line according to the embodiment. FIG. 9 is a diagram illustrating an example of a measurement process of a travel time of a vehicle according to the embodiment. FIG. 10 is a flowchart showing a detection procedure according to the embodiment. FIG. 11 is a flowchart showing an estimation procedure according to the embodiment. FIG. 12 is a diagram for explaining an overview of information processing according to the first modified example. FIG. 13 is a diagram for explaining an overview of information processing according to the second modified example. FIG. 14 is a diagram for explaining an overview of information processing according to the third modified example. FIG. 15 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of the information processing device.

Below, the information processing program, information processing device, and information processing system according to the present application will be described in detail with reference to the drawings. Note that the information processing program, information processing device, and information processing system according to the present application are not limited to these embodiments. In addition, the same parts in the following embodiments will be denoted by the same reference numerals, and duplicated descriptions will be omitted.

(Embodiment)
[1. Example of information processing system configuration]
Fig. 1 is a diagram showing an example of the configuration of an information processing system according to an embodiment. As shown in Fig. 1, the information processing system 1 includes an imaging device 10 and an information processing device 100. The imaging device 10 and the information processing device 100 are connected to each other via a network N in a wired or wireless manner so as to be able to communicate with each other.

The imaging device 10 captures images of roads. Specifically, the imaging device 10 captures video images of roadways such as expressways and general roads. For example, the imaging device 10 is a fixed camera that captures continuous video images for a predetermined period of time from a specific position. The imaging device 10 may be any type of camera capable of capturing images of roads, such as a fixed camera installed on a road, a security camera, an in-vehicle camera, or a camera mounted on a drone. Furthermore, the number of imaging devices 10 should be at least one.

The information processing device 100 is an information processing device that executes the information processing shown in FIG. 2. FIG. 2 is a diagram for explaining an overview of the information processing according to the embodiment. As shown in FIG. 2, the information processing device 100 acquires a moving image of a road from the imaging device 10. FIG. 2 shows one frame (still image, hereinafter also referred to as a frame image) G1 that constitutes the moving image. The frame image G1 shows two vehicles C1 and C2 traveling on a two-lane road R1. In addition, the frame image G1 shows a part of a white dashed line that divides the lanes in the center of the road R1. Here, the white dashed line refers to a line in which the white line area, which is an area where the white line is imaged, and the interval between the white lines are arranged at a regular interval. For example, the white dashed line may be a center line when the road has two lanes, or a lane boundary line when the road has two or more lanes.

Next, the information processing device 100 acquires a plurality of frame images (for example, 10, 20, 50, or 100 frame images) from the acquired video, and removes moving objects on the road, such as vehicles and people, by adding up the acquired plurality of frame images and taking the average of each pixel (step S1). FIG. 2 shows a composite image G2 after removing two vehicles C1 and C2 that were captured in the frame image G1. In the composite image G2, the vehicles have been removed, and only the white lines that separate the two lanes on the road R1 are captured on the road R1. The information processing device 100 also performs image processing such as a Hough transform on the composite image G2 to detect white line areas O1 and O2, which are areas in which two adjacent white lines are captured. The shapes of the white line areas O1 and O2 can be considered to be long lines parallel to the traveling direction of the road.

The information processing device 100 also estimates the trajectory of each vehicle by tracking each vehicle captured in the acquired video, and estimates the road area A1 based on the estimated trajectory of each vehicle (step S2). Here, the road area refers to an area in the image where the frequency of vehicles traveling exceeds a predetermined threshold. When the information processing device 100 estimates the road area A1, it may generate a composite image G3 in which the estimated road area A1 is drawn on the image of the composite image G2. As shown in FIG. 2, the shape of the road area A1 is elongated in a direction parallel to the road travel direction relative to its length in a direction perpendicular to the road travel direction. Furthermore, the length of the road area A1 in a direction perpendicular to the road travel direction fits within the width of the road.

Next, the information processing device 100 generates a circumscribing rectangle T1 that surrounds the road area A1 (step S3). For example, the circumscribing rectangle T1 is a minimum circumscribing rectangle with a smallest area. When the information processing device 100 generates the circumscribing rectangle T1, the information processing device 100 may generate a composite image G4 by drawing the generated circumscribing rectangle T1 on the image of the composite image G2. As shown in FIG. 2, since the shape of the road area A1 is elongated in a direction parallel to the traveling direction of the road, the circumscribing rectangle T1 is a rectangle whose long side is parallel to the traveling direction of the road.

Next, the information processing device 100 detects one of the two ends of the linear white line area O1 by moving the circumscribing rectangle T1 on the image in the direction of the long side of the circumscribing rectangle T1. In FIG. 2, the information processing device 100 detects a boundary (i.e., one of the boundaries E12 between the white line O1 and the space between the white lines) at which a crossing state in which one short side of the circumscribing rectangle T1 and the white line area O1 shown in the composite image G4 crosses each other changes to a non-crossing state in which one short side of the circumscribing rectangle T1 and the white line area O1 do not cross each other, by sliding the circumscribing rectangle T1 on the image. When the information processing device 100 detects a boundary E12 at which a crossing state changes between one short side of the circumscribing rectangle T1 and the white line area O1 to a non-crossing state, the information processing device 100 identifies the detected boundary E12 as the end of the white line area O1.

Next, the information processing device 100 detects the edge of the white line area O2 adjacent to the white line area O1 by further moving the circumscribing rectangle T1 on the image in the direction of the long side of the circumscribing rectangle. For example, the information processing device 100 detects a boundary E21 (i.e., one of the boundaries E21 and E22 between the white line O2 and the space between the white lines) at which a non-intersecting state in which one short side of the circumscribing rectangle T1 and the white line area O1 do not intersect changes to an intersecting state in which one short side of the circumscribing rectangle T1 and the white line area O2 intersect. Next, the information processing device 100 detects a boundary E22 (i.e., the other boundary E22 of the boundaries E21 and E22 between the white line O2 and the space between the white lines) at which a crossing state where one short side of the circumscribing rectangle T1 and the white line area O2 crosses changes to a non-crossing state where one short side of the circumscribing rectangle T1 and the white line area O2 do not cross by further moving the circumscribing rectangle T1 on the image in the direction of the long side of the circumscribing rectangle. In this way, the information processing device 100 detects the edge E22 on the same side as the edge E12 of the previously detected white line area O1, of the two edges E21 and E22 of the linear white line area O2. When the information processing device 100 detects a boundary E22 at which a crossing state where one short side of the circumscribing rectangle T1 and the white line area O2 crosses changes to a non-crossing state, the information processing device 100 specifies the detected boundary E22 as the edge of the white line area O2.

Next, when the information processing device 100 detects the edges of two adjacent white line areas O1 (border E12) and O2 (border E22), it generates two reference lines L1 and L2 based on the positions of the detected edges of the white line areas O1 and O2 (step S4). For example, the two reference lines L1 and L2 are straight lines parallel to the short sides of the circumscribed rectangle T1. When the information processing device 100 detects the edges of two adjacent white line areas O1 and O2, it may generate a composite image G5 in which the generated reference lines L1 and L2 are drawn at the positions of the detected edges of the white line areas O1 (border E12) and O2 (border E22).

Then, the information processing device 100 uses the two reference lines L1 and L2 to estimate speed information related to the traveling speed of the vehicle traveling on the road R1. In general, the length of the white lines and the length of the interval between the white lines are determined according to the type of road. The information processing device 100 acquires information indicating the length of the white lines and the length of the interval between the white lines according to the type of road. The information processing device 100 specifies the actual distance between the two reference lines L1 and L2 based on the information indicating the length of the white lines and the length of the interval between the white lines. The information processing device 100 also specifies the time required for the vehicle to travel between the two reference lines L1 and L2 from the moving image. The information processing device 100 calculates the traveling speed of the vehicle that traveled between the two reference lines L1 and L2 by dividing the specified actual distance between the two reference lines L1 and L2 by the time required for the vehicle to travel between the two reference lines L1 and L2.

Note that in the above-mentioned FIG. 2, when the information processing device 100 detects a boundary where one short side of the circumscribing rectangle T1 and the white line area change from an intersecting state to a non-intersecting state, the detected boundary is identified as the edge of the white line area, but this is not limited to the above. Specifically, when the information processing device 100 detects a boundary where one short side of the circumscribing rectangle T1 and the white line area change from a non-intersecting state to an intersecting state, the information processing device 100 may identify the detected boundary as the edge of the white line area.

In addition, in FIG. 2 above, the vehicle is described as an automobile, but the vehicle is not limited to an automobile. For example, the vehicle in this embodiment may include, in addition to an automobile, a motorized bicycle (motorcycle), a bicycle, a cart, or other vehicle that is towed by human or animal power or by another vehicle and does not operate on rails.

The configuration of the information processing device 100 according to the embodiment will now be described with reference to FIG. 1 again. As shown in FIG. 1, the information processing device 100 has a communication unit 110, a storage unit 120, and a control unit 130. Note that the information processing device 100 may also have an input unit (e.g., a keyboard, a mouse, etc.) that accepts various operations from a user of the information processing device 100, etc.

(Communication unit 110)
The communication unit 110 is realized by, for example, a network interface card (NIC), etc. The communication unit 110 is also connected to a network N by wire or wirelessly, and transmits and receives information to and from the imaging device 10, for example.

(Memory unit 120)
The storage unit 120 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 120 stores various programs (corresponding to an example of an information processing program). The storage unit 120 also stores images acquired by the acquisition unit 131 from the imaging device 10. The storage unit 120 also stores information related to a trained machine learning model M1 used to detect vehicles in images.

(Control unit 130)
The control unit 130 is a controller, and is realized, for example, by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), etc., by executing various programs (corresponding to an example of an information processing program) stored in a storage device inside the information processing device 100 using a storage area such as a RAM as a working area.

As shown in FIG. 1, the control unit 130 has an acquisition unit 131, a detection unit 132, an estimation unit 133, and a provision unit 134, and realizes or executes the information processing action described below. Note that the internal configuration of the control unit 130 is not limited to the configuration shown in FIG. 1, and may be other configurations as long as they perform the information processing described below.

(Acquisition unit 131)
The acquisition unit 131 acquires images from the imaging device 10 that captures images of a road. Specifically, the acquisition unit 131 acquires images from the imaging device 10 via the communication unit 110. For example, the acquisition unit 131 acquires moving images captured continuously for a predetermined period of time by the imaging device 10 installed at a fixed position. When the acquisition unit 131 acquires images, it stores the acquired images in the storage unit 120.

(Detection Unit 132)
The detection unit 132 detects a white line area, which is an area where a white line fixed on a road is captured, from the image acquired by the acquisition unit 131. In this way, the detection unit 132 detects a fixed object area (e.g., a white line area) from the image, which is an area where a fixed object (e.g., a white line) fixed on a road is captured. For example, the detection unit 132 acquires a moving image acquired by the acquisition unit 131. Next, the detection unit 132 acquires n different frame images (e.g., n=10, 20, 50, or 100, etc.) from the moving image. Next, the detection unit 132 adds up all the pixels of the acquired n frame images multiplied by (1/n). In this way, the detection unit 132 generates a composite image from which moving objects moving on the road are deleted (hereinafter, also referred to as an image from which a vehicle is deleted) by image processing that adds up the n frame images and takes the average of each pixel. Next, the detection unit 132 paints in the generated composite image other than the road area, which is an area where the road is captured. Next, the detection unit 132 converts the entire composite image into a grayscale image, extracts edges contained in the grayscale converted composite image, and then performs a Hough transform on the composite image from which the edges have been extracted to detect white line regions in the image, which are regions in which white lines are captured.

(Estimation unit 133)
The estimation unit 133 estimates speed information related to the traveling speed of a vehicle traveling on a road, using a plurality of reference lines on the image based on the positions of the white line areas detected by the detection unit 132. In this way, the estimation unit 133 estimates speed information related to the traveling speed of a vehicle traveling on a road, using a plurality of reference lines on the image based on the positions of fixed object areas (e.g., white line areas) detected by the detection unit 132. More specifically, the estimation unit 133 detects the edges of the white line areas based on the white line areas and circumscribing rectangles that surround road areas including areas in which the vehicle traveling frequency exceeds a predetermined threshold in the image, and estimates the speed information using a plurality of reference lines based on the positions of the detected edges of the white line areas.

Figure 3 is a diagram showing an example of a road area estimation process according to the embodiment. Image G11 shown in Figure 3 shows one frame constituting the moving image acquired by the acquisition unit 131. The estimation unit 133 acquires the moving image acquired by the acquisition unit 131. Next, the estimation unit 133 tracks each vehicle captured in the moving image to generate a line connecting the center coordinates of each vehicle, and estimates the generated line as the trajectory of each vehicle (step S21). The estimation unit 133 may generate a composite image G12 in which the line connecting the center coordinates of each vehicle is drawn.

Next, the estimation unit 133 extracts only pixels from the composite image G12 that have a darkness exceeding a predetermined threshold value based on the composite image G12 (step S22). The estimation unit 133 may generate a composite image G13 from which only pixels that have a darkness exceeding a predetermined threshold value are extracted.

Next, the estimation unit 133 extracts the contours of each region of the composite image G13 that is formed by the extracted pixels based on the composite image G13, and clusters each region based on the perimeter length of each region (step S23). The estimation unit 133 may generate a composite image G14 that visualizes the clustering of each region.

Next, the estimation unit 133 fills each of the clustered regions so that each region becomes a convex hull (step S24). The estimation unit 133 may generate a composite image G15 that visualizes the state in which each region is filled so that it becomes a convex hull.

The estimation unit 133 then extracts the area of the cluster with the largest area from among the areas filled in the convex hull, and estimates the area of the extracted cluster as the road area. In this way, the estimation unit 133 estimates the convex hull including the vehicle trajectory as the road area. In the example shown in FIG. 3, since there are two elements that belong to the cluster with the largest area, the estimation unit 133 estimates the one of the two elements with the largest area as the road area (step S25). The estimation unit 133 may generate a composite image G16 that visualizes the estimated road area.

The estimation unit 133 then superimposes the estimated road area on the image from which the vehicle has been removed, generated by the detection unit 132 (step S25). The estimation unit 133 may generate a composite image G31 in which the estimated road area has been superimposed on the image from which the vehicle has been removed.

Figure 4 is a diagram showing an example of a process for detecting the edge of a fixed object area in an embodiment. Image G31 shown in Figure 4 shows the composite image G31 generated in Figure 3. Based on the composite image G31, the estimation unit 133 generates a circumscribing rectangle that surrounds the estimated road area in the composite image G31 (step S31). When the estimation unit 133 generates a circumscribing rectangle, it may generate a composite image G41 by superimposing the generated circumscribing rectangle on an image from which the vehicle has been removed.

Next, the estimation unit 133 detects the edge of the white line area where the short side of the circumscribing rectangle and the white line area change from an intersection state where the short side of the circumscribing rectangle and the white line area intersect to a non-intersection state where the short side of the circumscribing rectangle and the white line area do not intersect, or the edge of the white line area where the non-intersection state changes to an intersection state, by moving the circumscribing rectangle on the image in the direction of the long side of the circumscribing rectangle. When the estimation unit 133 detects the edges of multiple white line areas by moving the circumscribing rectangle on the image in the direction of the long side of the circumscribing rectangle, it generates multiple reference lines based on the positions of each of the detected edges of the multiple white line areas (step S41). For example, the estimation unit 133 generates multiple reference lines parallel to the short side of the circumscribing rectangle at the positions of each of the detected edges of the multiple white line areas. The estimation unit 133 may generate a composite image G51 in which the generated reference lines are superimposed on the image in which the vehicle has been removed.

The estimation unit 133 may then generate multiple reference lines adjusted so that the angles of the reference lines are perpendicular to the white lines (step S42). For example, when the estimation unit 133 detects the edge of a white line area, it calculates the angle of a straight line along the longitudinal direction of the white line. The estimation unit 133 may then generate multiple reference lines arranged so that the angles of the reference lines are perpendicular to the calculated angle. The estimation unit 133 may generate a composite image G52 in which multiple reference lines adjusted so that the angles are perpendicular to an image from which the vehicle has been removed are superimposed. This allows the information processing device 100 to estimate speed information based on multiple reference lines perpendicular to the traveling direction of the road, thereby allowing for more accurate estimation of speed information.

Road standards also stipulate that the length of white lines on general roads in Japan is 5m, and the distance between the lines is 5m. In other words, on general roads in Japan, the ratio of the length of the white line area to the distance between the lines is 1:1, and the total length of the white line area and the distance between the lines is 10m. Road standards also stipulate that the length of white lines on domestic expressways is 8m, and the distance between the lines is 12m. In other words, on domestic expressways, the ratio of the length of the white line area to the distance between the lines is 1:1.5, and the total length of the white line area and the distance between the lines is 20m. In other words, the total length of the white line area and the distance between the lines on expressways is twice the total length of the white line area and the distance between the lines on general roads.

FIG. 5 is a diagram showing an example of a road type estimation process in an embodiment. The estimation unit 133 estimates the type of road based on the ratio between the length of the white line area and the length of the interval between the white lines. The image G61 shown in FIG. 5 shows one frame constituting the moving image acquired by the acquisition unit 131. The estimation unit 133 fills in the area of the image G61 other than the road area, which is the area where the road is captured. Next, the estimation unit 133 converts the entire composite image to grayscale. Next, the estimation unit 133 extracts edges included in the grayscale converted composite image. Next, the estimation unit 133 performs a Hough transform on the composite image from which the edges have been extracted, and detects white line areas, which are areas where the white lines are captured, in the composite image. Next, the estimation unit 133 performs a projective transform by giving coordinates to four points indicating the edges of the four white line areas included in the Hough transformed composite image. The image G62 shown in FIG. 5 is a composite image after the four points shown in the image G61 have been projectively transformed. The length of the white line area in image G62 after projective transformation shown in FIG. 5 is 73 pixels, and the length of the space between the white lines is 106 pixels. At this time, the estimation unit 133 calculates that the ratio between the length of the white line area in image G62 after projective transformation and the length of the space between the white lines is 1:1.45. The estimation unit 133 estimates that the road captured in image G62 is an expressway because the calculated ratio of 1:1.45 is closer to the ratio between the length of the white line area and the length of the space between the white lines on an expressway (1:1.5) than the ratio between the length of the white line area and the length of the space between the white lines on an ordinary road (1:1).

Then, the estimation unit 133 estimates speed information according to the estimated road type. For example, if the estimated road type is a general road, the estimation unit 133 specifies the actual distance between the two reference lines as 10 m. If the estimated road type is an expressway, the estimation unit 133 specifies the actual distance between the two reference lines as 20 m. The estimation unit 133 also specifies the time required for the vehicle to travel between the two reference lines from the video. The estimation unit 133 calculates the travel speed of the vehicle that traveled between the two reference lines by dividing the specified actual distance between the two reference lines by the time required for the vehicle to travel between the two reference lines.

FIG. 6 is a diagram showing an example of a detection process of a vehicle in an image using a machine learning model according to an embodiment. The estimation unit 133 detects a vehicle in an image using a trained machine learning model M1 that has been trained using images of a vehicle as training data. Specifically, the estimation unit 133 acquires a trained machine learning model M1 based on training data including a combination of an image of a vehicle and information indicating the position and class (vehicle) of the vehicle captured in the image. More specifically, the estimation unit 133 acquires a machine learning model M1 that has been trained to output, as output information, an image to which information indicating the position and class (vehicle) of the vehicle captured in the image is added when an image of a vehicle is input as input information to the machine learning model. For example, the estimation unit 133 refers to the storage unit 120 and acquires the trained machine learning model M1 stored in the storage unit 120. Next, when the estimation unit 133 acquires the trained machine learning model M1, the estimation unit 133 inputs the image acquired by the acquisition unit 131 into the trained machine learning model, and acquires, as an estimation result, an image to which information indicating the position and class (vehicle) of the vehicle output from the trained machine learning model M1 is added. In FIG. 6, the estimation unit 133 acquires, as the estimation result, an image to which coordinate information of a rectangle circumscribing each vehicle is added as the position information of each detected vehicle. For example, the estimation unit 133 may acquire, as the estimation result, an image to which coordinate information indicating the position of the center point of a rectangle circumscribing each vehicle is added as the position information of each detected vehicle.

The trained machine learning model M1 acquired by the estimation unit 133 is generated by machine learning using a neural network such as a convolutional neural network or a regression neural network, but is not limited to such examples. For example, the trained machine learning model M1 acquired by the estimation unit 133 may be generated using machine learning using a learning algorithm such as linear regression or logistic regression instead of a neural network.

Figure 7 is a diagram showing an example of a measurement process for a vehicle's travel distance according to an embodiment. In Figure 7, the estimation unit 133 estimates that the road captured in the image is an expressway. Then, since the estimation unit 133 has estimated that the type of road is an expressway, it determines the actual distance between reference line 1 and reference line 2 in the image to be 20 m. Then, since the estimation unit 133 has determined that the actual distance between reference line 1 and reference line 2 in the image to be 20 m, when the estimation unit 133 detects a vehicle passing between reference line 1 and reference line 2 in the image, it determines the travel distance of the detected vehicle to be 20 m.

FIG. 8 is a diagram showing an example of a process for determining whether a vehicle has passed through a reference line according to an embodiment. The estimation unit 133 records, in each frame image acquired by the acquisition unit 131, coordinate information (hereinafter also referred to as position information) indicating the position of a representative point of the vehicle in each frame image. For example, the estimation unit 133 may detect each vehicle in each frame image, and record the position coordinates of the center point of a rectangle circumscribing each detected vehicle as the position coordinates of the representative point in association with each frame image. In addition, the estimation unit 133 acquires a current frame image, and determines whether a line segment connecting a representative point of a specific vehicle in the acquired current frame image and a representative point of a specific vehicle in a next frame image that is adjacent to the current frame image in time intersects with the reference line. If the estimation unit 133 determines that a line segment connecting a representative point of a specific vehicle in the current frame image and a representative point of a specific vehicle in the next frame image intersects with the reference line, it determines that the specific vehicle has passed through the reference line. On the other hand, if the estimation unit 133 determines that the line segment connecting the representative point of the specified vehicle in the current frame image and the representative point of the specified vehicle in the next frame image does not intersect with the reference line, it determines that the specified vehicle has not passed through the reference line.

9 is a diagram showing an example of a measurement process of a travel time of a vehicle according to an embodiment. The estimation unit 133 estimates the travel time of a vehicle in an image based on the frame rate of a moving image captured by the imaging device 10. In FIG. 9, a case where the frame rate of a moving image is 30 (fps) will be described. The estimation unit 133 records a time corresponding to a frame (hereinafter also referred to as a first frame) at the moment when the vehicle passes the first reference line 1 on the road. For example, the estimation unit 133 records a time corresponding to the fifth frame out of 30 frames as the time corresponding to the frame at the moment when the vehicle passes the reference line 1. Next, the estimation unit 133 records a time corresponding to a frame (hereinafter also referred to as a second frame) at the moment when the vehicle passes the second reference line 2 on the road. For example, the estimation unit 133 records a time corresponding to the 17th frame out of 30 frames as the time corresponding to the frame at the moment when the vehicle passes the reference line 2. Next, the estimation unit 133 calculates the time difference between the time corresponding to the second frame at the moment when the vehicle passes the reference line 2 and the time corresponding to the first frame at the moment when the vehicle passes the reference line 1, and estimates that the time corresponding to the calculated time difference is the travel time required for the vehicle to pass between the reference line 1 and the reference line 2. For example, when the frame rate of the moving image is 30 (fps), the estimation unit 133 calculates the time difference between the time corresponding to the 17th frame and the time corresponding to the 5th frame as "(17-5)/30=0.4 seconds". Next, the estimation unit 133 estimates that the calculated time difference of "0.4 seconds" is the travel time required for the vehicle to pass between the reference line 1 and the reference line 2. In this way, the estimation unit 133 estimates that the travel time required for the vehicle to pass between the two reference lines is the time equivalent to the time difference between the time corresponding to the first frame in which the vehicle is imaged passing one of the two adjacent reference lines and the time corresponding to the second frame in which the vehicle is imaged passing the other of the two reference lines.

Note that, although FIG. 9 describes a case where the estimation unit 133 estimates the travel time of the vehicle in the image based on the frame rate of the moving image, the method by which the estimation unit 133 estimates the travel time of the vehicle in the image is not limited to this. For example, the estimation unit 133 may acquire the respective times at which the vehicle passes through reference line 1 and reference line 2, and estimate that the difference between the respective acquired times is the travel time required for the vehicle to pass between reference line 1 and reference line 2.

Then, the estimation unit 133 estimates traffic information for the road based on the estimated speed information. For example, the estimation unit 133 estimates traffic information such as congestion information for each road nationwide, road closures or accident occurrence status for each road, etc., based on the estimated speed information.

(Providing Unit 134)
The providing unit 134 provides the traffic information of the road estimated by the estimating unit 133. Specifically, the providing unit 134 transmits the traffic information of the road estimated by the estimating unit 133 to the terminal device of the user.

2. Information Processing Procedures
Fig. 10 is a flowchart showing a detection procedure according to the embodiment. As shown in Fig. 10, the acquisition unit 131 of the information processing device 100 acquires an image from the imaging device 10 that captures an image of a road (step S101). The detection unit 132 of the information processing device 100 deletes moving objects such as vehicles and people from the image acquired by the acquisition unit 131 (step S102). Next, the detection unit 132 detects a fixed object area, which is an area where a fixed object fixed on the road is captured, from the image from which the moving objects have been deleted (step S103).

11 is a flowchart showing an estimation procedure according to the embodiment. As shown in FIG. 11, the acquisition unit 131 of the information processing device 100 acquires an image from the imaging device 10 that captures an image of a road (step S201). The estimation unit 133 of the information processing device 100 estimates a road area based on the trajectory of the vehicle (step S202). Next, the estimation unit 133 generates a circumscribing rectangle of the road area (step S203). Next, the estimation unit 133 detects the edge of the fixed object area by moving the short side of the circumscribing rectangle in the long side direction of the circumscribing rectangle on the image (step S204). Next, the estimation unit 133 determines whether or not two or more edges of the fixed object area have been detected (step S205). When the estimation unit 133 determines that two or more edges of the fixed object area have been detected (step S205; Yes), the estimation unit 133 estimates speed information related to the traveling speed of the vehicle traveling on the road using two reference lines based on the positions of the edges of the two detected fixed object areas (step S206). On the other hand, if the estimation unit 133 determines that two or more edges of the fixed object region have not been detected (step S205; No), it repeats the process of step S204.

3. Modifications
The information processing device 100 according to the above-described embodiment may be implemented in various different forms other than the above-described embodiment. Therefore, other embodiments of the information processing device 100 will be described below. Note that the same parts as those in the embodiment will be denoted by the same reference numerals and descriptions thereof will be omitted.

3-1. First Modified Example
Fig. 12 is a diagram for explaining an overview of information processing according to the first modified example. In the above-mentioned embodiment, a case has been described in which the information processing device 100 estimates speed information using two reference lines L1 and L2 based on the positions of white line areas fixed on one road R1. Fig. 12 explains a case in which the information processing device 100 estimates speed information using two reference lines based on the positions of white line areas fixed on two adjacent roads R2 and R3.

Figure 12 shows a composite image G7 after the vehicle that was captured in the video has been removed. In Figure 12, the two adjacent roads R2 and R3 are captured in the composite image G7 after the vehicle has been removed. The composite image G7 also shows white line areas O11 and O12 on road R3, in which the white lines that separate the two lanes on road R2 on the left side are captured. The composite image G7 also shows white line areas O21 and O22 on road R3, in which the white lines that separate the two lanes on road R3 on the right side are captured.

Specifically, the detection unit 132 performs image processing such as a Hough transform on the composite image G7 to detect white line areas O11 and O12 on the left road R2, and white line areas O21 and O22 on the right road R3. Note that in FIG. 12, the shapes of the white line areas O11 and O12 can be regarded as long lines extending in a direction parallel to the traveling direction of road R2. Also, the shapes of the white line areas O21 and O22 can be regarded as long lines extending in a direction parallel to the traveling direction of road R3.

Although not shown, the estimation unit 133 estimates the trajectory of each vehicle by tracking each vehicle captured in the acquired video image, and estimates the road area of each road based on the estimated trajectory of each vehicle. Next, the estimation unit 133 generates a circumscribing rectangle that surrounds each road area. Next, the estimation unit 133 moves each circumscribing rectangle in the long side direction of each circumscribing rectangle on each road to identify the ends of two adjacent white line areas O11 and O12 on road R2 and the ends of two adjacent white line areas O21 and O22 on road R3. Next, the estimation unit 133 estimates the speed information using a reference line L3 that connects the end of the white line area O11 on road R2 and the end of the white line area O21 on road R3, and a reference line L4 that connects the end of the white line area O12 on road R2 and the end of the white line area O22 on road R3.

As a result, the information processing device 100 can reduce distance errors in an image caused by the effects of perspective compared to when multiple reference lines based on white line areas on a single road are used, allowing the information processing device 100 to estimate speed information more accurately.

3-2. Second Modification
13 is a diagram for explaining an overview of information processing according to the second modified example. In the above-mentioned embodiment, the information processing device 100 detects a white line area in which a white line fixed on a road is captured, and estimates speed information using a plurality of reference lines on an image based on the position of the detected white line area, but this is not limited to the above. Specifically, the information processing device 100 may estimate speed information using a plurality of reference lines on an image based on the position of any fixed object area, as long as the fixed object area is a fixed object area in which a fixed object including a reference point that is a reference for the distance in the traveling direction of the road in the image is captured, other than a white line. For example, the fixed object may be a white line of a pedestrian crossing or a white line of a stop line, in addition to a white line that divides a lane.

In FIG. 13, the information processing device 100 differs from the above-described embodiment in that, instead of the white line area, the information processing device 100 estimates speed information using multiple reference lines on the image based on the position of the sidewalk area in which the crosswalk is captured. FIG. 13 shows a composite image G81 after a vehicle that was captured in a moving image has been deleted. In FIG. 13, a crosswalk fixed on the road after the vehicle has been deleted is captured in the composite image G81. Note that in FIG. 13, the information processing device 100 performs the following processing if it is able to acquire data indicating the width of a crosswalk fixed on a straight road (i.e., the length in the direction of travel of the vehicle) from an external database or the like.

Specifically, the detection unit 132 performs image processing such as a Hough transform on the composite image G81 to detect seven white line areas O31 to O37 in which the seven white lines of the pedestrian crossing fixed on the road are captured. Note that in FIG. 13, the shape of each of the white line areas O31 to O37 can be regarded as a long line extending in a direction parallel to the traveling direction of the road.

Although not shown, the estimation unit 133 estimates the trajectory of each vehicle by tracking each vehicle captured in the acquired video image, and estimates the road area of the road based on the estimated trajectory of each vehicle. Next, the estimation unit 133 generates a circumscribing rectangle that surrounds the road area. Next, the estimation unit 133 moves the circumscribing rectangle on the road in the direction of the long side of the circumscribing rectangle to identify one end of each of the white line areas O31 to O37 and the other end of each of the white line areas O31 to O37. Next, the estimation unit 133 estimates speed information using a reference line L6 that connects one end of each of the white line areas O31 to O37 and a reference line L7 that connects the other end of each of the white line areas O31 to O37.

3-3. Third Modification
FIG. 14 is a diagram for explaining an overview of information processing according to the third modified example. In FIG. 14, the information processing device 100 estimates speed information using a plurality of reference lines on an image based on the positions of both ends of a sidewalk area, instead of estimating speed information using a plurality of reference lines on an image based on the positions of one end of a stop line located within a predetermined range from a crosswalk and one end of a sidewalk area, which is different from FIG. 13. FIG. 14 shows a composite image G82 after a vehicle that was captured in a moving image has been deleted. In FIG. 14, as in FIG. 13, a crosswalk fixed on a road after the vehicle has been deleted and a stop line (see FIG. 13) located within a predetermined range from the crosswalk are captured in the composite image G82. In FIG. 14, the information processing device 100 performs the following processing when it is able to acquire data indicating the distance between a line connecting the ends of a crosswalk fixed on a straight road and the stop line from an external database or the like.

Specifically, the detection unit 132 performs image processing such as a Hough transform on the composite image G82 to detect seven white line areas O31-O37 in which the seven white lines of a pedestrian crossing fixed on the road are captured, and a white line area O4 (see FIG. 13) in which a stop line located within a predetermined range from the pedestrian crossing is captured. Note that in FIG. 14, the shape of each of the white line areas O31-O37 can be regarded as a long line extending in a direction parallel to the traveling direction of the road. The shape of the white line area O4 can be regarded as a long line extending in a direction perpendicular to the traveling direction of the road.

Although not shown, the estimation unit 133 estimates the trajectory of each vehicle by tracking each vehicle captured in the acquired video, and estimates the road area of the road based on the estimated trajectory of each vehicle. Next, the estimation unit 133 generates a circumscribing rectangle that surrounds the road area. Next, the estimation unit 133 moves the circumscribing rectangle on the road in the direction of the longer side of the circumscribing rectangle to identify the end of each of the white line areas O31 to O37 that is closer to the stop line, and the end of the white line area O4. Next, the estimation unit 133 estimates the speed information using a reference line L7 that connects the ends of each of the white line areas O31 to O37 that are closer to the stop line, and a reference line L8 that is parallel to the end of the white line area O4.

4. Effects
As described above, the information processing device 100 according to the embodiment includes the detection unit 132 and the estimation unit 133. The detection unit 132 detects a fixed object area, which is an area in which a fixed object fixed on a road is captured, from an image acquired from the imaging device 10 that captures an image of a road. The estimation unit 133 estimates speed information related to the traveling speed of a vehicle traveling on a road, using a plurality of reference lines on the image based on the positions of the fixed object area detected by the detection unit 132.

As a result, the information processing device 100 can accurately estimate speed information regarding the traveling speed of a vehicle traveling on a road, even with only one imaging device.

The estimation unit 133 also detects the edges of the fixed object area based on a circumscribing rectangle that encloses a road area in the image, including an area where the vehicle traveling frequency exceeds a predetermined threshold, and the fixed object area, and estimates speed information using multiple reference lines based on the positions of the edges of the detected fixed object area.

This allows the information processing device 100 to accurately estimate speed information based on, for example, a white line fixed near the center of the road, without including straight lines at both ends of the road.

In addition, the estimation unit 133 detects the edge of the fixed object area where the short side of the circumscribing rectangle and the fixed object area change from an intersecting state where the short side of the circumscribing rectangle and the fixed object area intersect to a non-intersecting state where the short side of the circumscribing rectangle and the fixed object area do not intersect, or the edge of the fixed object area where the non-intersecting state changes to an intersecting state, by moving the circumscribing rectangle on the image in the direction of the long side of the circumscribing rectangle.

This allows the information processing device 100 to properly detect the edge of the fixed object area along the road travel direction.

In addition, when the estimation unit 133 detects the edges of multiple fixed object regions by moving the circumscribing rectangle on the image in the direction of the longer side of the circumscribing rectangle, it estimates the speed information using multiple reference lines based on the positions of each of the detected edges of the multiple fixed object regions.

This allows the information processing device 100 to accurately estimate speed information using multiple reference lines.

In addition, the estimation unit 133 estimates the speed information using multiple reference lines parallel to the short sides of the circumscribing rectangle.

This allows the information processing device 100 to estimate speed information more accurately.

The estimation unit 133 also estimates the road area based on the vehicle trajectory estimated by tracking the vehicle captured in the moving image.

This allows the information processing device 100 to appropriately estimate areas of the road where the frequency of vehicle travel exceeds a predetermined threshold.

The estimation unit 133 also estimates the convex hull that includes the vehicle trajectory as the road area.

This allows the information processing device 100 to appropriately estimate areas of the road where the frequency of vehicle travel exceeds a predetermined threshold.

In addition, the estimation unit 133 detects the edge of the fixed object region based on the minimum circumscribing rectangle that has the smallest area and the fixed object region.

This allows the information processing device 100 to accurately estimate the edges of fixed object regions.

The fixed object is a white line fixed on the road. The detection unit 132 detects a white line area, which is an area where a white line is imaged, as a fixed object area. The estimation unit 133 estimates the type of road based on the ratio between the length of the white line area and the length of the space between the white lines, and estimates the actual distance between multiple reference lines according to the estimated road type.

This allows the information processing device 100 to accurately estimate the actual distance between multiple reference lines in the image depending on the type of road, thereby making it possible to more accurately estimate speed information.

The estimation unit 133 also detects vehicles in the image using a trained machine learning model based on learning data including a combination of a vehicle image in which a specific vehicle is captured and information indicating the position and class of the specific vehicle captured in the vehicle image.

This allows the information processing device 100 to accurately detect vehicles in the image, thereby enabling more accurate estimation of speed information.

The estimation unit 133 also determines whether a line segment connecting a first representative point indicating the position of the vehicle in one frame image and a second representative point indicating the position of the vehicle in the next frame image that is adjacent in time to the one frame image intersects with a reference line, and if it is determined that the line segment intersects with the reference line, it determines that the vehicle has passed the reference line, and if it is determined that the line segment does not intersect with the reference line, it determines that the vehicle has not passed the reference line.

This allows the information processing device 100 to accurately determine whether the vehicle has passed the reference line, thereby enabling more accurate estimation of speed information.

The estimation unit 133 also estimates that the travel time required for the vehicle to pass between the two reference lines is the time corresponding to the time difference between the time corresponding to the first frame in which the vehicle passing one of the two adjacent reference lines is imaged, and the time corresponding to the second frame in which the vehicle passing the other of the two reference lines is imaged.

This allows the information processing device 100 to accurately determine the travel time required for the vehicle to pass between multiple reference lines, thereby enabling more accurate estimation of speed information.

The estimation unit 133 also estimates road traffic information based on the estimated speed information.

As a result, the information processing device 100 can estimate road traffic information with high accuracy based on accurate speed information.

5. Hardware Configuration
Moreover, the information processing device 100 according to the embodiment described above is realized by a computer 1000 having a configuration as shown in Fig. 15, for example. Fig. 15 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information processing device 100. The computer 1000 includes a CPU 1100, a RAM 1200, a ROM 1300, a HDD 1400, a communication interface (I/F) 1500, an input/output interface (I/F) 1600, and a media interface (I/F) 1700.

The CPU 1100 operates based on a program stored in the ROM 1300 or the HDD 1400, and controls each component. The ROM 1300 stores a boot program executed by the CPU 1100 when the computer 1000 is started, and programs that depend on the hardware of the computer 1000, etc.

HDD 1400 stores programs executed by CPU 1100 and data used by such programs. Communication interface 1500 receives data from other devices via a specified communication network and sends it to CPU 1100, and transmits data generated by CPU 1100 to other devices via the specified communication network.

The CPU 1100 controls output devices such as a display and a printer, and input devices such as a keyboard and a mouse, via the input/output interface 1600. The CPU 1100 acquires data from the input devices via the input/output interface 1600. The CPU 1100 also outputs generated data to the output devices via the input/output interface 1600.

The media interface 1700 reads a program or data stored in the recording medium 1800 and provides it to the CPU 1100 via the RAM 1200. The CPU 1100 loads the program from the recording medium 1800 onto the RAM 1200 via the media interface 1700 and executes the loaded program. The recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase change rewritable Disc), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

For example, when the computer 1000 functions as the information processing device 100 according to the embodiment, the CPU 1100 of the computer 1000 executes programs loaded onto the RAM 1200 to realize the functions of the control unit 130. The CPU 1100 of the computer 1000 reads and executes these programs from the recording medium 1800, but as another example, the CPU 1100 may obtain these programs from another device via a specified communication network.

Although several embodiments of the present application have been described in detail above with reference to the drawings, these are merely examples, and the present invention can be embodied in other forms that incorporate various modifications and improvements based on the knowledge of those skilled in the art, including the forms described in the disclosure section of the invention.

[6. Other]
Furthermore, among the processes described in the above embodiments and modifications, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by a known method. In addition, the information including the processing procedures, specific names, various data and parameters shown in the above documents and drawings can be changed arbitrarily unless otherwise specified. For example, the various information shown in each drawing is not limited to the illustrated information.

In addition, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

The information processing device 100 described above may be realized using multiple server computers, and depending on the functions, the configuration can be flexibly changed, such as by calling an external platform using an API (Application Programming Interface) or network computing.

The above-described embodiments and variations can be combined as appropriate to the extent that they do not cause inconsistencies in the processing content.

The above-mentioned "section, module, unit" can be read as "means" or "circuit". For example, the estimation unit can be read as estimation means or estimation circuit.

REFERENCE SIGNS LIST 1 Information processing system 10 Imaging device 100 Information processing device 110 Communication unit 120 Storage unit 130 Control unit 131 Acquisition unit 132 Detection unit 133 Estimation unit 134 Provision unit

Claims (12)

a detection step of detecting a fixed object area, which is an area in which a fixed object fixed on the road is imaged, from an image acquired from an imaging device that captures an image of the road;
an estimation step of detecting edges of the fixed object area based on the fixed object area detected by the detection step, a circumscribing rectangle surrounding a road area in the image including an area where a frequency of vehicles traveling on the road exceeds a predetermined threshold, and the fixed object area, and estimating speed information related to the traveling speed of the vehicle using a plurality of reference lines on the image based on the positions of the edges of the detected fixed object area;
An information processing program that causes a computer to execute the above. The estimation procedure comprises:
by moving the circumscribing rectangle in the direction of the long side of the circumscribing rectangle on the image, detecting an end of the fixed object region that changes from an intersecting state in which the short side of the circumscribing rectangle and the fixed object region intersect to a non-intersecting state in which the short side of the circumscribing rectangle and the fixed object region do not intersect, or detecting an end of the fixed object region that changes from the non-intersecting state to the intersecting state;
The information processing program according to claim 1 . The estimation procedure comprises:
when edges of a plurality of the fixed object regions are detected by moving the circumscribing rectangle in a direction of a longer side of the circumscribing rectangle on the image, the speed information is estimated using the plurality of reference lines based on the positions of the respective detected edges of the plurality of the fixed object regions;
3. The information processing program according to claim 1 or 2. The estimation procedure comprises:
estimating the velocity information using the plurality of reference lines parallel to the short sides of the circumscribing rectangle;
4. The information processing program according to claim 1. The estimation procedure comprises:
estimating the road area based on a trajectory of the vehicle estimated by tracking the vehicle captured in the image, which is a moving image;
5. An information processing program according to claim 1. The estimation procedure comprises:
A convex hull including the trajectory of the vehicle is estimated as the road area.
The information processing program according to claim 5. The estimation procedure comprises:
detecting an edge of the fixed object region based on a minimum circumscribing rectangle that has a minimum area of the circumscribing rectangle and the fixed object region;
7. An information processing program according to claim 1. the fixed object is a white line fixed on the road,
The detection step comprises:
A white line area is detected as the fixed object area, and the white line area is an area where the white line is captured.
The estimation procedure comprises:
a type of the road is estimated based on a ratio between a length of the white line area and a length of a space between the white lines, and an actual distance between the plurality of reference lines is estimated in accordance with the estimated type of the road.
An information processing program according to any one of claims 1 to 7. The estimation procedure comprises:
the step of determining whether a line segment connecting a first representative point indicating the position of the vehicle in one frame image constituting a moving image captured by the imaging device and a second representative point indicating the position of the vehicle in a next frame image adjacent in time to the one frame image intersects with the reference line, determining that the vehicle has passed through the reference line when it is determined that the line segment intersects with the reference line, and determining that the vehicle has not passed through the reference line when it is determined that the line segment does not intersect with the reference line.
An information processing program according to any one of claims 1 to 8. The estimation procedure comprises:
a time corresponding to the difference between a time corresponding to a first frame in which the vehicle has passed one of the two adjacent reference lines and an image of the vehicle has passed the other of the two reference lines and a time corresponding to a second frame in which the vehicle has passed the other of the two reference lines is estimated to be the travel time required for the vehicle to pass between the two reference lines;
10. An information processing program according to claim 1. a detection unit that detects a fixed object area, which is an area in which a fixed object fixed on the road is captured, from an image acquired from an imaging device that captures an image of the road;
an estimation unit that detects edges of the fixed object area based on the fixed object area detected by the detection unit, a circumscribing rectangle that encloses a road area in the image including an area where the frequency of vehicles traveling on the road exceeds a predetermined threshold, and the fixed object area, and estimates speed information related to the traveling speed of the vehicle using a plurality of reference lines on the image based on the positions of the edges of the detected fixed object area;
An information processing device comprising: An information processing system including an imaging device and an information processing device,
The imaging device includes:
Taking an image of the road,
The information processing device includes:
a fixed object area is detected from the image acquired from the imaging device, which is an area in which a fixed object fixed on the road is imaged; an edge of the fixed object area is detected based on the detected fixed object area, a circumscribing rectangle that encloses a road area in the image including an area in which a traveling frequency of vehicles on the road exceeds a predetermined threshold, and the fixed object area; and speed information relating to the traveling speed of the vehicle is estimated using a plurality of reference lines on the image based on the positions of the edges of the detected fixed object area.
Information processing system.

Images