TWI736138B - System and method for learning traffic safety - Google Patents

Abstract

A system for learning traffic safety includes a display device, a sensor, and a computation module. The sensor captures an image corresponding to a user. The computation module is communicatively connected to the display device and the sensor for displaying a traffic situation through the display dice. The computation module also determines an action of the user according to the image, and determines if the action complies with a traffic rule of the traffic situation to calculate a score.

Description

Traffic safety learning system and method

This disclosure is about using somatosensory interactive technology to allow users to learn and practice traffic safety behaviors and skills in a safe context.

Pedestrian traffic safety ability and knowledge are one of the important reasons for the accidental death of school children. For students with disabilities, mobility and traffic ability are important skills for independent living ability. For example, crossing the road, walking on the road, etc., this ability is also It will further affect their related life skills, such as shopping and the use of community resources. In the twelve-year national education, mobility and transportation skills are also one of the important learning content in the field of special needs.

At present, practical teaching is often carried out through oral descriptions, using photos and videos to set up simulated situations in classrooms for teaching, etc., which requires a lot of manpower to carry out, and the opportunities for active interaction with physically and mentally disabled students are also very limited. Taking it to the road for drills. For students and teachers with disabilities, in addition to safety concerns, it is also unable to provide a large number of training opportunities for students with disabilities in time, and there is currently no design development for mobility and transportation skills. Interactive technology teaching system.

The embodiment of the present invention provides a traffic safety learning system, which includes a display device, a sensor, and a calculation module. The sensor is used to capture at least one image corresponding to the user. The calculation module is used to communicate with the display device and the sensor to display the traffic situation through the display device, determine the user's action based on the above image, and determine whether the action complies with the traffic rules of the traffic situation to calculate the score.

In some embodiments, the calculation module only captures the upper extremity image or the lower extremity image in the above-mentioned images, and the above-mentioned motion is the upper extremity motion or the lower extremity motion.

In some embodiments, the above-mentioned traffic situation includes an intersection, a one-way turn, a circle, a straight, a level crossing, or a T-shaped road, and the traffic situation includes at least one obstacle.

In some embodiments, the above-mentioned image includes a depth image and a brightness image, and the depth image includes a plurality of depths. The calculation module is used to perform the following multiple steps: for every two adjacent depths, calculate the first gradient in the X direction and the second gradient in the Y direction of the two depths, and according to the first gradient and the second gradient Calculate the angle; add this angle to one of the multiple angle grooves between 0 and 360 degrees; and use the value of the angle groove as the feature vector of the depth image.

In some embodiments, the calculation module further sets multiple preset actions and executes the following multiple steps: detecting pedestrians in the brightness image; using the first classifier to calculate the pedestrians corresponding to the preset actions based on the brightness image The first confidence value; the second classifier is used to calculate multiple second confidence values corresponding to the preset actions of the pedestrian based on the depth image; and the first confidence value and the second confidence value are combined Value to determine which default action the user's action belongs to.

From another perspective, an embodiment of the present invention provides a traffic safety learning method for a computing module. The traffic safety learning method includes: capturing at least one image corresponding to a user through a sensor; and using a display device Display the traffic situation; determine the user's action based on the above-mentioned image; and determine whether the action conforms to the traffic rules of the traffic situation to calculate the score.

100: Traffic Safety Learning System

101: Teaching Trainer

102: User

110: calculation module

120: display device

130: Sensor

140: Cloud database

141: Parameters

142: Video data

143: History

201~204: Video

301~304: steps

[Fig. 1] is a schematic diagram showing a traffic safety learning system according to an embodiment.

[Fig. 2] is a schematic diagram showing the feature extraction of the brightness image and the depth image according to an embodiment.

[Fig. 3] is a flowchart of a traffic safety learning method according to an embodiment.

Regarding the “first”, “second”, etc. used in this text, it does not particularly mean the order or sequence, but only to distinguish elements or operations described in the same technical terms.

The traffic safety learning system proposed in this disclosure combines sports and somatosensory interactive technology, so that ordinary students or students with disabilities can practice traffic skills in various traffic situations in the classroom or in a safe environment. The functional design of the system is based on a task-oriented concept. The media built are common traffic situations and facilities in daily life. Each traffic situation is set as a block, and the blocks can be combined to allow the classroom to map the road according to the students’ abilities and needs. 径组合。 Path combination.

Fig. 1 is a schematic diagram illustrating a traffic safety learning system according to an embodiment. Please refer to FIG. 1, the traffic safety learning system 100 includes a calculation module 110, a display device 120 and a sensor 130. The computing module 110 can be a personal computer, a notebook computer, a server, or various computers with computing capabilities. The display device 120 may be a projector, a liquid crystal display, an organic light emitting diode display, a virtual reality display, an augmented reality display, and the like. The sensor 130 may include a visible light sensor, an infrared sensor, a depth sensor, and so on. For example, the depth sensor may include an infrared emitter and an infrared sensor, or include more than two visible light sensors, so that the depth of the scene can be calculated. The computing module 110 is communicatively connected to the display device 120 and the sensor 130. For example, data can be transmitted between these devices through any wired or wireless means.

First, the teaching trainer 101 can log in to the computing module 110 to build a map. The map includes one or more traffic situations. These traffic situations can include intersections, one-way turns, circles, straight ahead, level crossings, or T-shaped road, these traffic situations can be arranged arbitrarily. The teaching trainer 101 can also create obstacles in the traffic situation, set whether the obstacles are dynamic or static, and can also set the density of the obstacles. The teaching trainer 101 can also establish task goals (for example, crossing the road), and set the number and permutation of tasks. In some embodiments, the presentation mode of the map can also be set, which can be fixed orientation or dynamic orientation. In some embodiments, action parameters can also be set, which can be divided into upper limb mode or lower limb mode, that is, the traffic skills are practiced only according to the movements of the upper limbs or the lower limbs.

After the map is created, the computing module 110 can display the created traffic situation through the display 120, and at the same time capture the image corresponding to the user 102 through the sensor 130. The user 102 can be a general student or a student with a disability. , The present invention is not limited to this. In some embodiments, the sensor 130 includes a depth sensor and a visible light sensor, so the captured image includes a depth image and a brightness image. The gray scale of a pixel in the depth image represents the depth of the scene, and The gray scale of one pixel in the brightness image represents the brightness, and the brightness image can be a color image. The calculation module 110 detects the user 102 in the image according to the captured image, and determines the action of the user 102, such as forward, backward, sideward, turn left, turn right, stop, etc. The calculation module 110 also determines whether the actions of the user 102 comply with the traffic rules of the traffic situation to calculate the score. For example, if it is determined that the user 102 is moving forward, but there is an obstacle in front of the user, it means that the user 102 will hit the obstacle and will deduct points. On the contrary, if the user 102 will turn left or right to flash Over obstacles will add points. In other words, in some embodiments, points can be added or deducted based on whether the user will hit an obstacle. In some embodiments, the points can also be calculated according to current traffic rules, for example, points are deducted when running a red light, points are deducted when not walking on a zebra crossing, and so on.

In some embodiments, considering the diversity of limb movements of persons with disabilities, such as cerebral palsy and limb disorders, it is quite difficult to use the lower limbs to perform movements. Therefore, the system can select upper limbs or lower limbs according to students' abilities and needs. One of them to perform. Specifically, the computing module 110 only captures the upper limb image or lower limb image of the pedestrian in the image, according to the upper limb image Images or lower limb images are used to determine the upper limb movement or lower limb movement of the user, so that users with inconvenient lower limbs (or upper limbs) can only rely on upper limb movements (or lower limb movements) to complete the exercises.

After the user completes the exercise, the computing module 110 can collect relevant parameters 141 (which may include maps and traffic situations), video data 142 and history records 143 (which may include travel routes and scores), and store these data in the cloud data The library 140 is available for students and teachers to review and discuss later.

The above-mentioned detection of pedestrians and identification of pedestrian actions can be accomplished by using any image processing algorithm or machine learning algorithm. For example, the captured brightness image and depth image can be output to a machine learning model such as decision tree, random forest, multi-level neural network, convolutional neural network, support vector machine, etc. Etc., the present invention is not limited to this, and the output of the machine learning model is the location of the pedestrian, whether it is a pedestrian, or the user's actions. In this embodiment, the information in the brightness image and the depth image is combined, so that better detection and recognition results can be obtained, especially for the depth image, a new feature extraction method is proposed.

FIG. 2 is a schematic diagram showing feature extraction of a brightness image and a depth image according to an embodiment. Please refer to Figure 2, image 201 is a brightness image of pedestrians, image 202 shows the gradient of the brightness image 201, image 203 is a depth image, and image 204 shows the gradient of the depth image. The brightness can be seen from images 202 and 204 Images and depth images have different recognition capabilities in different parts of the body. Generally speaking, when the depth is similar (for example, under the feet), the image 204 is less able to distinguish pedestrians and non-pedestrians, but the depth difference is very large. The image 204 can provide better recognition ability if it is large (such as the head), while the image 202 is the opposite. In other words, the images 202 and 204 can complement each other to improve the detection and recognition capabilities.

In this embodiment, pedestrians in the brightness image 201 are first detected, and any algorithm, such as a convolutional neural network, can be used here. The position of the pedestrian is represented as a bounding box. This bounding box can also be applied to the depth image. The pixels in the bounding box can be used to capture feature vectors, especially for brightness images and depth images. Eigenvectors. The feature vector of the brightness image 201 can use any conventional method, such as Histogram of oriented gradients (HOG), and the following method can be used for the depth image 204.

First, the depth image 204 represents the depth of each pixel of gray, for every two adjacent second depth according to the equation (1), (2) calculating a first gradient △ _x and the Y direction in the X direction Gradient △ _y .

Among them, D(x,y) represents the depth at coordinates (x,y), D(x+1,y) and D(x-1,y) are the two adjacent depths mentioned above, D(x,y+ 1) and D(x,y-1) are also two adjacent depths. Next, an angle θ is calculated according to the first gradient and the second gradient, as shown in the following equation (3).

It is worth noting that the angle θ is in the range of -180 degrees to 180 degrees. Since 0 to -180 degrees is equivalent to 180 to 360 degrees, it can also be said that the angle θ is in the range of 0 degrees to 360 degrees. The angle of this range is taken because the pedestrian's body is always closer to the sensor than the background. The vector (△ _x ,△ _y ) will point from the inside of the body to the outside of the body. Under this calculation, the left and right sides of the body There will be different angles θ on both sides, thereby providing richer features. Next, the angle is accumulated to one of multiple angle bins between 0 and 360 degrees. For example, if there are a total of 10 angle grooves, the first angle groove is 0 degrees to 36 degrees, the second angle groove is 36 degrees to 72 degrees, and so on; if a certain angle θ is 24 degrees, Then the value of the first angle slot will accumulate 1, if a certain angle θ is 50 degrees, the value of the second angle slot will accumulate 1, and so on, the present invention does not limit the number of angle slots. In addition, an angle θ can be calculated for every two adjacent depths mentioned above. Therefore, multiple angles θ can be calculated for the pixels in the bounding box, and these angles will be added to the corresponding angle slot, that is to say, each The value of the angle groove represents how many angles θ are in the corresponding range in the depth image. The values of these angle slots form a vector (for example, the length is 10) as the feature vector of the depth image.

In this embodiment, multiple preset actions are set, such as forward, backward, side shift, turn left, turn right, stop, etc. The next step is to determine which preset action the pedestrian's action belongs to. Here, the first classifier is used to calculate the multiple first confidence values of pedestrians corresponding to the above-mentioned multiple preset actions based on the feature vector of the brightness image, and the second classifier is also used to calculate the pedestrian correspondence based on the feature vector of the depth image. To the multiple second confidence values of the multiple preset actions, the first classifier and the second classifier can adopt any machine learning model. In order to express it more clearly, x _{k is} used to represent the k-th pedestrian in the following. Although there is only one pedestrian under normal circumstances, this system can also be applied to multiple pedestrians, so a more general mathematical expression is used, where k is a positive integer. In addition, assuming that there are multiple classifiers and multiple preset actions, p _i,j (x _k ) represents the confidence value of calculating the k-th pedestrian belonging to the j-th preset action according to the i-th classifier, in other words The first confidence value is p _1,1 (x _k ),p _1,2 (x _k ),..., the above second confidence value is p _2,1 (x _k ),p _2,2 (x _k ),..., where i, j, and k are positive integers.

The above-mentioned first confidence value and second confidence value will be combined to determine which action the user's action belongs to. Specifically, the confidence values of different classifiers for the k-th pedestrian and the j-th preset action will be combined into a feature vector, expressed as p _jk = (p _1j (x _k ),p _2j (x _k ), ..., p _mj (x _k )), where m represents the number of classifiers, which is 2 in this embodiment. From another perspective, the feature vector p _jk includes the first confidence value and the second confidence value of the k-th pedestrian corresponding to the j-th preset action. Next, the feature vector p _jk is input to a support vector machine to calculate the confidence value for the k-th pedestrian and the j-th preset action, expressed as q _j (x _k ). In detail, _{the calculation of the confidence value q j} (x _k ) is as shown in the following equation (4).

q _j (x _k )=Σ _s y _s α _s . K(p _jk ,p _js )+b (4)

Equation (4) is the conjugate form of the support vector machine, where p _js represents the s-th support vector corresponding to the j-th preset action after the support vector machine is trained, and y _s is the s-th support vector Corresponding label (whether the mark is the j-th preset action), α _s is the Lagrange multiplier corresponding to the s-th support vector, b is a constant, K(.,.) is the core function, However, those with ordinary knowledge in the field should understand the conjugate form of support vector machines, and will not be described in detail here. In some embodiments, the above-mentioned core function K(.,.) adopts a radial basis function (RBF), but the present invention is not limited thereto. Next, according to the following equation (5), it is possible to determine which preset action the k-th pedestrian belongs to. In short, it is the preset action with the maximum confidence value q _j (x _k ), and the maximum confidence value is expressed as w( x _k ).

w(x _k )=arg max _j (q _j (x _k )) (5)

Through the above calculation, the information in the reference brightness image and the depth image can be combined, so the recognition accuracy of the action can be improved.

Fig. 3 is a flowchart of a traffic safety learning method according to an embodiment. Referring to FIG. 3, in step 301, an image corresponding to the user is captured through the sensor. In step 302, the traffic situation is displayed through the display device. In step 303, the user's action is determined based on the above-mentioned image. In step 304, it is determined whether the action conforms to the traffic rules of the traffic situation to calculate a score. However, each step in FIG. 3 has been described in detail as above, and will not be repeated here. It is worth noting that each step in FIG. 3 can be implemented as multiple program codes or circuits, and the present invention is not limited thereto. In addition, the method in FIG. 3 can be used in conjunction with the above embodiments, or can be used alone. In other words, other steps can also be added between the steps in FIG. 7.

Although the present invention has been disclosed in the above embodiments, it is not intended to limit the present invention. Anyone with ordinary knowledge in the relevant technical field can make some changes and modifications without departing from the spirit and scope of the present invention. The protection scope of the present invention shall be subject to those defined by the attached patent application scope.

100: Traffic Safety Learning System

101: Teaching Trainer

102: User

110: calculation module

120: display device

130: Sensor

140: Cloud database

141: Parameters

142: Video data

143: History

Claims (8)

A traffic safety learning system includes: a display device; a sensor for capturing at least one image corresponding to a user; and a calculation module for communicating with the display device and the sensor , Used to display a traffic situation through the display device, determine an action of the user based on the at least one image, and determine whether the action conforms to a traffic rule of the traffic situation to calculate a score, wherein the at least one image includes A depth image and a brightness image, the depth image includes a plurality of depths, and the calculation module is configured to perform a plurality of steps: for every two adjacent depths, calculate the first of the two depths in an X direction Gradient and a second gradient in the Y direction, and calculate an angle based on the first gradient and the second gradient; accumulate the angle to one of a plurality of angle slots between 0 and 360 degrees; and The value of the angle slot is used as the feature vector of the depth image. The traffic safety learning system according to claim 1, wherein the calculation module only captures an upper limb image or a lower limb image in the at least one image, and the movement is an upper limb movement or a lower limb movement. The traffic safety learning system according to claim 1, wherein the traffic situation includes intersection, one-way turn, circle, straight, level crossing Road or T-shaped road, and the traffic situation includes at least one obstacle. The traffic safety learning system according to claim 1, wherein the calculation module further sets a plurality of preset actions and executes a plurality of steps: detecting a pedestrian in the brightness image; using a first classifier according to the brightness The image calculates the plurality of first confidence values of the pedestrian corresponding to the preset actions; using a second classifier to calculate the plurality of second confidence values of the pedestrian corresponding to the preset actions according to the depth image; and merge The first confidence values and the second confidence values are used to determine that the action of the user belongs to one of the preset actions. The traffic safety learning system according to claim 4, wherein the calculation module determines the action according to the following equations (1) and (2), w(x _k )=arg max _j (q _j (x _k )) (1 ) q _j (x _k )=Σ _s y _s α _s . K(p _jk ,p _js )+b (2) where x _k represents the pedestrian, and p _jk includes the first confidence value and the second confidence value of the pedestrian corresponding to the jth predetermined action among the predetermined actions Confidence value, p _js represents the s-th support vector corresponding to the j-th preset action after support vector machine training, y _s is the label corresponding to the s-th support vector, and α _{s is} the s-th support The Lagrange multiplier corresponding to the vector, b is a constant, K(.,.) is the core function, and w(x _k ) is the confidence value corresponding to the action of the pedestrian. A traffic safety learning method for a calculation module. The traffic safety learning method includes: capturing at least one image corresponding to a user through a sensor, wherein the at least one image includes a depth image and a brightness image , The depth image includes a plurality of depths; for every two adjacent depths, a first gradient in an X direction and a second gradient in a Y direction of the two depths are calculated, and according to the first gradient and the The second gradient calculates an angle; adds the angle to one of a plurality of angle slots between 0 and 360 degrees; uses the value of the angle slots as the feature vector of the depth image; displays a traffic through a display device Context; judging an action of the user based on the at least one image; and judging whether the action conforms to a traffic rule of the traffic situation to calculate a score. The traffic safety learning method according to claim 6, further comprising: capturing only the upper limb image or the lower limb image in the at least one image, wherein the movement is an upper limb movement or a lower limb movement. The traffic safety learning method according to claim 6, wherein the traffic situation includes an intersection, a one-way turn, a circle, a straight, a level crossing, or a T-shaped road, and the traffic situation includes at least one obstacle.

Images