JP7306303B2 - judgment device - Google Patents

Description

The present invention relates to a judging device for judging the safety of bus passengers.

In order to prevent accidents such as occupants falling over on buses, it has been proposed to determine whether or not there is an upright occupant in the vehicle, and to determine that there is a danger if there is an upright occupant ( For example, see Patent Document 1).

JP 2016-107817 A

The occupant can maintain a relatively stable posture by holding a support such as a handrail while standing. However, with the technique described in Patent Document 1, there is a problem that it cannot be determined whether or not the occupant is holding the support, and thus the safety of the occupant cannot be accurately determined.

SUMMARY OF THE INVENTION The present invention has been made in view of this point, and an object of the present invention is to provide a determination device capable of improving the accuracy of determination of the safety of the occupants in the passenger compartment.

A determination device according to a first aspect of the present invention includes an acquisition unit that acquires a captured image of the inside of a bus, and a detection unit that detects a standing passenger based on the captured image acquired by the acquisition unit. a usage status specifying unit for specifying the usage status of the support member provided on the bus for supporting the passenger detected by the detection unit; and a determination unit that determines whether the occupant is in danger based on the above.

The detection unit detects the position of the hand of the occupant in a standing state, and the use status identification unit detects whether the occupant uses the support when the height of the hand is equal to or higher than the suspension height. It may be identified as a situation where The detection unit detects the position of the hand of the occupant in a standing state, and the use condition identifying unit identifies the use condition of the support device by the occupant based on the amount of displacement of the hand within a predetermined time. may

The determination device further includes a support identification unit that identifies a structure having a predetermined color in the captured image as the support, and the detection unit detects the position of the occupant's hand in the standing state, The usage status specifying unit specifies the usage status of the support device of the occupant based on the position of the support device specified by the support device specification unit and the position of the hand detected by the detection unit. good.

The detection unit is a support identification unit that identifies a structure having an edge as a support by identifying the positions of the hands of the occupant and analyzing the captured image in a state in which the occupant is not on the bus. and a storage unit that stores history information indicating a history of positions of the hands detected by the detection unit in the plurality of past captured images, wherein the support identifying unit stores the support by dividing the support into a plurality of segments. , and based on the number of times each of the plurality of hand positions indicated by the history information matches the position of the segment, the effective segment of the support is specified, and the usage status specifying unit is configured to: When the acquisition unit acquires the captured image, the usage status of the support device by the occupant is specified based on the position of the hand specified by the detection unit and the specified effective segment position in the captured image. may The detection unit detects an attribute of the occupant, and the determination unit determines whether the occupant is in danger based on whether the attribute of the occupant detected by the detection unit satisfies a predetermined condition. good.

ADVANTAGE OF THE INVENTION According to this invention, it is effective in improving the precision of a passenger|crew's safety determination in a vehicle interior.

It is a figure which shows the outline|summary of the determination apparatus of embodiment. 3 is a diagram showing the configuration of a bus; FIG. It is a figure which shows an example of the displacement amount of a hand. It is a flow chart which shows a processing procedure of judgment of passenger safety of a bus by a judging device. FIG. 11 is a diagram for explaining a method of identifying valid segments by a support identifying unit;

[Outline of determination device]
FIG. 1 is a diagram showing an overview of the determination device of this embodiment. FIG. 1 shows a captured image of the inside of a bus captured by an imaging unit mounted on a bus. The determination device is implemented by, for example, an ECU (Electronic Control Unit). The determination device detects the occupant of the bus based on the captured image generated by the imaging unit. The determination device identifies whether the detected occupant is in a standing state without sitting on the seat. Buses are provided with a plurality of supports for supporting passengers. Supports are, for example, straps or handrails. The determination device identifies the usage status of the support of the standing occupant based on the captured image generated by the imaging unit. The determination device determines whether the occupant is in a dangerous state based on the specified usage situation. In the example of FIG. 1, the occupant A in the front standing position holds the support. The determination device determines that the occupant A is not in a dangerous state because the occupant A in the standing state is holding the support.

The determination device specifies the usage status of the support, and determines whether the occupant is in a dangerous state based on the specified usage status. In this way, the determination device can suppress erroneous determination that the occupant is in a dangerous state when the occupant is holding the support and is in a safe state. Therefore, the determination device can improve the accuracy of determination of passenger safety.

[Bus configuration]
FIG. 2 is a diagram showing the configuration of the bus 100. As shown in FIG. A bus 100 includes an imaging unit 1 , a display unit 2 and a determination device 3 . The determination device 3 includes a storage section 31 and a control section 32 .

The imaging unit 1 generates a captured image of the inside of the bus. In the example of FIG. 1, the imaging unit 1 images the interior of the bus from the rear of the bus, but may also image the interior of the bus from the front of the bus. Also, a plurality of imaging units may be provided on the bus.

The display unit 2 is a display for displaying characters and the like. The storage unit 31 is composed of, for example, a ROM (Read Only Memory) and a RAM (Random Access Memory). The control unit 32 is, for example, a CPU (Central Processing Unit). The control unit 32 functions as an acquisition unit 321 , a detection unit 322 , a support identification unit 323 , a use state identification unit 324 , a determination unit 325 and a notification unit 326 by executing programs stored in the storage unit 31 .

The acquisition unit 321 acquires the captured image inside the bus 100 generated by the imaging unit 1 . Acquisition unit 321 acquires a captured image at predetermined time intervals. When a plurality of imaging units 1 are provided inside the bus 100 , the acquisition unit 321 acquires captured images generated by the respective imaging units 1 .

[Occupant detection]
The detection unit 322 detects the standing occupant based on the captured image acquired by the acquisition unit 321 . First, the detection unit 322 detects an occupant inside the bus 100 by a known technique based on the captured image. The detection unit 322 identifies whether or not the detected occupant is standing. The storage unit 31 stores in advance area information indicating a seating area where seats are arranged. The detection unit 322 reads the area information from the storage unit 31 . The detection unit 322 identifies the occupant detected at a position other than the seat area indicated by the read area information as standing. On the other hand, the detection unit 322 identifies the occupant detected at the position within the seat area indicated by the area information as being seated on the seat. Further, the detection unit 322 may detect the attributes of the occupant by performing image processing on the captured image acquired by the acquisition unit 321 using a known method. Attributes are, for example, sex, age, height, build, and the like. For example, the detection unit 322 may detect the age, which is one of the attributes, by performing face authentication on a standing passenger in order to determine whether the passenger is in danger, considering the age of the passenger. .

The detection unit 322 detects the position of the hand of the standing occupant. For example, the detection unit 322 uses the captured image generated by the imaging unit 1 as input data and the posture information including the position of the occupant's hand as output data, using a trained machine learning model to automatically extract the occupant's posture information. Output to the learning model. When a plurality of passengers are included in the captured image, the detection unit 322 causes the machine learning model to output the posture information of the plurality of passengers. The detection unit 322 may cause the machine learning model to output posture information including positions of other parts of the occupant such as shoulders and feet in addition to the positions of the occupant's hands.

[Identification of the position of the support]
The support identification unit 323 identifies positions of a plurality of supports inside the bus 100 . For example, the support identifying unit 323 analyzes a captured image (hereinafter, also referred to as a background image) generated by the imaging unit 1 and acquired by the acquiring unit 321 in a state in which no passenger is on the bus, thereby identifying edges. Identify the structure as a support. Further, the support specifying unit 323 may specify, as a support, a structure having a predetermined color in the background image among the specified structures. The predetermined color is, for example, orange or yellow.

The support specifying unit 323 generates first setting information for setting a display area corresponding to the position of the support on the captured image, and causes the storage unit 31 to store the first setting information in advance. When the acquisition unit 321 acquires the captured image, the support specifying unit 323 reads the first setting information from the storage unit 31 and specifies the position of the support based on the read first setting information.

Note that the support specifying unit 323 may set the position of the support by receiving the position of the support from the administrator. For example, the support specifying unit 323 sets the position of the support by displaying a background image on the display unit 2 and receiving the position of the support from the administrator.

[Identification of usage]
The use state identifying unit 324 identifies the use state of the supporting device of the standing occupant detected by the detecting unit 322 . For example, the usage status specifying unit 324 specifies the usage status of the support tool of the standing occupant based on the position of the support tool specified by the support tool specifying part 323 and the position of the hand specified by the detection unit 322. . More specifically, if the position of the support and the position of the hand of the occupant in the standing state match, the use status identifying unit 324 identifies that the occupant is using the support. If the position of the support does not match the position of the hand of the occupant in the standing state, the use state identification unit 324 does not identify that the occupant is using the support. In addition, if the distance between the position of the support and the position of the passenger's hand in the standing state is within a predetermined distance, the use status identifying unit 324 identifies that the occupant is using the support. good too. The predetermined distance is the maximum possible distance when the occupant's hand grips the support.

When the support is a strap, the ring moves within a predetermined range of motion, so the distance between the position of the occupant's hand and the position of the support specified by the support specifying unit 323 may fluctuate. . Therefore, the use state identification unit 324 may identify that the occupant is using the support when the height of the hand detected by the detection unit 322 is equal to or higher than the suspension height. The usage status identification unit 324 uses, for example, a value indicating a position higher than the handrail as the suspension height. The usage status identification unit 324 may use the shoulder height of the occupant as the suspension height. In this manner, the usage status identification unit 324 can improve the accuracy of identifying the usage status of the occupant's support tool when the support tool includes a strap.

[Specification of usage status based on displacement]
The usage status identification unit 324 may identify the usage status of the occupant's support device based on the amount of displacement of the hand detected by the detection unit 322 within a predetermined period of time. FIGS. 3A and 3B are diagrams showing an example of the amount of displacement of the hand. FIG. 3A is a diagram showing an example of occupant posture information output by the detection unit 322 to a learned machine learning model. In the example of FIG. 3A, the positions of the occupant's hands, elbows, shoulders, and other parts included in the posture information are indicated by circles.

FIG. 3(b) is a diagram showing posture information corresponding to a captured image generated at a timing later than the captured image used to generate the posture information of FIG. 3(a). FIG. 3(b) shows the situation when the passenger loses his balance due to sudden braking of the bus. In FIG. 3(b), for comparison with FIG. 3(a), the hand positions corresponding to FIG. 3(a) are indicated by hatched circles. In FIG. 3(b), arrows indicate the displacement of the hand position with reference to the position of the hand shown in FIG. 3(a).

The use state identifying unit 324 identifies that the occupant is using the support when the amount of displacement of at least one hand of the occupant within a predetermined period of time is equal to or less than the first reference value. It is assumed that the usage status identification unit 324 obtains the sum total of distances that the occupant's hands have moved within a predetermined period of time as the amount of displacement.

For example, for a plurality of occupants holding the support in the history information stored in the storage unit 31, the usage status identification unit 324 determines the amount of displacement of the hand holding the support within a predetermined time. Obtain the average value of the gripper. The usage status identification unit 324 obtains the non-holding person average value, which is the average of the amount of displacement of the hands of the occupants who do not hold the support in the history information within a predetermined period of time. The usage status identification unit 324 uses the intermediate value between the average value of the holder and the average value of the non-holder as the first reference value. The use state identifying unit 324 identifies that the occupant is not using the support when the amount of displacement of both hands of the occupant within a predetermined period of time exceeds the first reference value.

In this manner, the use status identification unit 324 determines whether or not the occupant is using the support without using the positional information of the support. Therefore, the administrator can omit the work of setting the position of the support. In addition, if the average amount of displacement of each part of the occupant's whole body, such as the shoulders and legs, within a predetermined period of time exceeds the second reference value, the usage status identifying unit 324 determines that the occupant is not using the support. It may be specified that there is

For example, the usage status identification unit 324 obtains the average value of the amount of displacement of each part of the whole body of the occupant holding the support in the history information within a predetermined time period for each occupant, and calculates the calculated average value for the plurality of occupants. The average value is obtained as the gripper average value. The usage status identification unit 324 obtains the average value of the amount of displacement of each part of the whole body of the occupant who is not gripping the support within a predetermined period of time for each occupant, and calculates the average value of the obtained average values for a plurality of occupants who are not gripping the support. It is calculated as the average value for each individual. The usage status identification unit 324 uses the intermediate value between the average value of the holder and the average value of the non-holder as the second reference value.

[Determination of Dangerous State]
The determination unit 325 determines whether the occupant inside the bus 100 is in a dangerous state based on the usage status of the supporting device of the standing occupant identified by the detection unit 322 . The determination unit 325 determines that the occupant is in a dangerous state on the condition that the usage state identification unit 324 identifies that the occupant in the standing state identified by the detection unit 322 is not using the support. judge. The determination section 325 outputs the determination result to the notification section 326 .

Further, the determining unit 325 determines whether the occupant is in a dangerous state on the condition that the occupant in the standing state is not using the support, but the present invention is not limited to this. The determination unit 325 may further determine whether the occupant is in a dangerous state based on whether the attribute of the occupant detected by the detection unit 322 satisfies a predetermined condition. Attributes are, for example, sex, age, height, build, and the like. For example, the determination unit 325 determines that the occupant in the standing state is not using the support, and the age of the occupant detected by the detection unit 322 is greater than or equal to the first age or less than the second age. , it may be determined that the occupant is in a dangerous condition. The first age is, for example, 65 years old. The second age is an age lower than the first age. The second age is, for example, 12 years old.

On the other hand, even when the determination unit 325 identifies that the occupant in the standing state is not using the support, the age of the occupant detected by the detection unit 322 is less than the first age and the age of the occupant is less than the first age. If the child is two years old or older, it may be determined that the child is not in a dangerous condition. In this way, the determination unit 325 can suppress excessive notifications by excluding occupants in the age group who are less likely to be seriously injured due to a fall or the like from the risk determination.

The notification unit 326 displays a message indicating that the occupant of the bus 100 is in a dangerous condition on the display unit 2 when the determination unit 325 determines that the occupant of the bus 100 is in a dangerous condition.

[Processing procedure for safety judgment by judging device]
FIG. 4 is a flow chart showing a processing procedure for judging the safety of the passengers of the bus 100 by the judging device 3. As shown in FIG. This processing procedure is started, for example, while the bus 100 is running with passengers on board.

First, the acquiring unit 321 acquires a captured image of the inside of the bus 100 generated by the imaging unit 1 (S101). The detection unit 322 detects the occupant of the bus 100 based on the captured image acquired by the acquisition unit 321 (S102). The determination unit 325 determines whether or not the detection unit 322 has identified that the occupant is standing (S103). When the detection unit 322 identifies that the occupant is in the standing state (YES in S103), the usage status identification unit 324 identifies the usage status of the occupant's support (S104).

The determination unit 325 determines whether or not the use state identification unit 324 has identified that the occupant is using the support (S105). If the determination unit 325 determines that the usage situation identification unit 324 has not identified that the occupant is using the support (NO in S105), the determination unit 325 determines that the occupant is in a dangerous state ( S106). The determination unit 325 determines whether or not there is another passenger detected by the detection unit 322 (S107). If the determination unit 325 determines that there is no other passenger detected by the detection unit 322 in the determination of S107 (NO in S107), the process ends.

If the detection unit 322 determines that the passenger is seated in the determination of S103 (NO in S103), the determination unit 325 proceeds to determination of S107. If the determination unit 325 determines that the use state identification unit 324 has identified that the occupant is using the support in the determination of S105 (YES in S105), the determination unit 325 proceeds to the determination of S107. If the determination unit 325 determines that there is another passenger detected by the detection unit 322 in the determination of S107 (YES in S107), the determination unit 325 returns to the determination of S103.

[Effects of the determination device of the present embodiment]
According to the present embodiment, the usage status identification unit 324 identifies the usage status of the support, and the determination unit 325 determines whether the occupant is in a dangerous state based on the usage status identified by the usage status identification unit 324. judge. Therefore, the determination unit 325 can suppress erroneous determination that the occupant is in a dangerous state when the occupant is holding the support and is in a safe state. Judgment accuracy can be improved.

In the present embodiment, an example will be described in which the notification unit 326 displays a message indicating that the occupant is in a dangerous state on the display unit 2 when the determination unit 325 determines that the occupant is in a dangerous state. bottom. However, the invention is not limited to examples of displaying messages. For example, when the determination unit 325 determines that the occupants are in a dangerous state, the behavior control unit (not shown) may limit driving such as sudden acceleration of the bus 100 that may cause the passengers to collapse. . For example, the behavior control unit may impose an upper limit on the acceleration due to rapid acceleration of the bus 100 when the determination unit 325 determines that the passenger is in a dangerous state. On the other hand, the behavior control unit does not impose an upper limit on the acceleration due to sudden acceleration of the bus 100 when the determination unit 325 does not determine that the occupant is in a dangerous state.

The behavior control unit may limit sudden braking that may cause the passenger to fall down, but may not limit sudden braking if there is a risk of collision or the like. When a message indicating that the occupant is in danger is displayed on the display unit 2, the driver brakes the bus 100 early while considering that the occupant may fall due to sudden braking or the like. to stop.

<Modified Example for Identifying Effective Segments of Support Tool>
A structure having an edge identified as a support by the support identification unit 323 may include a portion that is not suitable for the passenger to grip. Therefore, the support specifying unit 323 may divide the support into a plurality of segments, and among these segments, the segment arranged at a height equal to or higher than a predetermined value may be set as the effective segment of the support.

First, the support identifying unit 323 divides the identified structure into a plurality of segments. For example, the support specifying unit 323 divides the support into a plurality of segments so that the number of pixels included in each segment is as constant as possible. Of these segments, the support specifying unit 323 determines that the segments arranged at a height equal to or higher than a predetermined value are valid segments of the support. The support specifying unit 323 generates second setting information for setting a display area corresponding to the position of the effective segment included in the support in the captured image. The support identifying unit 323 uses, for example, a value corresponding to a height lower than the armrest of the seat as the predetermined value.

In addition, the support specifying unit 323 generates second setting information for setting a display area corresponding to the position of the effective segment of the support that is frequently gripped on the captured image, and stores this second setting information in the storage unit. 31 may be stored. The storage unit 31 stores history information indicating a history of hand positions detected by the detection unit 322 in a plurality of past captured images. For example, the storage unit 31 stores history information corresponding to captured images generated while the bus 100 is running for the last several weeks. This history information is created, for example, by tracking the positions of a plurality of hands detected by the detection unit 322 for a certain period of time or more.

The support specifying unit 323 determines that the occupant is gripping the support when each of the positions of the plurality of hands indicated by the history information matches the position of the segment of the support, and counts the number of grips. . The support identifying unit 323 identifies valid segments based on the counted number of grips. For example, the support identifying unit 323 identifies, as valid segments, segments for which the number of gripping times is equal to or greater than a threshold. The support identifying unit 323 determines the threshold based on, for example, the number of captured images used to create the history information and the total number of passengers included in the captured images.

FIG. 5 is a diagram for explaining a method of identifying effective segments by the support identifying unit 323. As shown in FIG. FIG. 5 shows segments A to G obtained by dividing the support. A range R in which many hand positions are distributed in the history information stored in the storage unit 31 is indicated by hatching. Since the segments A to D overlap with the range R, it is assumed that the number of gripping times of the segments A to D is equal to or greater than the threshold. In this case, the support identifying unit 323 identifies segments A to D as valid segments. On the other hand, since segments E to G do not overlap range R, it is assumed that the number of gripping times of segments E to G is less than the threshold. In this case, the support identifying unit 323 does not identify segments E to G as valid segments.

The support specifying unit 323 causes the storage unit 31 to store the second setting information indicating the display area of the specified valid segment on the captured image. When the acquisition unit 321 acquires the captured image, the support specifying unit 323 may read the second setting information from the storage unit 31 and specify the effective segment of the support based on the read second setting information.

The use state identifying unit 324 identifies the use state of the support device by the occupant using the position of the effective segment of the support device instead of the position of the support device. When the acquisition unit 321 acquires the captured image, the usage status identification unit 324 detects the position of the hand identified by the detection unit 322 and the position of the effective segment of the support identified by the support identification unit 323 in the captured image. to identify the use of occupant support.

For example, the usage status identification unit 324 identifies the usage status of the occupant's support device based on whether or not the position of the effective segment of the support device matches the position of the hand. In this way, the use status identifying unit 324 identifies whether or not the occupant is gripping the effective segment of the support, so that the captured image is generated when the occupant's hand accidentally overlaps a part of the support. In some cases, it is possible to suppress erroneous identification that the occupant is using the support.

In the present embodiment, the detection unit 322 uses a trained machine learning model that uses the captured image generated by the imaging unit 1 as input data and the posture information including the position of the occupant's hand as output data to detect the posture of the occupant. An example of outputting information to a machine learning model was described. The detection unit 322 may detect the occupant using the same machine learning model.

Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes are possible within the scope of the gist thereof. be. For example, all or part of the device can be functionally or physically distributed and integrated in arbitrary units. In addition, new embodiments resulting from arbitrary combinations of multiple embodiments are also included in the embodiments of the present invention. The effect of the new embodiment caused by the combination has the effect of the original embodiment.

1 imaging unit 2 display unit 3 determination device 31 storage unit 32 control unit 100 bus 321 acquisition unit 322 detection unit 323 support tool identification unit 324 usage identification unit 325 determination unit 326 notification unit

Claims (5)

an acquisition unit that acquires a captured image of the inside of the bus;
a detection unit that detects an occupant in a standing state based on the captured image acquired by the acquisition unit and specifies the position of the occupant's hand ;
a usage status identification unit that identifies the usage status of the support member provided on the bus for supporting the occupant detected by the detection unit;
a determination unit that determines whether the occupant is in danger based on the usage situation specified by the usage situation specifying unit;
a storage unit that stores history information indicating a history of positions of the hand detected by the detection unit in a plurality of past captured images;
A structure having an edge is identified as a support by analyzing the captured image in a state where no occupant is on the bus, the support is divided into a plurality of segments, and a plurality of segments indicated by the history information are identified. a support identification unit that identifies valid segments of the support based on the number of times each of the hand positions and the position of the segment match,
When the acquisition unit acquires the captured image, the usage situation specifying unit determines the position of the occupant based on the position of the hand specified by the detection unit and the specified position of the effective segment in the captured image. identify the use of supports,
judgment device. The usage status identifying unit identifies that the occupant is using the support when the height of the hand is equal to or higher than the suspension height,
The determination device according to claim 1. The usage status identifying unit identifies the usage status of the support device by the occupant based on the amount of displacement of the hand within a predetermined time.
The determination device according to claim 1 . further comprising a support identification unit that identifies a structure having a predetermined color in the captured image as the support ,
The usage status identifying unit identifies the usage status of the support device of the occupant based on the position of the support device identified by the support device identification unit and the position of the hand detected by the detection unit. ,
The determination device according to claim 1 or 2 . The detection unit detects attributes of the occupant,
The determination unit determines whether the occupant is in danger based on whether the attribute of the occupant detected by the detection unit satisfies a predetermined condition.
The determination device according to any one of claims 1 to 4 .

Images