JP2022158041A - Vehicle interior monitoring system and vehicle operation management system - Google Patents

Abstract

To provide a vehicle interior monitoring system and a vehicle operation management system that can determine whether or not there is space to ease congestion in a vehicle interior.SOLUTION: A ceiling center camera 18 (first imaging unit) images a plan view image of a vehicle interior from a center of a ceiling in the vehicle interior. A side camera 19 (second imaging unit) images a side view image of the vehicle interior from a peripheral edge of the ceiling or a side of a vehicle interior 40. A vehicle interior image recognition unit 36 recognizes a person and luggage included in the plan view image and the side view image. A congestion estimation unit 37 obtains, on the basis of the recognized person and luggage, a congestion rate of the vehicle interior. The vehicle interior image recognition unit 36 obtains a plan view section of the person and luggage recognized in the plan view image, and obtains a side view section of the person and luggage recognized in the side view image. A volume calculation unit 37E obtains, on the basis of the plan view section and the side view section of the person and luggage, volume of the person and luggage. A congestion rate calculation unit 37G calculates a congestion rate on the basis of a difference between total volume of the person and luggage and volume of the vehicle interior 40.SELECTED DRAWING: Figure 2

Description

Disclosed herein is a vehicle interior monitoring system and a vehicle operation management system including the system.

For example, Patent Literature 1 and Patent Literature 2 disclose a monitoring system for calculating the boarding ratio of passenger vehicles such as trains and buses. In this system, multiple surveillance cameras are provided above the vehicle. These surveillance cameras are arranged at regular intervals, and are installed so as to capture images from above to directly below.

Then, a person is detected from the image data captured by each surveillance camera, and the ratio of the area in the vehicle occupied by the person (occupancy rate) is calculated. The occupancy rate of the entire vehicle is obtained from the occupancy rate based on the image data of each surveillance camera.

JP 2013-25523 A JP 2012-146022 A

By the way, for example, if the luggage in the passenger compartment is short and short, moving it and placing it between the legs of the standing passengers will reduce the occupancy rate of the passenger compartment, in other words, alleviate congestion. I can expect it. However, height information of passengers, luggage, and the like is not reflected in a planar view image of the inside of the vehicle, which is captured by looking down from above the vehicle compartment.

Therefore, in this specification, a vehicle interior monitoring system and a vehicle operation management system including the system are disclosed, which can determine whether or not there is room for alleviating congestion in the vehicle interior.

A vehicle interior monitoring system is disclosed herein. The system comprises a first imager, a second imager, an image recognizer, and a congestion estimator. The first imaging device captures a planar view image of the interior of the vehicle from the central portion of the ceiling of the vehicle. The second imaging device captures a side-view image of the vehicle interior from the peripheral edge of the ceiling or the side of the vehicle interior. The image recognition unit recognizes a person and luggage included in the plan view image and the side view image. The congestion estimating unit obtains the congestion rate of the passenger compartment based on the recognized persons and luggage. The image recognition unit obtains a plan view section of the person and the package recognized in the plan view image, and obtains a side view section of the person and package recognized in the side view image. The congestion estimation unit includes a volume calculation unit and a congestion rate calculation unit. The volume calculation unit obtains the volume of the person and the package based on the plan view section and the side view section of the person and the package. The congestion rate calculation unit obtains the congestion rate based on the difference between the total volume of people and luggage and the volume of the passenger compartment.

According to the above configuration, in addition to the area occupied by people and luggage in a plan view, the volume-based congestion rate that takes into account height information is obtained. In the process of calculating the congestion rate, it is possible to extract short luggage by obtaining height information especially from the side view section of the luggage.

Moreover, in the above configuration, the congestion estimation unit may include a depth information acquisition unit and a coordinate conversion unit. The depth information acquisition unit acquires depth information of the plane view section of the person and the package along the optical axis of the first imaging device, and obtains depth information of the side view section of the person and the package along the optical axis of the second imaging device. depth information. The coordinate transformation unit transforms the plane coordinates of the plan view section and the side view section and the three-dimensional coordinates on the camera coordinate system based on the depth information into the three-dimensional coordinates on the world coordinate system, which is the coordinate system of the vehicle interior space. The volume calculator obtains the volume of the person and the package based on the plan view section and the side view section transformed into the world coordinate system.

In the two-dimensional image and the side view image, persons and items relatively close to the first imager and second imager appear larger than persons and items relatively far from these imagers. . The size of the images of the person and the luggage due to the distance to the first/second imaging device is eliminated by transforming the camera coordinate system into the world coordinate system.

Further, in the above configuration, the image recognition unit may set bounding boxes, which are rectangular areas surrounding the person and the luggage, obtained based on an SSD (Single Shot Multibox Detector) algorithm in the plan view section and the side view section. .

According to the above configuration, it is possible to obtain the volume of the person and the package using an existing algorithm.

Further, in the above configuration, the congestion estimation unit may include an adjacent baggage extraction unit. The proximate luggage extraction unit extracts proximate luggage, which is luggage whose volume overlaps the volume of the person by a predetermined ratio or more in the space of the world coordinate system. The congestion rate calculation unit excludes the extracted proximate parcels from the congestion rate calculation target.

Unlike the case where the volumetric regions of a person or a package are determined by their contours, when surrounded by a bounding box, if the person and the package are close to each other, their volumetric regions overlap each other. By excluding parcels with a large number of overlapped areas from the calculation of the congestion rate, it is possible to prevent the congestion rate from becoming excessively high.

Further, in the above configuration, the congestion estimation section may include a short baggage search section. When the congestion rate obtained by the congestion rate calculation section exceeds a predetermined upper limit threshold, the short baggage search section searches for the baggage for which the congestion rate is calculated in the side view section converted to the world coordinate system. Based on the height component, parcels with a height equal to or less than a predetermined threshold are searched.

According to the above configuration, it is possible to determine whether or not there is short luggage that can be moved between the legs of the standing passenger.

Further, in the above configuration, the congestion estimation unit may include an announcement control unit that outputs an announcement prompting the movement of the baggage into the vehicle interior when the baggage having a height equal to or lower than a predetermined threshold is found.

According to the above configuration, the output of the announcement promotes congestion relief.

Further, in this specification, a vehicle operation management system is disclosed that includes the vehicle interior monitoring system described above and an operation schedule adjustment section that can communicate with the vehicle interior monitoring system. The operation schedule adjusting unit can communicate with the congestion estimating unit of the vehicle interior monitoring system installed in each of the plurality of vehicles. Further, the operation schedule adjustment unit determines whether or not the operation schedule can be changed so that the train will pass through the stop at which the train is scheduled to stop next, based on the congestion rate received from each congestion estimation unit.

According to the above configuration, it is possible to operate according to the congestion rate, such as allowing a vehicle with a high congestion rate to pass through a stop scheduled to stop.

According to the vehicle interior monitoring system disclosed in this specification and the vehicle operation management system including the system, it is possible to determine whether or not there is room for alleviating congestion in the vehicle interior.

1 is a schematic diagram of a shared vehicle utilization system equipped with a vehicle interior monitoring system and an operation management system according to the present embodiment; FIG. FIG. 2 is a diagram illustrating functional blocks of the shared vehicle utilization system of FIG. 1; FIG. 3 is a functional block diagram illustrating subunits forming a congestion estimator; 1 is a perspective view illustrating an appearance of a passenger vehicle; FIG. It is a perspective view (perspective) which illustrates the passenger compartment of a passenger vehicle. It is a figure explaining camera arrangement in the vehicle interior of a passenger vehicle. It is a congestion rate estimation flow (1/2) performed by the vehicle interior monitoring system which concerns on this embodiment. 2 is a congestion rate estimation flow (2/2) executed by the vehicle interior monitoring system according to the present embodiment. It is a figure which illustrates a side view captured image. It is a figure which shows the example when image recognition based on an SSD algorithm is performed with respect to the side view captured image. It is a figure which illustrates a planar view captured image. It is a figure which shows the example when image recognition based on an SSD algorithm is performed with respect to a planar view captured image. FIG. 4 is a diagram for explaining inclinations in a vehicle interior space of a bounding box extracted from a planar-view captured image and a bounding box extracted from a front-view captured image; FIG. 4 is a diagram explaining a process of obtaining a volume from a bounding box; It is a figure which illustrates an operation adjustment flow performed by the operation management system concerning this embodiment. It is a figure (rear view) explaining another example of the matching process of the person and the luggage|packaging recognized from the planar view captured image, and the person and luggage|load recognized from the front view captured image. FIG. 11 is a diagram (plan view) explaining another example of the process of associating the person and the package recognized from the planar view captured image with the person and package recognized from the front view captured image. FIG. 11 is a functional block diagram of a system for utilizing shared vehicles, in which a vehicle interior monitoring system and an operation management system according to another example of the present embodiment are installed; It is a figure which illustrates an operation adjustment flow performed by the operation management system concerning another example of this embodiment.

Embodiments of the present invention are described below with reference to the drawings. The shapes, materials, numbers, and numerical values described below are examples for explanation, and can be changed as appropriate according to the specifications of the vehicle interior monitoring system and the vehicle operation management system including the system. In the following, like elements are given the same reference numerals in all drawings.

<Overall composition>
FIG. 1 illustrates the overall configuration of a shared vehicle utilization system including a vehicle interior monitoring system and a vehicle operation management system according to this embodiment. Further, FIG. 2 illustrates functional blocks of the passenger vehicle utilization system. The shared vehicle utilization system includes a shared vehicle 10 , a center server 50 , and a mobile terminal 70 . The passenger vehicle 10, the center server 50, and the mobile terminal 70 can communicate with each other using communication means such as the Internet 95 or the like.

As will be described in detail below, the passenger vehicle 10 travels along a predetermined route and stops at stops provided along the route. The passenger vehicle 10 is equipped with a vehicle interior monitoring system. As illustrated in FIGS. 1 and 2 , the vehicle interior monitoring system includes a ceiling center camera 18 , a side camera 19 , a vehicle interior image recognition section 36 and a congestion estimation section 37 .

As will be described later, the vehicle interior monitoring system calculates the congestion rate in the vehicle interior. This congestion rate is transmitted to the operation schedule display section 82 (see FIG. 2) of the mobile terminal 70 via the center server 50 . By transmitting the congestion rate to the mobile terminal 70, the user of the mobile terminal 70 can determine whether or not to board the shared vehicle 10 before boarding.

In the center server 50, the operation schedule adjusting unit 61 determines whether or not to change the operation schedule so that the train will pass through the next stop, based on the received congestion rate. For example, the operation schedule adjustment unit 61 changes the operation schedule so that the passenger vehicle 10 with a high congestion rate is allowed to pass through the stop at which it is scheduled to stop next. This suppresses a further increase in the congestion rate.

In the vehicle interior monitoring system, a ceiling center camera 18 provided in the vehicle interior of the passenger vehicle 10 captures a planar view image of the vehicle interior. Further, in this system, a side view image of the interior of the passenger compartment is captured by the side camera 19 provided in the interior of the passenger vehicle 10 . The occupied volume of the person (passenger) and luggage is obtained based on these planar view images and side view images. Furthermore, the congestion rate (volumetric congestion rate) is determined based on the difference between the sum of the occupied volumes of the people and luggage thus determined and the volume of the passenger compartment.

In addition, the vehicle interior monitoring system according to the present embodiment searches for short luggage that is relatively short. Then, when the congestion rate is high, an announcement is output in the passenger compartment prompting the movement of short luggage between the legs of standing passengers. Due to such movement of short luggage, the space in the passenger compartment previously occupied by short luggage becomes free space, thereby reducing congestion.

<mobile terminal>
The mobile terminal 70 is possessed by a user who is a user of the passenger vehicle 10 . The mobile terminal 70 may be, for example, a smartphone, and includes an input/output controller 71 that controls data input/output as its hardware configuration. The portable terminal 70 also includes a CPU 72 as an arithmetic device, and a ROM 75, a RAM 76, and a storage device 77 as storage units. The storage device 77 may be, for example, an SSD (Solid State Drive).

The mobile terminal 70 further includes an input section 73 and a display section 74 . For example, the mobile terminal 70 is provided with a touch panel in which the input unit 73 and the display unit 74 are integrated. Further, the mobile terminal 70 has a positioning section 78 and a clock 80 . The positioning unit 78 is, for example, a receiver that receives positioning signals from satellites in GNSS (Global Navigation Satellite System). Various hardware components of the mobile terminal 70 are connected to the internal bus 79 .

At least one of the ROM 75 and the storage device 77, which are storage devices, stores a program, and the CPU 72 executes the program to configure the bus application 81 illustrated in FIG. The bus application 81 has an operation schedule display section 82 and a congestion rate display section 83 .

The operation schedule display unit 82 displays the operation schedule of the shared vehicle 10 . For example, the operation schedule of the nearest stop to the current position of the mobile terminal 70 received by the positioning unit 78 is displayed. For example, an operation schedule for one hour after the current time obtained from the clock 80 is displayed. Further, as will be described later, the operation schedule display unit 82 displays a message to the effect that the shared vehicle 10 with a high congestion rate will pass through the stop.

The congestion rate display section 83 displays the congestion rate of the shared vehicle 10 . For example, the congestion rate of all the passenger cars 10 in operation is displayed. By displaying the congestion rate together with the operation schedule, the user can view the congestion rate of the nearest passenger vehicle 10 at the stop where the passenger is scheduled to board, and the congestion rate of the passenger vehicle 10 that stops next. It is possible to determine whether or not to board the passenger vehicle 10 according to the request.

<Center server>
Referring to FIG. 1, center server 50 is installed, for example, in a management company that provides a bus operation service using a plurality of passenger vehicles 10 . The center server 50 is composed of, for example, a computer (electronic computer). Referring to FIG. 1, center server 50 includes an input/output controller 51 for controlling data input/output as its hardware configuration. The center server 50 also includes a CPU 52 , an input section 53 , a display section 54 , a ROM 55 , a RAM 56 , a hard disk drive 57 (HDD), and a clock 59 . A storage device such as an SSD (Solid State Drive) may be used instead of the hard disk drive 57 . These components are connected to internal bus 58 .

At least one of the ROM 55 and the hard disk drive 57, which are storage devices, stores a program for managing operation. Execution of the program by the CPU 52 of the center server 50 or the like forms functional blocks in the center server 50 as illustrated in FIG.

That is, the center server 50 includes an operation schedule storage unit 65 and a dynamic map storage unit 66 as storage units. The center server 50 also includes an operation schedule adjustment unit 61, an operation schedule creation unit 62, an operation schedule provision unit 63, and an operation route creation unit 64 as functional units.

Here, each part of the operation schedule adjustment unit 61, the operation schedule creation unit 62, the operation schedule provision unit 63, the operation route creation unit 64, the operation schedule storage unit 65, and the dynamic map storage unit 66, and the above-described passenger vehicle 10 The onboard vehicle interior monitoring system is included in the vehicle operation management system according to the present embodiment.

Dynamic map data, which is map data, is stored in the dynamic map storage unit 66 . A dynamic map is a three-dimensional map, and stores, for example, the position and shape (three-dimensional shape) of a roadway. The three-dimensional shape of the roadway includes, for example, slope and width. The locations of lanes drawn on the roadway, pedestrian crossings, stop lines, etc. are also stored in the dynamic map. In addition, the positions and shapes (three-dimensional shapes) of structures such as stops, buildings, and traffic lights around the road are also stored in the dynamic map. In addition, parking lot locations and shapes are also stored in the dynamic map.

For example, dynamic maps use a geographic coordinate system that includes latitude and longitude. When the shared vehicle 10 automatically runs, the operation route creation unit 64 extracts dynamic map data from the dynamic map storage unit 66 and creates operation map data including travel routes and stop positions.

The operation schedule creation unit 62 creates an operation schedule (in other words, operation schedule) for the shared vehicle 10 . For example, when receiving the operation map data from the operation route creation unit 64, the operation schedule creation unit 62 calculates the estimated arrival time and Create an operational schedule that includes scheduled departure times. The created operation schedule is transmitted to the passenger vehicle 10 via the operation schedule providing unit 63 .

The operation schedule adjusting unit 61 is capable of communicating with a vehicle interior monitoring system installed in each of the plurality of passenger vehicles 10 . Based on the congestion rate received from the congestion estimation unit 37 of the vehicle interior monitoring system, the operation schedule adjustment unit 61 changes the operation schedule such that each passenger vehicle 10 passes through the next stop scheduled to stop. determine whether or not

For example, when the received congestion rate A% exceeds the upper limit threshold B%, the operation schedule adjustment unit 61 instructs the passenger vehicle 10 that has transmitted the congestion rate A% to pass through the next stop scheduled to stop. to change the operating schedule. The changed operation schedule is transmitted to the passenger vehicle 10 to be changed via the operation schedule providing unit 63 .

<Shared vehicle>
4 to 6 illustrate diagrams for explaining the structure of the passenger vehicle 10. FIG. 4 to 6, the longitudinal direction of the vehicle body is indicated by an axis indicated by symbol FR, and the vehicle width direction is indicated by an axis indicated by symbol RW (Right Width). Also, the vehicle height direction is indicated by an axis represented by symbol UP.

FIG. 4 illustrates a perspective view showing the external appearance of the front and left sides of the passenger vehicle 10. As shown in FIG. The passenger vehicle 10 is used as a passenger bus, and is provided with a pair of doors 41, 41 serving as an entrance/exit on the left side.

In addition, an exterior indicator 41A is provided in front of the vehicle. The vehicle exterior display 41A is also called a so-called digital signage, and is composed of a liquid crystal display or an LED display. As will be described later, when the congestion rate A% of the passenger vehicle 10 exceeds a predetermined upper limit threshold value B%, the congestion rate A% is displayed on the outside display 41A. Also, a message to the effect that the vehicle will pass through the stop at which the vehicle was scheduled to stop is displayed on the display outside the vehicle 41A.

FIG. 5 illustrates the inside of the passenger compartment 40 of the passenger vehicle 10. As shown in FIG. A plurality of seats 42 for passengers are provided along the wall surface in the passenger compartment 40 . A plurality of hangers 44 are provided on the ceiling to support standing guests. A handrail 46 is provided on the side wall of the compartment 40 .

A handrail 46 is provided so as to extend upward from the side wall of the compartment and run along the ceiling surface. A reference mark 48 is provided at the vehicle width direction central portion of the ceiling portion of the handrail 46 . A reference mark 48 indicates the origin position of a three-dimensional coordinate system in a world coordinate system, which will be described later.

The reference mark 48 is formed to have a size that allows the vehicle interior image recognition unit 36 (see FIG. 2) to recognize the mark in the images captured by the central ceiling camera 18 and the side cameras 19 (see FIG. 6). be. For example, the dimension of the reference mark 48 is determined so that it can be imaged even with the lower imaging resolution of the central ceiling camera 18 and the side cameras 19 .

Above the doors 41, 41, an in-vehicle indicator 41B is provided. The in-vehicle indicator 41B is composed of a liquid crystal display or an LED display, like the out-of-vehicle indicator 41A. As will be described later, during congestion, a message is displayed on the in-vehicle display 41B to prompt standing passengers to move short luggage between their legs. Furthermore, an in-vehicle speaker 45 is provided on the ceiling of the vehicle compartment 40 . As will be described later, during congestion, a voice message is output from the in-vehicle speaker 45 urging standing passengers to move short luggage between their legs.

A central ceiling camera 18 (first image pickup device) is provided in the central portion of the ceiling of the passenger compartment 40 . For example, the ceiling center camera 18 is provided at the center of the vehicle interior 40 in the vehicle width direction and at the center in the vehicle front-rear direction. For example, the ceiling center camera 18 has an image sensor such as a CMOS sensor or a CCD sensor.

For example, the ceiling center camera 18 may be a so-called 360° camera, and is capable of capturing an image of the entire passenger space of the passenger compartment 40 . For example, the central ceiling camera 18 includes the entire floor 43 of the passenger compartment 40 in its field of view.

The ceiling center camera 18 captures a plan view image of the vehicle interior 40 . For example, as shown in FIG. 11 to be described later, a plan view image of an angle looking down on the passenger compartment 40 from directly above is captured by the ceiling center camera 18 . As will be described later, image areas of a person (passenger) and luggage are recognized from this planar view image. In addition, planar sections of recognized persons and packages are determined.

Referring to FIG. 6 , a vehicle interior 40 is provided with a side camera 19 (second imaging device) in addition to the ceiling center camera 18 . The side camera 19 captures a side view image of the interior of the vehicle interior 40 from the peripheral edge of the ceiling of the vehicle interior 40 or from the side surface of the vehicle interior 40 . For example, the side camera 19 has an image sensor such as a CMOS sensor or a CCD sensor.

Note that the side view hereinafter is not limited to the viewpoint along the vehicle width direction (RW axis direction). In short, a mode in which the inside of the passenger compartment 40 is viewed in a substantially horizontal direction can be regarded as a side view. Geometrically, the three axes of UP, RW, and FR are used so that the angle formed with the RW and FR axes is smaller than the angle formed with the UP axis, that is, the RW or FR axis. A line-of-sight axis that seems to be tilted toward the axis side is widely perceived as a side view. Therefore, although the optical axis of the side camera 19 is inclined with respect to the FR axis and the RW axis, the angle of inclination is smaller than the angle of inclination with respect to the UP axis. treated as

In the example of FIG. 6, the side camera 19 is provided in front of the vehicle interior 40 and captures a front view image as a side view image. For example, a side view image as shown in FIG. 9 to be described later is captured by the side camera 19 . As will be described later, the image areas of the person (passenger) and luggage are recognized from this side view image. Additionally, a side view section of the recognized person and package is sought.

<Automatic driving control mechanism for shared vehicles>
Referring to FIGS. 1 and 2, passenger vehicle 10 is, for example, an automatic driving vehicle having an automatic driving function. For example, based on the standards of the Society of Automotive Engineers (SAE) of the United States, the passenger vehicle 10 is capable of operating from level 0 (driver performs all operations) to level 5 (fully automated). For example, when the passenger vehicle 10 is in operation, the level of automatic driving is set to level 4 or level 5, for example.

The shared vehicle 10 is used as a shared bus that stops at a stop and allows passengers to get on and off while automatically driving along a predetermined operation route. The passenger vehicle 10 is an electric vehicle that uses a rotating electric machine 17 (motor) as a drive source and a battery (not shown) as a power source. The passenger vehicle 10 also includes a steering mechanism 14B, a brake mechanism 14A, and an inverter 14C that controls the output of the rotating electric machine 17 as a travel control mechanism.

Furthermore, the passenger vehicle 10 includes an exterior camera 11A, a rider unit 11B, a proximity sensor 12, a positioning unit 13, and a control unit 20 as mechanisms for acquiring the position of the vehicle and grasping surrounding conditions.

For example, a passenger vehicle 10 is provided with sensor units 11 on its front surface, rear surface, and both side surfaces. The sensor unit 11 includes an exterior camera 11A and a lidar unit 11B.

The rider unit 11B is a sensor unit for automatic driving, and is a distance measuring unit capable of measuring the distance between the vehicle and objects around the vehicle. The lidar unit 11B uses LiDAR (Light Detection and Ranging), that is, a technique for measuring the distance to surrounding objects using laser light. The lidar unit 11B includes an emitter that emits infrared laser light toward the outside of the vehicle, a receiver that receives the reflected light, and a motor that rotates the emitter and receiver.

For example, the emitter emits infrared laser light toward the outside of the vehicle. When the laser light emitted from the emitter strikes an object around the passenger car 10, the reflected light is received by the receiver. The distance between the reflection point and the receiver is determined based on the time from illumination of the emitter to reception of light by the receiver. The emitter and receiver are rotated by a motor to scan the laser beam horizontally and vertically, thereby obtaining three-dimensional point cloud data about the surrounding environment of the passenger vehicle 10 .

The exterior camera 11A captures the same field of view as the lidar unit 11B. The vehicle exterior camera 11A includes an image sensor such as a CMOS sensor or a CCD sensor. The proximity sensors 12 are infrared sensors, for example, and are provided at the four corners of the passenger vehicle 10 in plan view, for example. For example, when the shared vehicle 10 arrives at the boarding place, the proximity sensor 12 detects a projecting object such as a sidewalk curbstone. This detection enables correct arrival control to stop the passenger vehicle 10 close to the curb. The positioning unit 13 is a system that performs positioning using artificial satellites, and uses, for example, GNSS similar to mobile terminals.

The control unit 20 may be, for example, an electronic control unit (ECU) of the passenger vehicle 10, and is composed of a computer (electronic calculator). As its hardware configuration, the control unit 20 includes an input/output controller 21 that controls data input/output. Moreover, the control part 20 is provided with CPU22, GPU23 (Graphics Processing Unit), and DLA24 (Deep Learning Accelerators) as an arithmetic unit. The control unit 20 also includes a ROM 25, a RAM 26, and a hard disk drive 27 (HDD) as storage units. A storage device such as an SSD (Solid State Drive) may be used instead of the hard disk drive 27 . These components are connected to internal bus 28 .

At least one of the ROM 25 and the hard disk drive 27, which are storage devices, stores a program for performing automatic operation control of the passenger vehicle 10. FIG. As the program is executed by the CPU 22 and the like of the control unit 20, functional blocks such as those illustrated in FIG. 2 are formed in the control unit 20. FIG.

That is, the control unit 20 includes a scan data analysis unit 30, a self-position estimation unit 31, and an autonomous driving control unit 33 as functional blocks for performing automatic driving control. The control unit 20 also includes a dynamic map storage unit 34 and an operation schedule storage unit 35 as storage units.

The scan data analysis unit 30 acquires an image captured by the exterior camera 11A. The scan data analysis unit 30 performs image recognition using a known deep learning method on the acquired captured image. By this image recognition, object detection and its attribute (vehicle, passerby, structure, etc.) recognition within the captured image are performed.

The scan data analysis unit 30 also acquires three-dimensional point cloud data from the lidar unit 11B. Further, the scan data analysis unit 30 creates peripheral data by superimposing the coordinates of the image-recognized captured image and the three-dimensional point cloud data. Peripheral data makes it possible to detect what kind of attribute an object is and how far it is from the passenger vehicle 10 . This peripheral data is sent to the autonomous travel control unit 33 .

The self-position estimation unit 31 acquires self-position information (latitude, longitude) from the positioning unit 13 . For example, the self-position estimation unit 31 acquires self-position information from an artificial satellite. Self-position information (self-vehicle position information) thus obtained is sent to the autonomous driving control unit 33 .

The dynamic map storage unit 34 stores operation route map data created by the operation route creation unit 64 of the center server 50 . This operational route map data includes dynamic map data. The operation schedule storage unit 35 also stores the operation schedule created by the operation schedule providing unit 63 of the center server 50 .

The autonomous driving control unit 33 uses the operation route map data stored in the dynamic map storage unit 34, the self-location information (vehicle location information) transmitted from the self-location estimation unit 31, and the information transmitted from the scan data analysis unit 30. Travel control of the passenger vehicle 10 is performed based on the acquired peripheral data. Also, when arriving at the stop, the shared vehicle 10 stops for a time according to the operation schedule.

<In-vehicle monitoring system>
The passenger vehicle 10 is equipped with a vehicle interior monitoring system. The hardware configuration of the system includes a central ceiling camera 18 (first image pickup device), a side camera 19 (second image pickup device), and a control section 20 . At least one of the ROM 25 and the hard disk drive 27, which are storage devices of the control unit 20, stores a program for monitoring the vehicle interior and estimating the congestion rate. As the program is executed by the CPU 22 and the like of the control unit 20, functional blocks such as those illustrated in FIG. 2 are formed in the control unit 20. FIG. That is, the control unit 20 includes a vehicle interior image recognition unit 36 and a congestion estimation unit 37 as part of the vehicle interior monitoring system.

The vehicle interior image recognition unit 36 recognizes the image areas of the person (passenger) and luggage from the plan view image captured by the ceiling center camera 18 and the side view image captured by the side camera 19 . For example, in the vehicle interior image recognition unit 36, an SSD (Single Shot Multibox Detector) using supervised learning is implemented as an image recognition algorithm.

Since the SSD algorithm is a known technology, it is briefly described here. The SSD algorithm performs position estimation and class estimation within the captured image. In other words, two types of estimation, ie, where the object is located in the image and what the attribute (class) of the object is, can be made in parallel, that is, in a single operation using a neural network (Single Shot ) is done.

In the SSD algorithm, instead of extracting the boundary line between the person and the package and the outside when detecting the person and the package, the boundary of the person and the package is defined by a rectangular area "bounding box" as illustrated in FIG. defined. For example, in the SSD algorithm, when estimating the position, a plurality of (for example, less than 10,000) rectangular frames with different sizes and shapes called default boxes are fitted on the image, and any of these frames can contain the object without excess or deficiency. is calculated. Furthermore, each bounding box displays the attribute (class) of the object captured in that box.

For example, according to the example of FIG. 10, in which image recognition processing is performed by the SSD algorithm on the side view image of FIG. is set. Further, bounding boxes 106A-106C (side view sections) of class "baggage" are set for the packages 102A-102C.

Further, for example, according to the example of FIG. 12, in which image recognition processing is performed by the SSD algorithm on the monoscopic image of FIG. 11, bounding boxes 108A to 108C (monoscopic section ) is set. Further, bounding boxes 110A-110C (planar view partitions) of class "baggage" are set for the packages 102A-102C.

Each bounding box is given a position parameter. That is, the box center coordinates C[Cx, Cy], the box width W, and the box height H are given as the position parameters of the bounding box. For this position parameter, a coordinate system on the image plane, that is, a value on the camera coordinate system is used.

Also, for example, the teacher data for the SSD algorithm includes a plurality of pairs of teacher data, each of which has a person image as input data and a class "Human" and a position parameter of a bounding box surrounding the person as output data. Further, the teacher data includes a plurality of pairs of teacher data, each of which has a package image as input data and a class "baggage" and a position parameter of a bounding box surrounding the package as output data.

The vehicle interior image recognition unit 36 may include a neural network implemented with an SSD algorithm for image recognition of planar images and a neural network implemented with an SSD algorithm for image recognition of side-view images. In this case, as training data for the former, images of a person and luggage viewed from above are used as input data. image is used as input data.

The vehicle interior image recognition unit 36 sets a bounding box, which is a rectangular area surrounding the person and the baggage, in the plan view section and side view section of the person and the baggage. The set bounding box is used for estimating the congestion rate.

Returning to FIG. 2 , the congestion estimation unit 37 obtains the congestion rate of the vehicle interior 40 based on the persons and luggage recognized by the vehicle interior image recognition unit 36 . The congestion estimator 37 comprises a plurality of subunits as exemplified in FIG. That is, the congestion estimation unit 37 includes a distortion/inclination correction unit 37A, a depth information acquisition unit 37B, a coordinate conversion unit 37C, a box combination extraction unit 37D, a volume calculation unit 37E, an adjacent luggage extraction unit 37F, a congestion rate calculation unit 37G, a short It has a baggage search section 37H and an announcement control section 37I.

The distortion/tilt correction unit 37A performs distortion correction and tilt correction as so-called camera calibration. For example, FIG. 11 illustrates a planar image captured by the central ceiling camera 18 . Due to the lens characteristics of the central ceiling camera 18 and the like, barrel-shaped distortion occurs in this planar view image. This distortion is eliminated by known distortion correction processing such as affine correction based on the lens characteristics of the central ceiling camera 18 .

Further, the distortion/tilt correction section 37A performs tilt correction. As illustrated in FIG. 13 , an imaging area 120 captured by the central ceiling camera 18 and an imaging area 122 captured by the side cameras 19 are tilted from an angle directly above if the central ceiling camera 18 is used, and if the side cameras 19 It is different from the angle from the side.

For example, when a perfect circle is drawn on the floor of the passenger compartment 40 and the central ceiling camera 18 captures an image of the imaging area 120 at an angle as shown in FIG. Is displayed. By using a conversion matrix that transforms this ellipse into a perfect circle, the tilt included in the planar image is corrected.

Similarly, when a perfect circle is drawn on a wall erected perpendicularly to the floor of the passenger compartment, when the side camera 19 captures an image of the imaging area 122 at an angle as shown in FIG. 13, the perfect circle of the wall is It is displayed as an ellipse in the imaging area 122 . By using a transformation matrix that transforms this ellipse into a perfect circle, the tilt included in the side view image is corrected.

The distortion/tilt correction unit 37A performs distortion correction and tilt correction on the plan view image captured by the ceiling center camera 18 and the side view image captured by the side camera 19 . The corrected plan view image and side view image are sent to the vehicle interior image recognition unit 36 (see FIG. 2), and image recognition of the person and luggage is executed based on the SSD algorithm described above.

The depth information acquisition unit 37B acquires depth information of each bounding box recognized by the vehicle interior image recognition unit 36 . For example, this depth information refers to position information along an axis orthogonal to the captured image plane. More specifically, for the bounding box (planar view section) extracted from the plan view image, the separation distance from the camera along the optical axis of the ceiling center camera 18 (first imaging device) is used as the depth information. Equivalent to. Similarly, for the bounding box (side view section) extracted from the side view image, the separation distance from the side camera 19 (second imaging device) along the optical axis corresponds to the depth information. .

Acquisition of such depth information uses, for example, a known stereo camera method. For example, when the central ceiling camera 18 and the side cameras 19 capture the same point in the vehicle interior 40, the distance from the central ceiling camera 18 and the side cameras 19 to the point is obtained using the principle of triangulation. can do

The depth information acquisition unit 37B acquires depth information of an arbitrary point within the bounding box using such a stereo camera method. For example, the depth information acquisition unit 37B acquires depth information at the center point C[Cx, Cy] of the bounding box, which is the plane coordinates of the camera coordinate system, and uses this as the depth information of the entire bounding box. The three-dimensional coordinates of each bounding box in the camera coordinate system can be obtained from this depth information and the position parameters of the bounding boxes described above.

The coordinate transformation unit 37C transforms the three-dimensional coordinates of each bounding box in the camera coordinate system into three-dimensional coordinates in the world coordinate system, which is the coordinate system of the vehicle interior space.

From the perspective of coordinate system transformation, the action of capturing an image of a real space with a camera can be considered as transforming (transferring) a three-dimensional space in the world coordinate system to a two-dimensional plane in the camera coordinate system. The depth information abstracted during this conversion is acquired by the depth information acquiring section 37B as described above, and as a result three-dimensional coordinates in the camera coordinate system are obtained.

The coordinate conversion unit 37C converts the three-dimensional coordinates of the bounding box in the camera coordinate system into the three-dimensional coordinates in the world coordinate system by performing conversion inverse to that performed when the camera captures the image. Here, the three-dimensional coordinates in the camera coordinate system refer to plane coordinates of the plan view section and the side view section plus depth information. Further, the transformation of this coordinate system can be executed using the existing coordinate transformation theory in the field of information processing engineering and the like.

For example, referring to the side view image of FIG. 9, in this image, a person 100A relatively close to the side camera 19 is larger than a person 100C relatively far from the camera. If a volumetric congestion rate, which will be described later, is obtained based on the size of an image based on such perspective, there is a risk that the congestion rate will be calculated that deviates from the accurate volume of each person. Therefore, by transforming the camera coordinate system into the world coordinate system in the coordinate transformation unit 37C, the size of the image caused by the perspective is eliminated.

In the world coordinate system, which is the spatial coordinate system in the vehicle interior 40, for example, the reference mark 48 (see FIG. 9) described above is set as the origin. Further, as illustrated in FIG. 14, in the bounding box 126 in the side view image, its center point Cs is indicated by coordinates [ _CFR , _CRW , _CUP ] in the world coordinate system. Furthermore, the width Ws is indicated as the length on the RW axis, and the height Hs is indicated as the length on the UP axis.

In the bounding box 124 in the planar image, the center point Cp is indicated by the coordinates [CFR, _CRW , _CUP ] in the world coordinate system, the width Wp is indicated as the length on the _FR axis, and the height Hp is shown as the length on the RW axis. The position parameters of the bounding box transformed into the world coordinate system are used for subsequent calculations.

The box combination extraction unit 37D extracts a pair of a bounding box in the plan view image and a bounding box in the side view image for obtaining the volume of the person and the package.

First, the box combination extraction unit 37D extracts only the bounding boxes of the class "person" among the plurality of bounding boxes. Furthermore, the box combination extraction unit 37D extracts a pair of bounding boxes using the position parameters of the bounding boxes extracted as classes.

For example, referring to FIG. 14, the box combination extraction unit 37D extracts C _FR −1/2W _P to C _FR +1/2W centered on the FR axis coordinate C _FR of the center point Cp of the bounding box 124 of the planar image. Set the range up to _P. Further, the box combination extractor 37D sets a range from C _RW -1/2H _P to C _RW +1/2H _P centered on the RW axis coordinate C _RW of the center point Cp. Next, the box combination extractor 37D extracts the bounding box 126 in the side view image that includes the center point Cs that satisfies these two range conditions, and extracts these bounding boxes 124 and 126 as a pair.

When the matching (pairing) of the bounding boxes of the class "person" is completed, the box combination extraction unit 37D extracts the bounding boxes of the class "luggage" and performs the same matching (pairing) as for the person described above. Run.

The volume calculation unit 37E obtains the volume of the person and the baggage from the bounding box (planar view section) of the plan view image and the bounding box (side view section) of the side view image paired by the box combination extraction unit 37D. . Note that the bounding box converted to the world coordinate system is used for the volume calculation.

Here, due to the accuracy of bounding box extraction by the SSD algorithm, the difference in magnification between the central ceiling camera 18 and the side cameras 19, etc., as illustrated in FIG. may have different dimensions from the bounding box 126 of .

Therefore, the volume calculation unit 37E may obtain the volume of the box by obtaining the volume by two different methods and then averaging them. For example, the volume calculator 37E obtains the first volume V1 by multiplying the width Wp and the height Hp of the bounding box 124 in plan view and the height Hs of the bounding box 126 in side view. Next, the volume calculator 37E obtains a second volume V2 by multiplying the width Ws and the height Hs of the bounding box 126 in side view by the width Wp of the bounding box 124 in plan view. Finally, the volume calculator 37E divides the sum of the first volume V1 and the second volume V2 by two. This gives the box volume Vb. Further, as position parameters of the box volume Vb, the center point Cv[ _{CV_FR} , _{CV_RW} , _{CV_UP} ], the length _LFR , the width _WRW , and the height _HUP are obtained.

The proximity baggage extraction unit 37F excludes baggage that is close to a person from objects for calculating the volumetric congestion rate. For example, the close parcel extraction unit 37F extracts parcels whose volume overlaps a person's volume by a predetermined ratio (for example, 30%) or more within the space of the world coordinate system. The extracted parcels (proximate parcels) are excluded from volumetric congestion rate calculation targets by the congestion rate calculator 37G.

For example, in image recognition using the SSD algorithm, position estimation is performed according to the bounding box, and the surrounding space is also captured in the bounding box in addition to the person and package, so such volume overlap can occur. If the volumetric congestion rate is calculated while allowing such volume overlap, the volumetric congestion rate may become excessively high. For this reason, luggage that is generally smaller in volume than a person is excluded from the calculation target of the volumetric congestion rate.

In addition, when prompting the movement of the short baggage between the legs of the standing passenger, which will be described later, the short baggage that has already been moved between the legs is sufficiently close to the person. Through the baggage extraction process, the baggage is excluded from the volume congestion rate calculation target. As will be described later, the short baggage to be moved between the legs of the standing customer is extracted from the calculation target of the volumetric congestion rate. In the process of extracting adjacent packages, they are excluded from extraction targets in advance.

The congestion rate calculation unit 37G obtains the total volume of boxes calculated by the volume calculation unit 37E and which are not excluded on the grounds that they are proximate packages. Furthermore, the congestion rate calculator 37G obtains the volumetric congestion rate A [%] based on the difference between the total box volume ΣVb and the volume Vc of the passenger compartment 40 . For example, the volumetric congestion rate A is obtained from (ΣVb/Vc)×100.

The short baggage search unit 37H searches for short baggage when the volumetric congestion rate A exceeds the upper limit threshold value B [%]. The upper threshold value B is set to a value of 80% or more and 100% or less, for example. The bounding box of the side-view image is used to search for short luggage. Specifically, reference is made to the height component Hs (see FIG. 14) of the bounding box (transformed to the world coordinate system) in the side-view image given the class "luggage". The short baggage search unit 37H extracts baggage (short baggage) whose height component Hs is equal to or less than a predetermined threshold height Hth. For example, the threshold height Hth is set to 50 cm.

When the volumetric congestion rate A exceeds the upper limit threshold value B [%] and short luggage is extracted, the announcement control unit 37I prompts the standing customer to move the short luggage between their legs. Output a message inside the vehicle. For example, a voice message is output from the in-vehicle speaker 45 (see FIG. 5), and the message text is displayed from the in-vehicle display 41B.

<Congestion rate estimation flow>
7 and 8 illustrate congestion rate estimation flows executed by the vehicle interior monitoring system according to the present embodiment. The distortion/inclination correction unit 37A (see FIG. 3) of the congestion estimation unit 37 acquires a side view image (see FIG. 9) of the vehicle interior 40 from the side camera 19 (S10). Next, the distortion/tilt correction unit 37A of the congestion estimation unit 37 performs the above-described distortion correction processing and tilt correction processing on the side view image (S11). The corrected side view image is transmitted to the vehicle interior image recognition section 36 .

The vehicle interior image recognition unit 36 recognizes the bounding box (side view section) surrounding the person and luggage in the corrected side view image using the SSD algorithm described above (S12). Furthermore, the depth information acquisition unit 37B acquires depth information of each bounding box using the stereo camera method described above (S13).

For example, as described above, the separation distance from the central ceiling camera 18 is obtained as depth information for the central point of the bounding box in the planar image. This separation is applied as depth information at all coordinate points of the bounding box.

Furthermore, the coordinate transformation unit 37C transforms the three-dimensional coordinates (image plane coordinates + depth information) of the center point and four vertices of the bounding box in the camera coordinate system into the world coordinates, which are the space coordinate system in the vehicle compartment, by the above-described transformation process. Convert to three-dimensional coordinates in the system (S14).

Next, the congestion estimation unit 37 acquires a planar view image (see FIG. 11) from the central ceiling camera 18 (S15). Further, the vehicle interior image recognition unit 36 and the congestion estimation unit 37 perform the same image processing as that for the side view image on the image. That is, distortion/angle correction (S16), image recognition of a bounding box (planar view section) surrounding a person and luggage (S17), depth information acquisition of the bounding box (S18), and conversion from the camera coordinate system to the world coordinate system. Various processes such as transformation (S19) are performed on the monoscopic image.

Furthermore, as described above, the box combination extraction unit 37D obtains a pair of a side view bounding box and a planar view bounding box based on the positional parameters of the bounding boxes after conversion into world coordinates (S20).

Next, the volume calculator 37E calculates the box volume as described above using the positional parameters of the paired side-view bounding box and planar-view bounding box on the world coordinate system (S21). The volume calculation unit 37E obtains the box volume Vb for all pairs of person and package extracted by the box combination extraction unit 37D.

Furthermore, as described above, the close baggage extracting unit 37F extracts baggage whose box volume Vb overlaps that of the person at a predetermined ratio, and excludes the baggage (close baggage) from the congestion rate calculation target (S22). Further, the congestion rate calculation unit 37G obtains the total sum ΣVb of the box volumes Vb that are objects of calculation (S23). Further, the congestion rate calculator 37G obtains a volumetric congestion rate A, which is the ratio of the total sum ΣVb of the box volume Vb to the passenger compartment volume Vc (S24).

The short baggage search unit 37H determines whether or not the volumetric congestion rate A obtained in step S24 exceeds the upper limit threshold value B (S25). If the volumetric congestion rate A is equal to or less than the upper limit threshold value B, the volumetric congestion rate A is transmitted to the operation schedule adjustment unit 61 of the center server 50 (see FIG. 2) (S29). Furthermore, the announcement control unit 37I causes the in-vehicle display 41B to display the volume congestion rate A (S30).

On the other hand, if the volumetric congestion rate A exceeds the upper limit threshold value B in step S25, the short baggage searching unit 37H refers to FIG. The height Hs is extracted from the image bounding boxes 106A-106C with reference to the position parameter (after transformation to the world coordinate system). Then, the short baggage search unit 37H determines whether or not there is a short baggage whose height Hs is equal to or less than a predetermined threshold height Hth (S26).

If all of the extracted heights Hs exceed the predetermined threshold height Hth, the short baggage search unit 37H sends the volumetric congestion rate A obtained in step S24 to the center server 50 (see FIG. 2). to the operation schedule adjusting unit 61 (S29).

On the other hand, in step S26, when the short baggage whose height Hs is equal to or less than the predetermined threshold height Hth is extracted, the announcement control unit 37I controls the height of the standing customer's feet of the extracted short baggage. A message urging the user to move between is output from the in-vehicle indicator 41B (see FIG. 5) and the in-vehicle speaker 45 (S27). Further, after waiting for a predetermined time (S28), the flow returns to step S10, and the volume congestion rate is calculated based on the side view image and the plan view image after the announcement.

As described above, in the vehicle interior monitoring system according to the present embodiment, when the volumetric congestion rate A exceeds the upper limit threshold value B, short luggage is searched for and the luggage is placed between the legs of the standing passenger. encouraged to move. As a result, the area occupied by the short baggage before the movement becomes an empty space, and congestion can be alleviated.

<Operation adjustment flow>
FIG. 15 illustrates an operation adjustment flow by the operation schedule adjustment unit 61 of the center server 50 when the volumetric congestion rate A is received from the congestion estimation unit 37 (see FIG. 2). Each step is given a symbol (S) indicating the center server 50 as its executing entity.

The operation schedule adjustment unit 61 acquires the volume congestion rate A from the congestion rate calculation unit 37G of the congestion estimation unit 37 (S30). Next, the operation schedule adjustment unit 61 determines whether or not the acquired volumetric congestion rate A exceeds a predetermined upper limit threshold value B (S32). This upper limit threshold B may be the same value as the upper limit threshold B used in step S25 of FIG.

When the volumetric congestion rate A≦the upper limit threshold B, the operation schedule adjusting unit 61 maintains the already set operation schedule as it is. Further, the operation schedule adjusting unit 61 transmits the volume congestion rate A to the bus application 81 of the mobile terminal 70 (S34).

On the other hand, in step S32, if the volumetric congestion rate A>the upper limit threshold B, the operation schedule adjusting unit 61 refers to the operation schedule of the passenger vehicle 10 that has transmitted the volumetric congestion rate A. Furthermore, the operation schedule adjustment unit 61 changes the operation schedule so that the passenger vehicle 10 passes through the next stop scheduled to stop (S36). The changed operation schedule is transmitted from the operation schedule providing unit 63 to the target passenger vehicle 10 .

The operation schedule adjusting unit 61 also transmits the volumetric congestion rate A and the changed operation schedule to the bus application 81 of the mobile terminal 70 (S38). Thereby, the user of the portable terminal 70 can know in advance that the shared vehicle 10 will pass through the stop without stopping.

<Another example of depth information acquisition method>
In the above-described embodiment, the depth information acquisition unit 37B (see FIG. 3) acquires the depth information of the bounding box by the stereo camera method using the central ceiling camera 18 and the side cameras 19. is not limited to this form.

For example, as illustrated in FIGS. 16 and 17, the floor 43 of the passenger compartment 40 is virtually divided into a plurality of regions of interest. In the illustration of FIGS. 16 and 17, the floor 43 is divided into 10 areas.

Then, the depth information acquisition unit 37B determines in which area of the floor 43 the person and the package are located, for example, in the side view image and the plan view image. The depth information assigned to each area is then assigned to the person and package in each area.

Alternatively, instead of the stereo camera method, the ceiling central camera 18 and the side cameras 19 may be used with 3D cameras such as RGB-D cameras. For example, in a TOF (Time of Flight) 3D camera, a short-pulse laser is irradiated, and based on the time it takes for the object to reflect and return to the light-receiving element, the distance between the camera and the object (i.e., depth information) is obtained. Desired.

<Another example of associating a plan view image and a side view image>
In the above-described embodiment, the box combination extraction unit 37D (see FIG. 3) uses the positional parameters of the bounding boxes as shown in FIG. ring). However, the vehicle interior monitoring system according to this embodiment is not limited to this form.

For example, tracking using the so-called optical flow method may be used. For example, when a person (passenger) gets into the passenger vehicle 10 from the doors 41, 41, the vehicle interior image recognition unit 36 executes image recognition of the person and luggage. A unique ID is assigned to each recognized person and package. Furthermore, the box combination extraction unit 37D traces the movement (trajectory) of the feature points of the person and package to which the ID is assigned. As a result, the trajectory of the same person and luggage can be traced on the two-dimensional image and the side view image. As a result, the box combination extraction unit 37D can execute association (pairing) of each person and package using the ID as a key.

<Another example of a shared vehicle utilization system>
FIG. 18 shows another example of the shared vehicle utilization system illustrated in FIG. In FIG. 18 , the operation schedule adjustment unit 61 provided in the center server 50 in FIG. 2 is provided in the passenger vehicle 10 . Other configurations are the same as those in FIG.

FIG. 19 illustrates an operation adjustment flow by the operation schedule adjustment unit 61 provided in the passenger vehicle 10. As shown in FIG. Each step of this flow is the same as each step in FIG. 15 as processing content. Specifically, steps S30, S32, S34, S36 and S38 in FIG. 15 correspond to steps S130, S132, S134, S136 and S138 in FIG. However, the execution subject of each step is replaced from the code (S) indicating the center server 50 to the code (B) indicating the passenger vehicle 10 . Further, in steps S134 and S138, the transmission destination of the congestion rate and the changed operation schedule is changed from the bus application 81 of the mobile terminal 70 to the center server 50. FIG.

10 passenger vehicle, 18 ceiling center camera, 19 side camera, 20 control unit, 36 vehicle interior image recognition unit, 37 congestion estimation unit, 37A tilt correction unit, 37B depth information acquisition unit, 37C coordinate conversion unit, 37D box combination extraction Section 37E Volume Calculation Section 37F Proximity Baggage Extraction Section 37G Congestion Rate Calculation Section 37H Short Baggage Search Section 37I Announcement Control Section 40 Vehicle Interior 41A Outside Display 41B Inside Display 45 Inside Speaker 48 reference mark, 50 center server, 61 operation schedule adjustment unit, 62 operation schedule creation unit, 63 operation schedule provision unit, 64 operation route creation unit, 65 operation schedule storage unit, 66 dynamic map storage unit, 70 mobile terminal, 81 bus application , 82 operation schedule display section, 83 congestion rate display section, 104A to 104C, 106A to 106C, 108A to 108C, 110A to 110C bounding box.

Claims (7)

a first imaging device that captures a planar image of the interior of the vehicle from the center of the ceiling of the vehicle;
a second imaging device that captures a side-view image of the inside of the vehicle interior from the peripheral edge of the ceiling or the side surface of the vehicle interior;
an image recognition unit that recognizes a person and luggage included in the planar view image and the side view image;
a congestion estimation unit that obtains a congestion rate of the vehicle compartment based on the recognized person and baggage;
with
The image recognition unit obtains a plan view section of the recognized person and package in the plan view image, and obtains a side view section of the recognized person and package in the side view image,
The congestion estimation unit,
a volume calculation unit that obtains the volume of the person and the package based on the plan view section and the side view section of the person and the package;
a congestion rate calculation unit that calculates the congestion rate based on the difference between the total volume of people and luggage and the volume of the vehicle compartment;
comprising a
In-vehicle monitoring system. The vehicle interior monitoring system according to claim 1,
The congestion estimation unit includes a depth information acquisition unit and a coordinate conversion unit,
The depth information acquisition unit acquires depth information along the optical axis of the first imaging device of the planar view section of the person and the package, and acquires depth information of the side view section of the person and the package from the second imaging device. acquire depth information along the optical axis of
The coordinate transformation unit converts three-dimensional coordinates on a camera coordinate system based on plane coordinates and depth information of the planar view section and the side view section into three-dimensional coordinates on a world coordinate system, which is the coordinate system of the vehicle interior space. Converted,
The volume calculation unit obtains the volume of the person and the baggage based on the plan view section and the side view section converted into the world coordinate system;
In-vehicle monitoring system. The vehicle interior monitoring system according to claim 2,
The image recognition unit sets a bounding box, which is a rectangular area surrounding a person and a package, obtained based on an SSD (Single Shot Multibox Detector) algorithm in the plan view section and the side view section.
In-vehicle monitoring system. The vehicle interior monitoring system according to claim 3,
The congestion estimating unit includes an adjacent luggage extracting unit,
The proximate luggage extracting unit extracts proximate luggage, which is luggage whose volume overlaps the volume of the person by a predetermined ratio or more in the space of the world coordinate system,
The congestion rate calculation unit excludes the extracted adjacent parcels from a target for calculating the congestion rate.
In-vehicle monitoring system. The vehicle interior monitoring system according to claim 4,
The congestion estimation unit includes a short luggage search unit,
When the congestion rate obtained by the congestion rate calculation section exceeds a predetermined upper limit threshold, the short baggage search section converts the baggage for which the congestion rate is calculated into the world coordinate system. Searching for packages having a height equal to or less than a predetermined threshold height based on the height component in the side view section;
In-vehicle monitoring system. The vehicle interior monitoring system according to claim 5,
The congestion estimating unit includes an announcement control unit that outputs an announcement urging the movement of luggage to the vehicle interior when a luggage having a height equal to or lower than the predetermined threshold height is searched,
In-vehicle monitoring system. A vehicle operation management system comprising: the vehicle interior monitoring system according to any one of claims 1 to 6; and an operation schedule adjusting unit capable of communicating with the vehicle interior monitoring system,
The operation schedule adjusting unit can communicate with the congestion estimating unit of the vehicle interior monitoring system installed in each of a plurality of vehicles, and based on the congestion rate received from each of the congestion estimating units, Determining whether to change the operation schedule so that the train will pass through the next scheduled stop,
Vehicle operation management system.

Images