JP6892258B2 - Operating state control device and operating state control method - Google Patents

Description

Embodiments of the present invention relate to an operating state control device and an operating state control method.

For example, as disclosed in Non-Patent Documents 1 and 2, a technique for detecting a person in an image taken by a camera is known. Patent Document 1 also discloses a technique for detecting a person in a wheelchair. In recent years, the development of image processing technology has been remarkable in this way, and it has become possible to acquire information such as the state, attributes, and possessions of a person as well as a person from an image.

It is expected that this kind of technology is applied to safe driving of a vehicle, and for example, the technology shown in Patent Documents 2 to 4 and the standard such as Non-Patent Document 3 are disclosed. Non-Patent Document 4 also introduces a technique for supporting driving at night by using an in-vehicle camera.

U.S. Pat. No. 7043084 International Publication No. 2016/121174 Pamphlet JP-A-2015-133050 International Publication No. 2014/054153 Pamphlet

Tomoki Watanabe, "High-precision person detection technology using new features", Toshiba Review Vol. 65 No. 4 (2010), p72-73 "Person tracking technology that can track people even if they are temporarily hidden from the camera's field of view" by Van Quocveto, Toshiba Review Vol. 70 No. 6 (2015), p56-57 Ministry of Land, Infrastructure, Transport and Tourism, Automobile Bureau press release material "We will establish standards for a" camera monitoring system "instead of rear-view mirrors, etc." ”, [Online], [Searched on November 30, 2016], Internet, <URL: http://www.mlit.go.jp/report/press/jidosha07_hh_000211.html> "Toyota | Safety Technology | Night View", [online], [Search on November 30, 2016], Internet, <URL: http://www.toyota.co.jp/jpn/tech/safety/technology/ technology_file / active / night_view.html> "Evaluation of communication policy after communication liberalization and research on future image of ICT society", [online], [Search on November 30, 2016], Internet, <URL: http: //www.soumu. go.jp/johotsusintokei/linkdata/h27_01_houkoku.pdf>

As mentioned above, in recent years, the technology to support the driving of the vehicle by the image recognition technology has been advanced, but in each case, the support of safe driving is the first priority, and the comfort and hospitality of the passengers are improved. There are few technologies that pay particular attention to it. For example, a technology that makes it possible to apply the knowledge obtained by image recognition to the field of so-called "autonomous driving" is in the process of development.

There are plans to provide vehicles that apply autonomous driving technology or autonomous driving support technology to the market in the near future. Along with that, services such as car sharing and rental cars equipped with an automatic driving system will be started. However, in providing this kind of service, there is no known technology that can reflect the attributes of passengers (which may include the driver) in driving.

For example, in car sharing / rental car, it is not possible to continuously use the vehicle desired by the driver. At best, you can specify the car model, which is not certain. In such an environment where a dedicated vehicle cannot be used, in order to use automatic driving functions such as route selection, alerting regarding acceleration / deceleration, or changing lanes, it is more important to consider not only safety but also passengers. Bring Furthermore, it will be possible to handle cases where passengers are using actuator technology (powered suits, etc.) that can assist athletic ability, which will appear as a walking aid in the near future, as described in Non-Patent Document 5. There is a need to.

The present invention has been made in view of the above circumstances, and an object of the present invention is to provide an operating state control device and an operating state control method with enhanced serviceability to passengers.

According to the embodiment, the operation state control device includes an image processing unit, a correlation processing unit, and a mode processing unit. The image processing unit processes the image data acquired by the camera including the outside of the vehicle in the field of view, and acquires the image information in the field of view of the passenger of the vehicle. The correlation processing unit generates the attribute information of the passenger by the correlation processing using the image information. The mode processing unit identifies the driving mode for the passenger based on the attribute information.

FIG. 1 is a functional block diagram showing an example of an operation state control device according to an embodiment. FIG. 2 is a diagram showing an installation example of the outside camera 31 and the inside camera 32. FIG. 3 is a diagram showing an installation example of the external camera 31. FIG. 4 is a flowchart showing an example of the processing procedure in the ECU 20. FIG. 5 is a flowchart showing an example of a processing procedure in the ECU 20. FIG. 6 is a diagram showing the correlation between the detection result of each sensor of FIG. 1 and the estimation result of the occupant's attribute in chronological order. FIG. 7 is a diagram showing an example of a prompt message displayed on the display 40. FIG. 8 is a diagram showing another example of the prompt message displayed on the display 40.

FIG. 1 is a functional block diagram showing an example of an operation state control device according to an embodiment. Hereinafter, the ECU (electronic control unit) designated by reference numeral 20 in FIG. 1 will be described as an example of the operating state control device.

The ECU 20 is connected to the automatic driving system 10 of the vehicle and the sensor system 30 provided in the vehicle. In addition, the vehicle includes a display 40, a speaker 50, and a light 70. Further, the vehicle may be provided with a rear door lock control unit 60 that controls locking / unlocking of the rear door.

The display 40 is installed between the driver's seat and the passenger seat, for example, and is a user interface such as a touch panel type LCD (Liquid Crystal Display). Various messages are given to the user (passenger) by a GUI (Graphical User Interface) displayed on the display 40 or a voice message from the speaker 50. Further, the user's instruction given from the GUI of the display 40 is passed to the automatic operation system 10 via the ECU 20.

The automatic driving system 10 is provided as an oncoming device of the ECU 20, and is responsible for controlling the automatic driving of the vehicle. For example, the automatic driving system 10 controls the driving of the vehicle according to the driving mode designated by the ECU 20.

The sensor system 30 includes an outside camera 31, an inside camera 32, a door unlock sensor 33, a door open / close sensor 34, a beacon receiver 35, a microphone 36, and a seat pressure sensitive sensor 37.

The vehicle exterior camera 31 is a camera that includes the outside of the vehicle in the field of view, and captures the inside of the field of view to acquire image data. That is, the outside camera 31 is a camera that photographs the periphery of the vehicle, and may at least capture a passenger approaching the door of the vehicle. It is also possible to share the camera monitoring system shown in Non-Patent Document 3 with the external camera 31. In particular, if it is a type that works with the illumination 70, a good image can be acquired even in a dark place such as at night, and a camera having a night-vision function may be used.

The in-vehicle camera 32 is an image sensor that includes the inside of the vehicle (inside the vehicle) in the field of view, and captures the inside of the field of view to acquire image data. That is, the in-vehicle camera 32 may be a camera that photographs the inside of the vehicle, and may at least capture the seating status of each seat in the vehicle. The number of in-vehicle cameras 32 is not limited. For example, if a camera installed at the front of a small vehicle can capture the state of the rear seats, it is not necessary to install a separate camera at the rear. The same applies to the external camera 31.

2 and 3 are views showing an installation example of the outside camera 31 and the inside camera 32. The external camera 31 can be attached to, for example, the position of the rearview mirror and used in combination with the camera monitoring system. The in-vehicle camera 32 is attached, for example, at a position overlooking the inside of the vehicle from the ceiling of the vehicle. A camera with a wide viewing angle such as a fisheye lens may be used.

The explanation will be continued by returning to FIG. The door lock release sensor 33 detects the state of the vehicle door lock. The door open / close sensor 34 is installed on each door of the vehicle and detects the open / closed state of each door. If the open / closed state of each door can be determined from the image data taken by the external camera 31 and the in-vehicle camera 32 and the information can be passed to the correlation processing unit 26, the door open / close sensor 34 may be omitted.

The beacon receiver 35 is loaded on the vehicle and receives the beacon signal from the beacon transmitter 80. The beacon transmitter 80 can be attached to a walking aid such as a wheelchair or crutches, and wirelessly transmits a beacon signal including information reflecting the attributes of the user. In the embodiment, the beacon transmitter 80 is carried by the passenger himself / herself, and transmits different signals according to the attributes of the passenger. As the beacon transmitter 80, a short-range wireless transmitter such as BLE (Bluetooth (registered trademark) Low Energy) can be used.

The presence of the walking aid can be directly confirmed based on the received beacon signal. Alternatively, the presence or absence of the walking aid may be determined by associating the beacon signal with the person detected by the camera outside the vehicle 31.

The microphone 36 is loaded on the vehicle, captures voices and sounds inside and outside the vehicle, and outputs a voice signal. The seat pressure sensor 37 is a pressure sensor provided for each seat of the vehicle, and detects the presence or absence of a seat-shaped passenger based on the detected pressure value. Information regarding the presence or absence of passengers on the seat is passed to the correlation processing unit 26.

Next, the ECU 20 has an image processing unit 21, a door unlock detection processing unit 22, a door open / close detection processing unit 23, a beacon reception processing unit 24, a voice processing unit 25, a correlation processing unit 26, and a user as processing functions according to the embodiment. It includes an interface unit 27, an operation mode selection processing unit 28, and a luggage carry-in support processing unit 29.

The image processing unit 21 processes the image data acquired by the vehicle exterior camera 31 to acquire image information mainly outside the vehicle of the passengers of the vehicle. Further, the image processing unit 21 processes the image data acquired by the in-vehicle camera 32 to acquire the image information in the vehicle. In particular, the image processing unit 21 determines the presence or absence of a child seat based on the image information in the vehicle, and passes the result of the determination to the correlation processing unit 26. It is technically possible to determine the presence or absence of an object such as a child seat by utilizing an existing image processing library or the like.

The door lock release detection processing unit 22 processes the detection information from the door lock release sensor 33. The door open / close detection processing unit 23 processes the detection information from the door open / close sensor 34.

The beacon reception processing unit 24 processes the beacon signal received by the beacon receiver 35 to acquire the beacon information. This beacon information is passed to the correlation processing unit 26. The voice processing unit 25 processes the voice signal output from the microphone 36 to acquire voice information.

The correlation processing unit 26 generates attribute information of the occupants of the vehicle by correlation processing using at least the image information obtained by imaging the outside of the vehicle output from the image processing unit 21. Correlation processing can be easily understood by those skilled in the art by applying a technique such as sensor fusion, which processes a plurality of sensing data in an integrated manner.

The passenger attribute information is, for example, information indicating age, gender, presence / absence of walking aids (wheelchair, crutches, etc.), and whether or not a pet is carried. In addition, a wide variety of attribute information can be considered according to the scene.

In particular, the correlation processing unit 26 generates the attribute information of the passenger by the correlation processing between the image information of the passenger and the image information in the vehicle. Further, the correlation processing unit 26 generates the attribute information of the passenger by the correlation processing using the beacon information.

Further, the correlation processing unit 26 can also generate the attribute information of the passenger by the correlation processing using the voice information. In particular, the correlation processing unit 26 may start the correlation processing when the door lock release is detected.

Further, the correlation processing unit 26 generates attribute information for each passenger who has boarded the vehicle from each door based on the time-series change of the open state and the closed state of the door of the vehicle. In addition, the correlation processing unit 26 generates attribute information of the passenger of the seat that is sensitive to the seat pressure sensor 37.

In particular, the correlation processing unit 26 generates attribute information regarding whether or not the occupant's walking aid is used. Further, the correlation processing unit 26 reflects the information regarding the presence / absence of the child seat in the correlation processing to generate the attribute information of the passenger.

Next, the user interface unit 27 is connected to the display 40 and the speaker 50 in the vehicle, and acts as an intermediary between the ECU 20 and the user to exchange various information. For example, the user interface unit 27 presents a plurality of operation modes extracted by the operation mode selection processing unit 28 to the user via the display 40 and the speaker 50. Further, the operation mode specified by using the touch panel of the display 40 is input to the ECU 20 via the user interface unit 27.

The operation mode selection processing unit 28 identifies the operation mode for the passenger based on the attribute information of the passenger generated by the correlation processing unit 26. In particular, the operation mode selection processing unit 28 extracts a plurality of operation modes based on the attribute information of the passenger, and notifies the user of the plurality of operation modes via the display 40 or the speaker 50 (user interface). When any of the plurality of operation modes displayed on the display 40 is specified by the user, the operation mode selection processing unit 28 specifies the designated operation mode as an operation mode for the passenger.

Further, the operation mode selection processing unit 28 notifies the automatic operation system 10 of the specified operation mode. More specifically, the driving mode selection processing unit 28 notifies the automatic driving system 10 of the combination of vehicle control parameters reflecting the specified driving mode. The vehicle control parameter may be, for example, at least one of an acceleration / deceleration range, a speed range, and a movement path, and various other parameters can be considered.

The luggage carry-in support processing unit 29 controls the rear door lock control unit 60 to open and close the door of the luggage compartment of the vehicle. For example, when it is determined from the image of the outside camera 31 that the passenger is accompanied by a large baggage, a wheelchair, or the like, the baggage carry-in support processing unit 29 opens the trunk or the rear hatch to support the storage of the baggage.

The ECU 20 estimates the attributes of the occupant based on the above functions and notifies the automatic driving system 10 of the result to indicate an appropriate driving mode. For example, if it is determined from the attribute information that an injured person is on board , a message suggesting a driving mode such as "gentle" driving with suppressed acceleration / deceleration is sent to the vehicle driver and passengers. It is displayed on the display 40.

Image processing unit 21, door unlock detection processing unit 22, door open / close detection processing unit 23, beacon reception processing unit 24, voice processing unit 25, correlation processing unit 26, user interface unit 27, operation mode selection processing shown in FIG. Each function of the unit 28 and the luggage carry-in support processing unit 29 can be stored in a memory (not shown) or the like in the form of a program that can be executed by a processor, for example. That is, the ECU 20 is a computer that realizes a function corresponding to each program by reading a program from a memory and executing the program.

The processor is a CPU (central processing unit), a GPU (Graphics Processing Unit), an integrated circuit for a specific application (Application Specific Integrated Circuit (ASIC)), or a programmable logic device (for example, a Simple Programmable Logic Device: It is a hardware resource such as SPLD), Complex Programmable Logic Device (CPLD), and Field Programmable Gate Array (FPGA). Next, the operation in the above configuration will be described.

4 and 5 are flowcharts showing an example of the processing procedure in the ECU 20. When the door lock of the vehicle is released in FIG. 4, the door lock release sensor 33 detects this and starts supplying power to the ECU 20 (step S1). Immediately after that, the ECU 20 is activated, and at the same time, the external camera 31, the in-vehicle camera 32, the door open / close sensor 34, the seat pressure sensor 37, and the like are also activated (step S2).

Subsequently, the external camera 31 captures a moving image of a person approaching the vehicle and outputs the image data to the image processing unit 21 of the ECU 20 (step S3). The image processing unit 21 processes the image data to generate image information, and estimates from the image information the presence / absence of a walking aid such as a wheelchair of an approaching person and attribute information such as age. The result is output to the correlation processing unit 26 (step SS5).

Here, when the presence of the luggage is estimated from the image of the external camera 31, the correlation processing unit 26 gives an instruction to the luggage carry-in support processing unit 29 to open the trunk and the rear hatch to support the luggage storage.

At the same time, the in-vehicle camera 32 also continuously captures the in-vehicle image after the start-up (step S4). The image data taken by the in-vehicle camera 32 is processed by the image processing unit 21 to generate image information. Based on the image information, the image processing unit 21 determines the presence or absence of a child seat or the like as a state before the passenger is seated, and outputs the result (step S6). The presence or absence of the child seat is notified to the correlation processing unit 26 (step S11).

Next, when the door of the vehicle is opened (step S7), the door open / close sensor 34 reacts (step S8), and the door open / close detection processing unit 23 outputs the open state of the door to the correlation processing unit 26 (step S9).

Upon receiving the notification of the open state of the door from the door open / close detection processing unit 23, the correlation processing unit 26 confirms the presence / absence of a passenger and the age estimation result from the image processing unit 21. The camera outside the vehicle 31 and the camera inside the vehicle 32 may be linked, and a moving image of a person on board may be tracked and photographed by the camera outside the vehicle 31 and the camera inside the vehicle 32. Then, the correlation processing unit 26 determines the presence or absence of a occupant for the seat in the vicinity of the opened door, and estimates the age of the occupant (step S10).

Subsequently, when the door is closed, the door open / close sensor 34 reacts, and the door open / close detection processing unit 23 detects the closed state of the door (step S12) and notifies the correlation processing unit 26 of that (step S21). ..

The correlation processing unit 26 has elapsed a certain period of time from the notification of the closed state of the door from the door open / close detection processing unit 23, or when the in-vehicle camera 32 detects that the passenger is seated in the driver's seat, or the outside camera. When other passenger candidates are no longer observed in 31, the presence or absence of passengers in each seat and the age estimation are determined (step S22). The confirmed passenger attributes are notified to the operation mode selection processing unit 28.

Then, the driving mode selection processing unit 28 selects the driving mode of the vehicle based on the attributes of the passenger, notifies the user of the contents via the display 40 and the speaker 50, and prompts the selection of the passenger (step S23). ). When the operation mode is specified by the passenger (step S24), the operation mode selection processing unit 28 notifies the automatic operation system 10 of the selection result (step S25).

FIG. 6 is a diagram showing the correlation between the detection result of each sensor of FIG. 1 and the estimation result of the occupant's attribute in chronological order. FIG. 6 shows processing at the left rear door of the vehicle. That is, the attributes of the passenger can be estimated for each door.

As shown in FIG. 6A, it is assumed that there are five judgment items, for example, door unlock, outside camera (left), door open (left rear), in-vehicle camera (left rear view), and all doors closed. (# 0 to # 4). When the door unlock is first detected (▽), the outside camera and the inside camera are activated, and a passenger is detected near the left rear door (○). Then, as shown in FIG. 6B, the presence / absence of the walking aid is determined from the image data of the camera outside the vehicle, and the presence / absence of the child seat is determined from the image data of the camera inside the vehicle.

When the time elapses in FIG. 6 (a) and the rear left door opens (▽), the presence or absence of a passenger and the age of the passenger are estimated in FIG. 6 (b). Further, when all the doors are closed after a lapse of time in FIG. 6A (all doors are closed (▽)), the presence / absence and age of a passenger and the presence / absence of a child seat are determined for the seat on the left rear side.

The procedure for a person to board a vehicle can be disassembled as follows, for example, (1) to (5).
(1) Approach the vehicle and move in front of the door. (2) Open the vehicle door. (3) Board the vehicle and take a seat. (4) Close the vehicle door. (5) Start driving the vehicle.

At this time, when the passenger is using a walking aid such as a wheelchair or crutches, a characteristic image and movement pattern are observed by the camera outside the vehicle 31. In addition, the act of removing the walking aid after opening the door will be observed.

On the other hand, when there is a child passenger, it is possible to judge by height from the captured image when the vehicle approaches and in front of the door. Further, when a seating aid such as a baby seat or a child seat is confirmed in the image taken by the in-vehicle camera 32, it can be determined that the infant is included in the passenger. Furthermore, the same judgment makes it possible to judge whether a pet is on board.

The attribute information determined for each door includes, for example, the items shown in the following table.

When the door open / close detection processing unit 23 notifies that all the doors have been closed, the correlation processing unit 26 generates the attribute information shown in Table 1 for each door. For example, from the detection result of the door lock release sensor 33, a small item such as the presence or absence of a passenger can be estimated. If there is a passenger, it is defined as 1, and if there is no passenger, it is defined as 0. From the image data of the camera outside the vehicle, the age and the presence or absence of walking aids can be estimated. The age and the presence or absence of a child seat can be estimated from the image data of the in-vehicle camera. Age can also be estimated by analyzing the sound from the microphone.

The attribute determination value Vm can be calculated using the numerical value (0/1) shown in the remarks column of Table 1.

For example, consider that an adult who does not have a walking aid such as a cane approaches a vehicle. In this case, the image processing unit 21 determines from the height of the approaching person that the approaching person is an adult, based on the image information from the external camera 31. As a result, 0 is generated as the value of the age estimation, and 0 is generated as no walking aid. It is determined from the image information from the in-vehicle camera 32 that the person is an adult, and 0 is generated as the age estimation value. If there is no child seat on the sheet, the correlation processing unit 26 generates 0 as a value.

At the same time, based on the voice data input from the microphone 36, the voice processing unit 25 discriminates each speaker and estimates the age. Here, if the passenger is determined to be an adult, 0 is generated as the value of the attribute information.

Further, when the door is closed, the presence or absence of a occupant from the door is determined, and a determination value of 1 is generated assuming that there is a occupant. As a result of the above process, items 1 = 1 and items 2 to 6 = 0 in Table 1 are obtained, and 0 is obtained as the attribute determination value Vm.

In another case, a passenger using crutches is approaching. In this case, crutches are detected from the image data from the external camera 31, and a determination value 1 is generated as item 3 in Table 1. For each item other than item 3, the same determination value as when an adult approaches is generated, and the attribute determination value Vm calculated in this case is 1.

Furthermore, as another case, consider the case where there is a child passenger. In this case, in item 2, item 4, and item 6 of Table 1, 1 is generated, and 3 is generated as the attribute determination value Vm.

The attribute determination value Vm is notified from the correlation processing unit 26 to the operation mode selection processing unit 28. When the attribute determination value Vm is not 0 (Vm> 0), the operation mode selection processing unit 28 determines that the use of the "gentle driving" mode for children, injured persons, and the like is recommended. Then, the operation mode selection processing unit 28 displays a message as shown in FIG. 7 or 8, for example, on the display 40.

FIG. 7 shows an example of a message notifying the passenger that driving with less burden is possible, and prompts the passenger to select “yes” or “no” from this screen. The same content may be notified to the passenger through the speaker 50 by voice. Also in FIG. 8, either (gentle operation) or (normal operation) can be selected.

When the passenger selects "Yes" from the screen of FIG. 7, the correlation processing unit 26 instructs the automatic driving system 10 to control the vehicle and select a route that reduces the burden on the passenger. That is, the automatic driving system 10 is notified of the optimum combination candidate of the vehicle control parameter according to the (gentle driving mode).
Next, some examples according to this embodiment will be described.

[First Example]
An example assuming rental car / car sharing using a vehicle with an automatic driving function or an automatic driving support function will be described.

When the passenger releases the door lock, the system of the present invention starts operation, and the ECU 20 starts collecting the attributes of the passenger. If it is determined that the passengers do not include injured persons, the vehicle will not make any special presentation, and unless the passengers operate / instruct, the vehicle will be controlled in the normal operation mode.

If it is estimated that there are passengers using walking aids such as crutches, wheelchairs, and canes, automatic driving such as lane change control with a smaller acceleration / deceleration rate than usual or with lateral acceleration suppressed. , Or provide driving assistance. In addition, when selecting a route to a destination, instead of prioritizing time, for example, a route with less stops due to traffic lights, a route with less traffic, or a route with less turns is selected and presented.

In addition, if it is estimated that there are passengers using walking aids, at the time of boarding, the passengers will be asked to confirm the unlock response by voice, etc. to store the walking aids, and the trunk or rear door will be locked. You may cancel it.

If it is estimated that there are children on board, time-priority route selection will be performed. Then, by providing the same driving support as above, we will present automatic driving or driving support that takes into consideration that children can spend their time safely in the car while shortening the riding time so as not to be foolish.

As a result, the driver can easily realize safe and comfortable driving for the passenger in an unfamiliar vehicle by being presented with the optimum driving state by the passenger and without performing complicated operations. .. Depending on the condition of the passenger, such as an injured person, it may take longer than usual, and it is the comfort and safety of the passenger to reduce the rate of acceleration / deceleration and minimize lateral shaking. It is important to secure the surface.

Systems installed in vehicles in recent years have become more sophisticated and complex. In particular, in an unfamiliar vehicle that is not used daily, it is difficult for the passenger to operate the vehicle mounting system. As a countermeasure, the screen display or voice output is used on the vehicle side to present the passengers with options for the functions provided in the vehicle, prompting the passengers to make a selection, or asking / wishing whether or not the passengers are injured. A method of confirming the contents to be performed and controlling the vehicle settings accordingly is conceivable.

However, presenting options / answering various questions tends to lead to time-consuming results from boarding to departure. Furthermore, it causes stress on the passengers due to incorrect answers to questions.

On the other hand, in the embodiment, the outside camera 31 is used to confirm the passenger outside the vehicle, that is, the state before boarding, and the attributes of the passenger are confirmed by a sensor system such as the in-vehicle camera 32 and the microphone 36. I made it. Then, a recommended driving mode is selected based on the result, and the optimum combination candidate of the vehicle control parameter is extracted and presented to the passenger. Then, the result of the selection based on the intention of the passenger is reflected in the control of the vehicle. This makes it possible to realize a vehicle system that provides comfort to all passengers.

In addition, even in a temporary usage environment such as car sharing / rental car, by acquiring the attributes of the passenger and proposing the most suitable one from the menu that reflects the attributes, it is optimal with a minimum of questions and answers. You will be able to determine the operation mode. Therefore, it provides a means for a system that can be easily utilized even by a non-specialist in a more sophisticated and complicated vehicle control system that differs depending on the vehicle type.

[Second Example]
For example, the above embodiment can be applied to an automatically driven taxi. While the self-driving taxi is in operation, the ECU 20 provided in the taxi may be kept in operation at all times.
The out-of-vehicle camera 31 discovers a person presumed to be a passenger and collects its attributes. When the passenger approaches and appears in the vehicle, the presence or absence of an injured passenger is determined from the collected attributes, and the same message as in the first embodiment is presented to the passenger. ..

By doing so, it is possible to eliminate the need to confirm the question and answer to the passengers in the self-driving taxi. In addition, it is possible to present the optimum driving state to the passenger, and it is possible to realize a quick start from the boarding and a safe and comfortable driving for the passenger.

If the technology according to the embodiment is not applied, a detailed question and answer session with the passenger is required due to the nature of the system in order to carry out the driving according to the passenger's condition in the self-driving taxi. In addition, it is necessary to control the driving mode by the judgment function of the vehicle or the judgment of a remote operator. This mechanism will be used after the passenger has boarded the vehicle, and it will take time to start.

On the other hand, according to the embodiment, the passenger only has to select the driving mode he / she desires from the presented message, so that he / she can enjoy a quick start and comfortable driving.

[Third Example]
An embodiment applicable to either rental car / car sharing using a vehicle with an automatic driving function or an automatic driving support function, or a taxi will be described.

For passengers who use actuator technology that assists their athletic ability as walking aids, or for passengers who normally use walking aids but do not wear (use) walking aids. In some cases, it may be difficult to determine the attribute only from the image data taken by the camera.

In this case, the beacon transmitter 80 is attached to the walking aid used by the passenger. Alternatively, the passenger himself carries a beacon transmitter 80 that emits a different signal according to the attributes such as the age of the passenger and the presence or absence of injury in advance. Then, by receiving the signal on the vehicle side and associating it with the image from the camera, the attributes of the passenger can be estimated. As a result, as in the first embodiment and the second embodiment, it is possible to present the optimum driving mode to the passenger and realize a quick start and safe and comfortable driving.

As described above, according to the embodiment, the external camera 31 is actively used, and the attribute estimation is started before the person is boarded. The images outside the vehicle are also used to determine whether or not wheelchairs and crutches are used. Therefore, it is possible to acquire detailed attributes that are difficult to detect only with the in-vehicle camera 32. Furthermore, the attributes of the passenger, which was difficult only with the in-vehicle camera 32, can be accurately acquired, and a more appropriate driving mode can be proposed.

Further, according to the embodiment, the passenger can determine the attributes of the occupant from the vehicle side and propose the functions provided in the vehicle without knowing the functions provided in the vehicle. Therefore, it is possible to provide a more comfortable means of transportation without causing the passenger to take time and effort.

Further, by starting the sensor system 30 and the ECU 20 from the time when the door lock is released, the collection of the attributes of the passenger can be started before boarding. Therefore, it is possible to propose an appropriate driving mode and select it by the passenger during the time until the departure becomes possible, and it is possible to provide the passenger with a stress-free usage environment.

From these facts, according to the embodiment, it is possible to provide an operation state control device and an operation state control method with enhanced serviceability to passengers.

The present invention is not limited to the above embodiment. For example, an automatic driving system can be read as an automatic driving support system. Further, if there is a camera having a performance capable of acquiring detailed attribute information to the extent necessary, the desired processing may be performed based only on the attribute information acquired by the camera.

The execution process is performed by recording the program for realizing each device and the system described above on a computer-readable recording medium, causing the computer system to read the program recorded on the recording medium, and executing the program. You may. The "computer system" referred to here may include hardware such as an OS and peripheral devices.

Although embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. This novel embodiment can be implemented in various other embodiments, and various omissions, replacements, and changes can be made without departing from the gist of the invention. This embodiment and its modifications are included in the scope and gist of the invention, and are also included in the scope of the invention described in the claims and the equivalent scope thereof.

10 ... Automatic operation system, 20 ... ECU, 21 ... Image processing unit, 22 ... Door lock release detection processing unit, 23 ... Door open / close detection processing unit, 24 ... Beacon reception processing unit, 25 ... Voice processing unit, 26 ... Correlation processing Unit, 27 ... User interface unit, 28 ... Operation mode selection processing unit, 29 ... Luggage loading support processing unit, 30 ... Sensor system, 31 ... Outside camera, 32 ... Inside camera, 33 ... Door unlock sensor, 34 ... Door open / close Sensor, 35 ... Beacon receiver, 36 ... Microphone, 37 ... Seat pressure sensor, 40 ... Display, 50 ... Speaker, 60 ... Rear door lock control unit, 70 ... Lighting, 80 ... Beacon transmitter.

Claims (14)

An image processing unit that processes image data acquired by a camera including the outside of the vehicle in the field of view to acquire image information in the field of view of the occupant of the vehicle.
A correlation processing unit that generates attribute information of the passenger by correlation processing using the image information,
A mode processing unit for specifying an operation mode for the passenger based on the attribute information is provided.
The mode processing unit
Based on the attribute information, the normal first operation mode and the second operation mode, which is less burdensome to the passenger than the first operation mode, are presented to the user interface.
An operation state control device that specifies an operation mode specified by a user in the user interface as an operation mode for the passenger. The image processing unit processes the image data acquired by the image sensor including the inside of the vehicle in the field of view to acquire the image information inside the vehicle.
The operating state control device according to claim 1, wherein the correlation processing unit generates attribute information of the passenger by correlation processing between the image information of the passenger and the image information in the vehicle. Further, it is provided with a beacon reception processing unit that processes a beacon signal received by a beacon receiver loaded on the vehicle to acquire beacon information.
The operating state control device according to claim 2, wherein the correlation processing unit generates attribute information of the passenger by the correlation processing using the beacon information. Further, it is provided with a voice processing unit that processes a voice signal output from a microphone loaded on the vehicle and acquires voice information.
The operating state control device according to claim 2, wherein the correlation processing unit generates attribute information of the passenger by the correlation processing using the voice information. Further, it is provided with a door lock release detection processing unit that processes detection information from a lock sensor that detects the door lock state of the vehicle.
The operation state control device according to claim 2, wherein the correlation processing unit starts the correlation processing when the door lock release is detected by the lock sensor. Further, it is provided with a door open / close detection processing unit that processes detection information from the door sensor that detects the open / closed state of the door of the vehicle.
The operating state according to claim 2, wherein the correlation processing unit generates attribute information of a occupant from the door based on a time-series change of the open state and the closed state of the door detected by the door sensor. Control device. The driving state control device according to claim 2, wherein the correlation processing unit generates attribute information of a occupant of a seat that is sensitive to a pressure sensor that detects pressure on the seat of the vehicle. The driving state control device according to claim 1, wherein the second driving mode is a driving mode in which the acceleration / deceleration rate is smaller than usual, or a lane change control in which lateral acceleration is suppressed. The driving state control device according to claim 8, wherein the mode processing unit notifies the automatic driving system of the vehicle of the specified driving mode. The driving state control device according to claim 8, wherein the mode processing unit notifies the automatic driving system of the vehicle of a combination of vehicle control parameters reflecting the specified driving mode. The driving state control device according to claim 10, wherein the vehicle control parameter is at least one of an acceleration / deceleration range, a speed range, and a movement path. The driving state control device according to claim 1, wherein the correlation processing unit generates attribute information regarding whether or not the passenger uses a walking aid. The image processing unit passes information on the presence or absence of a child seat acquired based on the image information in the vehicle to the correlation processing unit.
The operating state control device according to claim 2, wherein the correlation processing unit reflects information regarding the presence or absence of the child seat in the correlation processing to generate attribute information of the passenger. In the driving state control method applicable to the computer mounted on the vehicle,
The computer processes the image data acquired by the camera including the outside of the vehicle in the field of view to acquire the image information in the field of view of the passenger of the vehicle.
The computer generates attribute information of the passenger by correlation processing using the image information.
The computer identifies the driving mode for the passenger based on the attribute information.
Based on the attribute information, the computer presents to the user interface a normal first operation mode and a second operation mode in which the burden on the passenger is less than that of the first operation mode.
A driving state control method comprising: the computer specifying a driving mode specified by a user in the user interface as a driving mode for the passenger.

Images