JP7379253B2 - Behavior estimation system and behavior estimation method - Google Patents

Description

The present invention relates to a behavior estimation system and a behavior estimation method.

2. Description of the Related Art A technique is known in which the behavior of the head of a passenger riding a moving body and the behavior of the moving body are acquired, and the head motion performed by the passenger is estimated based on the difference between these behaviors.

For example, in the technique disclosed in Patent Document 1, a motion sensor of a mobile phone fixed to an instrument panel of a moving object detects the behavior of the moving object. Additionally, a sensor in a wearable device attached to the passenger detects the behavior of the passenger's head. Then, based on the difference between the detected behavior of the moving object and the behavior of the passenger's head, the head movement performed by the passenger is estimated.

JP2017-77296A

However, the technique disclosed in Patent Document 1 estimates the bodily behavior of the passenger using the behavior of the moving body detected by a motion sensor fixed at one location within the moving body. Since the behavior of the occupant's body caused by the behavior of the moving body differs depending on the position of the occupant on the moving body, the technique of Patent Document 1 has a problem in that the body behavior of the occupant cannot be accurately estimated.

The problem to be solved by the present invention is to provide a behavior estimation system and a behavior estimation method that can accurately estimate the physical behavior of a passenger of a moving body.

The present invention detects the physical behavior of a passenger riding a mobile body, detects the position of the passenger on the mobile body as the passenger position, detects the behavior of the mobile body at a plurality of installation positions of the mobile body, Based on the positional relationship between the passenger position and multiple installation positions on the mobile body and the behavior of the mobile body at the multiple installation positions, the behavior of the mobile body at the passenger position is estimated, and the detected body behavior and the behavior of the mobile body are estimated. The above problem is solved by estimating the bodily behavior of the occupant based on the behavior of the moving body at the occupant's position.

According to the present invention, it is possible to accurately estimate the physical behavior of a passenger of a moving body.

FIG. 1 is a diagram showing an example of the configuration of a behavior estimation system in the first embodiment. FIG. 2 is a schematic diagram showing the acceleration detected by the sensor when the vehicle turns left. FIG. 3A is a diagram showing an example of the positions of the plurality of moving body behavior detection devices 11 in this embodiment. FIG. 3B is a diagram showing an example of the positions of the plurality of moving object behavior detection devices 11 in this embodiment. FIG. 4 is a diagram showing the positional relationship between the interior of a vehicle traveling in the traveling direction and the passenger's line of sight. FIG. 5A is a diagram showing the interior of the vehicle as seen through the wearable device when the passenger is looking in the direction of travel of the vehicle. FIG. 5B is a diagram showing the interior of the vehicle as seen through the wearable device when the passenger is looking to the left with respect to the direction of travel of the vehicle. FIG. 6 is a diagram showing the positional relationship between the interior of a vehicle that is making a left turn and the passenger's line of sight. FIG. 7 is a diagram showing the interior of the vehicle as seen in the passenger's field of view when the display image is controlled based on the behavior of the passenger's head, including the behavior of the vehicle. FIG. 8 is a flowchart showing an example of a procedure for behavior estimation in this embodiment. FIG. 9 is a diagram illustrating an example of the configuration of a behavior estimation system in the second embodiment.

≪First embodiment≫
An embodiment of the behavior estimation system according to the present invention will be described based on the drawings.

The configuration of the behavior estimation system according to this embodiment will be explained using FIG. 1. FIG. 1 is a block diagram showing an example of the functional configuration of a behavior estimation system 1 in this embodiment.

The behavior estimation system 1 includes an in-vehicle device 10 and a wearable device 20. The in-vehicle device 10 is a device mounted on the vehicle 2. The in-vehicle device 10 includes at least a moving object behavior detection device 11, a controller 12, an in-vehicle communication device 13, a camera 14, and a seat sensor 15. Further, the wearable device 20 is a device that is attached to a passenger riding in the vehicle 2. The wearable device 20 also includes a passenger behavior detection device 21, a display section 22, and a communication device 23. The wearable device 20 is, for example, a glasses-type or goggle-type head-mounted display, and is attached to the passenger's head to detect the behavior of the passenger's head. In addition, the wearable device 20 causes the display unit 22, which is a display or the like, to display a display image of the virtual object so as to be superimposed on the scenery seen in the passenger's field of view. In the present embodiment, the behavior estimation system 1 is configured by a device mounted on a vehicle, but is not limited to a vehicle, and may be mounted on another moving body. For example, it may be a train, a ship, an airplane, etc.

Furthermore, the behavior estimation system 1 estimates the physical behavior of a passenger located at an arbitrary position within the vehicle 2, and controls image display on the display unit 22 based on the estimated physical behavior of the passenger. The body behavior by the passenger is the body behavior that occurs due to the actions performed by the passenger, that is, the active body behavior of the passenger. For example, the behavior of the head occurs when the occupant changes the direction of the head, and the behavior of the arms occurs when the occupant moves his arms. Further, as the behavior of the body, at least one of acceleration and angular velocity of the body is detected. In this embodiment, the sensor of the wearable device 20 attached to the passenger's head detects the behavior of the passenger's head. Therefore, when the vehicle 2 is stationary and the occupant moves his or her head, the sensor can detect the behavior of the occupant's head. However, when the vehicle 2 is running, the behavior of the occupant's head detected by the sensor includes not only the behavior of the occupant's head but also the behavior of the head caused by the behavior of the vehicle 2. That is, passive head behavior is also included. For example, even if the passenger moves his or her head, the movement of the passenger's head also occurs due to the acceleration, deceleration, or turning of the traveling vehicle 2, so the sensor detects the combination of the two movements. Detects the behavior of the passenger's head. In such a situation, it is difficult for the sensor to appropriately detect only the head behavior of the passenger. Therefore, conventionally, in addition to detecting the behavior of the passenger's head using the sensor of the wearable device 20, the behavior of the vehicle 2 is detected, and the vehicle By subtracting the behavior of 2, only the behavior of the head by the occupant is estimated.

However, the influence that the behavior of the vehicle 2 has on the behavior of the occupant's head differs depending on the occupant's position within the vehicle 2. For example, the behavior of the vehicle 2 differs depending on the position within the vehicle 2, as shown in FIG. FIG. 2 shows the acceleration sensors installed at three locations inside the vehicle 2 (front sensor 200, center sensor 201, and rear sensor 202) detecting the results when the vehicle 2 is turning around a leftward curve. FIG. 2 is a diagram schematically showing the acceleration of a vehicle. In Figure 2, the direction and magnitude of the acceleration detected by each sensor are decomposed into components in the X-axis direction (traveling direction of vehicle 2) and Y-axis direction (width direction of vehicle 2), and the direction and magnitude of the vector are It is expressed by For example, the direction of acceleration at the rear sensor 202 is opposite to the direction of acceleration at the front sensor 200 in the Y-axis direction. Further, the magnitude of the acceleration at the central sensor 201 is smaller than the magnitude of the acceleration at the front sensor 200 in the Y-axis direction. In this way, the behavior of the vehicle 2 differs depending on the position within the vehicle 2. Therefore, in order to accurately estimate the physical behavior of a passenger sitting at an arbitrary position in a vehicle, it is necessary to estimate different vehicle behaviors depending on the position of the passenger. The behavior estimation system 1 according to the present embodiment estimates the behavior of the vehicle at the position of the passenger, and subtracts the behavior of the vehicle at the position of the passenger from the behavior of the head of the passenger detected by the sensor. Estimating the head behavior of the occupant at the occupant's position. Thereby, in this embodiment, the physical behavior of the occupant of the moving body can be accurately estimated. In addition, when displaying a display image on the display unit 22 of the wearable device 20 according to the passenger's head behavior, the image is appropriately displayed in the passenger's field of vision based on the estimated body behavior of the passenger. It can be displayed. For example, when displaying a display image so as to overlap an object inside the vehicle that appears in the passenger's field of vision, the display position of the display image on the display unit 22 may be controlled in accordance with the behavior of the passenger's head. This is because the behavior of the passenger's head must be accurately estimated.

The moving object behavior detection device 11 detects the behavior of the vehicle 2. The moving object behavior detection device 11 includes, for example, an acceleration sensor 110 and a gyro sensor 111. Furthermore, the mobile object behavior detection device 11 is installed at a plurality of different installation positions within the vehicle 2, respectively. That is, a plurality of moving body behavior detection devices 11 are fixed to the vehicle 2. For example, at least three moving body behavior detection devices 11 each including a two-axis acceleration sensor 110 are installed at different positions on the same plane and the same circle. Note that the installation positions of the mobile object behavior detection devices 11 do not have to be different positions on the same plane and the same circle. Furthermore, six moving object behavior detection devices 11 each including a single-axis acceleration sensor 110 may be installed at different positions. The moving object behavior detection device 11 detects the behavior of the vehicle 2 at the installed installation position, that is, at least one of the acceleration and angular velocity of the vehicle 2. For example, as shown in FIG. 3A, a plurality of moving object behavior detection devices 11 are mounted on each of the four corners (front, rear, left, and right) of the frame of the vehicle 2, which can be treated as a rigid body. Furthermore, the present invention is not limited to this, and as shown in FIG. 3B, three moving object behavior detection devices 11 may be mounted on the vehicle 2. Thereby, the behavior of the vehicle 2 at the installation position of the vehicle 2 can be detected. The plurality of mobile object behavior detection devices 11 each detect the behavior of the vehicle 2 at the installed installation position, and transmit the detected values to the controller 12. At this time, the mobile object behavior detection device 11 also transmits the position of each device expressed in three-dimensional coordinates.

The behavior of the vehicle 2 includes behavior caused by the driving operation of the vehicle 2 and behavior caused by the external environment. Behaviors caused by driving operations of the vehicle 2 include, for example, acceleration of the vehicle 2 due to an accelerator operation, deceleration of the vehicle 2 due to a brake operation, and right/left turns of the vehicle 2 due to a steering operation. For example, when the vehicle 2 starts from a stopped state, the vehicle 2 is accelerated by operating the accelerator. A force accompanying the acceleration of the vehicle 2 is applied to the passenger, and an acceleration is generated in the wearable device 20 in the traveling direction of the vehicle 2 (also referred to as the longitudinal direction of the vehicle 2). Further, for example, in a scene where the vehicle 2 is decelerating, the vehicle 2 is decelerated by a brake operation. A force accompanying the deceleration of the vehicle 2 is applied to the passenger, and an acceleration is generated in the wearable device 20 in the traveling direction. The direction of acceleration generated in wearable device 20 due to acceleration of vehicle 2 and the direction of acceleration generated in wearable device 20 due to deceleration of vehicle 2 are opposite to each other. Further, for example, in a scene where the vehicle 2 turns left at an intersection, the vehicle 2 turns left by steering operation. A centrifugal force is applied to the passenger as the vehicle 2 turns left, and an acceleration occurs in the wearable device 20 in the right direction with respect to the direction of travel. Note that another example of the behavior caused by the driving operation of the vehicle 2 is vibration due to idling. In this case, wearable device 20 is subject to at least acceleration in the vertical direction of vehicle 2 .

Further, an example of a scene where acceleration occurs in the wearable device 20 due to the external environment is a scene where the vehicle 2 passes a step on the road surface such as a speed bump. For example, even if the vehicle 2 is traveling at a constant speed without accelerating, when passing a speed bump, the vehicle 2 moves up and down due to the difference in level of the speed bump. Even in a situation where the passenger is stationary and no acceleration is generated due to the passenger's movements, and the vehicle 2 is traveling at a constant speed and no acceleration is generated due to the driving operation of the vehicle 2, Force is applied to the occupant due to the external environment, and acceleration is generated in the wearable device 20. In the scene of the above example, acceleration in the vertical direction of the vehicle 2 is generated in the wearable device 20 .

Further, as the behavior of the vehicle 2, angular velocity may be detected. For example, in a scene where the vehicle 2 turns left at an intersection, the centrifugal force accompanying the left turn of the vehicle 2 is detected as an angular velocity. Additionally, if only one wheel runs over a step while the vehicle 2 is running, the body of the vehicle 2 will tilt in the width direction, so the angular velocity generated around the axis along the traveling direction of the vehicle 2 will not be detected. Ru.

Acceleration sensor 110 detects acceleration of vehicle 2 caused by driving operation of vehicle 2 and external environment. Specifically, the acceleration sensor 110 detects acceleration along the three axes, the X-axis, Y-axis, and Z-axis, which are perpendicular to each other. For example, the The axis is along the height direction (vertical direction). Further, the acceleration sensor 110 may detect acceleration in the axial direction about one axis or two axes. The acceleration acquired by the acceleration sensor 110 is transmitted to the controller 12. Note that in this embodiment, the sensor that detects the behavior of the vehicle may be only one of the acceleration sensor 110 and the gyro sensor 111.

The gyro sensor 111 detects the angular velocity of the vehicle 2 caused by a change in the attitude of the vehicle 2. The attitude change of the vehicle 2 detected by the gyro sensor 111 is a rotational behavior caused by the driver's operation such as acceleration/deceleration and steering. Specifically, the gyro sensor 111 detects angular velocities about the respective axes in the above-mentioned axial directions. The angular velocity includes an angular velocity of a roll angle centered on the X-axis, an angular velocity of a pitch angle centered on the Y-axis, and an angular velocity of a yaw angle centered on the Z-axis. The angular velocity acquired by the gyro sensor 111 is transmitted to the controller 12.

The camera 14 is installed to take images of the inside of the vehicle 2. Specifically, the camera 14 captures images of the inside of the vehicle 2 including the passenger at regular time intervals. For example, the camera 14 may be a camera device using an infrared LED and an infrared camera. Image data captured by the camera 14 is output to the controller 12.

The seat sensor 15 detects the load applied to the seat in the vehicle 2. Specifically, the seat sensor 15 outputs an ON value to the controller 12 when it is detected that a load of a predetermined value or more is applied to the seat. The seat sensor 15 can also be used when the camera 14 is in some kind of shadow and cannot capture an image of the passenger. Further, if only a sheet sensor is used, the cost of the sensor device can be reduced.

The controller 12 is capable of communicating with the moving object behavior detection device 11, the in-vehicle communication device 13, the camera 14, and the seat sensor 15. The controller 12 in this embodiment includes a computer having hardware and software, and this computer includes a ROM (Read Only Memory) that stores programs and a CPU (Central Processing Unit) that executes the programs stored in the ROM. and a RAM (Random Access Memory) that functions as an accessible storage device. In addition, as an operating circuit, instead of or in addition to the CPU, an MPU (Micro Processing Unit), a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array) etc. can be used. . The controller 12 is configured to include, as functional blocks, an occupant position acquisition section 120, a moving object behavior estimation section 121, an occupant behavior estimation section 122, and an image display control section 123, and realizes each of the above functions. Alternatively, each function is executed by cooperation between software for executing each process and hardware. Specifically, the controller 12 first determines the positional relationship between the position of the passenger in the vehicle 2 and the plurality of installation positions where the mobile object behavior detection device 11 is installed, and the behavior of the vehicle 2 at the plurality of installation positions. Based on this, the behavior of the vehicle 2 at the position of the passenger is estimated. Then, the controller 12 estimates the difference between the body behavior of the passenger and the behavior of the vehicle 2 at the position of the passenger as the body behavior of the passenger, and Control is performed to display a display image on the display unit 22 according to the direction of the user's line of sight. In this embodiment, the functions of the controller 12 are divided into four blocks, and the functions of each functional block will be explained. However, the functions of the controller 12 do not necessarily need to be divided into four blocks, and can be divided into three or less It may be divided into functional blocks or five or more functional blocks.

The passenger position acquisition unit 120 acquires the position of the passenger in the vehicle 2 as the passenger position based on information detected by the camera 14 and the seat sensor 15. For example, the passenger position acquisition unit 120 identifies the passenger using image recognition technology based on the image inside the vehicle 2 captured by the camera 14, and acquires the position of the passenger. In addition, the passenger position acquisition unit 120 acquires the passenger position of the passenger wearing the wearable device 20 by recognizing feature point markers attached to the wearable device 20 worn by the passenger. You can also do it. Further, the passenger position acquisition unit 120 may acquire the passenger position by specifying the position of the seat where the ON value is output based on the ON value output by the seat sensor 15. When there are multiple passengers in the vehicle, the passenger position acquisition unit 120 acquires the passenger position of each passenger. The passenger position within the vehicle 2 is represented by, for example, three-dimensional coordinates.

The moving object behavior estimation unit 121 estimates the behavior of the vehicle 2 at the passenger position acquired by the passenger position acquisition unit 120. Specifically, the mobile object behavior estimation unit 121 first acquires the passenger position acquired by the passenger position acquisition unit 120 and the plurality of installation positions where the plurality of mobile object behavior detection devices 11 are installed. At this time, the moving object behavior estimation unit 121 acquires the behavior of the vehicle 2 at the installation position in association with the position information of each moving object behavior detection device 11. Next, based on the passenger position and the positions of the plurality of moving object behavior detection devices 11, the moving object behavior estimation unit 121 calculates the passenger position and the plurality of moving objects in three-dimensional coordinates with the passenger position as the origin. The relative positional relationship with the position of the behavior detection device 11 is calculated. Then, the mobile object behavior estimation unit 121 calculates the vehicle behavior at the passenger position based on the behavior of the vehicle 2 at the plurality of installation positions and the relative positional relationship of each mobile object behavior detection device 11 with respect to the passenger position. 2, that is, the acceleration in each of the X, Y, and Z axis directions, and the angular velocity around each of the X, Y, and Z axes are calculated. Specifically, first, the mobile object behavior estimation unit 121 calculates the relative positional relationship between the passenger position and the positions of the plurality of mobile object behavior detection devices 11 in three-dimensional coordinates with the passenger position as the origin. calculate. At this time, if the vehicle 2, which is a rigid body, is rotating and an angular velocity is generated around the passenger position, the installation position of each moving object behavior detection device 11 is rotated according to the positional relationship with the passenger position. Angular velocity and centrifugal acceleration result. Next, the moving object behavior estimation unit 121 acquires the behavior of the vehicle 2 at the installation positions of the moving object behavior detection device 11 at different positions and directions. The behavior of the vehicle 2 detected by one moving body behavior detection device 11 is a value that is a combination of translational acceleration, gravitational acceleration, rotational angular velocity and centrifugal acceleration that differ depending on the positional relationship with the passenger position. Therefore, the influence of the rotational angular velocity and centrifugal acceleration, which differ depending on the positional relationship with the passenger position, cannot be separated by using only a single detected value. Therefore, the moving object behavior estimation unit 121 integrates the behavior values of each vehicle 2 at a plurality of installation positions, and calculates the position relative to the passenger position from the value detected by the moving object behavior detection device 11. The acceleration and angular velocity at the passenger position are calculated by removing the influence of rotational angular velocity and centrifugal acceleration that differ depending on the relationship.

The passenger behavior estimating unit 122 estimates the head behavior of the passenger. Specifically, the passenger behavior estimation unit 122 uses the behavior of the passenger's head detected by the passenger behavior detection device 21 and the behavior of the vehicle 2 at the passenger position estimated by the moving object behavior estimation unit 121. Calculate the difference from the behavior. That is, the difference between the behavior of the passenger's head and the behavior of the vehicle 2 at the passenger's position is estimated as the behavior of the head by the passenger. For example, when estimating the acceleration of the passenger's head, the passenger behavior estimation unit 122 estimates the acceleration of the passenger's head and the acceleration of the vehicle 2 at the passenger position in three axial directions (X-axis, The acceleration components of each of the Y-axis and Z-axis are obtained, and the difference between the acceleration of the passenger's head and the acceleration of the vehicle 2 at the passenger's position is calculated for each acceleration component. In addition, when estimating the angular velocity of the head of the occupant, the occupant behavior estimation unit 122 estimates the angular velocity of the occupant's head and the angular velocity of the vehicle 2 at the occupant's position, based on the angular velocity around three axes. are obtained, and the difference between the angular velocity of the passenger's head and the angular velocity of the vehicle 2 at the passenger's position is calculated for each axis.

The image display control unit 123 controls the display form of the display image of the virtual object displayed on the display unit 22 based on the head behavior of the passenger estimated by the passenger behavior estimation unit 122. The display form of the display image includes the display position of the display image on the display unit 22, the size of the display image, and the rotation angle of the display image. The image display control unit 123 first calculates the state of the passenger's head that changes depending on the passenger's head behavior estimated by the passenger behavior estimation unit 122, that is, the position of the head relative to the vehicle 2. , determine the orientation and tilt. Next, the image display control unit 123 specifies the visual field in the passenger's line of sight based on the condition of the passenger's head, and displays the display so that it overlaps the object reflected in the visual field in the passenger's line of sight. The display form of the display image of the virtual object displayed in 22 is specified. Then, the image display control unit 123 generates a control signal for displaying a display image based on the specified display format, and outputs the control signal to the in-vehicle communication device 13.

Here, virtual objects will be explained. A virtual object is a two-dimensional model (also referred to as a 2D model, 2D CG, etc.) or a three-dimensional model (also referred to as a 3D model, 3D CG, etc.) placed in real space, and is generated using a computer. This is an image. Virtual objects include both still images and moving images. In addition to virtual objects that do not actually exist (for example, anime characters), virtual objects include, for example, images of real people. Furthermore, in addition to objects representing people, virtual objects include, for example, physical objects such as information display boards at intersections. In this embodiment, by displaying a display image of a virtual object on the display unit 22 for the passenger who is viewing the scenery inside and outside the vehicle via the display unit 22 of the wearable device 20, the virtual object is displayed in the real space. Presenting a superimposed augmented reality (AR) space.

Next, the display form of the display image of the virtual object will be explained. The image display control unit 123 identifies the direction of the passenger's line of sight estimated from the behavior of the passenger's head, and controls the display form of the display image so that it overlaps with the object reflected in the field of view in the direction of the passenger's line of sight. . Specifically, the image display control unit 123 estimates the position, orientation, and inclination of the passenger's head from the head behavior of the passenger, and changes the head position, orientation, and inclination according to the estimated head position, orientation, and inclination. The position, size, and angle of the display image on the display unit 22 are controlled so that the display image is superimposed on the object reflected in the field of view. First, the display position of the display image will be explained. The display position of the display image is the position where the display image is displayed on the display unit 22 (display screen). For example, when a display image is displayed on the display unit 22 so as to overlap with an object inside the vehicle that is visible to the passenger, the image display control unit 123 identifies the position of the object inside the vehicle, and Based on the state of the head of , the position of the object on the display unit 22 corresponding to the position of the object in real space is specified as the display position. Next, the display form of the display image includes the size of the display image. By controlling the size of the displayed image and reproducing the sense of perspective in which nearby objects appear larger, it is possible to provide an AR space that more closely resembles reality. When a display image is displayed on an object in the vehicle, such as an instrument panel, the image display control unit 123 controls the size of the display image according to the forward and backward movement of the position of the passenger's head relative to the instrument panel. control When the passenger's head moves in a direction toward the instrument panel, the image display control unit 123 performs control to enlarge the displayed image, and the passenger's head moves in a direction away from the instrument panel. At times, the image display control unit 123 performs control to reduce the size of the displayed image. Furthermore, the display form of the display image includes the rotation angle of the display image. The rotation angle of the display image is the angle of the display image on the display unit 22. For example, consider a case where the rotation angle of the display image is controlled in accordance with the inclination of the object inside the car so that the display image is superimposed on the object inside the car. That is, in this case, when the vehicle 2 tilts, the image display control unit 123 controls the rotation angle of the display image according to the tilt of the vehicle 2, and when the tilt of the vehicle 2 does not change, the image display control unit 123 controls the rotation angle of the display image according to the tilt of the vehicle 2. Do not change the angle. Furthermore, although the tilt of the vehicle 2 does not change, when the display unit 22 (display screen) is tilted due to the passenger tilting his head, etc., the image display control unit 123 controls the display image to be tilted with respect to the object inside the vehicle. The rotation angle of the displayed image on the display unit 22 is controlled so that the displayed image is not visible.

An example of controlling the display form of a display image will be described below with reference to FIGS. 4 to 7. FIG. 4 is a diagram showing the relationship between the vehicle 2 and the line of sight direction of a passenger sitting in the driver's seat. In FIG. 4, the vehicle 2 includes a navigation device 300 and is traveling along a traveling direction L. In FIG. Further, in the vehicle 2, a passenger 100 wearing a wearable device 20 is sitting in the driver's seat. The line of sight direction P is a direction parallel to the direction of travel L, and is the direction of the line of sight when the passenger 100 is looking in the direction of travel. The line-of-sight direction Q is the line-of-sight direction when the passenger 100 is looking in a direction to the left of the line-of-sight direction P. Furthermore, in the wearable device 20, a virtual object R is displayed at a position on the display unit 22 that corresponds to the position of the navigation device 300 in the vehicle interior space. FIGS. 5A and 5B show the interior of the vehicle as viewed by the passenger 100 in FIG. 4 via the display unit 22 (display screen). FIG. 5A corresponds to the visual field of the passenger 100 in the line of sight direction P in FIG. 4, and shows the inside of the car as seen in the passenger's visual field when the virtual object R is displayed on the display unit 22. . FIG. 5B corresponds to the visual field of the passenger 100 in the viewing direction Q in FIG. There is. The virtual object R is, for example, an object that visually displays the route guidance provided by the navigation device 300 (for example, “2 kilometers ahead, turn left”), and is always displayed at the position of the navigation device 300 on the display unit 22. controlled so that Therefore, for example, when the line-of-sight direction of the passenger 100 changes from the line-of-sight direction P to the line-of-sight direction Q, the navigation device 300 on the display unit 22 corresponds to the visual field of the passenger 100. The position of moves to the right (Fig. 5B). Since the virtual object R is controlled to always be displayed at the position of the navigation device 300, the display position of the virtual object R is also controlled to move in accordance with the movement of the navigation device.

Further, as shown in FIG. 6, when the vehicle 2 is turning left, the head of the passenger 100 also experiences leftward acceleration due to the behavior of the vehicle 2 that causes leftward acceleration. When the passenger 100 does not move his head and does not change his line of sight from the traveling direction P, the state of the passenger's head relative to the vehicle 2, that is, the position, orientation, and inclination of the head, do not change, so the driver The inside of the car that appears in the field of vision should be the same as the inside of the car shown in FIG. 5A. However, when the display form of the display image is controlled simply based on the behavior of the passenger's head detected by the passenger behavior detection device 21, the , the display position of the virtual object R on the display unit 22 will move. That is, as shown in FIG. 7, the image display control unit 123 determines that the position of the passenger's head relative to the vehicle 2 has changed from the line-of-sight direction P to the line-of-sight direction P' due to leftward acceleration of the vehicle 2. Then, control is performed to move the display position of the virtual object R to the right. However, in reality, the position of the navigation device 300 on the display unit 22 does not change because the state of the passenger's head relative to the vehicle 2 has not changed. Therefore, the virtual object R is displayed at a position shifted to the right from the position of the navigation device 300 on the display unit 22. In contrast, in the present embodiment, by subtracting the head behavior caused by the vehicle behavior from the passenger's head behavior detected by the passenger behavior detection device 21, the The display position of the virtual object R can be controlled by Therefore, even if the vehicle 2 is turning left, the passenger 100 can visually recognize the interior of the vehicle shown in FIG. 5A. Furthermore, as shown in FIG. 2, when the vehicle 2 is turning left, the behavior of the vehicle 2 differs depending on the position within the vehicle 2. Therefore, although FIG. 7 shows an example in which the passenger behavior detection device 21 detects leftward acceleration when the vehicle 2 is turning left, the passenger 100 is not in the driver's seat but in the rear seat. If the passenger is sitting in the vehicle, the passenger behavior detection device 21 detects acceleration in the right direction. In this case, the display image of the virtual object on the display unit 22 is displayed so as to move to the left. Therefore, in order to correct such an image display, the controller 12 subtracts the acceleration in the right direction caused by the behavior of the vehicle from the acceleration of the head detected by the occupant behavior detection device 21. It is necessary to estimate the acceleration of the head by the person. However, in the conventional technology, the behavior of the vehicle is detected as acceleration in the left direction by a single sensor installed in the front, so it is not possible to accurately estimate the behavior of the head of the passenger in the rear seat. The invention according to this embodiment estimates the behavior of the vehicle 2 at the passenger position and calculates the difference between the behavior of the head of the passenger and the behavior of the vehicle 2 at the passenger position. Even if the occupant is in different positions, it is possible to estimate the head behavior of the occupant at each position. For example, in the above-mentioned example, the controller 12 estimates the head behavior of the occupant at the front of the vehicle 2 based on the leftward acceleration of the vehicle 2, and estimates the head behavior of the occupant at the rear of the vehicle 2. The behavior of the vehicle 2 is estimated based on the acceleration of the vehicle 2 in the right direction.

Note that in this embodiment, the display position of the virtual object is controlled based on the object inside the vehicle 2, but the display image of the virtual object is superimposed on the object outside the vehicle 2. Similar control is also performed when displaying the image. That is, in this embodiment, the position of the object outside the vehicle 2 is acquired as a relative position with respect to the position and orientation of the vehicle 2. Therefore, even when displaying a virtual object so as to overlap an object outside the vehicle 2, the image display control unit 123 can identify the state of the passenger's head with respect to the vehicle 2, thereby displaying the object outside the vehicle in real space. The position of the object outside the vehicle on the display unit 22 corresponding to the position is specified. Examples of objects outside the vehicle include vehicles, pedestrians, roads, signs, and buildings.

The in-vehicle communication device 13 is connected wirelessly or wired to the communication device 23 of the wearable device 20, and transmits and receives information and the like. For example, the in-vehicle communication device 13 transmits an image display control signal to the communication device 23 of the wearable device 20, and receives the detected bodily behavior of the passenger from the communication device 23.

The wearable device 20 is a device worn on the passenger's head, and is, for example, a goggle-type or glasses-type head-mounted display. Wearable device 20 includes at least a passenger behavior detection device 21, a display section 22, and a communication device 23. The wearable device 20 causes the display unit 22 to display an image of the virtual object based on the control signal output from the image display control unit 123. Further, the wearable device 20 may be a device worn on the arm of the passenger. The wearable device 20 worn on the arm detects arm behavior using the occupant behavior detection device 21 and transmits the detected arm behavior to the controller 12 . Further, if there are multiple passengers, each of them can wear and use the wearable device 20, and it is also possible to share the same virtual object with each other. In that case, each wearable device 20 detects the behavior of the passenger's head for each passenger and transmits it to the controller 12. Note that the wearable device 20 may be an AR or VR headset, a controller, a sensor device attached to clothing or directly on the body with a belt, or the like.

The passenger behavior detection device 21 detects the behavior of the passenger's body in the area where the wearable device 20 is attached. For example, the passenger behavior detection device 21 detects the behavior of the head to which the wearable device 20 is attached. The behavior of the passenger's head detected by the passenger behavior detection device 21 includes head behavior by the passenger and head behavior based on the behavior of the vehicle 2. The passenger behavior detection device 21 includes an acceleration sensor 210 and a gyro sensor 211. The passenger behavior detection device 21 outputs the detected behavior of the passenger's head to the controller 12.

Acceleration sensor 210 detects the acceleration of the passenger's head. Specifically, the acceleration sensor 210 detects acceleration along the three axes, the X-axis, Y-axis, and Z-axis, which are perpendicular to each other. The acceleration of the passenger's head detected by the acceleration sensor 210 includes acceleration due to the behavior of the passenger's head and acceleration due to the behavior of the vehicle 2. Therefore, for example, even if the passenger is stationary and does not move his head, if the passenger's head moves due to the behavior of the vehicle 2, the acceleration sensor 210 detects the acceleration of the passenger's head. The acceleration acquired by the acceleration sensor 210 is output to the communication device 23.

Gyro sensor 211 detects the angular velocity of the passenger's head. Specifically, the gyro sensor 211 detects angular velocities about the respective axes mentioned above. The angular velocity of the passenger's head produced by the gyro sensor 211 includes the angular velocity due to the behavior of the passenger's head and the angular velocity due to the behavior of the vehicle 2. Changes in the attitude of the vehicle 2 occur as a result of the driver's operations such as acceleration/deceleration and steering. Therefore, for example, even if the occupant is stationary and does not move his or her head, if the occupant's head moves due to the behavior of the vehicle 2, the gyro sensor 211 detects the angular velocity of the occupant's head. The angular velocity acquired by the gyro sensor 211 is transmitted to the communication device 23.

The display unit 22 is a part of the wearable device 20 for displaying images of virtual objects. Examples of the display unit 22 include a display and goggles. The display unit 22 displays a display image of the virtual object at a display position on the display based on the control signal transmitted by the image display control unit 123. Thereby, the passenger wearing the wearable device 20 can visually recognize the real space on which the virtual object is superimposed via the display unit 22.

Next, the procedure of the behavior estimation method according to this embodiment will be explained using FIG. 8. FIG. 8 is a flowchart showing the procedure of the behavior estimation method according to this embodiment. In this embodiment, when the passenger wears the wearable device 20 and starts the AR system that causes the display unit 22 to display a display image of a virtual object, the control flow starts from step S801. In this embodiment, the controller 12 repeatedly executes the control flow of FIG. 8 while displaying the virtual object.

In step S801, the moving object behavior detection device 11 detects the behavior of the vehicle 2. Specifically, the plurality of mobile object behavior detection devices 11 detect the behavior of the vehicle 2 at the respective installation positions using the acceleration sensor 110 and the gyro sensor 111. Then, the mobile object behavior detection device 11 outputs data on the detected behavior of the vehicle 2 to the controller 12. Further, the plurality of moving body behavior detection devices 11 repeatedly detect the behavior of the moving body at predetermined time intervals.

In step S802, the controller 12 acquires the position of the passenger inside the vehicle 2. Specifically, the controller 12 acquires the passenger position based on sensor information detected by the camera 14 and the seat sensor 15. If there are multiple passengers, the passenger position is acquired for each passenger.

In step S803, the controller 12 estimates the behavior of the moving body at the passenger position. Specifically, the controller 12 determines the positional relationship between the passenger position acquired in step S802 and the positions of the plurality of moving object behavior detection devices 11, and the positional relationship of the vehicle 2 at the plurality of installation positions detected in step S801. Based on the behavior, the behavior of the moving object at the passenger position is estimated by calculation.

In step S804, the passenger behavior detection device 21 detects the behavior of the passenger's head. The detected head behavior of the passenger includes head behavior by the passenger and head behavior due to the behavior of the vehicle 2. The passenger behavior detection device 21 then transmits the detected behavior of the passenger's head to the controller 12 via the communication device 23.

In step S805, the controller 12 estimates the behavior of the passenger's head. Specifically, the controller 12 calculates the difference between the behavior of the passenger's head detected in step S804 and the behavior of the moving object at the passenger's position estimated in step S803 as the head behavior by the passenger. Estimated as.

In step S806, the controller 12 performs display control of the display image of the virtual object to be displayed on the display unit 22 according to the head behavior of the passenger estimated in step S805. Specifically, the controller 12 calculates the state of the head, that is, the position, orientation, and rotation angle of the head relative to the vehicle 2, based on the head behavior of the passenger estimated in step S805, and calculates Based on the determined state of the passenger's head, a control signal for controlling the image form of the displayed image of the virtual object is generated. Then, the controller 12 transmits the generated control signal to the wearable device 20 via the in-vehicle communication device 13.

In step S807, the display unit 22 that has received the control signal via the communication device 23 displays a display image of the virtual object in a display format based on the control signal. For example, the display unit 22 displays an image that emphasizes an object outside the vehicle that appears in the passenger's field of vision.

Note that in this embodiment, the display form of the display image is controlled based on the head behavior of the passenger estimated by the passenger behavior estimation unit 122, but the present invention is not limited to this. The display form of the display image may be controlled based on the arm behavior of the passenger estimated by the estimation unit 122. For example, the passenger operates the display image displayed on the display unit 22 by the behavior of the arm on which the wearable device 20 is attached. Specifically, the image display control unit 123 controls the position of the display image displayed on the display unit 22 in conjunction with the behavior of the passenger's arms. In this case as well, if the behavior of the vehicle 2 is included in the behavior of the arms detected by the passenger behavior detection device 21, the display position of the display image will move in a direction not intended by the passenger. By estimating the arm behavior of the passenger by the passenger behavior estimation unit 122, the passenger can accurately manipulate the display position of the display image based on the passenger's arm behavior. Further, in this embodiment, the gesture of the passenger may be recognized based on the behavior of the passenger's arms estimated by the passenger behavior estimation unit 122.

As described above, in this embodiment, the passenger behavior detection unit is attached to a passenger riding a mobile object and detects the bodily behavior of the passenger, and the passenger's position in the mobile object is detected as the passenger position. a plurality of moving object behavior detection sections installed at a plurality of installation positions of a moving object to detect the behavior of the moving object at the installation positions; a moving object behavior estimating section that estimates the behavior of the moving object at the passenger position based on the positional relationship of the moving object and the behavior of the moving object at the plurality of installation positions; The vehicle includes a passenger behavior estimating unit that estimates the physical behavior of the passenger based on the behavior of the moving object at the passenger's position. Thereby, the physical behavior of the occupant of the moving body can be accurately estimated.

Further, in this embodiment, a display unit that displays an image and an image display that controls the display form of the image displayed by the display unit based on the body behavior of the passenger estimated by the passenger behavior estimation unit The apparatus further includes a control section. Thereby, when presenting an augmented reality space to a traveling passenger, it is possible to accurately display a display image of a virtual object according to the bodily behavior of the passenger.

Further, in this embodiment, the passenger behavior detection section is attached to the head of the passenger and detects the behavior of the head. Thereby, it is possible to perform display control of the display image according to the behavior of the head.

Further, in the present embodiment, the passenger behavior detection section includes a head gyro sensor that measures the angular velocity of the head as the head angular velocity, and the moving object behavior detection section measures the angular velocity of the moving object as the moving object angular velocity. The passenger behavior estimation unit estimates the head behavior of the passenger based on the head angular velocity and the mobile body angular velocity. This makes it possible to estimate the relative angle of the head and accurately estimate the orientation of the head within the vehicle.

Further, in this embodiment, the passenger behavior detection section includes a head acceleration sensor that measures the acceleration of the head as head acceleration, and the moving object behavior detection section measures the acceleration of the moving object as moving object acceleration. The passenger behavior estimation unit estimates the head behavior of the passenger based on the head acceleration and the mobile body acceleration. This makes it possible to estimate the relative position of the head and accurately estimate the position of the head within the vehicle.

Further, in the present embodiment, the passenger behavior detection section is provided in a wearable device mounted on the passenger, and the moving object behavior detection section is provided in a mobile device mounted on the moving object, and the passenger behavior detection section is provided in a wearable device mounted on the passenger. The unit is included in the mobile device. This allows the device installed in a mobile object to have the function of estimating the physical behavior of the occupants, making it possible to centrally estimate the behavior of multiple wearable devices. .

Further, in this embodiment, the passenger behavior detection section is attached to the arm of the passenger and detects the behavior of the arm. Thereby, display control of the display image can be performed according to the behavior of the arm.

≪Second embodiment≫
Next, the behavior estimation system 1 according to the second embodiment will be described using FIG. 9. The behavior estimation system 1 according to the second embodiment is a modification of the behavior estimation system 1 according to the first embodiment. The system has the same configuration as the first embodiment and operates in the same manner as the first embodiment, except that it differs from the behavior estimation system 1 according to the first embodiment in the points described below. The description of the form is cited as appropriate. The configuration of this embodiment that differs from the first embodiment is the passenger behavior estimation section. In the first embodiment, the passenger behavior estimation unit was a functional block that constitutes an in-vehicle device, but in the second embodiment, as shown in FIG. 9, it is a functional block that constitutes a wearable device 20. . That is, in the second embodiment, the wearable device 20 estimates the bodily behavior of the passenger. Therefore, in this embodiment, when there are multiple passengers, the physical behavior of the passengers is estimated in each of the plurality of wearable devices 20.

In this embodiment, when the moving object behavior estimating unit 121 estimates the behavior of the vehicle 2 at the passenger position, the controller 12 transmits the behavior of the vehicle 2 at the passenger position to the communication device 23 via the in-vehicle communication device 13. do. Further, when the wearable device 20 receives the behavior of the vehicle 2 at the passenger position via the communication device 23, the passenger behavior estimation unit 24 calculates the behavior of the head of the passenger and the behavior of the vehicle 2 at the passenger position. The difference between the two is estimated as the passenger's head behavior. Wearable device 20 then transmits the estimated head behavior of the occupant to controller 12 via communication device 23 . In addition, the wearable device 20 may include an image display control section as a functional block. In this case, the wearable device 20 displays the display displayed on the display section 22 based on the head behavior of the occupant. Controls the display format of images.

Note that the flowchart in this embodiment is the same as the flowchart in FIG. 8, but the specific processing of the behavior estimation method differs from the first embodiment in the following points. That is, in this embodiment, in step S803, the controller 12 estimates the behavior of the vehicle 2 at the passenger position, and then transmits the behavior of the vehicle 2 at the passenger position to the wearable device 20. Further, after the passenger behavior detection device 21 detects the behavior of the passenger's head in step S804, the wearable device 20 estimates the head behavior by the passenger in step S805, and The head behavior of the passenger is transmitted to the controller 12.

As described above, in the present embodiment, the passenger behavior detection section is provided in a wearable device worn by the passenger, and the moving object behavior detection section is provided in a mobile device mounted on a moving object, and the passenger behavior detection section is provided in a mobile device mounted on a moving object. The human behavior estimation unit is included in the wearable device. With this, when there are multiple wearable devices, each wearable device performs processing, so the processing load can be reduced and the processing speed can be improved.

Note that the embodiments described above are described to facilitate understanding of the present invention, and are not described to limit the present invention. Therefore, each element disclosed in the above embodiments is intended to include all design changes and equivalents that fall within the technical scope of the present invention.

DESCRIPTION OF SYMBOLS 1... Behavior estimation system 10... Vehicle-mounted device 11... Mobile object behavior detection device 12... Controller 120... Occupant position acquisition section 121... Mobile object behavior estimation section 122... Occupant behavior estimation section 123... Image display control section 13... Vehicle-mounted communication Device 14...Camera 15...Seat sensor 20...Wearable device 21...Occupant behavior detection device 22...Display section 23...Communication device

Claims (9)

a passenger behavior detection unit that is attached to a passenger on a moving object and detects the bodily behavior of the passenger;
a passenger position detection unit that detects the position of the passenger in the moving body as a passenger position;
a plurality of moving body behavior detection units that are installed at a plurality of installation positions of the mobile body and detect behavior of the mobile body at the installation positions;
Integrating the behavior of the mobile body at the plurality of installation positions based on the positional relationship between the passenger position and the plurality of installation positions in the mobile body, and the behavior of the mobile body at the plurality of installation positions. a moving object behavior estimation unit that calculates the behavior of the moving object at the passenger position by calculation ;
Behavior comprising: an occupant behavior estimating unit that estimates the body behavior of the occupant based on the body behavior detected by the occupant behavior detecting unit and the behavior of the moving body at the occupant position. Estimation system. The behavior estimation system according to claim 1,
a display section that displays an image;
A behavior estimation system further comprising: an image display control unit that controls a display form of the image displayed by the display unit based on the body behavior of the occupant estimated by the occupant behavior estimation unit. The behavior estimation system according to claim 1 or 2,
The passenger behavior detection unit is a behavior estimation system that is attached to the head of the passenger and detects the behavior of the head. The behavior estimation system according to claim 3,
The passenger behavior detection unit includes a head gyro sensor that measures the angular velocity of the head as a head angular velocity,
The moving object behavior detection unit includes a moving object gyro sensor that measures the angular velocity of the moving object as a moving object angular velocity,
The passenger behavior estimation unit is a behavior estimation system that estimates the behavior of the head by the passenger based on the head angular velocity and the moving object angular velocity. The behavior estimation system according to claim 3 or 4,
The passenger behavior detection unit includes a head acceleration sensor that measures the acceleration of the head as head acceleration,
The moving object behavior detection unit includes a moving object acceleration sensor that measures the acceleration of the moving object as a moving object acceleration,
The passenger behavior estimation unit is a behavior estimation system that estimates the behavior of the head by the passenger based on the head acceleration and the moving body acceleration. The behavior estimation system according to any one of claims 1 to 5,
The passenger behavior detection unit is included in a wearable device worn by the passenger,
The mobile body behavior detection unit is provided in a mobile device mounted on the mobile body,
The passenger behavior estimation unit is a behavior estimation system included in the mobile device. The behavior estimation system according to any one of claims 1 to 5,
The passenger behavior detection unit is included in a wearable device worn by the passenger,
The mobile body behavior detection unit is provided in a mobile device mounted on the mobile body,
The passenger behavior estimation unit is a behavior estimation system included in the wearable device. a passenger behavior detection unit that is attached to a passenger on a moving object and detects the bodily behavior of the passenger;
a passenger position detection unit that detects the position of the passenger in the moving body as a passenger position;
a plurality of moving body behavior detection units that are installed at a plurality of installation positions of the mobile body and detect behavior of the mobile body at the installation positions;
Movement of estimating the behavior of the mobile body at the passenger position based on the positional relationship between the passenger position and the plurality of installation positions in the mobile body, and the behavior of the mobile body at the plurality of installation positions. a body behavior estimation section;
an occupant behavior estimation unit that estimates the body behavior of the occupant based on the body behavior detected by the occupant behavior detection unit and the behavior of the moving body at the occupant position;
The passenger behavior detection unit is a behavior estimation system that is attached to the arm of the passenger and detects the behavior of the arm. Detects the physical behavior of passengers boarding a moving object,
Detecting the position of the passenger in the moving body as a passenger position,
Detecting the behavior of the moving body at a plurality of installation positions of the moving body,
Integrating the behavior of the mobile body at the plurality of installation positions based on the positional relationship between the passenger position and the plurality of installation positions in the mobile body, and the behavior of the mobile body at the plurality of installation positions. and calculating the behavior of the moving body at the position of the passenger,
A behavior estimation method for estimating the behavior of the body by the passenger based on the detected behavior of the body and the behavior of the moving object at the position of the passenger.

Images