JP2023066255A - Onboard display control system and onboard display control method - Google Patents

Abstract

To accurately select an object displayed on an onboard display.SOLUTION: Provided is an onboard display control system for controlling an onboard display which is installed in a vehicle interior. The onboard display control system comprises: one or a plurality of sensors for acquiring the state of an occupant in the vehicle interior; a line of sight direction identification unit for identifying a line of sight direction of the occupant on the basis of the output of the sensors; a finger identification unit for identifying a finger of the occupant on the basis of the output of the sensors; and a selection target specification unit for specifying, on the basis of a visible range of the occupant which is assumed from an identification result of the line of sight direction, a designated position on the display which is set from an identification result of the finger, and a positional relation with an object which is displayed on the display, an object as the target to be selected that exists within the visible range and is located at a position that corresponds to the designated position.SELECTED DRAWING: Figure 2

Description

The present invention relates to an in-vehicle display control system and an in-vehicle display control method.

Conventionally, a pointing device such as a mouse or a touch panel has been used to select an object displayed on a display. Also, a technique for performing an operation of selecting a desired target without contact is being considered.

For example, Patent Literature 1 describes: ``Providing a method for teaching the spatial position of an object by a pointing action that can be performed naturally and easily''; In the teaching method for surface-based determination, a finger of a human hand is extracted, indices are set on at least two points on the finger, two spatial coordinates of the finger are calculated, and the spatial coordinates are calculated by the stereo method. a three-dimensional straight line is applied to the spatial coordinates of the two points; will be taught as the position of."
In addition, Patent Document 2 describes, "It is an object of the present invention to provide a technique for accurately specifying a device that a driver desires to operate from among the in-vehicle devices and assisting the operation of the device. a line-of-sight direction acquisition unit that acquires the line-of-sight direction of the driver; a characteristic behavior acquisition unit that acquires characteristic behavior that is a characteristic behavior other than the line of sight of the driver; A device identification unit that identifies at least one in-vehicle device as an operation target device that the driver desires to operate, and a display control unit that displays the operation screen of the operation target device on a display device mounted in the vehicle. and,” is described.
U.S. Pat. No. 6,200,403 also presents "A User Interface Apparatus, Computer Program, Computer Readable Medium, and Method for Selecting Selectable Objects on a Display Screen. The Display Screen Displays One or More Selectable Objects." obtaining information about the tracked hand position and head or eye position of the user, determining whether the selectable object is at a first screen position based on the tracked hand position and head or eye position; where the first screen position is a position on the display screen such that the first hand at least partially blocks the user's view of the selectable object. determines that a selectable object has been selected."

JP 2005-069734 A Patent No. 6851482 JP 2017-539035 A

However, with the conventional technology, it is difficult for the passenger to select an object displayed on an in-vehicle display without malfunction. For example, since vehicle occupants do not always check the display or perform operations, selecting an object at an unnecessary timing will cause a malfunction. In addition, there are cases in which it is not possible to create a state in which an object exists on the line of sight and hand of the occupant because the degree of freedom in the seating position is low and it is obstructed by the steering wheel and the like.

SUMMARY OF THE INVENTION Accordingly, an object of the present invention is to provide an in-vehicle display control system and an in-vehicle display control method capable of accurately selecting an object displayed on an in-vehicle display.

In order to achieve the above object, one typical in-vehicle display control system of the present invention is an in-vehicle display control system for controlling an in-vehicle display installed in a vehicle interior, wherein the state of the occupant in the vehicle interior is , a line-of-sight direction identification unit that identifies the line-of-sight direction of the occupant based on the output of the sensor, and a finger identification unit that identifies the hand and fingers of the occupant based on the output of the sensor. and based on the positional relationship between the visible range of the occupant assumed from the identification result of the line-of-sight direction, the indicated position on the display set from the identification result of the finger, and the object displayed on the display, A selection target identifying unit that identifies an object located within the visible range and at a position corresponding to the indicated position as a selection target.

According to the present invention, an object displayed on an in-vehicle display can be accurately selected. Problems, configurations, and effects other than those described above will be clarified by the following description of the embodiments.

Explanatory drawing explaining the outline|summary of the vehicle-mounted display control system of Example 1 Schematic diagram of an in-vehicle display control system according to the first embodiment Flowchart showing the processing procedure of the in-vehicle display control system Explanatory diagram of indication position setting Selectable specific illustration Specific example of object selection (Part 1) Specific example of object selection (part 2) Specific example of object selection (Part 3) Examples of object manipulation First modification Second modification

An embodiment will be described below with reference to the drawings.

FIG. 1 is an explanatory diagram for explaining an outline of an in-vehicle display control system according to a first embodiment. A display 30 that is a display device is attached to the dashboard of the vehicle 10 . As an example, the display 30 has a display area extending from the driver's seat side to the front passenger's seat side of the dashboard. The in-vehicle display control system installed in the vehicle 10 acquires the state of the passenger U1 who is the driver using a sensor, identifies the position of the head of the passenger U1, the direction of the line of sight, and the fingers, and controls the display on the display 30. I do.

Specifically, the in-vehicle display control system processes an image captured by an exterior camera based on the position of the head of the occupant U1, and displays an image of the exterior of the vehicle, which is estimated to be visible from the viewpoint of the occupant U1, on the display 30. By making the display 30 virtually transparent, a field of view V2 is provided. As a result, the field of view V1 directly visible to the passenger U1 and the virtual field of view V2 are continuous, and the field of view of the passenger U1 can be expanded.

FIG. 1 shows a case where there is an obstacle D1 in front of the vehicle 10 . The obstacle D1 is shielded by the hood of the vehicle 10 and is not included in the directly visible field of view V1. However, by using the display on the display 30 as the virtual field of view V2, the occupant U1 can visually recognize the obstacle D1.

The image of the obstacle D1 displayed on the display 30 can be selected as an object to be operated. However, it is difficult to operate the display 30 by directly touching it due to the positional relationship with the seating position and the existence of the handle.

Therefore, the disclosed in-vehicle display control system provides a visible range of the occupant U1 assumed from the line-of-sight direction of the occupant U1, an indicated position on the display set based on the finger identification result, and a position of the object displayed on the display. Objects are selected by non-contact remote control based on the relationship.

Here, fingers are used in the sense of including at least one of hands and fingers. The pointed position on the display that is set based on the identification result of the finger is, for example, an object that exists ahead of the occupant's finger.
Further, in selecting an object, specifically, an object located within the visible range and at a position corresponding to the designated position may be specified as a selection target.

FIG. 2 is a configuration diagram of an in-vehicle display control system according to the first embodiment. As shown in FIG. 2 , the in-vehicle display control system has an exterior camera 11 , an interior camera 12 , a processing section 20 and a display 30 .

The vehicle exterior camera 11 is an imaging device that captures an image of the surroundings of the vehicle.
The in-vehicle camera 12 is an imaging device that captures an image of the interior of the vehicle, and functions as a sensor that acquires the state of the occupant in the vehicle.

The processing unit 20 has an occupant head identification unit 21 , an occupant hand/finger identification unit 22 , a virtual transparent display processing unit 23 , a selection target identification unit 24 and a target operation execution unit 25 . When the processing unit 20 is realized by a computer, a CPU (Central Processing Unit) executes a program to identify an occupant head identification unit 21, an occupant hand identification unit 22, a virtual transparent display processing unit 23, and a selection target identification unit. 24 and the target operation execution unit 25 are implemented.

The occupant's head identification unit 21 identifies the head position and line-of-sight direction of the occupant U<b>1 using an image captured by the in-vehicle camera 12 . That is, the occupant's head identifying section 21 includes a function as a line-of-sight direction identifying section.

The occupant hand identification unit 22 is a hand identification unit that identifies the hand and fingers of the occupant U1 using an image captured by the in-vehicle camera 12 . In finger identification, the fingertips and joints of each finger can be identified.

The virtual transparent display processing unit 23 processes the imaging result of the vehicle exterior camera 11 based on the position of the head output by the occupant head identification unit 21, and displays an image of the exterior of the vehicle that is estimated to be visible from the viewpoint of the occupant U1. By displaying on the display 30, the occupant U1 is provided with a field of view virtually transmitted through the display 30. - 特許庁

The selection target identification unit 24 assumes the visual range of the occupant from the line-of-sight direction identification result of the occupant head identification unit 21 , and sets the pointing position on the display from the finger identification result of the occupant hand identification unit 22 . Further, the display contents on the display 30 are obtained from the virtual transparent display processing unit 23 . Then, the selection target identifying unit 24 identifies an object located within the visible range and at a position corresponding to the indicated position as a selection target.

When setting the pointing position, the selection target specifying unit 24 sets the intersection of a straight line connecting two preset points included in the finger and the display surface of the display 30 as the pointing position.
When specifying the selection target, the selection target specifying unit 24 puts the object located outside the visible range into the selection unprepared state, the object located inside the visible range into the selection ready state, and instructs the object in the selection ready state. When the positions are superimposed, the object is selected.

The target operation execution unit 25 is an operation execution unit that identifies a gesture from the finger identification result of the occupant hand identification unit 22 and executes an operation associated with the identified gesture on a selection target.

Gestures include arbitrary actions by fingers, such as an operation of bringing two specific fingertips closer together, an operation of moving two specific fingertips away, extending and moving a specific number of fingertips, opening a hand, closing a hand, and the like. be able to.

Any operation can be used as the operation on the selection object, such as enlargement or reduction of the frame surrounding the selection object, enlargement or reduction of the image of the selection object, display of the image recognition result regarding the selection object, or the like.

FIG. 3 is a flow chart showing the processing procedure of the in-vehicle display control system. The processing unit 20 of the in-vehicle display control system repeatedly executes the processing procedure shown in FIG.

The processing unit 20 first acquires images from the vehicle exterior camera 11 and the vehicle interior camera 12 (step S101). After that, the processing unit 20 identifies the head position and line-of-sight method by the occupant head identification unit 21 and identifies the fingers by the occupant hand identification unit 22 for the image acquired from the in-vehicle camera 12 (step S202).

After step S202, the virtual transparent display processing unit 23 generates a virtual transparent display screen from the position of the occupant's head and the image of the camera outside the vehicle 11 (step S103), and causes the display 30 to display the virtual transparent display. Processing is performed (step S104).

After step S104, the selection target identification unit 24 identifies a selectable object from the virtual transparent display screen, and extracts a rectangular area surrounding the object as a target area (step S105).

After step S105, the selection target specifying unit 24 sets the visible range from the identification result of the line-of-sight direction of the passenger (step S106). Further, the selection target specifying unit 24 sets the indicated position from the identification result of the occupant's fingers (step S107).

After step 107, the selection target specifying unit 24 specifies an object located within the visible range and at a position corresponding to the indicated position as a selection target (step S108). The target operation execution unit 25 identifies a gesture from the finger identification result of the occupant hand identification unit 22, and executes an operation associated with the identified gesture on a selection target (step S109).

FIG. 4 is an explanatory diagram of setting the indicated position. The selection target specifying unit 24 sets the point of intersection between the straight line connecting two preset points included in the finger and the display surface of the display 30 as the pointing position P. FIG. In FIG. 4, the tip of the index finger and the third joint are identified by image processing and used as two points. An arbitrary point such as an arbitrary joint of the finger joints, the palm, or the center of mass of the hand can be identified and used to set the indicated position.

FIG. 5 is an explanatory diagram of specifying selection targets. The selection target specifying unit 24 extracts a rectangle surrounding the object as a target region. Then, if the target area is located outside the visible range, the target area is put into a selection unprepared state.

When the target region in the selection unprepared state is inside the visible range, the selection target specifying unit 24 puts the target region in the selection ready state. When the target region in the selection ready state is outside the visible range, the selection target specifying unit 24 puts the target region in the selection unprepared state.

When the pointing position enters the target area in the selection ready state, the selection target specifying unit 24 puts the target area in the selected state. If a deselection gesture is detected when there is a target area in the selected state, the selection target specifying unit 24 puts the target area in the selection ready state. As a deselection gesture, for example, clasping a hand (bending all fingers) or opening a hand (stretching all fingers) may be used.

If a gesture associated with an operation is detected when there is a target area in a selected state, the target operation execution unit 25 executes the corresponding operation. Then, after the operation is completed, it returns to the selected state.

6 to 8 are specific examples of object selection. In FIG. 6, the display 30 provided on the dashboard displays an image of the outside according to the viewpoint from the passenger U1. For example, an image of an obstacle D1 that exists on the other side of the hood is displayed on the display 30 as an object T1. Therefore, the occupant U1 can recognize the obstacle D1 as if it were seen through the hood.

FIG. 7 shows the visible range A1 of the occupant U1 by a hatched rectangle with a broken line. If the object T1 is outside the visible range A1, the object T1 is in a selection unprepared state. If the object T1 is inside the visible range A1, the object T1 is ready for selection.

FIG. 8 shows the relationship between the object in the selection ready state and the pointed position. If the pointing position is outside the rectangular area (target area) containing the object in the selection ready state, the object T1 is in the selection ready state. If the pointing position is inside the rectangular area (target area) including the object in the selection ready state, the object T1 is in the selected state.

FIG. 9 shows a specific example of object manipulation. FIG. 9 shows a case where an action of speaking the fingertips of the index finger and thumb (pinch-out action) is identified as a gesture. If a pinch-out motion in the vertical direction is detected, the target operation execution unit 25 vertically enlarges the rectangular area including the object. If the horizontal pinch-out motion is detected, the target operation execution unit 25 horizontally enlarges the rectangular area including the object. When an oblique pinch-out motion is detected, the target operation execution unit 25 expands the rectangular area including the object while maintaining the aspect ratio.

Although FIG. 9 shows a case where a rectangular area including an object is resized, any operation such as enlarging the object image itself by pinching out can be set.

In the description so far, an example of applying the present invention to a configuration in which a virtual field of view is provided by the display 30 arranged on the dashboard has been shown. Further, the process of setting a part of the display area of the display 30 as the visible range is exemplified. However, the present invention is not limited to this.

FIG. 10 is an explanatory diagram of a first modified example. In the configuration of FIG. 10, the display 40 provides information regarding the travel route of the vehicle. For example, map information. Also, the display area of the display 40 is smaller than the display 30 and smaller than the visible range. In this modified example, the selection target specifying unit puts the object displayed on the display 40 into the selection ready state on condition that the entire display area of the display 40 is included in the visible range A2. Then, when the specified position P2 is superimposed on the object, the object is selected and an operation by a gesture is accepted. As an example, it is possible to display an enlarged map by pinching out.

FIG. 11 is an explanatory diagram of a second modification. In the configuration of FIG. 11, the display 50 performs display output by projection onto the windshield. In this modified example, an object that the passenger is directly looking at can be recognized as an object and used as an operation target. In FIG. 11, the visible range A3 includes a sign as an object T2, and the designated position P3 is superimposed on the object T2. Therefore, the selection target specifying unit accepts the operation with the object T2 in the selected state. For example, it is possible to read aloud signs.

As described above, the disclosed system is an in-vehicle display control system that controls an in-vehicle display installed in a vehicle interior, and includes an in-vehicle camera as one or more sensors that acquire the state of an occupant in the vehicle interior. 12, an occupant head identification unit 21 as a line-of-sight direction identification unit that identifies the line-of-sight direction of the occupant based on the output of the sensor, and a finger identification unit that identifies the hand and fingers of the occupant based on the output of the sensor. The occupant's finger identification unit 22 as a unit, the visible range of the occupant assumed from the identification result of the line-of-sight direction, the indicated position on the display set from the identification result of the hand and fingers, and the displayed on the display a selection target specifying unit 24 for specifying an object located within the visible range and at a position corresponding to the indicated position as a selection target based on the positional relationship with the object.
With such a configuration and operation, an object displayed on the in-vehicle display can be accurately selected. Also, such selection can be done contactlessly and remotely.

Further, the disclosed system further includes a target operation execution unit 25 as an operation execution unit that identifies a gesture from the finger identification result and executes an operation associated with the identified gesture on the selection target.
Therefore, the occupant can perform an operation on the selected target.

Further, the sensor is an in-vehicle camera that captures an image of the interior of the vehicle, and the line-of-sight direction identification unit and the hand/finger identification unit identify the line-of-sight direction and the finger from the image of the occupant.
Therefore, it is possible to acquire the state of the occupant with a simple configuration, and the in-vehicle camera can be shared with other systems installed in the vehicle.

Further, the selection target specifying unit sets an intersection of a straight line connecting two preset points included in the finger and the display surface of the display as the indicated position.
Therefore, the indicated position can be easily set.

Further, the selection target specifying unit puts the object located outside the visible range into a selection-unprepared state, the object located inside the visible range into a selection-ready state, and puts the object in the selection-ready state into the selection-prepared state. When the designated position is superimposed, the object is selected.
In this manner, by referring to a plurality of elements and making a selection step by step, it is possible to prevent malfunction.

Further, according to the disclosed system, an image captured by the exterior camera 11 that captures images of the surroundings of the vehicle and the image captured by the exterior camera are processed based on the position of the occupant's head specified from the output of the sensor, a virtual transmissive display processing unit 23 that provides a field of view that is virtually transmitted through the display by displaying an image outside the vehicle that is estimated to be visible from the passenger's viewpoint on the display, and the visible range is , is part of the display area of the display.
According to such a configuration, a part of the vehicle body can be virtually transparently displayed using the display, and the displayed object can be remotely selected.

Further, according to the disclosed system, the display provides information related to the travel route of the vehicle, and the selection target specifying unit provides the selection target identification unit, on the condition that the entire display area of the display is included in the visible range, the To select an object displayed on a display.
According to such a configuration, a display smaller than the visible range, such as a screen of a navigation system, can be selected.

It should be noted that the present invention is not limited to the above embodiments, and includes various modifications. For example, the above-described embodiments have been described in detail in order to explain the present invention in an easy-to-understand manner, and are not necessarily limited to those having all the described configurations. Moreover, not only deletion of such a configuration but also replacement and addition of the configuration are possible.

For example, when there are passengers other than the driver, the present invention may be applied to the other passengers. Further, a specific passenger may be designated as an application target of the present invention among a plurality of passengers.

Also, the state of the occupant may be acquired from any sensor other than the in-vehicle camera, such as LIDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging), a stationary smartphone, or the like. Furthermore, it is also possible to use a combination of outputs from a plurality of sensors and a plurality of types of sensors.

11: vehicle exterior camera, 12: vehicle interior camera, 20: processing unit, 21: occupant head identification unit, 22: occupant hand identification unit, 23: virtual transparent display processing unit, 24: selection target identification unit, 25: target operation execution Part 30: Display

Claims (8)

An in-vehicle display control system for controlling an in-vehicle display installed in a vehicle interior,
one or more sensors for acquiring the state of an occupant in the vehicle;
a line-of-sight direction identification unit that identifies the line-of-sight direction of the occupant based on the output of the sensor;
a finger identification unit that identifies the occupant's fingers based on the output of the sensor;
Based on the positional relationship between the visible range of the occupant assumed from the identification result of the line-of-sight direction, the indicated position on the display set from the identification result of the finger, and the object displayed on the display A vehicle-mounted display control system, comprising: a selection target identifying unit that identifies an object located within a range and at a position corresponding to the indicated position as a selection target. 2. The in-vehicle display control according to claim 1, further comprising an operation execution unit that identifies a gesture from the finger identification result and executes an operation associated with the identified gesture on the selection target. system. The sensor is an in-vehicle camera that captures an image of the interior of the vehicle,
2. The in-vehicle display control system according to claim 1, wherein the line-of-sight direction identifying section and the hand/finger identifying section identify the line-of-sight direction and the hand/finger from the image of the occupant. 2. The in-vehicle vehicle according to claim 1, wherein the selection target specifying unit sets an intersection of a straight line connecting two preset points included in the finger and the display surface of the display as the indicated position. display control system. The selection target specifying unit places the object outside the visible range in a selection-unprepared state, places the object inside the visible range in a selection-ready state, and places the object in the selection-ready state in the designated position. 2. The in-vehicle display control system according to claim 1, wherein the object is placed in a selected state when the is superimposed on the object. an exterior camera that captures images of the surroundings of the vehicle;
Based on the position of the occupant's head specified from the output of the sensor, the image captured by the exterior camera is processed, and an image of the exterior of the vehicle that is estimated to be visible from the occupant's viewpoint is displayed on the display. a virtual transparent display processing unit that provides a field of view that is virtually transparent through the display;
further comprising
2. The in-vehicle display control system according to claim 1, wherein the visible range is a portion of the display area of the display. the display provides information about the travel route of the vehicle;
2. The in-vehicle display according to claim 1, wherein the selection target specifying unit enables selection of an object displayed on the display on condition that the entire display area of the display is included in the visible range. control system. An in-vehicle display control method for controlling an in-vehicle display installed in a vehicle interior,
obtaining occupant status in the vehicle interior from one or more sensors;
a line-of-sight direction identification step of identifying a line-of-sight direction of the occupant based on the output of the sensor;
a finger identification step for identifying the fingers of the occupant based on the output of the sensor;
Based on the positional relationship between the visible range of the occupant assumed from the identification result of the line-of-sight direction, the indicated position on the display set from the identification result of the finger, and the object displayed on the display A vehicle-mounted display control method, comprising: a selection target identification step of identifying an object located within a range and at a position corresponding to the indicated position as a selection target.

Images