JP2024018746A - Data processing device and data processing method - Google Patents

Abstract

An object of the present invention is to provide suitable feedback to vehicle occupants. [Solution] A data processing device includes a surrounding situation acquisition unit that acquires the surrounding situation of the own vehicle, and a hand recognition unit that recognizes a hand finger of a passenger riding in the own vehicle pointing at the outside of the own vehicle. , an object recognition unit that recognizes an object pointed by the hand based on an acquisition result by the surrounding situation acquisition unit and a recognition result by the finger recognition unit; and an object pointed to by the hand recognized by the object recognition unit. The invention is characterized by comprising an irradiation control section that controls the mode of light irradiated to the fingers according to the above. [Selection diagram] Figure 2

Description

The present invention relates to a data processing device and a data processing method.

Conventionally, a pointing device such as a mouse or a touch panel has been used to select an object displayed on a display. Further, a technology is being considered in which the operation of selecting a desired target is performed without contact.

For example, Patent Document 1 states, ``A pointing device is provided that can perform a pointing operation while confirming the pointing position by giving visual feedback.'' ``The hand gesture recognition device 12 recognizes the user's hand gestures and The three-dimensional position detecting unit 8 detects the three-dimensional position of the spot light projected by the laser emitting unit 2. The three-dimensional model generating unit 10 outputs position and orientation information. A three-dimensional model of a scene including light is generated.The support stand control unit 14 detects the position of the pointing target based on the three-dimensional model and the hand gesture position and orientation information, and determines the attitude of the support stand 16 including the laser emitting unit 2. control.''
Further, Patent Document 2 describes, "A user interface device, a computer program, a computer readable medium, and a method for selecting selectable objects on a display screen are presented. The display screen displays one or more selectable objects." Obtain information regarding the tracked hand position and head or eye position of the user. Based on the tracked hand position and head or eye position, determine whether the selectable object is at the first screen position. Here, the first screen position is a position on the display screen where the first hand at least partially blocks the user's view of the selectable object. It is determined that the selectable target has been selected.''

Japanese Patent Application Publication No. 2000-259337 JP 2017-539035 Publication

According to the conventional technology, a device can recognize a position on a display specified by a user by pointing or the like, and can feed back the recognition result by changing the display mode on the display.
However, in the conventional technology, when a user who is a vehicle occupant points outside the vehicle, even if the device recognizes the object pointed to, it is difficult to feed back the recognition result.

Therefore, an object of the present invention is to provide a data processing device and a data processing method that provide suitable feedback to a user who is a vehicle occupant.

In order to achieve the above object, one of the representative data processing devices of the present invention includes a surrounding situation acquisition unit that obtains the surrounding situation of the own vehicle, and a a hand recognition unit that recognizes a hand pointing to the outside of the vehicle; an object recognition unit that recognizes a target pointed by the hand based on an acquisition result by the surrounding situation acquisition unit and a recognition result by the hand recognition unit; The apparatus is characterized by comprising an irradiation control section that controls a mode of light irradiated to the finger according to the object pointed to by the finger recognized by the object recognition section.

According to the present invention, suitable feedback to vehicle occupants can be realized. Problems, configurations, and effects other than those described above will be made clear by the following description of the embodiments.

Explanatory diagram illustrating an overview of data processing in Example 1 Configuration diagram of data processing device of Example 1 Flowchart showing processing of data processing device Flowchart showing detailed processing of object recognition step of Example 1 Explanatory diagram illustrating an overview of data processing in Example 2 Configuration diagram of data processing device of Example 2 Flowchart showing detailed processing of object recognition step of Example 2

Examples will be described below with reference to the drawings.

FIG. 1 is an explanatory diagram illustrating an overview of data processing according to the first embodiment. FIG. 1 shows a state in which a user riding in a vehicle points outside the vehicle. A data processing device 10 mounted on a vehicle communicates information between an in-vehicle camera that images the inside of the vehicle and an outside camera that images the surroundings of the own vehicle.

The data processing device 10 acquires an image from an in-vehicle camera, analyzes the image, identifies the position of the user's eyes and fingers, and determines that a straight line connecting the eye position and the fingertip is the pointing direction. Here, the term "hand" is used to include at least one of a hand and a finger.
The data processing device 10 also acquires an image from a camera outside the vehicle, analyzes the image, and detects a parking area where the own vehicle can park.

The data processing device 10 recognizes the parking area as a target when the parking area is located in the pointing direction, that is, when the position pointed by the finger is included in the parking area. Furthermore, when a predetermined period of time has elapsed without the pointing direction deviating from the parking area, the data processing device 10 recognizes that a predetermined confirmation operation has been received for the target, and supports parking in the parking area.

Here, the data processing device 10 communicates with an irradiation unit that irradiates light onto the user's hand or finger, and can control the form of the irradiated light depending on the target pointed by the user. For example, in state A1 of FIG. 1, the user's hand H1 does not point to the parking area P1. At this time, the irradiation unit irradiates the user's hand H1 with white light.
After that, as shown in state A2, when the user's hand H1 points to the parking area P1, the irradiation unit irradiates the user's hand H1 with red light. Then, when a predetermined period of time has elapsed with the user's hand H1 pointing to the parking area P1, the irradiation unit irradiates the user's hand H1 with yellow light.

In this way, the data processing device 10 can feed back the recognition result of the object pointed at by the user by changing the form of light irradiated to the user's hand. Therefore, feedback with good visibility can be achieved even when the user points outside the vehicle. Note that, for example, a projector may be used as the irradiation unit. Moreover, as the mode of the irradiated light, in addition to color, blinking can also be used. Furthermore, characters and symbols may be projected onto the user's hand. Furthermore, "a state in which no light is irradiated" may be included as one of the plurality of embodiments.

FIG. 2 is a configuration diagram of the data processing device 10 of the first embodiment. The data processing device 10 includes a processing section 20, acquires information from an outside camera 11 and an inside camera 12, and uses an irradiation section 13 to irradiate a user's hand with light. The vehicle exterior camera 11 is an imaging device that photographs the surroundings of the vehicle. The in-vehicle camera 12 is an imaging device that captures an image of the interior of the vehicle, and functions as a sensor that acquires the status of the occupants in the vehicle interior. The irradiation unit 13 is a projector that irradiates light onto the hands and fingers of the occupant.

The in-vehicle camera 12 is installed at a position where it can image the eyes and hands of the occupant. For example, it may be installed near the room mirror or the room light. The irradiation unit 13 is installed at a position where it can irradiate light onto the occupant's hand so that the occupant can visually recognize it. For example, it may be installed near a room light or a headrest.

The processing section 20 includes a surrounding situation acquisition section 21 , an occupant recognition section 22 , an irradiation control section 23 , an object recognition section 24 , and a parking assistance execution section 25 . When the processing unit 20 is implemented by a computer, a CPU (Central Processing Unit) executes a program to perform the surrounding situation acquisition unit 21, occupant recognition unit 22, irradiation control unit 23, object recognition unit 24, and parking assistance execution. What is necessary is to realize the function corresponding to the section 25.

The surrounding situation acquisition unit 21 obtains the surrounding situation of the own vehicle. Specifically, the surrounding situation acquisition unit 21 acquires an image taken by the external camera 11 that images the surroundings of the own vehicle, and outputs it to the object recognition unit 24.

The occupant recognition unit 22 operates as a hand recognition unit that acquires an image captured by the in-vehicle camera 12 and recognizes a finger pointing toward the outside of the own vehicle by a user riding in the own vehicle. The occupant recognition unit 22 also recognizes the user's eyes. The occupant recognition unit 22 outputs the user's eye positions and finger positions to the object recognition unit 24 .

The irradiation control unit 23 controls the irradiation unit 13 to irradiate the finger recognized by the occupant recognition unit 22 with light. Specifically, the irradiation control unit 23 controls irradiation according to the output from the object recognition unit 24. The output from the object recognition unit 24 includes recognition results and finger positions. The irradiation control unit 23 controls the irradiation direction of the irradiation unit 13 according to the position of the finger, and controls the mode of the irradiated light according to the recognition result. The recognition results include "no target", "parking area recognized as target", and "confirmation operation accepted", and the irradiation control unit 23 changes the display mode of the light so that these recognition results can be identified. let

The object recognition unit 24 recognizes the object pointed to by the finger based on the acquisition result by the surrounding situation acquisition unit 21 and the recognition result by the occupant recognition unit 22.

Specifically, the object recognition unit 24 detects a parking area where the own vehicle can be parked from the image acquired by the surrounding situation acquisition unit 21. Any image processing technique may be used to detect the parking area. Further, the object recognition unit 24 determines that a straight line connecting the user's eye position and the fingertip is the pointing direction.
If there is no parking area in the pointing direction, the object recognition unit 24 determines the recognition result as "no object". When a parking area is located in the pointing direction, that is, when the position pointed by the finger is included in the parking area, the parking area is recognized as a target. The recognition result at this time is "the parking area is recognized as the target". Furthermore, when a predetermined period of time has elapsed without the pointing direction deviating from the parking area, the object recognition unit 24 recognizes that a predetermined confirmation operation has been received for the object. The recognition result at this time is "confirmation operation accepted".

The object recognition unit 24 outputs the recognition results and the positions of the fingers to the irradiation control unit 23 at any time. Further, when the recognition result is "confirmation operation accepted", data indicating the position of the parking area is output to the parking assistance execution unit 25.

The parking assistance execution unit 25 supports parking in the parking area when the object recognition unit 24 recognizes the parking area as a target and receives a predetermined confirmation operation. Specifically, the parking assistance execution unit 25 outputs an operation instruction to the drive control unit 30 based on the data indicating the position of the parking area output by the object recognition unit 24. This operation instruction is an instruction to control acceleration, deceleration, and steering of the own vehicle, and when the drive control unit 30 operates in response to this operation instruction, the own vehicle automatically travels to the parking area and stops.

FIG. 3 is a flowchart showing the processing of the data processing device 10. This process starts, for example, when an operation requesting automatic parking is received from the user, and ends when the parking area is determined and automatic driving starts.

When the process starts, the surrounding situation acquisition unit 21 executes a surrounding situation acquisition step of acquiring the surrounding situation of the own vehicle (step S101). Thereafter, the occupant recognition unit 22 executes a hand recognition step of recognizing a finger of a user riding in the own vehicle pointing toward the outside of the own vehicle (step S102). Thereafter, the object recognition unit 24 executes an object recognition step of recognizing the object pointed by the finger based on the acquisition result of the surrounding situation acquisition step and the recognition result of the hand recognition step (step S103). After that, the irradiation control unit 23 executes an irradiation step of irradiating the finger with light in a mode corresponding to the object pointed at by the finger (step S104). After that, the process from step S101 is repeated.

FIG. 4 is a flowchart showing detailed processing of the object recognition step of the first embodiment. First, the object recognition unit 24 detects a parking area where the own vehicle can be parked from the image acquired by the surrounding situation acquisition unit 21 (step S201). Further, the object recognition unit 24 determines that the straight line connecting the user's eye position and the fingertip is the pointing direction (step S202).

If there is no parking area in the pointing direction (step S203; No), the object recognition unit 24 sets the recognition result as "no object" (step S204), outputs the recognition result (step S208), and ends the object recognition step. do.

If a parking area is located in the pointing direction (step S203; Yes), the object recognition unit 24 recognizes the parking area as a target (step S205). After step S205, the object recognition unit 24 determines whether the state in which the parking area is the object continues for a predetermined period of time. If it has not continued for a predetermined period of time (step S206; No), the recognition result is output (step S208), and the object recognition step ends. The recognition result at this time is "recognition for the parking area".

If the state targeting the parking area continues for a predetermined period of time (step S206; Yes), the object recognition unit 24 recognizes that a predetermined confirmation operation has been received for the object (step S207), and outputs the recognition result. (Step S208), the object recognition step ends. The recognition result at this time is "confirmation operation accepted".

FIG. 5 is an explanatory diagram illustrating an overview of data processing according to the second embodiment. FIG. 5 shows a state in which a user riding in a vehicle points outside the vehicle. The difference from the first embodiment is that the user points to a facility as a destination, the data processing device 10a uses map data to identify the facility pointed to by the user, and sets the identified facility as the navigation destination. It is a point.

The data processing device 10a recognizes the facilities located in the direction indicated by the finger among the facilities included in the map data. Furthermore, when a predetermined period of time has elapsed without the pointing direction deviating from the facility, the data processing device 10a recognizes that a predetermined confirmation operation has been received for the target, and sets the facility as the destination.

The data processing device 10a includes an irradiation unit that irradiates light onto the user's hand or finger, and can control the form of the irradiated light depending on the object pointed at by the user. For example, in state B1 of FIG. 5, the user's hand H1 is not pointing at any facility. At this time, the irradiation unit irradiates the user's hand H1 with white light.
After that, as shown in state B2, when the user's hand H1 points to the facility T1, the irradiation unit irradiates the user's hand H1 with red light. Then, when a predetermined period of time has elapsed with the user's hand H1 pointing at the facility T1, the irradiation unit irradiates the user's hand H1 with yellow light.

In this way, similarly to the first embodiment, the data processing device 10a can feed back the recognition result regarding the object pointed at by the user by changing the form of light irradiated to the user's hand.

FIG. 6 is a configuration diagram of the data processing device 10a of the second embodiment. The data processing device 10a differs from the first embodiment in that the surrounding situation acquisition section 21a of the processing section 20a acquires map data and own vehicle position data from the navigation unit 40 without using the external camera 11. Another difference is that a destination setting section 26 is provided instead of the parking assistance execution section 25. Furthermore, the operation of the object recognition unit 24a is different. The other configurations and operations are the same as those in the first embodiment, so the same components are given the same reference numerals and the explanation will be omitted.

The surrounding situation acquisition unit 21a acquires map data around the own vehicle and position data of the own vehicle from the navigation unit 40, and outputs them to the object recognition unit 24a.
The object recognition unit 24a generates 3D model data of facilities around the own vehicle using the map data and the position data of the own vehicle. Note that the facility refers to something that can be set as a destination for a vehicle, such as a building, a store, or a park.

The object recognition unit 24a recognizes, as a target, a facility located in the direction indicated by the finger among the facilities included in the 3D model data.
Specifically, when there is no facility in the pointing direction, the object recognition unit 24 sets the recognition result to "no object". If a facility is located in the pointing direction, the facility is recognized as the target. The recognition result at this time is "recognition as a facility". Further, when a predetermined period of time has elapsed without the pointing direction deviating from the facility, the object recognition unit 24 recognizes that a predetermined confirmation operation has been received for the object. The recognition result at this time is "confirmation operation accepted".

The object recognition unit 24a outputs the recognition results and the positions of the fingers to the irradiation control unit 23 at any time. Further, when the recognition result is "confirmation operation accepted", data indicating the target facility is output to the destination setting unit 26.

When the object recognition section 24 recognizes a facility as a target and receives a predetermined confirmation operation, the destination setting section 26 outputs the facility as a destination to the navigation unit 40. The navigation unit 40 searches for a route to the output destination and provides guidance along the searched route.

The basic processing of the data processing device 10a is the same as in the first embodiment. The process of the second embodiment starts, for example, when an operation requesting destination setting is received from the user, and ends when the destination is determined and the process moves to route search.

FIG. 7 is a flowchart showing detailed processing of the object recognition step of the second embodiment. First, the object recognition unit 24a generates 3D model data of facilities around the own vehicle using the map data acquired by the surrounding situation acquisition unit 21 and the position data of the own vehicle (step S301). Further, the object recognition unit 24a determines that the straight line connecting the user's eye position and the fingertip is the pointing direction (step S302).

If there is no facility in the pointing direction (step S303; No), the target recognition unit 24a sets the recognition result as "no target" (step S304), outputs the recognition result (step S308), and ends the target recognition step. .

If a facility is located in the pointing direction (step S303; Yes), the target recognition unit 24a recognizes the facility as a target (step S305). After step S205, the target recognition unit 24a determines whether the state in which the facility is targeted continues for a predetermined period of time. If it has not continued for a predetermined period of time (step S306; No), the recognition result is output (step S308), and the object recognition step ends. The recognition result at this time is "recognize the facility as the target".

If the state targeting the facility continues for a predetermined period of time (step S306; Yes), the target recognition unit 24a recognizes that a predetermined confirmation operation has been received for the target (step S307), and outputs the recognition result ( Step S308), the object recognition step ends. The recognition result at this time is "confirmation operation accepted".

As described above, the disclosed data processing device includes the surrounding situation acquisition unit 21 that acquires the surrounding situation of the own vehicle, and the recognition unit 21 that acquires the surrounding situation of the own vehicle, and the recognition of the finger pointing to the outside of the own vehicle by the passenger riding in the own vehicle. an object recognition unit 24 that recognizes the object pointed by the hand based on the acquisition result by the surrounding situation acquisition unit and the recognition result by the hand recognition unit; The apparatus further includes an irradiation control section 23 that controls the mode of light irradiated to the finger according to the object pointed at by the finger recognized by the object recognition section 24.
Therefore, even if a user, who is a vehicle occupant, points outside the vehicle, the data processing device can recognize the target and feed back the recognition result in the form of light that illuminates the vehicle occupant's finger. Appropriate feedback can be achieved.

Further, according to the disclosed data processing device, the surrounding situation acquisition unit 21 acquires an image from the vehicle exterior camera 11, which is a camera that images the surroundings of the own vehicle, and the object recognition unit 24 extracts the image from the image. If a parking area where the own vehicle can be parked is detected, and the position indicated by the finger is included in the parking area, the parking area is recognized as the target, and the irradiation control unit 23 detects that the target recognition unit 24 The display mode of the light is changed depending on whether the parking area is recognized as the object.
The data processing device further includes a parking assistance execution unit that supports parking in the parking area when the object recognition unit 24 recognizes the parking area as the object and receives a predetermined confirmation operation.
Therefore, the data processing device can recognize the parking area pointed by the user, notify the recognition result, and support parking in the parking area.

Further, according to the disclosed data processing device, the surrounding situation acquisition unit 21 acquires map data around the own vehicle, and the object recognition unit 24 selects the hand and finger information from among the facilities included in the map data. A facility located in the direction indicated by is recognized as the target, and the irradiation control unit 23 changes the display mode of the light depending on whether the target recognition unit 24 recognizes the facility as the target. shall be.
The apparatus further includes a destination setting section 26 that sets the facility as a destination when the object recognition section 24 recognizes the facility as the object and receives a predetermined confirmation operation.
Therefore, the data processing device can recognize the facility pointed to by the user, notify the user of the recognition result, and set the facility as the destination.

Further, according to the disclosed data processing device, when the target pointed by the finger remains the same for a predetermined period of time, the target recognition unit 24 recognizes that a predetermined confirmation operation has been received for the target, and The control unit changes the mode of light irradiated to the finger when receiving the confirming operation.
In this way, the data processing device can accept a target confirmation operation by pointing or the like, and can feed back the acceptance of the confirmation operation to the user.

Note that the present invention is not limited to the above-described embodiments, and includes various modifications. For example, the embodiments described above are described in detail to explain the present invention in an easy-to-understand manner, and the present invention is not necessarily limited to having all the configurations described. Furthermore, it is possible not only to delete such a configuration but also to replace or add a configuration.

For example, when there are other occupants in addition to the driver, the present invention may be applied to the other occupants. Alternatively, a specific occupant may be designated as a target of the present invention among a plurality of occupants.

Further, the status of the occupant may be acquired from any sensor other than the in-vehicle camera, such as LIDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) or a stationary smartphone. Furthermore, outputs from multiple sensors and multiple types of sensors can be used in combination.

Furthermore, in the embodiment described above, the extension line between the eye and the fingertip is determined as the direction in which the user is pointing, but two points on the finger may be recognized and the extension line between these two points may be determined as the direction in which the user is pointing.

10: Data processing device, 11: External camera, 12: In-vehicle camera, 13: Irradiation unit, 20: Processing unit, 21: Surrounding situation acquisition unit, 22: Occupant recognition unit, 24: Object recognition unit, 25: Parking assistance execution part, 26: destination setting part, 30: drive control unit, 40: navigation unit

Claims (7)

a surrounding situation acquisition unit that obtains the surrounding situation of the host vehicle;
a hand recognition unit that recognizes a hand finger of a passenger on board the host vehicle pointing toward the outside of the host vehicle;
an object recognition unit that recognizes a target pointed by the finger based on the acquisition result by the surrounding situation acquisition unit and the recognition result by the finger recognition unit;
an irradiation control unit that controls a mode of light irradiated to the finger according to an object pointed to by the finger recognized by the object recognition unit;
A data processing device comprising: The data processing device according to claim 1,
The surrounding situation acquisition unit obtains an image from a camera that images the surroundings of the own vehicle,
The object recognition unit detects a parking area where the own vehicle can be parked from the image, and if the position pointed by the hand is included in the parking area, recognizes the parking area as the object;
The data processing device is characterized in that the irradiation control unit changes the display mode of the light depending on whether or not the object recognition unit recognizes the parking area as the object. The data processing device according to claim 2,
The data processing device further includes a parking assistance execution unit that supports parking in the parking area when the object recognition unit recognizes the parking area as the object and receives a predetermined confirmation operation. . The data processing device according to claim 1,
The surrounding situation acquisition unit acquires map data around the own vehicle,
The target recognition unit recognizes, as the target, a facility located in the direction indicated by the finger among the facilities included in the map data,
The data processing device is characterized in that the irradiation control unit changes the display mode of the light depending on whether or not the object recognition unit recognizes the facility as the object. The data processing device according to claim 4,
The data processing device further comprises a destination setting unit that sets the facility as a destination when the target recognition unit recognizes the facility as the target and receives a predetermined confirmation operation. The data processing device according to claim 1,
The target recognition unit recognizes that a predetermined confirmation operation has been received for the target when the target pointed by the hand remains the same for a predetermined time,
The data processing device is characterized in that the irradiation control unit changes the mode of light irradiated to the hand and fingers when the confirmation operation is received. The data processing device
a surrounding situation acquisition step of obtaining the surrounding situation of the own vehicle;
a hand finger recognition step of recognizing a hand finger of a passenger on board the host vehicle pointing at an outside of the host vehicle;
an object recognition step of recognizing an object pointed by the hand based on the acquisition result of the surrounding situation acquisition step and the recognition result of the hand recognition step;
an irradiation step of irradiating the finger with light in a mode depending on the object pointed at by the finger;
A data processing method characterized by comprising:

Images