JP2021135841A - Operating device - Google Patents

Abstract

To provide an operating device that can prevent a user from repeating a meaningless gesture.SOLUTION: An operating device 100 comprises: a detection unit 41a that detects a hand of a user based on information from a non-contact type sensor 10; a gesture specification unit 41b that specifies a gesture made by the hand detected by the detection unit 41a; an operation control unit 41c that specifies an operation corresponding to the gesture specified by the gesture specification unit 41b and outputs information indicating the details of the specified operation to a target of operation; and a projector 20. When the detection unit 41a detects the hand, the projector 20 projects a first image on the hand, and when the gesture specification unit 41b specifies the gesture, the projector projects a second image different from the first image on the hand.SELECTED DRAWING: Figure 1

Description

The present invention relates to an operating device.

As a conventional operation device, for example, Patent Document 1 describes a device that determines an operation by a user's gesture based on information from a non-contact sensor such as a camera. In the operating device described in Patent Document 1, the user can operate a predetermined device by performing a gesture in a preset detection area.

Japanese Patent No. 6559359

In the technique of determining the gesture operation in the preset detection area, the user may not be able to recognize that the part such as the hand performing the gesture is not in the detection area. In this case, the user may repeat a meaningless gesture that is not recognized by the device.

The present invention has been made in view of the above circumstances, and an object of the present invention is to provide an operating device capable of suppressing a user from repeating meaningless gestures.

In order to achieve the above object, the operating device according to the present invention is
A detector that detects the user's part in a preset detection area in the vehicle based on information from the non-contact sensor.
A gesture identification unit that identifies a gesture by the site detected by the detection unit, and a gesture identification unit.
An operation control unit that identifies an operation corresponding to the gesture specified by the gesture specifying unit and outputs information indicating the content of the specified operation to the operation target.
A projection unit capable of projecting an image onto an object existing in the vehicle is provided.
The projection unit
When the detection unit detects the site, a first image is projected onto the site.
When the gesture specifying portion identifies the gesture, a second image different from the first image is projected onto the portion.

According to the present invention, it is possible to prevent the user from repeating meaningless gestures.

It is a block diagram of the operation apparatus which concerns on one Embodiment of this invention. It is a schematic diagram which showed the vehicle which is equipped with the operation device which concerns on the said embodiment. It is a schematic diagram for demonstrating the detection area, the 1st area and the 2nd area of the operation apparatus which concerns on the said embodiment. It is a figure which shows the structural example of the 1st operation pattern and the 2nd operation pattern which concerns on the said embodiment. FIG. 5 is a schematic diagram showing an example of an image and a projected image displayed on the display unit according to the same embodiment. (A) to (d) are diagrams for explaining a transition example of the projected image according to the same embodiment. (A) and (b) are diagrams for explaining a transition example of the projected image according to the same embodiment. It is the flowchart which shows an example of the operation situation notification processing which concerns on the said embodiment. It is a flowchart which shows an example of the user identification process which concerns on the said embodiment. It is the flowchart which shows an example of the operation target notification processing which concerns on the said embodiment. (A) and (b) are diagrams for explaining the transition example of the projected image according to the modified example.

An embodiment of the present invention will be described with reference to the drawings.

As shown in FIG. 1, the operation device 100 according to one embodiment of the present embodiment is mounted on the vehicle 1 and enables the operation of the predetermined device by the gesture by the hand H (see FIG. 2) of the user U. be. As shown in FIG. 2, the operation using the operation device 100 is possible by both the driver U1 seated in the driver's seat 4 of the vehicle 1 and the passenger U2 seated in the passenger seat 5. In this embodiment, it is assumed that the vehicle 1 is an RHD (Right Hand Drive).

As shown in FIG. 1, the operation device 100 includes a non-contact sensor 10, a projector 20, a display unit 30, and a control device 40.

The non-contact sensor 10 is composed of a visible light camera, an infrared camera, a TOF (Time of Flight) camera, a millimeter wave sensor, and the like, and measures two-dimensional or three-dimensional information of a detection target (hereinafter, detection target information). It is transmitted to the control device 40. The non-contact sensor 10 is installed on the ceiling of the vehicle 1, for example. In this embodiment, the non-contact sensor 10 detects the hand H of the user U in the detection area Ad preset in the interior of the vehicle 1, as shown in FIG. Therefore, the space overlapping the detection area Ad becomes an input space that can be operated by the gesture of the user U. That is, the photographing range of the non-contact sensor 10 or the irradiation range of the electromagnetic wave includes the detection area Ad. The detection area Ad is set, for example, in the central portion of the front seat of the vehicle 1 and in a portion overlapping the center console 2 and the dashboard 3.

Under the control of the control device 40, the projector 20 projects an image onto an object existing in a projection area Ap preset in the interior of the vehicle 1. As shown in FIG. 2, the projection region Ap includes the detection region Ad and is set to a region wider than the detection region Ad. Further, the projection area Ap is set so as to overlap with a part of the display unit 30. As the projector 20, a known projector using a liquid crystal device, a digital mirror device, or the like can be used. The projector 20 is installed on the ceiling of the vehicle 1, for example.

The display unit 30 is composed of an LCD (Liquid Crystal Display), an OLED (Organic Light Emitting Diodes), and the like, and displays an image under the control of the control device 40. As shown in FIG. 2, the display unit 30 is provided at a position visible to both the driver U1 and the passenger U2 in the interior of the vehicle 1. The display unit 30 is provided on the dashboard 3 above the center console 2, for example.

The control device 40 is composed of a microcomputer that controls the overall operation of the operation device 100, and is composed of a control unit 41 composed of a CPU (Central Processing Unit) and the like, a ROM (Read Only Memory), a RAM (Random Access Memory), and the like. It includes a storage unit 42 that is configured. The ROM of the storage unit 42 stores data of the program PG, the first operation pattern PA, and the second operation pattern PB for executing the operation status notification process described later. Further, the ROM of the storage unit 42 stores image part data for generating the projected image of the projector 20 and the display image of the display unit 30. Further, the control device 40 includes a drive circuit for driving the projector 20 and the display unit 30, an input / output circuit for communicating with the device group 200, and the like, as a configuration (not shown). The configuration and arrangement of the control device 40 is arbitrary as long as it satisfies the functions described later. For example, the control device 40 may be realized by a plurality of computers communicating with each other.

The device group 200 includes a well-known in-vehicle system such as an ECU (Electronic Control Unit) for controlling each part of the vehicle 1, a car navigation (car navigation) device, an air conditioner (air conditioner) device for adjusting the temperature inside the vehicle, and a media player. ing. The device group 200 also includes a mobile terminal such as a smartphone or tablet terminal that is brought in by the user U and can communicate with the control device 40 wirelessly or by wire. For example, the control unit 41 communicates with each device constituting the device group 200, and causes the display unit 30 to display an image showing the status of each device, an image for operating each device, and the like. Further, the control unit 41 controls the display operation of the display unit 30 and the operation of the device group 200 according to the gesture made by the user U. In this embodiment, each device constituting the device group 200 and the display unit 30 are operation targets that can be operated by the gesture of the user U.

As shown in FIG. 1, the control unit 41 includes a detection unit 41a, a gesture identification unit 41b, an operation control unit 41c, a projection control unit 41d, and a display control unit 41e as main functions.

The detection unit 41a detects the hand H of the user U in the detection region Ad based on the detection target information from the non-contact sensor 10. For example, the detection unit 41a calculates the difference of the detection target information transmitted from the non-contact sensor 10 in a predetermined detection cycle for each cycle, and detects the region of the hand H. Then, the detection unit 41a extracts the coordinate information of the feature points such as the fingertip and the palm of the hand H based on the information indicating the detected region of the hand H. The method for detecting the region of the hand H and the method for extracting the feature points are arbitrary, and a known method can be used. For example, a pattern matching method or a feature point calculation method such as HOG (Histogram of Oriented Gradients), SIFT (Scale-Invariant Feature Transform), LBP (Local Binary Pattern), and Haar-Like can be used. When the detection unit 41a detects the user U's hand H in the detection area Ad, the detection area Ad does not have to include the entire user U's hand H, and the object is the hand H. It suffices that a part of the hand H is located in the detection region Ad to the extent that the detection unit 41a can be identified.

Further, the detection unit 41a can identify whether the hand H detected in the detection area Ad is the hand H1 of the driver U1 or the hand H2 of the passenger U2. Here, as shown in FIG. 3, the detection region Ad is composed of a first region A1 and a second region A2 classified by the division line B. The first region A1 is located on the right side of the dividing line B, and the second region A2 is located on the left side of the dividing line B. In this embodiment, since the vehicle 1 is RHD, the first region A1 is closer to the driver U1 than the second region A2, and the second region A2 is closer to the passenger U2 than the first region A1. Therefore, the hand H located in the first region A1 can be estimated as the hand H1 of the driver U1, and the hand H located in the second region A2 can be estimated as the hand H2 of the passenger U2. .. In consideration of this, the detection unit 41a identifies the hand H detected in the first region A1 as the hand H1 of the driver U1, and the hand H detected in the second region A2 is the hand H2 of the passenger U2. To identify. The coordinate information that defines each of the detection area Ad, the first area A1, and the second area A2 is stored in advance in the ROM of the storage unit 42.

The first area A1 may be set as an area where the hand H1 of the driver U1 is easily reachable and the hand H2 of the passenger U2 is hard to reach. On the contrary, the second region A2 may be set as an region where the hand H2 of the passenger U2 is easily reachable and the hand H1 of the driver U1 is hard to reach. That is, the position and shape of the dividing line B, which is the boundary line between the first region A1 and the second region A2, is arbitrary. For example, the dividing line B does not have to be linear. Further, the first region A1 and the second region A2 may be included in the detection region Ad, and for example, a configuration in which both regions are arranged at intervals may be adopted. When this configuration is adopted, when the detection unit 41a detects the hand H in a region other than the first region A1 and the second region A2 in the detection region Ad, the gesture by the detected hand H is invalidated. May be. When the vehicle 1 is an LHD (Left Hand Drive), the relationship between the first region A1 and the second region A2 may be set to be opposite to the above. Further, it is assumed that a plurality of hands H are detected at the same time in any of the first region A1 and the second region A2 (hereinafter referred to as one region). In this situation, for example, the detection unit 41a waits for the identification of the user U until the hands H included in one area become one, or identifies the user U based on the previously detected hand H. do it.

The gesture identification unit 41b identifies the gesture by the hand H detected by the detection unit 41a. For example, the gesture identification unit 41b creates an operation locus of the user U's hand H based on the coordinate information of the hand H extracted by the detection unit 41a. Then, the gesture specifying unit 41b compares the operation locus of the created hand H with the gesture pattern stored in the ROM of the storage unit 42, and identifies the gesture by the hand H. The gesture pattern is stored in the ROM in advance as data indicating a predetermined plurality of types of gesture trajectories. The gesture identification method by the gesture identification unit 41b is arbitrary, and for example, a known method such as an NN method (Nearest Neighbor algorithm), a k-NN method (k-Nearest Neighbor algorithm), or a pattern matching method can be used. can. The gesture mode that enables input is arbitrary, but for example, if it is various actions such as swipe, scroll, pinch-in, pinch-out, etc. performed in the detection area Ad, or a pose such as thumbs-up or peace sign. good.

The operation control unit 41c identifies the operation corresponding to the gesture specified by the gesture specifying unit 41b, and outputs information indicating the content of the specified operation to the operation target. Specifically, the operation control unit 41c refers to the operation pattern data stored in the ROM of the storage unit 42, and identifies the operation corresponding to the specified gesture. As the operation pattern, a first operation pattern PA and a second operation pattern PB for which the configuration is illustrated in FIG. 4 are provided.

The first operation pattern PA defines a plurality of types of operations permitted to the driver U1. The operations specified in the first operation pattern PA are an operation permitted only to the driver U1 (hereinafter referred to as a driver permission operation) and an operation permitted to both the driver U1 and the passenger U2 (hereinafter, both permission). It is called operation.) Includes. The driver permission operation is an operation related to the operation of the vehicle 1, and is, for example, a setting operation of a traveling mode, ADAS (Advanced Driver-Assistance Systems), wiper, headlight, and the like. The both permission operations are operations that do not hinder driving safety even when operated by the driver U1 and the passenger U2, and are, for example, setting operations of an air conditioner, music (music reproduction), radio, and the like. When the specified gesture is due to the hand H1 of the driver U1, the operation control unit 41c refers to the first operation pattern PA, identifies the operation corresponding to the specified gesture, and determines the content of the specified operation. The indicated signal is output to the operation target. For example, when the operation control unit 41c specifies a driving mode setting operation, the operation control unit 41c outputs a signal indicating the operation content to the ECU that controls the driving mode of the vehicle 1, and the ECU controls the driving mode accordingly. do. When the control of the travel mode of the vehicle 1 is completed according to the content of the travel mode setting operation, the ECU supplies an answerback signal to that effect to the operation control unit 41c.

The second operation pattern PB defines a plurality of types of operations permitted to the passenger U2. The operation defined in the second operation pattern PB includes an operation permitted only to the passenger U2 (hereinafter referred to as a passenger permission operation) and the above-mentioned both permission operation. The passenger permission operation is an operation that is not allowed by the driver U1 in order to ensure the safety of the vehicle 1 while traveling, and is an operation for using, for example, a car navigation system, video playback, e-mail, SNS, or the like. When the specified gesture is due to the hand H2 of the passenger U2, the operation control unit 41c refers to the second operation pattern PB, identifies the operation corresponding to the specified gesture, and the content of the specified operation. Is output to the operation target. For example, when the operation control unit 41c specifies a car navigation setting operation, the operation control unit 41c outputs a signal indicating the operation content to the car navigation device, and the car navigation device reflects the setting accordingly. When the control according to the content of the car navigation setting operation is completed, the car navigation device supplies an answerback signal to that effect to the operation control unit 41c.

The projection control unit 41d controls the operation of the projector 20 and projects an image onto an object existing in the projection area Ap.

Specifically, when the detection unit 41a detects the hand H, the projection control unit 41d provides a first image Ca for notifying the user U to that effect, as shown in FIGS. 5 and 6A. Project to any position on the hand H. For example, the projection control unit 41d projects the first image Ca as a band-shaped image projected on the wrist of the hand H. Here, when the detection unit 41a detects the hand H1 of the driver U1, the projection control unit 41d projects the first color image Ca1 having the first color on the detected hand H1 as the first image Ca. do. When the detection unit 41a detects the hand H2 of the passenger U2, the detected hand H2 is designated as the second color image Ca2 having a second color different from the first color as the first image Ca. Project. The first color and the second color are arbitrary as long as they are different colors, for example, one may be red and the other may be blue.

Further, when the gesture specifying unit 41b identifies the gesture, the projection control unit 41d hands H the second image Cb for notifying the user U to that effect, as shown in FIGS. 5 and 6 (b). Project to any position of. For example, the projection control unit 41d projects the second image Cb as a circular image projected on the fingertip of the hand H. In this embodiment, the second image Cb is, for example, an image having a color different from the first color and the second color described above, and is the same regardless of whether the user U is the driver U1 or the passenger U2. It is projected and displayed in the mode. The color of the second image Cb is arbitrary as long as it is a color different from the first color and the second color described above, and may be, for example, white or yellow.

Further, when the operation control unit 41c specifies the operation corresponding to the gesture, the projection control unit 41d projects the operation effective image Cc obtained by changing the second image Cb onto the hand H as shown in FIG. 6 (c). do. The operation effective image Cc is, for example, an image in which the second image Cb is stretched and enlarged. On the other hand, when the operation control unit 41c cannot specify the operation corresponding to the gesture, the projection control unit 41d is an image obtained by changing the second image Cb as shown in FIG. 6D, and the operation is effective. An operation-invalid image Cd having a display mode different from that of the image Cc is projected onto the hand H. The operation invalid image Cd is, for example, a blinking display of an image having the same shape as the second image Cb. The display mode of the operation valid image Cc and the operation invalid image Cd is arbitrary depending on the design. For example, the operation valid image Cc and the operation invalid image Cd may be displayed differently from each other, and at least one of the size, the display position, the color, and the lighting pattern may be different from the second image Cb.

Further, when the operation specified by the operation control unit 41c is a special operation of a predetermined type, the projection control unit 41d projects and displays the locus image Ce as shown in FIGS. 7A and 7B. Thereby, the target of the special operation can be notified to the user U. The special operation may be defined as, for example, one or a plurality of operations of a high importance of notifying the user U of the operation target, and may be stored in the ROM of the storage unit 42. For example, with respect to the wiper drive operation and the headlight lighting operation by the driver U1, it is possible to visually recognize that the operation has been performed by the driver U1, so that such an operation can be excluded from the special operation.

The locus image Ce includes a first locus image Ce1 from the user U's hand H toward the target image D, which will be described later, and a second locus image Ce2 from the target image D toward the hand H. .. The first locus image Ce1 is, for example, a locus-shaped image that moves from the fingertip of the hand H toward the target image D. The first locus image Ce1 is an image projected by the projector 20 when the operation corresponding to the gesture performed by the user U is specified. The second locus image Ce2 is, for example, a locus-shaped image that moves from the target image D toward the fingertip of the hand H. The second locus image Ce2 is an image projected by the projector 20 when the specified operation is reflected in the operation target. The display conditions of the first locus image Ce1 and the second locus image Ce2 will be described together with the operation status notification process described later.

The locus image Ce is projected onto an object in the vehicle 1, that is, the detected hand H, the center console 2, and the display unit 30, so as to connect the hand H and the target image D. For example, the projection control unit 41d projects the locus image Ce to a desired position based on the projection position information that defines the projection area Ap in the interior of the vehicle 1 stored in the ROM of the storage unit 42 in advance. The projection control unit 41d acquires the coordinate information indicating the position of the hand H1 from the detection unit 41a, and also acquires the position information indicating the display position of the target image D on the display unit 30 from the display control unit 41e. Then, the projection control unit 41d calculates the projection position corresponding to each of the coordinate information and the position information, and projects the locus image Ce based on the calculated projection position. Although FIGS. 7A and 7B show an example of representing a linear locus represented by the locus image Ce, the display mode of the locus image Ce is arbitrary. For example, the locus image Ce may represent a curved locus or a random locus as long as it goes from one of the detected hand H and the target image D to the other. Further, the locus image Ce is not limited to a linear shape, but may have a strip-shaped shape, a radial shape, or the like. Further, the locus image Ce does not have to be superimposed on both the hand H and the target image D as long as it is directed from one of the detected hand H and the target image D to the other. It may be displayed between the target images D.

As shown in FIG. 5, the display control unit 41e causes the display unit 30 to display a target image D showing information about a target that can be operated by the gesture of the user U. The information about the target includes not only the information indicating the target device itself, but also the operation items of the target device, the information indicating some functions of the target device, and the like. The target image D may have a configuration in which information about the target is represented by characters, figures, icons, and the like.

The target image D includes a first target image D1 showing information about the operation target specified in the first operation pattern PA, and a second target image D2 showing information about the operation target specified in the second operation pattern PB. ,including. FIG. 5 shows a first target image D1 showing that the driver U1 can operate the driving mode, and a second target showing that the passenger U2 can operate the mobile terminal connected to the control device 40. This is an example in which the image D2 is displayed on the display unit 30.

As shown in FIG. 5, the display control unit 41e causes the display unit 30 to display the first target image D1 in a manner including the same color as the color of the first color image Ca1 (the first color described above) or a related color. , The second target image D2 is displayed on the display unit 30 in a manner including the same color as the color of the second color image Ca2 (the above-mentioned second color) or a related color. As a result, the driver U1 can intuitively recognize that the target indicated by the first target image D1 can be operated by projecting the first color image Ca1 onto its own hand H1. .. Further, the passenger U2 can intuitively recognize that the target indicated by the second target image D2 can be operated by projecting the second color image Ca2 onto his / her hand H2. .. The color associated with the first color may be a similar color or a similar color of the first color. For example, a similar color of the first color may be a color having the same hue as the first color but different in lightness and saturation. Further, the similar color of the first color may be a color that is a combination of hues similar to the first color. This relationship is the same for the colors associated with the second color.

It should be noted that the mode in which the first target image D1 includes the same color as or related to the first color (hereinafter, also referred to as a corresponding color) is arbitrary, and parts such as frames and characters constituting the first target image D1 are included. It may have a corresponding color, or the entire first target image D1 may be displayed in the corresponding color. The display control unit 41e may constantly display the first target image D1 on the display unit 30, or may display the first target image D1 on the display unit 30 when a predetermined operation is received. Further, the display control unit 41e may switch the display of the first target image D1 to a display including the corresponding color of the first color image Ca1 on condition that the first color image Ca1 is projected on the driver U1. good. The same applies to the second target image D2.

Subsequently, the operation status notification process executed by the control unit 41 will be described with reference to FIGS. 8 to 10. When the operation status notification process shown in FIG. 8 is started, the control unit 41 first determines whether or not the user U has been specified (step S101). When the user U has not been specified yet (step S101; No), the control unit 41 executes the process shown in FIG. 9 as the user identification process (step S200), and determines whether or not the hand H is detected in the detection area Ad. Determine (step S201). When the hand H is detected (step S201; Yes), the control unit 41 identifies the user U based on the first region A1 and the second region A2 set in the detection region Ad as described above. (Step S202). After the process of step S202, or when the hand H is not detected (step S201; No), the control unit 41 ends the user identification process.

Returning to FIG. 8, when the user U is specified (step S101; Yes), the control unit 41 determines whether or not the hand H of the user U specified in the user identification process is detected (step S101; Yes). Step S102). Here, the specified user U may be at least one of the driver U1 and the passenger U2. When the hand H1 of the driver U1 and the hand H2 of the passenger U2 are detected at the same time, the control unit 41 executes the processes after step S102 in parallel for each specified user U.

When the specified user U's hand H is detected (step S102; Yes), the control unit 41 projects the first image Ca on the wrist of the hand H as shown in FIG. 6 (a) (step S103). , Notifies the user U that the hand H is in the detection area Ad. When the identified user U is the driver U1, the first color image Ca1 is projected as the first image Ca on the hand H1 of the driver U1. When the identified user U is the passenger U2, the second color image Ca2 is projected as the first image Ca on the hand H2 of the passenger U2. When the specified user U's hand H is not detected (step S102; No), the control unit 41 hides the first image Ca (step S104) and returns to step S101 to execute the process.

Following step S103, the control unit 41 identifies the gesture by the hand H detected in step S102 (step S105). When the gesture is specified (step S105; Yes), the control unit 41 projects the second image Cb on the fingertip of the hand H (step S106) as shown in FIG. 6 (b), and the gesture is given to the user U. Notify that it is being accepted. When the gesture cannot be specified (step S105; No), the control unit 41 hides the second image Cb (step S107), returns to step S101, and executes the process.

Following step S106, the control unit 41 refers to the operation pattern according to the user U (step S108). Specifically, the control unit 41 refers to the first operation pattern PA when the user U is the driver U1, and refers to the second operation pattern PB when the user is the passenger U2.

Subsequently, the control unit 41 determines whether or not the operation corresponding to the gesture specified in step S105 is within the operation pattern (step S109). When the operation corresponding to the gesture is specified by referring to the operation pattern (step S109; Yes), the control unit 41 outputs a signal indicating the content of the specified operation to the operation target, and executes the process of step S110. .. When the specified gesture is due to the hand H1 of the driver U1, the control unit 41 refers to the first operation pattern PA, identifies the operation corresponding to the specified gesture, and indicates the content of the specified operation. Output the signal to the operation target. As a result, the operation permitted by the driver U1 can be executed. When the specified gesture is due to the hand H2 of the passenger U2, the control unit 41 refers to the second operation pattern PB, identifies the operation corresponding to the specified gesture, and determines the content of the specified operation. The indicated signal is output to the operation target. As a result, the operation permitted to the passenger U2 can be executed.

In step S110, the control unit 41 determines whether or not the specified operation type is a predetermined special operation, and if it is not a special operation (step S110; No), the second image Cb shown in FIG. 6 (b). As shown in FIG. 6 (c), the operation effective image Cc of the modified mode is projected onto the hand H of the user U (step S111). On the other hand, when the operation corresponding to the gesture is not specified in step S109 (step S109; No), the control unit 41 changes the second image Cb shown in FIG. Is projected onto the hand H of the user U as shown in FIG. 6 (d) (step S112).

When the type of the specified operation is a special operation in step S110 (step S110; Yes), the control unit 41 executes the process shown in FIG. 10 as the operation target notification process (step S300), and FIG. 7 (a) shows. As shown in the above, the first locus image Ce1 is projected and displayed for a certain period of time (step S301). Subsequently, the control unit 41 determines whether or not a answerback signal indicating that the control according to the content of the operation has been completed has been received from the operation target that has output the signal indicating the content of the operation (step S302). ). When the answerback signal is received (step S302; Yes), the control unit 41 projects and displays the second locus image Ce2 for a certain period of time as shown in FIG. 7B (step S303). The period for projecting the first locus image Ce1 and the second locus image Ce2 may be, for example, a period of several seconds. After the execution of step S303 or when the answerback signal is not received (step S302; No), the control unit 41 ends the operation target notification process.

Returning to FIG. 8, after the processing of steps S111, S112, and S300, the control unit 41 returns to step S101 to execute the processing. The above operation status notification process is continuously executed, for example, while the operation device 100 is being activated.

The operation status notification process may be executed under certain conditions such as when the vehicle 1 is traveling. For example, the control unit 41 may execute the operation status notification process while the prime mover, which is the engine or motor of the vehicle 1, is being activated and when the parking brake of the vehicle 1 is released. Further, in the operation status notification process, the control unit 41 executes each process constituting steps S101, S102, and step S200 by the function of the detection unit 41a. Further, the control unit 41 executes step S105 by the function of the gesture identification unit 41b. Further, the control unit 41 executes steps S108 to S110 and S302 by the function of the operation control unit 41c. Further, the control unit 41 executes steps S103, S104, S106, S107, S111, S112, S301, and S303 by the function of the projection control unit 41d.

The present invention is not limited to the above embodiments and drawings. Changes (including deletion of components) can be made as appropriate without changing the gist of the present invention.

In the above embodiment, the operation device 100 includes the display unit 30 as its function, but it is optional to include the display unit 30. Even if the display unit 30 is not provided, or in addition to the display unit 30, a display device installed in the room of the vehicle 1 and capable of communicating with the operation device 100 displays the state of each device and each image for operation. Often, each image may be displayed on a display device brought into the room of the vehicle 1 by the user U. Further, an audio output unit may be provided to output audio for the state and operation of each device.

In the above embodiment, the example in which the second image Cb is projected on the fingertip of the hand H is shown, but the projection position of the second image Cb on the hand H is arbitrary. For example, as shown in FIG. 11A, the second image Cb may be projected onto the back of the hand H. Further, as shown in FIG. 11B, the operation effective image Cc may have an aspect in which an image part such as a frame is added to the second image Cb.

The shapes and projection positions of the first image Ca, the second image Cb, the operation valid image Cc, and the operation invalid image Cd are not limited to the above examples and are arbitrary. For example, the color of the second image Cb may be different for each user U as in the case of the first image Ca. Further, the first image Ca and the second image Cb do not have to be projected at different positions of the hand H. For example, the second image Cb may be a modified form of the first image Ca.

In the above, an example of identifying the user U based on the first region A1 and the second region A2 provided in the detection region Ad has been shown, but the method for identifying the user U is arbitrary. For example, it is possible to adopt a configuration in which information about the hand H of the person who uses the vehicle 1 is registered in the operation device 100 in advance and the user U is specified based on the registered information. Further, when the vehicle 1 is RHD, if the detected hand H is the left hand, it may be specified as the driver U1's hand H1, and if the detected hand H is the right hand, it may be specified as the passenger U2's hand H2. .. This is because when the detection region Ad is located in the center of the vehicle 1, the driver U1 is likely to make a gesture with the left hand and the passenger U2 is likely to make a gesture with the right hand. When the vehicle 1 is an LHD, the opposite relationship may be considered. A known method such as a pattern matching method can be used to determine whether the hand H is the left hand or the right hand.

In the above, an example in which a locus-shaped image directed from one of the hand H and the target image D to the other is projected as the locus image Ce has been shown, but the locus image Ce is from one of the hand H and the target portion described below. It may be a locus-shaped image toward the other side. The target unit may be an operation target (that is, each device or display unit 30 constituting the device group 200) or a related unit related to the operation target. The related part includes the target image D described above. Further, the related part may be, for example, an air outlet when the air conditioner device is the operation target, an indicator for notifying the travel mode when the ECU that controls the traveling mode is the operating target, or the like.

The arrangement of the non-contact sensor 10, the projector 20, and the display unit 30 in the vehicle 1 is arbitrary and can be changed as appropriate. Further, the vehicle on which the operation device 100 is mounted is not limited to the vehicle 1, and may be any vehicle other than the vehicle 1, such as a ship or an aircraft.

The program PG that executes the operation status notification process described above is assumed to be stored in the storage unit 42 in advance, but may be distributed and provided by a detachable recording medium. Further, the program PG may be downloaded from another device connected to the operating device 100. Further, the operating device 100 may execute each process according to the program PG by exchanging various data with other devices via a telecommunication network or the like.

In the above, the example in which the part of the human body detected by the detection unit 41a is the hand H of the user U has been described, but the present invention is not limited to this. As long as it is a part of the vehicle where gestures can be safely performed, it may be a foot or the like. The term "hand" in the present specification refers to a part of the human body beyond the shoulder, and includes not only fingers and palms but also elbows and the like.

In the above, the control unit 41 has shown an example of specifying the user U, but the operation status notification process may be executed without specifying the user U. For example, the processing of step S101 and step S200 of the operation status notification processing can be omitted. Further, although the example of projecting the locus image Ce in a predetermined case is shown above, the process of projecting the locus image Ce may be omitted. For example, the process of step S300 of the operation status notification process can be omitted.

The operating device 100 described above detects the hand H (an example of a part) of the user U in the detection area Ad preset in the vehicle 1 (an example of a vehicle) based on the information from the non-contact sensor 10. The detection unit 41a, the gesture identification unit 41b that identifies the gesture by the hand H detected by the detection unit 41a, and the operation corresponding to the gesture specified by the gesture identification unit 41b are specified, and information indicating the content of the specified operation is provided. It includes an operation control unit 41c that outputs to an operation target, and a projection unit that can project an image on an object existing in the vehicle 1 (for example, a projector 20 that operates under the control of the projection control unit 41d).

(1-1) In the operating device 100, the projection unit projects the first image Ca onto the hand H when the detection unit 41a detects the hand H, and the gesture identification unit 41b identifies the gesture. A second image Cb different from the first image Ca is projected onto the hand H. According to this configuration, the user U can grasp from the projected image that the hand H is detected and that the gesture is accepted. Therefore, it is possible to prevent the user U from repeating meaningless gestures.

(1-2) Further, when the operation control unit 41c specifies the operation corresponding to the gesture, the projection unit projects the operation effective image Cc obtained by changing the second image Cb onto the hand H. According to this configuration, it is possible to notify the user U that the operation is being accepted.

(1-3) Further, the projection unit is an image obtained by changing the second image Cb when the operation control unit 41c cannot specify the operation corresponding to the gesture, and is a display mode different from the operation effective image Cc. The operation invalid image Cd of is projected on the hand H. According to this configuration, it is possible to notify the user U that the operation is not accepted.

(1-4) Specifically, the operation valid image Cc and the operation invalid image Cd are different from each other in the display mode, and at least one of the size, the display position, the color, and the lighting pattern is different from the second image Cb.

(1-5) Further, the first image Ca and the second image Cb may be projected at different positions of the hand H.

(2-1) In the operating device 100, in the detection unit 41a, the hand H is the hand H1 of the driver U1 (an example of the first user) or the hand H2 of the passenger U2 (an example of the second user). Can be specified. Further, when the operation control unit 41c specifies the gesture by the hand H1 of the driver U1, the operation control unit 41c specifies the operation corresponding to the gesture based on the first operation pattern PA that defines a plurality of types of operations permitted to the driver U1. Then, when the gesture by the hand H2 of the passenger U2 is specified, the operation corresponding to the gesture is specified based on the second operation pattern PB that defines a plurality of types of operations permitted to the passenger U2. Further, at least a part of the first operation pattern PA and the second operation pattern PB is different. According to this configuration, the operation set according to the type of the user can be appropriately specified.

(2-2) Further, the projection unit projects the first color image Ca1 having the first color on the hand H1 of the driver U1, and the second color different from the first color is projected on the hand H2 of the passenger U2. The second color image Ca2 having the above is projected. According to this configuration, each of the plurality of users U can grasp the target that can be operated by itself.

(2-3) Further, the display unit 30 included in the operation device 100 displays the first target image D1 in a mode including the same color as the first color or a color related to the first color, and displays the second target image D2 in the second color. Display in a mode including the same color as or related to. According to this configuration, the user U can grasp the target that can be operated more intuitively.

(2-4) Specifically, the detection unit 41a identifies the hand H detected in the first region A1 of the detection region Ad as the hand H1 of the driver U1, and sets it as the first region A1 of the detection region Ad. Can identify the hand H detected in the different second region A2 as the hand H2 of the passenger U2.

(3-1) In the operation device 100, when the operation control unit 41c specifies an operation corresponding to the gesture, the projection unit projects a locus-shaped locus image Ce from one of the hand H and the target unit to the other. .. The target unit is an operation target or a related portion related to the operation target (for example, the target image D). According to this configuration, the user U can easily grasp the operated target.

(3-2) When the operation control unit 41c specifies the operation corresponding to the gesture, the projection unit projects the first trajectory image Ce1 from the hand H toward the target unit, and then the second projection unit from the target unit toward the hand H. The locus image Ce2 can be projected.

(3-3) Specifically, the projection unit projects the second locus image Ce2 when the operation specified by the operation control unit 41c is reflected in the operation target. According to this configuration, the user U can intuitively grasp that the operation is reflected.

(3-4) The projection unit projects the locus image Ce when the operation corresponding to the gesture specified by the operation control unit 41c is a special operation of a predetermined type, and the locus image when it is not a special operation. Do not project Ce. According to this configuration, it is possible to intuitively notify the highly necessary operation with the locus image Ce. In addition, it is possible to prevent the locus image Ce from being projected at every operation and causing trouble to the user U.

(3-5) The related part may be an image displayed on the display unit 30 in the vehicle 1 and may be a target image D showing information about the operation target.

In the above description, in order to facilitate the understanding of the present invention, the description of known technical matters has been omitted as appropriate.

1 ... Vehicle, 2 ... Center console, 3 ... Dashboard, 4 ... Driver's seat, 5 ... Passenger seat U ... User, U1 ... Driver, U2 ... Passenger, H, H1, H2 ... Hand 100 ... Operating device 10 ... Non Contact type sensor, 20 ... Projector, 30 ... Display unit 40 ... Control device, 41 ... Control unit, 42 ... Storage unit 41a ... Detection unit 41b ... Gesture identification unit 41c ... Operation control unit 41d ... Projection control unit 41e ... Display control unit Ad ... detection area, A1 ... first area, A2 ... second area Ap ... projection area B ... division line Ca ... first image, Ca1 ... first color image, Ca2 ... second color image Cb ... second image Cc ... Operation valid image Cd ... Operation invalid image Ce ... Trajectory image, Ce1 ... First trajectory image, Ce2 ... Second trajectory image D ... Target image, D1 ... First target image, D2 ... Second target image

Claims (5)

A detector that detects the user's part in a preset detection area in the vehicle based on information from the non-contact sensor.
A gesture identification unit that identifies a gesture by the site detected by the detection unit, and a gesture identification unit.
An operation control unit that identifies an operation corresponding to the gesture specified by the gesture specifying unit and outputs information indicating the content of the specified operation to the operation target.
A projection unit capable of projecting an image onto an object existing in the vehicle is provided.
The projection unit
When the detection unit detects the site, a first image is projected onto the site.
When the gesture specifying portion identifies the gesture, a second image different from the first image is projected onto the portion.
Operating device. When the operation control unit identifies an operation corresponding to the gesture, the projection unit projects an operation effective image obtained by changing the second image onto the portion.
The operating device according to claim 1. The projection unit
When the operation control unit identifies an operation corresponding to the gesture, the operation effective image is projected onto the portion.
When the operation control unit cannot specify the operation corresponding to the gesture, an operation invalid image having a display mode different from that of the operation valid image, which is an image obtained by changing the second image, is projected onto the portion. ,
The operating device according to claim 2. The operation valid image and the operation invalid image are different from each other in display mode, and at least one of the size, display position, color, and lighting pattern is different from the second image.
The operating device according to claim 3. The first image and the second image are projected at different positions of the portion.
The operating device according to any one of claims 1 to 4.

Images