JP7182442B2 - Instruction identification system - Google Patents

Description

TECHNICAL FIELD The present invention relates to a technique for specifying instructions of a user on board a vehicle.

Techniques have been developed for identifying instructions of users in vehicles in drive-throughs, parking lots, and the like.
Patent Literature 1 describes an automatic transaction device that optimizes relative positions with respect to various vehicles in order to improve operability.
Patent Document 2 describes a vehicle and a reader that facilitate mobile commerce by a user in the vehicle.

JP-A-7-272064 JP 2014-89722 A

2. Description of the Related Art In a drive-thru or a parking lot, a user in a car needs to give instructions to people and devices outside the car, such as a store clerk, a janitor, a guide terminal, and the like. At this time, it is common for the user inside the vehicle to open the window glass and indicate his/her intention to the device or the like outside the vehicle. However, when the window glass is opened, outside air containing exhaust gas, rain, etc. enters the vehicle, and the air inside the vehicle, whose temperature and humidity have been adjusted by the air conditioning equipment, may be disturbed. In addition, when a user inside the vehicle opens the window glass and gives instructions to the outside of the vehicle by voice, it may cause noise for nearby residents. Furthermore, when transmitting confidential information such as a personal identification number between inside and outside the vehicle, if the window glass is opened and the confidential information is communicated by voice, there is a possibility that someone will hear the voice, which poses a security problem. . On the other hand, if wireless communication using radio waves that pass through window glass is used to transmit information between inside and outside the vehicle, special equipment such as a modulation circuit and a demodulation circuit must be installed inside the vehicle.

One of the objects of the present invention is to enable a user in a vehicle to direct his/her intention to a device outside the vehicle with the window glass completely closed without installing any special equipment inside the vehicle. It aims to make it possible.

In order to solve the above-described problems, the present invention irradiates a window glass of a vehicle with light for displaying information prompting a user to instruct, and displays the information by reflected light of the light transmitted through the window glass, A first aspect of the present invention provides an instruction identifying system for photographing the window glass and identifying the instruction of the user from the photographed image.

According to the instruction identification system of the first aspect, without installing any special equipment in the vehicle, the user boarding the vehicle directs his or her intention toward the device outside the vehicle with the window glass completely closed. can do. In addition, according to the instruction identification system of the first aspect, a user in a vehicle directs the light transmitted through the window glass toward a device outside the vehicle using the reflected light reflected by an object inside the vehicle. They can direct their own will.

In the instruction specifying system of the first aspect, the information is displayed by reflecting the light transmitted through the windowpane on a windowpane different from the windowpane . good.

According to the instruction identification system of the second aspect, a user in a vehicle sees the information displayed by the light reflected by the window glass different from the window glass through which the light is transmitted, and points the information to the device outside the vehicle. can indicate the intention of

In the instruction identifying system of the first or second aspect, a configuration may be adopted as a third aspect in which the instruction is identified from a change over time in the position of the pointer moved by the user, which is included in the image. .

According to the instruction identification system of the third aspect, for example, when the pointer moved by the user does not move within a predetermined range for a predetermined period of time, a different instruction is identified.

In the instruction identifying system of any one of the first to third aspects, a configuration in which the instruction is identified from the shape of the pointer moved by the user, which is included in the image, may be employed as a fourth aspect. good.

According to the instruction identification system of the fourth aspect, the instruction is identified by, for example, the fact that the finger moved by the user is pressed against the window glass and deformed.
be done.

In the instruction specifying system of any one of the first to fourth aspects, a configuration may be adopted as a fifth aspect in which the instruction is specified based on the reflected wave of the electromagnetic wave irradiated toward the window glass. .

According to the indication specifying system of the fifth aspect, the position in the three-dimensional space between the indicator moved by the user and the window glass is specified by the reflected wave of the electromagnetic wave irradiated toward the window glass. Instructions are specified.

The figure which shows the example of a structure of the instruction|indication specific system 9 which concerns on one Embodiment of this invention. 4 is a flow chart showing the operation of the instruction specifying system 9; FIG. 1 is a schematic diagram of a vehicle C to which an instruction specifying system 9 is applied; FIG. FIG. 4 is a diagram showing an example of a screen DB 121 stored in a storage unit 12; 4 is a diagram showing an example of an instruction DB 122 stored in a storage unit 12; FIG. The figure explaining the path|route of the light which the irradiation apparatus 3 irradiated toward the windowpane W. FIG. The figure which shows the example which looked at the screen displayed on the windowpane W from the inside of a vehicle. The figure which shows the example of the windowpane W image|photographed from the outside of a vehicle. FIG. 10 is a diagram showing an example of a confirmation screen Px; The figure explaining other paths of the light which the irradiation apparatus 3 irradiated toward the windowpane W. FIG. FIG. 4 is a diagram for explaining the change over time of the position of the pointer moved by the user; FIG. 4 is a diagram for explaining changes in the shape of an indicator;

<Embodiment>
<Configuration of instruction specific system>
FIG. 1 is a diagram showing an example of the configuration of an instruction identification system 9 according to one embodiment of the present invention. An indication identification system 9 shown in FIG. The instruction identification system 9 is also connected to an external output device 8 .

The instruction specifying device 1 has a control section 11 , a storage section 12 and a connection section 13 . The control unit 11 is means for controlling the operation of each unit of the instruction specifying device 1 . The control unit 11 includes an arithmetic processing unit such as a CPU (Central Processing Unit) and a storage device such as a ROM (Read Only Memory) and a RAM (Random Access Memory). Run the program.

The storage unit 12 is storage means such as a solid state drive or a hard disk drive, and stores various programs, data, etc. read by the CPU of the control unit 11 . The storage unit 12 also stores a screen DB 121 and an instruction DB 122 .

The connection unit 13 is an interface for connecting the instruction specifying device 1 with the detection device 2 , the irradiation device 3 , and the imaging device 4 . Also, the connection unit 13 is connected to an output device 8 provided outside the instruction specifying system 9 by wire or wirelessly.

FIG. 2 is a flow diagram showing the operation of the instruction identification system 9. As shown in FIG. FIG. 3 is a schematic diagram of a vehicle C to which the instruction specifying system 9 is applied. The detection device 2 is, for example, a proximity sensor that detects the proximity of an object without contact, such as a photoelectric type, a capacitance type, or an ultrasonic wave type, or a load cell, a swing gate, or the like that detects the contacting object due to load or displacement. It is a detection sensor, and detects that the vehicle C exists at a predetermined position (step S101).

When the detection device 2 detects the vehicle C at a predetermined position, the irradiation device 3 irradiates the window glass of the vehicle C with light (step S102). This light is used to display information (operation screen) prompting the user to give instructions on the window glass.

In addition, the irradiation device 3 may always irradiate the light without being triggered by the detection by the detection device 2 . The user should just move the vehicle C so that the window glass is arranged at the position irradiated with this light.

In addition, the position of the window glass in the vehicle may differ, for example, in vertical and horizontal directions depending on the type of vehicle. Here, the irradiation device 3 may irradiate an image on an appropriate position as follows. That is, based on the image captured by the imaging device 4, the control unit 11 of the instruction specifying device 1 recognizes the position of the windowpane of the vehicle C using an edge detection algorithm or the like. Then, the instruction specifying device 1 sends a control signal to the irradiation device 3 for adjusting the light irradiation angle, focal length, etc., based on the information on the recognized position of the window glass. Thereby, the irradiation device 3 irradiates light onto an appropriate position where the window glass exists.

The photographing device 4 photographs the window glass irradiated with the light from the irradiation device 3 and supplies image data representing the photographed image to the instruction specifying device 1 . The photographing device 4 may start photographing at the timing of light irradiation by the irradiation device 3, or may photograph constantly.

When the control unit 11 of the instruction identification device 1 acquires the image data from the photographing device 4 via the connection unit 13, it identifies the instruction given by the vehicle user to the operation screen from the image indicated by the image data (step S103). . For example, from the image indicated by the acquired image data, the control unit 11 determines the position in the three-dimensional space of the information that prompts the user to give an instruction, which is displayed with the light described above, and the three-dimensional space of the pointer that the user moves. The upper position is specified, and the user's instruction is specified by referring to the screen DB 121 and the instruction DB 122 stored in the storage unit 12 . The position in the three-dimensional space is, for example, a position that includes depth direction information in addition to top, bottom, left, and right information as seen from the imaging device 4 . This information in the depth direction is specified based on, for example, the depth of focus. In addition, the instruction identification system 9 includes a plurality of photographing devices 4, and identifies the information in the depth direction based on the parallax angles of the images captured by two of the plurality of photographing devices 4. may

FIG. 4 is a diagram showing an example of the screen DB 121 stored in the storage unit 12. As shown in FIG. The screen DB 121 is a database that associates a screen displayed on the window glass as information prompting a user to give an instruction with light emitted by the irradiation device 3, and a user's instruction to the screen and a user's operation. Screen DB 121 includes screen list 1211 and area list 1212 .

The screen list 1211 is a list in which identification information (screen ID) of the screen described above and data (screen data) indicating the screen are stored in association with each other.

The area list 1212 associates an area included in the screen with operation information indicating the content of the operation indicated by the area when any position inside the area is pointed by the user's pointer. It is a list stored with the screen ID, and is provided for each screen ID of a plurality of screens.

When the control unit 11 of the instruction specifying device 1 acquires image data from the photographing device 4, the image displayed by the acquired image data overlaps the screen displayed on the window glass in the top, bottom, left, right, and depth directions. Alternatively, the position of the pointer of the user approaching within a predetermined distance is identified. The control unit 11 may specify the screen displayed on the window glass and the position of the indicator by image analysis processing that executes a so-called edge detection algorithm such as the Canny method. Further, the control unit 11 may identify the above-described screen and the position of the user's indicator from the difference between the image data respectively representing two images captured at different times.

After specifying the above screen and the position of the user's indicator, the control unit 11 determines whether the position of the indicator satisfies a determined condition. Then, when the position of the pointer satisfies the determined condition, the control unit 11 identifies a region on the screen corresponding to that position. Then, based on the screen DB 121, the control unit 11 identifies operation information indicating the content of the user's operation associated with the identified area.

FIG. 5 is a diagram showing an example of the instruction DB 122 stored in the storage unit 12. As shown in FIG. The instruction DB 122 shown in FIG. 5 includes a sequence of operation information (operation information sequence) determined by the order of the operation information, instruction information indicating the content of the instruction intended by the user, and when the instruction indicated by the instruction information is given. A database that associates and stores screen IDs (transition screen IDs) of screens to which the display for the user transitions.

The control unit 11 refers to the screen DB 121 and specifies instruction information indicating the content of the instruction intended by the user according to the order or combination of the operation information specified by the control unit 11 . Then, the control unit 11 specifies the transition screen ID associated with the specified instruction information, and instructs the irradiation device 3 to transition the display to the transition screen identified by the specified transition screen ID.

FIG. 6 is a diagram for explaining the path of the light irradiated toward the window glass W by the irradiation device 3. As shown in FIG. The windowpane W shown in FIG. 6 is a cross section, and the left side of the windowpane W is the inside of the vehicle, and the right side is the outside of the vehicle.

The incident light L1 is irradiated toward the window glass W from the irradiation device 3 installed outside the vehicle, and when incident on the window glass W, the incident light L1 passes through the window glass W and travels in the vehicle as transmitted light L2. In addition, the incident light L1 is reflected by the window glass W, and the reflected light L3 travels outside the vehicle.

The scattered light L4 is light scattered on the inner surface of the window glass W when the transmitted light L2 is transmitted through the window glass W. Since the window glass W is not uniform and the inner surface is dirty, the transmitted light L2 is scattered and visible to passengers inside the vehicle.

FIG. 7 is a diagram showing an example of viewing the screen displayed on the window glass W from inside the vehicle. The irradiation device 3 displays an operation screen P, which is information prompting the user to give an instruction, on the window glass W using scattered light. The operation screen P is a screen in which three buttons are displayed under the character string "Please select". In the example shown in FIG. 7, the user points with the index finger of the hand H, which is the indicator, to a button on which the letter "C" is drawn.

FIG. 8 is a diagram showing an example of the window glass W photographed from outside the vehicle. When the user performs the operation shown in FIG. 7, the photographing device 4 photographs the window glass W from outside the vehicle. At this time, the photographing device 4 photographs the operation screen P projected on the window glass W using the reflected light L3, which is the regular reflected light shown in FIG. 6, and the diffusely reflected light (not shown in FIG. 6) for the incident light L1. do. Since the operation screen P describes character strings that can be read by a user on board the vehicle, these character strings are reversed when the operation screen P is photographed from outside the vehicle.

In the image captured by the imaging device 4, there is a portion where the operation screen P and the index finger of the user's hand H overlap. The control unit 11 of the instruction identification device 1 analyzes the image captured by the imaging device 4 by referring to the screen DB 121, and identifies the region of the operation screen P including the position where the index finger of the hand H overlaps. After specifying the operation information associated with the specified area, the control unit 11 refers to the instruction DB 122 to specify the operation information string from the history of the previously specified operation information. specifies the instruction information and the transition screen ID associated with .

When the image shown in FIG. 8 is captured, the control unit 11 detects that the user presses neither the button Ba with the letter "A" nor the button Bb with the letter "B", but with the letter "C". It specifies that an instruction to select the button Bc on which characters are drawn is given.

When the user's instruction for the operation screen is specified (step S103), the control unit 11 causes the irradiation device 3 to emit light for displaying information (confirmation screen) prompting the user to confirm the instruction according to the content of the specified instruction. are irradiated (step S104). As a result, the operation screen P displayed on the window glass W (see FIGS. 7 and 8) is switched to the confirmation screen Px.

FIG. 9 is a diagram showing an example of the confirmation screen Px. As shown in FIG. 9, the confirmation screen Px includes a message area Mx consisting of a character string "C" and a character string "has been selected." It includes a button Bk on which the character string "Approve" is drawn and a button Bx on which the character string "Cancel" is drawn.

When the confirmation screen Px shown in FIG. The control unit 11 acquires the image data from the photographing device 4 and identifies an instruction given by the vehicle user to the confirmation screen Px (step S105).

The control unit 11 analyzes the content of the specified instruction and determines whether or not the user presses the button Bx on which the character string "cancel" is drawn (step S106). When determining that the button Bx has been pressed (step S106; YES), the control unit 11 returns the process to step S102. As a result, the content of the specified instruction is reset, and the operation screen P is displayed again.

On the other hand, if it is determined that the button Bx has not been pressed (step S106; NO), the control unit 11 analyzes the content of the specified instruction and confirms that the button Bk on which the character string "Approve" is drawn is the user's is pressed (step S107). When determining that the button Bk has not been pressed (step S107; NO), the control unit 11 causes the process to proceed to step S105.
back to As a result, the display of the confirmation screen Px is maintained, and the user's instruction to the confirmation screen Px is specified continuously.

On the other hand, when determining that the button Bk has been pressed (step S107; YES), the control unit 11 causes the output device 8 to output the content of the user's instruction to the operation screen P specified in step S103 (step S108). ) and terminate the process.
In step S105, if neither the button Bx nor the button Bk is pressed and the user's instruction is not specified even after a predetermined time has passed, the control unit 11 prompts the user to re-input. good too. In this case, the control unit 11 may cause the irradiation device 3 to emit light that causes the window glass W to display a message prompting re-input, for example, "Time is over. Please re-input." Then, for example, the control unit 11 may return the process to step S102 after the message prompting the re-input is displayed for a predetermined period of time. With this processing, the instruction specifying system 9 can prevent the lapse of a predetermined time even in a so-called hang-up state in which neither the button Bx nor the button Bk accepts the user's operation in step S105, for example. It is possible to get out of this state.

As described above, the instruction specifying device 1 causes the irradiation device 3 to irradiate light to display the operation screen P on the window glass, and the operation screen P and an indicator such as the user's hand H are connected to the photographing device 4. The user's instruction is specified by having the device capture a photograph and analyzing the photographed image. Therefore, in the instruction identification system 9, the instruction identification device 1 identifies an instruction indicating the intention of the user who has boarded the vehicle with the window glass closed without installing any special equipment inside the vehicle.

The configurations, shapes, sizes, and arrangement relationships described in the embodiments are merely schematic representations to the extent that the present invention can be understood and implemented. Therefore, the present invention is not limited to the described embodiments, but can be modified in various forms without departing from the scope of the technical idea indicated in the claims.

<Modification>
The above is the description of the embodiment, but the content of this embodiment can be modified as follows. Also, the following modifications may be combined.

<Modification 1>
In the above-described embodiment, the operation screen P is displayed by scattering light when the light emitted by the irradiation device 3 is transmitted through the window glass W, but the method of displaying the operation screen is not limited to this. For example, the operation screen P may be displayed by reflected light of light transmitted through the window glass W. FIG.

10A and 10B are diagrams for explaining other paths of the light irradiated toward the window glass W by the irradiation device 3. FIG. In the path shown in FIG. 10, the irradiated light (incident light L1) passes through the window glass W and becomes transmitted light L2, and the transmitted light L2 is further reflected on the surface of the object J to form scattered light L5 and reflected light L6. becomes.

In this case, on the windowpane W, the operation screen P (see FIGS. 7 and 8), which is information prompting the user to give instructions, includes scattered light L5 generated by reflection of the transmitted light L2 on the surface of the object J and reflected light It is displayed on the surface of the object J by at least one of L6. The photographing device 4 may photograph the operation screen P displayed on the surface of the object J and the user's indicator held thereon. At this time, the photographing device 4 may photograph the scattered light L5 and/or the reflected light L6 with the light transmitted through the windowpane of the vehicle C. FIG. This windowpane may be the windowpane W through which the incident light L1 is incident, or may be another windowpane.

<Modification 2>
Here, the object J in Modification 1 may be a windowpane different from the windowpane irradiated with light by the irradiation device 3 . For example, the object J may be a windowpane provided on the opposite side of the windowpane irradiated with light by the irradiation device 3 . In this case, when the irradiation device 3 irradiates light toward the window glass provided on the right front seat, for example, this light passes through the right window glass and traverses the interior of the vehicle. The window glass provided in the front seat is illuminated. The user can recognize the operation screen from the light reflected by the left window glass.

<Modification 3>
In the above-described embodiment, the control unit 11 identifies the screen displayed on the window glass and the position of the user's pointer superimposed on the screen from the image indicated by the image data obtained from the photographing device 4, and instructs the user. When the position of the body satisfies a predetermined condition, an area on the screen corresponding to that position is specified and an instruction is specified according to the area, but the mode of specifying the area is not limited to this. The control unit 11 may specify the user's instruction by specifying an area based on the temporal change in the position of the pointer moved by the user, which is included in the image described above. Further, the positions of the screen and the pointer specified by the control unit 11 are not limited to positions in the three-dimensional space, and may be positions in the two-dimensional space that do not include information such as depth direction information.

FIG. 11 is a diagram for explaining changes over time in the position of the pointer moved by the user. In the graph shown in FIG. 11, the upper graph shows the change over time of the X coordinate indicating the horizontal position of the pointer, and the lower graph shows the change over time of the Y coordinate indicating the vertical position of the pointer. be. In the period from time t0 to time t1 shown in FIG. 11, the X-coordinate and Y-coordinate of the pointer move beyond a predetermined range per unit time. In this state, the control unit 11 interprets that the pointer is not pointing to any area of the operation screen P, and continues monitoring the state of the pointer.

On the other hand, after time t1, the moving amount per unit time of the X-coordinate and Y-coordinate of the pointer does not exceed the determined range. When this state continues from time t1 to time t2, which is the time when the predetermined time has passed, the control unit 11 interprets that any region of the operation screen P has been pointed to by the pointer, and the region is displayed. identifies the user's instruction associated with that area.

<Modification 4>
In addition, the control unit 11 identifies an area pointed by the user's pointer from the shape of the pointer moved by the user, which is included in the image indicated by the image data acquired from the photographing device 4, and associates the area pointed by the user's pointer with this area. User instructions may be specified.

FIG. 12 is a diagram for explaining changes in the shape of the pointer. A finger H1, such as an index finger, of the user's hand H is a part used to specify which region of the operation screen P the user is pointing to. The finger H1 of the hand H is specified by the control unit 11 by matching the contour detected by the edge detection with the contour and a pre-stored contour pattern. When the finger H1 is not touching the window glass W, the finger H1 maintains its original curved surface as shown in FIG. 12(a).

On the other hand, when the finger H1 touches the window glass W and pushes the window glass W, a deformed portion H2 deformed along the window glass W is generated on the surface of the finger H1 (see FIG. 12(b)). Since this deformed portion H2 is along the windowpane W, it is flat. Since the deformed portion H2 has a different shape from other portions of the finger H1, in the image captured by the imaging device 4, for example, the boundary line and the inside of the deformed portion H2 have a different color and shape from those of the other portions. Appears in When the shape of the deformation portion H2 in this image is recognized by image processing, the control section 11 may specify the area pointed by the user's indicator and specify the user's instruction.

<Modification 5>
The designation identification system 9 may comprise a measurement device 5, as indicated by dashed lines in FIGS. The measuring device 5 irradiates the windowpane W with electromagnetic waves such as infrared rays, captures the reflected wave transmitted through the windowpane W, and is reflected by an object present in the vehicle. Based on the reflected wave, the measuring device 5 to this object. The connecting part 13 connects with the measuring device 5 .

For example, the control unit 11 of the instruction specifying device 1 calculates the ratio of the area occupied by the operation screen P displayed on the window glass W by the irradiation device 3 in the image captured by the imaging device 4. The distance from the photographing device 4 to the window glass W is estimated based on the ratio. Then, based on the estimated distance from the photographing device 4 to the windowpane W and the distance measured by the measuring device 5, the control unit 11 determines the positional relationship between the object that returned the reflected wave and the windowpane W. Identify. Then, when it is determined that the object and the window glass W are in contact with each other, the user's instruction may be specified by specifying the area pointed by the user's indicator.

<Modification 6>
The program executed by the control unit 11 of the instruction specifying device 1 is stored in a computer-readable record such as a magnetic recording medium such as a magnetic tape or a magnetic disk, an optical recording medium such as an optical disk, a magneto-optical recording medium, or a semiconductor memory. It may be provided stored on a medium. It is also possible to download this program via a communication line such as the Internet. Various devices other than the CPU may be applied as the control means exemplified by the control unit 11, and for example, a dedicated processor may be used.

DESCRIPTION OF SYMBOLS 1... Instruction specific apparatus 11... Control part 12... Storage part 121... Screen DB 1211... Screen list 1212... Area list 122... Instruction DB 13... Connection part 2... Detection apparatus 3... Irradiation apparatus , 4... Imaging device, 5... Measuring device, 8... Output device, 9... Instruction specifying system.

Claims (5)

A windowpane of a vehicle is irradiated with light for displaying information prompting a user to give an instruction, the information is displayed by reflected light of the light transmitted through the windowpane, the windowpane is photographed, and the photographed image is obtained from the photographed image. An instruction identification system that identifies user instructions. 2. The pointing and identifying system according to claim 1 , wherein said information is displayed by reflecting light transmitted through said windowpane from a windowpane different from said windowpane. 3. The instruction identification system according to claim 1 , wherein said instruction is identified from a change over time in the position of said pointer moved by said user included in said image. 4. The instruction identifying system according to any one of claims 1 to 3 , wherein the instruction is identified from the shape of the pointer moved by the user, which is included in the image. 5. The instruction specifying system according to any one of claims 1 to 4 , wherein said instruction is specified based on a reflected wave of an electromagnetic wave irradiated toward said window glass.

Images