JP2021179733A - Mounting device, processing method, and program - Google Patents

Abstract

To provide a portable device capable of efficiently sharing information when sharing desired information with other users at arbitrary position.SOLUTION: A mounting device 1 comprises projecting means 91 for projecting a projection image on a projection surface in a forward direction of a user, and at least two photographing means 92 for photographing the projection image at a different position.EFFECT: This directly projects the projection image on a projection surface in front of the user, so the user himself/herself can widely view the projection image without obstructing vision when confirming the projection image, providing the user with a suitable condition. Also, the projection image is projected on the projection surface, so the user can share information appearing on the projection image with other users.SELECTED DRAWING: Figure 9

Description

The present invention relates to a mounting device, a processing method, and a program.

In order to easily share information with others, mobile terminal users display information on their own mobile terminals such as smartphones and show the display displaying the information to others. Allows you to share the desired information with others.

A technique related to this technique is disclosed in Patent Document 1. Paragraph 0076 of Patent Document 1 discloses that the head-mounted projector includes a small projector and a camera. The patent document 1 discloses that the projected light is emitted to the lens surface of the spectacles.

Japanese Unexamined Patent Publication No. 2017-21861

There is a demand for a mobile device that efficiently shares information when sharing desired information with another person at any place without having a display.

Therefore, an object of the present invention is to provide a mounting device, a processing method, and a program that solve the above-mentioned problems.

According to the first aspect of the present invention, the mounting device includes a projection means for projecting a projected image on a projection surface in the front direction of the user, and at least two photographing means for capturing the projected image at different positions. ..

According to the second aspect of the present invention, in the processing method, the projection means of the mounting device projects the projected image on the projection surface in the front direction of the user, and at least two photographing means of the mounting device differ from the projected image. Take a picture at the position.

According to the third aspect of the present invention, the program controls the projection of the projection means for projecting the projected image onto the projection surface in the front direction of the user by using the information processing device provided in the mounting device. It functions as a shooting control means for controlling the shooting of at least two shooting means that shoot the projected image at different positions.

According to the present invention, it is possible to provide a portable device for efficiently sharing information when sharing desired information with another person at any place without providing a display.

It is the first figure which shows the appearance of the mounting apparatus by one Embodiment of this invention. It is a second figure which shows the appearance of the mounting apparatus by one Embodiment of this invention. It is a hardware function block diagram of the mounting apparatus according to one Embodiment of this invention. It is a figure which shows the hardware configuration of the information processing apparatus provided in the attachment apparatus by one Embodiment of this invention. It is a functional block diagram of the information processing unit by one Embodiment of this invention. It is the first figure which shows the use example of the mounting apparatus by one Embodiment of this invention. It is a second figure which shows the use example of the mounting apparatus by one Embodiment of this invention. It is a figure which shows the processing flow of the information processing apparatus by one Embodiment of this invention. It is a figure which shows the minimum structure of the mounting apparatus by one Embodiment of this invention. It is a figure which shows the processing flow by the mounting apparatus of the minimum composition by one Embodiment of this invention.

Hereinafter, the mounting device according to the embodiment of the present invention will be described with reference to the drawings.
FIG. 1 is the first diagram showing the appearance of the mounting device according to the same embodiment.
The wearing device 1 shown in FIG. 1 has a spectacle shape, and is provided with a projector 11 which is one aspect of the projection means between the lenses of the spectacle-shaped frame of the wearing device 1 and is provided on the spectacle-shaped frame. A camera 12 is provided at both ends of each of the lenses, which is one aspect of the photographing means. The projector 11 and the camera 12 may be provided at a position of a spectacle-shaped frame other than that shown in FIG.

FIG. 2 is a second view showing the appearance of the mounting device according to the same embodiment.
The mounting device 1 shown in FIG. 2 has a goggle shape, and is provided with a projector 11 which is an aspect of the projection means in the center of the goggle-shaped frame of the mounting device 1, and a lens provided in the goggle-shaped frame. Cameras 12, which are one aspect of photographing means, are provided at the left and right ends, respectively. The projector 11 and the camera 12 may be provided at a position of a goggle-shaped frame other than that shown in FIG.

The projector 11 projects a projected image on a projection surface in the front direction of the user who wears the mounting device 1. The projector 11 may be provided at a desired position of the mounting device 1 so that the projected image can be projected on the projection surface located in front of the user who wears the mounting device 1. Further, the camera 12 is at least two photographing means for photographing the projected image projected on the projection surface by the projector 11 at different positions. The camera 12 may be provided at a desired position of the mounting device 1 so that the entire projected image can be captured.

The wearing device 1 shown in FIGS. 1 and 2 may incorporate or mount the function of the information processing device 2 in a spectacle-shaped frame or a goggle-shaped frame. The function of the information processing device 2 is connected to the projector 11 and the camera 12 via a signal line.

FIG. 3 is a hardware function block diagram of the mounting device.
As shown in FIG. 3, in the mounting device 1, the projector 11 and the camera 12 are connected to the control unit 10, and the control unit 10 is connected to the information processing unit 13. The information processing unit 13 may control hardware such as the projector 11 and the camera 12 via the control unit 10. The information processing device 2 provided with the control unit 10 and the information processing unit 13 may be provided in a housing different from the mounting device 1. In addition, the mounting device 1 may include a hardware configuration such as a microphone, a speaker, an input device for the user to operate for inputting information, and a wireless device for transmitting and receiving wireless signals to and from other devices.

FIG. 4 is a diagram showing a hardware configuration of an information processing device provided in the mounting device.
As shown in FIG. 4, the information processing device 2 includes hardware such as a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, a database 104, and a communication module 105. It is a computer.

FIG. 5 is a functional block diagram of the information processing unit.
The information processing device 2 includes a projection control unit 21, a shooting control unit 22, a distortion correction unit 23, a color correction unit 24, a position detection unit 25, an application processing unit 26, a payment processing unit 27, and the like when the CPU 101 executes a program. Demonstrate the function of.

The projection control unit 21 controls the projection of the projector that projects the projected image onto the projection surface in the front direction of the user.
The shooting control unit 22 controls the shooting of at least two cameras 12 that shoot the projected images at different positions.

The distortion correction unit 23 acquires captured images obtained by capturing the projected images, detects the projected surface based on the distance to the corresponding point in the captured images, and projects the projected surface onto the projected surface based on the information representing the projected surface. Corrects the distortion of the projected image.
The color correction unit 24 corrects the amount of light used by the projector 11 for projecting the projected image based on the captured image acquired from any of the cameras 12. As a result, the color correction unit 24 changes the brightness of the projected image. The color correction unit 24 may control the projector 11 so as to change the color tone of the projected image based on the captured image.

The position detection unit 25 acquires captured images of captured images from each of the cameras 12, and determines the position of the projected image pointed to by the pointer on the projection surface based on the difference in the positions of pointers such as fingers reflected in the captured images. Identify.
The application processing unit 26 performs processing based on the input information according to the position of the projected image pointed to by the pointer.
The payment processing unit 27 projects the payment code as a projection image on the projection surface.

FIG. 6 is a first diagram showing a usage example of the mounting device.
As shown in FIG. 6, the user wears the spectacle-shaped wearing device 1 on the face. Further, FIG. 6 shows an image when the projected image is projected on the top plate of the desk, which is one aspect of the projection surface. The projector 11 of the mounting device 1 irradiates the projected image obliquely with respect to the desktop plane, but the information processing device 2 corrects the distortion so that the projected image is correctly projected in a rectangular shape. Further, the information processing apparatus 2 projects the projected image so that it becomes clear by increasing the brightness of the light in the projection of the projector 11 even if the desktop plane has a dark color tone.

FIG. 7 is a second diagram showing a usage example of the mounting device.
Similarly to FIG. 6, the user wears the spectacle-shaped wearing device 1 on the face. Further, FIG. 7 shows an image when a projected image is projected on a wall, which is one aspect of the projection surface. The information processing apparatus 2 corrects the irradiation angle with the projector 11 according to the distance to the wall surface and the projection angle so that the projected image is projected onto a rectangle of an appropriate size. Further, even if the wall surface has a dark color tone, the projected image is projected so as to be clear by increasing the brightness of the light in the projection of the projector 11.

FIG. 8 is a diagram showing a processing flow of the information processing apparatus.
In the information processing device 2 provided in the mounting device 1, the projection control unit 21 detects the start of projection of the projected image triggered by some instruction (step S101). The projection control unit 21 analyzes, for example, whether the sound obtained from the microphone is a sound indicating the start of projection, and if it is a sound indicating the start of projection, starts projecting the projected image. Alternatively, the projection control unit 21 detects the start of projection based on the signal from another device received via the wireless device, and when the start of projection is detected, the projection of the projected image is started. Alternatively, the projection control unit 21 analyzes the captured image acquired from the camera 12 activated in advance, and when it detects the start of projection, starts projecting the projected image. Alternatively, the projection control unit 21 detects the start of projection based on the instructions from the application processing unit 26 and the payment processing unit 27, and starts the projection of the projected image.

The projection control unit 21 acquires a projection image specified by an instruction from a storage unit such as a database 104 based on the detection of the start of projection. The projection control unit 21 outputs the acquired projection image to the projector 11 and outputs a projection instruction to the projector 11. The projector 11 inputs a projected image and a projection instruction from the projection control unit 21 and projects the projected image (step S102). The user adjusts the orientation of the face and the orientation of the body so that the projection direction of the mounted projector 11 faces the projection surface. Then, the projected image is projected on the projection surface.

The shooting control unit 22 outputs a shooting instruction to each camera 12 based on the above-mentioned instruction regarding the start of projection of the projector 11 or the above-mentioned projection start prior to that. Then, the camera 12 starts shooting. The camera 12 generates a captured image and outputs it to the information processing apparatus 2 (step S103). The control unit 10 acquires captured images from each camera 12. The control unit 10 associates the captured image with the identification information of the camera 12 that outputs the captured image, and records the captured image in a storage unit such as a RAM 103. The information processing apparatus 2 stores the captured image so that which of the two cameras 12 outputs the captured image can be known, and the captured image captured at the same time can be known by associating the time with each other.

The distortion correction unit 23 detects the instruction to start projection. Then, the distortion correction unit 23 reads a pair of captured images output from the two cameras 12 at almost the same time. The distortion correction unit 23 compares each captured image and detects a difference in position of a certain target point in each captured image in the corresponding captured image. The distortion correction unit 23 measures the distance from the origin to the target point in the three-dimensional virtual space based on a certain point in the mounting device 1 by triangulation based on the difference. The distortion correction unit 23 measures the distances to a plurality of set target points in the captured image in the same manner.

The distortion correction unit 23 calculates information representing the projection surface such as the normal direction of the projection surface in the virtual space and the formula indicating the plane of the projection surface based on the distance of each target point. Further, the distortion correction unit 23 calculates the area of the range of the projected image in the captured image. For example, the distortion correction unit 23 may specify a range of the projected image based on the brightness of the captured image, and calculate the area based on the range. The distortion correction unit 23 may calculate information indicating the shape of the projected image based on the range of the projected image. The distortion correction unit 23 determines whether or not the shape of the projected image reflected on the projected surface is projected by a predetermined shape based on the information representing the projected surface and the shape of the projected image. When the distortion correction unit 23 determines that the shape of the projected image reflected on the projection surface is not a predetermined shape, the distortion correction unit 23 corrects the shape of the projected image so that the shape of the projected image reflected on the projection surface becomes a predetermined shape. Is calculated (step S104). The correction parameters that correct the shape of the projected image are the enlargement ratio of the projected image, the length of the sides when the projected image is rectangular, and the enlargement ratio of the sides according to the change in the length of each side. good. The predetermined shape may be, for example, a rectangle (rectangle or square) in which the length of the side and the aspect ratio are determined. The distortion correction unit 23 may also calculate a correction parameter indicating the enlargement ratio of the projected image so that the area of the projected image becomes a predetermined size. The distortion correction unit 23 outputs a distortion correction instruction including correction parameters to the projector 11 (step S105). The projector 11 corrects the projected image using the correction parameters.

The color correction unit 24 also detects an instruction to start projection. Then, the color correction unit 24 reads at least one of the captured images output from the two cameras 12. The color correction unit 24 detects the brightness of the captured image, and calculates the light amount parameter of the light that the projector 11 irradiates the projected image so that the brightness becomes a predetermined threshold value (step S106). The color correction unit 24 outputs the light amount parameter to the projector 11 (step S107). If the brightness of the captured image is less than a predetermined threshold value, the color correction unit 24 outputs a light amount parameter indicating an increase value of the light amount according to the difference between the brightness of the captured image and the brightness indicated by the predetermined threshold value to the projector 11. .. If the brightness of the captured image is equal to or higher than a predetermined threshold value, the color correction unit 24 outputs a light amount parameter indicating a decrease value of the light amount according to the difference between the brightness of the captured image and the brightness indicated by the predetermined threshold value to the projector 11. .. Further, the color correction unit 24 may output a color tone correction value to the projector 11 so that the color information becomes a predetermined color based on the color information of each pixel of the captured image. As a result, the projector 11 projects the projected image of the amount of light or the color tone according to the instruction on the projection surface. The projector 11 changes the amount of light when projecting a projected image based on the amount of light parameter.

By the above processing, the mounting device 1 can automatically project the projected image projected on the projection surface so as to have a predetermined shape, a predetermined brightness, a predetermined color, and a predetermined size.

The position detection unit 25 also detects an instruction to start projection. Then, the position detection unit 25 reads the captured images output from the two cameras 12. The position detection unit 25 determines whether or not the shape of a predetermined pointer is captured in the projected image captured in each captured image. For example, the pointer may be a human finger. Alternatively, the pointer may be a pointer or the like. When the position detection unit 25 detects a pointer in the projected image in the captured image, the position detection unit 25 calculates the coordinates in the image corresponding to the tip of the pointer from each captured image.

The position detection unit 25 compares the coordinates of the tip of the pointer in the two captured images acquired from each camera 12 at approximately the same time, and refers to a certain point in the mounting device 1 by triangulation based on the difference in the coordinates of the pointer. The distance from the origin to the tip of the pointer in a three-dimensional virtual space is measured. In the position detection unit 25, the difference between the distance from the origin to the tip of the pointer in the virtual space and the distance from the origin to the projection plane corresponding to the position of the tip of the pointer is within a predetermined value, and the distances are substantially the same. Determine if it can be determined to be present. When the position detection unit 25 determines that the distance from the origin to the tip of the pointer in the virtual space and the distance from the origin to the projection surface corresponding to the position of the tip of the pointer match, the projected image in the captured image. The information of the position corresponding to the tip of the pointer of is output to the application processing unit 26 (step S108). The application processing unit 26 identifies a corresponding position in the projected image and performs processing according to the position. For example, when the button icon is displayed at that position, the application processing unit 26 detects that the user has pressed the button.

The mounting device 1 may project various projected images onto the projection surface. For example, in the information processing device 2 of the mounting device 1, the payment processing unit 27 outputs a two-dimensional code (payment code) used for the payment processing to the projection control unit 21 as a projection image based on a predetermined payment processing instruction. .. The projection control unit 21 controls the projector 11 to project the two-dimensional code as a projected image. It is assumed that the projection surface of the two-dimensional code is, for example, the top plate of the table of the cash register counter. The camera 12 captures the two-dimensional code. The distortion correction unit 23 corrects the distortion of the two-dimensional code as described above. The color correction unit 24 corrects the brightness and color information of the two-dimensional code. The two-dimensional code includes the payment amount and the user identifier. Alternatively, the two-dimensional code may simply include the identification of the user. Alternatively, the two-dimensional code may include an identifier of another payment agent, an identifier of the wearing device 1, and the like. The two-dimensional code may be code information for payment that includes at least one or a plurality of such information.

The cashier counter clerk reads the two-dimensional code projected on the top plate of the cashier counter stand with a reading device. The reading device outputs a two-dimensional code to the cash register device. As a result, the cash register device performs a known payment process using a two-dimensional code.

According to the process of the mounting device 1 described above, the projector 11 directly projects the projected image onto the projection surface in front of the user, so that when the user himself / herself confirms the projected image, the projected image can be displayed without obstructing the field of view. It looks wide and is suitable for users.
Moreover, since the projected image is projected on the projection surface, it is possible to share the information projected by the projected image with others.

Further, for a mobile terminal such as a general smartphone, it does not take time to take it out, and it is possible to share the information projected by the projected image with another person with both hands open.

In addition, if the projection surface is suitable for projecting the projected image, the projected image can be projected onto the desk, wall surface, body surface such as a car, and the body such as the palm or arm, and can be used at any time. Therefore, it is possible to change the size of the screen according to the area and distance of the projection surface.

Further, according to the above-mentioned processing of the mounting device 1, it is also suitable for code payment using a two-dimensional code such as a QR (Quick Response) code (registered trademark) or a bar code. Here, two methods are known for code payment, a user scan method and a store scan method. In the user scan method, the code posted by the store is read by the user's terminal, the user inputs the payment amount to the terminal held by the user, and the amount displayed on the terminal is displayed. You need to ask the clerk to confirm. However, in the virtual image projection method and the retinal projection method, which exist as the information display method used in the head-mounted display, the code information cannot be seen by the clerk, so the user can enter the payment amount into the terminal held by the user. I can't get confirmation and I can't make the payment. In addition, in the general smartphone payment method, the user inputs the usage amount into the smartphone held by the user, changes the smartphone and hands it to the clerk, and after the clerk confirms the payment amount displayed on the user's smartphone. , It is necessary to have the smartphone returned to the user, and the user has to change the smartphone again and perform the final payment operation using his / her own smartphone, which is a complicated exchange.

In the store scan method, a code in which payment information displayed on a terminal on the user side is embedded is read by a stationary or handheld code reader on the store side. However, in the above-mentioned virtual image projection method and retinal projection method, the payment cannot be completed because the image in which the code in which the payment information is embedded cannot be read from the store side. In addition, if the code reader on the store side is a stationary code reader with the reading unit facing up, the user operates the smartphone to insert the code with the payment information embedded in the payment method using a general smartphone. Display, switch the smartphone and turn the front and back, hold the smartphone over the reading part of the code reader on the store side, read the code, then switch the smartphone again and turn the front and back, and close the code display of payment information It is necessary to perform complicated handling.

However, according to the above-mentioned mounting device 1, the code reader on the store side directly displays a projected image for the projector 11 to display the code information for the stationary code reader with the reading unit facing upward. Since it is possible to irradiate the reading unit of the code reader and then proceed from inputting the payment amount to confirmation of the clerk and final payment processing, complicated terminal exchange is not required. For a handheld code reader that allows the code reader on the store side to point the reading unit downward, the projected image for the projector 11 to display the code information is displayed on the top plate of the counter. By irradiating, it is possible to proceed from reading the code information, inputting the payment amount, confirming the clerk, and final payment processing, so that complicated terminal exchange is not required.

FIG. 9 is a diagram showing the minimum configuration of the mounting device.
FIG. 10 is a diagram showing a processing flow by the mounting device having the minimum configuration.
The mounting device 1 may include at least a projection means 91 and at least two photographing means 92.
The projection means 91 projects a projected image on a projection surface in the front direction of the user (step S901).
The photographing means 92 photographs the projected image at different positions (step S902).

A computer system is provided inside the information processing device 2 provided in the above-mentioned mounting device 1. The process of each process described above is stored in a computer-readable recording medium in the form of a program, and the process is performed by the computer reading and executing this program. Here, the computer-readable recording medium refers to a magnetic disk, a magneto-optical disk, a CD-ROM, a DVD-ROM, a semiconductor memory, or the like. Further, this computer program may be distributed to a computer via a communication line, and the computer receiving the distribution may execute the program.

Further, the above program may be for realizing a part of the above-mentioned functions. Further, a so-called difference file (difference program) may be used, which can realize the above-mentioned function in combination with a program already recorded in the computer system.

1 ... Mounting device 2 ... Information processing device 10 ... Control unit 11 ... Projector 12 ... Camera 13 ... Information processing unit 21 ... Projection control unit 22 ... Shooting control unit 23 ... Distortion correction unit 24 ... Color correction unit 25 ... Position detection unit 26 ... Application processing unit 27 ... Payment processing unit

According to the first aspect of the present invention, the wearing device is in the shape of spectacles or goggles, is provided in the center of the spectacles or goggles, and projects a projected image onto a projection plane in the front direction of the user. The means, at least two photographing means for photographing the projected image at different positions of the left and right ends of the eyeglass shape or the goggles shape, and the photographed image obtained by capturing the projected image are acquired from the photographing means, respectively, and they are obtained. The projection plane is detected based on the distance to the corresponding point in the captured image, and the distortion correction means for correcting the distortion of the projection image projected on the projection plane based on the information representing the projection plane is provided.

According to the second aspect of the present invention, in the processing method, the wearing device is in the shape of spectacles or goggles, and the projection means of the wearing device is provided in the center of the spectacles or the goggles, and is directed toward the front of the user. projecting the projected image on the projection surface, at least two imaging means are different right and left end portions of the spectacle-shaped or the goggles shape position of the mounting device, photographs the projection image, the distortion correcting means of the mounting device , The captured image obtained by capturing the projected image is acquired from the imaging means, the projection surface is detected based on the distance to the corresponding point in the captured image, and the projection surface is based on the information representing the projection surface. The distortion of the projected image projected on the image is corrected .

According to a third aspect of the present invention, the program is provided with an information processing device provided in a spectacle-shaped or goggle-shaped wearing device in the center of the spectacle-shaped or goggles-shaped, and is directed toward the user's front. A projection means for projecting a projected image on a projection surface, at least two photographing means for photographing the projected image at different positions on the left and right ends of the eyeglass shape or the goggles shape, and the photographed image obtained by capturing the projected image. Distortion that is acquired from each means, detects the projection plane based on the distance to the corresponding point in the captured images, and corrects the distortion of the projection image projected on the projection plane based on the information representing the projection plane. It functions as a correction means.

Claims (10)

A projection means that projects a projected image onto the projection plane in the forward direction of the user, and
At least two shooting means for shooting the projected image at different positions, and
A mounting device equipped with. Formed in the shape of glasses or goggles,
The projection means is provided in the center of the spectacle shape or the goggle shape.
The mounting device according to claim 1, wherein the photographing means is provided at the left and right ends of the eyeglass shape or the goggles shape. The projection means is provided at a position of the own device capable of projecting a projected image on the projection surface located in front of the user who wears the own device.
The mounting device according to claim 1 or 2, wherein the photographing means is provided at a position of the own device capable of photographing the entire projected image. The captured images obtained by capturing the projected images are acquired from the photographing means, the projected surface is detected based on the distance to the corresponding point in the captured images, and the projected surface is formed based on the information representing the projected surface. Distortion correction means for correcting the distortion of the projected image to be projected, and
The mounting device according to any one of claims 1 to 3. A luminance correction means for correcting the brightness of the captured image based on the captured image obtained from the projected image from any of the photographing means.
The mounting device according to any one of claims 1 to 4. A position detecting means for acquiring a captured image obtained by capturing the projected image from the photographing means and specifying the position of the projected image pointed to by the pointer on the projection surface based on the difference of the pointers transferred to the captured images.
The mounting device according to any one of claims 1 to 5. An application processing means that performs processing based on input information according to the position of the projected image pointed to by the pointer, and
The mounting device according to claim 6. A payment processing means for projecting a payment code as a projection image onto the projection surface,
The mounting device according to any one of claims 1 to 7. The projection means of the mounting device projects the projected image on the projection plane in the front direction of the user, and the projection image is projected.
A processing method in which at least two photographing means of the mounting device capture the projected image at different positions. The information processing device provided in the mounting device,
A projection control means that controls the projection of the projection means that projects the projected image onto the projection surface in the front direction of the user.
A shooting control means for controlling the shooting of at least two shooting means for shooting the projected image at different positions, and a shooting control means.
A program that functions as.

Images