JP2014064248A - Image projection device and image projection method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To project, on a person capable of moving, an image corresponding to his/her motion.SOLUTION: A control unit 11 specifies a feature point B in an image in which a user's face is captured. Next, the control unit 11 obtains a correspondence relationship between the specified feature point B and a feature point A stored in a storage unit 12, and calculates a transform matrix for transforming projection image data. Then, the control unit 11 reads the projection image data from the storage unit 12, transforms the projection image data on the basis of the transform matrix, and projects an image from a wide-angle projector 18.

Description

  The present invention relates to an image projection apparatus used by being worn by a user, and an image projection method for the apparatus.

  As an image projection device used in a state of being mounted on a user's head, for example, a so-called head mounted display is known. Patent Document 1 discloses that the imaging direction of the imaging unit is made to follow the direction of the user's line of sight. Patent Document 2 discloses that an output image from a projector follows a user's line of sight.

JP 2000-59666 A JP 10-301055 A

  An image projected from a projection unit such as a projector is generally an image of a material used in a meeting attended by a plurality of people, for example. For example, an image corresponding to the movement of a person who can move is displayed. By projecting, it is conceivable to provide a service with an unprecedented interest.

  Therefore, an object of the present invention is to provide a mechanism capable of projecting an image corresponding to a movement of a human who can move.

  An image projection apparatus according to an aspect of the present invention includes an imaging unit that captures an image of a user wearing the device, a projection unit that projects an image toward the user wearing the device, and at least an imaging range of the imaging unit. Projection control means for projecting from the projection means a projection image corresponding to the characteristics of the image captured by the imaging means within the image projection range as a part of the image projection range.

  The image projection device may be mounted on the user's head, and the image projection range may include at least a part of the user's face.

  Prior to projecting an image by the projection unit, the projection control unit identifies a feature point in the user's image captured by the imaging unit, and based on the identified feature point, an image serving as a reference for the projection image is obtained. When deformed projection image data is generated and stored in association with the identified feature point, when the image is projected by the projection unit, the feature point in the user image captured by the imaging unit is identified. Then, the correspondence relationship between the identified feature point and the stored feature point is obtained, the stored projection image data is converted based on the obtained correspondence relationship, and the projection image after the conversion An image represented by data may be projected from the projection means.

  The projection control unit stores reference projection image data serving as a reference of a projection image in association with a feature point in an image represented by the reference projection image data, and based on a user image captured by the imaging unit, Identifying the feature points in the image and each part of the user, the position and inclination of each identified part, and the feature points stored in association with the identified feature points and the reference projection image data; And converting the stored reference projection image data based on the obtained position and inclination of the part and the correspondence relationship, and converting the image represented by the converted projection image data into the image You may make it project from a projection means.

  The projection control unit may determine one of a plurality of types of projection images as a projection image to be projected on the image projection range, and cause the projection unit to project the determined projection image.

  According to another aspect of the present invention, there is provided an image projection method including: an imaging unit that captures an image of a user; and a projection unit that projects an image toward the user. The projection unit projects at least a part of the imaging range as an image projection range, and projects a projection image corresponding to the feature of the image captured by the imaging unit in the image projection range.

  ADVANTAGE OF THE INVENTION According to this invention, the image according to the motion can be projected with respect to the person who can move.

1 is a front view, a side view, a top view, and a rear view of an image projection apparatus according to an embodiment of the present invention. The perspective view which shows a mode when the user has mounted | worn the image projector. The block diagram which shows the hardware constitutions of an image projector. The block diagram which shows the function structure of an image projector. The perspective view which shows a mode when an image is projected with respect to the user with which the image projector is mounted | worn. The flowchart which shows the procedure when an image projection apparatus registers projection image data. The flowchart which shows the procedure in case an image projection apparatus projects an image on a user's face. 10 is a flowchart showing a procedure of processing executed by the image projection apparatus in Modification 1; 10 is a flowchart showing a procedure of processing executed by the image projection apparatus in Modification 2.

[Embodiment]
1A is a front view of an image projection apparatus 1 according to an embodiment of the present invention, FIG. 1B is a side view, FIG. 1C is a top view, and FIG. 1D is a rear view. It is. FIG. 2 is a perspective view showing a state when the user is wearing the image projection apparatus 1. The image projecting device 1 is an image projecting device that is used while being worn on the head of the user 2, for example, and functions as a communication device capable of calling and data communication. The image projection apparatus 1 is mounted so as to sandwich the head of the user 2. That is, the image projection apparatus 1 is mounted on the head of the user 2 with a part called a nose part in contact with the nose 2a of the user 2 and a part called an arm part in contact with both ears 2b of the user 2. The

  A display unit 16 for displaying an image is provided on the front side of the image projector 1. The outer edge of the display unit 16 is covered with a component called a rim portion 16a. The display unit 16 and the rim unit 16a are positioned in front of the user 2 (shown by solid lines in FIG. 1C) and in a state of being separated from the front and retracted (shown by dotted lines in FIG. 1C). It is possible to displace between. The display unit 16 is made of a material that transmits light so that the user 2 can visually recognize the front of the user 2 through the display unit 16 even when the display unit 16 is positioned in front of the user 2. It has become.

  One wide-angle camera 17 and one wide-angle projector 18 are provided inside the rim portion 16a corresponding to each eye of the user 2 (the side facing the user when the user 2 wears the image projection device 1). . The wide-angle camera 17 is an imaging unit that images a user's face. The wide-angle projector 18 is a projection unit that projects an image toward the user's face. In this embodiment, when the user 2 wears the image projection device 1, the distance between the wide-angle camera 17 and the user 2 's face is relatively short, so that the widest possible range of the user 2's face can be captured. In addition, a wide-angle camera 17 using a wide-angle lens is employed. Similarly, a wide-angle projector 18 using a wide-angle lens is also adopted for the wide-angle projector 18 so that an image can be projected as wide as possible on the face of the user 2.

  FIG. 3 is a block diagram showing a hardware configuration of the image projection apparatus 1. As described above, the image projection device 1 functions as a communication device, and includes a control unit 11, a storage unit 12, a communication unit 13, a voice input / output unit 14, an operation unit 15, a display unit 16, A wide-angle camera 17 and a wide-angle projector 18 are provided. The control unit 11 is a computer including an arithmetic device such as a CPU (Central Processing Unit) and a storage device such as a ROM (Read Only Memory) and a RAM (Random Access Memory). The CPU controls the operation of each unit of the image projection apparatus 1 by executing a program stored in the ROM or the storage unit 12 using the RAM as a work area. The storage unit 12 is a writable nonvolatile storage means such as an EEPROM (Electronically Erasable and Programmable ROM) or an SRAM (Static Random Access Memory) equipped with a backup power source. In addition to the program executed by the control unit 11, the storage unit 12 stores various data described below. The communication unit 13 includes a wireless antenna and a communication circuit, and performs wireless communication with a base station of a mobile communication network, an access point of a wireless LAN (Loval Area Network), and the like. The voice input / output unit 14 includes a speaker, a microphone, a voice processing circuit, and the like, and performs voice input / output related to a call. The operation unit 15 includes operation elements such as keys and touch sensors, and supplies an operation signal corresponding to a user operation to the control unit 11. The control unit 11 performs processing according to the operation signal. The display unit 16 includes a liquid crystal panel and a liquid crystal drive circuit, and displays an image in response to an instruction from the control unit 11. The wide-angle camera 17 and the wide-angle projector 18 are as described above.

  FIG. 4 is a block diagram showing a functional configuration of the image projection apparatus 1. The image projecting apparatus 1 includes an image capturing unit 101, a projecting unit 102, and a projection control unit 103. The image capturing unit 101 is configured by the wide-angle camera 17, and the face of the user wearing the image projecting device 1 Image. The projection unit 102 is configured by the wide-angle projector 18 and projects an image toward the face of the user wearing the image projection apparatus 1. The projection control unit 103 is configured by the control unit 11 and the storage unit 12, and an image captured by the imaging unit 101 in the image projection range with at least a part of the imaging range of the imaging unit 101 as an image projection range. The projection unit 102 projects a projection image corresponding to the above feature. The projected image is an image that decorates the user's face. For example, when a clown face image is projected on the user's face shown in FIG. 2, the user's face has the appearance shown in FIG. Become. This clown face image moves in accordance with the movement of the user's face, that is, the facial expression.

  More specific processing is as follows. Prior to the projection of the image by the projection unit 102, the projection control unit 103 identifies a feature point in the user's image captured by the imaging unit 101, and based on the identified feature point, an image serving as a reference for the projection image is obtained. Deformed projection image data is generated and stored in the storage unit 104 together with the identified feature points. Then, when projecting an image onto the user by the projection unit 102, the projection control unit 103 identifies a feature point in the user image captured by the imaging unit 101, and stores the identified feature point and its preceding stage. A correspondence relationship with the feature points stored in 104 is obtained. Then, the projection control unit 103 converts the projection image data stored in the storage unit 104 based on the obtained correspondence relationship, and causes the projection unit 102 to project an image represented by the converted projection image data. . A feature point in an image is a set of points that are unique in the image as compared to an image of another object. For example, a corner or an intersection of two edges in a straight line or a curve corresponding to the contour of the image Inflection points.

Next, the operation of this embodiment will be described.
FIG. 6 is a flowchart showing a procedure when the control unit 11 registers projection image data in the storage unit 12 prior to image projection according to the procedure described in the program. First, when the user performs a predetermined operation while the image projection apparatus 1 is attached to the user 2, the control unit 11 acquires imaging data representing an image captured by the wide-angle camera 17 (step S11). At this time, the user keeps his / her face as stationary as possible with a very natural expression. Further, the imaging data is supplied from, for example, two wide-angle cameras 17, but since the viewpoints of these cameras are different, the control unit 11 assumes that the position of the camera viewpoint is known. By synthesizing one image data based on a predetermined image composition algorithm, image data representing an image when the user's face is viewed from the front is generated. In addition, the imaging data is not supplied from the wide-angle camera 17 and may be imaging data obtained by imaging with other cameras. For example, three types of imaging data are used: a face taken from the front, a profile taken from the right, and a profile taken from the left. In this case as well, since the viewpoints of these cameras are different as described above, the control unit 11 combines one piece of imaging data based on a predetermined image composition algorithm on the assumption that the position of the camera viewpoint is known. The imaging data representing the image of the user's face is generated. Depending on the required accuracy, for example, only face image data taken from the front may be used.

  Next, the control unit 11 generates a three-dimensional model of the user's face based on the imaging data and stores it in the storage unit 12 as face model data (step S12). This three-dimensional model, that is, face model data, is polygon data representing a three-dimensional object by a combination of polygonal figures.

  Here, a standard three-dimensional model of a human face (hereinafter referred to as a standard 3D model) is stored in the storage unit 12 in advance. This standard 3D model is also polygon data representing a three-dimensional object by a combination of polygonal figures, and many feature points in a standard human face image are associated with each other. The storage unit 12 stores projection image data including feature points in the projected image, that is, projection image data in which feature points are associated with a clown image as shown in FIG. Since this projection image data serves as a reference when generating projection image data corresponding to the movement of the user's face, it is hereinafter referred to as reference projection image data.

  The control unit 11 compares the standard 3D model stored in advance in the storage unit 12 with the face model data generated in step S12, and applies a known method such as edge detection or outer periphery inspection to the face model data. Therefore, in this face model data, a feature point (referred to as feature point A) corresponding to each feature point associated with the standard 3D model is specified and stored in the storage unit 12 (step S13). Thereby, in the captured user's face, the position corresponding to the feature point determined by the standard face (that is, the feature point in the user's face) can be known.

  Next, the control unit 11 generates projection image data obtained by modifying the shape of the projection image represented by the reference projection image data based on the difference between the feature point in the standard 3D model and the feature point in the face model data (step S14). ). More specifically, since the difference between the feature point in the projection image represented by the reference projection image data and the feature point in the face model data can be represented by a vector, each part of the projection image represented by the reference projection image data Projection image data is generated by moving only the direction and the magnitude according to the vector which is the difference between the feature points included in each part. In the generated projection image data, feature points on the user's face are associated with each other as in the case of the reference projection image data. The control unit 11 stores the generated projection image data in the storage unit 12.

  Next, FIG. 7 is a flowchart showing a procedure when the control unit 11 projects an image on the user's face according to the procedure described in the program. First, when the user performs a predetermined operation while the image projection apparatus 1 is attached to the user 2, the control unit 11 acquires imaging data representing an image captured by the wide-angle camera 17, as in step S11. (Step S21). At this time, unlike the case of imaging in step S11, the user does not need to be kept stationary with a natural expression.

  Next, the control unit 11 specifies a feature point (referred to as a feature point B) of the image represented by the imaging data using a known method such as edge detection or outer periphery inspection (step S22). The position of the feature point B identified here is different from the position of the feature point A in the user's natural expression image described above, and is the position of the feature point in the expression image arbitrarily determined by the user. Next, the control unit 11 obtains a correspondence relationship between the identified feature point B and the feature point A stored in the storage unit 12 in step S13 (step S23). That is, the control unit 11 determines which feature point in the latter feature point group A corresponds to the former feature point B.

  Next, the control unit 11 obtains a conversion matrix for converting the reference projection image data based on the difference in position between the corresponding feature points (step S24). Then, the control unit 11 reads the reference projection image data from the storage unit 12, and converts the reference projection image data based on the conversion matrix (step S25). Then, the control unit 11 supplies the converted projection image data to the wide-angle projector 18, and projects an image represented by the projection image data on the user's face (step S26). The processes in steps S21 to S26 are repeatedly executed until an instruction is given to end the projection process (step S27; YES).

  According to the embodiment, even when the user moves the face and changes the facial expression, the control unit 11 specifies the feature point B in the face image after moving, and based on the feature point B Since the converted image is projected onto the user's face, the image can be changed following the movement of the user's face. Therefore, if the user laughs, the image of the clown projected on the user also has a laughing expression.

  In addition, as a use scene when projecting an image on the user's face, an unusual disguise or decorative fun by projecting a clown image that is not related to the usual user as in the embodiment. In addition to the case of directing the image, for example, by projecting an image of the same color as the surrounding skin color on the spot or mole on the user's face, it may be difficult for others to see the spot or mole Conceivable.

[Modification]
The present invention is not limited to the aspect of the embodiment described above, and can be implemented in other aspects. The following is an example of another embodiment of the present invention. In addition, you may combine these modifications suitably as needed.

[Modification 1]
In the embodiment, the projection image data registration process that matches the facial features of the user is performed prior to the image projection, but this may not be performed. FIG. 8 is a flowchart showing a procedure when the control unit 11 projects an image according to the procedure described in the program in the first modification. In FIG. 8, the same processes as those in FIG. When the control unit 11 acquires imaging data representing an image captured by the wide-angle camera 17 (step S21), the control unit 11 specifies a feature point B in the image using a method such as edge detection or outer periphery investigation, and also determines the shape of the object. For example, each part of the face such as the nose, eyes, ears, eyebrows, and mouth is identified using a technique such as pattern recognition in consideration of the size (step S22a).

  Next, the control unit 11 determines how much the position of each specified part is different from a predetermined standard position, and the standard inclination in which the inclination (posture) of each specified part is predetermined. The degree of difference from (posture) is analyzed, and the correspondence between the feature points A and B associated with the standard 3D model stored in advance in the storage unit 12 is specified (step S23a). It is assumed that the standard position and inclination (posture) of each part are stored in advance in the storage unit 12.

  Then, the control unit 11 determines the reference projection image data based on the items specified in step S23a (that is, the difference between the position and inclination of each part and their standard values, and the correspondence between the feature point A and the feature point B). A conversion matrix for converting is obtained (step S24a). Then, the control unit 11 converts the reference projection image data read from the storage unit 12 based on the conversion matrix (step S25), supplies the converted projection image data to the wide-angle projector 18, and projects an image (step). S26). In this way, prior registration processing becomes unnecessary.

[Modification 2]
In the embodiment, the image to be projected to the user is determined in advance, but the user may determine this. FIG. 9 is a flowchart showing a procedure when the control unit 11 projects an image according to the procedure described in the program in the second modification. In the second modification, in addition to the projection image data of the clown, the storage unit 12 stores a plurality of types of standard 3D models such as a monster or a cartoon hero.

  9, the same processes as those in FIG. 6 are denoted by the same reference numerals, and steps S11 to S13 are the same processes as in FIG. After step S <b> 13, the control unit 11 causes the display unit 16 to display a standard 3D model type list stored in the storage unit 12, and causes the operation unit 15 to select a desired one from the list. In response to this selection operation, the control unit 11 determines the type of the standard 3D model (step S15), reads the 3D model of that type, generates reference projection image data, and stores it in the storage unit 12 (step S14). ). In this way, it is possible to project an image desired by the user. Note that the timing at which the user selects a desired one from the standard 3D model type list may be any timing before step S14.

  The user may store a plurality of types of reference projection image data in the image projection apparatus 1 through the processing of FIG. 9 and select a desired one from these during image projection. For example, the control unit 11 displays a list of reference projection image data stored in the storage unit 12 on the display unit 16 before step S21 in FIG. Allows the user to select. In response to this selection operation, the control unit 11 determines the reference projection image data to be used, and converts the reference projection image data based on the conversion matrix in step S25. In this way, it is possible to project an image desired by the user.

[Modification 3]
FIG. 5 shows an example in which an image is projected on almost the entire area of the user's face, but the projection light from the wide-angle projector 18 does not reach the entire area of the user's face due to the unevenness of the face surface. It is possible that an image can only be projected onto a part of the face (for example, around the eyes, nose, or forehead). However, such a case is also a range to which the present invention can be applied. That is, the image projection apparatus according to the present invention projects at least a part of the imaging range of the imaging unit as an image projection range, and projects a projection image corresponding to the characteristics of the image captured by the imaging unit in the image projection range. is there. Therefore, the image projection range only needs to include at least a part of the face of the user wearing the image projection apparatus, and is not limited to the face of the user, and may be any place of the user's human body. In the present invention, any image may be projected.

[Modification 4]
The wide-angle camera 17 and the wide-angle projector 18 may be provided in places other than the rim portion 16a. For example, when the rim portion 16a corresponding to both eyes of the user is connected by a component called a bridge portion, the wide-angle camera 17 and the wide-angle projector 18 may be provided on the rim portion 16a or the bridge portion.

[Modification 5]
The functions implemented by the image projection apparatus 1 are not limited to the example of the embodiment, and various functions are conceivable. For example, retinal projection display, gaze detection mechanism, iris authentication mechanism, bone conduction microphone and earphone, biological information measurement mechanism such as pulse wave, blood pressure, body temperature, biogas, ocular sensor, front full-field wide-angle camera, illuminance sensor, UV sensors, distance sensors, non-contact charging mechanisms, angular velocity sensors, geomagnetic sensors, positioning mechanisms, temperature sensors, humidity sensors, and the like.

[Modification 6]
The present invention can be understood not only as an image projection apparatus and an image projection method executed by the image projection apparatus, but also as a program executed by the image projection apparatus. The program can also be provided in a form recorded on a recording medium such as an optical disk, or in a form in which the program is downloaded to an image projection apparatus via a network such as the Internet and can be installed and used.

DESCRIPTION OF SYMBOLS 1 Image projector, 2 user, 2a nose, 2b ear, 11 control part, 12 memory | storage part, 13 communication part, 14 audio | voice input / output part, 15 operation part, 16 display part, 17 wide angle camera, 18 wide angle projector, 101 imaging Means, 102 Projection means, 103 Projection control means, 104 Storage means.

Claims (6)

Imaging means for imaging a user wearing the device;
Projecting means for projecting an image toward a user wearing the apparatus;
A projection control unit for projecting from the projection unit a projection image corresponding to the characteristics of the image captured by the imaging unit in the image projection range, wherein at least a part of the imaging range of the imaging unit is an image projection range. An image projection apparatus characterized by that. Worn on the user's head,
The image projection apparatus according to claim 1, wherein the image projection range includes at least a part of the user's face. The projection control means includes
Prior to projecting the image by the projection unit, a feature point in the user's image captured by the imaging unit is identified, and projection image data obtained by modifying an image serving as a reference of the projection image based on the identified feature point is obtained. Generated and stored in association with the identified feature points,
When projecting an image by the projection unit, a feature point in the user image captured by the imaging unit is specified, and a correspondence relationship between the specified feature point and the stored feature point is obtained. The stored projection image data is converted based on the obtained correspondence relationship, and an image represented by the converted projection image data is projected from the projection unit. The image projection apparatus described in 1. The projection control means includes
Reference projection image data serving as a reference for the projection image is stored in association with feature points in the image represented by the reference projection image data,
Based on the image of the user imaged by the imaging means, the feature point in the image and each part of the user are identified,
Obtaining the correspondence between the identified position and inclination of each of the parts, and the feature points that have been identified and stored in association with the reference projection image data;
The stored reference projection image data is converted based on the obtained position and inclination of the part and the correspondence relationship, and an image represented by the converted projection image data is projected from the projection means. The image projection apparatus according to claim 1, wherein: The projection control unit determines any one of a plurality of types of projection images as a projection image to be projected onto the image projection range, and causes the projection unit to project the determined projection image. The image projector of any one of -4. An image projection apparatus that includes an imaging unit that captures an image of a user and a projection unit that projects an image toward the user, wherein at least a part of the imaging range of the imaging unit is used as an image projection range. An image projection method comprising: projecting, from the projection unit, a projection image corresponding to a feature of an image captured by the imaging unit in an image projection range.

Images