JP2008090807A - Compact mobile terminal - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a compact mobile terminal provided with a camera and a projector carried or worn by a user to present projection images by use of projection surfaces different in distance with an intuitive and easy-to-understand operation. <P>SOLUTION: The projector projects a projection image by a light beam group including parallel light beams or light beams having a deep focus depth as one pixel. An image of a real space including the projection image is analyzed based on an image picked up by the camera, to detect positions in the image on a near side projection surface closest to the projector and a deep side projection surface farther than the near side projection image and a three-dimensional positional relationship between the both projection surfaces on the basis of the analysis result and the positional relationship between the camera and the projector. The content of the projection image is controlled depending on the positional relationship between the both projection surfaces and a state of the change thereof, while projecting in-focus images on both the projection surfaces using the projector. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a small portable terminal including a projector and a camera, and more specifically, includes a projector that projects an image to be presented to a user and a camera that inputs the projected image, and inputs and outputs from these. A small mobile terminal that performs image processing of an image, and a user who works while moving around the work space carries or wears the small mobile terminal, so that work objects, installation objects, and objects that the user has in the work space It is possible to give work procedures and instructions while projecting complex visual information directly onto the body part of the body and the body. The present invention relates to a small portable terminal that can change the content of a projected image by giving an instruction to select a part of the projected image by changing the positional relationship between the work object and the installation object. .

  From the viewpoint of skilled labor shortage and safety management, the necessity of work support interface is increasing more than ever. One of the promising interfaces to support workers involved in movement and work is intuitive and smooth dialogue with people in the real world, virtual world, or remote areas using visual sensors and visual displays. There is a wearable visual interface (portable / wearable visual interface) to be realized (Non-Patent Document 1). In the wearable visual interface, the functions of the input means from the user to the system and the reverse output means play important roles.

  The inventors previously used a portable active camera with laser-pointer (WACL), which is a wearable input / output interface in which a head unit in which a camera and a laser pointer are integrated is mounted on a small pan / tilt actuator (patent) Document 1 and Non-Patent Document 8) were proposed.

  The user of the small mobile terminal of the WACL can send the work environment and the work process surrounding the user to the remote instructor using video and audio, and the remote instructor Since the head unit with integrated camera and laser pointer can be operated from a remote location, it can be operated from a remote location, and the instructor continues to see the location he wants to observe without being affected by the movement of the user. In addition, the instructor can control the laser pointer to point a specific place or object to the user of WACL.

  The inventors conducted a user test for comparison with a headset composed of WACL, HMD (Head Mounted Display), and HMC (Head Mounted Camera), and the contents thereof are introduced in Non-Patent Document 2.

  According to this user test, in the collaborative work in the actual work environment, the work is mainly classified into three stages: “identification of object / location”, “explanation of procedure”, and “confirmation of understanding”. The WACL interface does not constrain the head and the field of view is open. In addition, because the instructions from the laser spot are directly presented in real space, it gives an excellent impression to the operator, while the ability to express the laser pointer. The user study revealed that the work time was increased in a scene where the “procedure explanation” was complicated due to its low height.

  In addition, the result that this problem is reduced by attaching an additional display device to a WACL user has been obtained by another user test (Non-Patent Document 3). In order to view additional display devices, it is necessary to move the line of sight, which may be a drawback depending on the work contents.

  On the other hand, apart from whether it is portable or wearable or installed, many interfaces have been proposed that directly project visual information more complex than a laser spot, such as letters and figures, onto a real object. It is possible to realize more intuitive and smooth dialogue and operation by combining or combining them.

  Non-Patent Document 4 proposes a system in which an installation-type projector and a hand-held device equipped with optical sensors at the four corners of a touch panel are combined. In that system, when a handheld device is inserted into an image projected from a projector onto a wall or desk, the handheld device is combined with the light sensor at its four corners and the brightness pattern changed locally at a high frequency in the image. The position and orientation of the mold apparatus in the three-dimensional space can be measured. Using the mouse or keyboard by changing the position / posture of the handheld device or operating the touch panel while viewing both the image projected on the wall, desk, etc. and the image projected on the handheld device Therefore, the contents of the projected image can be changed intuitively or the computer can be operated.

  In the portable terminal of Patent Document 2, a projector and a distance measuring unit that measures a projection distance to a screen (projection surface) are mounted on the portable terminal. When the distance to the projection surface is short by the distance measuring means, it is possible to reduce the amount of projection light to suppress power consumption, or to use the result of distance measurement for automatic focus adjustment of the projection lens.

  In the above-described invention and proposal, a light valve projector using a liquid crystal panel or DMD (Digital Mirror Device) is used as the projector. For example, in Non-Patent Document 5, a laser projector using a two-dimensional laser scanner is mounted on the head, and a clear image can be presented regardless of the projection distance. However, due to the limited performance of the scan mirror, character information is projected directly onto a relatively narrow area in the real world.

  If a system combining a light valve projector and a camera is called a procam system here, Non-Patent Document 6 and Non-Patent Document 7 are proposals combining a procam system and a user's hand. In Non-Patent Document 6, a rain pattern is projected on the floor by a light valve projector of a pro cam system installed on the ceiling.

  When the user points his / her palm to the ceiling within the reach of the projection light, the image obtained by the camera is used to calculate the position, size and orientation of the palm, and on the palm, for example, a nearby shop Sale information can be projected.

  On the other hand, Non-Patent Document 7 proposes to attach a procam system to the user's shoulder. The ProCam system is further equipped with a near-infrared projector, and the camera is equipped with an infrared filter. As a result, when the user reaches out in front of the ProCam system, the hand area in the infrared image obtained by the camera can be used relatively easily because the infrared reflection from the projector is observed strongly. The position of the person's hand can be detected, and an image can be projected on the hand.

Further, in the professional cam system of Non-Patent Document 7, visual information such as an operation button of a remote control is projected so as to correspond to each finger, and when the gesture for selecting the finger is performed, the operation button can be selected. A function is proposed.
Japanese Patent Laid-Open No. 2004-211984 JP 2005-292428 A Takeshi Kurata, Nobuchika Sakata, Hideaki Kuzuoka, Masakatsu Kojo, Takashi Otsuki, Takuichi Nishimura: "Wearable and Tangible Interface for Remote Collaborative Work", SICE 69th Pattern Measurement Subcommittee Meeting, pp.11-18 (2006 ) Takeshi Kurata, NobuchikaSakata, Masakatsu Kourogi, Hideki Kuzuoka, Mark Billinghurst: "Remote Collaboration using a Shoulder-Worn Active Camera / Laser", InProc. 8th IEEE International Symposium on Wearable Computers (ISWC2004) inArlington, VA, USA, pp.62- 69 (2004) Nobuchika Sakata, Takeshi Kurata, and Hideaki Kuzuoka: "Visual Assist with a Laser Pointer and Wearable Display for RemoteCollaboration", In Proc. 2nd international conference on Collaboration Technologies (CollabTech 2006) in Tsukuba, Japan, pp. 66-71 (2006) Lee, JC; Hudson, SE; Summet, JW; Dietz, PH, "Moveable Interactive Projected Displays Using Projector Based Tracking", ACM Symposium on User Interface Software and Technology (UIST), ISBN: 1-59593-271-2, pp. 63-72, October 2005 Hideki Ando, Tomohiro Amemiya, Taro Maeda, "Wearable Scanning Laser Projector for Annotation Display in AR", Transactions of the Virtual Reality Society of Japan, Vol. 10, No. 2, pp. 191-200, 2005. Yoko Ishii, Satoshi Kobayashi, Akira Nakayama, and Michi Hosoda, Hand Display Interface for Encountering Information, IEICE Technical Report, vol. 105, no. 256, MVE2005-42, pp. 89-93, September 2005 . Goro Yamamoto, Kosuke Sato: "Body interface using light projection on the palm" 2005 Information Processing Society of Japan Kansai Branch Visual Information Study Group, A-05, October 2005 Nobuchika Sakata, Takeshi Kurata, Takekazu Kato, Masakatsu Kourogi, and Hideaki Kuzuoka: "WACL: Supporting Telecommunications Using Wearable Active Camera with Laser Pointer", In Proc. 7th IEEE International Symposium on Wearable Computers (ISWC2003) in NY, USA, pp.53- 56 (2003)

  As described above, various projector systems and pro-cam systems have been proposed so far, but as a problem when using these systems, an operator who works while moving around the work space carries or wears it. When it is intended to be applied to a small portable terminal equipped with a projector and a camera to be used, there are the following problems to be solved.

  First, in the case of the small portable terminal of Patent Document 1 (WACL of Non-Patent Document 8), as described above, the information output device to be presented to the user is only a laser spot, and therefore the expression capability is poor. Not enough information can be provided to users.

  In the case of Non-Patent Document 4, for example, in consideration of realizing a similar function by combining a portable projector and a handheld device having optical sensors mounted on the four corners of the touch panel, in this case, the handheld device It is necessary to mount optical sensors at the four corners, and interaction using objects around the user or the user's hand cannot be performed.

  In Non-Patent Document 4, a light valve projector is used. However, because of its principle, the depth of focus is limited to a certain range, and therefore, it corresponds to a distance between a projection surface such as a wall or a desk and a handheld device. There are restrictions in use such as installing a projector at a certain distance and increasing the projection distance, or not increasing the distance between the projection surface such as a wall or a desk and the handheld device too much. If these restrictions are not satisfied, it is impossible to provide a focused image for both the image on the projection surface and the image on the handheld device.

  Considering these restrictions in terms of small mobile terminals, it is impractical to take a long projection distance because the projector must be installed on any part of the user's head or shoulders. Also, if the distance between the projection surface and the handheld device cannot be made too long, the available situations will be considerably limited. Further, since the system described in Non-Patent Document 4 is an installation type, it is assumed that the positional relationship between the projection surface (wall or desk) other than the handheld device and the projector is obtained in advance. This assumption is impractical for small mobile terminals.

  On the other hand, although the portable terminal of Patent Document 2 is not an installation type, since the distance measuring means outputs the distance to the screen and uses a light valve projector, the target projection surface is the target. It turns out that it is one plane (screen) in a real environment. Therefore, when there are a plurality of projection surfaces having different distances to the terminal, it is difficult to focus on all the projection surfaces.

  The head mounted laser projector proposed in Non-Patent Document 5 uses two galvanometer mirrors (two-dimensional laser scanners) to scan a laser light source and presents a clear image regardless of the projection distance. Although the galvanometer mirror has a relatively wide deflection angle, the image that can be projected at a time is limited to a narrow angle of view such as character information of several characters due to the limitation of the oscillation frequency. For this reason, it is not assumed that visual information is projected and presented at once in various places in the real environment or in the hands of users.

  Similarly to Non-Patent Document 4, Non-Patent Document 6 is an installation type system. Since the ProCam system using a light valve projector is installed on the ceiling, the projection distance can be made relatively long, and a focused image can be presented on both the palm and the floor. If you do it will be difficult.

  In addition, there is no means to measure anything other than the user's hand, and the measurement also takes into account video without taking into account the effect of the projected image or actively using the projected image. I am trying to calculate the position, size, and orientation of my palm from skin color information and shape information. For this reason, it is difficult to stably project an image on the palm in an actual environment.

  In Non-Patent Document 7, since the image is projected only on the hand, the focus of the light valve projector mounted on the shoulder may be limited to the range that can be reached by the user's hand. It is difficult to present projected images to both hands simultaneously. Non-Patent Document 7 does not assume a projection surface other than the hand. Unlike Non-Patent Document 6, near-infrared images are processed in order to prevent the projected image from adversely affecting image processing for hand measurement. Therefore, the hand measurement can be performed more stably than in Non-Patent Document 6, but the measurement of the hand becomes unstable in the situation where sunlight and other heat sources are observed.

  The present invention has been made in view of the various circumstances as described above, and an object of the present invention is to allow a user who works while moving around the work space to carry or wear the projection surfaces on the near side and the far side. An object of the present invention is to provide a small portable terminal equipped with a projector and a camera that can be operated intuitively and intelligibly while presenting an image using.

  In addition, another object of the present invention is that a user who works while moving around the work space can carry or wear it, and a work object or installation in the work space, a part of the user's object or body, can be complicated. Gives work procedures and instructions while directly projecting visual information, and changes the user's object, the shape of a part of the user's body, and the positional relationship with the work object and installation in the back. Thus, an instruction to select a part of the projection image is given, and a small portable terminal capable of changing the content of the projection image is thereby provided.

  In order to achieve the above object, as one aspect, a small portable terminal according to the present invention projects a projection image by a group of light beams including a parallel light beam or a light beam having a deep focal depth as one pixel, and the projection image. The real space including the camera, and analyzing the real space image including the projection image from the image captured by the camera, and based on the analysis result and the positional relationship between the camera and the projector, The position in the image of the near side projection plane that is the closest projection plane and the far side projection plane that is farther than the near side projection plane in the image, and the three-dimensional relationship between the near side projection plane and the near side projection plane A projection plane detecting means for detecting a correct positional relationship, and controlling the projector to project a focused projection image on both the front projection plane and the rear projection plane by the projector. A projection image control means for controlling the contents of the projection image in accordance with a positional relationship between the near side projection plane and the back side projection plane and a change state thereof, and a data processing means for supplying the projection image to the projector. It is set as the structure provided.

  In the above configuration, the projector is a laser projector using a two-dimensional scan mirror in which a high-frequency one-dimensional scan mirror and a low-frequency one-dimensional scan mirror are combined, and the scan direction of the high-frequency scan mirror is: The optical center of the camera and the optical center of the projector are configured to be substantially parallel to a straight line.

  In addition, the small portable terminal according to the present invention is further configured to include a mounting tool for mounting the small portable terminal on the chest or shoulder of the user.

  In the small portable terminal according to the present invention, the light ray projected by thinning out the number of light rays projected on the projection plane having a short projection distance with respect to the light ray group projecting each pixel of the projection image projected from the projector. The projector has a thinning means, and the projector projects a projected image under the control of the light thinning means, and the projected image is clearly projected on both the near-side projection surface and the back-side projection surface.

  As another aspect, in the small portable terminal of the present invention, a portion in the back side projection surface that can be observed from the camera although it cannot be observed from the user by being concealed by the near side projection surface. By projecting a geometric pattern light of any one of a straight line, a point cloud, a circle, a triangle, and a lattice pattern onto the region by controlling the projector, and photographing the pattern light with the camera, the partial region Or a concealment area projection means for measuring the distance between the camera and the partial area.

  As another aspect, in the small portable terminal according to the present invention, the camera is a fish-eye camera, and the projection plane detection unit converts the fish-eye image photographed by the fish-eye camera into a perspective projection image. Image conversion means for performing the image conversion of the image, and the projection plane detection means determines the three-dimensional positional relationship between the front projection plane and the rear projection plane based on the perspective projection image converted by the image conversion means. Configured to detect.

  The small portable terminal of the present invention configured as described above includes a projector that outputs an image to be presented to a user and a camera that inputs the image, and a user who works while moving around the work space By carrying or wearing the camera, it is possible to give work procedures and instructions while projecting complex visual information directly onto work objects and installations in the work space, and objects and parts of the user's body. Gives instructions to select a part of the projected image by changing the shape of the user's object or part of the user's body, and the positional relationship with the work object or installation object Thereby, the contents of the projected image can be changed.

  That is, in the present invention, the focal point is focused on both the near side projection plane (typically the user's hand) that is the projection plane closest to the projector and the far side projection plane that is farther than the near side projection plane. A parallel light group projecting unit that projects a parallel light beam or a light beam having a deep focal depth as a pixel as a projected image is projected by the projector, and includes a projection image projected by the projector. By taking a real-space image with a camera and analyzing the image, the positions of the front and back projection planes in the image are detected, and the positional relationship between the front and back projection planes The content of the projection image is controlled according to the above. Thereby, an instruction to select a part of the projection image or to change the content of the projection image can be given.

  Hereinafter, an embodiment of a small portable terminal according to the present invention will be described with reference to the drawings as an embodiment for carrying out the present invention. FIG. 1 is a diagram illustrating a system configuration of a small portable terminal according to the present invention. In FIG. 1, reference numeral 100 denotes a small portable terminal. The small portable terminal 100 is provided with a camera 1, a projector 2, and a self-contained sensor 3 as shown in the external view from the front side, and each is connected to the information processing system 4 although not shown. .

  The information processing system 4 is provided with a projection plane detection unit 5, a projection image control unit 6, and a wireless communication unit 8 as processing modules that are main system elements for executing information processing. As storage targets, a storage device is provided in which image data of an input image 700 captured and captured by the camera 1 and a projection source image 800 that is an image projected from the projector 2 is stored. Although not shown, a data processing means for performing these control processes and performing data processing for image processing information processing is provided.

  On the right side of FIG. 1, in order to illustrate the positional relationship between a plurality of projection planes of an image projected from the projector 2 of the small portable terminal 100, the small portable terminal 100 is illustrated by an appearance from the back side, and the small portable terminal is illustrated. The projection image projected from the projector 2 of the terminal 100 is shown. The camera 1 of the small portable terminal 100 captures a real space including a projection image projected from the projector 2. The projector 2 projects a projection image with a light beam group in which a parallel light beam or a light beam having a deep focal depth is one pixel. The self-contained sensor 3 measures acceleration, angular velocity, geomagnetism, etc., and acquires position information of the small mobile terminal 100. Of course, the arrangement of the camera 1, the projector 2, and the self-contained sensor 3 is not limited to the illustrated form, and may be changed. The self-contained sensor 3 may include an attitude sensor. Furthermore, by performing posture control by the stabilization control means using data from the posture sensor, it is possible to perform stabilization control for the position projected by the projector 2 and the image photographed by the camera 1. Known attitude sensors and stabilization control means can be used. The camera 1 is a wide-field camera.

  An information processing system 4 is built in or connected to the small portable terminal 100, and data processing for information processing of image processing is performed by the information processing system 4. The projection plane detection unit 5 provided in the information processing system 4 performs information processing for detecting the near side or the far side of the projection plane, and the projection image control unit 6 projects the projection image on the projection plane on the near side. To control whether to project the projection image on the rear projection surface. The wireless communication means 8 communicates with a remote instructor system (not shown), transmits the input image 700 taken by the camera 1 to the instructor system, and receives control information from the instructor system. Communication is performed to receive and supply the projection source image 800.

  1 shows a projection image projected from the projector 2 of the small portable terminal 100, and the small portable terminal 100 provided with the projector 2 and the near-side projection plane 200 which is the closest projection plane. The positional relationship between the user's hand and the far side projection plane 201 which is a far projection plane is shown. The back projection surface 201 shows a state in which the shadow of the user's hand (cast shadow 202) is shown in the projection image.

  The near-side projection plane 200 that is the projection plane closest to the projector 2 of the small mobile terminal 100 is, for example, a user's hand, which is one of the representative ones, and is farther than the near-side projection plane 200. The back side projection surface 201 is, for example, a wall surface facing the user.

  A projected image 300 is projected onto the near side projection surface 200 of the user's hand, and a projected image 400 is projected onto the back side projection surface 201 of the wall surface facing the user. The near-side projection surface 200 has a shadow portion with respect to the back-side projection surface 201, becomes a cast shadow 202 on the back-side projection surface 201, and is removed from the projection portion. In FIG. 1, the projection light arrival space 203 is a region that is a space having a shape close to a quadrangular pyramid. When the near-side projection surface 200 enters the projection light arrival space 203, the projection image is partially separated and a cast shadow 202 is generated.

  As described above, in the small mobile terminal 100, the projection source image 800 is projected by the projector 2 as a projection image of a light beam group in which a parallel light beam or a light beam having a deep focal depth is set as one pixel via the projection light arrival space 203. Since the projection is performed on the side projection plane 200 and the back projection plane 201, as described above, when a user's hand enters the projection light arrival space 203, the projection is performed on the front projection plane 200 of the user's hand. The image 300 is projected, and the projection image 400 is projected on the back side projection surface 201.

  Since the camera 1 captures a real space including these projected images (300, 400), these images are captured as the input image 700. The input image 700 is subjected to image processing by information processing of the projection plane detection means 5, and details will be described later, but an actual space image including the projection image is analyzed from the input image photographed by the camera 1, Based on the analysis result and the positional relationship between the camera 1 and the projector 2, the near-side projection plane 200 that is the projection plane closest to the projector 2 and the far-side projection plane that is farther than the near-side projection plane 200. Information processing for detecting a position in the image with respect to 201 and a three-dimensional positional relationship between the near side projection plane 200 and the back side projection plane 201 is performed.

  Further, the projection image control means 6 controls the projector 2 by the information processing as described later, and projects a focused projection image on both the front projection surface 200 and the back projection surface 201 by the projector 2. However, the content of the projected image to be projected is controlled in accordance with the positional relationship between the near side projection plane 200 and the back side projection plane 201 and the change state thereof.

  The projector 2 uses a parallel light beam or a light beam having a deep focal depth as one pixel, and projects a projection image (300, 400) by the light beam group. This functions as a parallel light beam projecting unit, and is a typical example. Uses a two-dimensional scan mirror that combines a high-frequency one-dimensional scan mirror and a low-frequency one-dimensional scan mirror that has a scan direction orthogonal to the scan direction, and a laser light source that outputs parallel rays or rays with a deep focal depth. Used a laser projector.

  As the projector 2, such a laser projector is used, but any other projector can be used as long as it functions as a parallel light group projecting unit that projects a parallel light beam or a light beam having a deep focal depth as one pixel and projects a projection image using the light beam group. The projection means may be used.

  The input image 700 is image data obtained by capturing a real space including the projection image 300, the projection image 400, and the cast shadow 202 using the camera 1. The projection source image 800 is a projection source image that is projected when the projector 2 is controlled by the projection image control means 6. The projection source image 800 is generated and processed by data processing means (not shown). Further, the wireless communication unit 8 supplies the external server.

  Projection plane detection means 5 for detecting the near side projection plane 200 and the back side projection plane 201 performs image analysis using the input image 700 and uses the result of the image analysis and the positional relationship between the camera 1 and the projector 2. Thus, the positions of the near side projection plane 200 and the back side projection plane 201 in the input image 700 are detected. At the same time, a three-dimensional positional relationship between the near side projection plane 200 and the back side projection plane 201 is detected. In the information processing of the projection plane detection unit 5, a known method for detecting the cast shadow 202 is used as will be described later.

  As another example of the projection plane detection means 5, image features such as edges and corners included in the projection source image 800 are detected in advance, and those image features are searched for and associated in the input image 700. By performing the information processing to be performed, the front side projection surface 200 and the back side projection surface 201 may be detected.

  As the image feature, for example, an edge detected using a Sobel operator, a feature point described by a feature point description method called a SIFT (Scale Invariant Feature Transform) key, or the like can be used. Thereby, the projection source image 800 can be handled in the same manner as pattern light such as a point cloud and slit light used in a general range finder. That is, the small portable terminal 100 can be used as a range finder using the active stereo method by the camera 100 and the projector 2.

  However, in this case, since the projection source image 800 is not appropriately designed as pattern light, only the distance of a region having an image feature can be measured. For the distance between the other regions, processing combined with a method using interpolation or cast shadow is executed based on the result of the region having image characteristics.

  The projection image control means 6 uses the positional relationship between the near-side projection plane 200 and the back-side projection plane 201 and the state of change thereof (discriminating these situations) to project projection images to be projected onto these projection planes. Control is performed to change the contents when the projection source image 800 is used in the application example. FIG. 2 is a diagram for explaining control for changing projection images projected on the projection surfaces of the near side projection surface and the back side projection surface in the small portable terminal of the present invention. This will be described with reference to FIG.

  FIG. 2 shows an exemplary embodiment. In the left side of FIG. 2, the user's hand, which is the front projection plane 200, is moved up, down, left, and right, and a part of the projection image 400 projected on the back projection plane 201 is projected on the front projection plane 200. Represents. An enlarged view of the front projection surface 200 is shown in the lower left of FIG. According to the enlarged view at the lower left of FIG. 2, in this case, a portion 300 written as “AIST Tsukuba” is projected from the projection image 400 on the rear projection surface 201 which is a satellite photograph so as to be placed on the front projection surface 200. The situation is shown. That is, a part of the projected image is cut out and shown.

  Here, for example, when the projection image control means 6 determines that the near side projection plane 200 is stationary at the same position for several seconds using the result of the projection plane detection means 5, the projection of the part written as “AIST Tsukuba” It can be recognized that the image 300 has been selected. Further, in this case, since the near-side projection plane 200 is detected by the projection image detecting means 5, the projection image control means 6 projects onto the near-side projection plane 200 as shown in the center part on the right side of FIG. The projected image 300 to be displayed can be switched from the projected image 300 to a projected image 302 and displayed. Here, as the projection image 302, a CG image showing a state when about half of the building existing in the portion (projection image 300) written as “AIST Tsukuba” is constructed by changing the displayed projection image. Projected.

  2 is an example in which the near side projection plane 200 that is the user's hand moves up, down, left, and right, but the right side view of FIG. FIG. 4 shows whether the projection image 400 moves toward or away from the far side projection surface 201 on which the projection image 400 is projected.

  In this case, as shown in the center part on the right side of FIG. 2, for example, in the state in which a projection image 302 of a CG image showing a state where about half of the building has been constructed is displayed, the near side projection surface 200 is placed closer to the near side. In other words, when moved away from the rear projection surface 201, the projection image control means 6 projects a projection image projected on the front projection surface 200, and shows a CG image showing a state when the building is completely constructed. On the contrary, when the front side projection surface 200 is moved so as to be closer to the back side projection surface 201, the projection image control means 6 is projected onto the front side projection surface 200. The projected image of the CG image is changed to a projected image 301 of the CG image showing a state before the building is constructed. By changing the projected image in this way, the time axis of the projected CG image can be manipulated.

  Such an intuitive operation can be performed while projecting a projected image in focus on both the near side projection surface and the far side projection surface by the projector 2 functioning as a parallel light group projection unit. . Such an application example is an unprecedented feature and is an excellent feature of the small portable terminal according to the present invention. Although the small portable terminal 100 is provided with the self-contained sensor 3, by measuring the momentum of the small portable terminal 100 with the self-contained sensor 3, this momentum is used as control information of the projection image control means 6. A projection image stabilization process may be performed such that the projection image 400 is projected onto the same position on the back projection plane 201 by information processing.

  In the description of FIGS. 1 and 2, the embodiment in the case where the back side projection surface 201 is a plane is described. However, for example, as shown in FIG. A projection surface that has a depth with respect to the direction and cannot be focused on all places, such as a light valve projector, may be used as the rear projection surface 201 here.

  FIG. 3 is a diagram for explaining an embodiment when the projection surface projected by the small portable terminal of the present invention is uneven. This will be specifically described with reference to FIG. Here, a situation is shown in which a user 600 wearing the small portable terminal 100 on the chest is working on three boxes placed on a desk 500. The annotation “Sell-by-date Jan 31, 2007” is projected from the projector 2 to the left box 451 from the projector 2 toward the left side of the user 600 by the small portable terminal 100 in the chest, and is displayed on the middle box 452. , An annotation “Outdated” is projected, and an annotation “Send to” is projected in the right box 453. Further, the map information 450 projected on the right box 453 and the desk 500 is projected so as to be visually connected by a line 454, and where to send the right box 453. Is intuitionally understood. In this case, for example, the user 600 can reach out to create the near side projection surface (200), select a part of the map, and obtain detailed information thereof.

  Such an interaction includes a small portable terminal 100 according to the present invention provided with a projector 2 that provides the function of a parallel light group projection unit, and a camera 1 that captures a projected image. This is realized for the first time by providing the information processing system with the projection plane detection means 5 for detecting, the projection image control means 6 for controlling the projection image projected on the near side projection plane or the back side projection plane, and the processing module for performing image processing. This is an excellent feature that is not possible in the past.

  The wireless communication means 8 transmits an input image 700 taken by the camera 1 to a remote collaborator (instructor) and receives projection images 300 and 400 projected from the remote collaborator. Then, it can be projected onto the work space by the projector 2. In this case, although not shown, an embodiment of a typical remote cooperative work system can be configured by using voice interaction means together with transmission / reception of such video.

  As described above, as a typical example of the projector 2 provided with the function as the parallel light group projection means, a high-frequency one-dimensional scan mirror and a low-frequency one-dimensional scan having a scan direction orthogonal to the scan direction are provided. A laser projector using a two-dimensional scanning mirror combined with a mirror and a laser light source that outputs a parallel beam or a beam having a deep focal depth is used. Here, in a projector used in this small portable terminal, The scan direction of the scan mirror is set substantially parallel to a straight line connecting the optical center of the camera 1 and the optical center of the projector 2. This makes it easier to detect a boundary of a cast shadow described later by image processing.

  Next, the near side projection plane or the far side projection plane is detected by the cast shadow 202 detection method of one typical embodiment of the projection plane detection means 5 for detecting the near side projection plane 200 or the back side projection plane 201. Information processing will be described with reference to FIG.

  FIG. 4 is a diagram illustrating an example of information processing for detecting the near side projection plane or the back side projection plane by the cast shadow detection method. In FIG. 4, reference numeral 801 denotes a projection source image. The projection source image 801 is applied to the palm which is the front projection plane 200 and the wall which is the back projection plane 201 by a small portable terminal (concept verification system) configured using a light valve projector using DMD as the projector 2. Projected. Reference numeral 702 denotes an input image obtained from the camera 1 at that time. Since the light valve projector has a shallow depth of focus, the distance between the projector 2 and the front projection surface 200 is approximately 30 cm, and the distance between the projector 2 and the rear projection surface is approximately 45 cm. I tried to fit.

  The cast shadow 30 of the projected near side projection surface 200 appears in the projection image projected on the back side projection surface. The boundary line 31 between the cast shadow 30 and the projected image on the far side projection plane depends on the change in the luminance of the projected image, but the luminance gradient in the direction orthogonal to the boundary line 31 is relatively large. The reflected light of the hand, which is a typical example of the near side projection surface 200, mainly comprises a diffuse reflection component, and further, the normal direction of the surface is orthogonal to the light beam in the vicinity of the boundary 32 and the reflected light is further weakened. A boundary 32 between the hand and the cast shadow 30 which is a typical example of the projection surface 200 has a relatively small luminance gradient. In consideration of these, information processing is performed to extract a contour by two-level binarization using a higher threshold value and a lower threshold value, and the portion where the vertical interval 33 of the two contours is substantially the same is the horizontal direction in the image. The cast shadow 30 is detected by determining that the location that exists continuously is the cast shadow 30.

  Of course, when the near side projection plane 200 is not a hand, the nature of the boundary 32 in the image is different. Therefore, it is possible to select a process for extracting the boundary 32 for each type of the near side projection plane 200. It is. Furthermore, if the shape of the cast shadow 30 is associated with the shape model of the near side projection plane 200, the scale of the near side projection plane 200 shown in the input image 702 can be known, so that the rough distance to the near side projection plane 200 can also be obtained. Prove. Further, the area excluding the front projection plane 200 and the cast shadow 30 is regarded as the back projection plane.

  As described above, when the front projection surface 200 is moved to the front or back as shown on the right side of FIG. 2 and the CG images 301 to 303 are switched and displayed, an absolute distance is not necessarily required. is not. For example, the operation in the depth direction can be performed only by changing the width 33 of the cast shadow 30 that can be observed from the camera 1. In FIG. 2, assuming that the state in which the CG image 302 is displayed is a reference state, the near side projection surface 200 is in the back when the width 33 of the cast shadow 30 is narrower than that state, and close when the cast shadow 30 is widened. Therefore, it is possible to carry out control to switch the CG images 301 to 303 under the control of the projection image control means 6. In this case, the relative relationship between the three-dimensional positions of the near side projection plane 200 and the back side projection plane 201 is obtained.

  FIG. 5 is a diagram for explaining an example of use of a small portable terminal according to the present invention. The experimental example shown in FIG. 5 is an example in which an experiment was performed under the same conditions as in FIG. Here, a scenario is assumed in which a moving user selects and interacts with a certain worker while grasping the positions of himself and other workers using a map application. In FIG. 5, an input image 701, an input image 702, and an input image 703 are input images at a certain point in time, and time advances in the order of the input image 701, the input image 702, and the input image 703. The projection source image 801 is the one at the time of the input image 701 and the input image 702, and the projection source image 802 is the one at the time of the input image 703. When an operation for putting the icon 304 indicating the cooperative worker on the map on the hand, for example, an operation such that the icon 304 appears in the palm of the front projection surface (input image 701, input image 702), the face of the cooperative worker is displayed. A picture 305 is displayed, and if it is left as it is for a while, the operation flow is such that the other party is called for a call.

  FIG. 3 illustrates an example in which the small portable terminal 100 is worn around the chest center of the user 600. However, when the small portable terminal 100 is worn instead of being held in the hand, the mounting position of the small portable terminal according to the present invention is objective. Therefore, the evaluation by the simulation was performed from the viewpoint of supporting the work accompanied by the actual object operation. As evaluation items, the breadth of field of view, the visibility of the work area, and the stability during operation were raised, and the weighted average of them was used as a comprehensive evaluation result.

  FIG. 6 is a diagram for explaining the result of the simulation evaluation from the viewpoint of supporting work involving real object operation in order to objectively determine the mounting position of the small mobile terminal. FIG. 6 shows an example of evaluation that visualizes the result of the simulation. Reference numeral 51 denotes the field of view, 52 denotes the visibility of the work area, 53 denotes the stability during the swing motion, and 54 denotes the results of the stability during the walking motion. 50 is a visualization diagram of a comprehensive evaluation result by the weighted average. In these figures, the darker is the better evaluation and the brighter is the bad evaluation. Based on this result, the small portable terminal 100 according to the present invention is basically used by attaching the small portable terminal to a portion 55 located further in front of the body on the chest or shoulder of the user. The small portable terminal 100 is provided with a mounting tool for such mounting.

  FIG. 7 is a diagram for explaining the operation of the light beam thinning means according to a modification of the projector. An outline of the operation of the beam thinning means is shown. The light thinning means 12 applies to the light beam group for projecting each pixel of the projection image projected from the projector so that the pixels of the projection image projected from the projector 2 overlap on the projection surface and the projection image does not become blurred. Thus, it is a control means for controlling the projector 2 so that the number of light rays projected onto a projection surface with a short projection distance is thinned out and projected. Reference numeral 40 denotes one parallel light beam. Further, in FIG. 7, for simplification of explanation, the front side projection surface 200 and the back side projection surface 201 are each represented by a plane, and only one scan line of the projector 2 is illustrated. In the case of two-dimensional scanning, thinning out can be performed in the same way. Further, the distance from the projector 2 to the front projection plane 200 and the distance from the projector 2 to the back projection plane 201 are 1: 2.

  For example, the light beam group output (40) of the projector 2 is set so that pixels can be displayed densely at the distance from the projector 2 to the back side projection surface 201 without overlapping each light ray on the back side projection surface 201. If this is the case, on the near side projection surface 200, by thinning out light rays at a rate of one in two, it is possible to prevent pixels from overlapping on the projection surface and blurring the projected image. This ratio can be determined by obtaining the distance to each projection plane by the projection plane detection means 5. In other words, by using the light thinning means 12, the projected image is clearly projected simultaneously on both the front side projection surface 200 and the back side projection surface 201.

  FIG. 8 is a diagram for explaining the operation of the concealment area photographing means of another modification of the projector. The outline of the operation | movement by a concealment area | region projection means is shown. Similarly to FIG. 7, only one scan line is shown, but the same concept can be adopted in the case of two-dimensional scanning. The concealed area projecting means 13 cannot be observed from the user viewpoint 601 because it is concealed by the near side projection plane 200, but a straight line or a point with respect to the partial area 60 in the back side projection plane 201 that can be observed from the camera 1. Information that measures the three-dimensional shape of the partial region 60 and the distance between the camera 1 and the partial region 60 by projecting geometric pattern light such as groups, circles, triangles, and lattice patterns and photographing with the camera 1 Process. Since the partial area 60 cannot be observed from the user viewpoint 601, even if visual information useful for the user is presented there, it is wasted. The hidden area projection means 13 obtains the position of the partial area 60 by the projection plane detection means 5 and projects the geometric pattern light. Thereby, for the partial region 60, the small portable terminal 100 can be used as a range finder using the active stereo method by the camera 1 and the projector 2.

FIG. 9 is a diagram illustrating a state in which pattern light is presented in a projection area that is hidden by a user's hand. FIG. 9A shows an image of the user's line of sight, and FIG. 9B shows an image taken by the camera. As described above, the near side projection surface 200 and the back side projection surface 201 are observed from the user's line of sight (image in FIG. 9A), but the back side projection surface 201 cannot be observed by the user. An area 60 that can be observed from the camera is present in the rear projection plane 201 (image in FIG. 9B). For example, as shown in FIG. 9B, pattern light is projected onto a part of such a region 60. The pattern light projected on the region 60 is blocked by the user's hand and cannot be observed by the user. Therefore, calibration is performed while the user does not notice using the image projected on the region 60. And 3D shape acquisition can be executed. Further, the cast shadow 202 of the near side projection plane 200 existing on the back side projection plane 201 can also be used as an important information source for knowing the relationship between both projection planes.

  A small portable terminal 100 according to the present invention is a system in which a projector and a camera are combined, and is a system called the above-described pro-cam system. The small portable terminal 100 provides a new wearable interface that does not restrain the head. With this wearable interface, new interaction techniques that seamlessly merge the real and virtual worlds will be provided to users anytime and anywhere. Regarding the details of the user interface of the wearable interface, the present inventors have proposed so-called BOWL ProCam (BOdy-Worn Laser Projector Camera) and have been proposing it while demonstrating its configuration in detail. It is. The experimental results so far will be described.

  Characteristically, the BOWL ProCam system projects a projected image of various contents using a laser projector with a deep focal depth, and captures the projected image with a high-definition fish-eye camera that makes it possible to grasp a wide range of situations. In addition, an attitude angle sensor effective for stabilizing the projected image is provided, and the user can stably operate a new interaction. We are also evaluating these configurations.

  The simulation experiment and the result will be described for evaluating the wearing part of the ProCam system. Evaluation is performed from the viewpoint of a ProCam system suitable for supporting work involving real object operations, and the chest is selected as the wearing site. Specifically, as an interface operation, the content presented on the projection surface such as a desk or a wall is blocked by the front projection surface represented by the user's hand and selected so as to be scooped out. This is to make it easier to do. This is an evaluation using a system developed using an active stereo method using a desktop image and an input image obtained by photographing the projected image and a light valve projector.

  BOWL ProCam (BOdy-Worn Laser Projector Camera), a new wearable input / output interface based on the results of user tests on WACL, is a laser projector, a high-definition wide-angle camera, a high-definition fisheye camera, an inertial sensor, or an attitude angle sensor. It consists of the stabilization device used and has a number of excellent features.

  Specifically, by using a high-definition wide-angle camera, the WACL feature that it is easy to grasp the work space regardless of the movement of the worker is maintained. As a result, the same object can be continuously observed and grasped, and projection image stabilization and work history recording can be facilitated. Compared with active cameras, there are problems such as motion blur and difficulty in exposure due to high definition and wide angle, but the advantage of not requiring an actuator is great.

  The features of the laser projector include not only the ability to project advanced visual information directly to the real world, but also the features such as deep depth of focus, easy miniaturization, and low power consumption. This laser projector employs a stainless steel mirror, a drive unit, and a small scanning mirror consisting of a piezoelectric element formed on the drive unit by the aerosol deposition (AD) method, compared with a general MEMS scanner. However, it is easy to realize high-speed scanning, high angle of view, and low cost.

  In addition, the ProCam system can acquire 3D structures in the real world through a combination of active stereo technology and shape-from-motion technology, so the system can intuitively interact with the user and the real environment better understood. Can provide techniques.

  When determining the mounting position of the camera, as a point to be taken into consideration, to obtain the image from the wearer's viewpoint, to capture the situation of the wearer's surroundings (front, back, overall), to capture the movement of the fingers and walking Therefore, it is set in consideration of various reasons, such as enabling it to be worn without difficulty every day. Many of these are determined empirically or experimentally, but in order to more objectively evaluate the camera mounting position, Mayol et al. Developed a simulator using a human articulated model consisting of 1800 polygons. And evaluate.

As described with reference to FIG. 6, the simulator of Mayol et al. Was improved from the viewpoint of a ProCam system for supporting work involving real object operations, and an experiment for evaluating the mounting position of BOWL ProCam was conducted. The evaluation items include the field of view, the visibility of the work area, the stability during operation, and the weighted average E as shown in the following equation. It was set as a comprehensive evaluation result.

Here, w _fov , w _hs and w _stb are the weights of the respective evaluation items. Here, emphasis is placed on the _visibility and stability of the work area, and w _fov = 0.1, w _hs = 0.5, and w _stb = 0.4. The field of view Efov is evaluated by the small self-occlusion ratio when an ideal omnidirectional camera is attached to each part of the body (upper left of FIG. 6). In the evaluation of the visibility of the work area, an area through which a straight line connecting both hands is passed as a typical work area when both arms are moved up and down in a state as shown in the lower left of FIG. Then, the inner product of the unit vector indicating the line-of-sight direction when each sample point in the region is viewed from the user's viewpoint and the unit vector indicating the line-of-sight direction (projection direction) when projected from the body part) Let the magnitude be an evaluation value E _hs . The stability during movement was evaluated by the small amount of displacement of each part in the walking movement (lower right of FIG. 6) and the turning movement of the neck (upper right of FIG. 6) (E _walk , E _look ).

  As a result, as shown in the center of FIG. 6, in the evaluation by this simulation, a result that the front upper part of the upper body was most suitable as the mounting position of the ProCam system was obtained. Although the result may vary slightly depending on the setting of the evaluation item and its weight, the “BOWL ProCam” system is designed according to this evaluation result.

  As described above with reference to FIG. 2, the interaction technique utilizing the feature of the laser projector having a large depth of focus and a wide angle of view includes a front projection surface represented by the user's hand (front projection surface 200), and a desk. Both the rear projection plane (the rear projection plane 201) such as a wall and a wall are effectively used. In these examples, the user selects his / her content by moving his / her hand up / down / left / right and blocks the projected image displayed on the wall, and moves it back and forth. Giving.

  In general, in a conventional light valve projector using liquid crystal, DMD, or the like, visual information is imaged on the front projection surface (hand) and the rear projection surface (wall surface) due to its optical characteristics. Although difficult, the problem can be reduced by using a laser projector that can project a projection image with light rays close to parallel rays. For this reason, it can be said that this interaction technique is suitable for the “BOWL ProCam” system. However, no matter what projector is used, if you do not know where the front projection plane is and where the rear projection plane is, you can select content or change only the visual information on the hand I can't let you.

  In order to implement the interaction technique unique to the “BOWL ProCam” system that implements a user interface using the front and rear projection planes, a projected desktop image and an input image obtained by capturing the projection image are used. The front surface is detected based on the active stereo method.

  When performing image processing data processing for detecting the near side projection surface and the back side projection surface and detecting the positional relationship, the distance between the projector and the palm is about 35-55 cm, and the distance between the back side projection surface Is set to approximately 65 cm so that projection can be performed with both images as focused as possible. In addition, a plane is assumed as the rear projection plane.

  In the small portable terminal here, the camera 1 uses a fisheye camera with a wide viewing angle, accurately captures the projected image including its peripheral portion, and performs image processing on the captured image. For this reason, although not shown, the projection plane detection unit 5 includes an image conversion unit that performs image conversion from a fish-eye image captured by a fish-eye camera to a perspective projection image. Then, the projection plane detection unit 5 detects a three-dimensional positional relationship between the near side projection plane and the far side projection plane based on the perspective projection image that has been image-converted by the image conversion unit.

FIG. 10 is a diagram for explaining a relationship between an imaging system of an equidistant projection type fisheye camera and a projection system of a projector. FIG. 11 shows an input image from the fisheye camera and perspective projection conversion. The resulting image is shown. Data processing for calculating an arithmetic expression related to the imaging system as shown in FIG. 10 is performed on the fish-eye image captured by the fish-eye camera, and conversion from the fish-eye image to a perspective projection image is performed. The distance l from the center of the image to the pixel xc is called an image height, and has a relationship of l = f _w θ in the equidistant projection method and l = f _w tan (θ) in the perspective projection method. Based on this relationship, conversion from a fisheye image to a perspective projection image is performed. An example of the conversion result is shown in FIG. FIG. 11A shows an input image from the fisheye camera. In the image-converted image shown in FIG. 11B, an example of a result obtained by performing only simple parametric correction by circle fitting on the outer periphery of the fisheye image is shown.

  12 to 13 are diagrams for explaining the parallelization processing of the projected image capturing portion and the feature point association processing, respectively. In the parallelization processing of the projected image capturing portion, a rectified image of only the area where the projected image is captured is generated so that the aspect ratio, resolution, and origin match the desktop image (FIG. 12 top). . The lower part of FIG. 12 shows an image obtained as a result of parallelizing the projection image picked up near the center of FIG.

Here, data processing for detecting the near side projection plane and the back side projection plane using the fisheye image data of the input image input from the fisheye camera will be described. In this process, the following steps 1 to 5 are performed to detect the near side projection plane and the back side projection plane.
Step 1: An input image from a fisheye camera is converted into a perspective projection image (FIG. 11).
Step 2: Parallelize only the imaging area of the projected image (bottom of FIG. 12)
Step 3: Extract feature points from the desktop image (upper figure)
Step 4: Perform stereo matching by normalized cross-correlation (bottom of FIG. 13)
Step 5: Estimate the depth of the front surface (bottom of FIG. 13)

  Next, as shown in FIG. 13, a feature point 220 in the desktop image is extracted from the projection image 300, stereo matching is performed by normalized cross-correlation, and the depth of the front surface is estimated. These processes will be described.

  In the stereo matching process here, stereo matching is performed using the parallelized image and the desktop image obtained in the previous stage. The parallelized image has a feature that combines a desktop image and a texture (such as a palm) of the projection surface, and is also affected by the reflection characteristics of the projection surface and the normal direction. Therefore, in this process, first, as a feature point extraction process, only points having two-dimensionally sufficient features are extracted from the desktop image. Here, a feature point is detected using a known “Harris operator”. FIG. 13A shows feature points obtained from a desktop image (image of FIG. 12A).

  The selected feature points are associated by stereo matching using normalized cross-correlation. Strictly speaking, if a texture having a luminance gradient and an antiphase that is included in a feature point accidentally exists on the projection plane, the search may fail in this method. Therefore, a desktop image usually has many feature points. In addition, it is assumed that a projection image including a large amount of texture enough to cancel the projection image is not used because it is not suitable for this interaction technique. If it is within that assumption, it will function normally.

  The image in FIG. 13B is an example of the result of stereo matching, and the parallax obtained by the association is indicated by a black line and a gray line. Black represents the parallax corresponding to the distance of the hand, and gray represents the other parallax. It can also be seen from this figure that the parallax of the feature points on the back projection plane is almost zero pixels. An ellipse 221 represents the dispersion of coordinate values in the parallelized image of feature points having a parallax that matches the hand distance. The center is an average value, the inner ellipse is 0.5σ, and the outer is 1σ. Thus, the feature points on the near side projection plane and the far side projection plane can be separated by the parallax (distance), and the existence range of the near side projection plane can also be obtained.

  Next, an overview of the entire system in which the demonstration experiment was conducted will be given. FIG. 14 is a diagram of a system configuration including the “BOWL ProCam” system and its periphery. As a feature of this system, first, a desktop image obtained by screen dumping is pasted on a polygon as a texture by a 3D graphic engine “OpenGL” and projected by a projector. This has an advantage that an image of an arbitrary application can be projected while performing various geometric corrections. In addition, since the main purpose here is remote cooperative work support, the system configuration is connected to a web service-based network system, and content sharing and communication between workers are also assumed.

In the experiment, the operator mainly performed the following four scenarios (A) to (D) on the map application (using Google Earth).
Scenario (A): An example of an interaction in which a call is started by selecting an icon of one of the collaborators.
Scenario (B): An example of interaction using a magnifying glass metaphor that removes the third floor and confirms the first floor.
Scenario (C): This is an example in which a photograph with position information stored in an SQL server is selected, and character region candidates in the enlarged photograph are further selected and input to the OCR application.
Scenario (D): This is an example in which a CG model acquired by the wearable system of another user stored in the SQL server with position information is selected, rotated, and browsed.

  FIGS. 15 to 17 show user viewpoint videos in scenarios (A) to (C). Projected images (a) before the user takes action as the user viewpoint images, images (b) in which the user presents the hand with the projection surface in front of the user, and further changes the projected image by moving the pointing hand An image (c) is shown. 15 shows the user viewpoint video of scenario (A), FIG. 16 shows the user viewpoint video of scenario (B), and FIG. 17 shows the user viewpoint video of scenario (C). Both the desktop image and the parallelized image have a resolution of 512 × 384 pixels. The block size of the “Harris operator” was 13 × 13 pixels, and the block size of normalized correlation was 19 × 19 pixels. Further, since the parallelized image cannot be strictly parallelized, the stereo matching scanning line has a width of one pixel at the left and right. Experiments were conducted in a general office environment, and fluorescent lights on the ceiling and nearby desk lights were present as ambient light.

  18 to 20 are diagrams showing detection results of the near side projection plane in the three scenarios corresponding to FIGS. 15 to 17. 18 shows the detection result of the near side projection plane in scenario (A), FIG. 19 shows the detection result of the near side projection plane in scenario (B), and FIG. 20 shows the near side projection in scenario (C). The detection result of a surface is shown. A projected image before the user takes action (a), an image (b) in which the user presents a hand as a projection plane, and an image (c) in which the projected image is changed by moving the pointing hand. Shows the result of processing.

  First, detection of the near side projection plane, but in each scenario, as a result of an experiment with 10 images including the hand and 4 images not including the hand, it can be seen that the hand (front side projection plane) is erroneously detected or not detected. There wasn't. Next, the error evaluation about the selection of projection content was performed with respect to 10 images in which the hand was detected. Here, an error is obtained by dividing the distance between the detected coordinates of the center of gravity of the hand and the center coordinates of the content desired to be selected by the user by the diagonal distance of the image. As a result, the average error was 6.6% and the standard deviation was 3.7% (FIGS. 18 to 20).

  FIG. 21 is a diagram illustrating a detection result of the near side projection plane when the depth is changed in the scenario (D). The center image shows the reference depth, the upper image shows the state when the hand is moved before it, and the lower image shows the state when the hand is moved further back. . As shown in FIG. 21, it can be confirmed that the projection plane can be correctly detected even when the depths of the front projection planes are different.

  As described above, by using the small portable terminal according to the present invention, a user who works while moving around the work space can carry or wear it, and the work object or installation in the work space, the object or body of the user can be one of them. Gives work procedures and instructions while projecting complex visual information directly to the part, etc., and further determines the shape of the user's object and part of the user's body, and the work object and installation in the back. By changing the positional relationship, it is possible to provide a small mobile terminal that can select a part of the projected image or change the contents of the projected image, reducing the shortage of skilled workers and safety management problems. Is very beneficial for.

It is a figure explaining the system configuration | structure of the small portable terminal by this invention. It is a figure explaining the control which changes the projection image projected on the projection surface of a near side projection surface and a back side projection surface in the small portable terminal of this invention. It is a figure explaining the Example in case the projection surface projected with the small portable terminal of this invention is unevenness | corrugation. It is a figure explaining an example of the method of detecting a near side projection surface or a back side projection surface by the cast shadow detection method. It is a figure explaining an example of utilization of the small portable terminal by this invention. It is a figure explaining the result of the simulation evaluation from a viewpoint of the assistance of the operation | work accompanying real object operation in order to judge the mounting position of a small portable terminal objectively. It is a figure explaining operation | movement of the light thinning-out means of the modification of a projector. It is a figure explaining operation | movement of the concealment area | region imaging | photography means of another modification of a projector. It is a figure explaining a mode that pattern light is shown to a projection field concealed by a user's hand. It is a figure which shows the relationship between the imaging system of the fish-eye camera of an equidistant projection system, and the projection system of a projector. It is a figure which shows the input image from a fisheye camera, and the image of the result of perspective projection conversion. It is a figure explaining the process of parallelization of a projection image imaging part. It is a figure explaining the process of matching of the feature point of a projection image imaging part. It is a figure of the system configuration containing the BOWL ProCam system and its periphery. It is a figure which shows the user viewpoint image | video in scenario (A). It is a figure which shows the user viewpoint image | video in scenario (B). It is a figure which shows the user viewpoint image | video in scenario (C). It is a figure which shows the detection result of the near side projection surface in scenario (A). It is a figure which shows the detection result of the near side projection surface in scenario (B). It is a figure which shows the detection result of the near side projection surface in scenario (C). It is a figure which shows the detection result of the near side projection surface when changing depth in scenario (D).

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Camera 2 Projector 3 Self-contained sensor 4 Information processing system 5 Projection surface detection means 6 Projection image control means 8 Wireless communication means 12 Ray thinning means 13 Concealed area projection means 30 Cast shadow 31 Boundary line 32 Boundary 33 Cast shadow width 40 Parallel rays 50 Visualization of overall evaluation 51 Visualization of the evaluation of the width of the field of view 52 Visualization of the evaluation of the visibility of the work area 53 Visualization of the stability evaluation during the swinging motion FIG. 54 Stability evaluation during the walking motion Visualization FIG. 55 Small mobile terminal mounting part 60 Partial area 100 Small mobile terminal 200 Front projection surface 201 Back projection surface 202 Cast shadow 203 Projected light arrival space 300 Projected image 301 CG image 302 CG image 303 CG image 304 Icon 305 Face photo 400 Projection image 450 Map information 451 Left box 452 True Box 453 the right of the box 454 along line 500 desk 600 user 601 user viewpoint 700 input image 701 input image 702 input image 703 input image 800 projected source image 801 projected source image 802 projected source image

Claims (6)

A projector for projecting a projection image with a group of rays of parallel rays or rays having a deep focal depth as one pixel;
A camera for photographing a real space including the projected image;
Analyzing a real-space image including a projection image from an image photographed by the camera, and based on the analysis result and the positional relationship between the camera and the projector, the near-side projection plane that is the projection plane closest to the projector And a projection plane detection means for detecting a position in the image and a three-dimensional positional relationship between the projection plane on the front side and the projection plane on the back side in the image. When,
While controlling the projector to project a projected image focused on both the front projection surface and the rear projection surface by the projector, the positional relationship between the front projection surface and the rear projection surface and the change thereof Projection image control means for controlling the content of the projection image according to the situation;
A small portable terminal comprising data processing means for supplying a projection image to the projector. The small portable terminal according to claim 1,
The projector is a laser projector using a two-dimensional scan mirror in which a high-frequency one-dimensional scan mirror and a low-frequency one-dimensional scan mirror are combined.
A small portable terminal, wherein a scanning direction of the high-frequency scanning mirror is substantially parallel to a straight line connecting the optical center of the camera and the optical center of the projector. In the small portable terminal according to claim 1 or 2, further,
A small portable terminal comprising a mounting tool for mounting the small portable terminal on a chest or shoulder of a user. The small portable terminal according to any one of claims 1 to 3, further comprising:
Light beam thinning means for thinning and projecting the number of light beams projected onto a projection surface with a short projection distance with respect to a light beam group for projecting each pixel of the projection image projected from the projector,
The projector projects a projected image under the control of the light thinning means,
A small portable terminal characterized in that a projected image is clearly projected on both the front and back projection planes. The small portable terminal according to any one of claims 1 to 4, further comprising:
It is not observable from the user by being concealed by the front side projection plane, but any of a straight line, a point cloud, a circle, a triangle, or a lattice pattern is applied to the partial area in the back side projection plane that can be observed from the camera. Concealed area projection for measuring the shape of the partial area or the distance between the camera and the partial area by projecting the geometric pattern light by controlling the projector and photographing the pattern light with the camera A small portable terminal comprising: means. The small portable terminal according to any one of claims 1 to 5,
The camera is a fisheye camera, and
The projection surface detection means includes image conversion means for performing image conversion from a fisheye image photographed by the fisheye camera to a perspective projection image,
A small portable terminal, wherein the projection plane detection unit detects a three-dimensional positional relationship between the near side projection plane and the far side projection plane based on the perspective projection image converted by the image conversion unit.