JP6393114B2 - Information processing system, information processing program, information processing method, and imaging apparatus - Google Patents

Description

  The present invention relates to an information processing system, an information processing program, an information processing method, and an imaging apparatus, and particularly relates to an information processing system, an information processing program, an information processing method, and an imaging apparatus that include a camera and a mirror at a position facing the camera. .

  An example of background art is disclosed in Patent Document 1. In the omnidirectional three-dimensional space recognition input device disclosed in Patent Document 1, an imaging unit that captures an omnidirectional image by an optical system including a mirror unit and a camera unit is provided, and an omnidirectional stereo image and a subject A three-dimensional coordinate value representing the shape and structure of is extracted.

JP 2010-256296 A

  However, in this background art, an omnidirectional stereo image and three-dimensional coordinate values representing the shape and structure of a subject are only extracted.

  A main object of the present invention is to provide a novel information processing system, information processing program, information processing method, and imaging apparatus.

  Another object of the present invention is to provide an information processing system and an information processing system capable of executing new information processing by using a partial image corresponding to a curved mirror and a partial image not corresponding to a curved mirror among camera images. A program, an information processing method, and an imaging apparatus are provided.

An information processing system according to a first invention includes a camera, a mirror , a first detection unit , and a contact determination unit . The mirror is installed facing the camera. The first detection unit is configured to detect, based on the first partial image corresponding to the mirror, and the second partial image adjacent to the first partial image and not corresponding to the mirror among the camera images captured by the camera. Detects the position of surrounding objects. The contact determination means determines whether or not the first object image for the object in the first partial image and the boundary between the first partial image and the second partial image are in contact with each other when the mirror side is installed on the installation surface. It is determined whether or not is in contact with the installation surface. When installing the mirror side on the installation surface, the positional relationship between the camera and the mirror and the light so that light traveling along the installation surface in a direction parallel to the installation surface is reflected by at least a part of the mirror. The shape of the mirror is determined.

According to the first invention, since the boundary between the first partial image corresponding to the mirror and the second partial image not corresponding to the mirror corresponds to the installation surface on which the camera is installed in the camera image, the object in the second partial image Whether the object is in contact with the installation surface can be determined according to whether the object is in contact with the boundary .

  A second invention is dependent on the first invention, and the first detection means includes a first position of the first object image for the object in the first partial image and a second object image for the object in the second partial image. The position of the object is detected based on the second position.

  According to the second aspect, the position of the object can be calculated based on the first partial image and the second partial image.

  A third invention is dependent on the first invention and further comprises a second detection means. The second detection means detects at least a boundary between the first partial image and the second partial image and a first positional relationship of the first object image with respect to an object in the first partial image. When the first object image is in contact with the boundary line, it can be seen that the object is in contact with the surface provided with the camera corresponding to the boundary line. The first detection means detects the position of the object according to the first positional relationship detected by the second detection means. That is, when the object is in contact with the surface on which the camera is provided, the height is set to 0, and the two-dimensional coordinates on the surface on which the camera is provided are calculated based on the second position of the second partial image. . When the object is not in contact with the surface on which the camera is provided, the three-dimensional position of the object is calculated based on the first partial image and the second partial image.

  According to the third aspect, the position of the object can be calculated by an appropriate method depending on whether the object is in contact with the surface on which the camera is provided.

  A fourth invention is dependent on the third invention, and the second detection unit further detects a second positional relationship between the reference position of the first partial image and the second object image with respect to the object in the second partial image. The first detection unit detects the position of the object based on the first positional relationship and the second positional relationship detected by the second detection unit.

  According to the fourth aspect, since the second positional relationship is further detected, the position of the object can be calculated by an appropriate method.

  A fifth invention is dependent on the fourth invention, and the first detection means is based on whether the first positional relationship indicates that the first object image is in contact with the boundary line and on the second positional relationship. The position of the object is detected.

  According to the fifth aspect, the position of the object can be calculated by an appropriate method depending on whether the object is in contact with the surface on which the camera corresponding to the boundary line is installed.

  A sixth invention is dependent on the first invention, and further includes third detection means. The third detecting means is a first position of the first object image for an object in the first partial image and an object in the second partial image on a straight line passing through the reference position of the first partial image in the camera image. A second position of the second object image is detected. The first detection means detects the position of the object based on the first position and the second position detected by the third detection means.

  According to the sixth aspect, the position of the object can be calculated based on the first partial image and the second partial image.

The seventh invention is dependent on any one of the first to sixth inventions, and the angle of view of the camera is 180 degrees.

According to the seventh aspect , it is possible to photograph a direction perpendicular to the optical axis of the camera.

The eighth invention is dependent on any one of the first to seventh inventions, and the camera can photograph 360 degrees around the camera.

According to the eighth aspect , it is possible to photograph an object existing around the camera.

A ninth invention is dependent on any one of the first to eighth inventions, and the first partial image and the second partial image include the camera so that an image of an object existing around the camera is included. The relative positional relationship of the mirror and the shape of the mirror are determined.

According to the ninth aspect , the position of the object can be detected based on the first partial image and the second partial image.

A tenth invention is dependent on any one of the first to ninth inventions, and the mirror has a curved surface.

According to the tenth aspect , an image around the camera can also be included in the first partial image.

An eleventh invention is according to the tenth invention, and the curved surface is a hyperboloid, a conical surface, an elliptical surface, or a composite surface of any two of these protruding toward the camera.

According to the eleventh aspect , a curved mirror can be used according to the application.

A twelfth invention is dependent on any one of the first to eleventh inventions, and the imaging device including the camera and the mirror is portable.

According to the twelfth aspect, the position of the object can be detected by installing the imaging device on an arbitrary installation surface.

A thirteenth invention is dependent on any one of the first to twelfth inventions, and the imaging device including the camera and the mirror is formed in a cylindrical shape.

According to the thirteenth aspect , processing for correcting an image due to the shape of the imaging device itself is unnecessary.

A fourteenth invention is according to any one of the first to thirteenth inventions, wherein the mirror is a curved mirror, and the first detection means includes an image obtained by correcting the first partial image with a predetermined algorithm, and the second portion. The position of the object is detected based on the image.

According to the fourteenth aspect , since the distortion of the image by the curved mirror is corrected, the position of the object can be detected without reducing or eliminating the error due to the distortion.

A fifteenth aspect of the invention is an information processing program executed by a computer of an information processing system including a camera and a mirror that is installed facing the camera, and when the mirror side of the information processing system is installed on the installation surface, The positional relationship between the camera and the mirror and the shape of the mirror are determined so that light traveling along the installation surface in a direction parallel to the installation surface is reflected by at least a part of the mirror, and the computer is Of the camera images captured by the camera, based on the first partial image corresponding to the mirror and the second partial image adjacent to the first partial image and not corresponding to the mirror, an object existing around the camera is displayed. detecting means for detecting a position, and when to set up the mirror side of the information processing system to the installation surface, the first object image for the object in the first partial image If, on whether the boundary of the first partial image and the second partial image are in contact, the object is to function as a contact determination means for determining whether or not to contact with the installation surface, an information processing program.

In the fifteenth invention, as in the first invention, it can be determined whether the object is in contact with the installation surface according to whether the object in the second partial image is in contact with the boundary. .

A sixteenth aspect of the invention is an information processing method for an information processing system including a camera and a mirror that is installed opposite to the camera, and is parallel to the installation surface when the mirror side of the information processing system is installed on the installation surface. The positional relationship between the camera and the mirror and the shape of the mirror are determined so that the light traveling along the installation surface in any direction is reflected by at least a part of the mirror, and (a) the image is captured by the camera A step of detecting a position of an object existing around the camera based on a first partial image corresponding to a mirror and a second partial image adjacent to the first partial image and not corresponding to a mirror among the camera images. If, (b) when is installed mirror side of the information processing system to the installation surface, and a first object image of the object in the first partial image, of the first partial image and the second partial image On whether the field Metropolitan contacting includes determining whether an object is in contact with the installation surface, an information processing method.

In the sixteenth invention, as in the first invention, it can be determined whether the object is in contact with the installation surface according to whether the object in the second partial image is in contact with the boundary. .

In a seventeenth aspect, based on a camera, a mirror installed facing the camera, and a camera image taken by the camera, it is determined whether an object existing around the camera is in contact with a predetermined surface. The camera and the mirror so that the camera image captured by the camera includes a first partial image corresponding to the mirror and a second partial image adjacent to the first partial image and not corresponding to the mirror. So that the light traveling along the predetermined surface in a direction parallel to the predetermined surface is reflected by at least a part of the mirror when the mirror side is installed on the predetermined surface. A positional relationship between the mirror and the shape of the mirror is determined, and when the contact determination unit sets the mirror side on a predetermined surface, the first object image of the object in the first partial image; On whether the boundary of the first partial image and the second partial image are in contact, the object determines whether contact with the predetermined surface, an information processing system.

According to the seventeenth aspect, it is possible to easily determine whether or not an object has contacted a predetermined surface.

An eighteenth invention is around 360 degrees enable photographing a wide-angle camera, disposed opposite the wide angle camera is parallel to the predetermined plane, the shape becomes a ray traveling direction along the predetermined plane Based on the formed mirror and a camera image taken by the wide-angle camera, a contact determination unit that determines whether an object existing around the wide-angle camera is in contact with a predetermined surface is provided. a first partial image corresponding to, the first portion adjacent to the image, to take pictures of the camera image containing the second partial image that does not correspond to the mirror, arranged wide-angle camera and a mirror, the contact determination means, a mirror-side Is placed on a predetermined surface, the object is determined by whether the first object image of the object in the first partial image is in contact with the boundary between the first partial image and the second partial image. It determines whether contact with an imaging device.

According to the eighteenth aspect, it is possible to capture a camera image that can easily determine whether or not an object has touched a predetermined surface.

According to this invention, in the camera image, the boundary between the first partial image corresponding to the mirror and the second partial image not corresponding to the mirror corresponds to the installation surface on which the camera is installed. Whether the object is in contact with the installation surface can be determined according to whether the object is in contact with the boundary .

  The above object, other objects, features, and advantages of the present invention will become more apparent from the following detailed description of embodiments with reference to the drawings.

FIG. 1 is an illustrative view showing an information processing system according to an embodiment of the present invention. FIG. 2 is an illustrative view showing one example of an external configuration of the imaging apparatus shown in FIG. FIG. 3 is a block diagram showing an electrical configuration of a camera unit included in the imaging apparatus shown in FIG. FIG. 4A is an illustrative view for explaining a photographing range of the imaging apparatus, and FIG. 4B is an illustrative view showing an example of a photographed image of the imaging apparatus. FIG. 5A is an illustrative view showing an example of a captured image when the installation surface on which the imaging device is installed is touched with a finger, and FIG. 5B is a diagram where the installation surface on which the imaging device is installed is touched with a finger. It is an illustration figure which shows an example of the picked-up image when it has not carried out. FIG. 6A is another example of a captured image of the imaging apparatus, and FIG. 6B is an explanatory diagram for explaining a method of calculating the position of a finger when touched. FIG. 7A is another example of a captured image of the imaging apparatus, and FIG. 7B is an explanatory diagram for explaining a method of calculating the finger position when not touching. FIG. 8 is an illustrative view showing one example of a memory map of a RAM built in the computer shown in FIG. FIG. 9 is a flowchart showing a part of the position detection processing of the CPU built in the computer shown in FIG. FIG. 10 is another part of the position detection process of the CPU built in the computer shown in FIG. 1, and is a flowchart subsequent to FIG. FIG. 11 is an illustrative view showing an example of position detection of another object using the information processing system of the embodiment. FIG. 12 is an illustrative view showing one example of a user interface using the information processing system of the embodiment. FIG. 13 is an illustrative view showing an example in which a dedicated display is provided on the imaging apparatus of the embodiment. FIG. 14 is an illustrative view showing an example C in which a display is detachably provided on the image pickup apparatus of the embodiment. FIG. 15A is an illustrative view showing an example in which a controller is configured by using an information processing system and a resinous operation unit, and FIG. 15B is an illustrative view showing a display example of a display window provided in the operation unit. It is. FIG. 16A is an illustrative view showing a state in which the imaging device is installed on the installation surface upside down as in the case of the example, and FIG. 16B was taken in the state shown in FIG. It is an illustration figure which shows an example of a picked-up image. 17A is an illustrative view showing an elliptical curved mirror, FIG. 17B is an illustrative view showing a conical curved mirror, and FIG. 17C is an example of a composite curved mirror. FIG. 17D is an illustrative view showing another example of a curved curved mirror having a composite surface. FIG. 18A is an illustrative view showing a schematic configuration diagram of another example of the information processing system, and FIG. 18B is an illustrative view showing an example of a photographed image of the camera shown in FIG. 18A. FIG. 19A is an illustrative view showing a schematic configuration diagram of another example of the information processing system, and FIG. 19B is an illustrative view showing an example of a photographed image of the camera shown in FIG. 19A. FIG. 20A is an illustrative view showing an example of an external configuration of a projector apparatus, and FIG. 20B is an illustrative view showing an example of an external configuration of an electronic apparatus integrally configured with an imaging apparatus and a projector apparatus. .

  Referring to FIG. 1, an information processing system 10 according to an embodiment of the present invention includes an imaging device 12 and a computer 14, and the imaging device 12 and the computer 14 are connected so as to be capable of wireless communication. However, the imaging device 12 and the computer 14 may be connected so as to be communicable by wire.

  As can be seen from FIG. 2, the imaging device 12 includes a camera unit 12a, a mirror portion 12b, and a fixing member 12c for fixing them, and is formed in a cylindrical shape whose both ends are closed by the camera unit 12a and the mirror portion 12b. Is done.

  The camera unit 12a is formed in a circular shape when viewed from above, and includes a camera 20 using an image sensor such as a CCD or a CMOS. As will be described later, the camera 20 is electrically connected to electronic components that perform imaging processing and communication processing, and such electronic components and the like are housed in the housing of the camera unit 12a. The mirror portion 12 b is formed in a circular shape when viewed from above, and includes a curved mirror 30 and a fixing member 32 that fixes the curved mirror 30. The curved mirror 30 is formed in a circular shape when viewed from above, and is provided to face the camera 20. The curved mirror 30 has a convex surface (in this embodiment, a hyperboloid) that protrudes toward the camera 20 side. The fixing member 12c has a cylindrical shape and is formed of a transparent resin or glass. The height (length) of the fixing member 12c is determined to be a predetermined length. As will be described later, when the imaging device 12 is placed on a predetermined surface, the light traveling along the predetermined surface in the light ray direction parallel to the predetermined surface is reflected at the end of the curved mirror 30. In addition, the positional relationship between the camera 20 and the curved mirror 30 and the shape (surface shape) of the curved mirror 30 are determined. Therefore, the height of the fixing member 12c is determined so as to have such a positional relationship. However, the light reflected by the curved mirror 30 enters the image sensor via the wide-angle lens 22 of the camera 20.

  FIG. 3 is a block diagram showing an electrical configuration of the camera unit 12a. The camera unit 12a includes a CPU 100. The CPU 100 is connected to the above-described camera 20 via the internal bus 102, and is connected to the ROM 104, the RAM 106, and the communication module 108.

  The CPU 100 governs overall control of the camera unit 12a. The ROM 104 stores a control program for the camera unit 12a. The RAM 106 is used as a work area and a buffer area for the CPU 100. The communication module 108 has a function of connecting to a wireless LAN, for example. Therefore, the imaging device 12 can communicate with an external computer (computer 14 in this embodiment) directly or via a network.

  Returning to FIG. 1, the computer 14 is a general-purpose personal computer, a server, or the like. In this embodiment, the computer 14 functions as a host computer of the imaging device 12. Although detailed description is omitted, the computer 14 includes components such as a communication module and an HDD together with the CPU 14a and the RAM 14b.

  For example, the imaging device 12 is placed or installed on a predetermined surface such as a desk or a wall surface (including a white board), and the computer 14 calculates the position of an object (object) existing around the imaging device 12. (To detect. In this embodiment, it is determined whether or not the user's finger is in contact (touch) with a predetermined surface, and the position of the finger (three-dimensional position in the real space) is calculated.

  In this embodiment, the position of the object existing around the imaging device 12 is calculated. However, it is simply determined whether the object existing around the imaging device 12 is in contact with a predetermined surface. It can also be determined whether an object exists around the device 12.

  FIG. 4A shows a part of an optical path of light (light beam) incident on the wide-angle lens 22 of the camera 20 when the imaging device 12 is installed on a certain installation surface. In this embodiment, the angle of view of the camera 20 is set to 180 degrees. Therefore, as indicated by the optical path L 1, light perpendicular to the optical axis of the camera 20 is also incident on the image sensor via the wide-angle lens 22. The light reflected by the curved mirror 30 is also incident on the image sensor through the wide angle lens 22. In this embodiment, as described above, the curved mirror 30 has a hyperboloid protruding toward the camera 20 and is parallel to the installation surface, and the light traveling along the installation surface is also imaged via the wide-angle lens 22. It is made a shape that can enter the element. However, as indicated by the optical path L2, the light that is parallel to the installation surface and travels along the installation surface is reflected at the end (outermost circumference) of the curved mirror 30.

  Therefore, in the captured image captured by the camera 20, the partial image C1 (see FIG. 4B) corresponding to the curved mirror 30 shows the camera 20 itself at the center, and the curved mirror 30 from the center. The environment around the imaging device 12 is captured up to the end. The periphery of the imaging device 12 means all directions (360 ° direction) with respect to the side surface direction of the cylinder. The same applies hereinafter. In addition, in the partial image C2 (see FIG. 4B) corresponding to a portion other than the curved mirror 30, the environment around the fixing member 32 of the curved mirror 30 and the imaging device 12 (side surface direction of the cylinder) is shown.

  The camera 20 has a wide-angle lens 22, and an object image is formed on an image sensor (image sensor) by the wide-angle lens 22. The imaging device converts an optical image corresponding to the formed subject image into an electrical signal, and thereby a captured image signal is obtained. Predetermined image processing such as A / D conversion, YUV conversion, gamma correction, and color correction is performed on the captured image signal, thereby generating captured image data corresponding to the captured image. This photographed image data is given from the camera 20 to the CPU 100, and the CPU 100 temporarily stores the photographed image data in the RAM 106, and then transmits it to the computer 14 using the communication module 108.

  FIG. 4B illustrates an example of a captured image (camera image) of the camera 20 provided in the imaging device 12 illustrated in FIG. The captured image includes a circular image indicated by a double circle. Among the captured images, the circular partial image C1 is an image (mirror image) of the curved mirror 30, and the donut-shaped partial image C2 adjacent to and surrounding the partial image C1 is the curved mirror 30. It is an image about parts other than. That is, the partial image C <b> 2 is not an object image reflected by the curved mirror 30 but an image of an object image directly captured by the camera 20.

  As described above, in the imaging device 12 of this embodiment, both the partial image C1 and the partial image C2 include the images of the objects existing around the imaging device 12 (FIGS. 2 and 5A). (See (B)), the positional relationship between the camera 20 and the curved mirror 30 and the shape of the curved mirror 30 are determined.

  In FIG. 4B, specific contents of the partial image C1 and the partial image C2 are omitted for simplicity. Actually, an object (for example, a desk, a wall, a door) and a person existing in a room where the imaging device 12 is arranged are photographed, and the photographed image is displayed. The same applies to other captured images described later.

  In the captured image, a square frame larger than the partial image C2 corresponds to the entire range of the imaging surface of the imaging element of the camera 20. In addition, of the entire range of captured images, the range that is not captured by the camera 20 provided in the imaging apparatus 12 of this embodiment (the range that exceeds the angle of view) is black, but is hatched in the drawing. . In other drawings, a black range in the captured image is omitted.

  In this embodiment, by using the partial image C1 and the partial image C2, an object (for example, a user's finger) existing around the imaging device 12 contacts (touches) the installation surface on which the imaging device 12 is installed. ) And the position of the object (three-dimensional position in the real space) can be calculated. However, the position of the object is a three-dimensional position in the local coordinate system calculated based on a predetermined position on the installation surface on which the imaging device 12 is installed (FIGS. 6A and 7A). reference). For example, the predetermined position serving as a reference is set at the intersection of the optical axis of the camera 20 and the installation surface, but is not limited to this. However, based on the intersection of the optical axis of the camera 20 and the installation surface, the calculation of the three-dimensional position of the object is relatively simple. In addition, the directions of the X axis and the Y axis in the local coordinate system coincide with the directions of the X axis (horizontal) and the Y axis (vertical) of the imaging surface (imaging device) of the camera 20, respectively.

  5A is an example of a captured image when the user touches the surface on which the imaging device 12 is installed with a finger (when touched), and FIG. 5B is an example of the captured image when the user is a finger. It is an example of the picked-up image when not touching the surface which installed (at the time of non-touch). However, in FIGS. 5A and 5B, only a part of the finger image in the partial image C1 is shown.

  As shown in FIG. 5A, at the time of touching, the finger image displayed in the partial image C1 touches the end of the curved mirror 30 image. Further, as shown in FIG. 5B, the finger image displayed in the partial image C <b> 1 is separated from the end of the image of the curved mirror 30 when not touched. This is because, as described above, the light is parallel to the installation surface, and the light traveling along the installation surface is reflected by the end of the curved mirror 30 and is incident on the image sensor through the wide-angle lens 22. .

  Next, a method for calculating the three-dimensional position of the local coordinate system for each of touching and non-touching will be described. For example, the XY plane of the local coordinate system is set to the installation surface on which the imaging device 12 is installed, and the center is set to the intersection of the center of the captured image, that is, the optical axis of the camera 20 and the installation surface. Further, the horizontal direction of the captured image is set to the X axis, and the vertical direction of the captured image is set to the Y axis. Then, the Z axis is set in a direction perpendicular to the XY plane.

  6A and 6B show the local coordinate system set in the partial image C1 and the partial image C2 when the installation surface of the imaging device 12 is touched (when touched) and the partial image C1. The position U of the object image (object image) and the position T of the object image in the partial image C2 are shown. However, the position of the object image is a position that is in contact with or closest to the installation surface of the imaging device 12, and is specifically a position corresponding to a fingertip.

  Here, for simplicity, the case where the position U and the position T of the object image are on the X axis will be described. However, when the position U and the position T are not on the X axis, the two-dimensional coordinates on the installation surface (XY plane) are based on the X axis and the straight line passing through the position U and the position T. It is expressed using an angle (rotation angle of local coordinates) θ. The same applies to FIGS. 7A and 7B described later.

  Here, it is assumed that the wide-angle lens 22 of the camera 20 is not distorted, and it is assumed that a conical curved mirror 30 is provided instead of the hyperbolic curved mirror 30. Further, it is assumed that the installation surface of the imaging device 12 and the end (outermost circumference) of the curved mirror 30 have the same height. The same applies to FIGS. 7A and 7B described later. Actually, distortion is also generated in the captured image due to distortion of the wide-angle lens 22 and the curved mirror 30, and thus the three-dimensional position is calculated after correcting the distortion. However, distortion correction may also be performed when the three-dimensional position is calculated. Therefore, it is possible to detect the three-dimensional position of the object while reducing or eliminating errors due to distortion.

  As shown in FIG. 6A, when the object image (position U) in the partial image C1 is in contact with the end of the partial image C1, the user touches the installation surface with a finger as described above. It is determined that Therefore, in this case, the height (Z coordinate) of the object in the local coordinate system is 0 (z = 0).

  In this case, the three-dimensional coordinates in the local coordinate system are calculated according to Equation 1. However, as described above, since the user touches the installation surface with a finger, the Z coordinate need not be calculated. Further, as shown in FIG. 6A, the length from the origin (center) of the local coordinate system to the position T of the object image in the partial image C2 is r (pixel), and the distance from the center to the outer periphery of the partial image C2 is set. The length is R (pixel). Also, as shown in FIG. 6B, the angle of view of the camera 20 is α, and the distance (fixed length (height)) h (mm) from the point M that is the base point of the angle of view α to the installation surface. It is as. The distance p (mm) to the actual object position T is calculated, and the three-dimensional position of the object is calculated using this distance.

  However, as described above, the angle of view α of the camera 20 of this embodiment is 180 degrees. However, if this is used as it is, the distance p (mm) becomes infinite in Equation 1, for example, , Α is set to 179 degrees or a value in the vicinity thereof (value less than 180 degrees). However, when the position of the object is calculated only within the range of the installation surface on which the imaging apparatus 12 is installed, the value of α is appropriately set according to the size of the installation surface (for example, the size of the desk). May be. Therefore, the angle of view α need not be limited to 180 degrees.

  As shown in FIG. 7A, when the object image (position U) in the partial image C1 is not in contact with the end of the partial image C1, the user touches the installation surface with a finger as described above. It is determined that it is not. In this case, as shown in FIGS. 7 (A) and 7 (B), the image appears to be farther from the center than the actual position T of the object (in real space). In such a case, the three-dimensional position of the object is calculated according to Equation 2. However, the radius (distance from the center to the end of the mirror image) of the partial image C1 is b (pixel), and the distance between the position U of the object image in the partial image C1 and the end of the partial image C1 is a (pixel). To do. Further, the height of the curved mirror 30 is d (mm), the distance from the center in the partial image C1 to the position T ′ of the apparent object image in the partial image C2 is r ′ (pixel), and in the partial image C1 The distance from the center to the position T ′ of the apparent object image in the partial image C2 is p ′ (mm).

  Thus, the method of calculating the three-dimensional position of the object in the local coordinate system differs depending on whether the object is in contact with the installation surface of the imaging device 12 or not. In this embodiment, such calculation processing is executed by the computer 14 that has received the captured image data from the imaging device 12.

  The imaging command is given from the computer 14 to the imaging device 12, and in response to this, the imaging device 12 executes imaging processing and transmits captured image data corresponding to the captured image to the computer 14. Alternatively, the imaging device 12 may perform imaging processing at predetermined time intervals and transmit the captured image data corresponding to the captured image to the computer 14 regardless of the imaging command from the computer 14.

  In this embodiment, the three-dimensional coordinates of the object in the local coordinate system are calculated using the center of the partial image C1 (captured image) as a reference position, but the reference position is a point on the outer periphery of the partial image C. That is, it may be a point on the boundary between the partial image C1 and the partial image C2. At this time, among the points on the outer periphery of the partial image C1, the position of the point having the shortest distance from the position U of the object image is the reference position. Similarly, the position of a point on the outer periphery of the image of the camera 20 itself included in the partial image C1 may be set as the reference position.

  FIG. 8 shows an example of the memory map 300 of the RAM 14b built in the computer 14 shown in FIG. As shown in FIG. 8, the RAM 14 b includes a program storage area 302 and a data storage area 304.

  An information processing program is stored in the program storage area 302, and the information processing program includes a communication program 302a, an image acquisition program 302b, a distortion correction program 302c, a contact determination program 302d, a position detection program 302e, and the like.

  The communication program 302a is a program for communicating with the imaging device 12 and other computers. The image acquisition program 302 b is a program for receiving captured image data transmitted from the imaging device 12 and storing it in the data storage area 304.

  The distortion correction program 302c is a program for correcting distortion of a captured image. This process corrects distortion caused by the wide-angle lens 22 and distortion caused by the curved mirror 30, and the pixel position is corrected. The coefficient for correction is empirically obtained in advance by a test or the like according to the wide-angle lens 22 and the curved mirror 30 to be used for each pixel.

  The contact determination program 302d is a program for determining whether the object is in contact with the installation surface of the imaging device 12. Although detailed description is omitted, for example, when the imaging device 12 is installed on the installation surface, a captured image (herein referred to as “reference image”) is acquired in a state where the object is not reflected, Whether or not the object exists around the imaging device 12 (the presence or absence of the object) can be detected by a difference between a reference image acquired in advance and a captured image captured thereafter. However, in this embodiment, since the position of the user's finger is calculated, the presence or absence of an object (finger) may be detected by skin color detection.

  The position detection program 302e is a program for calculating a three-dimensional position in the real space of an object existing around the imaging device 12. As described above, the position detection program 302e calculates the three-dimensional position of the local coordinate system according to the determination result of the contact determination program 302d.

  Although illustration is omitted, the program storage area 302 stores programs for various applications executed by the computer 14.

  The data storage area 304 stores photographed image data 304a, position data 304b, and reference image data 304c.

  The photographed image data 304a is photographed image data received from the imaging device 12 or photographed image data obtained by correcting distortion of the received photographed image data. The position data 304b is position (coordinate) data regarding a three-dimensional position of an object existing around the imaging device 12 calculated based on the captured image data. Therefore, when a plurality of objects exist around the imaging device 12, a three-dimensional position is calculated for each object, and position data for each object is stored. The reference image data 304c is image data corresponding to the above-described reference image.

  Although not shown, the data storage area 304 stores other data necessary for executing a program such as an information processing program, and is provided with a flag and a timer (counter).

  9 and 10 are flowcharts showing an example of the position calculation process of the CPU 14a built in the computer 14 shown in FIG. As shown in FIG. 9, when the position calculation process is started, the CPU 14a acquires a captured image in step S1. That is, the CPU 14 a receives the captured image data transmitted from the imaging device 12 and stores it in the data storage area 304.

  In the next step S3, distortion of the captured image is corrected. Therefore, the captured image data stored in the data storage area 304 is updated after correcting the distortion. In the next step S5, an object is detected. Here, the object is detected from the difference between the reference image and the captured image. However, when a plurality (N) of objects are detected, each object is managed so as to be identifiable in the subsequent processing. For example, a serial number (1, 2,..., N) is assigned to each object as identification information.

  The reference image data 304c corresponding to the reference image is acquired in advance and stored in the data storage area 304.

  In a succeeding step S7, it is determined whether or not there is an object. If “NO” in the step S7, that is, if there is no object, the position calculating process is ended as it is. On the other hand, if “YES” in the step S7, that is, if there is an object, the variable i is initialized in a step S9 (i = 1).

  Subsequently, in step S11 shown in FIG. 10, it is determined whether or not the object i is in contact with the installation surface based on the mirror image (partial image C1). That is, the CPU 14a determines whether the object image of the object i of interest is in contact with the end (outer periphery) of the partial image C1.

  In the next step S13, it is determined whether or not the determination result is contact. If “YES” in the step S13, that is, if the object i is in contact with the installation surface, the three-dimensional position of the object i is determined based on the image (partial image C2) outside the mirror image in a step S15. calculate. Here, the CPU 14a calculates the three-dimensional position of the object i according to Equation 1. However, since the object i is in contact with the installation surface, the Z coordinate (height) is zero. At this time, the calculated position data of the three-dimensional position is stored in the data storage area 304 in association with the identification information of the object i. The same applies to step S17 described later.

  On the other hand, if “NO” in the step S13, that is, if the object i is not in contact with the installation surface, in a step S17, the mirror image (partial image C1) and the image outside the mirror image (partial image C2) are displayed. Based on this, the three-dimensional position of the object i is calculated. Here, the CPU 14a calculates the three-dimensional position of the object i according to Equation 2.

  Then, in step S19, it is determined whether or not the variable i is N or more (i ≧ N). That is, the CPU 14a performs contact determination for all the objects i and determines whether or not a three-dimensional position has been calculated.

  If “NO” in the step S19, that is, if the variable i is less than N, in the step S21, the variable i is incremented by 1 (i = i + 1), and the processing of the steps S11 to S17 is executed for the next object i. To do. On the other hand, if “YES” in the step S19, that is, if the variable i is equal to or greater than N, the position calculating process is ended.

  Note that the position calculation processing shown in FIGS. 9 and 10 is executed in response to a request from the application or is executed at predetermined time intervals. In addition, the computer 14 (CPU 14a) may transmit a shooting command to the imaging device 12 prior to execution of the position calculation process.

  According to this embodiment, a curved mirror is arranged facing a camera equipped with a wide-angle lens, and an imaging device is installed between the image of the mirror part included in the photographed image of the camera and the image outside the mirror part adjacent thereto. Since it corresponds to the installation surface, the three-dimensional position of the object can be calculated by an appropriate calculation method depending on whether or not the object in the image of the mirror part is in contact with the end (outer periphery) of the image, that is, the boundary. .

  Further, according to this embodiment, since it is determined whether or not the object is in contact with the installation surface of the imaging apparatus, it can be determined whether or not the installation surface is touched with the user's finger. In such a case, an information processing system including an imaging device and a computer can be used as a pointing device.

  In this embodiment, the captured image data is transmitted from the image capturing apparatus to an external computer, and the touch determination and the position calculation are performed by the computer. However, the present invention is not limited to this. If the processing capacity of the CPU 100 and the capacity of the RAM 106 are sufficient, contact determination and position calculation may be performed by the imaging device 12 (camera unit 12a). In such a case, the single imaging device 12 (camera unit 12a) functions as an information processing device. In such a case, the information processing program, data, and the like as described above are stored in the RAM 106, and the position calculation processing shown in FIGS.

  Further, in this embodiment, it has been described that the imaging apparatus and the computer are connected so as to be communicable wirelessly or by wire, but they may be connected via a network such as the Internet. In such a case, the computer may acquire captured image data from a plurality of imaging devices and perform contact determination and position calculation for each imaging device.

  Furthermore, in this embodiment, it is assumed that the environment in which the imaging device is arranged is illuminated, but the mirror unit 12b and the imaging device 12 are installed so that the object and the position of the object can be detected even in a dark place. You may make it provide the imaging device with the illuminating device which illuminates a surface with white light.

  Furthermore, in this embodiment, the case where a camera using an image sensor such as a CCD or CMOS is used has been described. However, the present invention is not limited to this. For example, an IR camera can be used. In the case of using an IR camera, it is possible to reduce the influence of the brightness of the place where the imaging device is installed when detecting the objects existing around the imaging device and the positions of the objects. Further, if the IR camera is provided with an IR-LED that emits infrared rays on the installation surface of the mirror unit 12b and the imaging device 12, the object and the position of the object existing around the imaging device can be detected even in a dark place. Can do.

  In this embodiment, the height of the installation surface of the imaging device and the end (outermost circumference) of the curved mirror is assumed to be the same. However, the present invention is not limited to this. When the height of the edge of the curved mirror is different from the installation surface of the imaging device, the finger image is a preset reference line in the curved mirror image (partial image C1) (the boundary between the partial image C1 and the partial image C2). Whether or not the user's finger is in contact with (touching) the installation surface may be determined based on whether or not it is touching.

  Furthermore, in this embodiment, the imaging device 12 is placed on a horizontal installation surface such as a desk, but the present invention is not limited to this. The imaging device 12 may be pulled on a vertical surface such as a wall surface or a whiteboard. For example, it may be attached (installed) to a wall surface or a white board using a double-sided tape or a magnet. For example, when a magnet is used, the fixing member 32 may be made of a magnet. In this way, it is possible to calculate (detect) the presence or absence of contact of an object with a vertical surface such as a wall surface or a whiteboard, and the three-dimensional position of the object when the surface is an XY plane.

  Furthermore, in this embodiment, when the imaging device is arranged so that the curved mirror side is installed on the installation surface, the light parallel to the installation surface is reflected over the entire circumference, but the present invention is not limited to this. There is no need to The positional relationship between the camera and the curved mirror and the shape of the curved mirror may be determined so that light parallel to the installation surface is reflected by a part of the curved mirror (for example, ¼ turn, half turn, etc.). For example, if the part reflecting light parallel to the installation surface is a quarter of the curved mirror, install the imaging device so that the other part matches the corner of one of the whiteboards. For example, the presence or absence of an object on the whiteboard and the position of the object can be detected based on the photographed image. In addition, when the part that reflects light parallel to the installation surface is a half circumference of the curved mirror, the imaging device is placed on the side or the side so that the part other than the part is directed to the outside of any side of the whiteboard. If it is installed in the vicinity of the side, the presence or absence of an object on the whiteboard and the position of the object can be detected based on the photographed image.

  In this embodiment, the user's finger is detected as the object. However, the present invention is not limited to this. For example, as shown in FIG. 11, another object such as a figure may be detected. However, in FIG. 11, as an example, two figures 200 and 202 are arranged around the imaging device 12. The method for determining the presence / absence of an object using the captured image, the method for determining whether the object is in contact with the installation surface, and the method for calculating the position of the object are the same as in the above-described embodiment. However, when determining the presence or absence of an object, the presence or absence of the color or pattern of the figure (200, 202) may be detected in the captured image. However, instead of or together with the color or pattern, specific characters or figures attached to the figure (200, 202) are recognized from the photographed image to detect the presence of the figure (200, 202). May be. In addition, the type of the figure (200, 202) can be specified by recognizing a specific color, pattern, character or figure, or a combination of any two or more thereof.

  Furthermore, by using a sheet 210 as shown in FIG. 12, the information processing system 10 can be used as a user interface (UI). For example, a button design corresponding to an operation button is drawn on the paper 210. Further, the orientation and position of the imaging device 12 and the paper 210 are set to a predetermined relative orientation and position. As shown in the above-described embodiment, the computer 14 determines whether the user's finger has touched the installation surface of the imaging device 12. When determining that the computer 14 has touched, the computer 14 calculates the position of the finger that is in contact, and determines the touched button symbol on the paper 210. Then, the computer 14 outputs a command set to the button design (operation button) determined to be in contact with the user's finger to the application.

  Further, by changing the content drawn on the paper 210, it can be used as another UI. For example, a map similar to the map of the role-playing game executed in the virtual game space is drawn on the sheet 210, and the figures (200, 202) as shown in FIG. The figure (200, 202) is moved above. The computer 14 calculates the position of the figure (200, 202) on the sheet 210 and reflects the calculated position on the position of a character such as a player character in the virtual game space.

  As shown in FIG. 13, a dedicated display 220 can be provided and a touch operation can be executed using the display 220. However, FIG. 13 illustrates the hands of two different users. For example, the display 220 is connected to the computer 14, and the display is controlled by a display controller built in the computer 14 under the instruction of the CPU 14a. The display 220 in the example illustrated in FIG. 13 is formed in a cylindrical shape and is installed on the imaging device 12. For example, the display area of the display 220 can be used by one user or a plurality of users without being divided. In addition, the display area of the display 220 can be divided and used individually according to the number of users.

  Moreover, as shown in FIG. 14, you may make it provide the imaging device 12 so that a display (230a, 232a) can be attached or detached. However, FIG. 14 shows the hands of two different users. For example, in the example illustrated in FIG. 14, electronic devices (hereinafter referred to as “portable terminals”) (230, 232) with integrated displays (230a, 232a) such as smartphones and tablet PCs are installed on the imaging device 12. Is done. The portable terminals (230, 232) are communicable with the computer 14, execute application processing according to operation data transmitted from the computer 14, and display the execution results on the displays (230a, 232a). That is, in this case, the information processing system 10 including the imaging device 12 and the computer 14 is used as a common pointing device, and the computer 14 detects a touch operation for each user and responds to the detected touch operation. The operation data is transmitted (input) to the corresponding portable terminal. That is, in such a case, the information processing system 10 can be used as an input device of the mobile terminal (230, 232), and input can be performed while viewing the entire screen of the mobile terminal (230, 232).

  However, since the information processing system 10 (imaging device 12) is used in common by a plurality of users, a process for associating the user with the mobile terminals (230, 232) is necessary prior to use.

  Furthermore, as illustrated in FIG. 15A, the information processing system 10 can be used as a controller by using a resin-based operation unit 240 that does not have an electrical configuration. For example, the operation unit 240 is provided with a handle 240a and a lever 240b, and a display window 240c as shown in FIG. 15B is provided on the back side of the operation unit 240 shown in FIG. As shown in FIG. 15B, a marker 242 that moves to the left and right in conjunction with the operation of the handle 240a is disposed in the casing of the operation unit 240 at a position where the display window 240c is provided. Specifically, when the user turns the handle 240a to the right, the marker 242 is moved to the right in FIG. Therefore, the length of the marker 242 visible through the display window 240c is shortened. On the other hand, when the user turns the handle 240a to the left, the marker 242 is moved to the left in FIG. Therefore, the length of the marker 242 visible through the display window 240c is increased. The computer 14 determines which of the left and right handles 240a corresponds to the vertical length of the marker 242 visible through the display window 240c with respect to the vertical length of the display window 240 in the captured image captured by the imaging device 12. It is possible to know how much it is cut (rotation angle).

  Although not shown in the drawing, the lever 240b is similarly provided with a marker (different from the marker 242) that moves up and down in conjunction with the inclination of the lever 240b. This marker is also visible through the display window (different from the display unit 240c). Therefore, the tilt direction and the tilt amount of the lever 240b can be known from the captured image.

  Therefore, the computer 14 detects the direction in which the handle 240a is turned and the amount by which the handle 240a is cut from the captured image, and also detects the direction and amount of inclination of the lever 240b, and uses the detected information as in a car racing game. Give to the application. For example, when the operation unit 240 is used, an operation other than the touch operation can be performed by the information processing system 10. In addition, since there is no need to provide an electrical configuration in the operation unit 240, it is not necessary to supply power such as incorporating a battery (primary battery or secondary battery).

  In the example shown in FIG. 16A, the imaging device 12 is installed on the installation surface upside down as shown in the above-described embodiment. Even in this case, since the camera 20 having the angle of view of 180 degrees is provided, it can be determined whether the user is touching the installation surface based on the captured image. This is because the angle of view of the camera 20 is 180 degrees, and when the imaging device 12 is placed on a horizontal installation surface, light in a direction parallel to the installation surface can be captured (acquired).

  For example, as shown in FIG. 16A, when the user touches the installation surface, a captured image as shown in FIG. 16B is obtained. In this case, whether or not the user has touched the installation surface is determined based on whether or not the image of the user's finger is in contact with the end (outer periphery) of the partial image C2 in the partial image C2.

  FIG. 16B shows the position of the captured image when the user is touching the installation surface of the imaging device 12. In the partial image C2, an image of a portion corresponding to the user's fingertip is not captured. This is because, as shown in FIG. 16A, the photographing range of the camera 20 is translated above the installation surface by the thickness of the camera unit 12a.

  Therefore, strictly speaking, it cannot be determined whether or not the user's finger has touched the installation surface. For this reason, when using the imaging device 12 in the state shown in FIG. 16A, the computer 14 touches the installation surface because the finger image is in contact with the end of the partial image C2. Is determined. Thus, even when the imaging device 12 is installed in the opposite direction to the above-described embodiment, it is possible to determine whether or not there is a touch.

  Further, as shown in FIG. 16A, even when the imaging device 12 is installed, since light rays parallel to the installation surface are incident on the end of the curved mirror 30, the partial image C1 and the partial image C2 are displayed. When there is an object image that contacts the boundary (the end of the partial image C1), it can be detected that there is an object that contacts a surface parallel to the installation surface. Therefore, for example, it can be determined based on the partial image C1 that a figure (200, 202) as shown in FIG. 11 is placed.

  In the above-described embodiment, the hyperbolic curved mirror 30 is used. However, the present invention is not limited to this. For example, as shown in FIGS. 17A to 17D, curved mirrors (30a, 30b, 30c) having an elliptical surface, a conical surface, or a composite surface may be used.

  The hyperboloidal curved mirror 30 shown in the above-described embodiment is designed so that the light beam is parallel to the installation surface of the imaging device 12 at the end, and the object above the installation surface is reflected on the curved mirror 30. It will be. Therefore, the hyperboloid curved mirror 30 is effective when it is desired to recognize a three-dimensional object arranged on the installation surface, such as the figure (200, 202) shown in FIG.

  When the elliptical curved mirror 30a shown in FIG. 17A is designed so that the light beam is parallel to the installation surface at the end of the curved mirror 30a, as in the case of the hyperbolic curved mirror 30, the installation surface is viewed from above. Thus, an image is reflected on the curved mirror 30a. Therefore, it is effective when reading a flat object (for example, a card or the sheet 210 shown in FIG. 12) on the installation surface.

  In addition, since the conical curved mirror 30b shown in FIG. 17B has a flat slope, no distortion occurs in the height direction. For this reason, as described in the above embodiment, it is easy to calculate the height of the measurement point (for example, the fingertip) from the reflected image of the curved mirror 30b. However, the conical curved mirror 30b is also designed so that the light beam is parallel to the installation surface of the imaging device 12 at its end.

  As described above, the imaging device 12 is configured by selecting any one of a hyperboloid, an ellipsoid, and a conical surface according to an assumed use situation. FIG. 17C and FIG. It is also possible to synthesize (composite) curved surfaces as shown in FIG.

  In the curved mirror 30c having a composite surface shown in FIG. 17C, the curved surface is a hyperboloid in the vicinity of the installation surface (fixing member 32) and becomes an elliptical surface as it protrudes toward the camera 20 (upward). Is synthesized. In this way, the curved mirror 30c having the advantages of the hyperboloid and the ellipsoid is formed. However, this is only an example, and even with the curved mirror 30c having the same composite surface, as shown in FIG. 17D, conical surfaces inclined at different angles may be synthesized near and above the installation surface. .

  Further, in the above-described embodiment, the camera 20 capable of photographing all directions is provided, but the presence / absence of a touch can be similarly detected even with a camera that photographs one direction. FIG. 18A shows a state in which a mobile terminal 250 with a camera such as a mobile phone is placed on the charging stand 260. However, FIG. 18A shows only a part of the casing of the portable terminal 250 and the camera 252 provided in the casing, and the charging stand 260 also includes a part of the casing and the mirror 262a and the mirror provided in the casing. Only 262b is shown.

  As shown in FIG. 18A, the portable terminal 250 is placed on the charging stand 260 sideways. At this time, the camera 252 shoots the horizontal direction from the bottom surface of the mobile terminal 250, for example. In addition, an opening 260a for photographing light parallel to the installation surface on which the charging stand 260 is installed with the camera 252 is formed in a part of the casing of the charging stand 260. A mirror 262a and a mirror 262b for guiding parallel light into the angle of view of the camera 252 are arranged. In addition, an opening 260 b is formed in the other part of the casing of the charging stand 260 at a position facing the shooting direction of the camera 252 of the mobile terminal 250 mounted (mounted) on the charging stand 260.

  For example, in the state shown in FIG. 18A, the opening 260a has a vertical width from the lower end of the mirror 262a to the lower end of the mirror 262b, and the horizontal width is determined by the horizontal angle of view of the camera 252. The opening 260b is determined to have a size that guides the light reflected by the mirror 262b to the camera 252. Further, as can be seen from FIG. 18A, both the mirror 262a and the mirror 262b are flat mirrors, and are arranged so that the reflecting surfaces face each other.

  In the configuration as shown in FIG. 18A, light reflected by the mirror 262a and the mirror 262b is incident on substantially the lower half of the angle of view of the camera 252. Further, in the substantially upper half of the angle of view of the camera 252, since the photographing direction is not shielded by the casing of the charging stand 260, direct light is incident. Accordingly, as shown in FIG. 18B, the camera 250 captures a captured image including a partial image C1 corresponding to the mirror 262b and a partial image C2 corresponding to the mirror 262b. However, in FIG. 18B, images other than the finger image are omitted. In FIG. 18B, for simplicity, the partial image C2 is indicated by diagonal lines. In the situation shown in FIG. 18A, for example, the partial image C2 includes the second joint of the finger. To the image of the base of the finger.

  As shown in FIG. 18B, among the captured images, the lower end of the partial image C1 (captured image) corresponds to an installation surface on which the charging stand 260 is installed, and a finger image is displayed at the lower end of the partial image C1. When is touching, it is determined that the installation surface is touched. Further, when the finger image is not in contact with the lower end portion of the partial image C1, it is determined that the installation surface is not touched.

  In this case, the portable terminal 250 and the charging stand 260 function as the imaging device 12 shown in the above-described embodiment.

  Note that, for example, when a predetermined position is touched, the size of the finger image (finger thickness) of the partial image C1 or the finger image of the partial image C2 in the captured image captured by the camera 252. The size (finger thickness) is detected in advance. Then, by detecting the left and right positions of the finger when viewed from the camera 252 side based on the above-described predetermined position from the current captured image, and by comparing with the size of the finger image in the current captured image, It is possible to detect the position of the finger in the depth direction when viewed from the camera 252 side with the predetermined position as a reference.

  In addition, as shown in FIG. 19A, when a portable terminal 270 such as a mobile phone is vertically inserted into the charging base 280, it is shown in FIG. 18A and FIG. 18B. As in the case, the presence or absence of touch can be detected. However, FIG. 19A shows an outline of the portable terminal 270, and for the charging base 280, only a part of the casing and the curved mirror 282 and the wide-angle lens 284 provided in the casing are shown.

  As shown in FIG. 19A, the portable terminal 270 is inserted into the charging stand 280 vertically (a little obliquely). At this time, the camera 272 (only a lens is shown in FIG. 19A), for example, photographs downward from the bottom surface of the portable terminal 270. In addition, an opening 270 a for photographing light parallel to the installation surface on which the charging stand 280 is installed with the camera 272 is formed in a part of the casing of the charging stand 280. In addition, a curved mirror 282 for guiding the parallel light to the camera 272 is provided inside the housing. In addition, a wide-angle lens 284 for enlarging the angle of view of the camera 272 when the mobile terminal 270 is inserted into the charging base 280 is provided inside the housing.

  For example, in the state shown in FIG. 19A, the opening 280a has a vertical width from the bottom surface inside the casing of the charging base 280 to the ceiling surface, and the horizontal width of the charging base 280 (in FIG. The width in the direction perpendicular to the width).

  In the configuration shown in FIG. 19A, as shown in FIG. 19B, a captured image including a partial image C1 of the curved mirror 282 and a partial image C2 of a portion other than the curved mirror 282 is captured by the camera 272. Taken at. As shown in FIG. 19B, the boundary between the partial image C1 and the partial image C2 corresponds to the installation surface on which the charging stand 270 is installed. However, in FIG. 19B, images other than the finger image are omitted.

  Therefore, in the partial image C1, when the finger image is in contact with the boundary with the partial image C2 (the lower end portion of the partial image C1), it is determined that the user's finger is touching the installation surface. If the finger image is not in contact with the lower end of the partial image C1, it is determined that the user's finger is not touching the installation surface.

  In the captured image shown in FIG. 19B, the distance from the tip of the finger image of the partial image C2 to the boundary between the partial image C1 is the distance from the fingertip to the lower end of the curved mirror 282 in FIG. It corresponds to.

  Therefore, similarly to the case shown in the above-described embodiment, it is possible to determine whether or not the touch is performed using the partial image C1 and the partial image C2, and to calculate the position of the finger according to the determination result.

    In this case, the portable terminal 270 and the charging stand 280 function as the imaging device 12 shown in the above-described embodiment.

  In the example shown in FIGS. 19A and 19B, the curved mirror 282 is used, but a flat mirror may be used.

  Further, in the above-described embodiment, the case where the imaging device 12 capable of photographing all directions using the curved mirror 30 has been described. However, as another embodiment, the projector device 16 can be configured.

  As shown in FIG. 20A, the projector device 16 includes a projector unit 16a, a mirror portion 16b, and a fixing member 16c that fixes them, and has a cylindrical shape with both ends closed by the projector unit 16a and the mirror portion 16b. Formed.

  The projector unit 12a is formed in a circular shape when viewed from above, and includes a light source that respectively emits the three primary colors of RGB and a driving device that drives the light source, and electronic components including these are housed in a casing. Although the block diagram of the projector unit 16a is omitted, in the block diagram of the camera unit 12a shown in FIG. 3, the camera 20 may be replaced with a driving device, and a light source may be connected to the driving device.

  The curved mirror 50 protrudes toward the projector unit 16a and has an elliptical surface. The fixing member 16c has a cylindrical shape and is made of a transparent resin or glass. The height (length) of the fixing member 16c is determined to be a predetermined length. For example, when the projector device 16 is placed on a predetermined surface, the positional relationship between the projector unit 16a (light source) and the curved mirror 50 and the shape of the curved mirror 50 are so projected as to project a desired image on the predetermined surface. (Surface shape) is determined. Therefore, the height of the fixing member 16c is determined so as to have such a positional relationship.

  However, in the example shown in FIG. 20A, since the wide-angle lens 40 is provided in the projector unit 16a, an image can be projected around the projector device 16 (in the direction of the side surface of the cylinder). That is, the projector device 16 can project an image on an installation surface on which the projector device 16 is installed and a surface intersecting the installation surface.

  For example, by transmitting image data (hereinafter referred to as “projection image data”) to be projected from the computer 14 which is a host computer, the projector device 16 receives this and projects an image corresponding to the projection image data. can do. However, by storing projection image data in advance in a ROM (or a flash memory or the like) in the projector device 16, an image corresponding to the projection image data stored in advance can be projected. Therefore, an information processing system including the projector device 16 and the computer 14 is configured, or the projector device 16 functions as an information processing system.

  As shown in FIG. 20B, an electronic apparatus 60 including the imaging device 12 shown in the above-described embodiment and the projector device 16 shown in FIG. 20A can be configured. In the example shown in FIG. 20B, the projector device 16 is installed on the imaging device 12. For example, the optical axis of the camera 20 of the imaging device 12 and the optical axis of the light source of the projector unit 16a are matched. Moreover, the imaging device 12 and the projector device 16 are each connected so as to be communicable with the computer 14. That is, an information processing system including the computer 14 and the electronic device 60 is configured.

  In such an information processing system, for example, instead of the sheet 210 as shown in FIG. 12, the design of the button can be projected onto the installation surface of the electric device 60 by the projector device 16. Then, the computer 14 acquires a captured image of the imaging device 12, detects the position touched by the user based on the captured image, determines the touched button pattern, and determines the determined button pattern (operation button). Output the set command to the application. Therefore, a controller is realized by the electronic device 60 and the computer 14.

  Further, instead of the design of the button, a game map may be projected onto the installation surface by the projector device 16, and the figures (200, 202) as shown in FIG. 11 may be moved on the installation surface. In such a case, the position of the figure (200, 202) on the game map can be calculated from the captured image of the imaging device 12 and reflected in the processing of the application (game).

DESCRIPTION OF SYMBOLS 10 ... Information processing system 12 ... Imaging device 14 ... Computer 16 ... Projector apparatus 20 ... Camera 22, 40 ... Wide angle lens 30, 50 ... Curved surface mirror

Claims (18)

camera,
A mirror installed opposite the camera,
Based on the first partial image corresponding to the mirror and the second partial image that is adjacent to the first partial image and does not correspond to the mirror among the camera images captured by the camera, around the camera A first detecting means for detecting a position of an existing object; and a first object image of the object in the first partial image, the first partial image, and the second when the mirror side is installed on an installation surface. Contact determination means for determining whether or not the object is in contact with the installation surface depending on whether or not the boundary of the partial image is in contact;
When the mirror side is installed on the installation surface, the camera and the mirror are arranged such that light traveling along the installation surface in a direction parallel to the installation surface is reflected by at least a part of the mirror. An information processing system in which the positional relationship and the shape of the mirror are determined.   The first detection means is based on the first position of the first object image for the object in the first partial image and the second position of the second object image for the object in the second partial image. The information processing system according to claim 1, wherein the position of an object is detected. At least a second detection means for detecting a first positional relationship between the first partial image and the second partial image, and a first positional relationship of the first object image with respect to the object in the first partial image;
The information processing system according to claim 1, wherein the first detection unit detects a position of the object according to the first positional relationship detected by the second detection unit. The second detection means further detects a second positional relationship between a reference position of the first partial image and a second object image with respect to the object in the second partial image;
The information processing system according to claim 3, wherein the first detection unit detects the position of the object based on the first positional relationship and the second positional relationship detected by the second detection unit.   The first detection means detects the position of the object based on whether the first positional relationship indicates that the first object image is in contact with the boundary and the second positional relationship; The information processing system according to claim 4. The first position of the first object image for the object in the first partial image and the first position of the object in the second partial image on a straight line passing through the reference position of the first partial image among the camera images. Further comprising third detection means for detecting a second position of the two object images;
The information processing system according to claim 1, wherein the first detection unit detects the position of the object based on the first position and the second position detected by the third detection unit.   The information processing system according to claim 1, wherein an angle of view of the camera is 180 degrees.   The information processing system according to claim 1, wherein the camera is capable of photographing a surrounding 360 degrees.   The relative positional relationship between the camera and the mirror and the shape of the mirror are determined so that both the first partial image and the second partial image include an image of an object existing around the camera. An information processing system according to any one of claims 1 to 8.   The information processing system according to claim 1, wherein the mirror has a curved surface.   The information processing system according to claim 10, wherein the curved surface is a hyperboloid, a conical surface, an elliptical surface, or a composite surface of any two of them protruding toward the camera.   The information processing system according to claim 1, wherein an imaging apparatus including the camera and the mirror is portable.   The information processing system according to claim 1, wherein an imaging device including the camera and the mirror is formed in a cylindrical shape. The mirror is a curved mirror;
The said 1st detection means detects the position of the said object based on the image which correct | amended the said 1st partial image with the predetermined | prescribed algorithm, and the said 2nd partial image. Information processing system. An information processing program that is executed by a computer and a computer of an information processing system that includes a camera and a mirror that is installed facing the camera,
When for installing the mirror side of the information processing system to the installation surface, so that the light travels along the installation surface in a direction parallel to the installation surface is reflected at least part of the mirror, and the camera The positional relationship with the mirror and the shape of the mirror are determined,
Based on the first partial image corresponding to the mirror and the second partial image adjacent to the first partial image and not corresponding to the mirror, among the camera images captured by the camera, the computer detecting means for detecting a position of an object existing around the camera, and when to set up the mirror side of the information processing system to the installation surface, and a first object image of the object in the first partial image, said first An information processing program that functions as contact determination means for determining whether or not the object is in contact with the installation surface based on whether or not a boundary between the first partial image and the second partial image is in contact. An information processing method for an information processing system comprising a camera and a mirror installed opposite to the camera,
When for installing the mirror side of the information processing system to the installation surface, so that the light travels along the installation surface in a direction parallel to the installation surface is reflected at least part of the mirror, and the camera The positional relationship with the mirror and the shape of the mirror are determined,
(A) Based on a first partial image corresponding to the mirror and a second partial image adjacent to the first partial image and not corresponding to the mirror among the camera images captured by the camera. Detecting the position of an object existing around
(B) when for installing the mirror side of the information processing system to the installation surface, and a first object image of the object in the first partial image, the boundary of the first partial image and the second partial image The information processing method including the step of determining whether the said object contacts the said installation surface by whether it contacts. camera,
A mirror installed opposite to the camera, and contact determination means for determining whether an object existing around the camera is in contact with a predetermined surface based on a camera image taken by the camera ,
The camera and the mirror so that a camera image captured by the camera includes a first partial image corresponding to the mirror and a second partial image adjacent to the first partial image and not corresponding to the mirror. And place
When the mirror side is set on the predetermined surface, the camera and the camera are arranged so that light traveling along the predetermined surface in a direction parallel to the predetermined surface is reflected by at least a part of the mirror. The positional relationship with the mirror and the shape of the mirror are determined,
When the contact determination unit sets the mirror side on the predetermined surface, a first object image of the object in the first partial image and a boundary between the first partial image and the second partial image are obtained. An information processing system that determines whether or not the object is in contact with the predetermined surface based on whether or not the object is in contact. A wide-angle camera that can shoot 360 degrees around ,
A mirror that is disposed facing the wide-angle camera, is parallel to a predetermined surface, and is formed in a shape that is a direction of a light beam traveling along the predetermined surface ; and
Contact determining means for determining whether an object existing around the wide-angle camera is in contact with a predetermined surface based on a camera image captured by the wide-angle camera;
The wide-angle camera and the mirror are arranged so that a camera image including a first partial image corresponding to the mirror and a second partial image adjacent to the first partial image and not corresponding to the mirror can be taken by the wide-angle camera. And
When the contact determination unit sets the mirror side on the predetermined surface, a first object image of the object in the first partial image and a boundary between the first partial image and the second partial image are obtained. on whether contact, the object you determine if contact with the predetermined surface, the imaging apparatus.

Images