JP2016062589A - Information processing unit, optical device, information processing system, program, and object used therewith - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an excellent interface accommodating to components readily available in markets.SOLUTION: An information processing unit includes an image display part 110 disposed on one surface side, and imaging means 120 provided at a prescribed position on the one surface and capturing images of objects in a direction that the one surface faces. The imaging means 120 is equipped with an optical device 130 that guides light so as to capture at least a partial image of the image display part 110. The information processing unit causes the imaging means 120 to capture images including objects 153a and 153b located on a screen 111 of the image display part 110 and/or objects 153c and 153d located near the screen 111. The information processing unit specifies at least one of types, positions, directions, moving distances, rotation amounts and outlines of the objects based on the captured photographic images and controls the image display part 110 on the basis of information of the specified object.SELECTED DRAWING: Figure 7

Description

  The present invention relates to an information processing apparatus that performs processing based on an image photographed by a photographing means, an optical apparatus combined with a photographing means provided in the information processing apparatus, an information processing system including the information processing apparatus and the optical apparatus, and photographing by the photographing means. The present invention relates to a program used for processing based on a processed image and an object used therewith.

  Conventionally, an information processing apparatus that performs processing in accordance with the position and movement of a card or the like placed on a screen is known.

Patent Document 1 discloses an information output device that photographs a card placed on a stage surface with a camera disposed below the stage surface and identifies the position of the card. A projector is disposed under the stage surface, and projects an image corresponding to the position of the card onto the stage surface. When the user moves the card, for example, an image desired by the user is projected onto the stage surface.
The card functions as an interface for giving an instruction regarding video projection to the information output device. An instruction can be given to the information output device by an intuitive operation of moving the card.

JP 2008-212697 A

  However, the information output device of Patent Document 1 is designed and manufactured exclusively for an expensive optical system for specifying the position of a display or card for projecting an image in order to realize the excellent interface as described above. There is a need to. It is difficult to apply components that are readily available on the market.

  An object of the present invention is to provide an information processing apparatus that realizes an excellent interface that can use components that are easily available in the market. Moreover, the objective of this invention is providing the simple optical apparatus combined with the information processing apparatus which concerns on this invention. Another object of the present invention is to provide an information processing system that realizes an excellent interface at a low cost by combining the information processing apparatus according to the present invention with a simple optical device. Another object of the present invention is to provide a program that uses the information processing apparatus according to the present invention. It is providing the target object used with the information processing apparatus which concerns on this invention, an optical apparatus, an information processing system, or a program.

An information processing apparatus according to the first aspect of the present invention includes:
An image display unit arranged on one side;
An imaging unit that is provided at a predetermined position on the one surface and that captures a direction in which the one surface faces;
An information processing apparatus comprising:
An optical device that guides light to the photographing means so that at least a part of the image of the image display unit is photographed is attached;
An image including an object positioned on the screen of the image display unit, or an image including an object positioned on the screen and an object positioned in the vicinity of the screen;
Processing is performed based on the captured image.

An information processing apparatus according to the second aspect of the present invention includes:
Based on the photographed image, the type of the object, the position of the object located on the screen of the image display unit, the direction of the object located on the screen, and the amount of movement of the object located on the screen Identifying at least one of the amount of rotation of the object located on the screen and the contour of the object located on the screen;
Process based on information on the identified object.

In the information processing device according to the third aspect of the present invention,
The object is provided with an index including at least one of a graphic, a mark, an arrangement of marks, and a pattern of an automatic recognition code,
Identifying an object based on the image of the index included in the captured image,
Processing is performed based on the identified object.

In the information processing device according to the fourth aspect of the present invention,
The automatic recognition code includes at least one of a bar code, a two-dimensional code, and a dot code.

In the information processing device according to the fifth aspect of the present invention,
The object is provided with a dot pattern,
The dot pattern has one or more blocks including one or more information dots,
Information expressed by one or more information dots included in each block includes position information and / or code value of the block in the object.

In the information processing device according to the sixth aspect of the present invention,
For each information dot, basic information is defined based on at least one of the arrangement, size, shape, and color of the information dot.

In the information processing device according to the seventh aspect of the present invention,
The indicator has a characteristic of absorbing light of a predetermined wavelength, a characteristic of reflecting light of a predetermined wavelength, and a characteristic of emitting light having characteristics different from the light in response to light having the predetermined characteristic. At least one of the following.

In the information processing device according to the eighth aspect of the present invention,
The processing includes control of the image display unit.

In the information processing device according to the ninth aspect of the present invention,
The optical device includes at least one of a prism, a mirror, and an optical fiber.

In the information processing device according to the tenth aspect of the present invention,
The object includes at least one of a print medium, a thin plate-like medium, and a figure.

  The information processing apparatus according to the eleventh aspect of the present invention is plate-shaped.

An information processing apparatus according to a twelfth aspect of the present invention includes:
The indicator indicating the predetermined area of the object and the predetermined area of the object, or the indicator in contact with the predetermined area of the object and the predetermined area of the object are further specified based on the captured image. ,
Processing is performed based on the predetermined area of the identified object or based on the predetermined area of the identified object and the indicator.

In the information processing device according to the thirteenth aspect of the present invention,
The object is provided with a dot pattern,
The dot pattern has one or more blocks including one or more information dots,
Information expressed by one or more information dots included in each block includes position information of the block in the object,
When the predetermined area is specified based on the captured image, the predetermined area is specified based on block position information on the object.

An information processing apparatus according to a fourteenth aspect of the present invention includes:
The image display unit or a detection unit that detects a position on the screen of the indicator that is in contact with the object;
Processing is performed based on the detected position of the indicator on the screen and the information of the identified object, or the detected position of the indicator on the screen and the information of the identified object A predetermined area of the target object with which the indicator is brought into contact is specified based on the information, and processing is performed based on the specified predetermined target object area and information on the specified target object.

In the information processing device according to the fifteenth aspect of the present invention,
The detection means is a capacitive touch panel.

In the information processing device according to the sixteenth aspect of the present invention,
The photographed image forms a still image or a moving image.

In the information processing device according to the seventeenth aspect of the present invention,
A moving body video is shot by the shooting means,
When further identifying and processing the action of the action body based on the moving image, the action body is further based on the action of the identified action body.

In the information processing device according to the eighteenth aspect of the present invention,
The indicator is a finger.

In the information processing device according to the nineteenth aspect of the present invention,
The optical device can change a photographing region photographed by the photographing means.

In the information processing device according to the twentieth aspect of the present invention,
The change of the shooting area includes at least one of movement, enlargement, and reduction of the shooting area.

In the information processing device according to the twenty-first aspect of the present invention,
A calibration image including a plurality of identification images that can be distinguished from each other is displayed on the screen of the image display unit, and a reference point on the screen is designated by each of the plurality of identification images,
Identifying each identification image included in the photographed image based on the photographed image photographed by the photographing means, specifying a photographed image reference position indicating a position of a reference point designated by the identified identification image;
From the identification image included in the display image displayed on the screen of the image display unit, the same identification image as the identified identification image is identified, and the position of the reference point specified by the identification image identified as the same is determined. Specify the display image reference position to show,
Obtaining a reference correspondence relationship indicating correspondence between the specified captured image reference position and the specified display image reference position;
Based on the reference correspondence relationship, a position correspondence relationship indicating the correspondence between the position in the captured image and the position in the display image is obtained,
Information on the object is specified based on the positional correspondence and the captured image.

In the information processing device according to the twenty-second aspect of the present invention,
When obtaining the position correspondence relationship indicating the correspondence between the position (x, y) in the captured image and the position (X, Y) in the display image,
(X, Y) = F (x, y)
The function F that determines the position correspondence or the table that determines the position correspondence is defined.

In the information processing device according to the twenty-third aspect of the present invention,
The calibration image represents a grid line, a symbol array, or a dot pattern, and includes an image formed by the plurality of identification images.

In the information processing device according to the twenty-fourth aspect of the present invention,
The identification image forming the image representing the grid line is identified based on at least one of the line type, line color, line thickness, and line spacing.

In the information processing device according to the twenty-fifth aspect of the present invention,
The plurality of identification images forming an image representing the grid lines respectively represent rectangular lines surrounding a plurality of grid areas partitioned by the grid lines,
The identification image is identified based on a ratio of rectangular sides.

In the information processing device according to the twenty-sixth aspect of the present invention,
The identification image forming the image representing the symbol arrangement is identified based on at least one of a symbol shape, a symbol size, a symbol color, and a symbol arrangement pattern.

In the information processing device according to the twenty-seventh aspect of the present invention,
An image including a dot pattern is displayed on the screen of the image display unit,
The dot pattern has one or more blocks including one or more information dots,
Information expressed by one or more information dots included in each block includes information including a position where the block is displayed.

In the information processing device according to the twenty-eighth aspect of the present invention,
Based on the photographed image photographed by the photographing means, specify part or all of the outline of the image on the screen of the image display unit included in the photographed image,
Based on a part or all of the specified contour, a position correspondence relationship indicating a correspondence between a position in a captured image photographed by the photographing unit and a position in a display image displayed on the screen of the image display unit is obtained.
Information on the object is specified based on the positional correspondence and the captured image.

In the information processing device according to the twenty-ninth aspect of the present invention,
The optical device is provided with an irradiating unit that irradiates an imaging region imaged by the imaging unit.

In the information processing device according to the thirtieth aspect of the present invention,
The irradiation unit irradiates light having a predetermined wavelength.

In the information processing device according to the thirty-first aspect of the present invention,
The optical device is provided with a filter that transmits light of the first wavelength and blocks light of the second wavelength, and the imaging region is imaged through the filter.

  The optical device according to one aspect of the present invention is combined with the information processing device according to any of the first to thirty-first aspects.

  An information processing system according to one aspect of the present invention is combined with the information processing apparatus according to the first to thirty-first aspects, the optical apparatus according to the one aspect, and the information processing apparatus according to the first to thirty-first aspects. Equipped with a target object.

  A program according to an aspect of the present invention is combined with the information processing system according to the above aspect, and causes the information processing apparatus to execute the processing.

  The object according to one aspect of the present invention is combined with the information processing apparatus according to the first to thirty-first aspects.

  By using the information processing apparatus, the optical apparatus, the information processing system, the program, and the target object of the present invention including the photographing unit, an excellent interface can be realized using components that can be easily obtained on the market.

1 is a plan view of an information processing system according to an embodiment of the present invention. 1 is a side view of an information processing system according to an embodiment of the present invention. It is a perspective view of an optical device concerning one embodiment of the present invention, and (a) is a figure showing the state where a cover was removed. (B) is a figure which shows the state which attached the cover. (C) is a figure which shows the state which attached the irradiation means. The perspective view of the optical apparatus containing a prism based on another embodiment of this invention. The perspective view of the optical apparatus containing the optical fiber module based on another embodiment of this invention. It is a figure which shows the optical apparatus which concerns on another embodiment of this invention, (a) is a perspective view of the optical apparatus which has an attachment part which clamps information processing apparatus. (B) is a side view of the optical device showing a state in which an optical device having an attachment portion for holding the information processing device is attached to the information processing device. (C) is a perspective view of an optical device having a concave mirror. (D) is a perspective view of an optical device having a prism having a light incident surface formed in a convex shape. (A) Photographs an image of an object according to an embodiment of the present invention located on the screen of the image display unit and an object according to an embodiment of the present invention located outside the screen. The perspective view of the information processing system which concerns on one embodiment of this invention which shows a mode to do. (B) is a perspective view of the information processing system which concerns on one Embodiment of this invention which shows a mode that the image of the target object which concerns on one Embodiment of this invention located on the screen of an image display part is image | photographed. (C) is a figure which shows an example of the picked-up image image | photographed by the imaging means. (D) is a figure which shows another example of the picked-up image image | photographed by the imaging means. 1 is a block diagram of an information processing apparatus according to a first embodiment of the present invention. 3 is a flowchart showing the operation of the information processing apparatus according to the first embodiment of the present invention. The figure which shows the example of the image displayed on the image display part controlled based on the information of a target object. The block diagram of the information processing apparatus which concerns on the 2nd Embodiment of this invention. The figure which shows an example of a dot pattern. It is a figure which shows a response | compatibility with arrangement | positioning of an information dot, and the value of an information, (a) is a figure which shows a response | compatibility in the example which can arrange | position 8 information dots. (B) is a figure showing correspondence in an example in which information dots can be arranged in four ways. (C) is a figure showing correspondence in another example in which information dots can be arranged in four ways. The figure which shows the response | compatibility with the arrangement | positioning of an information dot, and the value of information in the example which can arrange | position 16 information dots. It is a figure which shows a response | compatibility with the arrangement | positioning of an information dot, and the value of information in the example which can arrange | position 0-8 information dots to eight predetermined positions, (a) is information at a certain two position, respectively. The figure which shows the value of the information corresponding to the arrangement | positioning which positions a dot, and the arrangement | positioning. FIG. 7B is a diagram illustrating an arrangement in which information dots are positioned at four positions and information values corresponding to the arrangement. (C) is a figure which shows the value of the information corresponding to the arrangement | positioning which positions an information dot in each of five certain positions, and the arrangement | positioning. (A) is a top view which shows a mode that an example of the calibration image containing the image of a grid line is displayed on the screen of the information processing apparatus which concerns on the 4th Embodiment of this invention. FIG. 7B is a diagram illustrating a captured image of the information processing apparatus captured by the imaging unit in a state where an example of a calibration image including a grid line image is displayed on the screen. (A) is a figure showing an example of a reference point in a display image. (B) is a figure which shows an example of the reference point in a picked-up image. (C) is a figure which shows an example of the table which determines a positional correspondence. (A) is a figure which shows the image of the reference | standard point in a picked-up image, and the card | curd mounted on the screen. (B) is a figure which shows the positional relationship of the reference point in a display image, and the representative point of the card | curd mounted on the screen, and also shows the reference point of the image which converted the picked-up image with the function. (A) is a figure which shows an example of the arrangement | sequence of the symbol contained in a calibration image. (B) is a figure which shows an example of the position which arrange | positions the symbol of (a), and also shows an example of the arrangement | sequence of the dot contained in a calibration image. (A) is a figure which shows an example of a picked-up image. FIG. 5B is a diagram illustrating an image obtained by converting a captured image with a function.

  An information processing apparatus 100 according to a first embodiment of the present invention, an optical apparatus 130 according to an embodiment, and an information processing system according to an embodiment will be described. FIG. 1 is a plan view of the information processing system. FIG. 2 is a side view of the information processing system. FIG. 3A and FIG. 3B are perspective views of the optical device 130. The optical device 130 is combined with the information processing device 100. The information processing system includes an information processing device 100 and an optical device 130. The information processing apparatus 100 includes an image display unit 110 that displays an image, a photographing unit 120 that captures an image, and an optical device 130. The image display unit 110 includes, for example, a liquid crystal display. The photographing unit 120 includes, for example, a CCD camera or a C-MOS camera. The photographed image photographed by the photographing means 120 forms a still image or a moving image. The photographing unit 120 is provided with an image storage unit that stores a photographed image photographed by the photographing unit 120. The image display unit 110 and the photographing unit 120 are stored in a plate-shaped housing 140. The information processing apparatus 100 is plate-shaped. The image display unit 110 is disposed on one surface side of the information processing apparatus 100 (the upper surface 142 side of the housing 140). The screen 111 of the image display unit 110 is exposed. The screen 111 faces in the direction in which the one surface of the information processing apparatus 100 faces. The imaging unit 120 is provided at a predetermined position on the one surface of the information processing apparatus 100 and images the direction in which the one surface is facing. The photographing means 120 is disposed on the periphery of the plate-shaped housing 140. The housing 140 is provided with a photographing hole 141, and the photographing means 120 performs photographing through the photographing hole 141.

  The optical device 130 guides light to the photographing unit 120 so that at least a part of the image on the screen 111 of the image display unit is photographed. The optical device 130 is attached to the housing 140 of the information processing apparatus 100. The optical device 130 is placed on the upper surface 142 of the housing 140 via the base 131b. The portion 131a of the base 131b is formed so as to avoid the photographing hole 141 of the housing 140. A support portion 131c that supports the base 131b is provided on the lower surface of the base 131b. The support part 131c can change the inclination of the base 131b. The base 131b and the support portion 131c constitute an attachment portion 131. The optical device 130 is attached to the housing 140 by the attachment portion 131.

  In order to detachably attach the optical device 130 to the information processing apparatus 100, means different from the attachment portion 131 shown in FIGS. 1, 2, 3A, and 3B are used as the optical device 130 and information. You may provide in at least one with the processing apparatus 100. FIG. For example, the support unit 131c may be omitted, and the optical device 130 may be provided with means for detachably fixing the base 131b to the upper surface 142 of the housing 140. In this case, a means for changing the inclination of the base 131b is not necessary, and the configuration of the optical device 130 can be simplified. The optical device 130 may not be detachably attached to the information processing apparatus 100. For example, the optical device 130 may be fixed to the information processing apparatus 100 by a predetermined method.

  A column 132 is rotatably attached to the attachment portion 131. When the optical device 130 is placed, the longitudinal direction of the support column 132 is a direction that intersects the upper surface 142 of the housing 140. The support column 132 can be rotated around the longitudinal direction. A support portion 133 is attached to the support column 132 to rotatably support the mirror 134. The mirror 134 guides light to the photographing means 120 for photographing. The rotation axis of the mirror 134 is parallel to the screen 111 of the image display unit 110. The support part 133 is slidable in the longitudinal direction of the support column 132.

  The mirror 134 can rotate in two axial directions and can slide in the longitudinal direction of the support column 132. That is, three degrees of freedom are realized. Thereby, the optical device 130 can change the imaging region of the imaging unit 120. Using the biaxial rotation and the longitudinal slide, the optical device 130 can change the imaging area so that at least one of movement, enlargement, and reduction of the imaging area is performed. The degree of freedom for moving the mirror can be set arbitrarily. Zero or more degrees of freedom can be set. An optical device that realizes such a degree of freedom can be realized by using a well-known mechanism. The mirror may be provided with a mechanism that can rotate around an axis parallel to the screen and that can slide in a direction perpendicular to the screen. In this case, two degrees of freedom are realized.

  A cover 135 that covers the mirror 134 is attached to the support portion 133. Since the cover 135 is disposed so as to cover the photographing hole 141, it is possible to prevent light from external illumination or the like from directly entering the photographing unit 120. The cover 135 and the support part 133 may be integrally formed.

  As shown in FIG. 3C, an irradiation unit 136 that irradiates the imaging region of the imaging unit 120 may be attached to the support unit 133. When a sufficient amount of light for photographing the object cannot be secured, the object can be clearly photographed by irradiation.

  The optical device 130 may include a prism 134b shown in FIG. 4 or an optical fiber module 134c having an optical fiber shown in FIG.

  In order to detachably attach the optical device to the information processing apparatus 100, a clip that holds the information processing apparatus 100 may be used. As shown in FIGS. 6A and 6B, the attachment portion 131 has an abutment portion 131d that abuts against the lower surface of the housing 140 instead of the support portion 131c. The contact portion 131d extends from the base 131b, and the contact portion 131d and the base 131b sandwich the housing 140. FIG. 6C shows an optical device 130b having a concave mirror. The concave mirror of the optical device 130b is designed so that the entire screen 111 is photographed when the optical device 130b is placed at a predetermined position on the upper surface 142 of the housing 140 in a predetermined direction. FIG. 6D shows an optical device 130c having a prism with a light incident surface formed in a convex shape. Similar to the optical device 130b, the convex surface of the optical device 130c is designed so that the entire screen 111 is photographed when the optical device 130b is placed at a predetermined position on the upper surface 142 of the housing 140 in a predetermined direction. ing. The optical devices 130b and 130c realize a wide field of view when photographing the screen 111. For this reason, it is possible to easily photograph a desired region simply by placing it at a predetermined position in a predetermined direction. In the optical device shown in FIG. 3, FIG. 4, FIG. 5, FIG. 6 (a) and FIG. 6 (b), the orientation and position of optical elements such as mirrors and prisms are adjusted in order to photograph a desired region. Although it is necessary, there is no need to adjust the optical devices 130b and 130c. Further, since no means for adjusting the orientation and position of the optical element is required, the structure is simple. In order to attach the optical devices 130 b and 130 c to the information processing apparatus 100, the optical devices 130 b and 130 c may be attached to the upper surface 142. In that case, an adhesive that can easily peel off the optical devices 130b and 130c from the upper surface 142 and can be attached again can be used. Moreover, you may use the clip similar to the attaching part 131 shown by Fig.6 (a) and FIG.6 (b). As described above, the embodiments of various optical devices have been described. However, as long as at least part or all of the screen 111 can guide light so as to be included in the imaging region of the imaging unit 120, any structure may be used. Also good.

  The image capturing unit 120 captures an image including an object located on the screen 111 of the image display unit 110 and / or an object located outside the screen 111. The object located outside the screen 111 is preferably located near the screen. This is particularly desirable when both an object located on the screen 111 and an object located outside the screen 111 are photographed simultaneously. The object includes at least one of a print medium, a lamellar medium, and a figure. FIG. 7A shows a state where an object according to an embodiment of the present invention is photographed. In the example illustrated in FIG. 7A, the imaging unit 120 captures images of the objects 153 a and 153 b located on the screen 111 of the image display unit 110 and / or the objects 153 c and 153 d located outside the screen 111. be able to. The object 153a is a card. The card is a print medium, and is a thin plate-like medium. The object 153b is a coin. The object 153c is a figure. The object 153d is a glass. FIGS. 7C and 7D are examples of images taken by the photographing unit 120. FIG. If the function of the optical device 130 that changes the shooting area of the shooting means 120 is used, images of various areas can be shot.

  As shown in FIG. 7B, even if the photographing unit 120 expands the photographing region to the outside of the screen 111 so that the indicator pointing to the object (for example, the card 153a) on the screen 111 can be photographed from outside the screen 111. Good. A user's finger 154 may be used as the indicator, or a touch pen may be used. In order to change the imaging region, the orientation of the optical device 130 may be changed, and an optical device having a structure for condensing light and / or a material for condensing light so that a wide area can be photographed. The formed optical device may be prepared in advance.

A program according to an embodiment of the present invention that uses the information processing apparatus 100 will be described.
FIG. 8 is a block diagram of the information processing apparatus 100. The information processing apparatus 100 includes an image analysis unit 161, a control unit 162, a storage unit 163, and an audio reproduction unit 164. The storage unit 163 stores a program that causes the information processing apparatus 100 to execute image analysis performed by the image analysis unit 161 and control performed by the control unit 162. The storage unit 163 further includes image data and / or moving image data displayed on the image display unit 110, image data and / or moving image data used for analyzing the captured image by the image analyzing unit 161, and audio reproduction. Audio data to be reproduced by the unit 164 is stored. The program, image data, moving image data, and audio data can be downloaded from a predetermined server using a communication means (not shown).

  FIG. 9 is a flowchart showing the operation of the information processing apparatus 100. The information processing apparatus 100 causes the image capturing unit 120 to capture an image including the object, and processes information related to the captured image based on the captured image. For example, at least one of the image display unit 110 and the audio reproduction unit 164 is controlled based on the captured image. The program is read from the storage unit 163, and information processing is performed by the control unit 162 based on the program. Image data, moving image data, and audio data are appropriately read from the storage unit 163 and used by the control unit 162 and the image analysis unit 161. The control unit 162 causes the photographing unit 120 to capture an image including the objects 153a and 153b positioned on the screen 111 of the image display unit 110 and / or the objects 153c and 153d (FIG. 7A) positioned outside the screen 111. Photographing is performed (step 171). An image including the objects 153a and 153b located on the screen 111 may always be taken. In that case, an image including the objects 153a and 153b located on the screen 111 or an image including the objects 153a and 153b located on the screen 111 and the objects 153c and 153d located near the screen 111 is captured. The Next, the control unit 162 causes the image analysis unit 161 to analyze the image. Based on the image of the object photographed by the photographing unit 120 (the image stored in the image storage unit 121), the image analysis unit 161 includes the type, position, orientation, movement amount, rotation amount, and contour of the object. Information on an object including at least one of the items is specified (step 172). Examples of the types of objects include cards, figures, and coins. When the type of the object is specified, it is specified whether the object is a card or a figure. When the photographing unit 120 always captures an image including the objects 153a and 153b located on the screen 111 in step 171, the type of the object (the object located on the screen 111 is obtained in step 172. When both of the objects 153a and 153b and the objects 153c and 153d located outside the screen 111 are photographed, at least one kind of the objects 153a, 153b, 153c and 153d, only the objects 153a and 153b are photographed. In this case, at least one kind of the objects 153a and 153b.), At least one position of the objects 153a and 153b located on the screen 111, at least one direction of the objects 153a and 153b, and the objects 153a and 153b At least one movement amount, at least one rotation amount of the objects 153a and 153b, and the object 15 a, information of the object comprising at least one of the at least one contour of 153b are identified. The control unit 162 controls at least one of the image display unit 110 and the audio reproduction unit 164 based on the information on the target specified by the image analysis unit 161 (Step 173). As described above, the captured image can form a moving image. If the amount of movement of the object is specified in step 172, the position of the object that changes from moment to moment is analyzed based on the moving image. When the amount of rotation is specified, the direction of the object that changes from moment to moment is analyzed based on the moving image.

  For example, when the card 153a on which a tank picture is drawn is placed at a predetermined position on the screen 111, the control unit 162 controls the audio reproduction unit 164 so that the audio reproduction unit 164 reproduces the start sound of the tank engine. . At the same time, the image display unit 110 is controlled so that an image in which soil smoke rises around the card 153a is displayed. As shown in FIG. 10, when the card 156 on which the tank is drawn is moved and / or rotated on the screen 111, the track of the endless track along the track of the card 156 is accordingly changed. An image 156a may be displayed, or an image 156b may be displayed in which a tank drawn on the card 156 fires toward the vehicle 157 displayed on the screen 111 as a moving image.

  As shown in FIG. 1, a predetermined area 151 (in the example of FIG. 1, an area where a star mark “☆” is drawn) of a card 153 (object) is indicated by an indicator 154 (for example, a user's finger). When the predetermined area 151 is pointed from a position away from the predetermined area 151, the control unit 162 performs predetermined control accordingly. A pen may be used instead of the finger. The indicator 154 may contact the predetermined area 151. A predetermined area 151 (an area near the tank cannon) indicated by the indicator 154 is specified based on the captured image. Based on the identified area, the control unit 162 can control at least one of the image display unit 110 and the audio reproduction unit 164. For example, control is performed so that a sound like a cannon firing is reproduced and a spark image is displayed in the vicinity of the area 151.

  The image analysis unit 161 can specify whether the indicator 154 points to the predetermined region 151 from a position away from the predetermined region 151 or whether the indicator 154 is in contact with the predetermined region 151. When specifying, the image analysis unit 161 specifies based on the shadow of the indicator 154 included in the captured image. When it is specified that the indicator 154 points to the predetermined area 151 from a position away from the predetermined area 151, the image analysis unit 161 specifies (in reality) the direction in which the indicator 154 points to the predetermined area 151, The predetermined area 151 is specified based on the direction. When it is determined that the indicator 154 is in contact with the predetermined area 151, the image analysis unit 161 analyzes the captured image and (in reality) is determined to be positioned directly below the indicator 154. This area is specified as the predetermined area 151. In order to specify whether the indicator 154 is away or in contact, a detection unit that detects the position on the screen 111 of the indicator 154 that contacts the object 153 located on the screen 111 of the image display unit 110 is used. May be. The detection means is provided on the screen 111 and includes a capacitive touch panel. If the position of the indicator 154 on the screen 111 is detected by the detection means, the indicator 154 is specified to come into contact. If no position is detected, the indicator 154 is identified as being away.

  The image analysis unit 161 can specify not only the predetermined area 151 of the card 153 but also the indicator 154 based on the photographed image photographed by the photographing unit 120. The predetermined area 151 can be specified in the same manner as described above. When specifying the indicator 154, the image analysis unit 161 specifies the indicator 154 by comparing the image data and / or moving image data stored in the storage unit 163 with the image of the indicator 154. For example, the image analysis unit 161 compares the image of the indicator 154 with the image data of the user's finger or the pen image data stored in the storage unit 163 and determines that the indicator 154 is the user's finger. To do. This comparison / collation may be determined by extracting feature points between the image of the indicator 154 and the stored image data. The control unit 162 can control at least one of the image display unit 110 and the audio reproduction unit 164 based on the specified predetermined area 151 and the indicator 154. In this case, if the specified indicator is a red touch pen, the control unit 162 performs a certain control (for example, a control in which a spark image is displayed in the vicinity of the region 151), and the specified indicator is a blue touch pen. In such a case, another control (for example, a control that displays an image showing the number of remaining cannons in the vicinity of the area 151) can be performed.

  You may control based on operation | movement of the action body expressing a motion. A pen or a user's hand can be used as the operating body. The image analysis unit 161 adds information on the target object (information including at least one of the type, position, orientation, movement amount, rotation amount, and outline of the target object) based on the captured image forming the moving image. Thus, it is possible to further specify the action of the action object photographed by the photographing means 120. The control unit 162 can control at least one of the image display unit 110 and the audio reproduction unit 164 based on the action of the specified operating body. For example, a circle is drawn clockwise or counterclockwise with the moving object, the moving object is moved along the screen 111, or the moving object is moved away from or closer to the screen 111. The image analysis unit 161 compares the image data and / or moving image data stored in the storage unit 163 with the moving image of the photographed moving body to identify the movement of the moving body. When specifying, the speed of the moving body and the trajectory of the moving body can be specified. The control unit 162 can perform control based on the speed of the operating body and the trajectory of the operating body. When the user's hand is used as the operating body, the control may be performed based on the movement of the soap (for example, change from goo to par). The trajectory of the moving body may be specified by extracting feature points that characterize the motion.

  Next, an information processing apparatus 200 according to the second embodiment of the present invention will be described. Most of the configuration of the information processing apparatus 200 according to the second embodiment is the same as that of the information processing apparatus 100 according to the first embodiment. Components having the same configuration and function as the components of the information processing apparatus 100 according to the first embodiment are denoted by the same reference numerals.

  The configuration of the information processing apparatus 200 will be described with reference to FIGS. 1 and 2. The information processing apparatus 200 further includes detection means for detecting the position on the screen 111 of the indicator 154 (user's finger) that contacts the object 153 located on the screen 111 of the image display unit 110 (FIG. The detection means is not shown in FIGS. 1 and 2). The detection means is provided on the screen 111 and includes a capacitive touch panel. The detection means detects the position on the screen 111 of the object 153 and / or the indicator 154 that contacts the object 153 or the screen 111 such as an optical touch panel and a pressure-sensitive touch panel in addition to the capacitive touch panel. If possible, any means may be used.

  FIG. 11 is a block diagram of the information processing apparatus 200. The image analysis unit 161 can specify information on the card 153 including at least one of the type, position, orientation, movement amount, rotation amount, and contour of the card 153 based on the captured image. For example, information including the position of the card 153, that is, the position on the screen 111 and the orientation of the card 153 is specified. Next, the image analysis unit 161 applies the finger of the user based on the information on the card 153 (position and orientation on the screen 111) and the position of the user's finger on the screen 111 detected by the detection unit 265. The area 151 of the card 153 in contact is specified. The control unit 162 controls at least one of the image display unit 110 and the audio reproduction unit 164 based on the identified area 151 of the card 153 and information on the card 153 (position and orientation on the screen 111). For example, when the user places a finger on the area 151 of the card 153 (an area in the vicinity of the tank cannon), a sound is reproduced as if the cannon fired, and a spark image is displayed in the vicinity of the area 151. Is made.

  When the indicator 154 contacts an area on the screen 111 in the vicinity of the object 153 (the indicator 154 is not in contact with the object 153), the information processing apparatus 200 operates as follows. The image analysis unit 161 identifies information on the card 153 (information including the position and orientation on the screen 111) based on the captured image. The control unit 162 controls at least one of the image display unit 110 and the audio reproduction unit 164 based on the information on the specified card 153 and the position of the user's finger on the screen 111 detected by the detection unit 265. For example, when the user places a finger on the area on the screen 111 near the area 151 of the card 153 (the area near the tank cannon), a sound like a cannon firing is reproduced, and a spark is generated near the area 151. Control is performed so that an image is displayed.

  Next, an information processing apparatus according to a third embodiment of the present invention and an object according to an embodiment of the present invention used therewith will be described. Most of the configuration of the information processing apparatus according to the third embodiment is the same as that of the information processing apparatus 100 according to the first embodiment. Components having the same configuration and function as the components of the information processing apparatus 100 according to the first embodiment are denoted by the same reference numerals.

  The object of the present embodiment is provided with an index for specifying the object. The indicator includes at least one of a graphic, a mark, a mark arrangement, and a pattern of an automatic recognition code. When the object on which the index is provided is imaged by the imaging unit 120, the information processing apparatus identifies the object based on the image of the index included in the captured image. The information processing apparatus performs processing based on the identified object.

  For example, a card (object) provided with a cat graphic index is placed at the center of the screen 111 of the image display unit 110, and a card (object) provided with a dog graphic index is the end of the screen 111. It is assumed that it is mounted on. A predetermined ID number for identifying the card, for example, “1” is set for the cat card. The dog card has an ID number “2”. The storage unit 163 (FIG. 8) stores an ID number, index (graphic) image data, and a correspondence relationship between the ID number and the index. When these cards are photographed, the image analysis unit 161 compares the index image included in the photographed image with the index image data stored in the storage unit 163, and the image of the screen 111 in the photographed image is compared. It is specified that the image of the graphic index of the cat stored in the storage unit 163 is included in the image of the card located at the center. Similarly, the image analysis unit 161 identifies that the image of the card graphic positioned at the edge of the image on the screen 111 in the captured image includes the image of the dog graphic index stored in the storage unit 163. . Based on the correspondence between the ID number and the index stored in the storage unit 163, the image analysis unit 161 has the ID number of the center card of the screen 111 as “1” and the ID number of the card at the end of the screen 111 as “ 2 ”. In this way, the object is specified. The storage unit 163 stores a program associated with the card ID number, image data, moving image data, audio data, and the like. When the card ID number is specified, the control unit 162 performs control based on a program corresponding to the card ID number. For example, the character “cat” is displayed in the area of the screen 111 near the card with the ID number “1” (cat card), and the card near the card with the ID number “2” (dog card). Control is performed such that the character “dog” is displayed in the area of the screen 111.

  Even when a mark is used as an index, an object can be specified in the same manner as when a graphic is used as an index. The mark is, for example, a heart shape or a diamond shape. The object can be specified by the difference in the marks.

  As described above, an array of marks can be used as an index. The array of marks is, for example, a polygon, and marks are arranged at the positions of the vertices of the polygon. The vertex of the polygon coincides with a point representing the mark, for example, the centroid of the mark. It is assumed that an array of marks is provided on two cards (objects). One card is provided with three marks, and the three marks are respectively arranged at the positions of the assumed vertices of an equilateral triangle. The other card is also provided with three marks, and the three marks are respectively arranged at the positions of the vertices of the assumed isosceles triangle. The equilateral triangle arrangement card is placed at the center of the screen 111, and the isosceles triangle arrangement card is placed at the end of the screen 111. The image analysis unit 161 specifies a card based on the mark shape and the mark arrangement (in this case, a polygonal shape). An ID number “3” is set for an equilateral triangle arrangement card, and an ID number “4” is set for an isosceles triangle arrangement card. The storage unit 163 stores an ID number, mark shape data, mark array data, and a correspondence between them. When these cards are photographed, the image analysis unit 161 identifies the mark shape and mark arrangement provided on each card based on the photographed image. The image analysis unit 161 compares the specified mark shape and mark arrangement with the mark shape data and mark arrangement data stored in the storage unit 163, and determines the correspondence stored in the storage unit 163. Based on this, it is specified that the ID number of the card at the center of the screen 111 is “3” and the ID number of the card at the end of the screen 111 is “4”. When the card ID number is specified, the control unit 162 performs control based on a program corresponding to the card ID number.

  The shape of an assumed polygon (hereinafter referred to as an assumed polygon) is disturbed when photographed. When the mark array is photographed from an oblique direction, the photographed assumed polygon shape (polygon shape having a vertex representing the photographed mark) is different from the actual assumed polygon shape. . For example, when an equilateral triangle is photographed from an oblique direction, the shape of the photographed equilateral triangle is an isosceles triangle. The image analysis unit 161 estimates the actual assumed polygonal shape from the captured image using the contour shape of the captured screen 111 and the like. The image analysis unit 161 uses the estimated actual assumed polygonal shape when specifying the mark arrangement based on the captured image.

  By using the mark arrangement in this way, a plurality of cards (objects) can be distinguished from each other. The arrangement of points representing the mark (hereinafter referred to as representative point arrangement) is preferably unique. That is, when two cards are arbitrarily selected, the representative point arrangement of one card is enlarged, moved in parallel, and rotated, the representative point arrangement cannot be superimposed on the representative point arrangement of the other card. Is desirable.

  The indicator can be printed using ink that is invisible on the object. As the ink used for printing the index, a second characteristic that is different from the first light in response to the first light having the characteristic that reacts to the light having the predetermined wavelength and the first light having the predetermined characteristic. An ink having at least one of the characteristics of emitting light may be used. Further, an index may be provided on the object using a material having such characteristics instead of ink. For example, when an index is printed with ink having a characteristic of absorbing light of a predetermined wavelength, light of a predetermined wavelength is absorbed in the target index portion, and light of the predetermined wavelength is diffusely reflected in other portions. Is done. The object is photographed through a filter that transmits at least light of a predetermined wavelength. That is, the index can be specified by not capturing light of a predetermined wavelength only for the pixel corresponding to the index portion of the object. Further, the indicator is printed with invisible ink having a characteristic of emitting a second light having a characteristic different from the first light in response to the first light having a predetermined characteristic. The index is photographed by emitting visible light having a wavelength different from the color of the object, and the index is specified.

  The automatic recognition code used for the indicator includes at least one of a bar code, a two-dimensional code, and a dot code. A dot pattern which is a pattern used for a dot code will be described with reference to FIGS. A dot pattern consists of a plurality of dots. The line segments and curves drawn in FIGS. 12 to 15 are given for convenience of explanation, and are not included in the dot pattern.

  FIG. 12 is a diagram showing a dot pattern according to the present embodiment. The dot pattern shown in FIG. 12 is called GRID1. As shown in FIG. 3C, the optical device 130 can be provided with an irradiation means 136 for irradiating the imaging region. As the irradiation unit 136, an LED capable of irradiating light having a predetermined wavelength onto an object on which a dot pattern has been applied can be used. The photographing means 120 can photograph with light of a predetermined wavelength. The optical device 130 can be provided with a filter 137 that transmits light of the first wavelength and blocks light of the second wavelength. The filter 137 can be provided in the portion 131 a so as to cover the imaging hole 141 of the housing 140. The shooting area is shot through the filter 137. Of course, the dots may be printed with a color scheme that can identify the dots, and the dot pattern may be photographed without providing the irradiation means 136 and the filter 137.

  The dot pattern 1 is generated by arranging fine dots, that is, key dots 2, information dots 3, and reference grid point dots 4 according to a predetermined rule by a dot code generation algorithm. As shown in FIG. 12, on the medium (object) on which the dot pattern 1 is provided, a plurality of line segments extending in one direction and parallel to each other and a plurality of line segments extending in a direction orthogonal to this direction and parallel to each other. A lattice formed by is assumed. Various dots are arranged based on this lattice. These dots constitute one block. Reference grid point dots 4 are arranged at the grid points. Key dots 2 are arranged at positions shifted from the grid points. In the example of FIG. 12, the key dots 2 are arranged at the corners of the block, but may be arranged anywhere in the block. Reference grid point dots 4 are arranged in each of the four corners of a rectangular grid area partitioned by lines forming the grid. A virtual lattice point is set at the center of each lattice region (intersection of rectangular diagonal lines). Information dots 3 are arranged around the virtual grid point with the virtual grid point as a reference. FIG. 12 shows four blocks arranged in close proximity. The grid frame that forms the basis of each block is indicated by a bold line. Adjacent blocks share the reference grid point dot 4 and the key dot 2 on the grid frame.

Since the reference grid point dots 4 are arranged in a grid pattern, the photographed reference grid point dots 4 are regularly arranged. However, due to distortion of the lens of the photographing unit 120, deformation of the medium provided with the reference grid point dots 4, the array of the captured reference grid point dots 4 is deformed. The deformation of the arrangement of the reference grid point dots 4 causes an error in the analysis of the dot pattern 1. In particular, when four reference lattice point dots arranged at the four corners of the lattice region (located at the corners of the rectangle) are photographed from an oblique direction, the lattice region surrounded by these reference lattice point dots depending on the photographing angle. The degree of deformation increases. Such deformation can be corrected by converting the captured image using a predetermined function f. In order to obtain the function f, first, four reference grid point dots estimated to be arranged at the four corners of the grid area are selected from the photographed reference grid point dots. The positions of the four reference grid point dots are converted, and a function f is obtained so that these are positioned at the corners of the square. The position of the reference grid point dot before conversion in the captured image is (Xi ′, Yi ′) (i = 1, 2, 3, 4), and the position of the reference grid point dot after conversion in the captured image (position at the corner of the square) (Xi, Yi) (i = 1, 2, 3, 4), the function f is (Xi, Yi) = f (Xi ′, Yi ′).
Defined by

  Thus, if the reference grid point dot 4 is used, even if the lens of the photographing unit 120 is distorted or the angle at which the dot pattern is photographed is large, the error in analyzing the dot pattern 1 can be reduced. Can do.

  As shown in FIG. 12, the key dots 2 are arranged at positions shifted from the lattice points in a predetermined direction. By extracting the key dot 2 from the captured image, the direction and size of the block can be specified. When the dot pattern 1 is photographed and the direction of displacement of the key dot 2 is specified in the photographed image, the direction or the related direction is specified as the block direction. The amount of shift of the key dot 2 is, for example, about 20% of the interval between adjacent reference grid point dots. The amount of deviation is not limited to this, and can be varied according to the size of the block. Key dot 2 is a dot representing a block. When defining information in a block, the position of the block on the medium on which the dot pattern 1 is provided can be included in the information (in this embodiment, an ID number for specifying the object is included) ).

  The information dot 3 is a dot for expressing various information. One or more information dots 3 are included in the block. Each information dot 3 individually defines basic information such as numerical information. Various information, for example, the coordinate value of the position of the block, is expressed by the basic information defined in the one or more information dots 3 included in one block. The basic information of the information dot 3 is defined based on at least one of the arrangement, size, shape, and color of the information dot 3. When the basic information is numerical information, the basic information has a value.

  A case where the basic information is defined based on the arrangement of the information dots 3 will be described. The information dots 3 are arranged with reference to virtual lattice points. The virtual lattice point is set at the center of a rectangular lattice region divided by lines forming the lattice. Assume a vector having a virtual grid point as a start point and an information dot 3 position as an end point. The basic information defined in the information dot 3 is specified based on the direction and length of the vector. The vector direction is specified with reference to a predetermined reference direction. As the reference direction, the direction of displacement of the key dots 2 (block direction) is used. A vector, that is, an information value (binary value) corresponding to the arrangement of the information dots 3 is determined in advance. In order to facilitate the specification of the position of the information dot 3, the information dot 3 may be positioned only at some predetermined positions. In the example shown in FIG. 13A, the information dots 3 can be positioned only at eight positions with the virtual lattice point 5 as a reference. In FIG. 13A, the virtual lattice point 5 is drawn with a white circle mark “◯”. The virtual lattice point 5 is a virtual point, and the mark “◯” is not included in the dot pattern 1.

  Since the information dot 3 can take eight states, 3-bit information can be expressed. In FIG. 13A, the value of information corresponding to each state is shown as a binary value. Since one block contains 16 information dots (FIG. 12), one block can express 3 bits × 16 pieces = 48 bits of information.

  The number of states that the information dot 3 can take is not limited to eight. For example, as shown in FIG. 13B and FIG. Since the information dot 3 can take four states, 2-bit information can be expressed. One block can express 2 bits × 16 pieces = 32 bits of information. The number of states that the information dot 3 can take may be 16.

  In the example of FIG. 13, the length of the vector (distance from the virtual dot 5 of the information dot 3) is constant. In the example of FIG. 14, the information dot 3 can take two states, that is, whether the length of the vector is long or short. The information dots 3 can be arranged in eight predetermined directions. That is, in the example of FIG. 14, the information dot 3 can take 2 × 8 = 16 states and can express 4 bits. The length of the longer vector is preferably about 25% of the interval between adjacent reference grid point dots 4, and the length of the shorter vector is preferably about 15% of the interval between adjacent reference grid point dots 4. . However, it is desirable that the length value of the long and short vectors is larger than the diameter value of the information dot 3.

  FIG. 15A shows a state in which two information dots are arranged. FIG. 15B shows a state in which four information dots are arranged. FIG. 15C shows a state in which five information dots are arranged.

  Another embodiment of the object to which the index is applied will be described. A dot pattern is used as an index. In the above-described embodiment of the target object with the index, the dot number information dot information used as the index includes an ID number for identifying the target object. The information dot information of the dot pattern provided on the object includes position information and / or code values on the object.

  As a code value, for example, an ID number for specifying an object can be used. A dot pattern and a graphic (for example, a cat graphic) are overlaid and printed on two cards (objects). Print the same cat graphic on the two cards. If a dot pattern is used, cards on which the same graphic is printed can be distinguished. “1” is set as the ID number for one card, and “2” is set as the ID number for the other card. That is, the code value of one card is “1”, and the code value of the other card is “2”. When these cards are photographed, for example, the characters “Twin Brothers” are displayed in the area of the screen 111 near the card whose ID number is “1”, and the card whose ID number is “2”. Control is performed such that the characters “Twin Brothers” are displayed in the area of the nearby screen 111.

  One or more blocks are printed on the surface of the card (object) to which the dot pattern is applied, and the information represented by the information dot 3 included in each block includes the position information of the block on the card. Can be made. For example, the information on the information dot 3 included in the block can include the coordinates of any reference grid point dot 4 included in the block on the card. Using this, the position of the predetermined area on the card can be specified based on the dot pattern image applied to the predetermined area of the photographed card. At this time, the image analysis unit 161 (FIG. 8) acquires, from the image, information represented by the information dot included in the block applied to the predetermined area, that is, the position information of the block on the card. A predetermined area is specified based on the information. In this way, a predetermined area of the card using the dot pattern, for example, the predetermined area 151 of the card with which the indicator 154 (user's finger) is in contact (see FIGS. 1 and 2). The dot pattern is not shown. ) Can be specified. Since the position of fine dots is used for specifying the area, the area can be specified with extremely high accuracy.

  In the embodiment of the object to which the above index is applied, the optical device 130 can be used to shoot with the shooting area enlarged. Thereby, a fine dot can be more reliably imaged.

  Next, an information processing apparatus according to the fourth embodiment of the present invention will be described. Most of the configuration of the information processing apparatus according to the fourth embodiment is the same as that of the information processing apparatus 200 according to the second embodiment. Components having the same configuration and function as those of the information processing apparatus 200 according to the second embodiment are denoted by the same reference numerals.

  The information processing apparatus according to the fourth embodiment includes a detection unit 265 (FIG. 11) that detects the position of the indicator 154 on the screen 111. Moreover, the dot pattern is given to the target object similarly to 3rd Embodiment. As described above, the predetermined area on the object indicated by the indicator can be specified by using the detection unit 265 or the dot pattern.

  Next, an information processing apparatus 500 according to the fifth embodiment of the present invention will be described. Most of the configuration of the information processing apparatus according to the fifth embodiment is the same as that of the information processing apparatus 100 according to the first embodiment. Components having the same configuration and function as the components of the information processing apparatus 100 according to the first embodiment are denoted by the same reference numerals.

  In the information processing system, the card placed on the screen 111 is photographed by the photographing means 120 together with the screen 111, for example, the corner of the card is specified from the photographed photographed image, and the actionon image is displayed around the corner. Is done. When an action image is displayed around the corner of the card, it is necessary to determine which of the pixels constituting the display image to be displayed on the screen 111 should be displayed. In order to determine this, it is necessary to determine which pixel of the captured image corresponds to which pixel of the display image. In order to obtain this correspondence, a calibration image is used.

  A calibration image is displayed on the screen 111 (FIGS. 1 and 2) of the image display unit 110. The calibration image is photographed by the photographing unit 120 together with the object 153 located on the screen 111 and stored in the image storage unit 121 (FIG. 8). The calibration image associates the position in the display image displayed on the screen 111 (the position of the pixel in the display image) with the position in the captured image stored in the image storage unit 121 (the position of the pixel in the captured image). Used for. The calibration image includes a plurality of identification images that can be distinguished from each other.

  The calibration image can include an image representing a grid line. Here, the grid line refers to a plurality of lines including a plurality of lines extending in one direction and arranged in parallel and a plurality of lines extending in another direction intersecting with the one direction and arranged in parallel. The interval between the parallel lines may or may not be constant.

  FIG. 16A shows a state in which a calibration image including an image representing a grid line is displayed on the screen 111 as a display image. An image representing a cross line passing through the lattice points is an identification image. Each of the plurality of identification images designates a reference point on the screen 111. The reference point on the screen 111 is a virtual point geometrically set on the screen 111. The reference point is a point that serves as a reference for specifying an arbitrary position on the screen 111 shown in FIG. In the example of FIG. 16A, each identification image specifies an intersection of vertical and horizontal lines represented by the identification image, that is, a lattice point as a reference point. An identification image is set for each grid point. The identification image is identified based on at least one of a line type, a line color, a line thickness, and a line interval. The calibration image is formed by a collection of such identification images.

  In the example of FIG. 16B, the identification image captured by the imaging unit 120 and included in the captured image stored in the image storage unit 121 is identified based on the type of line and the thickness of the line. The horizontal line and the vertical line are arranged at equal intervals. There are no lines of the same type and thickness. If the type (for example, dotted line, chain line, broken line, solid line, double line) and the thickness are specified, the horizontal line (for example, the horizontal line 512a of the thin two-dot chain line) can be specified. The position where each horizontal line is displayed is determined in advance. Therefore, the vertical position of the region on the image 111a of the screen 111 on which the horizontal line is displayed can be specified based on the type and thickness of the horizontal line. The same applies to the vertical lines. Based on the type and thickness of the vertical line (for example, a thin solid vertical line 513a), the horizontal position of the area on the image 111a of the screen 111 on which the vertical line is displayed can be specified. If this is used, the position of the reference point on the image 111a of the screen 111 on which the grid point 514a is displayed can be specified based on the type and thickness of the vertical line 513a and the horizontal line 512a passing through the grid point 514a. . In this way, the position of each reference point can be specified. If the positional relationship between each reference point and the object located on the screen 111 can be identified, the position and orientation of the object on the screen 111 can be identified. Similarly, the amount of movement and the amount of rotation of the object can be specified.

  In FIG. 16B, the portion of the screen 111 closer to the photographing means 120 becomes larger and the portion farther away becomes smaller. Therefore, although the grid lines are deformed, the position of each reference point in FIG. 17B in the captured image (the pixel address of the reference point shooting frame buffer) is specified based on the type and thickness of the grid lines. be able to. The image analysis unit 161 (FIG. 8) identifies each identification image included in the captured image based on the captured image, and specifies a captured image reference position indicating the position of the reference point designated by the identified identification image. . The screen image 111a included in the captured image includes an image representing a thin two-dot chain horizontal line 512a and a thin solid vertical line 513a. The image analysis unit 161 identifies the horizontal line 512a based on the type of line and the thickness of the line, and specifies that the horizontal line 512a is a thin two-dot chain horizontal line. The image analysis unit 161 identifies the vertical line 513a based on the type of line and the thickness of the line, and specifies that the vertical line 513a is a thin solid vertical line. That is, the identification image formed by the horizontal line 512a and the vertical line 513a is identified. Next, the image analysis unit 161 determines the position (captured image reference position (xr1, yr1)) in the captured image of the reference point designated by the identified identification image, that is, the lattice point that is the intersection of the horizontal line 512a and the vertical line 513a. Identify. Next, the image analysis unit 161 identifies the same identification image as the identified identification image from the identification images included in the display image displayed on the screen 111.

  The image display unit 110 displays an image on the screen 111 based on the display image data. The display image data includes identification image display information for displaying a plurality of identification images. Further, the display image data includes information on the position of the reference point (grid point) designated by each identification image in the display image (information on the position of the pixel displaying the image of the area around the lattice point). The identification image display information of each identification image includes information for specifying the identification image, for example, information on the type of line and the thickness of the line. The image analysis unit 161 uses the horizontal line 512 and the vertical line having the same line type and line thickness as the specified horizontal line 512a and vertical line 513a (the captured image in FIG. 16B) based on the display image data. 513 (the display image in FIG. 16A) is specified. At this time, the identification image display information of the identification image formed from the horizontal line 512 and the vertical line 513 (display image) is specified from the identification image display information of the plurality of identification images. That is, the same identification image (formed by the horizontal line 512 and the vertical line 513 in the display image) as the identified identification image (formed by the horizontal line 512a and the vertical line 513a in the captured image) is specified. The image analysis unit 161 displays the position (display image reference position (Xr1, Yr1)) in the display image of the reference point (lattice point 514a) designated by the identification image (horizontal line 512a, vertical line 513a) identified as the same. Specify based on image data.

  The image analysis unit 161 obtains a reference correspondence relationship indicating the correspondence between the specified captured image reference position (xr1, yr1) shown in FIG. 17B and the specified display image reference position (Xr1, Yr1). A reference correspondence is obtained for each identification image. That is, the reference correspondence relationship indicating the correspondence between the captured image reference position (xri, yri) and the display image reference position (Xri, Yri) (i = 1, 2, 3,...) Is obtained.

  Note that it is also possible to acquire a captured image having an aspect ratio comparable to that of a display image without distortion using the special optical device 130. Further, for any identification image, by comparing the identification image in the captured frame buffer with the identification image in the display frame buffer, the pixel address of the reference point in the captured frame buffer (captured image reference position) And a correspondence relationship (reference correspondence relationship) between the pixel address of the reference point in the display frame buffer (display image reference position). The position is handled with coordinate values. The xy coordinates are set for the captured image in the imaging frame buffer, and the XY coordinates are set for the display image in the display frame buffer. Once the correspondence between the reference position xy coordinates-XY coordinates is obtained, it is not necessary to display the calibration image on the screen 111 thereafter.

  A conversion function associating the xy coordinates of the captured image reference position with the XY coordinates of the display image reference position or a conversion table as shown in FIG. 17C is obtained and used to determine which pixel of the captured image in the captured frame buffer. It is possible to specify which pixel address (position coordinate) of the display image in the display frame buffer corresponds to the address (position coordinate). In other words, when an object placed on the screen 111, for example, a card is photographed, the position coordinates of the image of the object in the photographing frame buffer are specified, so that the display frame buffer corresponding to the specified position coordinates is stored. The position coordinates in the display image (the position of the card on the screen 111) can be specified. Further, regarding the direction in which the card is placed, for example, in the captured image, the rotation angle of the card relative to the vertical row or horizontal row of reference points is specified, or the positions of at least two points on the card are specified. The card direction on the screen 111 can be specified.

  The position on the screen 111 of the geometrically characteristic point of the card placed on the screen 111, for example, the point forming the center or the frame (contour) can be specified as follows. In the following, the center position of the card is specified. FIG. 18A shows a reference point extracted from the photographed image and an image of the card placed on the screen 111. Four reference points surrounding the card coordinate value (xc, yc) obtained by obtaining the card coordinate value (xc, yc) in the shot image indicating the center of the card by predetermined image processing or pattern recognition in the shot image in the shooting frame buffer. The position coordinates of A1, A2, A3 and A4 are selected. Next, the position coordinates of the four reference points B1, B2, B3, B4 shown in FIG. 18B in the display image in the display frame buffer corresponding to these four reference points A1, A2, A3, A4 are shown. This is specified by the conversion table in FIG. The four reference points B1, B2, B3, and B4 of the display frame buffer surround the center of the card in the display frame buffer placed on the screen 111. Therefore, the position coordinates of the four reference points A1, A2, A3, A4 of the photographing frame buffer of FIG. 18A and the four reference points B1, B2, B3, B4 of the display frame buffer of FIG. Based on the position coordinates and the position coordinates (xc, yc) of the center of the card in the captured image in the captured frame buffer of FIG. 18A, the display of FIG. The card coordinate values (Xc, Yc) in the display image in the frame buffer can be accurately obtained. On the other hand, if the conversion functions X = fx (x, y) and Y = fy (x, y) are obtained based on the conversion table, the card coordinate values (Xc, Yc) can be determined. Such a conversion function includes a method of mapping from a predetermined coordinate system (coordinates of the captured image) to another coordinate system (coordinates of the display image), a method of obtaining the coordinates of the display image by normalizing the captured image, and the like. Use it to find out. When the card coordinate value (Xc, Yc) is obtained using the conversion function, the card coordinate value (xc, yc) is obtained by performing image processing or pattern recognition on the captured image, and the card coordinate value is obtained using the conversion function. (Xc, yc) may be converted into card coordinate values (Xc, Yc). Further, the captured image is converted by a conversion function (the coordinate system of the pixels forming the captured image is converted from the xy coordinate system shown in FIG. 18A to the XY coordinate system shown in FIG. 18B), and converted into the converted image. Alternatively, the center of the card may be obtained by performing image processing or pattern recognition, and the card coordinate value (Xc, Yc) may be acquired.

  In order to specify the position of the object (card) on the screen 111, when a representative point is set at a predetermined position of the card in the photographed image, only geometrically characteristic portions such as the center and frame of the card are used. Instead, the representative point may be set at a predetermined position on the screen 111 corresponding to the position of the graphic drawn on the card (for example, the position ahead of the arrow of the arrow graphic). In this case, a card is extracted from the captured image stored in the captured frame buffer using image recognition, and it is determined whether or not an image similar to a previously recorded arrow exists in the graphic drawn on the card. The image is processed by image processing and pattern recognition, and the arrow is recognized. Of course, the target graphic may be directly specified by the image obtained by converting the photographed image (FIG. 18B) without extracting the card. A predetermined position (representative point) of the graphic may be indicated by an indicator. Similarly, the position of the indicator 154 and the predetermined area 151 on the screen 111 (FIG. 1) can also be specified using a conversion table or a conversion function. When the card is moved on the screen 111, the locus on the screen 111 of the point set on the card can also be specified. If the trajectory of a plurality of points is specified, the amount of movement and the amount of rotation can be specified.

The conversion from the coordinates of the captured image to the coordinates of the display image will be described in more detail. Pixels that form an image of the area around the lattice point 514a in the captured image and pixels that form an image of the area around the lattice point 514 in the display image Corresponds one-to-one. The image analysis unit 161 obtains a position correspondence indicating the correspondence between the position (x, y) in the captured image and the position (X, Y) in the display image based on the reference correspondence. When finding the position correspondence
(X, Y) = F (x, y)
A function F defined by is obtained. The function F determines the position correspondence. If the predetermined algorithm as described above is used, the function F can be obtained. By combining the position correspondence and the captured image stored in the image storage unit 121, at least one of the type, position, orientation, movement amount, rotation amount, and contour of the object can be specified. At least one of the image display unit 110 and the audio reproduction unit 164 can be controlled based on the information on the identified object. For example, by using the function F that determines the positional correspondence, the pixel is located in the surrounding pixels ((xai, yai) (i = 1, 2, 3,...) Of the target image included in the captured image. The pixel (pixel located at (Xai, Yai) (i = 1, 2, 3,...)) Corresponding to (pixel) is obtained ((Xai, Yai) = F (xai, yai) ( i = 1, 2, 3,...)), it is possible to perform control such that a predetermined image is displayed around the object. A table may be used instead of the function F in order to determine the position correspondence. An original display image as shown in FIG. 20B can be obtained by converting the photographed image as shown in FIG. 20A using the function F or a table. 20A and 20B show a card and a coin placed on the screen 111, an indicator (user's finger) indicating the card, and a car image displayed on the screen 111. Yes. Actually, only the car image is displayed in the original display image.

  A symbol (graphic) as shown in FIG. 19A may be used as an identification image of the calibration image instead of the grid lines (in this case, the calibration image includes an image of an array of symbols). The image of each symbol is used as an identification image. The reference point designated by the identification image can be set, for example, at the center of gravity of the symbol. The identification image is identified based on at least one of a symbol shape, a symbol size, a symbol color, and a symbol arrangement pattern. The symbols are arranged on lattice points (not shown). The symbols are arranged at equal intervals. In the example of FIG. 19A, the identification image is identified based on the symbol size and the symbol shape. There are no symbols of the same type and size. The position where each symbol is displayed may be arranged at the position of the lattice point of the lattice as shown in FIG. Based on the symbol type and the symbol size, the position coordinates of the reference point on the screen 111 on which the identification image is displayed can be specified. By using a highly accurate algorithm such as the above-described interpolation method, mapping method, and method of obtaining the coordinates of the display image by normalizing the captured image from the symbols arranged in a grid pattern at the position of the reference point, When the deformation of the captured image is a linear deformation, even if symbols are placed at least at the four corners of the display image during calibration (symbols are placed at the four corners of the screen in the captured image), the deformed captured image is converted. The original display image can be roughly reproduced.

  In FIG. 19B, only dots are arranged at the positions of the reference points arranged in a grid pattern, but when these dots are used as a calibration image, if all the dots are imaged, those images are captured. The correspondence relationship between the coordinate value of the captured image reference position and the coordinate value of the display image reference position can be easily recognized by the arrangement order of the reference points.

  Although not shown, the symbol interval does not have to be constant. The identification image can be identified based on the symbol arrangement pattern.

  In order to obtain the position correspondence relationship indicating the correspondence between the position in the captured image and the position in the display image, an image including the dot pattern shown in FIG. 12 may be displayed on the screen 111 of the image display unit 110. The dot pattern has one or more blocks including information dots representing information. The information represented by the information dots included in each block includes information including the position where the block is displayed. Based on the captured image, the position of a certain block in the captured image is specified. The position where the block is displayed can be acquired from the information represented by the information dot included in the block. As a result, it is possible to acquire a position correspondence relationship indicating the correspondence between the position of the block in the captured image and the position in the display image. It should be noted that the dot pattern placement accuracy (dot interval and dot size) used here is naturally within the resolution of the display image and the picked-up image. It is desirable that the dot interval is about several cm and the dot size is about 1 mm to about several mm. Further, a highly accurate conversion table can be created by interpolation calculation as described in International Publication No. 2010/061584.

  In addition, in order to obtain a position correspondence indicating the correspondence between the position in the captured image and the position in the display image, the shape of the contour of the screen 111 of the captured image display unit 110 can be used. Based on the captured image, a part or all of the contour of the image on the screen 111 of the image display unit 110 included in the captured image is specified. Based on a part or all of the specified contour, a position correspondence relationship indicating a correspondence between the position in the captured image and the position in the display image displayed on the screen of the image display unit can be obtained.

1 dot pattern 2 key dot 3 information dot 4 reference grid point dot 5 virtual grid point 100, 200, 500 information processing device 110 image display unit 111 screen 120 photographing unit 121 image storage unit 130 optical device 140 housing 141 photographing hole 142 housing Upper surface 153, 153a, 153b, 153c, 153d of body Object 151 Predetermined area 161 of object 161 Image analysis unit 162 Control unit 163 Storage unit 164 Audio reproduction unit 265 Detection means

Claims (35)

An image display unit arranged on one side;
An imaging unit that is provided at a predetermined position on the one surface and that captures a direction in which the one surface faces;
An information processing apparatus comprising:
An optical device that guides light to the photographing means so that at least a part of the image of the image display unit is photographed is attached;
An image including an object positioned on the screen of the image display unit, or an image including an object positioned on the screen and an object positioned in the vicinity of the screen;
An information processing apparatus that performs processing based on the captured image. Based on the photographed image, the type of the object, the position of the object located on the screen of the image display unit, the direction of the object located on the screen, and the amount of movement of the object located on the screen Identifying at least one of the amount of rotation of the object located on the screen and the contour of the object located on the screen;
The information processing apparatus according to claim 1, wherein the information processing apparatus performs processing based on information on the identified object. The object is provided with an index including at least one of a graphic, a mark, an arrangement of marks, and a pattern of an automatic recognition code,
Identifying an object based on the image of the index included in the captured image,
The information processing apparatus according to claim 1, wherein the information processing apparatus performs processing based on the identified object. The information processing apparatus according to claim 3, wherein the automatic recognition code includes at least one of a bar code, a two-dimensional code, and a dot code. The object is provided with a dot pattern,
The dot pattern has one or more blocks including one or more information dots,
5. The information expressed by one or more information dots included in each block includes position information and / or code value of the block in the object. The information processing apparatus according to item 1. 6. The information processing apparatus according to claim 5, wherein basic information is defined for each of the information dots based on at least one of an arrangement, size, shape, and color of the information dots. The indicator has a characteristic of absorbing light of a predetermined wavelength, a characteristic of reflecting light of a predetermined wavelength, and a characteristic of emitting light having characteristics different from the light in response to light having the predetermined characteristic. The information processing apparatus according to claim 3, wherein the information processing apparatus has at least one of the following. The information processing apparatus according to claim 1, wherein the processing includes control of the image display unit. The information processing apparatus according to claim 1, wherein the optical apparatus includes at least one of a prism, a mirror, and an optical fiber. The information processing apparatus according to claim 1, wherein the object includes at least one of a print medium, a thin plate-like medium, and a figure. The information processing apparatus according to claim 1, wherein the information processing apparatus has a plate shape. The indicator indicating the predetermined area of the object and the predetermined area of the object, or the indicator in contact with the predetermined area of the object and the predetermined area of the object are further specified based on the captured image. ,
The information according to any one of claims 1 to 11, wherein processing is performed based on a predetermined area of the specified object or based on the predetermined area of the specified object and the indicator. Processing equipment. The object is provided with a dot pattern,
The dot pattern has one or more blocks including one or more information dots,
Information expressed by one or more information dots included in each block includes position information of the block in the object,
13. The information processing apparatus according to claim 12, wherein when the predetermined area is specified based on the captured image, the predetermined area is specified based on block position information on the object. The image display unit or a detection unit that detects a position on the screen of the indicator that is in contact with the object;
Processing is performed based on the detected position of the indicator on the screen and the information of the identified object, or the detected position of the indicator on the screen and the information of the identified object A predetermined area of an object with which the indicator is brought into contact is specified based on the information, and processing is performed based on the specified area of the predetermined object and information on the specified object. The information processing apparatus according to any one of claims 1 to 13. The information processing apparatus according to claim 14, wherein the detection unit is a capacitive touch panel. The information processing apparatus according to claim 1, wherein the captured image forms a still image or a moving image. A moving body video is shot by the shooting means,
The information processing apparatus according to claim 16, wherein when the action of the action body is further specified and processed based on the moving image, the action body is further based on the action of the specified action body. The information processing apparatus according to claim 12, wherein the indicator is a finger. The information processing apparatus according to any one of claims 1 to 18, wherein the optical device is capable of changing a shooting region shot by the shooting unit. The information processing apparatus according to claim 19, wherein the change of the shooting area includes at least one of movement, enlargement, and reduction of the shooting area. A calibration image including a plurality of identification images that can be distinguished from each other is displayed on the screen of the image display unit, and a reference point on the screen is designated by each of the plurality of identification images,
Identifying each identification image included in the photographed image based on the photographed image photographed by the photographing means, specifying a photographed image reference position indicating a position of a reference point designated by the identified identification image;
From the identification image included in the display image displayed on the screen of the image display unit, the same identification image as the identified identification image is identified, and the position of the reference point specified by the identification image identified as the same is determined. Specify the display image reference position to show,
Obtaining a reference correspondence relationship indicating correspondence between the specified captured image reference position and the specified display image reference position;
Based on the reference correspondence relationship, a position correspondence relationship indicating the correspondence between the position in the captured image and the position in the display image is obtained,
21. The information processing apparatus according to claim 1, wherein information on an object is specified based on the position correspondence relationship and the captured image. When obtaining the position correspondence relationship indicating the correspondence between the position (x, y) in the captured image and the position (X, Y) in the display image,
(X, Y) = F (x, y)
The information processing apparatus according to claim 21, wherein a function F for determining the position correspondence or a table for determining the position correspondence defined by (1) is obtained. 23. The information processing apparatus according to claim 21, wherein the calibration image represents a grid line, an array of symbols, or a dot pattern, and includes an image formed by the plurality of identification images. 24. The identification image forming the image representing the lattice line is identified based on at least one of a line type, a line color, a line thickness, and a line interval. The information processing apparatus described in 1. The plurality of identification images forming an image representing the grid lines respectively represent rectangular lines surrounding a plurality of grid areas partitioned by the grid lines,
The information processing apparatus according to claim 23, wherein the identification image is identified based on a ratio of rectangular sides. The identification image forming an image representing the symbol arrangement is identified based on at least one of a symbol shape, a symbol size, a symbol color, and a symbol arrangement pattern. Item 24. The information processing apparatus according to Item 23. An image including a dot pattern is displayed on the screen of the image display unit,
The dot pattern has one or more blocks including one or more information dots,
27. The information represented by one or more information dots included in each block includes information including a position where the block is displayed. Information processing device. Based on the photographed image photographed by the photographing means, specify part or all of the outline of the image on the screen of the image display unit included in the photographed image,
Based on a part or all of the specified contour, a position correspondence relationship indicating a correspondence between a position in a captured image photographed by the photographing unit and a position in a display image displayed on the screen of the image display unit is obtained.
The information processing apparatus according to any one of claims 1 to 27, wherein information on an object is specified based on the position correspondence relationship and the captured image. The information processing apparatus according to any one of claims 1 to 28, wherein the optical device is provided with an irradiation unit that irradiates an imaging region imaged by the imaging unit. 30. The information processing apparatus according to claim 29, wherein the irradiation unit irradiates light having a predetermined wavelength. 2. The optical device is provided with a filter that transmits light of a first wavelength and blocks light of a second wavelength, and the imaging region is imaged through the filter. The information processing apparatus according to any one of claims 30 to 30. 32. The optical device according to any one of claims 1 to 31. 32. An information processing system comprising: the information processing apparatus according to claim 1; an optical apparatus; and an object. 34. A program that is combined with the information processing system according to claim 33 and causes the information processing apparatus to execute the processing. 32. The object according to any one of claims 1 to 31.

Images