JP2012216029A - Program, information storage medium, terminal, server, and marker display body - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a program, an information storage medium, a terminal, and a marker display body that are capable of reducing a processing load and more accurately detecting a marker.SOLUTION: A first rectangular area circumscribed to a first color area formed by plural pixels satisfying a first color condition and a second rectangular area circumscribed to a second color area formed by plural pixels satisfying a second color condition are extracted from a captured image, and it is determined that there is a marker in the case of a positional relationship in which either one of the first rectangular area and the second rectangular area includes the other.

Description

  The present invention relates to a program, an information storage medium, a terminal, a server, and a marker display.

  Conventionally, there is a technique in which a two-dimensional code generated based on predetermined format information is captured by a camera and the code is recognized from the captured image. However, the prior art requires straight line component detection processing, intersection line detection processing, and the like, and this detection processing has a very high processing load. Further, in the process of detecting a color mark as described in Patent Document 1, a color that is not originally a mark may be detected, and the mark may be erroneously determined.

JP 2002-269561 A

  The present invention has been made in view of the problems as described above. The object of the present invention is to reduce a processing load and to detect a marker more accurately, an information storage medium, a terminal, It is to provide a server and a marker display.

  (1) The present invention is a program for performing processing for detecting at least one marker from a captured image captured by an imaging unit, and a first color formed by a plurality of pixels that satisfy the first color condition A process of extracting a first rectangular area circumscribing the area and a second rectangular area circumscribing the second color area formed by a plurality of pixels that satisfy the second color condition from the captured image is performed. The marker functions as a marker determination unit that performs a process of determining whether or not there is the marker based on the positional relationship between the rectangular region extraction unit and the first rectangular region and the second rectangular region, and the marker The present invention relates to a program for determining that the marker is present when one of the first rectangular area and the second rectangular area is in a positional relationship including the other. The present invention also relates to an information storage medium storing the program and a terminal configured as each unit.

  According to the present invention, when the first and second rectangular areas are extracted and one of the first and second rectangular areas is in a positional relationship including the other, it is determined that there is a marker. The marker can be detected while reducing the processing load compared to the prior art. In addition, according to the present invention, since the rectangular area is extracted based on the first and second color conditions, compared to the case where the marker is detected based on the shading and the luminance value (white, black, gray, etc.), Markers can be detected with increased accuracy. Furthermore, according to the present invention, since it is determined whether or not there is a marker using two color conditions, it is possible to detect the marker with higher accuracy than in the case where only one color condition is determined.

  (2) Further, in the program, the information storage medium, and the terminal according to the present invention, the marker determination unit is configured so that the marker is based on a positional relationship between the center position of the first rectangular area and the center position of the second rectangular area. You may make it perform the process which determines whether there exists. According to the present invention, whether or not there is a marker is determined based on the positional relationship between the center position of the first rectangular area and the center position of the second rectangular area. It can be determined whether or not.

  (3) In the program, the information storage medium, and the terminal according to the present invention, when the marker determination unit has a distance between the center position of the first rectangular area and the center position of the second rectangular area within a predetermined distance. In addition, it may be determined that there is the marker. According to the present invention, when the distance between the center position of the first rectangular area and the center position of the second rectangular area is within a predetermined distance, it is determined that there is a marker. It can be determined whether or not.

  (4) In the program, the information storage medium, and the terminal according to the present invention, when the rectangular area extraction unit has a color value of a pixel within a first color value area including the first color value, If the color value of a pixel is within the second color value range including the second color value, it may be determined that the second color condition is satisfied. According to the present invention, the first and second rectangular regions can be easily extracted.

  (5) In the program, the information storage medium, and the terminal of the present invention, the first color value and the second color value may be in a complementary color relationship. That is, if the first color value and the second color value have a complementary color relationship, the first and second rectangular regions can be easily extracted, and the marker can be detected with high accuracy.

  (6) In the program, the information storage medium, and the terminal of the present invention, the marker determination unit may determine whether there are a plurality of the markers.

  (7) The program, the information storage medium, and the terminal according to the present invention may be configured such that when the marker determination unit determines that there are a plurality of markers, the code information from the captured image is based on the positional relationship between the plurality of markers. The computer may be caused to function as a code recognition processing unit that performs processing for recognizing. According to the present invention, when the marker determination unit determines that there are a plurality of markers, it is possible to perform processing for recognizing code information from the captured image based on the positional relationship between the plurality of markers.

  (8) In the program, the information storage medium, and the terminal according to the present invention, the marker determination unit has the first marker when the second rectangular area is in a positional relationship including the first rectangular area. When the first rectangular area has a positional relationship including the second rectangular area, the process for determining that there is a second marker is performed, and the code recognition processing unit When it is determined by the determination unit that there is one first marker and two or more predetermined number of second markers, the code is based on the positional relationship between the first marker and each second marker. Processing for recognizing information may be performed.

  According to the present invention, when the marker determination unit determines that there is one first marker and two or more predetermined number of second markers, the positional relationship between the first marker and each second marker Therefore, the code information can be recognized in a more correct environment.

  (9) The program, information storage medium, and terminal of the present invention generate an augmented reality image by synthesizing the captured image and the virtual image when the marker determination unit determines that there are a plurality of markers. The computer may further function as a drawing unit that performs the processing. According to the present invention, an augmented reality image can be generated when a plurality of markers are detected.

  (10) The program, the information storage medium, and the terminal of the present invention include the first marker when the marker determination unit has a positional relationship in which the second rectangular area includes the first rectangular area. When the first rectangular area is in a positional relationship including the second rectangular area, a process for determining that there is a second marker is performed, and the drawing unit performs the marker determination unit When it is determined that there is one first marker and two or more predetermined number of second markers, the process of generating the augmented reality image may be performed. According to the present invention, when the marker determination unit determines that there is one first marker and two or more predetermined number of second markers, the process of generating the augmented reality image is performed, so that a more correct environment Below, augmented reality images can be provided.

  (11) Further, the program, the information storage medium, and the terminal according to the present invention provide code information from the captured image based on the positional relationship of the plurality of markers when the marker determination unit determines that there are a plurality of markers. The computer functions as a code recognition processing unit that performs processing for recognizing the image, and the drawing unit performs processing for generating a virtual image based on the code information, and combines the captured image and the virtual image for expansion. A real image may be generated. According to the present invention, since a process for generating a virtual image is performed based on code information, an augmented reality image in which a virtual image corresponding to the code information and a captured image are combined can be generated.

  (12) The present invention is a server that performs processing for detecting at least one marker, and a communication control unit that receives a captured image from a terminal via a network, and a plurality of adjacent ones that satisfy the first color condition A first rectangular area circumscribing a first color area composed of pixels and a second rectangular area circumscribing a second color area composed of a plurality of adjacent pixels that satisfy the second color condition A rectangular area extracting unit that performs processing of extracting from an image, and a marker determining unit that performs processing of determining whether or not there is the marker based on a positional relationship between the first rectangular area and the second rectangular area; The marker determination unit determines that there is the marker when either one of the first rectangular region and the second rectangular region includes the other, and the communication control unit , Send the processing result to the terminal To about the server.

  (13) The present invention is a marker display body composed of a plurality of markers detected by a terminal, wherein a plurality of markers having the same center point and composed of two circles having different diameters are arranged at predetermined intervals. The inner circle of one of the plurality of markers is colored in the first color, and the portion of the outer diameter circle except the inner circle is colored in the second color, and the remaining markers The present invention relates to a marker display body in which the inner circle is colored in the second color, and the portion of the outer diameter circle excluding the inner circle is colored in the first color.

  According to the present invention, a marker composed of two circles having the same center point and different diameter lengths is arranged, and the marker is an inner diameter circle of a first color and an outer diameter circle. The portion excluding the inner circle is colored in the second color, or the inner circle is colored in the second color, and the portion of the outer diameter circle except the inner circle is colored in the first color. ing. Therefore, it is possible to provide a marker display body capable of detecting a marker considering the first and second colors from any direction. That is, it is possible to provide a marker display body in which markers that can be detected with higher accuracy than one color marker are arranged. The marker display body may be a two-dimensional display body or a three-dimensional display body.

  (14) Moreover, the marker display body of the present invention may be a marker display body characterized by showing predetermined code information within a range surrounded by the respective markers. According to the present invention, it is possible to provide a marker display body that recognizes code information when a plurality of markers are detected by a terminal.

  (15) Moreover, the marker display body of the present invention may be a marker display body characterized in that the first color is red and the second color is green. ADVANTAGE OF THE INVENTION According to this invention, the marker display body by which the marker which is easy to detect with a terminal is arrange | positioned can be provided.

FIG. 1 is an example of a functional block diagram of a terminal according to the present embodiment. 2A and 2B are examples of the terminal according to the present embodiment. 3A and 3B are examples of the terminal according to the present embodiment. FIG. 4 is an example of a marker display. 5A and 5B are explanatory diagrams of markers. 6A, 6B, and 6C are explanatory diagrams of code information. FIG. 7 is an explanatory diagram of a first rectangular area. FIG. 8 is an explanatory diagram of a second rectangular area. FIG. 9A is a flowchart of rectangular area extraction processing. FIG. 9B is a flowchart of rectangular area extraction processing. FIG. 9C is a flowchart of rectangular area extraction processing. FIG. 9D is a flowchart of rectangular area extraction processing. 10A to 10D are explanatory diagrams of rectangular area extraction processing. 11A, 11B, and 11C are explanatory views of the positional relationship between the first and second rectangular regions. 12A, 12B, and 12C are explanatory views of the positional relationship between the first and second rectangular regions. FIG. 13 is an explanatory diagram of a situation in which a marker display body is imaged. FIG. 14A is an explanatory diagram for explaining processing for extracting first and second rectangular regions from a captured image obtained by capturing a marker display body. FIG. 14B is an explanatory diagram for explaining processing for extracting the first and second rectangular regions from the captured image obtained by capturing the marker display body. FIG. 14C is an explanatory diagram for explaining processing for extracting the first and second rectangular regions from the captured image obtained by capturing the marker display body. FIGS. 15A, 15 </ b> B, and 15 </ b> C are explanatory diagrams for explaining an example in which code information is recognized based on the positional relationship between markers. FIG. 16A is an explanatory diagram of a captured image. FIG. 16B is an explanatory diagram for explaining an example in which an augmented reality image is generated by combining a captured image and a virtual image. FIG. 16C is an explanatory diagram of an augmented reality image. FIG. 17 is a flowchart of this embodiment. 18A, 18B, 18C, 18D, and 18E are application examples of marker display bodies. FIG. 19 is an explanatory diagram for explaining an example in which an augmented reality image is generated by combining a captured image and a virtual image. 20A, 20B, and 20C are explanatory diagrams of adjacent and overlapping relationships. FIG. 21 is an explanatory diagram of a network system.

  Hereinafter, this embodiment will be described. In addition, this embodiment demonstrated below does not unduly limit the content of this invention described in the claim. In addition, all the configurations described in the present embodiment are not necessarily essential configuration requirements of the present invention.

1. Configuration FIG. 1 shows an example of a functional block diagram of a terminal (a computer, a game device, a mobile phone, a smartphone, a mobile terminal, a portable game device, and an image generation device) of this embodiment. Note that the terminal of this embodiment may have a configuration in which some of the components (each unit) in FIG. 1 are omitted.

  The imaging unit 150 forms an image of light emitted from an object on a light receiving plane of an imaging element by an optical system such as a lens, photoelectrically converts light and darkness of the image into an amount of electric charge, sequentially reads out the electric signal, and Convert. Then, the imaging unit 150 performs processing to output a color image (input image) defined in a color space (format) such as RGB, YUV, or HSV to the captured image storage unit 173. The imaging unit 150 performs a process of outputting a color image captured at a predetermined cycle (for example, every 1/60 seconds) to the captured image storage unit 173. That is, processing for sequentially storing color images in the captured image storage unit 173 is performed.

  The input unit 160 is an input device (controller) for inputting input information from the player (operator), and outputs the input information of the player to the processing unit. The input unit 160 of this embodiment includes a detection unit 162 that detects input information (input signal) of the player. The input unit 160 includes, for example, a lever, a button, a steering, a microphone, a touch panel display, and the like. The input unit 160 may include a vibration unit that performs a process of vibrating based on the vibration signal.

  The input unit 160 may be an input device including an acceleration sensor that detects triaxial acceleration, a gyro sensor that detects angular velocity, and the imaging unit 150. For example, the input unit 160 may be held and moved by the player, or may be worn by the player and moved. The input unit 160 may be an input device imitating an actual tool such as a sword-type controller or gun-type controller held by the player, or a glove-type controller worn by the player (attached to the hand of the player). Good. The input unit 160 also includes a terminal (a mobile phone, a mobile terminal, a portable game device) integrated with an input device.

  The storage unit 170 serves as a work area for the processing unit 100, the communication unit 196, and the like, and its function can be realized by a RAM (VRAM) or the like.

  The information storage medium 180 (computer-readable medium) stores programs, data, and the like, and functions as an optical disk (CD, DVD), magneto-optical disk (MO), magnetic disk, hard disk, and magnetic tape. Alternatively, it can be realized by a memory (ROM). The processing unit 100 performs various processes of the present embodiment based on a program (data) stored in the information storage medium 180. The information storage medium 180 can store a program for causing a computer to function as each unit of the present embodiment (a program for causing a computer to execute processing of each unit).

  The display unit 190 outputs an image generated according to the present embodiment, and its function can be realized by a CRT, LCD, touch panel display, HMD (head mounted display), or the like.

  The sound output unit 192 outputs the sound generated by the present embodiment, and its function can be realized by a speaker, headphones, or the like.

  The communication unit 196 performs various controls for communicating with the outside (for example, other terminals and servers), and the functions are realized by hardware such as various processors or communication ASICs, programs, and the like. it can.

  Note that a program or data for causing a computer to function as each unit of the present embodiment stored in the information storage medium or storage unit of the server is received via the network, and the received program or data is received by the information storage medium 180. Or may be stored in the storage unit 170. The case where the terminal is functioned by receiving the program or data as described above is also included in the scope of the present invention.

  The processing unit 100 (processor) performs processing such as game processing, image generation processing, or sound generation processing based on input data from the input unit 160, a program, and the like.

  The processing unit 100 performs various processes using the main storage unit 171 in the storage unit 170 as a work area. The functions of the processing unit 100 can be realized by hardware such as various processors (CPU, DSP, etc.), ASIC (gate array, etc.), and programs.

  The processing unit 100 includes a rectangular area extraction unit 110, a marker determination unit 111, a code recognition processing unit 112, a drawing unit 120, and a sound processing unit 130. Note that some of these may be omitted.

  The rectangular area extraction unit 110 is formed by a first rectangular area circumscribing the first color area formed by a plurality of pixels that satisfy the first color condition and a plurality of pixels that satisfy the second color condition. A process of extracting the second rectangular area circumscribing the second color area from the captured image is performed.

  Here, the rectangular area extraction unit 110 determines that the first color condition is satisfied when the color value of the pixel is within the first color value range including the first color value, and the color value of the pixel is When the second color value is within the second color value range including the second color value, it is determined that the second color condition is satisfied. Note that it is desirable that the first color value and the second color value have a complementary color relationship. For example, the first color value may be a red color value, and the second color value may be a green color value.

  The marker determination unit 111 performs a process of determining whether or not there is a marker based on the positional relationship between the first rectangular area and the second rectangular area. For example, the marker determination unit 111 determines that there is a marker when one of the first rectangular region and the second rectangular region has a positional relationship including the other.

  In addition, the marker determination unit 111 may perform processing for determining whether there is a marker based on the positional relationship between the center position of the first rectangular area and the center position of the second rectangular area. . That is, the marker determination unit 111 may determine that there is a marker when the distance between the center position of the first rectangular area and the center position of the second rectangular area is within a predetermined distance.

  In addition, the marker determination unit 111 performs a process of determining that there is a first marker when the second rectangular area includes the first rectangular area, and the first rectangular area is the second rectangular area. When the positional relationship includes the rectangular area, a process of determining that the second marker is present is performed.

  When the marker determination unit 111 determines that there are a plurality of markers, the code recognition processing unit 112 performs processing for recognizing code information from the captured image based on the positional relationship between the plurality of markers.

  In addition, when the marker determination unit 111 determines that there is one first marker and two or more predetermined numbers (for example, three) of second markers, the code recognition processing unit 112 Based on the positional relationship between the marker and each second marker, processing for recognizing code information is performed.

  The drawing unit 120 performs processing for arranging an object in an object space (virtual three-dimensional space). For example, the object placement unit performs processing for placing a display object such as a building, a stadium, a car, a tree, a pillar, a wall, and a map (terrain) in the object space. Here, the object space is a virtual game space, for example, a space in which objects are arranged in three-dimensional coordinates (X, Y, Z) such as a world coordinate system and a virtual camera coordinate system.

  For example, the object placement unit places an object (an object composed of a primitive such as a polygon, a free-form surface, or a subdivision surface) in the world coordinate system. Also, for example, the position and rotation angle (synonymous with direction and direction) of the object in the world coordinate system is determined, and the rotation angle (X, Y, Z axis rotation) is determined at that position (X, Y, Z). Position the object at (Angle).

  The drawing unit 120 performs drawing processing based on the results of various processes performed by the processing unit 100, thereby generating an image and outputting the image to the display unit 190. In other words, the drawing unit 120 generates an image that can be seen from the virtual camera in the object space.

  For example, the drawing unit 120 receives object data (model data) including vertex data (vertex position coordinates, texture coordinates, color data, normal vector, α value, etc.) of each vertex of the object (model). Based on the vertex data included in the object data, vertex processing (shading by a vertex shader) is performed. When performing the vertex processing, vertex generation processing (tessellation, curved surface division, polygon division) for re-dividing the polygon may be performed as necessary.

  In the vertex processing, according to the vertex processing program (vertex shader program, first shader program), vertex movement processing, coordinate transformation, for example, world coordinate transformation, visual field transformation (camera coordinate transformation), clipping processing, perspective transformation (projection transformation) ), Geometry processing such as viewport conversion is performed, and based on the processing result, the vertex data given to the vertex group constituting the object is changed (updated or adjusted).

  Then, rasterization (scan conversion) is performed based on the vertex data after the vertex processing, and the surface of the polygon (primitive) is associated with the pixel. Subsequent to rasterization, pixel processing (shading or fragment processing by a pixel shader) for drawing pixels (fragments forming a display screen) constituting an image is performed. In pixel processing, according to a pixel processing program (pixel shader program, second shader program), various processes such as texture reading (texture mapping), color data setting / change, translucent composition, anti-aliasing, etc. are performed, and an image is processed. The final drawing color of the constituent pixels is determined, and the drawing color of the perspective-transformed object is output (drawn) to the image buffer 172 (buffer that can store image information in units of pixels; VRAM, rendering target, frame buffer). . That is, in pixel processing, per-pixel processing for setting or changing image information (color, normal, luminance, α value, etc.) in units of pixels is performed. Thereby, an image that can be seen from the virtual camera (given viewpoint) in the object space is generated.

  Note that the vertex processing and pixel processing are realized by hardware that enables polygon (primitive) drawing processing to be programmed by a shader program written in a shading language, so-called programmable shaders (vertex shaders and pixel shaders). Programmable shaders can be programmed with vertex-level processing and pixel-level processing, so that the degree of freedom of drawing processing is high, and expressive power is greatly improved compared to conventional hardware-based fixed drawing processing. Can do.

  The drawing unit 120 performs geometry processing, texture mapping, hidden surface removal processing, α blending, and the like when drawing an object.

  In the geometry processing, processing such as coordinate conversion, clipping processing, perspective projection conversion, or light source calculation is performed on the object. Then, the object data (positional coordinates of object vertices, texture coordinates, color data (luminance data), normal vector, α value, etc.) after geometry processing (after perspective projection conversion) is stored in the storage unit 170. .

  Texture mapping is a process for mapping a texture (texel value) stored in the storage unit 170 to an object. Specifically, the texture (surface properties such as color (RGB) and α value) is read from the storage unit 170 using texture coordinates or the like set (given) to the vertex of the object. Then, a texture that is a two-dimensional image is mapped to an object. In this case, processing for associating pixels with texels, bilinear interpolation or the like is performed as texel interpolation.

  As the hidden surface removal processing, hidden surface removal processing can be performed by a Z buffer method (depth comparison method, Z test) using a Z buffer (depth buffer) in which Z values (depth information) of drawing pixels are stored. . That is, when drawing pixels corresponding to the primitive of the object are drawn, the Z value stored in the Z buffer is referred to. Then, the Z value of the referenced Z buffer is compared with the Z value at the drawing pixel of the primitive, and the Z value at the drawing pixel is a Z value (for example, a small Z value) on the near side when viewed from the virtual camera. In some cases, the drawing process of the drawing pixel is performed and the Z value of the Z buffer is updated to a new Z value.

  α blending (α synthesis) is a translucent synthesis process (usually α blending, addition α blending, subtraction α blending, or the like) based on an α value (A value).

  For example, in α blending, the drawing color (overwriting color) C1 to be drawn in the image buffer 172 and the drawing color (background color) C2 already drawn in the image buffer 172 (rendering target) are set to α values. Based on this, a linear synthesis process is performed. That is, if the final drawing color is C, it can be obtained by C = C1 * α + C2 * (1−α).

  The α value is information that can be stored in association with each pixel (texel, dot), for example, plus alpha information other than color information. The α value can be used as mask information, translucency (equivalent to transparency and opacity), bump information, and the like.

  In particular, when the marker determination unit 111 determines that there are a plurality of markers, the drawing unit 120 of the present embodiment performs a process of generating an augmented reality image by combining the captured image and the virtual image. For example, the drawing unit 120 according to the present embodiment performs a process of generating an augmented reality image when the marker determining unit 111 determines that there is one first marker and two or more predetermined numbers of second markers. Do.

  In addition, the drawing unit 120 of the present embodiment performs a process of generating a virtual image based on the code information recognized by the code recognition processing unit 112, and generates an augmented reality image by synthesizing the captured image and the virtual image. You may make it do.

  The sound processing unit 130 performs sound processing based on the results of various processes performed by the processing unit 100, generates a game sound such as BGM, sound effect, or sound, and outputs the game sound to the sound output unit 192.

  Note that the terminal according to the present embodiment may be controlled so that the game can be played in a single player mode in which only one player can play or in a multiplayer mode in which a plurality of players can play. For example, in the case of controlling in the multiplayer mode, game processing may be performed by transmitting / receiving data to / from other terminals via a network, or one terminal may receive input information from a plurality of input units. Processing may be performed based on this.

2. Overview In the present embodiment, processing for detecting at least one marker from a captured image captured by the imaging unit 150 of the terminal 10 is performed. For example, the terminal 10 is a game device as shown in FIGS. 2A and 2B, a portable terminal (smart phone) as shown in FIGS.

  And the terminal 10 of this embodiment images the marker display body 20 as shown in FIG. 4 as a subject, and detects the marker M displayed on the marker display body 20. The terminal 10 can continue imaging in a predetermined cycle (for example, 1/60 second unit), and the terminal 10 can detect the marker M in the predetermined cycle.

  The terminal 10 includes a first rectangular area circumscribing the first color area formed by a plurality of pixels that satisfy the first color condition and a first pixel that is formed by a plurality of pixels that satisfy the second color condition. The second rectangular area circumscribing the second color area is extracted from the captured image captured by the imaging unit 150.

  And the terminal 10 performs the process which determines whether there exists a marker based on the positional relationship of a 1st rectangular area and a 2nd rectangular area. That is, the terminal 10 performs a process of determining that there is a marker when one of the first rectangular area and the second rectangular area has a positional relationship including the other. In this way, it is possible to determine whether or not there is a marker without detecting a straight line component and a crossing component from the captured image. That is, the processing load can be reduced as compared with the prior art. Since it is determined whether or not there is a marker based on the positional relationship between the first and second rectangular regions extracted based on the two color conditions, the marker is detected (recognized with higher accuracy than in the prior art). )can do. Hereinafter, the marker detection process of this embodiment will be described in detail.

3. Description of Marker Display First, the marker display 20 that is a subject of the imaging unit 150 will be described. As shown in FIG. 4, the marker display body 20 has a plurality of markers M arranged at predetermined intervals. In other words, in the example of FIG. 4, four markers M are formed at the corners of a quadrangle (square).

  And as shown to FIG. 5 (A) (B), the marker M is comprised by two circle | round | yen which has the same center point O and differs in the length of a diameter. That is, the marker M is composed of a circle with a radius r1 centered on the point O and a circle with a radius r2 centered on the point O (r1 <r2). For example, the radius r2 can be twice as long as the radius r1.

  And the marker display body 20 of this embodiment has arrange | positioned the 1st marker M1 from which a color scheme differs, and the 2nd marker M2. That is, as shown in FIG. 5 (A), the first marker M1 is arranged with the inner color circle in the first color and the outer diameter circle excluding the inner diameter circle in the second color, As shown in FIG. 5B, for each of the remaining second markers, the inner diameter circle is colored in the second color, and the portion of the outer diameter circle excluding the inner diameter circle is colored in the first color. ing.

  It is desirable that the first color and the second color have a complementary color relationship. For example, the first color can be red R and the second color can be green G. The complementary color relationship refers to a relationship positioned in the opposite direction in the hue circle. Complementary colors include not only green for red but also purple for yellow and orange for blue.

  That is, the first marker M1 colors the circle with the radius r1 in red R, and colors the portion of the circle with the radius r2 excluding the circle with the radius r1 (ring-shaped portion) into the green G. In addition, the second marker M2 colors the circle with the radius r1 to green G, and colors the portion of the circle with the radius r2 excluding the circle with the radius r1 (ring-shaped portion) to the red R.

  In particular, when the terminal 10 detects a marker, the marker display body 20 of the present embodiment uses one of the four markers M as a first marker and the remaining three as second markers. The positional relationship between the markers M can be easily detected.

  Further, code information is shown on the marker display body 20. For example, as shown in FIG. 4, code information indicated by white or black is displayed in a range surrounded by four markers M.

  The code information can be a numerical value, a character, a barcode, a pictograph, or a design. The code information of the present embodiment is 8-bit information. For example, as shown in FIG. 6A, 8 squares 0 to 7 other than the central square among 9 squares composed of 9 squares in 3 rows and 3 columns are associated with each 8-bit position. White and 0 are displayed in black. The central cell is used as a parity bit. In this way, for example, code information as shown in FIGS. 6B and 6C can be displayed.

  In the example of FIG. 4, an example (print medium) of a marker display body printed on paper is shown. However, the marker display body is an object, for example, a medium, an image, and paper displayed in two dimensions. , A bulletin board, a board, or a three-dimensional display object (ball, marble, building block, light source). For example, in the case of a display image, it may be a two-dimensional image or a stereoscopic image that looks stereoscopic to a viewer.

4). Detailed Description of Processing for Extracting Rectangular Area In the present embodiment, first and second rectangular areas are extracted in order to determine whether or not there is a marker. For example, a first rectangular region circumscribing a first color region formed by a plurality of pixels that satisfy the first color condition and a second color region formed by a plurality of pixels that satisfy the second color condition A process of extracting the second rectangular area circumscribing from the captured image is performed. In other words, a pixel satisfying the first color condition is detected, and a first rectangular area including the first color area formed by the detected pixel and a pixel satisfying the second color condition are detected from the captured image. Then, the second rectangular area including the second color area formed by the detected pixels is extracted from the captured image.

  Whether the terminal 10 of the present embodiment satisfies all the pixels of the captured image in a predetermined order in a red color condition (an example of the first color condition) or a green color condition (an example of the second color condition). Judging. The terminal 10 uses a plurality of adjacent pixel groups that satisfy the red color condition as a red region R (an example of a first color region), and when the plurality of red regions R are adjacent, the captured image A plurality of adjacent red regions R are connected so as to be parallel to the Y-axis and X-axis, a rectangle is set, and this rectangle is extracted as a red rectangular region BR (an example of a first rectangular region).

  Similarly, the terminal 10 sets a plurality of adjacent pixel groups satisfying the green color condition as a green region G, and when the plurality of green regions G are adjacent, the terminal 10 is parallel to the Y axis and the X axis of the captured image. A plurality of adjacent green regions G are connected so as to form a rectangle, and this rectangle is extracted as a green rectangular region BG (an example of a first rectangular region).

  That is, the red rectangular area BR is a rectangular area (first circumscribed rectangular area, first bounding box, first square) circumscribing the red area R in the captured image, and the green rectangular area BG is in the captured image. This is a rectangular area circumscribing the green area B (second circumscribed rectangular area, second bounding box, second square). That is, when the plurality of red regions are connected, the red rectangular region BR is set based on the red region having the maximum length in the X-axis direction and the Y-axis direction. Further, when connecting a plurality of green regions, the green rectangular region BR is set based on the green region having the maximum length in the X-axis direction and the Y-axis direction.

  For example, as illustrated in FIG. 7, when the first mark is captured by the imaging unit 150, the red rectangular region BR of the red region R that is the red portion of the captured image is extracted and the green portion of the captured image is displayed. A green rectangular region BG of the green region G is extracted.

  Also, as shown in FIG. 8, when the second mark is imaged by the imaging unit 150, the red rectangular area BR of the red area R, which is the red part of the captured image, is extracted and the green part of the captured image is displayed. A green rectangular region BG of the green region G is extracted.

4-1. First and second color conditions The terminal 10 of the present embodiment scans pixels from the upper left pixel of the captured image in order from left to right, and when one line scan is completed, the next line is scanned from left to right. Scan pixels sequentially in the direction. Finally, the pixel at the lower right of the captured image is scanned and the process ends.

  Then, the terminal 10 determines whether each pixel satisfies the first color condition or the second color condition. The first color condition is a condition that the color value of the pixel is in the first color value range including the first color value. For example, the red color condition that the UV value of the pixel is in the red value range including the red color value is set as the first color condition. That is, when the pixel color value is in the red value range including red, vermilion, wine red, etc., it is detected as a pixel that satisfies the first color condition.

  The second color condition is a condition that the color value of the pixel is within the second color value range including the second color value. For example, the green color condition that the UV value of the pixel is in the green value range including the green color value is set as the second color condition. That is, when the color value of the pixel is a green region including a green primary color, a light green color, a dark green color, and the like, the pixel is detected as a pixel that satisfies the second color condition.

  As described above, in the present embodiment, the second color value is a color that has a complementary color relationship with the first color value. However, the second color value does not necessarily have a complementary color relationship. If there is a complementary color relationship, the two colors can be easily identified, and the accuracy of marker detection can be increased. In short, when the first condition is satisfied, the second color condition is not satisfied, and when the second color condition is satisfied, the first color condition is not satisfied. The second color condition is defined.

4-2. Extraction of Rectangular Area In the present embodiment, it is determined whether or not the first or second color condition is satisfied for all pixels of the captured image, and the red rectangular area BR and the green rectangular area BG are extracted.

  In the present embodiment, it may be determined whether each pixel of the captured image satisfies the first and second color conditions with reference to a lookup table. This is because the use of a lookup table enables efficient processing. For example, a numerical value (0 or 1) indicating whether or not the first color condition is satisfied is set in association with each color value (U value or V value when defined in the YUV format), and each color is set. A lookup table in which a numerical value (0 or 1) indicating whether or not the second color condition is satisfied is set in correspondence with the value is prepared.

  When determining whether or not the first and second conditions are satisfied based on the color value of the YUV value, the Y value (luminance value) is greater than or equal to a predetermined Y value (predetermined luminance value) or within a given range. May be acquired, and it may be determined whether or not the acquired UV value of the pixel satisfies the first and second conditions. In this way, since the dark color is excluded, the accuracy of detecting the marker can be increased.

  When each color value is defined in RGB, a lookup table in which a numerical value (0 or 1) indicating whether or not the first color condition is satisfied is set in association with each RGB value; A lookup table in which a numerical value (0 or 1) indicating whether or not the second color condition is satisfied may be prepared. Also, when each color value is defined by HSV, a lookup table may be prepared in association with each value in the same manner as RGB.

4-2-1. Rectangular Area Extraction Processing Next, the flow of processing for extracting a rectangular area in the present embodiment will be described with reference to FIGS.

  First, FIG. 9A is a flowchart showing an outline of rectangular area extraction processing. In the rectangular area extraction process, scanning is performed line by line from the top to the bottom. That is, rectangular area setting control (step S1) is performed, rectangular area deletion control (step S2) is performed, and rectangular area combination control (step S3) is performed. It is determined whether or not all the lines have been scanned (step S4). If all the lines have not been scanned (step S4: no), the process proceeds to the next line (step S5), and the process proceeds to step S1. Return. If all the scans have been completed (step S4: yes), the process is terminated.

4-2-2. Rectangular Area Setting Control FIG. 9B is a flowchart showing the rectangular area setting control (step S1). In the rectangular area setting control, step processing is performed for each pixel in the order from left to right on the line until processing of all the pixels on the line is completed. First, it is determined whether or not the pixel color value is within the red value range (step S11). If the color value of the pixel is within the red value range (yes in step S11), the process proceeds to step S13. On the other hand, if the color value of the pixel is not within the red value range (no in step S11), it is determined whether the color value of the pixel is within the green value range (step S12). If the color value of the pixel is in the green value range (yes in step S12), the process proceeds to step S13. If the color value of the pixel is not in the green value range (no in step S12), the process proceeds to step S16.

  And it is judged whether it is the same color and adjoining of any rectangular area (step S13). Here, in step S13, being adjacent to one of the rectangular areas means that the pixel is in contact with a part of the rectangular area in the X-axis direction and the Y-axis direction, or in the X-axis direction and the Y-axis direction. This includes the case where the pixel is located one pixel away from the rectangular area (slight difference).

  If it is the same color and adjacent to any of the rectangular areas (yes in step S13), the rectangular area is expanded (step S14). That is, when the N-th line and the N + 1-th line are adjacent to each other, a process of expanding the rectangular area by one pixel in the height direction (Y-axis direction) is performed and adjacent to the line direction (X-axis direction). In this case, a process of expanding the rectangular area by one pixel in the line direction is performed so as to circumscribe the rectangular area adjacent to the pixel.

  For example, as shown in FIG. 10 (A), a rectangular region BG1 including a green region generated in the Nth line is set, and as shown in FIG. 10 (B), each pixel in the (N + 1) th line is scanned and BG2 As a result, the rectangular region BG1 is adjusted so as to be expanded in the X-axis direction (one pixel in the left direction and the right direction) and adjusted to be expanded in one pixel in the Y-axis direction (height direction). Then, as shown in FIG. 10C, it is expanded to the rectangular area BG3.

  On the other hand, if it is the same color and not adjacent to any rectangular area (no in step S13), a new rectangular area is set (step S15). That is, a new rectangular area circumscribing the pixel being scanned is set.

  Then, it is determined whether or not scanning of all pixels on the line has been completed (step S16). If scanning of all pixels on the line has not been completed (no in step S16), the next pixel is processed. As an object (step S17), the process returns to step S11. On the other hand, when the scanning of all the pixels on the pixel line is completed (Yes in step S16), the process is terminated.

4-2-3. Rectangular Area Deletion Control FIG. 9C is a flowchart showing a flow of rectangular area deletion control processing (step S2). The rectangular area deletion control is performed in order to reduce the noise and detect the marker more accurately by treating the rectangular area of a predetermined size (for example, one pixel and two pixels) as dust.

  First, it is determined whether there is a rectangular area that may be adjacent to or partially overlap the rectangular area (step S21). If there is no rectangular area that may be adjacent to or partially overlap the rectangular area (No in step S21), it is determined whether the rectangular area is smaller than a predetermined size (step S22). If the rectangular area is smaller than the predetermined size (yes in step S22), a process of deleting the rectangular area as dust is performed (step S23). On the other hand, if there is a rectangular area that may be adjacent to or partially overlap the rectangular area (Yes in step S21), if the rectangular area is not smaller than the predetermined size (no in step S22), the process proceeds to step S24. move on.

  Then, it is determined whether or not processing has been completed for all rectangular areas (step S24). If processing has not been completed for all rectangular areas (no in step S24), the next rectangular area is processed. (Step S25), and the process returns to Step S21. On the other hand, when the process is completed for all the rectangular areas (yes in step S24), the process is terminated.

4-2-4. Rectangular Area Combining Control FIG. 9D is a flowchart showing the processing flow of rectangular area combining control (step S3). The rectangular area combination control is a process for combining adjacent rectangular areas between lines in a rectangular area set on a line.

  First, it is determined whether or not the Nth rectangular area is adjacent to or overlaps with the Mth rectangular area of the same color (step S31).

  If the Nth rectangular area is adjacent to or overlaps with the Mth rectangular area of the same color (Yes in step S31), the Nth rectangular area is combined with the Mth rectangular area (step S32). ). That is, two rectangular areas are defined as one rectangular area.

  On the other hand, if the Nth rectangular area is not adjacent or overlapped with the Mth rectangular area of the same color (no in step S31), whether or not the processing has been completed for all rectangular areas on the line. If it is determined (step S33) and the process is not completed (no in step S33), the process returns to step S31 with the next rectangular area (step S34) as the processing target.

  For example, as shown in FIG. 10A, when the rectangular area BG1 is set in the line N and the rectangular area BG2 is set in the line N + 1 as shown in FIG. 10B, the rectangular area BG1 of the line N Since the rectangular area BG2 of the line N + 1 is adjacent or overlapped, as shown in FIG. 10C, the rectangular area BG1 and the rectangular area BG2 are combined to newly set the rectangular area BG3.

  Then, as shown in FIG. 10D, when the rectangular area BG4 is set in the line N + 2, as shown in FIG. 10E, a new rectangular area obtained by combining the rectangular area BG3 and the rectangular area BG4. BG5 is set.

  Note that the adjacent relationship means, for example, a case where at least a part of pixels in two rectangular areas are adjacent as shown in the rectangular area a and the rectangular area b in FIG. In such a case, the rectangular area a and the rectangular area b are combined to set a new rectangular area BG6 as shown in FIG. Further, the overlapping relationship means a case where at least a part of surfaces (regions) of two rectangular regions overlap as shown in the rectangular region BG6 and the rectangular region c in FIG. In such a case, the rectangular area BG6 and the rectangular area c are combined, and a new rectangular area BG7 is set as shown in FIG.

5. 5. Detailed description of marker detection process 5-1. Marker determination based on inclusion relationship The terminal 10 according to the present embodiment is a process in which the terminal 10 determines that there is a marker when any one of the red rectangular region BR and the green rectangular region BG has a relationship including the other. I do.

  For example, the terminal 10 determines whether or not the green rectangular area BG includes the red rectangular area BR, and determines that the first marker M1 is present when the green rectangular area BG includes the red rectangular area BR. .

  Specifically, as shown in FIG. 11A, the coordinate values (X, Y) of the vertices P1 to P4 of the green rectangular area BG1, and the red rectangular area BR1 as shown in FIG. Are compared with the coordinate values (X, Y) of the respective vertices Q1 to Q4 to determine whether or not the green rectangular region BG1 includes the red rectangular region BR1.

  For example, it is determined whether or not the vertices Q1 to Q4 of the red rectangular area BR1 are located within the vertices P1 to P4 of the green rectangular area BG1. That is, as shown in FIG. 11C, when the vertices Q1 to Q4 of the red rectangular area BR1 are located within the vertices P1 to P4 of the green rectangular area BG1, the green rectangular area BG1 is a red rectangle. Since the region BR1 is included, it is determined that there is the first marker M1.

  On the other hand, when one or more of the vertices Q1 to Q4 of the red rectangular area BR1 are located outside the vertices P1 to P4 of the green rectangular area BG1, the green rectangular area BG1 replaces the red rectangular area BR1. Judge that it is not included.

  Further, the terminal 10 determines whether or not the red rectangular area BR includes the green rectangular area BG, and determines that the second marker M2 is present when the red rectangular area BR includes the green rectangular area BG. .

  More specifically, as shown in FIG. 12A, the coordinate values (X, Y) of the vertices P5 to P8 of the green rectangular area BG2, and the red rectangular area BR2 as shown in FIG. Are compared with the coordinate values (X, Y) of the respective vertices Q5 to Q8, it is determined whether or not the red rectangular area BR2 includes the green rectangular area BG2.

  For example, it is determined whether or not the vertices P5 to P8 of the green rectangular area BG2 are located within the vertices Q5 to Q8 of the red rectangular area BG2. That is, as shown in FIG. 12C, when the vertices P5 to P8 of the green rectangular area BG2 are located within the vertices Q5 to Q8 of the red rectangular area BR2, the red rectangular area BR2 is the green rectangular area. Since BG2 is included, it is determined that the second marker M2 is present.

5-2. Marker determination based on center position In addition, in the terminal 10 of the present embodiment, the distance between the center position of the red rectangular area BR and the center position of the green rectangular area BG is within a predetermined distance in order to detect the marker with higher accuracy. A process of determining whether or not there is a marker when the distance between the center position of the red rectangular area BR and the center position of the green rectangular area BG is within a predetermined distance. Do.

  For example, as shown in FIG. 11A, the center position K1 of the green rectangular region BG1 is obtained based on the vertices P1 to P4 of the green rectangular region BG1. Then, as shown in FIG. 11B, the center position L1 of the red rectangular area BR1 is obtained based on the vertices Q1 to Q4 of the red rectangular area BR1. Then, as shown in FIG. 11C, when the distance between the center position K1 of the green rectangular area BG1 and the center position L1 of the red rectangular area BR1 is within a predetermined distance (within a predetermined number of pixels), the marker M1 is Judge that there is. In other words, when the distance between the center position L1 of the red rectangular area BR1 and the center position K1 of the green rectangular area BG1 is a predetermined distance or more, the first mark is not determined.

  Further, as shown in FIG. 12A, the center position K2 of the green rectangular region BG2 is obtained based on the vertices P5 to P8 of the green rectangular region BG2. Then, as shown in FIG. 12B, the center position L2 of the red rectangular area BR2 is obtained based on the vertices Q5 to Q8 of the red rectangular area BR2. Then, as shown in FIG. 12C, when the distance between the center position K2 of the green rectangular area BG2 and the center position L2 of the red rectangular area BR2 is within a predetermined distance (within a predetermined number of pixels), the marker M2 Judge that there is. In other words, when the distance between the center position K2 of the green rectangular area BG2 and the center position L2 of the red rectangular area BR2 is equal to or greater than a predetermined distance, the second mark is not determined.

5-3. Effect of Marker Determination of Present Embodiment As described above, the terminal 10 has a positional relationship in which one of the red rectangular area BR and the green rectangular area BG includes the other, and the red rectangular area BR and the green rectangular area BG. By determining whether or not there is a marker on the condition that each center position is within a predetermined distance (within a predetermined number of pixels), the marker can be detected with higher accuracy.

  FIG. 13 shows an example of a marker display body 20, a red pencil, and an eraser (an eraser partially wrapped with green wrapping paper) on the desk in the real world. For example, when the imaging unit 150 of the terminal 10 captures the image in the V1 direction at the position C1, the image capture in the V2 direction at the position C2, and the image capture in the V3 direction at the position C3, the markers M1 and M2 are displayed. Can be detected.

  For example, FIG. 14A is a conceptual diagram illustrating a red rectangular area BR and a green rectangular area GR on the captured image G1 when the imaging unit 150 of the terminal 10 captures an image in the V1 direction at the position C1. In FIG. 14A, one of the red rectangular area BR and the green rectangular area BG includes the other, and the center positions of the red rectangular area BR and the green rectangular area BG are within a predetermined distance ( Since the markers M1 and M2 are determined on the condition that they are within a predetermined number of pixels, the markers M1 and M2 of the marker display body can be accurately detected. That is, the red rectangular area BR of the red pencil and the green rectangular area BG extracted from the green of the eraser wrapping paper can be prevented from being detected as markers.

  FIG. 14B is a conceptual diagram showing a red rectangular area BR and a green rectangular area GR on the captured image G2 when the imaging unit 150 of the terminal 10 captures an image in the V2 direction at the position C2. Also in this case, the markers M1 and M2 of the marker display body can be accurately detected.

  FIG. 14C is a conceptual diagram showing a red rectangular area BR and a green rectangular area GR on the captured image G3 when the imaging unit 150 of the terminal 10 captures an image in the V3 direction at the position C3. Also in this case, the markers M1 and M2 of the marker display body can be accurately detected. Further, even when the imaging direction of the imaging unit 150 is such that the markers M1 and M2 appear to be crushed in the captured image G3 as a result of imaging in a direction in which the marker display body 20 is looked down obliquely, the red rectangular area BR, green The rectangular area BG can be extracted.

  Further, as shown in FIGS. 3A and 3B, the markers M1 and M2 of the marker display body 20 are composed of two circles having the same center point and different diameter lengths. Since red or green and the outer diameter circle except for the inner diameter circle are colored in green or red, the imaging direction is changed as shown in FIGS. 14 (A), (B), and (C). However, the markers M1 and M2 can be detected.

5-4. Other examples of marker determination 5-4-1. Marker determination based on vertical / horizontal ratio of rectangular area Note that the terminal 10 has a vertical / horizontal (length in the X-axis direction, length in the Y-axis direction) length between the red rectangular area BR and the green rectangular area BG. Whether or not there is a marker may be determined based on the ratio (ratio).

  For example, at least one of the ratio between the vertical length of the green rectangular area BG and the vertical length of the red rectangular area BR and the ratio between the horizontal length of the green rectangular area BG and the horizontal length of the red rectangular area BR are: When it is within the range of the first predetermined ratio (for example, 1.75: 1 to 2.25: 1), it is determined that there is the first marker M1.

  In the example of FIG. 11C, the ratio between the length of P4-P3 (the vertical length of the green rectangular region BG1) and the length of Q4-Q3 (the vertical length of the red rectangular region BR1) is 2. Since it is 1 to 1, it is determined that there is the first marker M1. Similarly, since the ratio between the length of P4-P1 (the horizontal length of the green rectangular region BG1) and the length of Q4-Q1 (the horizontal length of the red rectangular region BR1) is also 2: 1. It can be determined that there is the first marker M1.

  Further, at least one of the ratio between the vertical length of the green rectangular area BG and the vertical length of the red rectangular area BR and the ratio between the horizontal length of the green rectangular area BG and the horizontal length of the red rectangular area BR are: When it is within the range of the second predetermined ratio (for example, 1: 1.75 to 1: 2.25), it is determined that there is the second marker M2.

  In the example of FIG. 12C, the ratio between the length of P8-P7 (vertical length of the green rectangular region BG2) and the length of Q8-Q7 (vertical length of the red rectangular region BR2) is 1. Since it is a pair 2, it is determined that there is a second marker M2. Similarly, since the ratio between the length of P8-P5 (the horizontal length of the green rectangular region BG2) and the length of Q8-Q5 (the horizontal length of the red rectangular region BR2) is 1: 2. It is determined that there is the second marker M2.

5-4-2. Marker determination based on distance ratio of each vertex of rectangular area In addition, the terminal 10 determines whether the marker is based on the distance ratio in the X-axis direction and the distance ratio in the Y-axis direction of each vertex with the red rectangular area BR and the green rectangular area BG. It may be determined whether or not there is.

  For example, if the terminal 10 has a positional relationship in which the green rectangular area BG includes the red rectangular area BR, the distance between the vertices of the red rectangular area BR and the green rectangular area BG in the X axis direction and the Y axis direction And at least one of the distance ratio in the X-axis direction and the distance ratio in the Y-axis direction is within a predetermined distance ratio range (for example, 1: 1.75: 1 to 1: 2.25: 1). It is determined that there is a first marker M1.

  In the example of FIG. 11C, the ratio of the length A1 between the P3-Q3 on the X axis, the length A2 between the Q3-Q2 on the X axis, and the length A3 between the Q2-P2 on the X axis is a predetermined distance. Since it is within the range of the ratio (for example, 1: 1.75: 1 to 1: 2.25: 1), it is determined that there is the first marker M1. Further, since the ratio of the length B1 between the P4 and Q4 of the Y axis, the length B2 of the Q4 and Q3 of the Y axis, and the length B3 of the Q3 and P3 of the Y axis is within the predetermined distance ratio, the first It is determined that there is a marker M1.

  In addition, when the terminal 10 has a positional relationship in which the red rectangular area BR includes the green rectangular area BG, the distance between the vertices of the red rectangular area BR and the green rectangular area BG in the X-axis direction and the Y-axis direction. And at least one of the distance ratio in the X-axis direction and the distance ratio in the Y-axis direction is within a predetermined distance ratio range (for example, 1: 1.75: 1 to 1: 2.25: 1). It is determined that there is a second marker M2.

  In the example of FIG. 12C, the ratio of the length a1 between Q7 and P7 on the X axis, the length a2 between P7 and P6 on the X axis, and the length a3 between P6 and Q6 on the X axis is a predetermined distance. Since it is within the range of the ratio, it is determined that there is the second marker M2. Further, since the ratio of the length b1 between Q8 and P8 of the Y axis, the length b2 of P8 and P7 of the Y axis, and the length b3 of P7 and Q7 of the Y axis is within the predetermined distance ratio, the second It is determined that there is a marker M2.

5-4-3. Marker determination based on area ratio of rectangular area Further, the terminal 10 determines whether the marker is based on the ratio of the area to the red rectangular area BR and the green rectangular area BG (area ratio, area ratio), or a predetermined area ratio range. It may be determined whether or not there is.

  For example, the ratio between the area of the green rectangular region BG and the area of the red rectangular region BR is within a first predetermined area ratio (for example, 4 to 1) or a range of the first predetermined area ratio (for example, 4: 1 to 1). 5: 1), it is determined that the first marker M1 is present. On the other hand, the ratio between the area of the green rectangular region BG and the area of the red rectangular region BR is the first predetermined area ratio or the first If it is not within the range of the predetermined area ratio, it is determined that there is no first marker M1.

  Further, the ratio of the area of the green rectangular region BG and the area of the red rectangular region BR is within a second predetermined area ratio (for example, 1 to 4) or a second predetermined area ratio (for example, 1: 4 to 1: 5), it is determined that there is the second marker M2, while the ratio of the area of the green rectangular region BG to the area of the red rectangular region BR is the second predetermined area ratio or the second When it is not within the range of the predetermined area ratio, it is determined that there is no second marker M2.

5-4-4. Application Example of Combination for Marker Determination That is, if the terminal 10 has a positional relationship in which the green rectangular area BG includes the red rectangular area BR, (a) the center position of the green rectangular area and the center position of the red rectangular area (B) The ratio of the vertical and horizontal lengths of the green rectangular area BG to the vertical and horizontal lengths of the red rectangular area BR is the first predetermined ratio. (C) The distance ratio between the vertices of the X and Y axes of the green rectangular area and the red rectangular area is within a predetermined distance ratio range. (D) The area of the green rectangular area BG and the red color. When at least one condition is satisfied that the ratio of the rectangular area BR to the area of the rectangular area BR is within the range of the first predetermined area ratio or the first predetermined area ratio, it is determined that the first marker M1 is present. If all the conditions (a) to (d) are satisfied, the first marker M1 can be detected with higher accuracy.

  Further, when the terminal 10 has a positional relationship in which the red rectangular area BR includes the green rectangular area BG, (f) the distance between the center position of the green rectangular area and the center position of the red rectangular area is within a predetermined distance. (G) The ratio of the vertical / horizontal length of the green rectangular area BG to the vertical / horizontal length of the red rectangular area BR is within the range of the second predetermined ratio. ) The distance ratio between the vertices of the X and Y axes of the green rectangular area and the red rectangular area is within a predetermined distance ratio. (G) Ratio of the area of the green rectangular area BG to the area of the red rectangular area BR Is within the range of the second predetermined area ratio or the second predetermined area ratio, it is determined that there is the second marker M2 when at least one condition is satisfied. In addition, if all the conditions (f) to (g) are satisfied, the first marker M2 can be detected with higher accuracy.

6). Detailed Description of Code Recognition Processing In the present embodiment, processing for recognizing code information (processing for acquiring code information) is performed when at least one marker is detected. In particular, the terminal 10 of the present embodiment performs a process of recognizing code information when it is determined that there are a predetermined number of first markers M1 and a predetermined number of second markers M2. For example, when it is determined that there is one first marker M1 and three second markers M2, the terminal 10 performs processing for recognizing code information. In other words, if the first marker M1 is not detected, if there are a plurality of first markers M1, or if the number of detected second markers is two or less, or four or more, the code information is Do not perform recognition processing.

  First, as shown in FIG. 15A, when one first marker M1 and three second markers M2 are detected, the marker position PM1 of the first marker M1 and the second marker Identification numbers are assigned to the positions PM2-1, PM2-2, and PM2-3 of M2.

  For example, as shown in FIG. 15B, since there is only one position PM1 of the first marker M1, it can be uniquely determined. Therefore, first, the identification number “0” is assigned to the position PM1 of the first marker M1. Then, “1”, “2”, and “3” in the order of counterclockwise rotation from the position PM1 to the positions PM2-1, PM2-2, and PM2-3 of the three second markers. Give. For example, based on whether the position PM1 and the positions PM2-1, PM2-2, and PM2-3 are connected to each other and each line is in an intersecting relationship or whether each line is in a parallel relationship. “1”, “2”, and “3” are assigned to the positions PM2-1, PM2-2, and PM2-3 of the two second markers in order of counterclockwise (counterclockwise) from the position PM1.

  Thus, by assigning identification numbers to the positions PM1, PM2-1, PM2-2, and PM2-3, it is possible to specify the arrangement direction of the squares of the marker display bodies in the actual space.

  Then, as shown in FIG. 15C, the terminal 10 is based on the position PM1 of the first marker M1 and the positions PM2-1, PM2-2, and PM2-3 of the second marker M2. The areas A0 to A8 on the captured image are specified, and binarized to 1 (white) or 0 (black) based on the luminance values (or color values) of the areas A0 to A8. Get the value (0 or 1). That is, the 8-bit value of code information is obtained from each of the areas A0 to A8. The area A0 is used as a parity bit to perform error detection processing.

  Since the terminal 10 is assigned the position PM1, and the identification numbers of the positions PM2-1, PM2-2, and PM2-3, the direction of the code information can be determined, and the areas A0 to A8 can be accurately determined. Can be specified, and the 8-bit value of the code information can be acquired.

  When the code information is recognized in this manner, for example, a process of generating an augmented reality image based on the code information, a process of executing a command for presenting information, and the like can be performed.

7). Detailed description of processing for generating augmented reality image 7-1. Description of Augmented Reality Image In this embodiment, when at least one marker is detected, a process of generating an augmented reality image by superimposing a captured image and a virtual image is performed. Here, augmented reality is Augmented Reality (AR), which indicates a technology for additionally presenting information to a real environment using a computer (terminal) and an environment for which information is additionally presented.

  For example, it is assumed that a captured image is displayed on the screen of the terminal 10 by the imaging unit 150 as illustrated in FIG. Then, in this embodiment, as shown in FIG. 16B, this captured image and a virtual image including a character (an example of an object) are combined (superimposed) to generate an augmented reality image. In this way, as shown in FIG. 16C, an augmented reality image as if there is a virtual object can be provided through the imaging unit 150, although no character exists in the real world.

7-2. Processing for performing display control based on marker detection processing When it is determined that the terminal 10 of the present embodiment has a predetermined number of first markers M1 and a predetermined number of second markers M2, A process for generating an augmented reality image is performed. For example, when it is determined that there is one first marker M1 and three second markers M2, the terminal 10 performs a process of generating an augmented reality image as illustrated in FIG. In other words, when the first marker M1 is not detected, when there are a plurality of first markers M1, and when the number of detected second markers is two or less or four or more, an augmented reality image The process of displaying the captured image as shown in FIG. 16A is performed without performing the process of generating.

  In the present embodiment, since the marker detection process is performed at a predetermined period (for example, a drawing frame rate, 1/60 second period), an augmented reality image can be generated in real time.

7-3. Processing for performing display control based on code recognition processing Further, in the present embodiment, when at least one marker is detected and code information is recognized, whether or not an augmented reality image is generated based on the code information. It may be determined. For example, the terminal 10 recognizes the code information when it is determined that there is one first marker M1 and three second markers M2. When the code information is recognized, an augmented reality image is generated based on the code information and displayed on the display unit. On the other hand, when the code information cannot be recognized, a captured image is displayed.

  In the present embodiment, as shown in FIGS. 6B and 6C, a plurality of types of code information are prepared in advance. Therefore, an augmented reality image may be generated based on the code information.

  For example, when the code information based on FIG. 6B is acquired, a process of generating an augmented reality image obtained by synthesizing the virtual image including the first object and the captured image is performed. On the other hand, when the code information based on FIG. 6C is acquired, a process of generating an augmented reality image obtained by synthesizing the virtual image including the second object and the captured image is performed. In this way, various augmented reality images can be generated according to the type of code information.

7-4. Generation of Virtual Image In this embodiment, the virtual image used for the augmented reality image may be a two-dimensional image or a three-dimensional image that can be seen from a virtual camera arranged in a three-dimensional virtual space (world coordinate system). Also good.

  In the present embodiment, the arrangement position, virtual camera position, orientation, and angle of view of an object in the three-dimensional virtual space based on the positions (positions PM1, PM2-1, PM2-2, and PM2-3) of a plurality of markers. At least one of these may be determined, and a virtual image that can be seen from the virtual camera in the three-dimensional virtual space may be generated. In this way, a more natural augmented reality image can be generated.

8). Process Flow of the Present Embodiment The process flow of the present embodiment will be described using FIG. First, a process of storing the captured image captured by the imaging unit 150 in the captured image storage unit 173 is performed (step S41). Next, a process of extracting a red rectangular area and a green rectangular area on the captured image is performed (step S42). Next, a process of detecting the first mark M1 and the second mark M2 is performed based on the positional relationship between the red rectangular area and the green rectangular area (step S43).

  Next, it is determined whether one first mark M1 and three second marks M2 have been detected (step S44). If one first mark M1 and three second marks M2 are detected (Y in step S44), a code information recognition process (step S45) is performed. On the other hand, when one first mark M1 and three second marks M2 are not detected (N in step S44), a process of displaying the captured image on the display unit is performed (step S49). The process ends.

  After the code information recognition process (step S45), a process for determining an object is performed based on the code information (step S46). Then, the captured image and the virtual image including the determined object are combined to generate an augmented reality image (step S47). And the process which displays an augmented reality image on a display part is performed (step S48). The process ends here.

9. Application example 9-1. Application Example of Marker Display Body As shown in FIG. 18A, the marker display body 20 of the present embodiment may be formed by arranging a plurality of markers at the corners of a triangle. . For example, as shown in FIG. 18A, one may be the first marker M1, and the remaining two may be the second marker M2, which may be arranged at the corners of the equilateral triangle.

  As shown in FIG. 18B, the marker display body 20 of the present embodiment may be formed by arranging markers at three corners of each corner of a rectangle. For example, one may be the first marker M1, the other two may be the second marker M2, and each of the markers M1 and M2 may be arranged at the corner of a regular square.

  As shown in FIG. 18C, the marker display body 20 of the present embodiment may be formed by arranging a marker at each corner of a pentagon. For example, as shown in FIG. 18C, one may be the first marker M1, the remaining four may be the second markers M2, and the markers M1 and M2 may be arranged at the corners of a regular pentagon.

  Further, as shown in FIG. 18D, the marker display body 20 of the present embodiment may be formed by arranging a marker at each corner of the hexagon. For example, as shown in FIG. 18D, one may be the first marker M1, the remaining five may be the second markers M2, and the markers M1 and M2 may be arranged at the corners of a regular hexagon.

  Moreover, the marker display body 20 of this embodiment may arrange | position the marker M1 in one corner among each corner of a regular polygon, and may arrange | position the marker M2 in each remaining corner. It is because it becomes easy to obtain accurate information by arranging many markers.

  In the present embodiment, the number of markers to be detected may be defined in accordance with the marker display body. That is, the number of first markers M1 and the number of second markers M2, which are conditions for recognizing code information and conditions for generating an augmented reality image, are determined in accordance with the marker display.

  For example, in the case of FIGS. 18A and 18B, the terminal 10 determines that there are “one” first marker M1 and “two” second markers M2. A process for recognizing code information and a process for generating an augmented reality image are performed. In other words, when the first marker M1 is not detected, when there are a plurality of first markers M1, or when the number of detected second markers is one or three or more, the code information is recognized. And the process of generating an augmented reality image is not performed.

  Similarly, in the case of FIG. 18C, the terminal 10 determines that there is “one” first marker M1 and “four” second markers M2, and the code information Recognition processing and processing for generating an augmented reality image. In other words, if the first marker M1 is not detected, if there are a plurality of first markers M1, or if the number of detected second markers is three or less, or five or more, the code information is The process of recognizing and the process of generating an augmented reality image are not performed.

  Similarly, in the case of FIG. 18D, the terminal 10 determines that there is “one” first marker M1 and “five” second markers M2, and the code information Recognition processing and processing for generating an augmented reality image. In other words, when the first marker M1 is not detected, when there are a plurality of first markers M1, and when the number of detected second markers is four or less or six or more, the code information is The process of recognizing and the process of generating an augmented reality image are not performed.

9-2. Application Example of Augmented Reality Image In the present embodiment, as illustrated in FIG. 19, the terminal 10 recognizes code information when at least one marker is detected (for example, one first marker M1 and three If it is determined that there is the second marker M2 and the code information is recognized), additional information corresponding to the code information ("store name", "URL", "phone number", "store evaluation information") The augmented reality image may be generated by synthesizing the virtual image including the store information) and the captured image.

  When the terminal 10 recognizes the code information, the terminal 10 transmits the code information to the server via the network, and additional information (“store name”, “URL”, “phone number”) corresponding to the code information from the server. , Store information such as “store evaluation information”, and the augmented reality image may be generated by combining the captured image and the received virtual image indicating the additional information.

9-3. Application example of code information In addition to the object and the additional information, the code information is associated with a virtual moving image, a character (sentence), a voice, and the like, and the terminal 10 performs code information based on the recognized code information A virtual moving image reproduction process, a character display process, and an audio reproduction process associated with the video may be performed.

  In the present embodiment, the example in which the code information is described in the range surrounded by each marker M of the marker display body 20 has been described. However, the code information is described in a given range of the marker display body 20. May be. For example, the code information may be described outside the range surrounded by each marker M of the marker display body 20.

10. Server In this embodiment, as shown in FIG. 21, the server 20 in a client-server network system may be applied to marker detection processing. That is, the server 20 may perform a process performed by the terminal 10 (for example, at least one process of the processing unit 100).

  For example, as shown in FIG. 21, the server 20 is connected to a plurality of terminals 10 (10A to 10C) via a network (for example, the Internet), and the server 20 receives captured images (captured image data) from the terminals 10. Receive.

  For example, the server 20 performs a process of determining whether or not there is a marker based on the received captured image. More specifically, the server 20 includes a first rectangular area circumscribing the first color area composed of a plurality of adjacent pixels that satisfy the first color condition, and a plurality of adjacent areas that satisfy the second color condition. The second rectangular area circumscribing the second color area consisting of pixels is extracted from the received captured image, and the marker is determined based on the positional relationship between the first rectangular area and the second rectangular area. A process for determining whether or not there is is performed. For example, the server 20 determines that there is a marker when one of the first rectangular area and the second rectangular area has a positional relationship including the other.

  Further, the server 20 may transmit the processing result to the terminal 10. The processing result is information obtained based on the captured image. For example, the server 20 receives at least one of a determination result (whether there is a marker) based on the captured image (the presence / absence of a marker), code information, object information determined based on the code information, and an augmented reality image. The processing result may be transmitted to the terminal 10.

  For example, the server 20 performs a process of transmitting 1 to the terminal 10 as a determination result when it is determined that there is a marker and 0 when it is determined that there is no marker.

  Further, the server 20 may perform a code information recognition process for recognizing code information from the captured image based on the detected positional relationship between the markers, and perform a process of transmitting the code information to the terminal 10. Further, the server 20 may determine an object based on the code information, and may transmit information on the determined object to the terminal 10.

  Further, the server 20 may generate an augmented reality image obtained by synthesizing the captured image and the virtual image including the determined object, and perform processing for transmitting the generated augmented reality image to the terminal 10.

  Note that the terminal 10 performs processing to transmit the image captured by the imaging unit 150 and the captured image stored in the storage unit 170 (or the information storage medium 180) to the server 20, and receives the processing result from the server 20. Process. Then, the terminal 10 performs a given process based on the received processing result.

10 terminal, 20 marker display,
M marker, M1 first marker, M2 second marker,
100 processing unit, 110 rectangular area extraction unit, 111 marker determination unit,
112 code recognition processing unit, 120 drawing unit, 130 sound processing unit,
170 storage unit, 171 main storage unit, 172 image buffer, 173 captured image storage unit,
150 imaging unit, 160 input unit, 162 detection unit,
180 information storage medium, 190 display unit,
192 sound output unit, 196 communication unit

Claims (17)

A program for performing a process of detecting at least one marker from a captured image captured by an imaging unit,
A first rectangular area circumscribing a first color area formed by a plurality of pixels satisfying the first color condition and a second color area circumscribed by a plurality of pixels satisfying the second color condition A rectangular area extracting unit that performs processing for extracting the second rectangular area from the captured image;
Based on the positional relationship between the first rectangular area and the second rectangular area, the computer functions as a marker determination unit that performs a process of determining whether or not the marker is present,
The marker determination unit is
A program characterized by determining that the marker is present when one of the first rectangular area and the second rectangular area is in a positional relationship including the other. In claim 1,
The marker determination unit is
The program which performs the process which determines whether the said marker exists based on the positional relationship of the center position of a 1st rectangular area, and the center position of a 2nd rectangular area. In claim 1 or 2,
The marker determination unit is
A program for determining that the marker is present when a distance between a center position of the first rectangular area and a center position of the second rectangular area is within a predetermined distance. In any one of Claims 1-3,
The rectangular area extraction unit
Determining that the first color condition is satisfied when a color value of the pixel is within a first color value range including the first color value;
A program for determining that the second color condition is satisfied when a color value of a pixel is within a second color value range including the second color value. In claim 4,
A program characterized in that the first color value and the second color value have a complementary color relationship. In any one of Claims 1-5,
The marker determination unit is
A program for determining whether or not there are a plurality of the markers. In any one of Claims 1-6,
When the marker determination unit determines that there are a plurality of markers, the computer functions as a code recognition processing unit that performs processing for recognizing code information from the captured image based on a positional relationship between the plurality of markers. A program characterized by In claim 7,
The marker determination unit is
When the second rectangular area is in a positional relationship including the first rectangular area, a process for determining that there is a first marker is performed,
When the first rectangular area is in a positional relationship including the second rectangular area, a process for determining that there is a second marker is performed.
The code recognition processing unit
When it is determined by the marker determination unit that there is one first marker and two or more predetermined number of second markers, based on the positional relationship between the first marker and each second marker, A program for recognizing the code information. In any one of Claims 1-8,
When the marker determination unit determines that there are a plurality of markers, the computer further functions as a drawing unit that performs a process of generating an augmented reality image by combining the captured image and the virtual image. Program to do. In claim 9,
The marker determination unit is
When the second rectangular area is in a positional relationship including the first rectangular area, a process for determining that there is a first marker is performed,
When the first rectangular area is in a positional relationship including the second rectangular area, a process for determining that there is a second marker is performed.
The drawing unit
The program which performs the process which produces | generates the said augmented reality image, when it determines with the said marker determination part having one 1st marker and 2 or more predetermined number of 2nd markers. In claim 9 or 10,
When the marker determination unit determines that there are a plurality of markers, the computer functions as a code recognition processing unit that performs processing for recognizing code information from the captured image based on a positional relationship between the plurality of markers.
The drawing unit
A program that performs a process of generating a virtual image based on the code information, and generates an augmented reality image by combining the captured image and the virtual image.   A computer-readable information storage medium, wherein the program according to any one of claims 1 to 11 is stored. A terminal that performs a process of detecting at least one marker from a captured image captured by an imaging unit,
A first rectangular area that circumscribes a first color area that includes a plurality of adjacent pixels that satisfies the first color condition and a second color area that includes a plurality of adjacent pixels that satisfy the second color condition A rectangular area extracting unit that performs processing for extracting the second rectangular area to be extracted from the captured image;
Based on the positional relationship between the first rectangular area and the second rectangular area, the computer functions as a marker determination unit that performs a process of determining whether or not the marker is present,
The marker determination unit is
A terminal characterized by determining that the marker is present when one of the first rectangular area and the second rectangular area is in a positional relationship including the other. A server that performs processing for detecting at least one marker,
A communication control unit that receives a captured image from a terminal via a network;
A first rectangular area that circumscribes a first color area that includes a plurality of adjacent pixels that satisfies the first color condition and a second color area that includes a plurality of adjacent pixels that satisfy the second color condition A rectangular area extracting unit that performs processing for extracting the second rectangular area to be extracted from the captured image;
A marker determination unit that performs a process of determining whether or not there is the marker based on a positional relationship between the first rectangular region and the second rectangular region,
The marker determination unit is
When one of the first rectangular area and the second rectangular area is in a positional relationship including the other, it is determined that the marker is present,
The communication control unit
A server that transmits a processing result to the terminal. A marker display body composed of a plurality of markers detected by the terminal,
A plurality of markers that are composed of two circles having the same center point and different diameter lengths are arranged at predetermined intervals,
The inner diameter circle of one of the plurality of markers is colored in the first color, and the outer diameter of the circle except the inner diameter circle is colored in the second color,
About the remaining markers, the marker display body is characterized in that the inner diameter circle is colored in the second color, and the portion of the outer diameter circle excluding the inner diameter circle is colored in the first color. In claim 15,
A marker display body characterized in that predetermined code information is shown within a range surrounded by the markers. In claim 15 or 16,
The marker display body, wherein the first color is red and the second color is green.