JP2011204047A - Augmented reality system, marker terminal photographing terminal, augmented reality method, and information recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To appropriately change an object image combined with an image indicating the situation of a real world in an augmented reality image.SOLUTION: In the marker terminal 211, an encoding part 212 encodes information to be transmitted to the column of graphic codes, a marker generation part 213 generates a marker image for which a reference image is combined with each of the graphic codes, and a marker display part 214 displays the generated marker image. In the photographing terminal 231, a photographing part 232 photographs a real world, a marker recognition part 233 recognizes the marker image from a photographed image, and a decoding part 234 obtains the column of the graphic codes from the successively recognized marker images and decodes the transmitted information. An image generation part 235 generates an object image in the transmitted information, combines the object image with the photographed image as a background and generates the augmented reality image.

Description

  The present invention relates to an augmented reality system, marker terminal, imaging terminal, and augmented reality method suitable for appropriately changing an object image combined with an image representing the state of the real world in an augmented reality (AR) image. In addition, the present invention relates to a computer-readable information recording medium on which a program that realizes these on a computer is recorded.

  Conventionally, an augmented reality technique has been proposed in which an image of a real world is generated by synthesizing a virtual object image against a background of an image of the real world.

  In augmented reality technology, a figure of a predetermined shape called a marker is placed in the real world. The photographing device photographs the real world including this marker. Therefore, the marker is arranged in a state of being perspective-transformed in the photographed image.

  When the marker included in the photographed image is recognized, the perspective transformation parameters are also acquired.

  Then, this perspective transformation is applied to the object associated with the marker.

  Then, when the object image obtained with the captured image as a background is synthesized, an augmented reality image is generated. In the augmented reality image, the object image is drawn at a position where the marker is hidden or near the marker.

  Such augmented reality technology is disclosed in the following documents.

JP 2005-234757 A

  Here, there is a strong demand for changing the shape, pattern, color, etc. of the object associated with the marker. There is also a desire to change the shape, pattern, color, etc. of an object according to the distance between the marker and the imaging device.

  The present invention solves the above-described problems, and in an augmented reality image, an augmented reality system, a marker terminal, and an imaging suitable for appropriately changing an object image combined with an image representing the state of the real world It is an object of the present invention to provide a terminal, an augmented reality method, and a computer-readable information recording medium in which a program for realizing these is implemented by a computer.

  In order to achieve the above object, the following invention is disclosed in accordance with the principle of the present invention.

  The augmented reality system according to the first aspect of the present invention uses a graphic code including a one-dimensional code or a two-dimensional code, has a marker terminal and a photographing terminal, and is configured as follows.

  Here, the one-dimensional code encodes information by changing the interval and thickness when arranging a plurality of line segments, and corresponds to a so-called bar code.

  Also, the two-dimensional code is a color in which a plurality of barcodes are arranged in a direction perpendicular to the direction in which the lines are arranged, or a square or rectangle is divided into a plurality of cells (cells), Information is encoded by changing it to black or white (or colorless), and is sometimes called a data matrix. In Japan, a QR code (registered trademark) is widely used as a two-dimensional code.

  The marker terminal is a terminal that displays a marker image for augmented reality on the screen, and the photographing terminal captures the marker image displayed on the screen and the surrounding real world to generate an augmented reality image. Is. Both of them are typically realized by operating a predetermined program on a portable game machine or personal computer.

  Here, the marker terminal includes an encoding unit, a marker generation unit, a marker display unit, and a repetition control unit, and the imaging terminal includes an imaging unit, a marker recognition unit, a decoding unit, and an image generation unit.

  Then, when an information amount that can be expressed by one graphic code is specified, the encoding unit encodes information to be transmitted to the photographing terminal into a graphic code string that can express the specified information amount. .

  Here, the amount of information that can be expressed by one graphic code corresponds to, for example, the number of digits that can be expressed in the case of a barcode, and corresponds to the number of cells in the case of a two-dimensional code. For example, the amount of information that can be expressed by a two-dimensional code composed of 4 rows and 4 columns is 4 × 4 = 16 bits.

  On the other hand, information to be transmitted to the photographing terminal may have to be expressed by a larger number of bits (for example, 128 bits).

  In such a case, the encoding unit encodes information to be transmitted to the photographing terminal using a sequence of 128/16 = 8 two-dimensional codes.

  In practice, the graphic code typically includes a bit representing the number of the graphic code in the column, a parity bit, an error correction bit, and the like. In this case, if 128 bits of information are to be encoded with a 4 × 4 graphic code, more than 8 graphic codes are required.

  In addition, the figure code typically has a fixed number of vertical and horizontal dots regardless of the amount of information that can be expressed. For example, the width and height of each cell of the two-dimensional code of the 4 × 4 cell is 7/4 = 1.75 times the width and height of each cell of the two-dimensional code of the 7 × 7 cell. It becomes length.

  Further, the marker generation unit sequentially generates a marker image obtained by combining a code image in which each graphic code included in the encoded sequence is drawn with a predetermined size and a predetermined reference image.

  Here, the reference image is an image having a shape suitable for AR recognition. Typically, a square shape is drawn with a thick line to form a frame shape, and a figure for expressing a direction is added to the frame. What was done is used.

  At the time of composition, as will be described later, it is possible to employ a technique such as embedding a code image in the frame shape of the reference image or arranging the reference image and the code image next to each other.

  As described above, the code image has the same width and height regardless of the amount of information that can be expressed, and therefore the width and height of the marker image are also constant.

  And a marker display part displays the produced | generated several marker image on a screen in the said produced | generated order.

  As described above, since the information to be transmitted is represented by a plurality of graphic codes, a plurality of marker images are sequentially displayed on the screen.

  Further, the repetition control unit repeats the process of designating the information amount in the encoding unit and displaying the plurality of marker images in order on the marker display unit for each of a plurality of predetermined information amounts.

  For example, if four kinds of information amounts of “4 rows and 4 columns”, “5 rows and 5 columns”, “6 rows and 6 columns”, and “7 rows and 7 columns” are designated to the encoding unit, The sequence of the marker images is sequentially displayed.

  On the other hand, the photographing unit sequentially photographs the real world.

  In general, the photographing unit shoots a moving image and shoots the real world, so that the marker terminal displaying the marker image on the screen is also photographed.

  Further, the marker recognizing unit recognizes the marker image displayed on the screen of the marker terminal from each of the sequentially captured images.

  For the recognition of the marker image, it is possible to apply a normal AR recognition technique. However, as will be described later, it is also possible to adopt a method in which the reference image is first recognized and then the code image is recognized. Is possible.

  Further, when the marker images are sequentially recognized, the decoding unit acquires a code image included in the marker image, classifies the graphic image included in the code image for each information amount that can be expressed, and classifies the classified image. For each row of graphic codes, the information transmitted from the marker terminal is decoded.

  In other words, the decoding unit represents which graphic code represents “4 rows 4 columns” “5 rows 5 columns” “6 rows 6 columns” “7 rows 7 columns” “8 rows 8 columns”. Are identified by bits representing the order in the column, parity bits, and the like, and are classified according to their types to decode the original information.

  Then, the image generation unit generates an object image associated with the decoded information, and generates an augmented reality image obtained by synthesizing the object image with the captured image as a background.

  In a general AR technique, there is one object associated with one marker, and only the direction and size of the object are changed according to the direction and size of the marker photographed. Then, an object image is generated based on the transmitted information.

  Therefore, the shape, color, pattern, etc. of the object itself can be changed. For example, when a game character is adopted as an object, it is possible to change the character's posture, facial expression, clothes, and the like.

  According to the present invention, in an augmented reality image, an object image combined with an image representing the state of the real world can be appropriately changed.

  In the augmented reality system of the present invention, the photographing terminal can further include a distance estimation unit and can be configured as follows.

  Here, the distance estimation unit selects the largest information amount that can be expressed by the graphic code from among the graphic code sequences from which the information can be decoded, and the marker terminal and the photographing terminal are selected from the maximum information amount. And estimate the distance between.

  The distance estimated here is not a distance based on coordinates but a distance based on good visibility. That is, a distance that is determined to be close when the observation target looks clear and clear, and far when it is blurred or unclear is adopted.

  For example, even if a marker image of the same size is captured, the recognized graphic code is only “4 rows and 4 columns” and when it can recognize “8 rows and 8 columns”. The former will be “far” compared to the latter.

  For example, the amount of information that can be expressed by the recognized graphic code changes due to the influence of light and darkness, dust, smoke, diffuse reflection, etc. at the time of shooting, and the “distance” also changes.

  Then, in the photographing terminal, the image generation unit generates an object image based on the decoded information and the estimated distance.

  For example, when the estimated distance is “far”, it is considered that if the sharpness of the object image is changed and blurred, an image that matches the real world of the background can be obtained. In addition, the expression, posture, dialogue, etc. of the character represented by the object may be changed based on the “distance” of the estimated distance.

  According to the present invention, the object image can be appropriately changed based on the distance based on the good visibility between the marker terminal and the photographing terminal based on the information amount of the recognized graphic code.

  In the augmented reality system of the present invention, when the information can be decoded, the imaging terminal notifies the marker terminal of the fact, and the marker terminal includes a plurality of repetition control units based on the notification from the marker terminal. The frequency of designating each of the information amounts to the encoding unit can be changed.

  For example, whenever the information can be decoded, the photographing terminal notifies the marker terminal of the fact by transmitting the information by beep sound or wireless communication.

  By detecting beep sound with a microphone or receiving information by wireless communication, the marker terminal can know what amount of code image the photographing terminal can recognize.

  Then, the frequency of designation of the information amount is changed so that the information amount is transmitted as high as possible among the recognized information amount, that is, the code image having many cells.

  According to the present invention, it is possible to transmit information as fast as possible by notifying the marker terminal of whether or not recognition is possible.

  In the augmented reality system of the present invention, the reference image can be configured to be synthesized so as to surround the graphic code.

  The present invention relates to a preferred embodiment of the present invention. When the reference image has a frame shape, a marker code having a compact shape is generated by arranging a graphic code therein.

  In the augmented reality system of the present invention, the reference image can be composed so as to be adjacent to the graphic code.

  The present invention relates to a preferred embodiment of the above invention, and by arranging the reference image and the figure code in a close position, the figure code is recognized after AR recognition of only the reference figure using an existing library. Will be able to.

  Further, in the augmented reality system of the present invention, the reference image and the figure code have different colors, and the marker recognizing unit cancels the color of the figure code in each of the sequentially captured images, or the reference image By applying a filter in which the color of the image is emphasized and extracting the result of perspective transformation of the reference image from the application result, the region where the marker image is arranged in the image before the application is recognized, and the decoding unit Can be configured to restore the code image by performing the inverse transformation of the perspective transformation on the image of the recognized area, and to decode the information from the restored code image.

  For example, when an existing AR library recognizes a marker from a monochrome image, consider a mode in which a black figure is adopted as a reference image and a figure code is drawn in red.

  In this case, a filter process is performed to remove the red component from the photographed image and generate a monochrome image with the other color components. Then, it is possible to recognize the reference image by simply using the existing AR library.

  If the reference image can be recognized by the AR, the original form of the figure code can be restored by performing reverse transformation of the perspective transformation on the original photographed image. Then, the information is decrypted from the restored form.

  According to the present invention, for example, it is possible to transmit information using a graphic code while using an existing AR library.

  A marker terminal according to another aspect of the present invention is a marker terminal in the augmented reality system.

  A photographing terminal according to another aspect of the present invention is a photographing terminal in the above augmented reality system.

  The augmented reality method according to another aspect of the present invention is executed by a marker terminal and a photographing terminal using a graphic code including a one-dimensional code or a two-dimensional code.

  Here, the marker terminal includes an encoding unit, a marker generation unit, a marker display unit, and a repetition control unit, and the imaging terminal includes an imaging unit, a marker recognition unit, a decoding unit, and an image generation unit.

  On the other hand, the augmented reality method includes an encoding step, a marker generation step, a marker display step, a repetition control step, a photographing step, a marker recognition step, a decoding step, and an image generation step.

  Here, in the marker terminal, in the encoding step, when an information amount that can be expressed by one graphic code is specified, the encoding unit expresses the information to be transmitted to the photographing terminal. Encode into a sequence of possible graphic codes.

  On the other hand, in the marker generation step, the marker generation unit sequentially generates a marker image obtained by synthesizing a code image in which each graphic code included in the encoded sequence is drawn with a predetermined size and a predetermined reference image. To do.

  Further, in the marker display step, the marker display unit displays the plurality of generated marker images on the screen in the order in which they are generated.

  In the repetition control step, the repetition control unit designates the information amount for each of the plurality of predetermined information amounts with respect to the marker generation step, and the marker display step displays the plurality of marker images. Repeat the process of displaying in order.

  On the other hand, in the photographing terminal, in the photographing process, the photographing unit sequentially photographs the real world.

  Further, in the marker recognition step, the marker recognition unit recognizes a marker image displayed on the screen of the marker terminal from each of the sequentially captured images.

  In the decoding process, when the marker images are sequentially recognized, the decoding unit acquires the code image included in the marker image, and classifies the code image according to the amount of information that can be expressed by the graphic code included in the code image. The information transmitted from the marker terminal is decoded for each of the classified graphic code columns.

  On the other hand, in the image generation step, the image generation unit generates an object image associated with the decoded information, and generates an augmented reality image obtained by synthesizing the object image with the captured image as a background.

  A computer-readable information recording medium according to another aspect of the present invention causes a marker program to cause a first computer to function as the marker terminal and a second computer to function as an imaging terminal for the marker terminal. The shooting program is recorded.

  These programs can be recorded on a computer-readable information storage medium such as a compact disk, a flexible disk, a hard disk, a magneto-optical disk, a digital video disk, a magnetic tape, and a semiconductor memory.

  These programs can be distributed and sold via a computer communication network independently of the computer on which the program is executed. The information storage medium can be distributed and sold independently from the computer.

  According to the present invention, in an augmented reality image, an augmented reality system, a marker terminal, a photographing terminal, an augmented reality method, and these suitable for appropriately changing an object image combined with an image representing the state of the real world, and these It is possible to provide a computer-readable information recording medium in which a program for realizing the above is recorded.

It is a schematic diagram which shows schematic structure of a typical information processing apparatus. It is a mimetic diagram showing an outline composition of an augmented reality system concerning this embodiment. It is explanatory drawing which shows the mode of the code image showing a two-dimensional code. It is explanatory drawing which shows the mode of the code image showing a two-dimensional code. It is explanatory drawing which shows the mode of the code image showing a two-dimensional code. It is explanatory drawing which shows the mode of the code image showing a two-dimensional code. It is explanatory drawing which shows the mode of a reference | standard image. It is explanatory drawing which shows the mode of a marker image. It is explanatory drawing which shows the mode of a marker image. It is a flowchart which shows the flow of control of the marker display process performed in the information processing apparatus which implement | achieves a marker terminal. It is explanatory drawing of the image showing the mode of the real world image | photographed so that the marker terminal which is displaying the marker image on the screen of a liquid crystal display may be included. It is explanatory drawing of the image showing the mode of the real world image | photographed so that the marker terminal which is displaying the marker image on the screen of a liquid crystal display may be included. It is explanatory drawing showing the result of having applied the color filter to the picked-up image. It is explanatory drawing which shows the code image currently drawn in the original picked-up image. It is explanatory drawing showing the result of having performed reverse conversion with respect to the image | photographed code image. It is explanatory drawing which shows the mode of an augmented reality image. It is a flowchart which shows the flow of control of the imaging | photography process performed with an imaging | photography terminal. It is explanatory drawing which shows the mode of an augmented reality image.

  Embodiments of the present invention will be described below. In the following, for ease of understanding, an embodiment in which the present invention is realized using a game information processing device will be described. However, the embodiment described below is for explanation, and the present invention is described. It does not limit the range.

  Therefore, those skilled in the art can employ embodiments in which each or all of these elements are replaced with equivalent ones, and these embodiments are also included in the scope of the present invention.

(Information processing device)
FIG. 1 is a schematic diagram illustrating a schematic configuration of a typical information processing apparatus that can function as a marker terminal or a photographing terminal according to the augmented reality system of the present embodiment by executing a program. Hereinafter, a description will be given with reference to FIG.

  An information processing apparatus 101 shown in the figure is a portable multimedia terminal, and includes a central processing unit (CPU) 102, a random access memory (RAM) 103, a read only memory (ROM) 104, an input device 105, and an image processing unit. 106, a liquid crystal display 107, an audio processing unit 108, headphones 109, a cassette reader 110, a ROM cassette 111, an external memory 112, an RTC (Real Time Clock) 113, a wireless LAN (Local Area Network) interface 114, and a camera 115.

  By mounting the ROM cassette 111 in which the program according to the present embodiment is recorded on the cassette reader 110 of the information processing apparatus 101, the marker terminal and the photographing terminal according to the present embodiment are realized.

  Here, the CPU 102 controls each unit of the information processing apparatus 101 and performs various arithmetic processes and determination processes.

  When the information processing apparatus 101 is turned on, the CPU 102 executes an IPL (Initial Program Loader) recorded in the ROM 104, and the ROM cassette 111 connected via the cassette reader 110 in the course of the processing. The process is transferred to the program recorded in

  The game program and multimedia information reproduction program executed by the information processing apparatus 101 are generally provided by the ROM cassette 111, but can be prepared in the ROM 104 in advance.

  Further, a program group called BIOS (Basic Input Output System) is prepared in the ROM 104, and the input device 105, the image processing unit 106, and the audio processing unit 108 can be controlled.

  The RAM 103 is an area for storing temporary information, and the external memory 112 is an area for storing nonvolatile information. As the external memory 112, a hard disk or the like that is mainly built in the information processing apparatus 101, or various memory cards or the like that are inserted into or removed from the information processing apparatus 101 may be used.

  The input device 105 is generally realized by various buttons, a keyboard, a mouse, a joystick, or the like, but a touch screen formed integrally with the liquid crystal display 107 can also be used as the input device 105. .

  The image processing unit 106 displays various character information and image information on the screen of the liquid crystal display 107 under the control of the CPU 102. In general, the image processing unit 106 has a vertical synchronization interrupt cycle (typically 30 minutes). The pixel information stored in the frame buffer prepared in the RAM 103 is reflected on each pixel of the liquid crystal display 107 every 1 second or 1/60 second).

  Note that the display destination of the image is not necessarily limited to the liquid crystal display 107, and various display monitors such as a television apparatus connected to the image processing unit 106 and a CRT (Cathode Ray Tube) may be used. Is possible.

  In general, an instruction input from the user is performed by moving the cursor displayed on the liquid crystal display 107 by operating the move button of the input device 105, moving the cursor to a desired menu item, and selecting the menu item by operating the enter button. The cursor is not necessary when using a touch screen. Further, when the function assigned to each button of the input device 105 is determined in advance, the display on the liquid crystal display 107 is not necessarily required.

  The audio processing unit 108 outputs audio data prepared in the RAM 103, the ROM 104, the ROM cassette 111, and the external memory 112 to the headphones 109. As audio data, it is possible to use PCM (Pulse Code Modulation) data obtained by digitizing audio waveform data, MP3 (MPeg audio layer-3) data obtained by compressing PCM data to reduce the size, and the like. In addition, data such as MIDI (Music Instruction Data Interface) data that defines the pitch, tone length, volume, and timbre type is prepared, and the sound source waveform data prepared in advance is selected and transformed accordingly. It is also possible to adopt a method for reproducing the image.

  The RTC 113 measures the current date and time, and generally adjusts the time when the information processing apparatus 101 is used for the first time. The RTC 113 is connected to an NTP (Network Time Protocol) server via the wireless LAN interface 114. It is also possible to adopt a mode in which the time is automatically adjusted by connecting.

  There is also an RTC 113 having a function for generating an alarm interrupt. When the set time comes, an alarm interrupt occurs, and the CPU 102 temporarily suspends the currently executing program, executes a preset interrupt handler, and then resumes the interrupted program.

  The wireless LAN interface 114 is connected to the Internet via a wireless LAN access point prepared at home, company, or street corner, and establishes a communication path ad hoc with another information processing apparatus 101 disposed in the vicinity. It is possible to perform communication on a one-to-one basis.

  The camera 115 realizes the function of a digital camera by the information processing apparatus 101, and a technique such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) image sensor can be applied. The captured image is temporarily expanded in the RAM 103 and then stored in the external memory 112 or the like.

  In addition, a business computer, a mobile phone, a PDA (Personal Data Assistant), a portable marker terminal, an entertainment terminal, a multi-function television device, a DVD (Digital Versatile Disc) player, a portable music player, etc. It can also be employed as an information processing apparatus in which a photographing terminal is realized.

(Augmented reality system)
FIG. 2 is a schematic diagram showing a schematic configuration of the augmented reality system according to the present embodiment. Hereinafter, a description will be given with reference to FIG.

  As shown in the figure, the augmented reality system 201 includes a marker terminal 211 and a photographing terminal 231.

  Here, the marker terminal 211 is realized by the CPU 102 executing the marker terminal program read from the ROM cassette 111 in the information processing apparatus 100 to control each unit.

  Here, the photographing terminal 231 is also realized by the CPU 102 executing the photographing terminal program read from the ROM cassette 111 in the information processing apparatus 100 and controlling each unit.

  Note that the information processing apparatus 100 for realizing the marker terminal 211 and the photographing terminal 231 does not necessarily have the same configuration. For example, the marker terminal 211 may be a portable terminal that does not have a camera mounted thereon, and the photographing terminal 231 may be a portable personal computer connected to a USB (Universal Serial Bus) camera. It is possible to adopt.

  The marker terminal 211 is a terminal that displays a marker image in AR technology. In the conventional AR technology, the marker is generally printed on paper or the like and placed on a wall, the ground, a table, or the like. In this embodiment, the liquid crystal display of the information processing apparatus 100 that realizes the marker terminal 211 is used. A marker image is displayed at 107. In the present embodiment, the marker image changes with the passage of time.

  On the other hand, the imaging terminal 231 captures the situation in the real world including the marker terminal 211 with the camera 115 of the information processing apparatus 100, and synthesizes a virtual object image at the position of the marker image captured in the image. Thus, an augmented reality image is generated and can be presented to the user.

  In the augmented reality system according to the present embodiment, a one-dimensional code or a two-dimensional code can be used as a graphic code.

  Here, the one-dimensional code encodes information by changing the interval and thickness when arranging a plurality of line segments, and corresponds to a so-called bar code.

  In addition, the two-dimensional code means that a plurality of barcodes are arranged in a direction perpendicular to the direction in which the lines are arranged, or a square or rectangle is divided into a plurality of cells, and the color applied to each cell is black or white (or Information is encoded by changing it to (colorless), and is sometimes called a data matrix. In Japan, a QR code (registered trademark) is widely used as a two-dimensional code.

  Hereinafter, in order to facilitate understanding, a simple two-dimensional code will be described as an example.

(Marker terminal)
Below, after outlined about the structure of the marker terminal 211, the process performed in the marker terminal 211 is demonstrated in detail.

  As illustrated in FIG. 2, the marker terminal 211 includes an encoding unit 212, a marker generation unit 213, a marker display unit 214, and a repetition control unit 215.

  In addition, the marker terminal 211 includes a notification receiving unit 216 as an optional element. The function of the notification receiving unit 216 will be described in the second embodiment.

  First, when the repetition control unit 215 specifies “amount of information that can be expressed by one graphic code”, the encoding unit 212 generates an object image to be combined by the photographing terminal 231 accordingly. Information is encoded into a sequence of graphic codes.

  Therefore, the encoding unit 212 is realized when the CPU 102 cooperates with the RAM 103 or the like.

  In this embodiment, a game character is adopted as an object to be synthesized in augmented reality. In this case, the information for generating the object image includes, for example, the following information.

  (1) Character type parameter represented by the object. The basic shape of the character is determined by the type parameter.

  (2) A posture parameter representing the posture of the character's head, arms, legs, torso, etc. Based on this parameter, changing the position and orientation of the bone that defines the basic shape of the character changes the character's posture.

  (3) The emotion parameter of the character. Based on this parameter, moving the bone that represents the shape of the character's eyes and mouth, or changing the color of the texture pasted on the cheek surface changes the character's facial expression.

  (4) Lines from the character. As part of the object image, a message can be synthesized like a comic balloon.

  (5) Clothing parameters of the character. Based on this parameter, the surface arranged at the periphery of the character and the texture pasted on the surface change.

  The amount of information that can be expressed by one graphic code corresponds to, for example, the number of digits that can be expressed in the case of a barcode, and corresponds to the number of cells in the case of a two-dimensional code. For example, the amount of information that can be expressed by a two-dimensional code composed of 4 rows and 4 columns is 4 × 4 = 16 bits.

  In the present embodiment, as the two-dimensional code, a code that can express four types of information such as “4 rows and 4 columns”, “5 rows and 5 columns”, “6 rows and 6 columns”, and “7 rows and 7 columns” is adopted. Depending on the application and the performance of the hardware, it is possible to appropriately select other numbers of rows and columns, or to increase or decrease the number of types of two-dimensional codes.

  3A, FIG. 3B, FIG. 3C, and FIG. 3D show the state of code images representing two-dimensional codes of “4 rows and 4 columns”, “5 rows and 5 columns”, “6 rows and 6 columns”, and “7 rows and 7 columns”, respectively. It is explanatory drawing shown. Hereinafter, description will be given with reference to these drawings.

  As shown in these drawings, the code image 301 includes a plurality of cells 302, and each cell 302 is painted in a colorless (white in the drawing) or colored (black in the drawing).

  In this figure, in order to show the whole area | region of the code image 301, the dotted line is drawn on the edge, but actually this dotted line is not used. In a two-dimensional code, a line is generally not drawn at the boundary between the cells 302.

  One-bit information can be expressed depending on which color the one cell 302 is painted on. Accordingly, a 4 × 4 two-dimensional code as shown in FIG. 3A has an information amount of 4 × 4 = 16 bits, and a 5 × 5 two-dimensional code as shown in FIG. 3B has 5 × 5 = 25. An information amount of 6 bits and 6 columns as shown in FIG. 3C is 6 × 6 = 36 bits, and an information amount of 7 rows and 7 columns as shown in FIG. X7 = 49 bits of information can be expressed respectively.

  In the present embodiment, the same length is used for both the height and width of the code image 301. Accordingly, the size of one cell 302 is the largest for the 4 × 4 two-dimensional code shown in FIG. 3A and the smallest for the 7 × 7 two-dimensional code shown in FIG. 3D.

  In general, when the code image 301 is photographed by the camera 115, the image information is degraded. The degree of deterioration varies depending on the distance and relative angle between the camera 115 and the code image 301, the transparency of the air therebetween, whether dust or steam exists, and the like.

  However, the larger the cell 302 is, the less degradation is considered. Therefore, the possibility that the original information can be correctly restored from the image of the code image 301 photographed by the camera 115 is most likely in the 4 × 4 code image 301 shown in FIG. 3A, and the 7th row shown in FIG. 3D. Seven rows of code images 301 have the lowest probability.

  On the other hand, the amount of information that can be transmitted by one code image 301 is the smallest in the 4 × 4 code image 301 shown in FIG. 3A and the 7 × 7 code image 301 shown in FIG. 3D is the largest.

  Therefore, a plurality of code images 301 are used to transmit information.

  For example, consider a case where eight code images 301 of 4 rows and 4 columns are used. In this case, 3 bits are required to represent the number of each code image 301 (0th to 7th). Also, 1 bit is adopted as the parity bit of each code image 301. Then, the remaining information amount that can be embedded in one code image 301 is 4 × 4-3-1 = 12 bits. Accordingly, the entire information transmitted to the photographing terminal 231 by a column in which four 4 × 4 code images 301 are arranged is 12 × 8 = 96 bits.

In order to transmit the same amount of information, the following number and format are adopted in other two-dimensional codes.
(1) 5 sheets in the case of 5 rows and 5 columns. If 3 bits are used to represent 0th to 4th and 1 bit is used for parity, (5 × 5-3-1) × 5 = 105 bits can be expressed.
(2) 3 in the case of 6 rows and 6 columns. If 2 bits are used to represent 0th to 3rd and 1 bit is used for parity, (6 × 6-2-1) × 3 = 99 bits can be expressed.
(3) 3 in the case of 7 rows and 7 columns. If 2 bits are used to represent 0th to 3rd and 1 bit is used for parity, (7 × 7-2-1) × 3 = 138 bits can be expressed.

  In the case of 5 rows and 5 columns, 105-96 = 9 bits, in the case of 6 rows and 6 columns, 99-96 = 3 bits, and in the case of 7 rows and 7 columns, there is 138-96 = 42 bits. . Therefore, if a technique such as embedding an error correction code in the surplus bits is employed, the reliability of information transmission can be improved. When embedding the error correction code, it is possible to omit the parity bit and use the same as the bit for the error correction code.

  The marker generation unit 213 sequentially generates a marker image obtained by combining a code image in which each graphic code included in the sequence encoded by the encoding unit 212 is drawn with a predetermined size and a predetermined reference image. Generate.

  Therefore, the marker generation unit 213 is realized by the operation of the image processing unit 106 and the like under the control of the CPU 102, and the generated marker image is temporarily stored in the RAM 103.

  FIG. 4 is an explanatory diagram showing the state of the reference image. Hereinafter, a description will be given with reference to FIG.

  In the present embodiment, as the reference image 401, an image in which a reference graphic having a shape suitable for AR recognition when an existing library is used is drawn. As shown in the figure, the reference image 401 has a shape in which each side 402 of a square is drawn with black (colored) having a constant thickness, and a small square 403 is drawn with black (colored) inside three corners. However, this shape can be changed as appropriate.

  5A and 5B are explanatory diagrams showing the state of the marker image. Hereinafter, a description will be given with reference to FIG.

  A marker image 501 shown in FIG. 5A is an image in which the code image 301 is embedded in the frame of the reference image 401, that is, the reference image 401 is drawn so as to surround the code image 301.

  When adopting the marker image 501 shown in FIG. 5A, it is desirable to draw the reference image 401 and the code image 301 in different colors. As described above, when drawing with different colors, after the marker image 501 is captured, a part corresponding to the code image 301 can be easily removed or attenuated by applying an appropriate color filter to the captured result. Because.

  On the other hand, a marker image 501 shown in FIG. 5B is obtained by arranging the code image 301 and the reference image 401 so as to be adjacent to each other with a certain distance.

  In this aspect, the reference image 401 can be recognized only by using conventional AR recognition for the imaging result of the marker image 501.

  Therefore, the original code image 301 can be restored from the imaging result of the marker image 501 using the recognition result.

  Further, the marker image 501 and the code image 301 may be drawn with the same color.

  In the present embodiment, since the height and width of the code image 301 and the height and width of the reference image 401 are constant, the height and width of the marker image 501 are constant.

  Then, the marker display unit 214 displays the generated plurality of marker images 501 on the screen in the order of generation.

  Therefore, the marker display unit 214 is realized by the RAM 103, the image processing unit 106, and the liquid crystal display 107 working together under the control of the CPU 102.

  It is desirable that the time length for displaying one marker image 501 on the screen is appropriately set by obtaining an appropriate value by experiment or the like according to the recognition time in the photographing terminal 231, but the time length of several frames to several tens of frames. (That is, several times to several tens of times of the vertical synchronization period).

  Now, for each of a plurality of predetermined information amounts, the repetition control unit 215 repeats the process of designating the information amount to the encoding unit 212 and displaying the plurality of marker images on the marker display unit 214 in order. Therefore, the repetition control unit 215 is realized by the CPU 102.

  In the present embodiment, the repetition control unit 215 gives four types of information amounts of “4 rows and 4 columns”, “5 rows and 5 columns”, “6 rows and 6 columns”, and “7 rows and 7 columns” to the encoding unit 212. By specifying, the marker display unit 214 sequentially displays four types of columns of the marker image 501 respectively.

  The function of the notification receiving unit 216 will be described in an embodiment described later.

  FIG. 6 is a flowchart showing the flow of control of marker display processing executed in the information processing apparatus 101 that implements the marker terminal 211. Hereinafter, a description will be given with reference to FIG.

  When this process is started, the CPU 102 performs various initializations (step S601). This initialization includes settings such as reading out the parameters of the character object to be transmitted to the photographing terminal 231 from the external memory 112 into the RAM 103.

  Next, the CPU 101 repeats the following processing (step S602).

  First, the CPU 102 selects one of the four types of information amounts (step S603). The simplest selection method is to sequentially select four types of information in round robin. Other methods will be described in examples described later.

  The selection of the information amount in step S603 corresponds to designation of the information amount for the encoding unit 212 by the repetition control unit 215.

  Next, the CPU 101 encodes the parameters stored in the RAM 103, that is, the information to be transmitted to the photographing terminal 231 with the graphic code of the information amount selected in step S603, and obtains a graphic code string ( Step S604).

  As described above, in this embodiment, the information to be transmitted is 96 bits, and when the amount of information that can be expressed corresponds to “4 rows and 4 columns”, the sequence of 8 graphic codes is “5”. In the case of “row 5 column”, five columns are generated, and in the case of “6 row 6 column” and “7 row 7 column”, three columns are generated.

  Next, the following processing is repeated in order for each graphic code included in the graphic code column (step S605).

  First, the CPU 102 and the image processing unit 106 generate a code image 301 representing the graphic code in the RAM 103 (step S606).

  Next, the CPU 102 and the image processing unit 106 combine the generated code image 301 and the reference image 401 to generate a marker image 501 (step S607).

  Thereafter, the process waits until a vertical synchronization interrupt occurs (step S608). During this standby, other processes can be appropriately executed in a coroutine manner.

  When the vertical synchronization interrupt occurs, the generated marker image 501 is transferred from the RAM 103 and displayed on the screen of the liquid crystal display 107 (step S609).

  Thereafter, the CPU 102 waits until a predetermined number of times (N times) of vertical synchronization interrupts are generated (step S610). With this processing, if the vertical synchronization interrupt period is 1/60 second, one marker image 501 is displayed for N / 60 seconds.

  This display time may be longer than the time required for the imaging terminal 231 to recognize one marker image 501, restore the code image 301, and obtain information represented by the code image 301. desirable. It is most typical to determine N as a constant by determining the time length by experiment. Other aspects will be described later.

  Note that other processes can be executed in a coroutine during the standby in step S610, as in the standby in step S608.

  In this way, when one marker image 501 is displayed on the screen for a predetermined time (N / 60 seconds) of the liquid crystal display 107, the processes in and after step S605 are performed until all the graphic codes included in the column are processed. Repeat (step S611).

  When repetition is performed for all the graphic codes included in the obtained graphic code string, the CPU 102 updates parameters to be transmitted to the RAM 103 photographing terminal 231 if necessary (step S612), and step S602. The subsequent processing is repeated (step S613).

  By repeating such processing, the marker terminal 211 sequentially displays graphic codes of various numbers of lines and digits representing the latest parameter information on the screen.

  On the other hand, the photographing terminal 231 shoots a moving image of the real world at an angle such that the screen of the liquid crystal display 107 of the marker terminal 211 enters the field of view. Hereinafter, the configuration and processing of the photographing terminal 231 will be described in detail.

(Shooting terminal)
As illustrated in FIG. 2, the imaging terminal 231 according to the present embodiment includes an imaging unit 232, a marker recognition unit 233, a decoding unit 234, and an image generation unit 235.

  As shown in the figure, the photographing terminal 231 includes a distance estimation unit 236 and a notification report unit 237 as elements that can be omitted. Note that the functions of the distance estimation unit 236 and the notification report unit 237 will be described in an embodiment described later.

  First, the photographing unit 232 sequentially photographs the real world.

  In general, the photographing unit 232 performs moving image photographing to photograph the real world. At this time, the user adjusts the position and orientation of the photographing terminal 231 so that the marker terminal 211 enters the photographing range. Therefore, the captured image includes a marker image 501 displayed on the screen of the liquid crystal display 107 by the marker terminal 211.

  Therefore, the photographing unit 232 is realized by the camera 115 under the control of the CPU 102, and a photographed image that is a photographing result is temporarily stored in the RAM 103.

  FIG. 7A and FIG. 7B are explanatory diagrams of images representing the state of the real world photographed so as to include the marker terminal 211 displaying a certain marker image 501 on the screen of the liquid crystal display 107 by the photographing unit. Hereinafter, a description will be given with reference to FIG.

  In the photographed image 701 shown in this figure, a marker terminal 211 realized by the information processing apparatus 101 is photographed, a table 721 on which the marker terminal 211 is installed, a handrail 722, and a mountain 723 in the distance. Have been filmed.

  As shown in the figure, a marker image 501 displayed on the liquid crystal display 107 is arranged on the photographed image 701 by being perspective-projected. Note that “perspective projection” here means projection in consideration of refraction by the lens of the camera 115, and for example, a phenomenon such as an image being out of focus and blurring may occur.

  However, as in the present embodiment, in a situation where a situation in the real world is photographed in addition to the marker image 501, the “perspective projection” is regarded as a perspective projection by one-point perspective or a parallel projection, and AR image recognition is performed. It is also possible to proceed.

  In the marker image 501 shown in FIG. 7A, the code image 301 and the reference image 401 are drawn in different colors, and the marker image 501 shown in FIG. The code image 301 is surrounded by the reference image 401.

  In the marker image 501 shown in FIG. 7B, the code image 301 and the reference image 401 are drawn in the same color, and the marker image 501 shown in FIG. The code image 301 is arranged adjacent to the reference image 401.

  Now, the marker recognizing unit 233 recognizes the marker image 501 displayed on the screen of the marker terminal 211 from the image 701 thus shot.

  For recognition of the marker image, a normal AR recognition technique can be applied. However, it is also possible to adopt a method of first recognizing the reference image and then recognizing the code image.

  Therefore, the marker recognizing unit 233 is realized by the cooperation between the RAM 103 in which the image 701 is temporarily stored and the CPU 102.

  In the captured image 701 shown in FIG. 7B, the reference image 401 is displayed on the liquid crystal display 107 as a graphic independent of the code image 301. Therefore, by using various AR recognition libraries, the area where the reference image 401 is projected is recognized from the captured image 701, the original reference image 401, the reference image 401 included in the captured image 701, The perspective transformation between and the inverse can be calculated.

  As described above, by arranging the reference image 401 and the code image 301 at positions close to each other, it is possible to recognize the graphic code after performing AR recognition of only the reference graphic using the existing library.

  On the other hand, in the captured image 701 shown in FIG. 7A, the code image 301 is embedded inside the reference image 401. This is because when the reference image 401 has a frame shape, the marker image 501 having a compact shape is generated by arranging the code image 301 therein. However, there are cases where the existing AR recognition library cannot be used as it is.

For example, when the AR recognition library recognizes only a monochrome image, there is an upper limit on the number of recognizable marker image 501 patterns. Incidentally, in this embodiment, since 96-bit information is transmitted, if all the marker images 501 are simply registered as patterns, at least ²⁹⁶ patterns are registered in the AR recognition library for each number of rows and columns. This must be done, but this is virtually impossible.

  In such a case, a color filter is applied to the captured image 701.

  For example, consider a case where the code image 301 is drawn in red and the reference image 401 is drawn in black. The simplest color filter in this case is to remove the red component from the captured image 701. In other words, when the RGB component of each pixel of the photographed image 701 is (r, g, b), the color filter converts the pixel into a monochrome pixel of lightness (g + b) / 2. Such a color filter can be appropriately adjusted according to the color in which the code image 301 is drawn.

  FIG. 8 is an explanatory diagram showing the result of applying a color filter to the captured image 701 shown in FIG. 7A. Hereinafter, a description will be given with reference to FIG.

  As shown in this figure, in the result image 702 obtained by applying the color filter to the photographed image 701, the code image 301 has disappeared, and it appears as if the reference image 401 is arranged independently.

  Therefore, as in the case of AR recognition of the captured image shown in FIG. 7B, various AR recognition libraries can be used when recognizing the captured image shown in FIG. 7A.

  In general, even if a color filter is applied to the captured image 701, the code image 301 is not always completely deleted. Even if the original code image 301 is drawn in a certain color depending on the conditions under which the image is taken by the camera 115, the color of the code image 301 in the photographed image 701 is generally slightly different from this. is there.

  However, the color of the code image 301 can be reduced by applying a color filter.

  Therefore, even when an existing AR recognition library is used, the possibility that the region on which the reference image 401 is projected can be recognized from the captured image 701 by applying a color filter. .

  In this way, a perspective transformation (or an affine transformation approximating this) from the original reference image 401 to the reference image 401 included in the photographed image 701 and its inverse transformation are calculated. These transformations are expressed by a transformation matrix, and the values of the elements of the transformation matrix are temporarily stored in the RAM 103.

  When the marker image 501 is recognized as described above, the decoding unit 234 acquires the code image 301 included in the marker image 501, and the code image 301 is “4 rows × 4 columns” “5 rows × 5 columns”. "6 rows and 6 columns" and "7 rows and 7 columns" which figure code represents and what figure code it represents.

  Therefore, the CPU 102 is realized by cooperating with the RAM 103 in which various images and conversion matrices are temporarily stored.

  Specifically, first, in the original photographed image 701, an area where the code image 301 is drawn is obtained. The reference image 401 has already been AR-recognized, and orientation information is given to the reference image 401. Therefore, if the position and orientation of the reference image 401 are used as a reference, the area where the code image 301 is arranged can be easily obtained.

  FIG. 9 is an explanatory diagram showing the code image 301 drawn in the original captured image 701. In general, the image quality of the code image 301 in the captured image 701 is deteriorated due to the influence of dust, irregular reflection, hot water, and the like between the marker terminal 211 and the imaging terminal 231 and the number of pixels of the camera 115.

  Next, the inverse transformation obtained as described above is applied to the area. FIG. 10 is an explanatory diagram showing the result of performing inverse transformation on the photographed code image 301. Hereinafter, a description will be given with reference to FIG.

  As shown in the figure, a restored image 751 is obtained by performing inverse transformation. The restored image 751 is an image in which various noises are added to the original code image 301.

  Then, decoding of the obtained restored image 751 is attempted. A general two-dimensional code decoding technique can be applied to the process of decoding the original code from the two-dimensional image. That is, pattern matching is performed assuming that the restored image 751 is one of the graphic codes “4 rows and 4 columns”, “5 rows and 5 columns”, “6 rows and 6 columns”, and “7 rows and 7 columns”. The matching result is the decryption result. Depending on the shooting situation, decoding from the restored image 751 may fail.

  In this way, from the restored image 751, which graphic code represents “4 rows, 4 columns”, “5 rows, 5 columns”, “6 rows, 6 columns”, or “7 rows, 7 columns”. When it is a graphic code or what is the specific value of the bit string included, it is transmitted by arranging the bit string based on the order of the obtained graphic code for each number of rows and columns. The entire information can be decrypted.

  As described above, the bit string typically includes a parity bit and an error correction code. By using these pieces of information, it is possible to transmit information as correctly as possible.

  When the parameter information for representing the shape of the object transmitted from the marker terminal 211 can be decoded in this way, the image generation unit 235 generates an object image associated with the decoded information and captures the image. An augmented reality image is generated by synthesizing the object image with the set image as a background.

  Therefore, the CPU 102 functions as the image generation unit 235 in cooperation with the RAM 103 and the image processing unit 106. Typically, the augmented reality image generated here is displayed on the liquid crystal display 107 of the information processing apparatus 101 that implements the photographing terminal 231, but may be stored only in the external memory 112 or the like as appropriate.

  FIG. 11 is an explanatory diagram showing a state of an augmented reality image. Hereinafter, a description will be given with reference to FIG.

  As shown in the figure, the augmented reality image 801 is generated by drawing an object image 851 in accordance with the position and orientation of the marker image 501 in the captured image 701 as a background.

  In this figure, the object image 851 is combined with the photographed image 701 in FIG. 7A, but the augmented reality image 801 based on the photographed image 701 in FIG. 7B is also created by the same method.

  In the general AR technique, there is one object associated with one marker, and only the direction and size of the object are changed according to the direction and size in which the marker is photographed. Then, the object image 851 is generated based on the transmitted parameter information.

  Therefore, the shape, color, pattern, etc. of the object itself included in the augmented reality image 801 can be changed. For example, when a game character is adopted as an object, it is possible to change the character's posture, facial expression, clothes, and the like.

  FIG. 12 is a flowchart showing a flow of control of the photographing process executed by the photographing terminal 231. Hereinafter, a description will be given with reference to FIG.

  When this process is started, the CPU 102 performs various initializations (step S901). In this initialization, for example, processing such as reading information such as bones, skins, and textures constituting the character object from the ROM cassette 111 into the RAM 103 is performed.

  In addition, since the information transmitted from the marker terminal 211 is divided into the graphic codes in the graphic code string of the above four types of rows and columns, the information acquired from the graphic codes received so far Is temporarily stored in the RAM 103.

  Then, the CPU 102 repeats the following process (step S902).

  First, the CPU 102 controls the camera 115 to perform image capturing, and temporarily stores the captured image 701 as the image capturing result in the RAM 103 (step S903).

  Next, the CPU 102 applies a color filter to the captured image 701 in order to recognize the marker image 501 from the captured image 701, and obtains a result image 702 in the RAM 103 (step S904). Note that this processing is unnecessary in an aspect in which the code image 301 is not arranged inside the reference image 401.

  Then, AR recognition is tried for the obtained result, and calculation of a perspective transformation matrix for the original reference image 401 is tried (step S905).

  Here, if AR recognition is successful (step S905; success), a perspective transformation matrix from the original reference image 401 to the reference image 401 in the photographed image 701 is obtained, and an inverse matrix of the matrix is calculated. By doing so, inverse transformation is acquired (step S906).

  Then, the CPU 102 extracts the drawing area of the code image 301 in the photographed image 701 (step S907), and obtains the result in the RAM 103.

  Then, the inverse image obtained in step S905 is applied to the obtained drawing region image to obtain a restored image 751 (step S908).

Further, two-dimensional code recognition is tried for the restored image 751 (step S909). If the two-dimensional code recognition is successful (step S909; success), the following information is acquired and stored in the RAM 103.
(1) Information on how many rows and columns the graphic code by the restored image 751 is. That is, the amount of information that can be expressed by the restored image 751.
(2) Information on the order in which the restored image 751 indicates the figure code number in the transmitted figure code string.
(3) A bit string represented by the restored image 751.

  In the present embodiment, the two-dimensional code recognition is performed based on the degree of matching between each pattern from 4 rows 4 columns to 7 rows 7 columns and the restored image 751. Furthermore, it is possible to determine whether or not the recognition has failed from the relationship between the bit pattern and the parity bit in the bit string as a result of the restoration attempt.

  Next, it is checked from the information stored in the RAM 103 whether or not all the bit strings have been acquired in the order in the columns for the graphic codes having the current number of rows and columns (step S910).

  When all the bit strings have been acquired (step S910; Yes), the setting information regarding the object in the RAM 103 is updated using the bit string as parameter information transmitted from the marker terminal 211 (step S911), and the process proceeds to step S912.

  On the other hand, when the entire bit string has not yet been obtained (step S910; No) or when the two-dimensional code recognition has failed (step S909; failure), the parameter information is not updated, and the process directly proceeds to step S912.

  Then, an object image 851 is generated from the setting information related to the object in the RAM 103 and the perspective transformation matrix obtained by the AR recognition in step S904 (step S912).

  Based on the setting information of the character object, determine the direction of each bone of the character, determine the shape of the skin from the coordinates of the control points linked with the bone, paste the texture on the skin, and images such as lines as necessary Is added to generate an object image 851. At this time, the perspective transformation matrix determines the relative position of the viewpoint with respect to the object and the relative direction of the line of sight.

  Then, the CPU 102 controls the image processing unit 106 to generate an augmented reality image 801 by synthesizing the object image 851 with the captured image 701 stored in the RAM 103 as a background ( The process proceeds to step S913) and step S914.

  On the other hand, when AR recognition fails (step S905; No), the captured image 701 is handled as it is as the augmented reality image 801 (step S916), and the process proceeds to step S914.

  Thereafter, the CPU 102 waits until a vertical synchronization interrupt occurs (step S914), and then controls the image processing unit 106 to transfer the augmented reality image 801 to the liquid crystal display 107, so that the augmented reality image 801 is displayed on the monitor screen. (Step S915), and the process proceeds to step S917.

  In step S914, the CPU 102 can execute other processing in a coroutine manner.

  For this reason, when AR recognition is successful, based on the latest information transmitted from the marker terminal 211, a character object that changes its shape, color, pattern, facial expression, etc., is combined with the real world state. When the drawn augmented reality image 801 is displayed on the monitor screen and AR recognition fails, the object image 851 is not synthesized, so that the photographed image 701 representing the state of the real world photographed by the camera 115 is directly liquid crystal. It is displayed on the monitor screen of the display 107.

  By repeating the processing from step S902 (step S917), an image obtained by combining the object image 851 with the captured image 701 is displayed on the monitor screen of the liquid crystal display 107 while the marker image 501 is successfully recognized. While the image is displayed and the marker image 501 fails to be recognized, the captured image 701 is displayed as a moving image as it is.

  As described above, according to the present embodiment, in the augmented reality image, it is possible to appropriately change the object image combined with the image representing the state of the real world.

  For example, when a game character is used as an object image, it is possible to display an augmented reality character as a video by changing its facial expression and posture as time passes, etc. It becomes.

  In the above-described embodiment, information is transferred from the marker terminal 211 to the imaging terminal 231 using a plurality of types of graphic code strings having different expressible information amounts.

  In the present embodiment, the frequency of successful information transfer is measured for each of the graphic codes of the number of rows and the number of columns, and the “distance” between the marker terminal 211 and the imaging terminal 231 is estimated based on the result. Is.

  That is, the photographing terminal 231 further includes a distance estimation unit 236. Here, the distance estimation unit 236 selects the largest information amount that can be expressed by the graphic code from among the graphic code sequences from which the information can be decoded, and the marker terminal 211, The distance between the photographing terminal 231 is estimated.

  The distance estimated here is not a distance based on coordinates but a distance based on good visibility. That is, a distance that is determined to be close when the observation target looks clear and clear, and far when it is blurred or unclear is adopted.

  For example, even when a marker image of the same size is captured, when the recognized graphic code is only “4 rows and 4 columns” and when “7 rows and 7 columns” are recognized, The former will be “far” compared to the latter.

  For example, the amount of information that can be expressed by the recognized graphic code changes due to the influence of light and darkness, dust, smoke, diffuse reflection, etc. at the time of shooting, and the “distance” also changes.

  Specifically, at the time of initialization in step S901, a variable v is secured in the RAM 103, and its value is set to zero.

  In step S910 of the flowchart, if all bit strings have been acquired for the graphic code string of c rows and c columns (step S910; Yes), the value of v is set to c if v ≦ c. If c <v, the value of v is multiplied by a constant not less than 0 and less than 1 (typically a constant of about 0.8 to 0.9), or the value of v is a predetermined constant (typically It is reduced by a constant of about 0.5 to 1.).

  When such processing is performed, the variable v indicates that “the distance between the marker terminal 211 and the imaging terminal 231 is a barely visible distance if it is a graphic code of v rows and v columns”.

  In the present embodiment, it is desirable to change the object image 851 based on the “distance” obtained in this way.

  For example, the sharpness at the time of object image generation is set such that the longer the “distance”, that is, the smaller v is, the more obscured the object image 851 is, and the larger v is, the clearer the object image 851 is. You can change or apply a blur filter.

  When such processing is performed, the texture (distance) of the captured image 701 and the texture (distance) of the object image 851 can be matched, so that when the augmented reality image 801 is obtained by combining the two, A more “real” image can be obtained.

  As another method, there may be a method of newly adding an image representing “distance” to the object image 851.

  FIG. 13 is an explanatory diagram showing a state of an augmented reality image. Hereinafter, a description will be given with reference to FIG.

  In the augmented reality image 801 shown in this figure, a semi-transparent image 852 is added to the object image 851. The translucent image 852 can give the object image 851 an effect that represents a situation in which, for example, the character is hidden behind hot water in a hot spring and the character cannot be clearly seen.

  Further, in the object image 851 in this figure, the facial expression and posture of the character are also changed according to the “distance”.

  In addition, a method may be employed in which the “distance” is longer, that is, the smaller v is, the smaller the object image 851 is, and the larger v is, the larger the object image 851 is.

  In the conventional AR technique, the size of the object image 851 is determined in conjunction with the size of the marker image 501 in the captured image 701. In this method, the size of the marker image 501 is determined. In addition, the size of the object image 851 can be changed depending on the clarity of the marker image 501.

  In the above embodiment, the information indicating which graphic code string can be recognized and which graphic code string cannot be recognized is used only by the photographing terminal 231, but in the present embodiment, this information is transmitted to the marker terminal 211. Return it.

  In other words, the notification reporting unit 237 of the imaging terminal 231 determines that when all the bit strings have been acquired for the graphic code string of c rows and c columns in Step S910 (Step S910; Yes), the marker terminal 211 is notified. Notice.

  Then, the notification reception unit 216 of the marker terminal 211 receives this notification.

  For the exchange of notifications, for example, the information processing apparatus 101 connected to a speaker or buzzer (not shown) instead of the headphones 109 was used as the photographing terminal 231 and the transmitted information could be decoded. If it is found in step S910, a notification sound indicating that is emitted.

  On the other hand, as the marker terminal 211, the notification sound is monitored by the microphone by using the information processing apparatus 101 connected to the microphone (not shown).

  In the marker terminal 211, it is only necessary to monitor the input from the microphone between a step S612 and a step S613 in the flowchart described above for a certain period of time. If the above notification sound can be recognized at this stage, “the information can be transmitted to the photographing terminal 231 if the number of lines / columns of the graphic code string used until just before” can be transmitted. Recognize.

  Therefore, in the present embodiment, the probability that “the number of rows / columns authorized to transmit information” is selected in step S603 is increased. By performing such processing, the repetition control unit 215 changes the frequency of designating each of the plurality of information amounts to the encoding unit 212 based on the notification from the marker terminal 211.

  For example, the frequency can be changed by the following method. That is, in step S601, an array a is prepared in the RAM 103, and the value of each element is set to an appropriate positive constant. a [0] is the frequency of “4 rows and 4 columns”, a [1] is the frequency of “5 rows and 5 columns”, a [2] is the frequency of “6 rows and 6 columns”, and a [3] is , “7 rows and 7 columns”, respectively.

  Then, when it is found that the information transmission by the graphic code string of c rows and c columns is successful between step S612 and step S613, a positive constant (typically 0.8 to 0) is added to all elements of the array. ˜0.95.) And 1 is added to the element a [c-4]. Thereafter, a predetermined constant (typically about 0.5 to 1.0) may be added to all the elements.

  On the other hand, if it is found that the information transmission by the graphic code string of c rows and c columns has failed, a positive constant of 0 or more and less than 1 is typically set in the element a [c-4] (typically about 0.8 to 0.95). ).

  In step S603, when selecting the number of rows and columns, a [0] to a [3] are considered as probability density distributions (histograms), random numbers are generated, and “4 rows and 4 columns” “ One of “5 rows and 5 columns”, “6 rows and 6 columns”, and “7 rows and 7 columns” is selected.

  In the present embodiment, if information transmission using the graphic code string of c rows and c columns is successful, the probability that c rows and c columns will be selected subsequently increases. In addition, since the number of other rows and columns is not selected at all, even when the environment changes, it is possible to search for an information amount suitable for transmission. For this reason, graphic codes having as many rows and columns as possible are gradually selected.

  Note that the frequency of selection is corrected if the method is not limited to the above, and if the information transmission by the graphic code sequence of c rows and c columns is a method that increases the possibility of selecting c rows and c columns. It can be adopted as a technique to do.

  Instead of voice notification, for example, a mode in which the wireless LAN interface 114 is used to connect the marker terminal 211 and the photographing terminal 231 by ad hoc communication and notification is performed using wireless communication may be employed. .

  According to the present embodiment, information can be transmitted as fast as possible by notifying the marker terminal 211 from the imaging terminal 231 that recognition is possible.

  As described above, according to the present invention, in an augmented reality image, an augmented reality system suitable for appropriately changing an object image combined with an image representing the state of the real world, a marker terminal, a photographing terminal, an augmented image It is possible to provide an actual method and a computer-readable information recording medium in which a program that realizes these in a computer is recorded.

101 Information processing apparatus 102 CPU
103 RAM
104 ROM
105 Input Device 106 Image Processing Unit 107 Liquid Crystal Display 108 Audio Processing Unit 109 Headphone 110 Cassette Reader 111 ROM Cassette 112 External Memory 113 RTC
114 Wireless LAN Interface 115 Camera 201 Augmented Reality System 211 Marker Terminal 212 Encoding Unit 213 Marker Generation Unit 214 Marker Display Unit 215 Repeat Control Unit 216 Notification Accepting Unit 231 Imaging Terminal 232 Imaging Unit 233 Marker Recognition Unit 234 Decoding Unit 235 Image Generation Unit 236 distance estimation unit 237 notification report unit 301 code image 302 cell 401 reference image 402 square side 403 small square 501 marker image 701 captured image 702 result image 751 restored image 801 augmented reality image 851 object image 852 translucent image

Claims (10)

An augmented reality system having a marker terminal and a photographing terminal using a graphic code consisting of a one-dimensional code or a two-dimensional code,
(A) The marker terminal is
When an information amount that can be expressed by one graphic code is specified, an encoding unit that encodes information to be transmitted to the photographing terminal into a sequence of graphic codes that can express the specified information amount,
A marker generation unit that sequentially generates a marker image obtained by combining a code image in which each graphic code included in the encoded sequence is drawn with a predetermined size and a predetermined reference image;
A marker display unit for displaying the generated plurality of marker images on the screen in the order of generation;
For each of a plurality of predetermined information amounts, a repetition control unit that repeats the process of designating the information amount in the encoding unit and sequentially displaying a plurality of marker images on the marker display unit,
(B) The photographing terminal
Shooting unit that sequentially captures the real world,
A marker recognition unit for recognizing a marker image displayed on the screen of the marker terminal from each of the sequentially photographed images;
When the marker images are sequentially recognized, a code image included in the marker image is acquired, and classified according to the amount of information that can be expressed by the graphic code included in the code image. A decoding unit for decoding information transmitted from the marker terminal,
An augmented reality system comprising: an image generation unit that generates an object image associated with the decoded information and generates an augmented reality image obtained by synthesizing the object image with the captured image as a background. The augmented reality system according to claim 1,
The imaging terminal selects a graphic code sequence in which the information can be decoded, and selects the largest information amount that can be expressed by the graphic code. From the maximum information amount, the marker terminal and the imaging terminal are selected. And a distance estimation unit for estimating a distance between
In the imaging terminal, the image generation unit generates the object image based on the decoded information and the estimated distance. The augmented reality system according to claim 1,
When the information can be decoded, the imaging terminal notifies the marker terminal to that effect,
In the marker terminal, the repetition control unit changes the frequency of designating each of the plurality of information amounts to the encoding unit based on a notification from the marker terminal. The augmented reality system according to any one of claims 1 to 3,
The augmented reality system, wherein the reference image is synthesized so as to surround the graphic code. The augmented reality system according to any one of claims 1 to 3,
The augmented reality system, wherein the reference image is synthesized so as to be adjacent to the graphic code. An augmented reality system according to claim 4 or 5,
The reference image and the graphic code are different colors,
The marker recognizing unit applies a filter that cancels the color of the graphic code or emphasizes the color of the reference image to each of the sequentially captured images, and the reference image is perspective-transformed from the application result. By extracting the result, the area where the marker image is arranged in the image before application is recognized,
The augmented reality system, wherein the decoding unit restores a code image by performing inverse transformation of the perspective transformation on the image of the recognized area, and decodes information from the restored code image.   The marker terminal in the augmented reality system according to any one of claims 1 to 5.   The imaging terminal in the augmented reality system according to any one of claims 1 to 5. An augmented reality method executed by a marker terminal and a photographing terminal using a one-dimensional code or a graphic code composed of a two-dimensional code, wherein the marker terminal includes an encoding unit, a marker generation unit, a marker display unit, and repeat control The imaging terminal includes an imaging unit, a marker recognition unit, a decoding unit, and an image generation unit, and the augmented reality method includes:
(A) In the marker terminal,
When the information amount that can be expressed by one graphic code is designated, the encoding unit encodes information to be transmitted to the photographing terminal into a graphic code string that can express the designated information amount. Encoding process,
A marker generating step in which the marker generating unit sequentially generates a marker image obtained by combining a code image in which each graphic code included in the encoded sequence is drawn with a predetermined size and a predetermined reference image;
A marker display step in which the marker display unit displays the plurality of generated marker images on the screen in the order of generation;
A process in which the repetition control unit designates the information amount for each of a plurality of predetermined information amounts and sequentially displays a plurality of marker images in the marker display step. It has a repeat control process that repeats,
(B) In the photographing terminal,
A photographing process in which the photographing unit sequentially photographs the real world;
A marker recognition step in which the marker recognition unit recognizes a marker image displayed on the screen of the marker terminal from each of the sequentially captured images;
When the decoding unit sequentially recognizes the marker image, the decoding unit obtains a code image included in the marker image, classifies the graphic code included in the code image for each information amount that can be expressed, and classifies the classified image. A decoding step of decoding information transmitted from the marker terminal for each row of graphic codes
The image generation unit includes an image generation step of generating an object image associated with the decoded information and generating an augmented reality image obtained by synthesizing the object image with the captured image as a background. Augmented reality method. A marker program for causing the first computer to function as a marker terminal using a graphic code consisting of a one-dimensional code or a two-dimensional code, and an imaging program for causing the second computer to function as an imaging terminal for the marker terminal. A recorded computer-readable information recording medium,
(A) The marker program causes the first computer to
When an information amount that can be expressed by one graphic code is specified, an encoding unit that encodes information to be transmitted to the photographing terminal into a sequence of graphic codes that can express the specified information amount,
A marker generation unit that sequentially generates a marker image obtained by combining a code image in which each graphic code included in the encoded sequence is drawn with a predetermined size and a predetermined reference image;
A marker display unit for displaying the generated plurality of marker images on the screen in the order of generation;
For each of a plurality of predetermined information amounts, specify the information amount in the encoding unit, and function as a repetition control unit that repeats the process of displaying a plurality of marker images in order on the marker display unit,
(B) The photographing program causes the second computer to
Shooting unit that sequentially captures the real world,
A marker recognition unit for recognizing a marker image displayed on the screen of the marker terminal from each of the sequentially photographed images;
When the marker images are sequentially recognized, a code image included in the marker image is acquired, and classified according to the amount of information that can be expressed by the graphic code included in the code image. A decoding unit for decoding information transmitted from the marker terminal,
An information recording medium that generates an object image associated with the decoded information and functions as an image generation unit that generates an augmented reality image obtained by synthesizing the object image with the captured image as a background .

Images