JP2003125361A - Information processing device, information processing method, information processing program, and information processing system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information-processing system, an information-processing device, and the like capable of adding a desired effect reliably even when live video transmissions are distributed. SOLUTION: The information processing device is characterized by having: an image-taking-in means that takes in moving image information taken by a build-in camera or an external camera; an extracting means that extracts two dimensional coded information contained in the moving image information; an image processing means that processes the moving image information based on the two dimensional coded information extracted by the extracting means; a transmitting means that transmits the moving image information processed by the image processing means in a streaming format. Information for adding the effect is contained in the two dimensional coded information and this can be contained in moving picture information picked up, so that, for example, pictures along with a performer can be taken and the desired effect can automatically be synthesized without performing any troublesome operations.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an information processing apparatus, an information processing method, an information processing program, and an information processing method capable of adding various display effects to moving image information captured by an image pickup device such as a camera or a camcorder and delivered live. Regarding processing system.

[0002]

2. Description of the Related Art An information communication network such as the Internet is used to deliver images used by individuals and groups in real time. At the time of this real-time distribution, various video devices such as a structure in which a small camera is attached to a personal computer and a structure in which a video signal from an imaging device such as a camera or a camcorder can be sent out through a wireless or wired connection are currently used. Is used, and it is possible to deliver live to various viewers by sending the video signal to a server on the Internet.

When performing such live distribution, not only is the image from the camera simply transferred and distributed, but
By superimposing previously prepared effects such as images and sounds on the display image, it is possible to enrich the video expression. Generally, in order to add an effect, a desired effect is created in advance, and the photographer or editor of the distribution image operates to superimpose the effect, and the effect is distributed to the distribution image. It overlaps. The user can select the desired effect from the user interface for selecting the effect while monitoring the camera image etc. delivered in real time, and make a visual effect on the image. You can get an auditory effect by overlapping. Among these effects, effects on images include, for example, those that perform arithmetic processing on image data such as black and white, mosaic, sepia, negative / positive inversion, mirror, emboss, and comparison of flowers, hearts, crackers, glasses, etc. It is generally used to superimpose simple images.

[0004]

By adding such an effect to the image, the user can enrich the expression by the image and make the viewer of the image more entertaining. However, when an image used by an individual or a group is distributed in real time using an information communication network such as the Internet, in actuality, an effective effect is added only to a subject performing a relatively static operation. I can't. That is, in the current live video, when the software on the distribution side tries to apply some effect to the video, the effect can only be a predetermined effect in a predetermined area. For example,
Even if it is attempted to superimpose only the face of a moving person on the mosaic, such a thing is difficult because it is not known which part in the moving image is the face of the person.

Even so, when the user wants to intentionally add a mosaic effect to the person's face, the user must select the mosaic effect from among the numerous effects and specify the face position in the image, which is a troublesome editing work. You will need it. When the face of a person moves around in the camera image, it is necessary to manually set the position of the face at any time, and it is almost impossible for the mosaic to follow the face. As described above, in the current technology, the effect according to the camera image is dealt with by manually adding it.

In the conventional live video distribution,
A user who is not a performer usually manually selects and turns on the effect. That is, normally, the performer himself / herself reflected in the camera cannot select the effect or turn the effect on / off. Therefore, the operating user must do all the work of controlling the images such as effects, especially when the performer himself is the image provider of the live distribution, adding special effects such as a remote controller to add the effects. Various devices are required, and their operation is difficult.

In view of the above technical problems, the present invention is directed to an information processing apparatus, an information processing method, and an information processing method capable of reliably adding a desired effect even when live video distribution is performed. It is intended to provide a processing program and an information processing system.

[0008]

An information processing apparatus of the present invention comprises an image capturing means for capturing moving image information, an extracting means for extracting two-dimensional code information contained in the moving image information, and the extracting means. It has an image processing means for processing the moving image information according to the extracted two-dimensional code information, and a sending means for sending the moving image information processed by the image processing means in a streaming format. To do.

According to the information processing apparatus of the present invention, the information for adding the effect is included in the two-dimensional code information, and the two-dimensional code information can be included in the captured moving image information. In other words, it can be copied with the performers. The two-dimensional code information includes, for example,
Effect selection information for specifying the effect, relative position information regarding the relative position between the two-dimensional code information and the effect display position,
The effect setting information such as effect addition range information regarding the display range of the effect to be added is included, and the two-dimensional code information further includes coordinate information. Therefore, the effect addition can be controlled using the two-dimensional code information, the real-time effect control can be automatically performed, and the performer can also control the effect.

Also, the information processing method of the present invention comprises the steps of taking in moving image information, extracting two-dimensional code information contained in the moving image information, and the step of extracting the two-dimensional code information in accordance with the extracted two-dimensional code information. It is characterized by including a step of processing the moving image information and a step of transmitting the processed moving image information in a streaming format.

Also in the information processing method of the present invention, since information for adding an effect can be included in the two-dimensional code information, it is automatically included in the captured moving image information. It is possible to add various effects. The two-dimensional code information includes, for example, effect selection information for specifying an effect, relative position information regarding a relative position between the two-dimensional code information and the effect display position, and effect addition range information regarding a display range of an effect to be added. The effect setting information such as the above is included, and the coordinate information is included in the two-dimensional code information. Therefore, the effect addition can be controlled using the two-dimensional code information, the real-time effect control can be automatically performed, and the performer can also control the effect.

Further, in the information processing program of the present invention, according to the procedure of taking in the moving image information, the procedure of extracting the two-dimensional code information contained in the moving image information, and the extracted two-dimensional code information. It is characterized by comprising a procedure of processing the moving image information and a procedure of transmitting the processed moving image information in a streaming format.

According to the information processing program of the present invention, the moving image information can include the two-dimensional code information, and the information for adding the effect is included in the two-dimensional code information to automatically add the effect. Is possible. Therefore, the effect addition can be controlled using the two-dimensional code information, the real-time effect control can be automatically performed, and the performer can also control the effect.

Further, another information processing apparatus of the present invention is an image capturing means capable of transmitting the captured moving image information in a streaming format including a time stamp for specifying a frame, and the moving image information. And a sending means for sending the time stamp together with the effect setting information included in the extracted two-dimensional code information.

The time stamp is data for specifying a frame of moving image information in time series, and by including this time stamp in the moving image information transmitted in the streaming format, the moving image information is transmitted. The above-mentioned effect addition will be realized.

Still another information processing apparatus of the present invention is: an image processing unit for specifying moving image information in frame units by a time stamp and processing moving image information in frame units according to effect setting information; And a sending means for sending the moving image information processed by the processing means in a streaming format.

In this information processing apparatus, the time stamp and the effect setting information are received, and the image processing means realizes the addition of the effect in real time according to the time stamp and the effect setting information. The moving image information processed by the image processing means is transmitted in a streaming format and can be viewed on a required display device or the like.

Further, the information processing system according to the present invention includes image capturing means capable of transmitting captured moving image information in a streaming format including a time stamp for specifying a frame, and the moving image information. Extraction means for extracting the included two-dimensional code information in frame units, sending means for sending the time stamp together with the effect setting information included in the extracted two-dimensional code information, and the time stamp sent from the sending means. Specifies the moving image information on a frame-by-frame basis, and distributes the moving image information processed by the image processing unit in a streaming format, and an image processing unit that processes the moving image information on a frame-by-frame basis according to the effect setting information. And a distribution means.

According to the information processing system, the sending means can deliver the information via the information communication network such as the Internet or the communication line, and the moving image information can be specified in the frame unit by the time stamp. The image processing means can process the moving image information in frame units according to the effect setting information.

[0020]

BEST MODE FOR CARRYING OUT THE INVENTION An information processing apparatus, an information processing method, an information processing program, and an information processing system of the present invention will be described with reference to the drawings. First, the system configuration will be roughly divided into a case where the effect database is provided in the personal computer on the distribution side and a case where the effect database is provided in the effect server on the network. Note that these systems are merely examples, and various modifications of the information processing system and the like of the present invention are possible.

FIG. 1 is a diagram showing an outline of the live distribution system of the present embodiment, and is a configuration diagram when an effect database is provided in a distribution-side personal computer. The configuration of the distribution source in the system according to the present embodiment has a local computer 11 which is an ordinary personal computer, and the effect database 1 stored in the local computer 11 such as an internal or external hard disk.
Two are provided. As will be described later, when the 2D code 15 which is the two-dimensional code information is read, the corresponding item is searched in the effect database 12 and the effect to be displayed is selected. The local computer 11 may have a structure integrally provided with a camera,
A camera may be connected and used. Also,
The local computer 11 may be a device itself such as a camera or a camcorder having a network function, and can be replaced with various electronic devices such as a mobile phone and a personal digital assistant having an imaging function. The local computer 11 is, for example, a laptop computer, and a CCD (Charge Coupled D
evice) A video camera is provided. 2D code 15
Is captured by the CCD video camera of the local computer 11, the pattern of the 2D code 15 is recognized from the image data obtained as a result, and a predetermined process corresponding to the pattern is executed.

The connection destination of the local computer 11 is a streaming server 13, which is a server provided in a part of an information communication network such as a LAN, a WAN, or the Internet. In this streaming server 13,
The live video provided from the local computer 11 can be distributed in streaming format to a single viewer or multiple viewers. Specifically, the viewer can access the streaming server 13 via the terminal 14 and view the video on the display device attached to the terminal 14.

The effect database 12 provided in the local computer 11 includes effect selection information regarding effects, relative position information regarding the relative position between the two-dimensional code information and the effect display position, and effect addition regarding the display range of the effect to be added. The effect setting information such as range information is held in the database formation. The effect selection information about an effect indicates an ID (identification number or 2D code ID) of 2D data, and one ID of 2D data corresponds to one effect. For example, 2
When the ID of the D data is “0001001”, the pet type animation display expressed in three dimensions is designated, and the data stored in the predetermined folder of the memory can be read and displayed. This effect selection information not only specifies the content to be displayed, but also specifies which effect is specifically added. For example, not only the superimposition of image files, but also the replacement of the background, the calculation processing of a certain display area, and the image files to be overlaid are not limited to two-dimensional images, and animation images that change in the time axis direction,
It can be applied to various effects such as three-dimensional images and three-dimensional computer graphics in which shadows and lightness are changed in gradation to be recognized stereoscopically.

The relative position information of the effect setting information is data for designating where to set the effect display position with the position of the imaged 2D code as the origin. For example, the relative position information is a three-dimensional vector (x, It is information having data of y, z). Alternatively, a time stamp or the like may be used to include time coordinates to control the delay time, the display switching speed, and the animation progress speed. With this relative position information, even if the effect position is a part of the entire screen, the effect position can be arbitrarily specified. The relative position information is not limited to the three-dimensional vector (x, y, z) data, and may be the two-dimensional vector data.

The effect addition range information of the effect setting information is information relating to the display range of the effect to be added, and is information for specifying the range to which the effect is added based on the origin of the specified relative position. is there. For example, when the effect is a mosaic, the number of pixels in the vertical direction and the number in the horizontal direction are specified such that the mosaic is applied to ± 50 pixels in the vertical direction and ± 100 pixels in the horizontal direction.

Such an effect database 12
Can be arbitrarily set before the user of the local computer 11 starts the live distribution. For example,
It is also possible to match the effects that are frequently used or favorite effects with the 2D code that you have, or to start multiple effects simultaneously with one 2D code. Further, it is possible to control not only one type of 2D code but also a plurality of 2D code groups.

Here, examples of some effects will be described. The effects described here are only examples,
It is just a few of the many effects that the user can choose from. One of the effects available in the system of this embodiment is a mosaic. In the case of this mosaic effect, the position of the 2D code is recognized, the range to which the effect is added is read from the effect database 12, and the range is subjected to mosaic by the image processing means. As a method of using such a mosaic, it can be used when it is not desired to clearly show a part such as a face. For example, by setting relative position information and effect addition range information of the effect database 12. It is also possible to add a mosaic to the position of the face just when wearing the T-shirt. This mosaic is an effect that is generated by applying arithmetic processing to an image.In addition, arithmetic processing for image data such as black and white, sepia, negative / positive inversion, mirror, emboss, watercolor, old film, and waveform processing is also an effect. It is also possible to select as.

Further, the effects that can be used in the information processing system of this embodiment include a two-dimensional image. This recognizes the position of the 2D code on the screen and pastes a bitmap or other image file linked in advance from the effect database 12 while using the relative position information. In this case, the size of the 2D code is recognized from the coordinates of the four corners of the 2D code, and the size of the image file is converted according to the recognized size of the 2D code. Here, the image file called on the screen may be a file containing characters or symbols, or conversely may be composed of characters or symbols. In this case, a message or advertisement can be displayed corresponding to the 2D code.

Furthermore, other effects that can be used in the information processing system of this embodiment include three-dimensional computer graphics. First, the position and coordinates of the 2D code on the screen are recognized, and three-dimensional computer graphics prepared in advance according to the position and coordinates are overlaid. The coordinates indicate the direction of the 2D code. For example, by changing the direction of the 2D code on the screen, it is possible to control the corresponding 3D computer graphics so as to change the displayed direction. . In particular, in the case of 3D computer graphics, unlike the effects of image files and arithmetic processing effects such as mosaics, the 3D information of the real world image captured by the 2D code is taken into consideration. When computer graphics are shown by a character, it is possible to obtain an effect as if the character were present in the real world.

Still another effect that can be used in the information processing system of this embodiment is background replacement. With this effect, when a person or the like is recognized as a subject, the background portion is further recognized, and only the background portion is replaced with a background image prepared in advance. In the effect database 12, which image file is used to replace the background image is designated or associated in advance, and the background image can be selected by reading the 2D code. The background image can be replaced by changing the 2D code, and the background image can be prevented from being replaced by not displaying the 2D code on the camera. When replacing the background image, prepare the background image in advance, take an image in the absence of a performer as initialization processing, take the difference between the initialized image without the performer and the current image, It can be performed by recognizing a non-existing portion as a background and replacing it with a background image prepared in advance. With such a background replacement function,
The program distributor can prepare a background image so that, for example, moving images can be distributed as if they were being broadcast from the beach while in their own room.

Next, the 2D code will be described. 2D
By using the code, effect setting information such as effect selection information about the effect, relative position information about the relative position between the two-dimensional code information and the effect display position, and effect addition range information about the display range of the effect to be added can be obtained. The information can be provided to the read device, and further, the coordinate information that is the direction of the 2D code according to the positions of the four corners, and the distance information such as the distance depending on the size can be provided.

First, as shown in FIG. 2, the 2D code has a rectangular range in which one block is one unit and the length is 9.5 blocks long and the horizontal direction is 7 blocks long. Inside, a cell portion A and a logo portion B are arranged one block apart. In the cell part A, both the vertical and horizontal directions are 7
There are 2 square cells within a square of block length.
Dimensionally patterned and arranged. In the logo part B, a large rectangular cell having a length of 1.5 blocks in the vertical direction and a length of 7 blocks in the horizontal direction is arranged, for example, for the name of an object or for advertising. The mark (logo), etc. are written.

Such a 2D code can be read by application software developed on the OS 56C such as Windows (registered trademark), Mac, and Linux. The application software is a hard disk drive (HD
D) 56. As shown in FIG. 3, the hard disk drive (HDD) 56 has an electronic mail program 56A, an autopilot program 56B, an operation system software (OS) 56C, a two-dimensional barcode database (hereinafter abbreviated as a 2D code database) 56D. A finder application program (hereinafter abbreviated as FA program) 56E, a two-dimensional bar code associate application program (hereinafter abbreviated as 2D code program) 56F, and a management tool program 56G are stored. The effect database 12 described above can be constructed as a part of the 2D code database 56D.

The 2D code database 56D contains 2D
Code effect selection information (hereinafter referred to as 2D code ID)
In addition, an executable file (a program) (for example, a file having an extension such as .exe or .com) or a document file (for example, .txt, .html, .jpg) that is set in association with the 2D code The name of a file having an extension) is stored. This executable is 2
It is an execution file (hereinafter referred to as a 2D code-related execution file) set so as to be automatically activated corresponding to the D code when it is identified. Further, the document file is a document file (hereinafter, referred to as a 2D code related document file) set to be automatically opened corresponding to the 2D code when the 2D code is identified. In the following, 2
When there is no particular need to distinguish between the D-code related execution file and the 2D-code related document file, both are collectively described as a 2D-code related file. 2D
The code database 56D also stores information accompanying the 2D code, such as information describing the contents of the 2D code related file (hereinafter referred to as 2D code memo information) and the title of the 2D code related file. In this example, the 2D code database 56D is
DLL (Dynamic Link Library).

The FA program 56E is, for example, 2D from the image data stored in the VRAM of the graphic chip.
It is a program for extracting image data of a code portion and automatically executing a 2D code related file corresponding to the extracted 2D code. FA program 56E also has 2
API for developing the D code program 56F (Applic
application programming interface).

The 2D code program 56F is a 2D code related execution file capable of exchanging data with the FA program 56E via the API of the FA program 56E. The management tool program 56G is a program for registering data necessary for executing the FA program 56E and the 2D code program 56F. Management tool program 56G or specified
It has a GUI (Graphical User Interface). OS 56C in the hard disk drive 56, FA program 5
6E and the like, for example, the power switch is operated, OS56C
Is started (booted up), and the FA program 56
In the process of starting E, these programs are RA
Sequentially transferred and stored in M.

Next, the 2D code recognition processing by the FA program 56E will be described with reference to the flowchart of FIG. In this 2D code recognition processing, the FA program 56E stored in the hard disk drive 56 is stored in the RAM in response to a predetermined program starting operation by the user.
It is transferred to, stored in, and executed by the CPU.

After the user executes the program start operation, CC the surface to which the 2D code of the object is attached.
When directed to the D video camera, the image is picked up by the CCD video camera and subjected to a predetermined process in the processing unit. In step S1, the FA program 56E processes an image in which the image in the normal mode (the image of the aiming indicating the position of the 2D code) is superimposed on the image of the image data which is processed by the processing unit and drawn in the VRAM of the graphic chip. It is displayed on a display unit such as an LCD or a CRT as a finder window (FIG. 6, but this figure shows the hyper search mode).

In step S2, the FA program 56
E determines whether or not a predetermined program end operation has been performed by the user, and when determining that the end operation has not been performed, the process proceeds to step S3. In step S3, the FA program 56E causes the finder window (3
20 × 240 pixels) The entire image data (FIG. 5A) is captured (the image data is transferred from VRAM to RAM).

In step S4, the FA program 56
E is whether or not a camera operation button such as an image pickup button is pressed (whether or not the hyper search mode is set)
If it is determined that the imaging button is pressed, the process proceeds to step S5. In step S5, the FA program 56E displays an image indicating that the hyper search mode has been set in the finder window. That is, as shown in FIG. 6, the FA program 56E creates a rectangular frame (80 × 60 pixels, FIG. 5B) in the central portion of the finder window that indicates the range in which the image is enlarged to detect the 2D code. , "Hyper Search" is displayed in the lower right part. By displaying such a rectangular frame, the user can easily adjust the 2D code to an appropriate position. It should be noted that the 2D code may be set to be constantly searched for instead of the hyper search mode.

In step S6, the FA program 56
E generates an image (320 × 240 pixels, FIG. 5C) in which the image data of the central portion (80 × 60 pixels) of the finder window is magnified 4 times vertically and horizontally. In step S7, the FA program 56E is 320 × 24.
The 2D code is detected from the image of 0 pixels (the image captured in step S3 or the image enlarged in step S6).

In step S8, the FA program 56
E determines whether or not the 2D code was detected in step S7, and when it is determined that the 2D code is not detected, E displays the message to the user and displays the distance to the CCD video camera of the object to which the 2D code is attached to the user. Encourage you to adjust the angle. Then, the process returns to step S1 and the subsequent processes are repeated. Thereafter, when it is determined that the 2D code can be detected, the process proceeds to step S9.

In step S9, the FA program 56
E identifies the pattern of the 2D code and the pattern indicates
ID (2D code ID), two-dimensional coordinates (P _ix , P _iy ) indicating the positions of four vertices P _i (i = 0, 1, 2, 3) of the 2D code in the finder window and the focus of the CCD video camera Gets the distance (hereinafter, also referred to as "focal" as appropriate). However, the upper left vertex of the 2D code is P0,
Let the upper right vertex be P1, the lower left vertex be P2, and the lower right vertex be P3.

In step S10, the FA program 5
6E determines whether or not the acquired 2D code ID has already been registered (whether or not the 2D code database 56D stored in the hard disk drive 56 has the 2D code ID acquired in step S9). If it is determined that it has already been registered, the process proceeds to step S11. In step S11, the FA program 56E executes the registered process. For example, FA program 56E
Is the 2D code program 56 corresponding to the 2D code ID
A three-dimensional object display application program as F (hereinafter abbreviated as 3D application) is started.

The 3D application is an application program for displaying a three-dimensional object on the finder window, and is associated with the 2D code ID supplied as an argument to display the three-dimensional object data (used in Microsoft Direct3D). X file which is a standard file format), sound data to be played (WAVE file), XYZ coordinates for displaying a 3D object in the viewfinder window (the origin of the specified part of the 2D code in the viewfinder window) It manages information such as the coordinate system) and the speed of animation.

The processing of the 3D application will be described with reference to the flowchart of FIG. Step S2
In 1, the 3D application is FA program 5
Receive 2D code ID as an argument supplied from 6E. In step S22, the 3D application generates a 3D object display window on the finder window. In step S23, 3D
The application is the 2D supplied in step S21.
Read the 3D object data etc. corresponding to the code ID. In step S24, the 3D application determines the two-dimensional coordinates (P _ix , P _iy ) of the 2D code on the finder window supplied from the FA program 56E and CC.
D Receives the focal length of the video camera.

In step S25, the 3D application calculates the coordinates on the finder window for displaying the 3D object using the two-dimensional coordinates of the 2D code on the finder window received in step S24. Specifically, the coordinates of the four vertices of the 2D code in the real three-dimensional coordinate system are ω _i, and
The coordinates of the four vertices of the 2D code in the camera coordinate system with the lens surface of the CCD video camera facing the Z axis as the origin are C _i, and the 2D code on the finder window (hereinafter referred to as the finder window coordinate system) If the coordinates of the four vertices of P are set to P _i (i = 0, 1, 2, 3), the coordinates ω _{i in} the real three-dimensional coordinate system and the coordinates C _i in the camera coordinate system are expressed by the following equation (1). Be related. C _i = R * ω _i + T (1) Here, R is a rotation matrix representing the rotation of the camera coordinate system with respect to the real three-dimensional coordinate system, and T is the movement vector (real three-dimensional coordinate) of the CCD video camera. The distance between the origin of the system and the origin of the camera coordinate system). First, the rotation matrix R and the movement vector T are calculated.

The coordinates Pi in the finder window coordinate system and the coordinates Ci in the camera coordinate system have the relationship shown in the following equation (2).

Here, the vectors ω ₃₀ , ω ₂₀ , and ω ₁₀ are real 3
Using the coordinate ω _z of the dimensional coordinate system, it is defined as shown in the following expression (3).

Since the coordinates ω ₀ , ω ₁ , ω ₂ , ω ₃ exist on the same plane, there are s and t such that ω ₃₀ = sω ₁₀ + tω ₂₀ (4). However, in this case, since the coordinates ω ₀ , ω ₁ , ω ₂ , and ω ₃ are the coordinates of the four vertices of the rectangle, s = t = 1.

The vectors ω ₃₀ , ω ₂₀ , and ω ₁₀ are the coordinates Ci of the camera coordinate system and the vectors C ₃₀ , C ₂₀ , and C _10.
Can be represented by.

By the way, the coordinates C _i in the camera coordinate system
Can be expressed as k _i (scalar amount) times the coordinates P _i in the finder window coordinate system, as shown in the following equation (6). C _i = k _i * P _i (i = 0, 1, 2, 3) (6)

Here, by substituting the equation (5) into the equation (4), the following equation (7) is obtained. _{C 3 -C 0 = s (C} 1 -C 0) + t (C 2 -C 0) ··· (7)

Substituting equation (6) into equation (7), the following equation (8)
To get _{k 3 * P 3 -k 0 *} P 0 = s (k 1 * P 1 -k 0 * P 0) + t (k 2 * P 2 -k 0 * P 0) ··· (8)

Solving equation (8) for P3, P₀k
₀(1-s-t) / k₃+ P₁s * k₁/ K ₃+ P₂t * k₂/
k₃= P₃ ... (9).

When the equation (9) is expressed in a determinant, the following equation (1
0).

[0057]

[Equation 1] Here, if (P ₀ P ₁ P ₂ ) on the left side of the equation (10) is set to V and its inverse matrix V ⁻¹ is multiplied on both sides of the equation (10) from the left side, the following equation (11) is obtained. .

[0058]

[Equation 2] Further, V ^-1 * P ₃ on the right side of the equation (11) is (a ₀ a ₁ a
₂ ), the following equation (12) is obtained.

By the way, | ω ₃₀ | is the length of the diagonal line of the 2D code, and | ω ₃₀ | = | k ₃ * P ₃ −k ₀ * P ₀ |

Therefore, if equation (12) is applied to k ₀ on the right side of equation (13), then | ω ₃₀ | = | k ₃ * P ₃ −a ₀ * k ₃ * P ₀ / (1- _{s-t) | = k 3} | P 3 -a 0 * P 0 / (1-s-t) | get a ... (14).

│ω30│ and │P3−a0 * in equation (14)
Since the value of P0 / (1-s-t) | is known, the scalar amount k3 can be obtained. Further, by substituting the obtained scalar quantity k3 into equation (12), the scalar quantities k0, k1, k2
To get The obtained scalar quantity ki (i = 0,1,2,3)
By substituting into equation (6), the coordinates Ci (i =
0, 1, 2, 3) is obtained. Here, the rotation matrix R can be calculated as a matrix for converting the vectors ω ₃₀ and ω ₁₀ into the vectors C ₃₀ and C ₁₀ , respectively. Also, the movement vector T
Can be calculated as T = Σ (Ci−R * ωi) / 4 from the equation (1).

Using the rotation matrix R and the movement vector T thus obtained, the coordinates ω in the real three-dimensional coordinate system
i is converted to the three-dimensional coordinates of the camera coordinate system, and the coordinates Pi in the finder window coordinate system are calculated,
It progresses to step S26 of FIG. In step S26, the 3D application displays the corresponding 3D object at the coordinates of the finder window coordinate system calculated in step S25, as shown in FIG. With such a 3D application process, it is possible to display a composite image as if a 3D object were present in the camera image displayed in the viewfinder window.

The 3D object to be displayed is Mi
Since it is a standard file format (X file) used in crisoft Direct3D, the user can create it using arbitrary software. Also, 3D
Animations are also supported. Furthermore, at the same time, a sound file (WAVE file) can be specified and played back, so 3D in the finder window
It is possible to realize an expression as if the object were talking.

Returning to the explanation of FIG. When it is determined in step S10 that the acquired 2D code ID is not already registered, the process proceeds to step S12, and the FA program 56E executes a process of accepting an input of setting to associate a predetermined process with the 2D code ID. Then, step S1
Return to.

In step S2, the FA program 56E ends the 2D code recognition process when it is determined that the user has performed the end operation.

When the FA program 56E determines in step S4 that the image pickup button is not pressed, it skips steps S5 and S6. in this case,
In step S7, the 2D code is detected from the image captured in step S3.

In the above description, it is not mentioned whether the image pickup button in step S4 is pressed halfway or fully, but, for example, the image pickup button or the shutter button is pressed halfway. The hyper search mode may be set when the button is pressed, and the 2D code may be detected when the button is fully pressed.

As described above, by enlarging the central portion of the captured image and detecting the 2D code, the distance from the CCD video camera that can recognize the 2D code is about 10 cm when the image is not enlarged. It can be doubled to about 20 cm.

In this embodiment, since the process of enlarging the captured image puts a heavy load on the CPU, the enlarging process is executed only when the image pickup button is pressed. As a result, the recognition rate can be improved only when necessary (when the 2D code cannot be recognized in the normal mode).

The information processing system of the present embodiment can add the effect corresponding to not only two dimensions but also three dimensions by the program of the code recognition and the coordinate recognition of the 2D code as described above. It should be noted that these recognition programs are merely examples, and code recognition and coordinate recognition are possible by other calculation methods and procedures.

Next, the work of live distribution will be described with reference to FIGS. 9 to 16. First, in the present embodiment, a preparation stage before live distribution and an automatic effect addition operation during live distribution are realized.

First, in the preparatory stage before live distribution, various effect setting information corresponding to the 2D code 15 shown in FIG. 9 is input to the effect database 12.
The various effect setting information includes effect selection information (ID) about the effect, relative position information (position) about the relative position between the two-dimensional code information and the effect display position, and an effect addition range regarding the display range of the effect to be added. Information (range) and the like are input, and the effect database 12 stores the content (content), position, and range corresponding to the effect selection information (ID) in the format of the table 12T. The effect database 12 is rewritable, and when changing the effect addition range,
Enter the change information for that range.

FIG. 10 is a flow chart in the preparation stage before live distribution. Operate the local computer 11,
First, start the application software. At this stage, on the LCD display section of the local computer 11, a numerical value input screen of the 2D code is displayed, and the effect selection information (2D code ID) is input by operating the keyboard or the numeric keypad or directly holding the 2D code in front of the camera (step S3).
1).

When this 2D code ID is entered,
It is determined whether or not an effect is set for the input 2D code ID (step S32), and if the effect is already set (NO), it is determined whether or not overwriting is possible (step S33). ). If overwriting is not possible, the process returns to step S31 and the 2D code ID is input again.

If the effect is not set in step S32 (YES) and if overwriting is possible, the process proceeds to step 34 and the effect type is set. This effect type can be set, for example, in mosaic, two-dimensional image (bitmap image), three-dimensional computer graphics (3D
CG), background replacement, etc. are selected.

After setting the type of effect, step S3
In step 5, the corresponding data files are linked and set. When the effect is a mosaic, it is not necessary to link to the data file because only the required arithmetic processing is performed. If the effect is a bitmap image, the corresponding bitmap image file is set,
Effects are three-dimensional computer graphics (3D
In the case of CG), the corresponding three-dimensional computer graphics is selected, and when the effect is background replacement,
The selected background image file is selected.

Next, in step S36, the position where the effect is applied is set. This position means relative position information regarding the relative position between the two-dimensional code information and the effect display position, and gives the distance and direction from the origin when the 2D code is read in three-dimensional coordinates (or two-dimensional coordinates). It is a thing. With this setting, for example, when a performer appears in a T-shirt and a 2D code is printed on the T-shirt, a mosaic is created so that the face is not exactly visible from the relative position between the T-shirt and the face. You will be able to call.

After inputting this relative position information, step S37.
As shown in, the range to which the effect is added is set. The setting of the range to which this effect is added corresponds to the input of the effect addition range information of the effect setting information. For example, when adding a mosaic, the mosaic is set to ± 50 pixels in the vertical direction and ± 100 pixels in the horizontal direction. Specify the number of pixels in the vertical and horizontal directions as if to multiply. As a setting method, it is possible to specify the number of pixels as described above, but it is also possible to trim a part of the screen on the preview screen or to divide by the color of the captured video. Further, instead of the area range on the screen, a temporal range may be set in advance.

By setting such setting information before distribution, by holding the 2D code over the camera during live distribution, it is possible to apply a real-time effect corresponding to the held 2D code. Further, since the setting information constitutes a database, it can be shared by a plurality of computers, can be stored in various media, and can be a transmission / reception target or an attached file.

Next, the operation during live distribution will be described with reference to FIGS. 9 and 11. As shown in FIG. 9, when proceeding with live distribution, the local computer 11
Is connected to the streaming server 13 via the Internet. When the connection to the streaming server 13 is made, the camera attached or connected to the local computer 11 is operated to capture the camera image 18A. In this camera image 18A, the 2D code 15 is displayed on the screen by the performer or photographer when an effect is to be added.

When the 2D code 15 is reflected in the range of the camera image 18A in this way, the two-dimensional code information is extracted by the finder application program that has been activated as described above. When the two-dimensional code information is extracted from the image 18B, the position, coordinates (direction or posture), size, etc. of the 2D code 15 are recognized from the patterns at the four corners of the 2D code 15. This pattern recognition work is carried out by the operation of the CPU as the work of the extraction means, and the data table 12T of the effect database 12 set in the above-mentioned preparation stage.
The processing according to is performed. The distribution screen 18C shows a screen processed by the image processing means. In the example shown in FIG. 9, a case where a pet type animated character called “3D peach” is selected and displayed at a predetermined position is shown. Shows. The image information on the distribution screen 18C is transmitted to the streaming server 13 in a streaming format, and the streaming server 13 performs live distribution to a single viewer or a plurality of viewers.

FIG. 11 is a flow chart of effect addition work in the local computer 11 during live distribution.
It shows processing in each frame of image data. First, when the software is started, the 2D code imaged on the camera is searched (step S41). As a result of this search, it is determined whether or not the 2D code can be recognized (step S42). If the 2D code is not recognized, the procedure returns to step 41 and the same is performed for the next frame. Take action.

When the 2D code can be recognized in step S42, the process proceeds to step S43, and the effect information corresponding to the recognized 2D code is retrieved from the effect database 12 in the local computer 11. That is, by recognizing the 2D code, the 2D code ID, which is the effect selection information, is read, and it is determined in step S44 whether or not the effect information exists. When the effect information does not exist for the read 2D code, in other words, the ID of the undefined effect
If is recognized, the procedure returns to step 41 and the same operation is performed for the next frame.

If the effect information exists in step S44 (YES), the type, position, and range of the effect for the 2D code ID are read as set, and the procedure S
At 45, a data file corresponding to the effect type is read. When the effect is a bitmap image, the corresponding bitmap image file is loaded, when the effect is 3D computer graphics (3DCG), the corresponding 3D computer graphics is loaded, and when the effect is background replacement A background image file to be selected is read in. When the effect type is a calculation process such as mosaic, the data file to be read is part or the whole of the screen, but it is also possible to skip to step S49 without reading the data here.

According to the read operation of the data file, it is determined in step S46 whether or not the corresponding data file exists. If the data file does not exist (NO), the procedure returns to step 41 and the operation is performed again for the next frame. On the contrary, if the data file exists (YES), the position (2D coordinate) of the recognized 2D code is acquired (step S47). The position of the 2D code recognized here is the procedure S in the flow of FIG.
Similar to the operation of 24, it is the two-dimensional coordinates (P _ix , P _iy ) of the 2D code on the finder window supplied from the FA program 56E.

Next, in the case of three-dimensional computer graphics (3DCG) which requires a three-dimensional display effect, the three-dimensional coordinates are read out in step S48. This operation corresponds to the step S25 in the flow of FIG. 7 described above, and specifically, using the rotation matrix R and the movement vector T, the
The coordinates ωi in the dimensional coordinate system are converted into the three-dimensional coordinates in the camera coordinate system, and the coordinates Pi in the finder window coordinate system are calculated to obtain the necessary three-dimensional coordinates.

Subsequently, in step S49, the effect reflecting the effect information is drawn based on the coordinates based on the 2D code. The drawing of the effect reflecting the effect information is performed by the image processing means, and is advanced by rewriting the display data in the VRAM to the data including the effect under the control of the CPU in the local computer 11. An image with this effect is distributed via the Internet (step S50). Such image processing is advanced for each frame and is repeatedly performed for each frame.

Here, a concrete example of the effect is shown in FIG.
2 to 16, a brief description will be given with reference to FIG.
FIG. 4 is a diagram showing a camera input image, which is an image captured by a camera connected to the local computer 11. Figure 1
In 2, the performer is wearing a T-shirt, and the 2D code is printed from the chest to the abdomen on the front of the T-shirt. In this state, the camera copies the 2D code together with the performers, and the effect addition is realized by the effect addition program.

FIG. 13 is a diagram in which a mosaic is added as an effect. In the information processing apparatus according to the present embodiment, the two-dimensional code information is read while referring to the effect database 12 and the effect addition processing of the image is performed, thereby creating an image in which the mosaic of FIG. 13 is applied to the performer's face. It From the state of FIG. 12 to the state of FIG. 13, the program automatically advances, and the relative position information regarding the relative position between the two-dimensional code information and the effect display position is set in advance. As described above, for example, a performer can appear by wearing a T-shirt on which a 2D code is printed, and a mosaic can be applied so that the face can be easily identified from the relative position data of the T-shirt and the face. In this state of FIG. 13, even when the performer moves within the screen, the T-shirt on which the 2D code is printed also moves, and therefore the display position provided by the 2D code also moves. Therefore, the position of the mosaic is also automatically moved, and it is possible to display the effect following the movement of the performer without the camera operator changing the effect range each time.

FIG. 14 shows an example of pasting an image file, and in this FIG. 14 as well, the image of the cat-shaped robot is displayed so as to be overwritten on the face of the performer.
The image file is the data of the image file prepared in advance before the live distribution. For example, as shown in FIG.
It is possible to easily superimpose the image of the cat-shaped robot on the face based on the relative position data of the T-shirt and the face simply by wearing the T-shirt on which the 2D code is printed. Further, the distance of the 2D code is grasped from the 2D code based on the coordinate data of the four corners of the 2D code, and the size of the image of the cat robot is enlarged or reduced to an appropriate size and displayed from the distance.

FIG. 15 is a diagram showing a case of three-dimensional computer graphics (3DCG). In FIG. 15, the T-shirt part of the performer is displayed so as to be overwritten with an image of an animation-like 3D frog. This anime-like 3D
The frog image is created by computer graphics and is drawn from data. When performing live distribution, performers move,
When facing vertically or horizontally, the position of the T-shirt on which the 2D code is printed and the direction of its normal vector will change, and the movement of 3D computer graphics faithful to the movement of the performer will be expressed. Will be done. Further, not only images but also audio files can be made to follow, and performers can freely control within a set range.

FIG. 16 is a diagram showing replacement of the background. In FIG. 16, the background of the performer is displayed so as to be overwritten on the beach landscape. This beach landscape is an image drawn by extracting the effect selection information of the 2D code printed on the T-shirt worn by the performer. The performer is recognized in advance by being separated from the background portion, and the image of FIG. 16 is created by replacing the background image with the background image that has been facilitated in advance. It is possible to restore the background image to the original image by hiding the 2D code of the T-shirt worn by the performer, and the performer can wear another kimono with a different 2D code printed to change the background. Switching to an image can also be automatically performed.

Next, referring to FIGS. 17 to 19,
As a second embodiment, an information processing system when the effect database is in an effect server on the network will be described. FIG. 17 is a diagram showing an outline of the information processing system of this embodiment. In the system of the present embodiment, first, as a video distribution side configuration, a computer 21 with a camera, an effect server 26 having an effect database 22, a streaming server 23 for Internet live video distribution, and an effect used by a user It has a billing server 27 for managing the number of times of use and time and billing. Also, as a configuration on the side of receiving the video, there is a terminal 24 for connecting to the streaming server 23 via the Internet,
The viewer can view the delivered video through the display unit of the terminal 24.

The computer 21 may have a structure in which a camera is integrally provided, or may be one in which a camera is connected and used. The computer 21 may be a device itself such as a camera or a camcorder having a network function, and may be replaced with various electronic devices such as a mobile phone and a personal digital assistant (PDA) having an image capturing function. The computer 21 is, for example, a laptop computer, and a CCD is provided at the center of the upper end of the display unit.
(Charge Coupled Device) A video camera is provided. The 2D code 25 is imaged by the CCD video camera of the computer 21, the pattern of the 2D code 25 is recognized from the image data obtained as a result, and a predetermined process corresponding to the pattern is executed.

From this computer 21, the 2D code information is sent to the effect server 26 on the Internet, and the video signal itself is sent to the streaming server 23 on the Internet. Effect server 26
Further, user information, which is information for identifying the user, is also sent to the user. The effect server 26 is a server including the effect database 22, and stores, for example, characters, computer graphics, photographs, etc. produced by a required creator together with the assigned 2D code ID,
It is configured to provide in the form of an electronic file when accessed.

The effect database 22 provided in the effect server 26 includes effect selection information regarding effects, relative position information regarding the relative position between the two-dimensional code information and the effect display position, and effect addition regarding the display range of the effect to be added. The effect setting information such as range information is held in the database formation. The effect selection information about an effect indicates a 2D code ID (identification number or 2D code ID), and one ID of the 2D code corresponds to one effect. For example, when the ID of the 2D code is “0001001”, the pet type animation display expressed in three dimensions is designated, and the data stored in the predetermined folder of the server can be read and displayed. . This effect selection information not only specifies the content to be displayed, but also specifies which effect is specifically added. For example, not only the superimposition of image files, but also the background replacement, the calculation processing of a certain display area, and the image files to be overlaid are not limited to two-dimensional images. Various effects such as three-dimensional images and three-dimensional computer graphics, in which the brightness changes with gradation and are three-dimensionally recognized, can be supported.

The relative position information of the effect setting information is data for designating where to set the effect display position with the position of the imaged 2D code as the origin. For example, the relative position information is a three-dimensional vector (x, It is information having data of y, z). Alternatively, a time stamp or the like may be used to include time coordinates to control the delay time, the display switching speed, and the animation progress speed. With this relative position information, even if the effect position is a part of the entire screen, the effect position can be arbitrarily specified. The relative position information is not limited to the three-dimensional vector (x, y, z) data, and may be the two-dimensional vector data.

The effect addition range information of the effect setting information is information regarding the display range of the effect to be added, and is information for specifying the range to which the effect is added based on the origin of the specified relative position. is there. For example, when the effect is a mosaic, the number of pixels in the vertical direction and the number in the horizontal direction are specified such that the mosaic is applied to ± 50 pixels in the vertical direction and ± 100 pixels in the horizontal direction.

Such an effect database 22
Can be arbitrarily set before the user of the computer 21 starts the live distribution, and it is possible for the user to upload the one created by the user to the effect server 26. By arranging the database 22 on the Internet and uploading various contents such as characters, computer graphics, and photographs created by the creator together with the assigned 2D code ID, the user can use various effects. Therefore, the creator can be given a place to show works, and the next charging server 27 can also collect charges according to the frequency of use.

The streaming server 23 can deliver the live video provided from the computer 21 to one or more viewers in a streaming format. Specifically, the viewer can access the streaming server 23 via the terminal 24 and watch the video on the display device attached to the terminal 24. When performing live distribution from the streaming server 23,
Data about an effect is provided from an effect server 26 having an effect database 22.
At this time, the effect server 26 and the billing server 27 can work together to record which user, to what extent, which effect was used, and feed this back to the creator as billing information.

The operation of this system will be described with reference to FIG. In this system as well, the preparation stage before live distribution and the automatic effect addition operation at the time of live distribution are realized. However, in the preparation stage before live distribution, one of the procedures performed by the user is different from the previous embodiment. The department is designed to be done by the creator.

First, before the live distribution, the effect database 22 in which the 2D code ID and the effect information are associated with the effect server 26 on the network.
Exists. Fun effects created by creators
By preparing the content on the effect database 22, the user does not have to create the effect himself / herself, and the high-quality effect created by the creator can easily appear in the live video.

The effect created by the creator is almost the same as the effect in the first embodiment. That is, the effects that can be used in the system of the present embodiment include effects that perform arithmetic processing such as mosaic, two-dimensional images such as animations and characters, and three-dimensional (two
Dimension) Computer graphics, background replacement images, etc. are possible. It is also possible to combine audio effects and the like. Effect database 22
The uploading of the effect to is performed by the creator himself sending electronic data from his terminal 28 via the Internet. At this time, the effect database 22 stores the 2D code ID as effect selection information.
In addition, various effect setting information such as relative position information regarding the relative position between the 2D code ID and the effect display position and effect addition range information regarding the display range of the effect to be added is stored. Further, in this embodiment, the creator of each content is also recorded, and the author information is shared with the billing server 27.

Next, at the time of live image distribution, after the application is started, the effect server 2 having the effect database 22 from the distribution-side computer 21.
6 and the streaming server 23 for live distribution, and also via the effect server 26 to the billing server 27 that holds billing information for the creator. User identification information is transmitted to the billing server 27.

During live video distribution, the user or performer holds the 2D code over the camera. Then, the 2D code recognition software in the distribution side computer 21 becomes 2D.
Recognize the code ID, position (2D coordinates) and size,
Calculate (3D coordinates). The distribution-side computer 21 has two
The D code ID and the camera image are transmitted to the effect server 26. The effect server 26 matches the received 2D code ID with the 2D code ID in the effect database 22 held in the server 26, and the matching I
When there is D, the corresponding effect information is acquired.

When the effect information is found, the effect server 26 reads out the kind of effect, and performs background replacement, mosaic, bitmap image, 3DCG effect and the like. Background image, bitmap image, 3
In the DCG mode, the data file is read from the database. Mosaic, bitmap image, 3DCG
In mode, read the position where the effect is applied.
In the mosaic mode, the range to which the effect is applied is read out. The distribution-side computer 21 subsequently acquires the position (2D coordinate), size, and attitude (3D coordinate) in the image of the 2D code, and transmits the position to the effect server 26. These calculation methods are the same as the methods of the previous embodiment.

The effect server 26 receives the position information, and superimposes the effect on the camera image by using the effect setting information based on this position. At the time of overlapping, an effect is added so that the time stamp at the time of recognizing the 2D code ID received from the distribution side computer 21 and the time stamp of the video correspond. The superimposed images are transmitted to the streaming server 23. The streaming server 23 delivers the video live as in the conventional personal delivery system.

By repeating these operations for each frame of the camera image, even if the ID or position of the 2D code changes, it is possible to live-distribute the image with the corresponding effect. In addition, the effect database 22 is stored in the effect server 2 on the network.
By setting the number to 6, the user can not only save the effort of creating the effect by himself but also can easily apply the high quality effect created by the creator by holding the 2D code over the camera.

Next, a billing system when such a creator creates an effect and is used by the user will be described. When the software is activated, the distribution-side computer 21 connects to the effect server 26 and transmits the user information. When the 2D code is recognized, the ID is transmitted to the effect server 26, but the effect server 26
If the effect corresponding to the 2D code exists in the effect database 22, the charging server 27
The user information and the 2D code ID to notify that the effect is used by the user.

The billing server 27 identifies the effect used by the 2D code ID and collectively manages the number of times of use and time. Further, based on the user information sent from the effect server 26, the effect use fee is charged according to the number of times and the time the effect is used by the user. Since the effect is freely created by the creator, the usage fee is returned to the creator according to the number of times and time the effect is used. In this embodiment, the correspondence between the 2D code ID and the effect is stored in the effect database 22 of the effect server 26,
By managing the number of times / time of use by the charging server 27, it is possible to charge the user according to the number of times / time of use of the effect and return to the creator, and a profit is generated by the difference. .

FIG. 19 is a more specific flow chart showing the operation of the system when live distribution is performed on the Internet. First, the user activates the distribution-side computer 21 to activate the software. Next, in step S61, the user's computer 21 is connected to the streaming server 23. In step S62, it is determined whether or not the connection between the computer 21 and the streaming server 23 has succeeded, and if unsuccessful, an error dialog is displayed on the display unit of the user's computer 21 and processed in step S63. To finish.

If it is determined in step S62 that the connection between the computer 21 and the streaming server 23 has succeeded (YES), the process proceeds to step S64. Here, the user's computer 21 is connected to an effect server 26 having an effect database 22. Next step S6
In 5, the computer 21 and the effect server 26
It is determined whether or not the connection has been successful, and if unsuccessful, an error dialog is displayed on the display unit of the user's computer 21 in step S63, and the process ends.

When it is determined in step S65 that the connection between the computer 21 and the effect server 26 has succeeded (YES), the process proceeds to step S66. Here, the user's computer 21 is connected to the billing server 27. In the next step S67, the computer 21 and charging server 27
It is determined whether or not the connection has been successful, and if unsuccessful, an error dialog is displayed on the display unit of the user's computer 21 in step S63, and the process ends.

When it is determined in step S67 that the connection between the computer 21 and the billing server 27 has succeeded (YE
For S), the procedure proceeds to step S68. In step S68, the user information is sent to the connected billing server 27, and the number of times the effect is used and the time are recorded together with the connection state.

In the next step S69, the video signal is sent to the streaming server 23 and the effect server 26, and the live distribution of the video becomes possible in this state. Hereinafter, step S70
Effects are added and billing is performed by the following procedure. The time stamp is used to specify and process images on a frame-by-frame basis, and is used for superimposing images on the effect server 26.

First, in step S70, the 2D code existing in the image photographed by the camera is searched. This image is searched in frame units. Next, in step S71, it is determined whether or not the 2D code can be recognized, and if the 2D code cannot be recognized (NO), the procedure returns to step S70 to search for the 2D code existing in the next image. Move to. When the 2D code can be recognized in step S71 (YE
In step S), the effect database 22 of the effect server 26 is queried for effect information corresponding to the 2D code ID recognized in step S72. This effect information includes effect selection information about the effect, that is, 2
Effect setting information such as relative position information (position) regarding the relative position between the D code ID, the two-dimensional code information and the effect display position, and effect addition range information (range) regarding the display range of the effect to be added is included. Further, it may also include effect addition prohibition information for a specific user. In conjunction with the inquiry to the effect database 22, the time stamp of the frame of the image is transmitted to the effect server 26 in step S73.

Next, the effect server 26 determines whether or not the effect information corresponding to the transmitted 2D code ID exists (step S76). If there is no corresponding effect information (NO), computer 2
1 is notified, and the process returns to step S70 to start processing the next image frame.

If the corresponding effect information is present (YES), the contents corresponding to the corresponding 2D code ID in step S77 (for example, effect type, mosaic, bitmap image, 3DCG, background replacement), relative position information, And effect addition range information for effect database 2
Get from 2. Next, in step S78, the charging server 2
Data transmission for the used 2D code ID to 7 is performed. The billing server 27 receives this data transmission, increments the number of times the effect of the 2D code ID used in step S83 is used, and saves it.

After data transmission of the used 2D code ID to the billing server 27, in step S79, the data file is read according to the type of effect. When the effect is a bitmap image, the corresponding bitmap image file is loaded, when the effect is 3D computer graphics (3DCG), the corresponding 3D computer graphics is loaded, and when the effect is background replacement A background image file to be selected is read in.

According to the read operation of the data file, it is determined in step S80 whether or not the corresponding data file exists. If the data file does not exist (NO), the procedure returns to step 70 and the operation is performed again for the next frame.

If the data file exists in step S80 (YES), 2D coordinates and 3D coordinates for the effect server 26 are acquired. At this time, the user's computer 21 acquires the position (2D coordinate) of the recognized 2D code (step S74). The position of the 2D code recognized here is the two-dimensional coordinates (P _ix , P _iy ) of the 2D code on the finder window supplied from the FA program 56E, as in the operation of step S24 in the flow of FIG. Is. Further, in the case of three-dimensional computer graphics (3DCG) that requires a three-dimensional display effect, the three-dimensional coordinates are read in step S75. This operation is step S25 in the flow of FIG. 7 described above.
Specifically, using the rotation matrix R and the movement vector T, the coordinates ωi in the actual three-dimensional coordinate system are converted into the three-dimensional coordinates in the camera coordinate system, and the coordinates Pi in the finder window coordinate system are calculated. Then, the required three-dimensional coordinates are obtained.

Using the 2D code position (2D coordinates) and three-dimensional coordinates calculated in this way, the effect server 26 draws an effect reflecting the effect information based on the coordinates based on the 2D code in step S81. To do.
The drawing of the effect that reflects this effect information is performed by the image processing means, for example, the effect server 2
Under the control of the CPU in 6, the display data in the VRAM is rewritten to the data including the effect to proceed. At this time, the time stamp of the frame in which the 2D code is recognized is referred to, and the effect in which the effect information is reflected is added to the camera image in the corresponding image frame based on the coordinates of the 2D code. By using the time stamp,
Reliable processing can be performed even on a server located at a remote place. Next, in step S82, the image on which the effect is superimposed is sent to the streaming server 23 frame by frame. In the streaming server 23, processing based on the time stamp is in progress, and the effect server 2
When there is a frame to which an effect is added from 3, the image from the effect server 23 is prioritized,
Alternatively, the frame having the same time stamp is overwritten and distributed via the Internet (step S87).

Regarding the operation of the charging server 27, the procedure S
After receiving the data transmission for the 2D code ID used in 83, in step S84, the used 2D code I
The number of times of use and the time of use of the effect for D are recorded. Periodically or as required, the user information is charged according to the usage time (step S85). For the effect, a usage unit price can be set for each effect, and user information can also be managed to enable accurate billing. Similarly, in step S86, a copyright fee and other fees are paid to the creator who provided the effect on a regular basis or according to the amount charged as needed. With such a system, it is also possible to collect effects with better designs.

In the above embodiment, mosaics, bitmap images, and 3DCG (three-dimensional computer graphics) are given as examples of 2D code effects, but various other effects can be added. For example, if the ID of the 2D code is made to correspond to the pen color, moving the 2D code makes it possible to draw on a virtual canvas. It is also possible to virtually play a musical instrument and create a music program by associating the ID of the 2D code with the musical instrument or tone color and associating the position and posture of the 2D code with the volume or scale.

Further, in the above-mentioned embodiment, the Internet live video distribution is mainly described, but the present invention can be applied not only to the Internet live video distribution but also to normal television broadcasting. That is, although the information processing system of the present invention is particularly intended for Internet live video distribution,
The 2D code effect can also be applied to existing television broadcasting. Further, although the information processing system of the present invention assumes live video distribution, graphics and effects by image recognition are not limited to live video distribution. The idea of recognizing images in the real world and fusing graphics to them can be applied to everyday life. Vivid graphics emerge on familiar walls, 3D characters appear as if they are there, and surprising effects like movie appear in front of you from image recognition to real world. It becomes possible by acquiring information and superimposing graphics on it.

In the above embodiment, the effect information is embedded and the 2D code is used as the image recognizing means, because the 2D code easily obtains information such as ID, position, orientation, and size. As long as these pieces of information can be obtained, other image recognition methods can be substituted.

Further, as an application range of the present invention, the higher the degree of image recognition, the more various effects can be applied. Ie. In the above-described embodiment, only the background and the 2D code are recognized for image recognition, but various effects can be applied if more advanced recognition is possible. For example, if the facial expressions and gestures of the performers can be recognized, a character that reacts to the movements of the performers will be able to appear in the program.

Further, in the information processing apparatus and the information processing system of the above-mentioned embodiment, since Internet live video distribution is assumed, a camera is used as an input device and image recognition from the camera is performed. However, a sensor other than a camera can be used as the input device. For example, it is possible to obtain the movement of the performer using a magnetic sensor or infrared rays and apply an effect according to the movement. You can also recognize the voice and apply an effect accordingly. In this case, effects can be applied not only to images but also to voice and music. Also,
The video transmission side with the input device is exemplified by a notebook PC, but in principle, a mobile phone with a sensor such as a camera,
It is also possible to use a device such as a PDA and is not limited to a personal computer. Furthermore, regarding the device on the viewer side, a note P is used as a terminal on the viewer side to which the video is distributed.
C, not limited to desktop PC, mobile phone, PD
A, You can also watch the live-streamed images on a television receiver. Further, the present invention is not limited to a laptop personal computer with a built-in camera, and may be realized by connecting an external camera to a desktop personal computer. Further, not only a live image captured by a camera but also a medium such as a hard disk drive (HDD) may be recorded once and then the 2D code may be recognized.

[0129]

As described above, by combining 2D code recognition technology with live video distribution, the captured video is not distributed as it is, but virtual graphics and effects that are not actually present are superimposed. Combined live video distribution is realized. Since the operation uses the 2D code recognition technology, the effect can be automatically combined on the screen without any complicated operation for the user, and the effect can be controlled not only by the photographer but also by the performer. Will be realized.

[Brief description of drawings]

FIG. 1 is a schematic diagram showing an example of an information processing system of the present invention.

FIG. 2 is a schematic diagram showing an example of a 2D code.

FIG. 3 is a schematic diagram showing a program stored in a hard disk drive of a computer used in an example of the information processing system of the present invention.

FIG. 4 is a flowchart of a 2D code recognition process used in an example of the information processing system of the present invention.

FIG. 5 is a schematic diagram showing a window in a 2D code recognition process.

FIG. 6 is a diagram showing a screen state in 2D code recognition processing.

FIG. 7 is a flowchart of 3D application processing used in an example of the information processing system of the present invention.

FIG. 8 is a diagram showing a case where a character in a virtual space is displayed on the screen as a result of the 3D application processing.

FIG. 9 is a schematic diagram showing an example of an information processing system of the present invention together with video processing.

FIG. 10 is a flow chart of creating an effect database before live distribution used in an example of the information processing system of the present invention.

FIG. 11 is a flowchart showing an effect addition process during live distribution used in an example of the information processing system of the present invention.

FIG. 12 is a diagram showing a camera input screen when an example of the information processing system of the present invention is used.

FIG. 13 is a diagram showing a screen to which a mosaic effect is added when an example of the information processing system of the present invention is used.

FIG. 14 is a diagram showing a screen to which a bitmap image is added when an example of the information processing system of the present invention is used.

FIG. 15 is a diagram showing a screen to which three-dimensional computer graphics are added when an example of the information processing system of the present invention is used.

FIG. 16 is a diagram showing a screen on which a background is replaced when an example of the information processing system of the present invention is used.

FIG. 17 is a schematic diagram showing another example of the information processing system of the present invention.

FIG. 18 is a schematic diagram showing another example of the information processing system of the present invention together with a distribution screen.

FIG. 19 is a flowchart of an effect addition operation during live distribution used in the another example of the information processing system of the present invention.

[Explanation of symbols]

11 local computer 12, 22 Effect database 13, 23 Streaming server 14, 24 terminals 15, 25 2D code 21 computer 26 Effect Server 27 Billing server

   ─────────────────────────────────────────────────── ─── Continued front page    (72) Inventor Keigo Ihara             6-735 Kita-Shinagawa, Shinagawa-ku, Tokyo Soni             -Inside the corporation (72) Inventor Takanori Nishimura             6-735 Kita-Shinagawa, Shinagawa-ku, Tokyo Soni             -Inside the corporation (72) Inventor Yasushi Miyajima             6-735 Kita-Shinagawa, Shinagawa-ku, Tokyo Soni             -Inside the corporation F term (reference) 5C023 AA09 AA10 AA11 AA18 BA11                       CA01 CA05                 5C063 AB05 AC10 CA23 CA36 DA01                       DA03                 5C064 BA01 BB01 BC01 BC18 BC23                       BD13

Claims (38)

[Claims] 1. An image capturing unit that captures moving image information, an extracting unit that extracts two-dimensional code information included in the moving image information, and the extracting unit that extracts the two-dimensional code information included in the moving image information according to the two-dimensional code information extracted by the extracting unit. An information processing apparatus comprising: an image processing unit that processes moving image information; and a sending unit that sends the moving image information processed by the image processing unit in a streaming format. 2. The information processing apparatus according to claim 1, wherein the two-dimensional code information includes effect setting information. 3. The information processing apparatus according to claim 2, wherein the effect setting information includes effect selection information that specifies an effect to be added. 4. The information processing apparatus according to claim 2, wherein the effect setting information includes relative position information regarding a relative position between the two-dimensional code information and the effect display position. 5. The information processing apparatus according to claim 2, wherein the effect setting information includes effect addition range information regarding a display range of an effect to be added. 6. The information processing apparatus according to claim 1, wherein the two-dimensional code information includes coordinate information. 7. The information processing apparatus according to claim 6, wherein a desired effect is linked to the coordinate information. 8. The information processing apparatus according to claim 1, wherein the extraction means includes a database for processing the moving image information according to the two-dimensional code information. 9. The information processing apparatus according to claim 1, wherein the two-dimensional code information is extracted in frame units of the moving image information, and the image processing unit processes the moving image information in frame units. . 10. A step of fetching moving image information, a step of extracting two-dimensional code information included in the moving image information, and a step of processing the moving image information according to the extracted two-dimensional code information. And a step of transmitting the processed moving image information in a streaming format. 11. The information processing method according to claim 10, wherein the two-dimensional code information includes effect setting information. 12. The information processing method according to claim 11, wherein the effect setting information includes effect selection information that specifies an effect to be added. 13. The information processing method according to claim 11, wherein the effect setting information includes relative position information regarding a relative position between the two-dimensional code information and the effect display position. 14. The information processing method according to claim 11, wherein the effect setting information includes effect addition range information regarding a display range of an effect to be added. 15. The information processing method according to claim 10, wherein the two-dimensional code information includes coordinate information. 16. The information processing method according to claim 15, wherein a desired effect is associated with the coordinate information. 17. The information processing method according to claim 10, wherein the extraction of the two-dimensional code information is performed based on a database for processing the moving image information. 18. The information according to claim 10, wherein the two-dimensional code information is extracted in frame units of the moving image information, and the moving image information is processed by processing the moving image information in frame units. Processing method. 19. A procedure of taking in moving image information, a step of extracting two-dimensional code information included in the moving image information, and a step of processing the moving image information according to the extracted two-dimensional code information. An information processing program, comprising: a step of transmitting the processed moving image information in a streaming format. 20. The information processing program according to claim 19, wherein the two-dimensional code information includes effect setting information. 21. The information processing program according to claim 20, wherein the effect setting information includes effect selection information that specifies an effect to be added. 22. The information processing program according to claim 20, wherein the effect setting information includes relative position information regarding a relative position between the two-dimensional code information and the effect display position. 23. The information processing program according to claim 20, wherein the effect setting information includes effect addition range information regarding a display range of an effect to be added. 24. The information processing program according to claim 19, wherein the two-dimensional code information includes coordinate information. 25. The information processing program according to claim 24, wherein a desired effect is linked to the coordinate information. 26. The information processing according to claim 19, wherein the two-dimensional code information is extracted in frame units of the moving image information, and the moving image is processed in frame units of the moving image information. program. 27. An image capturing means capable of outputting captured moving image information in a streaming format including a time stamp for specifying a frame; and two-dimensional code information included in the moving image information in frame units. An information processing apparatus, comprising: an extracting unit that extracts the time stamp and an effect setting information that is included in the extracted two-dimensional code information. 28. A step of enabling transmission of captured moving image information in a streaming format including a time stamp for specifying a frame; and extracting two-dimensional code information included in the moving image information in frame units. An information processing method comprising: a step; and a step of transmitting the time stamp together with the effect setting information included in the extracted two-dimensional code information. 29. A procedure for enabling transmission of captured moving image information in a streaming format including a time stamp for identifying a frame, and extracting two-dimensional code information included in the moving image information in frame units. An information processing program comprising a procedure and a procedure for transmitting the time stamp together with the effect setting information included in the extracted two-dimensional code information. 30. Image processing means for specifying moving image information in frame units by a time stamp and processing moving image information in frame units in accordance with effect setting information,
An information processing apparatus comprising: a sending unit that sends the moving image information processed by the image processing unit in a streaming format. 31. The information processing apparatus according to claim 30, wherein the image processing means comprises a database for processing the moving image information according to the two-dimensional code information. 32. The charging device according to claim 30, further comprising a charging unit for performing a charging process according to an effect addition state when processing the moving image information in the frame unit according to the effect setting information. Information processing equipment. 33. A step of specifying moving image information in frame units by a time stamp and processing the moving image information in frame units according to effect setting information; and sending the processed moving image information in a streaming format. An information processing method, comprising: 34. The information processing according to claim 33, wherein when the moving image information is processed in the frame unit according to the effect setting information, a billing process is performed according to an effect addition state. Method. 35. A procedure of specifying moving image information in frame units by a time stamp and processing the moving image information in frame units according to effect setting information; and sending the processed moving image information in a streaming format. An information processing program, characterized by including the following procedure. 36. The information processing according to claim 35, wherein when the moving image information is processed in the frame unit according to the effect setting information, a billing process is performed according to an effect addition state. program. 37. Image capturing means capable of outputting captured moving image information in a streaming format including a time stamp for specifying a frame; and two-dimensional code information included in the moving image information in frame units. In the frame unit of the moving image information based on the time stamp sent from the sending unit, and the sending unit sending the time stamp together with the effect setting information included in the extracted two-dimensional code information. And an image processing unit that processes moving image information in frame units according to the effect setting information and a distribution unit that distributes the moving image information processed by the image processing unit in a streaming format. Information processing system. 38. The method according to claim 37, further comprising a charging means for performing a charging process according to an effect addition state when the moving image information is processed in the frame unit according to the effect setting information. Information processing system.