JP2009093376A - Signal processing system, signal processing method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a signal processing system, a signal processing method and a program for seamlessly displaying a CG on an image with respect to the free movement of a camera. <P>SOLUTION: An arithmetic part 23 is configured to calculate an inter-coordinate conversion parameter for converting the position on the (n+1)th coordinates into the position on the n-th coordinates based on the n-th conversion parameter for converting the position on the n-th coordinates specified by the n-th light source group into the position on the coordinates specified by itself and the (n+1)th conversion parameter for converting the position on the (n+1)th coordinates specified by the (n+1)th light source group into the position on the coordinates specified by itself. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a signal processing system, a signal processing method, and a program for superimposing three-dimensional computer graphics (3D-CG) on a real image captured by a camera in real time.

  A technique for superimposing and displaying three-dimensional computer graphics (3D-CG) in real time on a real image photographed by a camera based on three-dimensional information of a photographing space has been proposed. These technologies are called Augmented Reality, and they can be applied to various applications such as games, entertainment devices, and information devices, as they provide a new world feeling through the fusion of the real world and the virtual world. Expected.

  In these techniques, in order to display 3D-CG on a real image, first, a coordinate system (world coordinate) serving as a reference for three-dimensional arrangement information is set in a space in which the image is taken, and a camera in the coordinate system is set. The three-dimensional position, the line-of-sight direction, and the tilt angle are calculated. Based on the information, a 3D-CG object is virtually arranged at a predetermined position in the real space, and the CG when viewed from the actual position and direction of the camera is superimposed on the image and displayed.

  In order for the camera system to execute such overlay display in real time in response to the movement of the camera or the subject, a target object serving as a reference for setting the coordinate system is set in the shooting space, and the target object is set. Should be easily detected. At present, several methods have been proposed as means for that purpose. For example, a method has been proposed in which a card on which a two-dimensional code is printed is placed in a shooting space, and the camera determines the code, shape, and the like by image processing, thereby calculating card placement information with respect to the camera (for example, Non-patent document 1). There has also been proposed a method of extracting a feature point from a captured image and analyzing the feature point from the appearance (see, for example, Non-Patent Document 2). In this method, the positions of the paper are calculated by extracting the positions of the four corners of the rectangular paper by image processing.

  However, since these methods are based on image processing, the image is unclear due to the brightness of the ambient light, or the resolution cannot be obtained depending on the orientation and distance of the target, and the image cannot be analyzed. Therefore, various restrictions are imposed on the usage environment such as the arrangement of cameras and targets.

  On the other hand, a method using an LED (Light Emitting Diode) light source as a marker has been proposed as a method for avoiding such problems and enabling stable and robust target detection in various environments (for example, patents). Reference 1). This is because an LED light source that transmits data encoded with a blinking pattern of light is arranged in the imaging space, the data is received by an image sensor that constitutes the camera, and the light receiving pixel position is analyzed. Three-dimensional information is calculated. According to this method, if the LED has sufficient luminous intensity and has high directivity, it is possible to detect the target regardless of the brightness of the environment or the arrangement of the camera. Furthermore, since image processing is not required, the target is recognized at high speed, and CG display with high real-time characteristics is possible.

Impress Watch, “SCEJ, PS3“ THE EYE OF JUDGMENT ”card game that reads the card layout in real time and reflects it in real time”, [online], August 25, 2006, [Search September 1, 2007], Internet <URL: http://www.watch.impress.co.jp/game/docs/20060825/teoj.htm> The Journal of the Institute of Image Information and Television Engineers Vol.51, No.7 (1997) pp.1086-1095 (1997) JP 2004-56343 A

  However, as a disadvantage of the method of setting the target in the shooting space as described above, if the target moves out of the shooting space due to camera movement, panning, movement of the target itself, etc., the coordinate system is set. There is a problem that becomes impossible. That is, CG is displayed when the target is present in the shooting space, but the CG display disappears when the target is removed. The shooting angle of view of the camera is usually about 50-60 degrees. However, assuming an application that is frequently moved in a camera mounted on a mobile terminal, the camera image can be easily displayed on the target. It is possible that the corners will fall off, making it less practical.

  The present invention has been proposed to solve such a problem. A signal processing system and a signal processing method capable of seamlessly displaying CG on an image with respect to a free movement of the camera, and Provide a program.

  In order to solve the above-described problems, a signal processing system according to the present invention includes an input unit that inputs identification information and a pixel position of each light source of a light source group in which light sources are arranged in a predetermined positional relationship, and each of the light sources. Calculation means for calculating a self-coordinate conversion parameter for converting a position on the coordinate specified by the light source group to a position on the coordinate specified by the light source group based on the identification information and the pixel position. When the identification information and the pixel position of each light source are input from one light source group and the second light source group, the position on the first coordinate defined by the first light source group A first self-coordinate conversion parameter for converting to the position of the second self-coordinate, and a second self-coordinate conversion parameter for converting the position on the second coordinate specified by the second light source group to a position on the coordinate specified by the second light source group Based on the second coordinate position It is characterized by calculating the inter-coordinate conversion parameter for converting the position on the serial first coordinate.

  The signal processing method according to the present invention includes an input step of inputting identification information and a pixel position of each light source of a light source group in which the light sources are arranged in a predetermined positional relationship, a first light source group, and a second light source. When the identification information and the pixel position of each light source are input from the group, the position on the coordinates defined by the first light source group based on the identification information and the pixel position of each light source And a second self-coordinate conversion parameter for converting a position on the coordinate specified by the second light source group to a position on the coordinate specified by the second light source group. A coordinate for converting the position on the second coordinate to the position on the first coordinate based on the self-coordinate parameter calculating step, the first self-coordinate conversion parameter, and the second self-coordinate conversion parameter. Coordinates for calculating conversion parameters It is characterized by having a conversion parameter calculating step.

  In addition, the program according to the present invention includes an input step of inputting identification information and pixel position of each light source of a light source group in which light sources are arranged in a predetermined positional relationship, a first light source group, and a second light source group. If the identification information and the pixel position of each light source are input from, the position on the coordinates that the first light source group defines on the coordinates based on the identification information and the pixel position of each light source Self-coordinates for calculating a first self-coordinate transformation parameter to be converted into a second self-coordinate transformation parameter for transforming a position on the coordinate specified by the second light source group into a position on the coordinate specified by the self Inter-coordinate conversion for converting the position on the second coordinate to the position on the first coordinate based on the parameter calculating step and the first self-coordinate conversion parameter and the second self-coordinate conversion parameter Between coordinates to calculate parameters To perform a conversion parameter calculation process on the computer.

  According to the present invention, since the inter-coordinate conversion parameter for converting the position on the second coordinate to the position on the first coordinate is calculated, CG is seamlessly displayed on the image with respect to the free movement of the camera. can do.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
(First embodiment)
FIG. 1 is a schematic diagram showing a configuration of a signal processing system. This signal processing system has a light source group 10 and a camera 20 in which four LEDs (Light Emitting Diodes) (L_A to L_D) are arranged on the same plane, and the light source group 10 is installed in an imaging region of the camera 20. Has been.

  In the light source group 10, for example, LEDs (L_A to L_D) are fixed to a flat board as shown in FIG. Moreover, it is not restricted to this, You may be fixed and installed in the wall etc. For example, the four LEDs are arranged in a square shape with one side having a length of S. Each LED repeats blinking at high speed, and by coding data in the blinking pattern, different ID identification information (ID_A to ID_D) of 8 bits, for example, is transmitted.

  On the other hand, the camera 20 can recognize which LED forms an image on which pixel by detecting information from the light source group 10 by each pixel of the image sensor simultaneously with acquiring a normal image ( For example, refer to JP2003-169251.). As described above, in the camera 20, by analyzing the positional relationship between the pixels formed by the four LEDs, it is possible to calculate the position and orientation (gaze direction, inclination) of the camera 20 with respect to the light source group 10. Become.

  FIG. 2 is a block diagram illustrating a configuration of the signal processing device. This signal processing device is installed in, for example, the camera 20 and includes a sensor 21, a signal processing unit 22, a calculation unit 23, a memory 24, and a display unit 25.

  The sensor 21 images the light source group 10 described above, and outputs an image signal (RAW data) and an optical communication signal.

  The signal processing unit 22 includes an image signal processing unit 221 that processes RAW data and a decoding processing unit 222 that decodes an optical communication signal. The image signal processing unit 221 converts the RAW data into a predetermined format such as YUV422. The decode processing unit 222 decodes the optical communication signal to detect the ID number of each light source, and detects the pixel position of each light source from the image.

  The calculation unit 23 includes a camera position / posture detection unit 231, a conversion parameter calculation unit 232, a 3DCG position coordinate calculation unit 233, and a 3DCG composition processing unit 234. The camera position / orientation detection unit 231 acquires the positional relationship of the light source group from the ID number and pixel position of each light source, and converts the position on the coordinate specified by the light source group into the position on the coordinate specified by the camera itself A coordinate conversion parameter is calculated. The conversion parameter calculation unit 232 calculates an inter-coordinate conversion parameter between coordinates defined by one light source group and coordinates defined by the other light source group, as will be described later. The 3DCG position coordinate calculation unit 233 converts 3DCG coordinates defined by the light source group based on its own coordinate conversion parameters and inter-coordinate conversion parameters. The 3DCG synthesis processing unit 234 synthesizes 3DCG into an image signal.

  The memory 24 includes a parameter storage unit 241 that stores its own coordinate conversion parameters, inter-coordinate parameters, and the like, and a storage unit 24 that stores a 3DCG position predetermined by the light source group.

  The display unit 25 includes a display such as an LCD (Liquid Crystal Display) and displays an image in which 3DCG is synthesized.

  Next, the relationship of the coordinate system between the light source group 10 and the camera 20 will be described in detail. 3 and 4 are diagrams showing the relationship between the light source group and the coordinate system of the camera 20, and FIG. 5 is a flowchart showing the coordinate conversion process.

  Here, a coordinate system indicating the position and orientation of the camera 20 is a camera coordinate system (CC: Camera Coordinate), the direction of the imaging surface (coupling plane) is the xy axis, and the optical axis direction is the z axis. The imaging surface of the camera is set at a position away from the CC origin by the focal length f to the −z side. On the other hand, the coordinate system of the light source group 10 is a world coordinate (WC: World Coordinate) that defines absolute coordinates in real space, the LED position of L_A is the origin, the direction of L_B is the x axis, the direction of L_D is the y axis, The direction perpendicular to the plane of the group 10 is taken as the z-axis.

  If the orthogonal unit vector that defines the x, y, z axes of WC on the CC is (ux, ui, uz) and the origin of WC is d1 = (d1, d2, d3), an arbitrary point on WC ( A calculation formula for converting xw, yw, zw) into a coordinate value (xi, yi, zi) on CC can be expressed by formula (1-1).

  Here, U = (ux, uy, uz) represents a rotational transformation, and d1 represents a translation component. Further, (xi, yi, zi) on the CC is converted into a pixel position (Xi, Yi) on the imaging surface (sensor) by the expression (1-2).

  The value of each element of the conversion parameters U and d1 changes sequentially according to the movement of the camera position and posture. Each time, the coordinate value (0, 0) ( If information of the light receiving pixel position (x0, y0) (x1, y1) (x2, y2) (x3, y3) on the imaging plane corresponding to (S, 0) (S, S) (0, S) is obtained. From these, it is possible to calculate the elements of the conversion parameter in real time as shown in equation (1-3).

  When the conversion parameter is obtained, the inverse conversion operation is performed as in the expressions (1-4) to (1-7), so that the camera position (xc, yc, zc) on the WC, the camera Vector components indicating the line-of-sight direction (xv, yv, zv) and the upper direction (xu, yu, zu) of the camera can be calculated.

  Normally, in a 3D graphics API (Application Program Interface) such as OpenGL (Open Graphics Library), a 3D-CG object placed at an arbitrary position on the WC is determined if the camera position and orientation parameters are determined with respect to the WC. Can be displayed as a figure viewed from the camera. Therefore, 3D-CG can be superimposed and displayed on the actual image based on the light source group.

  Here, as shown in FIG. 4, if a light source group 11 different from the light source group 10 is prepared, local coordinates (LC1: Local Coordinate) defined by the light source group 11 on the WC defined by the light source group 10. 1) can be set. The ID identification information of the LED light sources installed in the light source group 11 is distinguished from the light source group 10 as (ID_E to ID_H), and the same calculation as that for the light source group 10 is performed, so that the camera coordinates for the light source group 11 are obtained. The conversion matrix E and the translation component d2 can be calculated.

  If this conversion parameter is obtained, the inverse conversion operation is performed as in equations (1-9) to (1-11), so that the camera position (xc_e, yc_e, zc_e) on the LC 1 Each vector component indicating the line-of-sight direction (xv_e, yv_e, zv_e) and the upper direction of the camera (xu_e, yu_e, zu_e) can be calculated.

  Further, since the camera parameters for the light source group 10 are given by U and d1, coordinate values (xe, ye, ze) on LC1 are converted into coordinate values (xw, yw, zw) on WC by combining these conversion parameters. Can be converted to

  If this conversion parameter is obtained, the inverse conversion operation is performed as in equations (1-13) to (1-15), so that the camera position (xc, yc, zc) on the WC and the camera Each vector component indicating the line-of-sight direction (xv, yv, zv) and the upper direction (xu, yu, zu) of the camera can be calculated.

  Next, arithmetic processing in the signal processing device will be described with reference to the flowchart shown in FIG. In step S11, the decode processing unit 222 detects the imaging position on the camera imaging plane of each LED (L_A to L_D) from the input ID number and pixel position.

  In step S <b> 12, the camera position / posture detection unit 231 determines whether the LEDs (L_A to L_D) are on the WC based on the imaging positions on the camera imaging plane and the above expressions (1-1) to (1-3). A conversion parameter, a conversion matrix U, and a translation component d1 for converting an arbitrary point (xw, yw, zw) into a coordinate value (xi, yi, zi) on CC are calculated.

  Further, the camera position / orientation detection unit 231 performs the inverse conversion operation on the basis of the conversion parameters U and d1 as in Expressions (1-4) to (1-7), so that the camera on the WC Each vector component indicating the position (xc, yc, zc), the viewing direction of the camera (xv, yv, zv), and the upper direction (xu, yu, zu) of the camera is calculated (step S13).

  Here, when the light source group 10 that defines WC and the light source group 11 that defines LC1 exist in the imaging region of the camera 20, the decode processing unit 22 receives the input ID number as in the case of the light source group 10. And the imaging position on the camera imaging plane of each LED (L_E to L_H) is detected from the pixel position.

  In step S14, the camera position / posture detection unit 231 determines the positions of the LEDs (L_E to L_H) on the camera imaging plane and the above formulas (1-1) to (1-3). A conversion parameter, a conversion matrix E, and a translation component d2 for converting an arbitrary point (xe, ye, ze) on LC1 into a coordinate value (xi, yi, zi) on CC are calculated (formula (1-8)). reference.).

In step S15, the conversion parameter calculation unit 232, any point on LC1 (xe, ye, ze) coordinate values on WC and (xw, yw, zw) conversion parameter for converting the transformation matrix _{M 1} and translation calculating the component _{D 1} (equation (1-12) reference.). The transformation matrix M ₁ and the translation component D ₁ are stored in the parameter storage unit 241. Further, the conversion parameter calculation unit 232 performs the inverse conversion operation on the basis of the conversion parameters M ₁ and D _{1 as shown} in the equations (1-13) to (1-15), so that the camera on the WC Each vector component indicating the position (xc, yc, zc), the viewing direction of the camera (xv, yv, zv), and the upper direction of the camera (xu, yu, zu) is calculated.

In step S16, the 3DCG position coordinate calculation unit 233 converts the position of the 3DCG model defined on the LC1 onto the WC. Specifically, the 3DCG position coordinate calculation unit 233 reads the coordinates of the 3DCG model defined on the LC1 from the 3DCG position coordinate storage unit 242 and, based on the conversion parameters M ₁ and D ₁ , any point on the LC1. Is converted into coordinates on the WC (see Expression (1-12)). The 3DCG synthesis processing unit 234 synthesizes the 3DCG model with the image from the image signal processing unit 221 and outputs the synthesized 3DCG model to the display unit 25.

  Thereby, for example, 3D-CG defined by the light source group 11 can be superimposed and displayed in the virtual space defined by the WC, and the 3D-CG can be controlled by the movement of the light source group 11. It becomes possible.

(Second Embodiment)
FIG. 6 is a schematic diagram illustrating changes in the imaging region in the second embodiment. In the first embodiment, when the light source group 10 and the light source group 11 exist within the angle of view at the same time, a relative relationship between the two light source groups is calculated and the virtual space defined in the WC is calculated. 3DCG defined by the light source group 11 is superimposed and displayed. However, in the second embodiment, when the light source group 10 is out of the imaging region, the WC coordinate system based on the light source group 10 is used. Will continue to be inherited. In addition, since the structure of the signal processing apparatus in 2nd Embodiment is the same as that of the structure of the signal processing apparatus shown in FIG. 2, it attaches | subjects the same code | symbol and abbreviate | omits description.

  7A and 7B are flowcharts showing the operation of the signal processing apparatus according to the second embodiment. Here, as shown in FIG. 6, a case is considered where the shooting region of the camera 20 moves from the direction of the light source group 10 to the direction of the light source group 11 by panning the camera 20. Steps S11 to S15 are the same as the processing in the first embodiment. That is, steps S20 to S26 are added to the operation shown in FIG.

  First, in step S20, the camera position / orientation detection unit 231 detects from the ID number of each light source that the light source group 10 and the light source group 11 exist in the imaging region of the camera. That is, the LED ID number (ID_A to ID_D) of the light source group 10 and the LED number (ID_E to ID_H) of the light source group 11 are detected.

  In step S20, in the case of the imaging region 1 where both the light source group 10 and the light source group 11 exist, the process proceeds to step S11.

Steps S11 to S15 are the same as the processing in the first embodiment. That is, in the imaging region 1, any point on LC1 (xe, ye, ze) coordinate values on WC and (xw, yw, zw) conversion parameter for converting the, the transformation matrix _{M 1} and the parallel movement component _{D 1} Calculate (see Formula (1-12)). Here, the element values of the conversion parameters U, d 1, E, and d 2 change sequentially with the movement of the camera 20. However, since the conversion parameters M ₁ and D ₁ represent the rotational transformation and translation component between CC and LC ₁ respectively, if the relative relationship between the light source group 10 and the light source group 11 is fixed, The element value is constant regardless of the movement of the camera 20.

When the light source group 10 (ID_A to ID_D) that defines WC is not detected and only the light source group 11 (L_E to L_H) that defines LC1 is detected (step S21), the process proceeds to step S22 shown in FIG. The matrix M ₁ and the translation component D ₁ are stored in the parameter storage unit 241. As described above, when the light source group 10 is out of the imaging region (imaging region 2), the conversion parameters M ₁ and D ₁ may be stored in the parameter storage unit 241.

In step S23, the conversion parameter calculation unit 232, based on the transformation matrix _{M 1} and the parallel movement component _{D 1,} by performing the inverse transform operation (1-13) as formula - (1-15) below, Vector components indicating the camera position (xc, yc, zc) on the WC, the viewing direction of the camera (xv, yv, zv), and the upper direction (xu, yu, zu) of the camera are calculated.

  In step S24, the 3DCG position coordinate calculation unit 233 reads 3DCG model data from the 3DCG position coordinate storage unit 242, and determines whether the definition coordinate system of the 3DCG model is WC or LC1.

When the definition coordinate system of the 3DCG model is LC1, the process proceeds to step S25, and the 3DCG model defined on LC1 is converted to WC. Specifically, the 3DCG position coordinate calculation unit 233 reads 3DCG model data defined on the LC1 from the 3DCG position coordinate storage unit 242, and selects an arbitrary point on the LC1 based on the conversion parameters M ₁ and D _1. The coordinates are converted into coordinates on the WC (see Expression (1-12)). Then, the 3DCG synthesis processing unit 234 synthesizes the 3DCG model with the image from the image signal processing unit 221 and outputs it to the display unit 25.

  When the definition coordinate system of the 3DCG model is WC, the process proceeds to step S <b> 26, and the 3DCG synthesis processing unit 234 synthesizes the 3DCG model with the image from the image signal processing unit 221 and outputs it to the display unit 25.

When the imaging region moves in this manner and the light source group 10 deviates from the imaging region (imaging region 2), conventionally, the WC cannot be defined by converting the LC1 coordinate system into the WC coordinate system. It is possible to continue to display the 3DCG object that was not displayed on the screen. That is, by storing the conversion parameters M ₁ and D ₁ , the coordinate values on the LC 1 can be converted into the coordinate values on the WC. Even if the 3DCG object set on the LC 1 is moved to the imaging area 2, it can be displayed seamlessly.

(Third embodiment)
In the third embodiment, coordinate conversion is relayed using local coordinates using the same method as in the second embodiment, and the WC coordinates are inherited even if the camera 20 moves further. In addition, since the structure of the signal processing apparatus in 3rd Embodiment is the same as that of the signal processing apparatus shown in FIG. 2, it attaches | subjects the same code | symbol and abbreviate | omits description.

  FIG. 8 is a schematic diagram illustrating changes in the imaging region in the third embodiment. Here, as shown in FIG. 8, in a situation where the three light source groups 10, 11, and 12 are fixedly arranged, the angle of view (imaging area) of the camera 20 is panned from the light source group 10 to the light source group 12. The case where it does is demonstrated.

The conversion parameter calculation unit 232 converts the arbitrary point (xf, yf, zf) on the LC2 into the coordinate value (xe, ye, ze) on the LC1, similarly to the conversion between the WC and the LC1. M ₂ and D ₂ are calculated.

  If this conversion parameter is obtained, the inverse conversion operation is performed as shown in equations (3-2) to (3-4), so that the camera position (xc_f, yc_f, zc_f) on LC2 and the camera Each vector component indicating the line-of-sight direction (xv_f, yv_f, zv_f) and the upper direction of the camera (xu_f, yu_f, zu_f) can be calculated.

Further, since the camera parameters for the light source group 11 are given by M ₁ and D ₁ , the coordinate values (xf, yf, zf) on LC2 are converted into the coordinate values (xw, yw, zw).

  If this conversion parameter is obtained, the calculation is performed as shown in equations (3-6) to (3-8), so that the camera position (xc, yc, zc) on the WC and the line-of-sight direction of the camera ( xv, yv, zv), vector components indicating the upper direction (xu, yu, zu) of the camera can be calculated.

  9A and 9B are flowcharts illustrating the operation of the signal processing device according to the third embodiment. Here, steps S20 to S22 are the same as the processes with the same reference numerals shown in FIG. That is, steps S31 to S34 are added to the operations shown in FIGS. 7A and 7B.

In step S31, when the camera 20 pans and the light source group 11 and the light source group 12 enter the same imaging region, similarly to the processing of step S20 to step S15 shown in FIG. 7A, between the LC1, LC2, and CC, respectively. The conversion parameters E and F are calculated. That is, conversion parameters M ₂ and D ₂ for converting an arbitrary point (xf, yf, zf) on LC2 into a coordinate value (xe, ye, ze) on LC1 are calculated (see Expression (3-1)). ).

Light source group 11 which defines the LC1 (L_E~L_H) is not detected, if only the light source group 12 which defines the LC2 (L_I~L_L) is detected (step S32), the transformation matrix _{M 2} and the parallel movement component _{D 2} Is stored in the parameter storage unit 241 (step S33).

In step S < _b > 34, the conversion parameter calculation unit 232 performs an inverse conversion operation based on the conversion matrix M < _b > ₂ and the translation component D < _b > _{2 as} in Expressions (3-6) to (3-8), Vector components indicating the camera position (xc, yc, zc) on the WC, the viewing direction of the camera (xv, yv, zv), and the upper direction (xu, yu, zu) of the camera are calculated.

  About step S23-step S25, it is the same as that of the process of the same code | symbol shown to FIG. 7B.

  Thus, in the situation where the three light source groups 10, 11, and 12 are fixedly arranged, the reference coordinate system is used even when the angle of view (imaging area) of the camera 20 pans from the light source group 10 to the light source group 12. Can be maintained.

(Fourth embodiment)
The fourth embodiment is generalized so as to be applicable even when the shooting area of the camera 20 changes over a wider range than the above-described method. In other words, the light source groups 1 to n are arranged at intervals of at least two at the same time in the camera photographing area within the movement range of the camera 20 (the longest allowable interval between the light source groups depends on the distance to the camera 20. Change). Then, if the conversion parameters M _k and D _k (1 ≦ k ≦ n) between the two light source groups are sequentially stored according to the movement of the camera 20, only the light source group n is finally in the imaging region. When present, any point (xn, yn, zn) on LCn can be converted to a coordinate value on WC.

  The conversion formula between WC and LCn generalized using the n coordinate systems LC1 to LCn is expressed by Formula (3-9).

  Also, each vector component indicating the camera position (xc_n, yc_n, zc_n), the camera viewing direction (xv_n, yuv_n, zv_n), and the upper direction (xu_n, yu_n, zu_n) of the camera on LCn is (3 -10) It can be calculated by the equations (3-12).

  Further, each vector component indicating the camera position (xc, yc, zc) on the WC, the viewing direction of the camera (xv, yv, zv), and the upper direction (xu, yu, zu) of the camera is (3 It can be calculated by the formulas (-13) to (3-15).

  10A and 10B are flowcharts illustrating the operation of the signal processing apparatus according to the fourth embodiment. Here, step S20 to step S24 and step S31 to step S33 are the same as the processes with the same reference numerals shown in FIGS. That is, step S41 and step S42 are added to the operations shown in FIGS. 9A and 9B.

In step S41, when the camera 20 further pans and the light source group n (n is an integer equal to or greater than 1) and the light source group n + 1 enter the same imaging region, the processing in steps S20 to S33 shown in FIGS. 9A and 9B. In the same manner, conversion parameters M _{3 to} M _n and D _{3 to} D _n are calculated. The transformation matrix M _n and the translation component D _n are stored in the parameter storage unit 241.

In step S42, the conversion parameter calculation unit 232, based on the transformation matrix _{M n} and translation component _{D n,} by performing the inverse transform operation (3-13) as formula - (3-15) below, Vector components indicating the camera position (xc, yc, zc) on the WC, the viewing direction of the camera (xv, yv, zv), and the upper direction (xu, yu, zu) of the camera are calculated.

  About step S23-step S25, it is the same as that of the process of the same code | symbol shown to FIG. 9B.

  If the WC is defined by one light source group in the real space as described above, the WC can be inherited in an arbitrary imaging region. For example, in the omnidirectional direction, a CG based on the WC coordinate system is displayed on the image. It is possible to display seamlessly.

  Here, when the moving distance of the camera is long and it is necessary to relay many LCs for conversion to WC, it is not always necessary to perform conversion calculation in the order of the physical arrangement of the light source groups. When there are a large number of light source boards, multiple types of relay routes may be used as options for conversion to WC. At that time, if the route with the fewest relay coordinates is extracted and applied in the camera system, the calculation error is minimized. Is possible.

  In addition, the light source group may be installed at least within the range where the shooting area of the camera is considered to move according to the application.For example, when a camera mounted on a mobile terminal is assumed, the camera is moved up and down. There is no need to assume that the light source is directed, and a method of arranging necessary and sufficient light source groups only in the horizontal direction is conceivable.

  Moreover, although the above method performs position detection using the LED light source as a marker over the entire photographing range, it can also be used in combination with other methods. For example, for a specific area of the shooting range, a coordinate system may be determined by a two-dimensional bar code or image processing method such as feature point extraction.

  In the above-described embodiment, the sensor that acquires the actual image is the same as the sensor that performs optical communication. However, the sensors are not necessarily the same. For example, as shown in FIG. 11, the image sensor 212 and the optical communication sensor 211 are arranged close to each other, and the detection area of the optical communication sensor 211 is made larger than the imaging area of the image sensor 212 (view angle β> α).

  Accordingly, even if the light source group is out of the imaging area of the image sensor 212, the position of the WC can be held as long as the light source group does not deviate from the detection area of the optical communication sensor 211. In general, the angle of view of the image sensor 212 is limited to allow distortion of the image, applications, and the like. Therefore, although the degree of freedom of setting is small, the optical communication sensor 211 is basically not restricted by these, and the image It is possible to set a large angle. Therefore, the installation interval of the light source groups may be adjusted according to the detection area of the optical communication sensor 211, so that it can be installed at a large interval without being restricted by the imaging area of the image sensor 212. Can be reduced.

  In the above-described embodiment, one coordinate system is defined by the four LED light sources of the light source group. However, the four LED light sources are not necessarily fixed to the same board and the same plane. If the respective three-dimensional arrangement positions are known, they may be scattered in the space. For example, if there are four light sources in the imaging region and the coordinate values in the respective world coordinates are known, the conversion parameters U and d1 are calculated if the pixel positions on the imaging surface of the respective LED light sources are known. (Refer to the formula (1-1).) In other words, the position of each light source in the light source group may be stored as a position on the coordinates defined by the light source group, and the light source may be identified based on the optical communication signal to obtain a predetermined positional relationship. Thereby, since the conversion parameter is uniquely determined, the position of the camera on the world coordinates can be calculated. In addition, if other four LED light sources that define the local coordinate system are set therein, the local coordinate system (LC1) that can control 3DCG independently in the world coordinate system as in the above-described embodiment. Can be defined. That is, when the camera pans, even if a light source whose position in the world coordinate system is known deviates from the shooting area, another light source enters the shooting area by moving the camera, so that the world coordinate system is included in the shooting area. If there are at least four light sources whose positions are known, the world coordinate system can be defined. Therefore, the world coordinates can be displayed seamlessly with respect to the movement of the camera within a wide area.

  Further, the calculation of the above world coordinate system may be operated in combination with the coordinate system conversion means defined by the board to which the light source group is fixed, instead of using only individually arranged LED light sources. In other words, when there are four individually arranged light sources in the imaging area, the above-described method of storing the coordinate position is used, and when there are two light source boards, coordinate conversion calculation between the light source boards is executed. do it.

  It is also possible to acquire the coordinate position of each LED of the light source board that defines the world coordinates (WC) or each light source board that defines each local coordinate (LC) by optical communication. That is, each LED light source forming the light source board transmits the coordinate information in the world coordinates (WC) or the local coordinates (LC) simultaneously with transmitting the ID information. For example, in the case of the light source board shown in FIG. 1, the light source L_A is (0, 0, 0), L_B is (S, 0, 0), L_C is (S, S, 0), and L_D is (0, S, 0). ) Position coordinates are transmitted as data. Further, assuming that the arrangement of the light sources is standardized as a square as shown in FIG. 1, each LED may simply transmit the length S of one side of the square.

  In the above-described embodiment, the world coordinates are calculated using only the LED light source, but in combination with the optical communication of the LED light source, information from other sensors that detect the movement of the camera and You may use together. For example, the translation direction and distance of the camera can be obtained by integrating twice the measurement value obtained by the accelerometer. The rotation direction and angle of the camera can be obtained by integrating an angular velocity meter. Therefore, when these sensors are mounted on the camera and the required light source group does not exist in the shooting area as the camera moves, the conversion parameters are calculated from the required translation and rotation by these sensors. Interpolation is also possible.

  In the above-described embodiment, the hardware configuration has been described. However, the present invention is not limited to this, and arbitrary processing may be realized by causing a CPU (Central Processing Unit) to execute a computer program. Is possible. In this case, the computer program can be provided by being recorded on a recording medium, or can be provided by being transmitted via the Internet or another transmission medium.

It is a schematic diagram which shows the structure of a signal processing system. It is a block diagram which shows the structure of a signal processing apparatus. It is a figure which shows the relationship between the light source group and the coordinate system of a camera. It is a figure which shows the relationship between the light source group and the coordinate system of a camera. It is a flowchart which shows operation | movement of the signal processing apparatus in 1st Embodiment. It is a schematic diagram which shows the change of the imaging region in 2nd Embodiment. It is a flowchart which shows operation | movement of the signal processing apparatus in 2nd Embodiment (the 1). It is a flowchart which shows operation | movement of the signal processing apparatus in 2nd Embodiment (the 2). It is a schematic diagram which shows the change of the imaging area in 3rd Embodiment. It is a flowchart which shows operation | movement of the signal processing apparatus in 3rd Embodiment (the 1). It is a flowchart which shows operation | movement of the signal processing apparatus in 3rd Embodiment (the 2). It is a flowchart which shows operation | movement of the signal processing apparatus in 4th Embodiment (the 1). It is a flowchart which shows operation | movement of the signal processing apparatus in 4th Embodiment (the 2). It is a figure which shows the detection range of an image sensor and an optical communication sensor.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Light source group, 11 Light source group, 12 Light source group, 20 Camera, 21 Sensor, 22 Signal processing part, 22 Decoding process part, 23 Computation part, 24 Storage part, 25 Display part, 211 Optical communication sensor, 212 Image sensor, 221 Image signal processing unit, 222 decode processing unit, 231 camera position / attitude detection unit, 232 conversion parameter calculation unit, 233 3DCG position coordinate calculation unit, 234 3DCG synthesis processing unit, 241 parameter storage unit, 242 3DCG position coordinate storage unit

Claims (14)

Input means for inputting identification information and pixel position of each light source of a light source group in which the light sources are arranged in a predetermined positional relationship;
A calculation means for calculating a self-coordinate conversion parameter for converting a position on the coordinate specified by the light source group into a position on the coordinate specified by the light source group based on the identification information of each light source and the pixel position;
When the identification information and the pixel position of each light source are input from the first light source group and the second light source group, the calculation means determines the position on the first coordinate defined by the first light source group. A first self-coordinate conversion parameter for converting to a position on a coordinate specified by itself, and a second for converting a position on a second coordinate specified by the second light source group to a position on the coordinate specified by the second light source group. And a coordinate conversion parameter for converting a position on the second coordinate to a position on the first coordinate based on the own coordinate conversion parameter. A light source group in which at least four light sources are arranged in a predetermined positional relationship;
The signal processing system according to claim 1, wherein the light source transmits an optical communication signal including identification information of the light source by a blinking pattern of light. An imaging means for acquiring an image of the light source group;
3. The signal according to claim 2, further comprising detection means for detecting an identification information of each light source by decoding an optical communication signal from an image of the light source group and detecting a pixel position where each light source forms an image. Processing system. Storage means for storing the inter-coordinate conversion parameter,
4. The signal according to claim 3, wherein the computing means converts the position on the second coordinate to the position on the first coordinate based on the inter-coordinate conversion parameter stored in the storage means. Processing system. When the identification information and the pixel position of each light source are input from the second light source group and the third light source group, the calculation means, the second self-coordinate conversion parameter, and the third light source group Based on the third self-coordinate conversion parameter for converting the position on the third coordinate defined by the current position to the position on the coordinate defined by itself, the position on the third coordinate is determined on the second coordinate. Calculate the inter-coordinate conversion parameter to convert to the position,
Based on the inter-coordinate conversion parameter and the inter-coordinate conversion parameter for converting the position on the second coordinate to the position on the first coordinate, the position on the third coordinate is set on the first coordinate. The signal processing system according to claim 4, wherein the signal processing system is converted into a position.   The signal processing system according to claim 4, wherein the arithmetic unit synthesizes an image preliminarily defined by the light source group with an image acquired by the imaging unit.   When the arithmetic means is in a state of receiving only the optical communication signal of the second light source group from the state of receiving the optical communication signal from the first light source group and the second light source group, the coordinates 5. The signal processing system according to claim 4, wherein an inter-conversion parameter is stored in the storage means.   5. The signal processing system according to claim 4, wherein the inter-coordinate conversion parameter is an element value of a rotation conversion matrix and a parallel movement vector between the first coordinate and the second coordinate.   The signal processing system according to claim 2, wherein the light source group is arranged at an interval at which the receiving unit can receive optical communication signals from at least two light source groups.   The signal processing system according to claim 2, wherein the light source group includes four light sources arranged in a square shape at a predetermined interval.   The signal processing system according to claim 2, wherein the light source transmits an optical communication signal including coordinate information of the light source. Receiving means for receiving the optical communication signal; and imaging means for acquiring an image of the light source group;
The signal processing system according to claim 2, wherein an image receiving area of the optical communication signal in the receiving unit is larger than an imaging area in the imaging unit. An input step of inputting identification information and pixel position of each light source of a light source group in which the light sources are arranged in a predetermined positional relationship;
Coordinates defined by the first light source group based on the identification information and the pixel position of each light source when the identification information and the pixel position of each light source are input from the first light source group and the second light source group. A first self-coordinate conversion parameter for converting the upper position into a position on a coordinate specified by itself and a position on the coordinate specified by the second light source group into a position on the coordinate specified by the second light source group; A self-coordinate parameter calculating step of calculating two self-coordinate conversion parameters;
Coordinates for calculating an inter-coordinate conversion parameter for converting a position on the second coordinate to a position on the first coordinate based on the first self-coordinate conversion parameter and the second self-coordinate conversion parameter A signal processing method comprising: an inter-conversion parameter calculation step. An input step of inputting identification information and pixel position of each light source of a light source group in which the light sources are arranged in a predetermined positional relationship;
Coordinates defined by the first light source group based on the identification information and the pixel position of each light source when the identification information and the pixel position of each light source are input from the first light source group and the second light source group. A first self-coordinate conversion parameter for converting the upper position into a position on a coordinate specified by itself and a position on the coordinate specified by the second light source group into a position on the coordinate specified by the second light source group; A self-coordinate parameter calculating step of calculating two self-coordinate conversion parameters;
Coordinates for calculating an inter-coordinate conversion parameter for converting a position on the second coordinate to a position on the first coordinate based on the first self-coordinate conversion parameter and the second self-coordinate conversion parameter A program that causes a computer to execute an inter-conversion parameter calculation step.

Images