JP2006033329A - Optical marker system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an optical marker system which can presume the position and the attitude of a plurality of moving imaging means without using a special imaging means. <P>SOLUTION: LED markers 21-2m are mounted at predetermined positions, and transmit identification information and time information which are assigned single-mindedly to a self by changing the luminescence state. At this point, cameras 31-3n photograph the image including the luminance points of the LED markers 21-2m, and detect the flashing states of the LED markers 21-2m from the images photographed by candidate point extracting units 41-4n and marker tracing units 51-5n. An information detector 61 extract the identification information and the time information of the LED markers 21-2m, detect the positions of the LED markers 21-2m on the image based on the extracted identification information and time information; and presumes the position and the attitude of the cameras 31-3n based on the identification information, the time information, and the position of the LED markers 21-2m detected by a camera position attitude presuming unit 71. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an optical marker system that transmits time information and identification information.

  In recent years, with the spread of video cameras, camera-equipped mobile phones, and the like, opportunities for taking pictures have increased. For example, in an event such as a concert or an athletic meet, videos from various viewpoints by a large number of cameras can be obtained. In order to store and process these videos and extract a specific video sequence or create a new image, the marker is recorded while shooting the scene video by shooting the scene including a predetermined marker. Various proposals have been made to acquire information to be transmitted.

For example, the ID (identification information) of the marker is acquired while capturing an image using a high-speed image sensor (see Non-Patent Document 1), or the marker arranged in a square is captured using a normal visible light camera. Searching for the shape of the marker arrangement is performed (see Non-Patent Document 2).
Nobuyuki Matsushita et al., “Id cam: Smart camera that can acquire scene and id at the same time”, linguistic theory, Vol. 43, no. 12, 2001, pp. 3664-pp. 3674 Tsuyoshi Aoki, “Infrared tags read by cameras and their applications”, Interactive System and Software VIII, 2000, pp. 131-pp. 136

  However, in the former method, it is necessary to use a dedicated image sensor, and it cannot be applied to an image shot with a general camera. In the latter method, since the marker arrangement shape is detected, the number of markers that can be arranged in the scene is limited, and the method cannot be applied when the position and orientation of the camera change.

  An object of the present invention is to provide an optical marker system that can synchronize between a plurality of independently photographed images and can calculate corresponding points between the plurality of images.

  Another object of the present invention is to provide an optical marker system that can estimate the positions and orientations of a plurality of moving imaging means without using special imaging means.

  The optical marker system according to the present invention is attached to a predetermined position and transmits identification information uniquely assigned to itself by changing a light emission state and time information for specifying a shooting time. A plurality of light emitting means, a plurality of photographing means for photographing an image including a light emitting point of the light emitting means, a light emission state of the light emitting means is detected from an image photographed by the photographing means, and time information and the light emitting means Identification information and detection means for detecting the position on the image.

  In the optical marker system according to the present invention, a plurality of light emitting means attached at predetermined positions change the light emission state, thereby identifying identification information and photographing time assigned uniquely to itself. The time information for specifying is transmitted, the plurality of photographing means take images including the light emitting point of the light emitting means, detect the light emission state of the light emitting means from each image taken by the photographing means, and the time information and light emission The identification information of the means and the position on the image are detected.

  In this way, it is possible to obtain identification information of each light emitting means observed in an image photographed by each photographing means, and to specify the photographing time of the image photographed by each photographing means. This makes it possible to calculate the time lag between the images taken independently and to obtain a pseudo-synchronization, and further to calculate the corresponding points between multiple images using the identification information of each light emitting means, A wide-field moving image is generated by connecting a plurality of images, a relative three-dimensional position and posture between photographing means are estimated, and a three-dimensional position of a light emitting means having each identification information is measured. Thus, the three-dimensional absolute position and orientation of the photographing means can be estimated from the projection position on the image of the light emitting means.

  It is preferable to further include estimation means for estimating the position and orientation of the photographing means based on the identification information, time information and position of the light emitting means detected by the detection means.

  In this case, the position and orientation of the photographing means are estimated based on the detected identification information, time information, and position of the light emitting means. Here, since each light emitting means is attached at a predetermined position and the three-dimensional position of each light emitting means is known, the three-dimensional position and the two-dimensional position of the light emitting means on the image at each photographing time are determined. The position and orientation of each imaging means can be estimated sequentially. As a result, it is possible to estimate the positions and orientations of a plurality of moving imaging means without using special imaging means.

  The plurality of light emitting units transmit time information for specifying the same time, and the detecting unit displays time information in a section in which the light emission states of the plurality of light emitting units are in a pattern representing the same time on the time axis. It is preferable to extract.

  In this case, since the section in which the light emission states of the plurality of light emitting means are patterns representing the same time on the time axis is the transmission period of the time information, the time information can be accurately extracted separately from the identification information.

  The plurality of light emitting means transmit time information for specifying the same time, and the detecting means detects a light spot that matches a predetermined pattern from an image photographed by the photographing means. While tracking as a candidate point, a candidate point whose light emission pattern becomes a pattern representing the same time on the time axis is detected as a light point of the light emitting means, and identification information and time information of the detected light point are extracted. It is preferable.

  In this case, while tracking a light spot that matches a predetermined pattern from the photographed image as a light spot candidate point of the light emitting means, the light emitting state emits a candidate point that becomes a pattern representing the same time on the time axis. Since the light spot of the light emitting means is detected, the light spot of the light emitting means can be detected with high accuracy without erroneously detecting a light spot other than the light emitting means. Time information can be accurately extracted.

  The plurality of light emitting means transmit identification information and time information so that a lighting probability falls within a predetermined range, and the detecting means is a light that matches a predetermined pattern from an image photographed by the photographing means. While tracking points as light spot candidate points of the light emitting means, candidate points whose lighting probability falls within a predetermined range are detected as light spots of the light emitting means, and identification information and time information of the detected light spots are extracted It is preferable to do.

  In this case, a candidate point whose lighting probability falls within a predetermined range is detected as a light spot of the light emitting means while tracking a light spot that matches a predetermined pattern from the photographed image as a candidate point of the light spot of the light emitting means. Therefore, the light spot of the light emitting means can be detected with high accuracy without erroneously detecting the light spot other than the light emitting means, and the identification information and time information are accurately extracted from the detected light spot of the light emitting means. can do.

  Preferably, the plurality of light emitting units transmit identification information and time information by turning on or off infrared light, and the plurality of photographing units capture images using infrared light and visible light. .

  In this case, since identification information and time information can be transmitted using infrared light that is not directly visible to humans, the identification information and time information can be transmitted without causing visual interference with human behavior in the scene. It can be transmitted reliably.

  According to the present invention, it is possible to obtain identification information of each light emitting means observed in an image photographed by each photographing means, and to specify the photographing time of the image photographed by each photographing means. This makes it possible to calculate the time lag between the images taken independently and to obtain a pseudo-synchronization, and further to calculate the corresponding points between multiple images using the identification information of each light emitting means, A wide-field moving image is generated by connecting a plurality of images, a relative three-dimensional position and posture between photographing means are estimated, and a three-dimensional position of a light emitting means having each identification information is measured. Thus, the three-dimensional absolute position and orientation of the photographing means can be estimated from the projection position on the image of the light emitting means.

  According to the present invention, by photographing a plurality of light emitting means with the photographing means, identification information of each light emitting means can be obtained together with the photographing time information from the light emission pattern of the light emitting means recorded in the photographed image. That is, when a plurality of light emitting units transmit common time information, the time information can be more stably extracted using the redundancy. Further, from the identification information shown as the light emission pattern of the light emitting means, it is possible to uniquely identify which of the light emitting means each of the light emitting means recorded in the captured image is.

  The above basic features of the present invention enable a wide range of applications. Here, some of them will be exemplified first. One application mode uses the time information and the identification information directly. For example, each video is obtained for a plurality of video sequences obtained by independently photographing the same scene with a plurality of photographing means simultaneously. By using the position of each light emitting means recorded in the row, the corresponding points between the plurality of videos can be calculated, and the plurality of video rows can be connected to generate a wide-field moving image. Time information is used for pseudo-synchronization between videos, and identification information is used for specifying the corresponding light emitting means in a plurality of video sequences.

  Another application mode is to estimate the relative positional relationship between a plurality of photographing means using time information and identification information. For example, in each captured image that is pseudo-synchronized with time information. By using the position of each light emitting means in the image, corresponding points between a plurality of images are calculated, and the geometric position (relative position and orientation) between the imaging means is obtained from the calculated positional relationship of the corresponding points. be able to.

  As another application mode, the three-dimensional position of each light emitting means is used in addition to the time information and the identification information. For example, the projection of the light emitting means on the image by measuring the three-dimensional position of each light emitting means in advance. The three-dimensional absolute position and posture of the photographing means can be estimated from the position.

  Below, an implementation example is shown about the application form which estimates the relative position and attitude | position of a several imaging means using time information and identification information among the said application forms. In addition, as a method for calculating the relative position and orientation from a plurality of corresponding points between images, a method using epipolar geometry, a method using plane projective transformation, and the like are known, but here a target three-dimensional point (light emitting means) The method by plane projective transformation that makes it possible to estimate the relative position and orientation relatively easily using the constraint condition that) is on the same plane will be described.

  Hereinafter, an optical marker system according to an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing a configuration of an optical marker system according to an embodiment of the present invention.

  The optical marker system shown in FIG. 1 includes a lighting control device 11, m (m is an integer of 2 or more) LED markers 21 to 2m, n (n is an integer of 2 or more) cameras 31 to 3n, n Candidate point extraction units 41 to 4n, n marker tracking units 51 to 5n, an information detection unit 61, a camera position and orientation estimation unit 71, and an image synthesis unit 81 are provided.

  The LED markers 21 to 2m are configured by LEDs (Light Emitting Diodes) that emit infrared light, a drive circuit, and the like, and are attached to a predetermined position in a shooting scene, for example, on the same plane in the present embodiment. The three-dimensional positions of the markers 21 to 2m are stored in the camera position / orientation estimation unit 71 in advance.

  The lighting control device 11 is composed of a computer or the like that executes a predetermined blinking control program, and has identification information uniquely assigned to each of the LED markers 21 to 2m and all the LED markers 21 to 2m. The blinking states of the LED markers 21 to 2m are individually controlled so as to transmit the same time information. The configuration of the lighting control device is not particularly limited to the above example, and part or all of the lighting control device may be configured by a dedicated hardware circuit, and time information is set at the same timing using a predetermined reference clock or the like. In the case where the lighting control device 11 can be sent out, the lighting control device 11 may be individually provided in the LED markers 21 to 2 m and the lighting control device 11 may be omitted.

  The light emitted from the LED markers 21 to 2m by the blinking control of the lighting control device 11 is preferably infrared light. In this case, since identification information and time information can be transmitted using infrared light that is not directly visible to humans, the identification information and time information can be transmitted without causing visual interference with human behavior in the scene. It can be transmitted reliably. Further, light in a shorter wavelength region (wavelength region closer to visible light) in the infrared region is more preferable, and infrared light having a wavelength of 890 nm is used in this embodiment. In this case, since a normal visible light video camera has sufficient detection sensitivity in the wavelength range, the identification information and time information can be accurately obtained while shooting a scene using the normal visible light video camera. Can be detected.

  The light emitted from the LED marker is not particularly limited to the above example, and light in the visible light range may be used. In this case, the light emitted from the LED marker can be photographed using a wavelength range in which a normal visible light video camera has sufficient sensitivity, and identification information can be obtained using a normal visible light video camera. In addition, time information can be detected with high accuracy.

  Here, a blinking pattern of the LED markers 21 to 2m for transmitting identification information and time information will be described. FIG. 2 is a diagram illustrating an example of a blinking pattern of identification information and time information. The blinking frequency of the LED markers 21 to 2m and the frame rate of the cameras 31 to 3n coincide with each other, and the identification information and time information are coded by a blinking pattern that is continuous in units of 1 second (30 frames). That is, the first 10 bits are time information, and the latter 20 bits are identification information.

  As identification information, 8 bits, that is, 256 types of IDs are assumed, they are redundantly encoded with 20 bits, and different identification information is transmitted from each of the LED markers 21 to 2m. As the time information, the current time is converted into a hexadecimal number, one digit (4 bits) is redundantly encoded with 10 bits, and the same time information is transmitted from all the LED markers 21 to 2m.

  In the above encoding, the lighting probabilities p of the LED markers 21 to 2m are in a certain range in order to distinguish the light spots (LED emission points) of the LED markers 21 to 2m from other light spots (light spots other than the markers). The blinking pattern is set to be within (pL <p <pH). Further, in order to facilitate marker tracking described later, a blinking pattern is set so as not to turn off continuously for k frames (for example, k = 3) or more. Under these conditions, the identification information and the time information are redundantly encoded so that the Hamming distance is maximized for both the identification information and the time information. The configuration of the identification information and the time information is not particularly limited to the above example, and the identification information and the time information are transmitted by changing the lighting time. Various changes such as transmitting time information or transmitting identification information and time information by a color change using light having different color components are possible.

  Referring again to FIG. 1, the cameras 31 to 3n are composed of a normal visible light video camera (for example, a CCD camera ELMO CN-42H manufactured by Nortech Systems) having sufficient sensitivity to infrared light. Is done. The cameras 31 to 3n are individually operated by different users and take images including any of the LED markers 21 to 2m. The configuration of the camera is not particularly limited to the above example, and can be variously changed. For example, when the LED marker emits visible light, it has sufficient sensitivity only for visible light. Even if a camera is used, or a visible light camera and an infrared light camera are integrally configured to shoot a scene with a visible light camera, an infrared light camera may shoot an LED marker. Good.

  The candidate point extraction units 41 to 4n, the marker tracking units 51 to 5n, the information detection unit 61, the camera position and orientation estimation unit 71, and the image synthesis unit 81 are configured by a computer or the like that executes a position estimation process and the like described later. The extraction units 41 to 4n and the marker tracking units 51 to 5n are provided for the cameras 31 to 3n, and the image data of the cameras 31 to 3n are processed in parallel.

  In addition, the configuration of each block such as the candidate point extraction unit is not particularly limited to the above example, part or all may be configured from a dedicated hardware circuit, and if the processing speed is sufficiently high, Various changes such as changing the parallel processing by the candidate point extraction unit and the marker tracking unit to sequential processing are possible. In addition, a storage device for storing video data captured by the cameras 31 to 3n is added, and the processing after the candidate point extraction units 41 to 4n is performed offline using the image data stored in the storage device. The video data captured by the cameras 31 to 3n is transmitted to a predetermined server via a network such as the Internet, and the processing after the candidate point extraction units 41 to 4n is performed on the server online or offline. Also good.

  The candidate point extraction unit 41 selects light spots that match a predetermined pattern from an image sequence that is captured by the camera 31 and includes a scene including the LED markers 21 to 2m, as candidate points for the light spots of the LED markers 21 to 2m. The marker tracking unit 51 tracks the extracted candidate points. Other candidate point extraction units 42 to 4n and marker tracking units 52 to 5n operate in the same manner as described above.

  Specifically, since the lit LED markers 21 to 2m are sufficiently bright in the input image, the candidate point extraction units 41 to 4n use a predetermined template in the luminance component Vt of the image It observed at time t. The two-dimensional position (x, y) at which the score calculated by the following formula becomes a certain value or more is extracted as a candidate point of the light spot of the LED markers 21 to 2m. Here, ε is a threshold value of the pixel value that is determined to be the light spot of the LED markers 21 to 2m.

  FIG. 3 is a diagram illustrating an example of a template used in the candidate point extraction units 41 to 4n illustrated in FIG. In the candidate point extraction units 41 to 4n, for example, the candidate points of the light spots of the LED markers 21 to 2m are extracted using the 5 × 5 template or the 7 × 7 template shown in FIGS. can do.

  Referring to FIG. 1 again, the marker tracking units 51 to 5n extract the blinking patterns of the LED markers 21 to 2m by tracking the light spot candidate points of the LED markers 21 to 2m in time series. First, it is assumed that the movement of the light spot of the marker p (LED markers 21 to 2m) on the image can be approximated by a constant linear motion within a short time (εt), and the marker tracking units 51 to 5n Is used to estimate the two-dimensional observation position xp (t) and velocity vp (t) on the image at time t of the marker p being tracked. For example, if the estimated values of position and velocity at time t-Δt are xp (t−Δt) and vp (t−Δt), respectively, the estimated value at time t is xp (t) = xp (t−Δt) + vp (T−Δt) · Δt, vp (t) = (xp (t−Δt) −xp (t−2Δt)) / Δt.

  Next, when the marker tracking units 51 to 5n observe a point of x (t + Δt) −xp (t) + Δvp (t) <εx (where εx is a predetermined threshold) in the next frame (time t + Δt). The candidate point x (t + Δt) is continuously tracked as an observation of the marker p, and when the candidate point does not exist, it is determined that the marker p is not lit.

  In addition, since the LED markers 21 to 2m are not continuously turned off for k frames or more, the marker tracking units 51 to 5n fail to track the LED markers 21 to 2m when the lighting state is not observed in the period of 2k frames or more. It is determined that the LED markers 21 to 2m are out of the image, and the tracking is finished.

  For candidate points that are not associated with the LED markers 21 to 2m during tracking, the marker tracking units 51 to 5n may start from unsupported candidate points in an image captured during the period from time t-εt to time t. Search is made for candidate point sequences that follow the fast linear motion, and when they are found, tracking is started as observation points by the same LED marker. This process is executed for all light spots, and a full search is performed.

  The information detection unit 61 extracts the identification information and time information of the LED markers 21 to 2m from the blinking state of the light spots of the LED markers 21 to 2m tracked by the marker tracking units 51 to 5n. FIG. 4 is a diagram illustrating an example of a blinking pattern string of the LED markers 21 to 2m illustrated in FIG. The clock shown in FIG. 4 is a reference clock for blinking the LED markers 21 to 2m. As shown in FIG. 4, the hatched portion of the blinking pattern sequence of the LED markers 21 to 2m (markers 1 to m in the figure) becomes the same pattern, this section becomes the time information section, and the adjacent time information sections are identified. This is an identification information section in which information is transmitted. The blinking pattern is not particularly limited to this example, and various changes can be made. For example, in order to give some redundancy to the code, a part expressed by 1 bit (0, 1) is expressed by 2 bits (00, 01, 10, 11), and “01” and “10” are original expressions. The original code “101” is “000100”, “000111”, “001000”, “001011” by matching “00” of “0” and “00” and “11” with “1” of the original expression. ”,“ 110100 ”,“ 110111 ”,“ 111000 ”, and“ 1111011 ”.

  The information detection unit 61 extracts a section indicating the same blinking pattern from the blinking pattern sequence of the plurality of LED markers 21 to 2m obtained from the cameras 31 to 3n as a time information section, and decodes information in the time information section. To extract time information. This time information is the shooting time of each frame. Moreover, the information detection part 61 extracts between the adjacent time information areas as an identification information area, decodes the information in this identification information area, and extracts the identification information of each LED marker 21-2m.

  Furthermore, the information detection part 61 detects the projection position of the LED markers 21-2m on the image image | photographed with each camera 31-3n for every imaging | photography time using the extracted identification information and time information. At this time, the information detection unit 61 checks the correlation between the lighting probability p of each light spot and the blinking pattern with other light spots in the time information section, and the lighting probability p is out of a certain range (p ≦ pL, pH ≦ p ) And a light spot whose flashing pattern does not match in the time information section is excluded, and only the LED markers 21 to 2m are extracted.

  The camera position / orientation estimation unit 71 uses the time information detected by the information detection unit 61 to perform time alignment between a plurality of images captured by the cameras 31 to 3n, and obtains the identification information detected by the information detection unit 61. The LED markers 21 to 2m are associated between images at the same shooting time. The camera position / orientation estimation unit 71 stores the three-dimensional positions of the LED markers 21 to 2m in advance in association with the identification information of the LED markers 21 to 2m, and detects the LED markers 21 to 21 detected by the information detection unit 61. The 3D positions of the LED markers 21 to 2m are read from the identification information of 2m, and the positions of the cameras 31 to 3n based on the read 3D position and the 2D positions of the LED markers 21 to 2m detected by the information detection unit 61 And the posture are estimated.

  In addition, as a process for estimating the positions and orientations of the cameras 31 to 3n by the camera position / orientation estimation unit 71, a known process can be used, for example, 2 using a marker point sequence observed between two cameras. The fundamental matrix between two cameras (QTLuong and ODFaugeras. The Fundamental Matrix: theory, algorithms and stability analysis. International Journal of Computer Vision, Vol.17, No.1, pp.43 -75,1995), and calculates the plane projection (Homography) between two cameras using a marker sequence arranged on the same plane (J. Weng, N. Ahuja and TSHuang. Motion and Structure) From Point Correspondences with Error Estimation: Planar Surfaces. See IEEE trans. On Signal Processing, Vol. 38, No. 12, pp. 2691-2717, 1991).

  The image composition unit 81 composes a plurality of images taken by the cameras 31 to 3n based on the positions and orientations of the cameras 31 to 3n estimated by the camera position and orientation estimation unit 71, and creates a new image. As the image composition processing by the image composition unit 81, a known process can be used. For example, a virtual viewpoint video is created using a fundamental matrix between cameras (Hideo Saito, Takeo Kanade, It is possible to use a technique such as virtualization of dynamic events by a large number of cameras, Information Processing Research Report, CVIM 119-016, pp. 117-124, 1999).

  In the present embodiment, a method using the above-mentioned planar projection is adopted, and the LED markers 21 to 2m are arranged on the same plane. In the present embodiment, the lighting control device 11 and the LED markers 21 to 2m correspond to an example of the light emitting unit, the cameras 31 to 3n correspond to an example of the photographing unit, the candidate point extracting units 41 to 4n, and the marker tracking. The units 51 to 5n and the information detection unit 61 correspond to an example of a detection unit, and the camera position / posture estimation unit 71 corresponds to an example of an estimation unit.

  Next, position estimation processing by the optical marker system shown in FIG. 1 will be described. FIG. 5 is a flowchart for explaining position estimation processing by the optical marker system shown in FIG.

  First, with the LED markers 21 to 2m transmitting identification information and time information respectively, the cameras 31 to 3n take images including any of the LED markers 21 to 2m, and the candidate point extraction units 41 to 4n Then, a light spot that matches a predetermined pattern is extracted from the captured image sequence as a light spot candidate point of the LED markers 21 to 2m (step S11). And something different.

  Next, the marker tracking units 51 to 5n track the candidate points of the light spots of the LED markers 21 to 2m in time series on the assumption that the light spots of the LED markers 21 to 2m perform a uniform linear motion (step S12). ), The light spot of the LED marker 21-2m is detected using the feature of the blinking pattern of the light spot of the LED marker 21-2m (step S13).

  Next, the information detection unit 61 extracts the identification information and time information of the LED markers 21 to 2m from the blinking patterns of the light spots of the LED markers 21 to 2m tracked by the marker tracking units 51 to 5n, and extracts the extracted identifications. Using the information and the time information, the two-dimensional positions of the LED markers 21 to 2m are detected at each photographing time (step S14).

  Next, the camera position / orientation estimation unit 71 performs time alignment between a plurality of images taken by the cameras 31 to 3n using the time information detected by the information detection unit 61, and is detected by the information detection unit 61. Using the identification information, the LED markers 21 to 2m are associated with each other at the same shooting time (step S15). Next, the camera position / orientation estimation unit 71 reads the three-dimensional positions of the LED markers 21 to 2 m from the identification information of the LED markers 21 to 2 m detected by the information detection unit 61, and reads the read three-dimensional positions and the information detection unit 61. The positions and postures of the cameras 31 to 3n are estimated based on the two-dimensional positions of the LED markers 21 to 2m detected by the above, and then the processing after Step S11 is continued, and the positions and postures of the cameras 31 to 3n are changed. Estimated sequentially.

  Through the above processing, in the present embodiment, the cameras 31 to 3n capture images including the light emission points of the LED markers 21 to 2m that transmit the identification information and time information, and the LEDs are captured from the images captured by the cameras 31 to 3n. The blinking patterns of the markers 21 to 2m are detected to extract the identification information and time information of the LED markers 21 to 2m, and the two-dimensional positions of the cameras 31 to 3n are taken for each photographing time based on the extracted identification information and time information. Since the three-dimensional positions and orientations of the cameras 31 to 3n are sequentially estimated based on the detected identification information, time information, and position of the detected light emitting means, a plurality of cameras that move without using special imaging means The positions and orientations of 31 to 3n can be estimated.

  Next, the estimation accuracy of the position and orientation of the camera by the position estimation process will be described with a specific example. FIG. 6 is a schematic diagram illustrating an arrangement example of LED markers. As shown in FIG. 6, after arranging six LED markers M1 to M6 around a rectangular area of 300 mm × 500 mm, a video sequence of a scene including the six LED markers M1 to M6 is photographed, and the photographed image is displayed. On the other hand, the position estimation process described above was performed.

  FIG. 7 is a diagram illustrating the trajectories of the six LED markers M1 to M6 extracted from the captured image sequence, and FIG. 8 is a diagram illustrating the blinking pattern of the six LED markers M1 to M6. As shown in FIG. 7, the trajectories of the six LED markers M1 to M6 were obtained. Further, a portion surrounded by a rectangle in FIG. 8 is a section extracted as a time information section having a signal pattern common to the six LED markers M1 to M6, and time information is extracted from this time information section. The section between the time information sections is the identification information section, and the identification information of each LED marker M1 to M6 is extracted from this identification information section.

  Next, while fixing one camera C1 of the two cameras C1 and C2 and moving and rotating the other camera C2, the video sequence of the scene including the six LED markers M1 to M6 is shot and shot. The position estimation process described above was performed on the image. FIG. 9 is a schematic diagram showing the positional relationship between the two cameras C1 and C2. As shown in FIG. 9, in the initial state, two cameras C1 and C2 are set to the same position and posture (excluding height) with respect to the marker MK (LED markers M1 to M6), and then the camera C2 is set. It is rotated in the horizontal plane while moving in the X-axis direction, the internal parameters of the cameras C1 and C2 are known, and the three-dimensional positions of the cameras C1 and C2 and the two-dimensional positions of the LED markers M1 to M6 in the image are used. Posture was estimated.

  FIG. 10 is a diagram showing estimation results of the relative position and relative posture of the camera C2 with respect to the camera C1 obtained at this time. As shown in FIG. 10, the X axis that is the moving direction of the camera C2 and the azimuth angle (Azimuth) that is the rotation direction have changed, but the other Y axis and Z axis, the elevation angle (Elevation), and the rotation angle (Rotation). Are almost constant, and it was confirmed that the three-dimensional motions of the cameras C1 and C2 can be continuously tracked using the information of the LED markers M1 to M6.

  Note that the position and orientation of the camera obtained by the present invention can be used for various purposes, for example, integration of captured images by a large number of people at an event venue, synthesis of live-action images and computer graphics in studio shooting, etc. It can be applied to various video processing.

It is a block diagram which shows the structure of the optical marker system by one embodiment of this invention. It is a figure which shows an example of the blink pattern of identification information and time information. It is a figure which shows an example of the template used in the candidate point extraction part shown in FIG. It is a figure which shows an example of the blink pattern row | line | column of the LED marker shown in FIG. It is a flowchart for demonstrating the position estimation process by the optical marker system shown in FIG. It is a schematic diagram which shows the example of arrangement | positioning of an LED marker. It is a figure which shows the locus | trajectory of six LED markers extracted from the picked-up image sequence. It is a figure which shows the blink pattern of six LED markers. It is a schematic diagram which shows the positional relationship of two cameras. It is a figure which shows the estimation result of the relative position of two cameras shown in FIG. 9, and a relative attitude | position.

Explanation of symbols

DESCRIPTION OF SYMBOLS 11 Lighting control apparatus 21-2m LED marker 31-3n Camera 41-4n Candidate point extraction part 51-5n Marker tracking part 61 Information detection part 71 Camera position and orientation estimation part 81 Image composition part

Claims (6)

An optical marker system that transmits time information and identification information,
A plurality of light emitting means attached to a predetermined position and transmitting identification information uniquely assigned to itself by changing a light emission state and time information for specifying a photographing time;
A plurality of photographing means for photographing an image including a light emitting point of the light emitting means;
An optical marker system comprising: time information, identification information of the light emitting means, and detection position for detecting a position on the image by detecting a light emission state of the light emitting means from an image photographed by the photographing means.   An optical marker system, further comprising: an estimation unit that estimates the position and orientation of the photographing unit based on the identification information, time information, and position of the light emitting unit detected by the detection unit. The plurality of light emitting means transmit time information for specifying the same time,
3. The optical marker system according to claim 1, wherein the detection unit extracts time information in a section in which light emission states of the plurality of light emission units are patterns representing the same time on a time axis. The plurality of light emitting means transmit time information for specifying the same time,
The detection means traces a light spot that matches a predetermined pattern from an image photographed by the photographing means as a light spot candidate point of the light emission means, and the light emission state represents the same time on the time axis. The optical marker system according to claim 1 or 2, wherein a candidate point to be a pattern is detected as a light spot of the light emitting means, and identification information and time information of the detected light spot are extracted. The plurality of light emitting means transmit identification information and time information so that a lighting probability falls within a predetermined range,
The detection means tracks candidate points whose lighting probability falls within a predetermined range while tracking light spots that match a predetermined pattern from the image photographed by the photographing means as light spot candidate points of the light emitting means. 5. The optical marker system according to claim 1, wherein the optical marker system is detected as a light spot of the light emitting means, and identification information and time information of the detected light spot are extracted. The plurality of light emitting means transmit identification information and time information by turning on or off infrared light,
The optical marker system according to claim 1, wherein the plurality of imaging units capture an image using infrared light and visible light.

Images