JP2012128557A - Subject position measuring apparatus, display system and subject position measuring method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To stably determine a position of a portion of a subject such as the head of a person to be photographed through a simple system when determining the position of the portion in a subject position measuring apparatus and subject position measuring method.SOLUTION: A portion of a subject is regarded as a sphere and a position of that portion is determined. Specifically, an apparatus extracts a contour image of the subject in each of images of that portion of the subject photographed using a camera in at least two directions in that image. Further, information on a contact position where the contour image and a search straight line extending in a first direction in the image are brought into contact is determined, thereby determining a first virtual straight line extending from the camera and being in contact with that portion of the subject for each of the contour images in a photographing space. Moreover, in the photographing space, a position of an intersection of second virtual straight lines resulting from moving the first virtual straight line in a direction corresponding to the first direction in parallel by a predetermined distance respectively, or a position of a point where the second virtual straight lines become closest to each other is determined as a center position of that portion.

Description

  The present invention relates to a subject position measuring device that obtains the position of a part of the photographed subject by regarding a part of the photographed subject as a spherical body, a display system using the subject position measuring device, and a subject position measuring method.

  Conventionally, various methods for tracking the position of the head of a subject person on a moving image taken by a camera or the like have been proposed. Such tracking of the position of the head of the subject person can be effectively used in a motion parallax stereoscopic display so that the appearance of the virtual object changes according to the motion parallax of the viewer. Motion parallax relates to the amount of change in the appearance of the depth of a virtual object that changes as the position of the viewer's viewpoint changes.

For example, a three-dimensional human head tracking method using a polygon model is known as a tracking method of a human head region. Specifically, in this method, a three-dimensional polygon model is projected on a captured two-dimensional image, and the color and depth information of the three-dimensional polygon model at that time are used, so that the subject person on the captured image is displayed. The three-dimensional position and the face direction are estimated.
In addition to automatically tracking the face of the subject person in real time based on the information of the images taken by the two cameras and measuring the movement of the face part, Also known is a system that extracts information from a captured image and tracks its characteristics.

Atsushi Ohba, Kazunori Terada, Akira Ito, “3D Human Head Tracking Using Polygon Model”, [online], Robotics and Mechatronics Lecture (ROBOMEC05), [November 19, 2010 search], Internet (URL) : Http://www.elf.info.gifu-u.ac.jp/terada/publications/robomec05O.pdf) face LAB (SeeingMachines product), [online], [Searched on November 19, 2010], Internet (URL: http://www.creact.co.jp/jpn/facelab/index_f.html)

  When the method for tracking the position of the head of a subject person on a moving image taken by a camera or the like as described above is used in a motion parallax stereoscopic display, the position coordinates in the space of the center position of the head of the subject person Based on the above, the viewpoint position of the subject person in the virtual space is determined, and the video of the virtual object is displayed in consideration of motion parallax. At this time, if the coordinate position that defines the center position of the head to be measured fluctuates, there may be a problem that the display image of the virtual object fluctuates even if the subject person does not move the head.

The 3D human head tracking method using the polygon model and the process of tracking the face of the subject person in the system are generally susceptible to subtle fluctuations in the illumination conditions of the subject person, and therefore the head of the measurement target. Even if the part is not moving, the coordinate position of the head may fluctuate.
In order to suppress the fluctuation, a low-pass filter may be used. However, when the head actually moves, the coordinate position of the head may not follow the moving head in real time.

  Accordingly, the present invention provides a subject position measuring apparatus that can stably determine the position of a part of the subject such as the head of the photographed person by a simple method different from the conventional method. It is another object of the present invention to provide a subject position measuring method and a display system using the subject position measuring apparatus.

One aspect of the present invention is a subject position measuring apparatus that obtains a position of a part of a photographing subject as a spherical body. The device is
A plurality of cameras that photograph a subject to be photographed from at least two directions;
A data processing unit that determines a position of the part of the photographed subject by regarding a part of the photographed subject as a spherical body from a plurality of images photographed by the camera.
The data processing unit
In each of the plurality of images, a contour extraction unit that extracts a contour image of a photographing subject in the image;
Contact position information in the image where the contour image and a search straight line for searching for the contour image in contact with the first image in the first direction come into contact is obtained. A virtual straight line extraction unit for obtaining a first virtual straight line extending from each of the contours of the photographic subject for each of the contour images using the contact point position information;
An intersection of the second virtual lines obtained by translating the first virtual line obtained for each of the contour images by a predetermined distance in a direction in the imaging space corresponding to the first direction, or the second virtual line A position calculation unit that obtains the position of the point at which the straight lines are closest to each other as the center position of the part of the shooting subject in the shooting space.

Another embodiment of the present invention is a display system that displays video. The system
The subject position measuring device;
A data processing device including a video generation unit that generates a video signal of a virtual object to be viewed from the shooting subject according to the center position of the portion of the shooting subject;
A screen display device for displaying an image of the virtual object on the screen using the generated video signal.

Furthermore, another aspect of the present invention is a subject position measuring method for obtaining a position of a part of a photographing subject as a spherical body. In this method,
By photographing a subject to be photographed from at least two directions using a camera, a plurality of images viewed from at least two directions are acquired,
In each of the plurality of images, extract a contour image of the photographic subject in the image,
Contact position information in the image where the contour image and a search straight line for searching for the contour image in contact with the first image in the first direction come into contact is obtained. A first imaginary straight line extending from each and contacting the part of the photographic subject is obtained for each of the contour images using the contact position information,
An intersection of the second virtual lines obtained by translating the first virtual line obtained for each of the contour images by a predetermined distance in a direction in the imaging space corresponding to the first direction, or the second virtual line The position of the point at which they are closest to each other is obtained as the center position of the part of the shooting subject in the shooting space.

  According to the subject position measuring apparatus and subject position measuring method of the above aspect, when obtaining the position of a part of the photographic subject, the position of the part can be obtained stably. For this reason, according to the display system using the subject position measuring device and the subject position measuring method, it is possible to prevent the display image of the virtual object from fluctuating and causing discomfort to the viewer.

It is a figure explaining the outline of the display system of this embodiment. (A) is a block block diagram of the display system shown in FIG. 1, and (b) is a functional block diagram of the display system shown in FIG. It is a block diagram which shows the structure of the data processing part of the to-be-photographed object position measuring apparatus of this embodiment. It is a figure explaining an example of the calculation of the contact position information of the outline image performed with the to-be-photographed object position measuring apparatus of this embodiment. It is a figure explaining an example of the position measurement of the head which the photographic subject position measuring device of this embodiment performs. It is a flowchart which shows an example of the flow of the to-be-photographed object position measuring method of this embodiment.

  The subject position measuring apparatus, display system, and subject position measuring method of the present invention will be described below.

(Display system)
FIG. 1 is a diagram illustrating an outline of a display system 10 according to the present embodiment. FIG. 2A is a block configuration diagram of the display system 10, and FIG. 2B is a functional block diagram of the display system 10.
The display system 10 is a system including a motion parallax stereoscopic display that changes the appearance of a virtual object to be displayed according to the motion parallax of the viewer V. For example, when the viewer V's viewpoint slightly moves to the upper left, the display system 10 displays the virtual object to be displayed according to the viewer V's viewpoint so that the viewer V sees the virtual object from the upper left. Corrects the depth appearance and shading and displays it.

The display system 10 includes a pair of cameras 12a and 12b, a data processing device 14, and a screen display device 16.
The pair of cameras 12 are fixed in space in order to photograph the viewers V who are viewing the display image from different directions.
The data processing device 14 generates a video signal of a virtual object to be displayed that can be seen by the viewer V according to the position of the head of the viewer V.
As shown in FIG. 2A, the data processing device 14 is formed by a computer having a CPU 15a, a ROM 15b, a RAM 15c, a storage unit 15d, an input / output unit 15e, and a bus 15f. The CPU 15a, ROM 15b, RAM 15c, storage unit 15d, and input / output unit 15e are connected to each other via a bus 15f. The input / output unit 15e is connected to cameras 12a and 12b, a screen display device 16, and an input operation system (not shown) including a mouse and a keyboard.

  The ROM 15b stores an OS and control software for the CPU 15a to execute each process when the computer is activated. The RAM 15c temporarily stores the results of various data processing described later. The storage unit 15d stores a program for executing data processing to be described later in the data processing device 14. In addition, the storage unit 15d stores the results of various data processing described later. Note that a program for executing data processing to be described later may be stored in the ROM 15b instead of the storage unit 15d. The CPU 15a controls and manages the operation of each part of the computer, and further controls and manages data processing to be described later.

  The data processing device 14 which is the computer reads out and executes the program stored in the storage unit 15d, thereby functionally forming the data processing unit 14a and the video generation unit 14b as shown in FIG. 2B. . That is, the data processing unit 14a and the video generation unit 14b are software modules, and are portions that are virtually formed when the CPU 15a executes the program.

The data processing unit 14a obtains the position of the viewer V's head from images from two directions taken by the cameras 12a and 12b, regarding the viewer V's head as a spherical body. The data processing unit 14a provides information on the position of the head of the viewer V to the video generation unit 14b.
The video generation unit 14b generates a video signal of a virtual object to be displayed that can be viewed by the viewer V according to the position of the head of the viewer V with respect to the input video signal of the virtual object. The video signal input to the video generation unit 14b includes, for example, the direction of the illumination light source irradiated in the virtual space, the light of the illumination light, in addition to the data represented by the three-dimensional coordinates of the virtual object in the virtual space. Information such as intensity and spectral distribution is also included. That is, the video generation unit 14b converts the three-dimensional data into two-dimensional data according to the position of the head of the viewer V using the input video signal, and performs a shadow process, thereby performing a two-dimensional virtual Generate an object video signal.
The screen display device 16 displays an image of the virtual object on the screen using the two-dimensional video signal generated by the data processing device 14.

  As will be described later, the data processing unit 14a can stably determine the position of the head when determining the position of the head of the viewer V, and even if the head actually moves, Can be obtained by following the coordinate position in real time. Therefore, since the screen display device 16 does not fluctuate the coordinate position of the head of the viewer V as in the conventional device, the displayed virtual object will not be fluctuated and displayed. The virtual object image changes in real time as the head moves.

(Subject position measurement device)
As shown in FIG. 2B, the subject position measuring device 20 of the present embodiment is a part of the data processing device 14 and is mounted on the display system 10, as indicated by a dotted line in the figure. And a pair of cameras 12a and 12b and a data processing unit 14a.
The cameras 12a and 12b shoot the viewer V, who is the subject to be photographed, from two different directions. The reason why the viewer V is photographed from two different directions is to determine the position of the head of the viewer V in the photographing space.
The data processing unit 14a obtains the position of the viewer V's head from the two images captured by the cameras 12a and 12b, regarding the head of the viewer V as the subject to be photographed as a spherical body.

FIG. 3 is a functional block diagram showing the configuration of the data processing unit 14a.
As described above, the data processing unit 14a is a software module that is functionally formed by the computer that is the data processing device 14 executing the program stored in the storage unit 15d. The data processing unit 14 a includes a contour extraction unit 24, a virtual straight line extraction unit 26, and a position calculation unit 28. That is, the contour extraction unit 24, the virtual straight line extraction unit 26, and the position calculation unit 28 are software modules formed by executing a program in the data processing device 14. A hardware module using a dedicated circuit or the like may be used as part of the contour extracting unit 24, the virtual straight line extracting unit 26, and the position calculating unit 28 instead of the software module.

  The contour extraction unit 24 extracts a contour image of the viewer V in each of the two images taken by the cameras 12a and 12b. The contour extracting unit 24 uses the background image when extracting the contour image of the viewer V. In this background image, an image obtained when the viewer V is not within the shooting range of the cameras 12a and 12b is taken in advance by the cameras 12a and 12b and stored in the RAM 15c or the storage unit 15d in advance. The contour extracting unit 24 obtains a difference between the image taken by the viewer V taken by the cameras 12a and 12b and the background image not taken by the viewer V, and binarizes the difference image. An outline image of the viewer V is extracted. By binarizing the difference image using a predetermined threshold, it is possible to clearly distinguish the contour image of the viewer V from the other images. The contour image of the viewer V is extracted for each image of the viewer V taken by the cameras 12a and 12b.

  Specifically, the contour extracting unit 24 obtains a difference image of the luminance data by obtaining a difference between the luminance data in each pixel in the image taken by the viewer V and the background image. In the difference image, the pixel value is greatly different between a portion where the viewer V is shown and a portion where the viewer V is not shown. Therefore, in the binarized image, a portion where the viewer V is shown is, for example, a white region, and a portion where the viewer V is not shown is, for example, a black region. In the following description, an example will be described in which a portion where the viewer V is shown is a white region, and a portion where the viewer V is not shown is a black region. The threshold used for binarization can be prevented from being erroneously extracted as a contour image by adjusting a part of the background image other than the viewer V by adjusting according to the illumination condition for photographing. The extracted contour image of the viewer V may be subjected to enlargement / reduction processing for enlarging the white region after reducing the white region of the contour image as noise reduction processing. The process of reducing the white area of the contour image is, for example, a process for forcibly setting a pixel of interest to a black pixel when the pixel of interest is in contact with a black pixel. The process of enlarging the white area of the contour image is, for example, a process of forcibly making the pixel of interest a white pixel when the pixel of interest is in contact with a white pixel.

The contour extraction unit 24 creates a difference image using the luminance data, but instead of the luminance data, the difference image of the hue data may be obtained using the difference of the hue data, or the luminance data and the hue data may be obtained. You may obtain | require the difference image using both. In this case, a common part between the contour image obtained from the difference image of the luminance data and the contour image obtained from the difference image of the hue data can be obtained as the contour image of the subject V.
In this embodiment, the contour image of the viewer V is extracted using the image of the presence or absence of the viewer V. As another method, for example, the position of the head of the viewer V is determined by the eyes, nose, mouth. Further, the contour image of the viewer V may be extracted by extracting using image feature amounts (color and shape) related to hair and the like.

  The virtual straight line extraction unit 26 extracts a contour image extracted from each of the two images of the viewer V taken by the cameras 12a and 12b, and a search straight line that extends in the horizontal direction in the difference image and searches for a region of the contour image. The contact position information in the difference image that touches is obtained. The virtual straight line extraction unit 26 obtains the contact position information as follows, for example.

FIG. 4 is a diagram illustrating an example of calculation of contact point position information of a contour image.
In FIG. 4, the contour image 30 is extracted from the binarized difference image 32. Further, the difference image 32 includes white regions W _{1 to} W ₄ other than the contour image 30 of the viewer V, which are formed by a change in illumination conditions or noise or the like with respect to the background image.
In this state, the virtual straight line extraction unit 26 moves in the horizontal direction to search the region of the contour image 30 while moving from the upper side to the lower side in FIG. 4 (in the vertical direction in the difference image 32). Move _s . Thereby, the virtual straight line extraction unit 26 obtains contact position information in the difference image 32 where the search straight line L _s contacts the contour image 30. Specifically, the virtual straight line extraction unit 26 repeatedly moves the search straight line L _s one pixel at a time in the vertical direction so that the search straight line L _s extending in the horizontal direction crosses the contour image 30. At this time, the virtual straight line extraction unit 26 uses the average value of the center position in the horizontal direction of the region where the search straight line L _s crosses the contour image 30 as a part of the contact position information indicating the coordinate position of the contact position (information in the horizontal direction). ). Further, the virtual straight line extraction unit 26 obtains the vertical position of the search straight line L _s with respect to this average value as a part of the contact position information (information in the vertical direction).
More specifically, the virtual straight line extraction unit 26 repeatedly moves the search straight line L _s one pixel at a time in the vertical direction, and the search straight line L _s is the same binarized data as that of the viewer V's head. When the value, that is, the number of repetitions that cross the same white area and the search straight line L _s continuously crosses the area reaches a predetermined number of times, the area is determined as the head area of the viewer V.
As shown in FIG. 4, the determination is made using the search straight line L _s under the above conditions when the white areas W _{1 to} W ₄ other than the contour image 30 of the viewer V exist, as shown in FIG. This is because the regions W _{1 to} W ₄ are not erroneously determined as the region of the head of the viewer V. In the difference image, the white areas W _{1 to} W ₄ have a very wide range in the same manner as the head area of the viewer V. Therefore, the virtual straight line extraction unit 26 uses, as a condition for determining the head, that the number of repetitions that the search straight line L _s continuously crosses the white area reaches a predetermined number. Therefore, although the predetermined number of times used for the determination is determined in advance, it is preferable to change depending on the shooting conditions such as the shooting magnification of the cameras 12a and 12b.

As described above, the virtual straight line extraction unit 26 obtains the average value of the center positions in the horizontal direction of the region where the search straight line L _s crosses the contour image 30 as information in the horizontal direction of the contact position information. This contact point position information is strictly different from the position of the first point of contact with the uppermost part of the contour image 30 while the search straight line L _s moves from top to bottom in FIG. 4 pixel by pixel. However, since the uppermost portion of the contour image 30 does not necessarily correspond to the actual contact of the head due to the influence of noise or the like, the virtual straight line extraction unit 26 is used to obtain more stable and accurate contact position information. Obtains the average value obtained by the above method as information in the horizontal direction of the contact point position information. Of course, if the difference image 32 does not contain noise, the virtual straight line extraction unit 26 determines the horizontal position of the point where the search straight line L _s first contacts the uppermost portion of the contour image 30 in the horizontal direction of the contact position information. You may decide as information in.

Further, in the difference image, there may be a region in which the width shown in FIG. 4 is wider than the actual head region because the white regions W _{1 to} W ₄ are connected to the head region. is there. Further, there may be a region where the horizontal width is shorter than the actual head region due to the influence of the background image used to create the difference image. In these cases, these areas are determined to be the areas of the contour image 30 that the search straight line L _s crosses, and the average value of the center position obtained as the contact position information becomes inaccurate. For this reason, the virtual straight line extraction unit 26 can perform the following processing. That is, the virtual straight line extracting unit 26, the search when the straight line L _s crosses the contour image 30 continuously for a predetermined number of times in the horizontal direction, the average values for the horizontal cross width of the region where search straight L _s crosses the contour image 30 And obtain the standard deviation. The virtual straight line extraction unit 26 further obtains the average value of the center position in the horizontal direction by limiting the horizontal crossing width within the allowable range obtained from the average value and the standard deviation. . The virtual straight line extraction unit 26 determines the obtained average value as information in the horizontal direction of the contact point position information. That is, the virtual straight line extraction unit 26 excludes a region where the horizontal transverse width is not included in the allowable range obtained from the average value and the standard deviation. When the average value is λ and the standard deviation is σ, the allowable range is λ−c · σ to λ + c · σ (c is a given constant in the range of 0.5 to 1.5) and To do. Or when removing the area | region where the width | variety of the horizontal direction expanded compared with the actual width | variety, the said allowable range shall be (lambda) + c * (sigma) or less.
In the present embodiment, the search line L _s extends in the horizontal direction in the difference image and moves one pixel at a time in the vertical direction, but the search line L _s extends in the direction inclined in the difference image. You may move in the direction orthogonal to the direction. Further, the search straight line L _s moves by a distance corresponding to one pixel, but may move by a distance corresponding to a plurality of pixels or by a preset distance.

  Further, the virtual straight line extraction unit 26 extends from the photographing center of each of the cameras 12a and 12b in the photographing space in which the viewer V is photographed using the obtained contact point position information (position information in the horizontal direction and the vertical direction). A first virtual straight line in contact with the head of the viewer V is obtained for each contour image 30. Since the points in the images taken by the cameras 12a and 12b are straight lines extending from the respective photographing centers of the cameras 12a and 12b when the taken images are not distorted, the points in the images are in a three-dimensional shooting space. Inside, it is converted to a straight line. For this reason, the virtual straight line extraction unit 26 can easily obtain the first virtual straight line using the contact point position information.

The virtual straight line extraction unit 26 performs calibration in order to obtain an accurate first virtual straight line in the photographing space when the first virtual straight line is obtained using the contact position information in the photographed image. Specifically, when the virtual straight line extraction unit 26 obtains the first virtual straight line from the contact position information, information on distortion aberration caused by the photographing lenses of the cameras 12a and 12b, photographing positions of the cameras 12a and 12b, photographing magnification, and the like. Make a correction that accurately reflects. More specifically, the virtual straight line extraction unit 26 obtains the first virtual straight line from the contact position information in the image by performing calibration using, for example, a well-known Zhang algorithm. Zhang's algorithm is http://research.microsoft.com/en-us/um/people/zhang/calib/ (searched on November 21, 2010)), http://research.microsoft.com/en- us / um / people / zhang / Papers / TR98-71.pdf (searched on November 21, 2010)).
In the Zhang algorithm, a matrix for correcting a pixel scale amount and an image distortion amount, a rotation of the shooting space, and a space between a two-dimensional position coordinate on the shot image and a three-dimensional position coordinate in the shooting space It is related beforehand using the translation parameter of translation. For example, the checker board is photographed from three or more different positions using the cameras 12a and 12b, a position coordinate sequence in the photographing space of the intersection of the checkers, and a two-dimensional position coordinate in each photographed image. This is done by identifying the matrix and the value of the conversion parameter using columns.
Therefore, the contact position information in the photographed image is calculated using the matrix and the conversion parameter values obtained by performing the calibration in advance, thereby calculating the relationship, so that the cameras 12a and 12b. It can be converted into an accurate straight line in the photographing space extending from each photographing center and in contact with the head of the viewer V, that is, a first virtual straight line.

  The position calculation unit 28 translates the first virtual line obtained for each of the contour images 30 of the viewer V by a predetermined distance in the direction in the imaging space corresponding to the horizontal direction in the difference image 32. Ask for. The position calculation unit 28 further uses the obtained position of the intersection of the second virtual lines or the position of the point where the second virtual lines are closest to each other as the center position of the head of the viewer V in the shooting space. Ask.

FIG. 5 is a diagram for explaining the position measurement of the head H of the viewer V performed by the subject position measurement apparatus 20.
As shown in FIG. 5, first virtual straight lines L ₁ and L ₂ in a shooting space extending from the shooting center of each of the cameras 12 a and 12 b and in contact with the head H of the viewer V are obtained by the virtual straight line extraction unit 26. . At this time, the contact points P ₁ and P _{2 at which} the first virtual straight lines L ₁ and L ₂ are in contact with the head H are at different positions because the orientations of the cameras 12a and 12b for photographing the viewer V are different. However, the center position of the head H is the same even if the directions in which the cameras 12a and 12b capture the viewer V are different. Therefore, in order to obtain the center position, the position calculation unit 28 obtains the position of the intersection of the second virtual lines or the position of the point where the second virtual lines are closest.

That is, the position calculation unit 28 converts a straight line obtained by translating the first virtual straight lines L ₁ and L ₂ by a predetermined distance in the direction in the imaging space corresponding to the horizontal direction in the difference image 32 into the second virtual straight lines L ₃ and L ₃ . determined as L _4. At this time, the predetermined distance is a radius R _h of the sphere when the head H of the viewer V is regarded as a spherical shape and approximated. If the size of the head H is a human, it is determined to be substantially constant, for example, 18 cm. Therefore, the value of the radius R _h of such a sphere is determined from the average size of the human head. .
Accordingly, since the second virtual straight lines L ₃ and L ₄ pass through the center position of the head H or the vicinity thereof, the position calculation unit 28 determines the position of the intersection P ₃ between the second virtual straight lines L ₃ and L ₄ , Alternatively, the position of the point at which the second virtual straight lines L ₃ and L ₄ are closest to each other is obtained as the center position of the head H. The coordinate information of the center position is provided to the video generation unit 14b of the display system 10 as described above.
Here, the position of the point at which the second virtual straight lines L ₃ and L ₄ are closest to each other is the center position on the line segment where the distance between the second virtual straight lines L ₃ and L ₄ is the smallest, or the above line segment. Is the position of the point on the line segment that divides.
The above is the description of the subject position measuring apparatus 20.

(Subject position measurement method)
FIG. 6 is a flowchart showing an example of the flow of the subject position measurement method performed by the subject position measurement 20 of the present embodiment.
First, for each of the cameras 12a and 12b, the matrix described above that associates between the two-dimensional position coordinates on the image photographed by the cameras 12a and 12b using the Zhang algorithm and the three-dimensional position coordinates in the photographing space. Get the value of each element and the value of conversion parameter. As described above, the matrix and the conversion parameter values are obtained by photographing the checkerboard from three or more different positions, and the position coordinate sequence in the photographing space of the intersection of the checkers and the two-dimensional in each photographed image. It is performed by identifying using the position coordinate sequence of.

  Next, the cameras 12a and 12b capture the viewer V, and the data processing unit 14a acquires an image (step S10). The photographed image data of the viewer V is stored in the RAM 15c or the storage unit 15d. The data processing unit 14a determines whether or not the storage of the image data from the cameras 12a and 12b has been completed (step S20). The data processing unit 14a waits until all the image data of the viewer V is stored in the RAM 15c or the storage unit 15d from the cameras 12a and 12b. When all the image data of the viewer V is provided from the cameras 12a and 12b and stored in the RAM 15c or the storage unit 15d (Yes in step S20), the contour extracting unit 24 creates the contour image 30 of the viewer V. Extract (step S30). As described above, the contour image 30 is extracted by creating an image in which the difference between the image captured by the viewer V and the background image not captured by the viewer V is binarized. The contour extracting unit 24 stores the extracted contour image 30 in the storage unit 15d. The contour image 30 is formed, for example, as a white region in the binarized difference image. The binarized difference image including the contour image 30 is stored in the RAM 15c or the storage unit 15d.

Next, the virtual straight line extraction unit 26 calculates contact point position information where the search straight line L _s contacts the head H of the viewer V in the difference image (step S40). Specifically, the virtual straight line extraction unit 26 repeatedly performs the movement of the search straight line L _s extending in the horizontal direction by a distance corresponding to one pixel in the vertical direction in the difference image 32. At this time, the search line L _s crosses the region having the same binarized data value as the head H of the viewer V, and the search line L _s continues to cross the region continuously is a predetermined number of times. When reaching, the region is determined as the region of the head H of the viewer V. At this time, the virtual straight line extraction unit 26 obtains the average value of the center positions in the horizontal direction of the transverse width at which the search straight line L _s crosses the contour image 30 as information in the horizontal direction of the contact position information. Further, the virtual straight line extraction unit 26 obtains the position in the vertical direction of the search straight line L _{s at} the average value as information in the vertical direction of the contact position information. In order to obtain the contact position information of the viewer V's head more accurately, the virtual straight line extraction unit 26 obtains an average value and a standard deviation regarding the crossing width that the search straight line L _s crosses the region, and out of the crossing width, The average value of the center position in the horizontal direction related to the transverse width included in the allowable range determined from the average value and the standard deviation is preferably determined as information in the horizontal direction of the contact position information. Thereby, the virtual straight line extraction unit 26 can prevent an inaccurate average value including an area different from the actual area of the head H from being obtained.

Next, the virtual straight line extraction unit 26 extends from the photographing center of each of the cameras 12a and 12b in the photographing space photographed by the cameras 12a and 12b, using the obtained contact position information, and the head H of the viewer V. First imaginary straight lines L ₁ and L _{2 in contact with} are obtained (step S50). Specifically, the virtual straight line extraction unit 26 converts the values of each element of the matrix obtained in advance to relate the two-dimensional position coordinates on the photographed image and the three-dimensional position coordinates in the photographing space. The first virtual straight lines L ₁ and L ₂ are obtained from the contact point position information using the parameter values.

Next, the position calculating unit 28 translates the first virtual straight lines L ₁ and L ₂ into the second virtual straight lines L ₃ and L ₄ that are translated by a predetermined distance in the direction in the imaging space corresponding to the horizontal direction in the difference image 32. Is calculated (step S60). Here, the predetermined distance for translating the first virtual straight lines L ₁ and L ₂ is a radius R _h of the sphere when the head H of the viewer V is approximated as a spherical shape, and is a human head. The average radius of the part.
The second imaginary straight line L _3, L _4, the camera 12a, required for first each virtual straight line L _1, L ₂ obtained from the difference image for each 12b. Therefore, the position calculation unit 28 determines whether all the second virtual straight lines L ₃ and L ₄ have been calculated for each of the cameras 12a and 12b (step S70). The position calculation unit 28 waits until all the second virtual straight lines L ₃ and L ₄ are calculated.

When the second virtual straight lines L ₃ and L ₄ are all calculated (Yes in step S70), the position calculating unit 28 calculates the coordinates of the center position of the viewer V's head (step S80). Specifically, the position calculating unit 28, the second virtual straight line L _3, L ₄ intersections of, or the position of the point where the second virtual straight line L _3, L ₄ are closest together, the head of the viewer V As the center position of the part, the coordinates of this position are calculated. Since the shape of the head H of the viewer V is processed as a spherical shape, the second virtual straight lines L ₃ and L ₄ often have no intersection. Therefore, the position calculation unit 28 obtains the position of the point at which the second virtual straight lines L ₃ and L ₄ are closest to each other as the center position of the viewer V's head.
The coordinates of the center position of the head H calculated in this way are provided to the video generation unit 14b as shown in FIG. 2, for example, and the video in which the appearance of the virtual object is changed by the motion parallax of the viewer V Used to generate a signal.
In this embodiment, the cameras 12a and 12b are used to photograph the viewer V from two directions, but three or more cameras are used to photograph the viewer V from three or more directions. The first virtual straight line and the second virtual straight line may be obtained, and the center position of the head H of the viewer V may be obtained using these second virtual straight lines.

The subject position measuring device 20 and the subject position measuring method of the present embodiment are arranged so that the two first virtual straight lines in the photographing space are in contact with the head H of the viewer V when photographing with the cameras 12a and 12b from two directions. L ₁ and L ₂ are obtained, and the position of the intersection between the two second virtual lines translated by a predetermined distance or the second virtual lines L ₃ and L ₄ are obtained by calculating the first virtual lines L ₁ and L _2. The position of the closest point is obtained as the position of the head H. For this reason, in the present embodiment, the position of the head H of the viewer V can be obtained by simple calculation as compared with the conventional case, and there are factors that are affected by subtle fluctuations in lighting conditions as in the conventional case. Hard to enter. For this reason, this embodiment can obtain | require the position of the viewer's V head H stably.

Since the subject position measuring device 20 and the subject position measuring method obtain the first virtual straight lines L ₁ and L ₂ by obtaining the contact position information in the image where the contour image and the search straight line L _s are in contact with each other, the image is taken from the contour image 30. The first virtual straight lines L ₁ and L ₂ in the space can be easily obtained.

Furthermore, the object position measuring device 20 and the object position measuring method, as search linearly L _s crosses the contour image 30, by performing repeatedly be distance moved corresponds to one pixel search straight line L _s in a vertical direction The average value of the center position in the horizontal direction of the region where the search straight line L _s crosses the contour image 30 is obtained as information in the horizontal direction of the contact point position information. Therefore, the present embodiment, the search straight line L _s as compared with the case of obtaining the point in contact with the contour image 30, can be accurately determined and stable contact position information.

Furthermore, the object position measuring device 20 and the object position measuring method, the search line L _{when s} traverses the contour image 30 continuously for a predetermined number of times in the vertical direction in the difference image 32, the transverse width of the area where the search line L _s crosses Find the mean and standard deviation for. At this time, the subject position measuring device 20 and the subject position measuring method are such that the difference image 32 relates to a region in which the transverse width is included in the allowable range obtained from the average value and the standard deviation among the regions that the search straight line L _s crosses. Information in the horizontal direction of the contact position information is obtained using the average value of the center positions in the horizontal direction. For this reason, in this embodiment, it can prevent calculating | requiring the incorrect average value containing the area | region different from the area | region of the actual head H. FIG.

The subject position measurement device 20 and the subject position measurement method repeatedly move the search line L _s in the vertical direction in the difference image 32 by a distance corresponding to one pixel, and the search line L _s is the head of the viewer V. When the number of times that the search line L _s crosses the region having the same binarized data value as H and the search line L _s continuously crosses the region reaches a predetermined number, the region is the region of the head H of the viewer V Is determined. For this reason, this embodiment has a low possibility of erroneously extracting the region of the head H of the viewer V.
In this embodiment, a person and a person's head are used as the subject and a part of the subject. However, the present invention is not limited to the person and the person's head, and any other subject or part of the subject may be considered as a spherical shape. .

The said embodiment discloses the content shown below.
(Appendix 1)
A subject position measuring device for obtaining a position of the part by regarding a part of the photographing subject as a spherical body,
A plurality of cameras that photograph a subject to be photographed from at least two directions;
A data processing unit for determining a position of the part of the photographing subject by regarding a part of the photographing subject as a spherical body from a plurality of images photographed by the camera;
The data processing unit
In each of the plurality of images, a contour extraction unit that extracts a contour image of a photographing subject in the image;
Contact position information in the image where the contour image and a search straight line for searching for the contour image in contact with the first image in the first direction come into contact is obtained. A virtual straight line extraction unit for obtaining a first virtual straight line extending from each of the contours of the photographic subject for each of the contour images using the contact point position information;
An intersection of the second virtual lines obtained by translating the first virtual line obtained for each of the contour images by a predetermined distance in a direction in the imaging space corresponding to the first direction, or the second virtual line A position calculation unit that obtains a position of a point at which straight lines are closest to each other as a center position of the portion of the shooting subject in the shooting space;

(Appendix 2)
The virtual straight line extraction unit moves the search straight line in a second direction orthogonal to the first direction so that the search straight line extending in the first direction crosses the contour image. By repeating this, the average value of the center position in the first direction of the region where the search straight line crosses the contour image is obtained, and the obtained average value is obtained as information in the first direction of the contact point position information. The subject position measuring apparatus according to Supplementary Note 1, defined as:

(Appendix 3)
The virtual straight line extracting unit traverses the region where the search straight line crosses a region having the same binarized data value as the contour image, and the search straight line repeats moving in the second direction. The subject position measuring apparatus according to appendix 2, wherein the area is determined as the partial area of the photographic subject when the number of repetitions crossing the predetermined number of times is reached.

(Appendix 4)
The virtual straight line extraction unit obtains a transverse width that the search line crosses the region every time the search line crosses the region having the same binarized data value as the contour image, and the search straight line is the second line. When the number of repetitions that cross the region continuously when the movement is repeated in a direction reaches a predetermined number of times, an average value and a standard deviation of the transverse width are obtained, and the transverse width among the regions that the search straight line has traversed Note that the average value of the center position in the first direction is determined as the information in the first direction of the contact position information, limited to a region included in the allowable range obtained from the average value and the standard deviation. The subject position measuring apparatus according to any one of 1 to 3.

(Appendix 5)
Any one of appendices 1 to 4, wherein the contour extraction unit extracts the contour image by binarizing a difference between a captured image of the captured subject and a background image of the captured subject that has not been captured. The subject position measuring apparatus according to Item 1.

(Appendix 6)
A display system for displaying images,
The subject position measuring device according to any one of appendices 1 to 6,
A data processing device including a video generation unit that generates a video signal of a virtual object to be viewed from the shooting subject according to the center position of the portion of the shooting subject;
A display system comprising: a screen display device that displays an image of the virtual object on a screen using the generated video signal.

(Appendix 7)
A subject position measuring method performed by a subject position measuring device that obtains a position of the part by regarding a part of the photographing subject as a spherical body,
By photographing a subject to be photographed from at least two directions using a camera, a plurality of images viewed from at least two directions are acquired,
In each of the plurality of images, extract a contour image of the photographic subject in the image,
Contact position information in the image where the contour image and a search straight line for searching for the contour image in contact with the first image in the first direction come into contact is obtained. A first imaginary straight line extending from each and contacting the part of the photographic subject is obtained for each of the contour images using the contact position information,
An intersection of the second virtual lines obtained by translating the first virtual line obtained for each of the contour images by a predetermined distance in a direction in the imaging space corresponding to the first direction, or the second virtual line A method for measuring a position of a subject, characterized in that a position of a point where they are closest to each other is obtained as a center position of the part of the photographing subject in the photographing space.

(Appendix 8)
When obtaining the first virtual straight line, obtaining contact position information where the contour image extracted from each of the plurality of images and a search straight line extending in the first direction in the image and searching for the contour image are in contact with each other. The subject position measuring method according to appendix 7, wherein the first virtual straight line is obtained by:

(Appendix 9)
When obtaining the first virtual straight line for each of the contour images, the search straight line is orthogonal to the first direction so that the search straight line extending in the first direction crosses the contour line. By repeatedly performing a predetermined distance movement, an average value of center positions in the first direction of the region where the search line crosses the contour image is obtained, and the obtained average value is used as the contact position information. The subject position measuring method according to appendix 8, wherein the subject position is determined as information in the first direction.

(Appendix 10)
When the first virtual straight line is obtained for each of the contour images, the search straight line crosses an area having the same binarized data value as the contour image, and the search straight line moves in the second direction. The subject position measuring method according to appendix 9, wherein the region is determined to be the partial region of the photographic subject when the number of repetitions that cross the region continuously reaches a predetermined number when repeated.

(Appendix 11)
When obtaining the first virtual straight line for each of the contour images, each time the search straight line crosses an area having the same binarized data value as the contour image, a transverse width across which the search straight line crosses the area is obtained. When the number of repetitions that cross the region continuously when the search line repeatedly moves in the second direction reaches a predetermined number of times, an average value and a standard deviation of the transverse width are obtained, and the search line The average value of the center position in the first direction is limited to the area in which the transverse width is included in the allowable range obtained from the average value and the standard deviation, and the average value of the center position in the first position of the contact position information. The subject position measuring method according to any one of appendices 8 to 10, which is obtained as information in one direction.

(Appendix 12)
When extracting the contour image, the contour image is extracted by binarizing a difference between a captured image of the photographic subject and a background image of the photographic subject that has not been captured. The subject position measuring method according to any one of the preceding claims.

  As described above, the subject position measuring device, the display system, and the subject position measuring method of the present invention have been described in detail. However, the present invention is not limited to the above-described embodiment, and various improvements and modifications can be made without departing from the gist of the present invention. Of course, changes may be made.

10 Display System 12a, 12b Camera 14 Data Processing Device 15a CPU
15b ROM
15c RAM
15d Storage unit 15e Input / output unit 16 Screen display device 20 Subject position measurement device 24 Contour extraction unit 26 Virtual straight line extraction unit 28 Position calculation unit 30 Contour image 32 Difference image

Claims (6)

A subject position measuring device for obtaining a position of the part by regarding a part of the photographing subject as a spherical body,
A plurality of cameras that photograph a subject to be photographed from at least two directions;
A data processing unit for determining a position of the part of the photographing subject by regarding a part of the photographing subject as a spherical body from a plurality of images photographed by the camera;
The data processing unit
In each of the plurality of images, a contour extraction unit that extracts a contour image of a photographing subject in the image;
Contact position information in the image where the contour image and a search straight line for searching for the contour image in contact with the first image in the first direction come into contact is obtained. A virtual straight line extraction unit for obtaining a first virtual straight line extending from each of the contours of the photographic subject for each of the contour images using the contact point position information;
An intersection of the second virtual lines obtained by translating the first virtual line obtained for each of the contour images by a predetermined distance in a direction in the imaging space corresponding to the first direction, or the second virtual line A position calculation unit that obtains a position of a point at which straight lines are closest to each other as a center position of the portion of the shooting subject in the shooting space;   The virtual straight line extraction unit moves the search straight line in a second direction orthogonal to the first direction so that the search straight line extending in the first direction crosses the contour image. By repeating this, the average value of the center position in the first direction of the region where the search straight line crosses the contour image is obtained, and the obtained average value is obtained as information in the first direction of the contact point position information. The subject position measuring device according to claim 1, defined as:   The virtual straight line extracting unit traverses the region where the search straight line crosses a region having the same binarized data value as the contour image, and the search straight line repeats moving in the second direction. 3. The subject position measurement apparatus according to claim 2, wherein when the number of repetitions crossing a predetermined number of times is reached, the region is determined as the partial region of the photographing subject.   The virtual straight line extraction unit obtains a transverse width that the search line crosses the region every time the search line crosses the region having the same binarized data value as the contour image, and the search straight line is the second line. When the number of repetitions that cross the region continuously when the movement is repeated in a direction reaches a predetermined number of times, an average value and a standard deviation of the transverse width are obtained, and the transverse width among the regions that the search straight line has traversed Is determined only as an area within an allowable range obtained from the average value and the standard deviation, and an average value of the center position in the first direction is obtained as information in the first direction of the contact position information. Item 4. The subject position measuring apparatus according to any one of Items 1 to 3. A display system for displaying images,
The subject position measuring device according to any one of claims 1 to 4,
A data processing device including a video generation unit that generates a video signal of a virtual object to be viewed from the shooting subject according to the center position of the portion of the shooting subject;
A display system comprising: a screen display device that displays an image of the virtual object on a screen using the generated video signal. A subject position measuring method performed by a subject position measuring device that obtains a position of the part by regarding a part of the photographing subject as a spherical body,
By photographing a subject to be photographed from at least two directions using a camera, a plurality of images viewed from at least two directions are acquired,
In each of the plurality of images, extract a contour image of the photographic subject in the image,
Finding contact position information in the image where the contour image extracted from each of the plurality of images and a search straight line that searches for the contour image extending in the first direction in the image contact each other, A first virtual straight line extending from each of the cameras and in contact with the portion of the subject to be photographed is obtained for each of the contour images using the contact position information in the photographed photographing space,
An intersection of the second virtual lines obtained by translating the first virtual line obtained for each of the contour images by a predetermined distance in a direction in the imaging space corresponding to the first direction, or the second virtual line A method for measuring a position of a subject, characterized in that a position of a point where they are closest to each other is obtained as a center position of the part of the photographing subject in the photographing space.

Images