JP2000097637A - Attitude position detecting device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable permit use in a computer with low-speed processability by providing an attitude position determining means to determine angle information and translational position information on an object to be measured. SOLUTION: The image of the periphery of a device is picked up by an image information obtaining means to obtain image information. Next, image information obtained by the image information obtaining means at a different point in time is compared and analyzed by an image analyzing means to detect angle information indicating the inclination of the device and translational position information indicating the position movement of the device. In addition, an angle information detecting means is capable of detecting angle information without using the image information, and a translational position information detecting means is capable of detecting translational position information without using the image information. Then, through the use of the angle information and translational position information from the image analyzing means, the angle information from the angle information detecting means, and the translational position information from the translational position information detecting means, the angle information and translational position information of an object to be measured are determined.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a posture detecting device for detecting a posture of a moving body in a three-dimensional space, and more particularly, to a posture detecting device.
A virtual reality system that detects the posture of a human head wearing a head mounting display (HMD), generates an image based on the information, and displays the image as if a virtual space exists in front of the eyes ( In a VR (VR) system, the present invention relates to a self-contained posture / position detecting device for providing a system that can be used anywhere without further limiting the place.

[0002]

2. Description of the Related Art Conventionally, the following are known as posture and position detecting devices. First, there is a reference point installation type posture position detecting device in which a transmitter serving as a reference point for light, ultrasonic waves, a magnetic field, or the like is provided at a location around an operator, and a position corresponding to the transmitter is detected.

On the other hand, a system using an angular velocity sensor such as a gyro has been developed as a conventional technique of a self-contained posture position detecting device. Since the gyro sensor outputs the angular velocity with respect to the rotation of the gyro itself, the displacement relative angle can be obtained by integrating this information.

[0004] Therefore, by arranging the gyro sensors in a space of three axes orthogonal to each other, it becomes possible to obtain a rotational attitude position with respect to this space. The outputs of these gyro sensors are output as positive and negative signals with respect to the direction of rotation around a certain reference voltage, but the reference offset signal drifts due to changes in temperature or humidity, The characteristics change due to the magnetic field, and the output signal itself fluctuates due to vibration such as sound.

However, since the difference of the output value with respect to the offset signal is always used for calculating the angle, if these drifts and noises are included, the angle information which is the calculation result will be included in the angle information. Large errors accumulate.

Therefore, conventionally, measures for canceling these drift signals have been proposed. One of the methods is to use information from a direction sensor as an absolute direction to cancel a drift component of a gyro.

As this azimuth sensor, a system using a GPS or a geomagnetic sensor has been developed. this is,
For example, JP-A-5-71964, JP-A-4-14
No. 934, for example.

[0008] The GPS is a system that specifies the position of the GPS itself while catching signals from several satellites, but cannot be used in places where radio waves are difficult to reach, such as in a building or a tunnel.

On the other hand, the terrestrial magnetism sensor uses the information output by the MR elements arranged on two axes perpendicular to the terrestrial magnetism vector with respect to the terrestrial magnetism vector to determine the magnetic field in the direction of the terrestrial magnetism vector substantially north of the earth. Since the intensity can be detected, the azimuth direction can be obtained from two signals of each MR element.

That is, the respective magnetic intensities are represented by X, Y
Then, the azimuth (θ) is obtained by the equation (1). θ = arcTan (Y / X) (1) Incidentally, the geomagnetic vector has an error called a declination with respect to the geographical north, which is unique to each region on the earth.

In a posture detecting device used in a car navigation system, these must be corrected and used. In a virtual display system for a VR system, the system is used as reference azimuth information in a certain local space. No absolute direction accuracy is required.

Therefore, by comparing the displacement of the azimuth information with the displacement angle information of the gyro, it is possible to detect an error due to drift of the gyro or the like. In addition, when the angle data, which is the gyro integration information, is reset, and the geomagnetic vector at that time is measured, the absolute position can be corrected by comparing with the gyro integration result. .

However, a problem with the geomagnetic sensor is that the compass using the geomagnetic sensor must be placed in a horizontal plane for measurement. This is similar to the declination angle described above, in which the geomagnetic vector has a tilt called the elevation angle in a vertical plane peculiar to each region on the earth, and around Tokyo, it faces downward about 48 degrees from north. ing.

This inclination is a factor that occurs as an azimuth error when the compass is inclined from a horizontal plane. In systems such as car navigation, the car is basically moving in a horizontal plane and can be used as is.

However, in the VR system, it is necessary to detect even a tilted state in order to measure a three-dimensional posture of a person. FIG. 4 shows an example of the VR system.

FIG. 1 is a view showing a virtual image of a desktop image of a computer displayed in a space in front of an operator. Here, when the operator turns right, a screen on the right side of the desktop image appears on the HMD, when the operator turns the opposite side, a screen on the left side appears, and when the operator turns further up and down, a screen in that direction appears. You can feel as if there is a large desktop space in front of you.

In FIG. 4, reference numeral 51 denotes a virtual screen, 52 denotes a real screen, 53 denotes an icon,
4 is a window and 60 is an HMD. FIG. 5 shows coordinate axes in the three-dimensional space at this time.

That is, FIG. 5 shows that the forward direction of the operator is +
The figure shows three-dimensional space coordinates with the X axis, the + Y axis in the left direction, and the + Z axis in the upward direction. In addition, the rotation direction of the right screw with respect to each axis is defined as + roll, + pitch (Pi).
tch) and + Yaw (Yaw).

When the operator turns left and right to look at the left and right screens, the direction changes in the Yaw direction. However, when the operator moves the head up and down to look in the vertical direction on the screen, the Pitch direction changes.

Further, as a function corresponding to the gimbal mechanism, there is a method of a software gimbal mechanism which measures an inclination angle by an acceleration sensor and corrects the azimuth output (θ) of the geomagnetic sensor based on the inclination information.

As a device in which a gyro, a geomagnetic sensor and an acceleration sensor are combined, there is a conventional example disclosed in JP-A-8-178687. However, in this case, three magnetic sensors are required, and the output for each axis is X. Y and Z, and the inclination with respect to the horizontal plane at this time is Ro
Assuming ll and Pitch, the general correction equation is equation (2).

Here, Xh and Yh are values obtained by correcting the strength of the magnetic vector into the horizontal plane. Xh = X * cos (Roll) + Y * sin (Roll) * sin (Pitch) -Zcos (Roll) * sin (Pilch) Yh = Y * cos (Roll) + Z * sin (Roll) = arcTan (Yh / Xh) (2) When the acceleration sensor is tilted by (θg) with respect to the gravity vector G, the acceleration sensor has a gravitational acceleration component (G · cos) corresponding to a length projected perpendicular to an axis in the detection direction. (Θ
g)), and a motion acceleration component due to the acceleration / deceleration motion is generated.

In order to cancel the motion acceleration component, it is necessary to perform a filtering process. By adopting such a configuration, it is theoretically possible to obtain a posture in a three-dimensional space without contradiction.

The acceleration sensor uses the output of the inclination information based on the gravity of the earth, that is, is used as an inclination sensor. Japanese Patent Laid-Open No. 6-89 describes a method for detecting the movement of a moving object by image processing.
No. 342 discloses a conventional example.

Conventionally, there has been proposed a method of measuring a posture of a moving body by arranging a special mark or a light source in a space around the moving body and detecting a motion vector by image processing.

[0026]

However, the VR shown in FIG.
When a geomagnetic sensor is used to detect the posture and position shown in FIG. 5 to realize the system, an error occurs because the azimuth output is inclined from the horizontal direction. Therefore, in general, in a geomagnetic sensor, an attempt is made to reduce errors by mounting a mechanical gimbal mechanism for maintaining a horizontal state. However, mounting a mechanical mechanism in a detection device is small and lightweight. It is a difficult point in terms of conversion.

Also, the geomagnetic sensor is vulnerable to external disturbances, is easily affected by magnetic field distortion due to nearby irons, magnetized objects, buildings, and the like. Also, the geomagnetic sensor itself has an error due to an external magnetic field. There are many problems in using it as a typical direction sensor.

However, in an application in which the device is used only in a certain local area, there is no problem even if the absolute direction is different, but there is a motor, a CRT, or the like nearby or a magnetized object. Even if the robot moves, the magnetic field is disturbed. Even if it is used as a relative reference direction, it is necessary to pay attention to the surrounding environment.

Further, the geomagnetic sensor has a problem that the reaction speed is slow. Furthermore, an apparatus in which a gyro, a geomagnetic sensor, and an acceleration sensor are combined also has a problem as a magnetic sensor as described above.

In addition to the above-mentioned problems, there is a large problem in utilization. That is, the direction sensor is
To show the relative value to the absolute geographical direction on the earth, for example, if you are in a train or car, the direction of the vehicle to the earth changes even if you do not move yourself,
The posture changes without permission.

In a portable computer system,
In a system using a display screen using VR spatial display technology, in such an environment, an image cannot be fixed in front of the user himself / herself, and the image moves irrespective of the user's intention.

As described above, when the acceleration sensor is used on the earth as a tilt sensor, no problem occurs in a horizontal state because human beings always act on gravity. In the case where an inclination occurs on a mountain path or the like, or when an attempt is made to use it in outer space where gravity does not reach, it cannot be used again.

In the prior art as a self-contained sensor for detecting the attitude of a moving body as described above, the azimuth of a geomagnetic sensor serving as a relative or absolute reference is corrected in order to correct drift of a gyro sensor. When the reference information is used, there is a problem that the reference information is shifted due to the influence of magnetic disturbance or an error caused by the inclination with respect to the horizontal plane.

Also, GPS cannot be used in places where radio waves are difficult to reach, such as in a building or in a tunnel. In addition, since the azimuth sensor and the acceleration sensor are based on gravity and the magnetic field of the earth, the GPS sensor cannot be used in such places. Can only be used in

Further, when an attempt is made to use the mobile terminal in a moving object such as a vehicle, not the attitude of the operator with respect to the vehicle but the attitude of the vehicle itself is detected. It moves.

In the detection of the posture of a moving object by image processing, measurement cannot be performed if computation cannot be performed due to conditions such as feature points of a captured peripheral image.

In the conventional method of measuring the attitude of a moving object by image processing, the position of a reference mark as a feature point in space is always compared with a feature point detected from a previous image. In order to find the motion vector, if there is no overlap between the previous image and the next image, the motion vector cannot be detected, and it is necessary to extract the feature points again and proceed to the next image input processing. Since the limit of the motion detection speed is determined by the performance of the image processing speed, a high-speed image processing system is required to perform image processing in response to human motion.

As described above, conventionally, in image processing, a device capable of high-speed arithmetic processing is required.
A lot of so-called heavy arithmetic processing is required, and a computer having a very high processing capability is required, and there are many problems in making a portable processing device.

The present invention has been made by paying attention to the above problems, and has as its object to provide a posture and position detecting device which can be used favorably in any environment.

The present invention has been made in view of the above problems, and has as its object to provide a posture / position detecting apparatus which can be used even with a computer having a low processing capacity. .

[0041]

According to the present invention, in order to solve the above-mentioned problems, (1) the measuring object is mounted or built into the object to be measured, and moves with the movement of the object to be measured; In a posture position detection device that detects posture position information of an object, an image information acquisition unit that can acquire image information by imaging the periphery of the device,
Image analysis means for comparing and analyzing the image information at different points in time acquired by the image information acquisition means to detect angle information representing the inclination of the device and translational position information representing the position movement of the device; and Angle information detecting means capable of detecting angle information without using the image information; translational position information detecting means capable of detecting translational position information without using the image information; and angle information and translation position from the image analyzing means. Information, the angle information from the angle information detection means, using the translation position information from the translation position information detection means, the attitude position determination means to determine the angle information and the translation position information of the measurement object, Is provided.

The invention (1) corresponds to the first embodiment of the invention described later. According to the invention of (1), the information of the angle information detecting means and the translational position information detecting means which has no reference point and detects relative posture change or translation displacement information in space is used. By associating information by image analysis means capable of having an absolute reference point in space with respect to the measurement object,
It is possible to obtain a posture position with respect to an absolute reference point in space.

According to the present invention, in order to solve the above-mentioned problems, (2) the angle information detecting means comprises: an angular velocity detecting means for detecting angular velocity information on three orthogonal axes in space; First attitude position calculating means for calculating angle information of the apparatus, wherein the translational position information detecting means comprises:
The image analysis method includes: acceleration detection means for detecting acceleration information in a translation direction on three axes orthogonal to space; and second attitude / position calculation means for calculating angle information and translation position information of the apparatus from the acceleration information. The means includes: an image input means for inputting a captured image in a time series; a feature point extracted from the input image, the feature point information including coordinate information is recorded, and the previously recorded feature point information is stored. A searchable feature point detecting means; and a third attitude position calculating means for calculating angle information and translational position information of the apparatus from the displacement of the position of the feature point, wherein the attitude position determining means includes the angular velocity detecting means. Alternatively, it is determined whether or not the apparatus is stopped based on the output of the acceleration detecting means, and based on the determination, the calculation method of the first posture position calculation means or the calculation method of the second posture position calculation means is determined. Stop determining means for outputting further information, and correction calculating means for calculating information from the first attitude position calculating means, the second attitude position calculating means, and the third attitude position calculating means. (1)
An attitude position detecting device according to the above is provided.

The invention (2) corresponds to the first embodiment of the invention described later. According to the invention of (2), the angular velocity and the intermediate data are used as auxiliary data for image processing when the angular velocity is detected by the angular velocity detecting means and when the translational position is detected by the acceleration detecting means. By doing so, it becomes possible to perform image processing while reducing the calculation load.

Therefore, according to the invention of (2),
Since the load of the arithmetic processing is smaller than that of an apparatus using only image processing, high-speed processing is possible, and a low-speed processing system can be configured.

Further, according to the invention of (2), it is possible to correct a gyro drift or the like by judging a stop state by comparing with a motion vector in image processing. Further, according to the invention of (2), as another effect, a relative posture position with respect to the moving body can be detected even in the moving body.

Further, according to the present invention, in order to solve the above-mentioned problems, (3) mounted on or built into a measurement object, moved with the movement of the measurement object, and moved to the posture position of the measurement object. In a posture and position detection device for detecting information, a detection unit for detecting posture and position information of the device, an image acquisition unit capable of imaging the periphery of the device and acquiring image information, and image information acquired by the image acquisition unit Feature point extracting means for extracting more feature points; attitude position calculating means for calculating second attitude position information from a change in the position of the feature point between different pieces of image information obtained by the image obtaining means;
A posture position detecting device comprising: posture position determining means for determining posture position information by correcting the posture position information with the second posture position information.

The invention (3) corresponds to the first embodiment of the invention described later. According to the invention of (3), when the coordinate information for the same feature point is obtained, the accuracy increases as the translation information increases, and the error can be reduced as the number of measurement data increases. It is possible to reduce the error of the coordinate information to be obtained on the fixed space.

According to the invention of (3), since the feature point information obtained from the two-dimensional image processing includes the posture parameter information of the camera, the feature point information is converted into information from the fixed coordinate origin. By storing the information, the processing when comparing the attitude angle information of the angular velocity detecting means or the translation information from the velocity detecting means with the reference reference information is simplified, and the processing speed is increased.

[0050]

Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 and FIG. 11 are block diagrams of the whole and main parts showing the configuration of the first embodiment according to the present invention.

FIG. 2 is an overall block diagram showing the configuration during the operation of camera posture correction. FIG. 3 shows the flow of various modes relating to image processing. 1 and 11
As shown in the figure, 3 in the space of the moving object as the measurement object
The angle detecting means 100 for calculating the attitude angle information of the axis includes an angular velocity detecting means 11 composed of three angular velocity sensors arranged in a space on three orthogonal axes, and a moving body based on the angular velocity information from the angular velocity detecting means 11. And first attitude calculating means 15 for calculating the attitude angle information.

The translation position detecting means 101 for calculating three-axis translational position information on the space of the moving body includes acceleration detecting means 26 for detecting acceleration information in the translational direction of the space on three orthogonal axes. The second attitude calculating means 27 calculates tilt information and translation information of the moving object from the acceleration information from the acceleration detecting means 26.

The image processing posture detecting means 102, which is installed on the moving body itself and calculates six-axis image posture information of translation and posture of the moving body from the image around the moving body, stores the peripheral image on the imaging surface. Image input means 1 configured to include a camera unit for projecting and an image pickup device (CCD or the like) for converting the projected image into a video signal, and for inputting the images in time series.
7 and a feature point detecting means 18 for extracting a plurality of feature points from the image input from the image input means 17, storing the feature points, and simultaneously searching for and detecting the coordinates of the previously stored feature points. A motion vector detecting means 19 for detecting a motion vector of an image from a two-dimensional displacement of a feature point by the feature point detecting means 18; and a moving object based on the angular velocity information, the acceleration information, and the feature point information. And third attitude calculating means 20 for calculating the attitude in the rotation direction and the translation direction.

Further, the position and angle information from the angle detecting means 100, the translational position information from the translational position detecting means 101, and the image and posture information from the image processing and posture detecting means 102 provide information on the space. Attitude calculation means 103 for calculating 6-axis attitude information of the moving body in the vehicle
The attitude calculation method of the first attitude calculation means 15 and the attitude calculation method of the second attitude calculation means 27 are based on the information from the motion vector detection means 19 and the information from the angular velocity detection means 11 and the acceleration detection means 26. And a correction calculating means 21 for calculating information from each of the first attitude calculating means 15, the second attitude calculating means 27 and the third attitude calculating means 20. ing.

FIG. 10 shows the positional relationship between the angular velocity sensor and the acceleration sensor as the angular velocity detecting means 11 and the acceleration detecting means 26 in the moving body posture position detecting device constructed as described above.

Here, as the angular velocity sensor, X, Y,
FIG. 1 shows a piezoelectric vibration gyro sensor and the like for each of the three axes of Z.
0 is used. The acceleration sensor is a minute electromechanical sensor made by micromachine technology or the like. One is a two-axis type device for detecting acceleration in the X-axis and Y-axis directions, and the other is a Y-axis device. And a one-axis type device capable of detecting the is used as arranged as shown in FIG.

Since the gyro sensors are arranged in a space on three orthogonal axes, the gyro sensor outputs an angular velocity with respect to the rotational movement around each of the arranged axes, and the sign changes according to the rotational direction around the axis.

Equation (3) shows a relational expression for obtaining the angle (θ) from the gyro output (Vg). An offset, which is a reference output at rest, is held for each gyro, and the offset (Vo) and a difference signal of the gyro output become an angular velocity signal (ω), and an angle is obtained by integrating the angular velocity signal (ω).

As a result of this calculation, the relative displacement of the posture in the space is obtained. ω = As · (Vg + Vo + Vt + Vn) θ = Σ (ω) (3) where As is a gyro scale factor including a time (Δt) factor, Vt is a fluctuation component, Vn
Indicates a noise component.

The offset which is the reference output varies depending on the unit, and it is necessary to store and correct the offset for each unit. Further, it fluctuates due to a temperature change, and the fluctuation component (Vt) and the noise component (Vn) become error components.

Therefore, since these error components are integrated into the gyro angle information, the values fluctuate even in a stopped state, and appear as output drift errors.

Therefore, the difference output is subjected to low-pass filtering (LPF) to obtain a noise component (V
High frequency (fh) such as n) and vibration components applied to the device are cut and used as first information in the first attitude calculation means 15.

At the same time, the second information, which is cut at a lower frequency (fl), such as a drift component, is obtained as an offset value, and a difference from the second information is obtained.
Errors can be reduced.

However, the low-pass filter (L
If the cutoff frequency (fl) of PF) is set high,
Although the effect of reducing such a large drift is obtained, the operation signal due to the original posture change is also cut, so that it does not respond to a slow operation.

Therefore, the information of the low-pass filter is stored as an offset only when it is determined by the stop determination control means 16 that will be described later that the apparatus is stopped. By doing so, the cut-off frequency (fl) of the low-pass filter can be set low, it is possible to detect even lower-speed movement, and fluctuation due to drift is reduced.

The acceleration sensor can be used in two ways. One is a function as a tilt sensor using the gravitational acceleration of the earth's gravity, and the other is a motion acceleration sensor function generated by inertial motion.

Speed information is obtained by integrating the motion acceleration information, and position information is obtained by further integrating. For example, considering an acceleration sensor in the X-axis direction, a gravitational acceleration component (Gx) contributing to the X-axis acceleration sensor at this time and an output (V
a), acceleration sensor scale factor (As), inclination of the X-axis acceleration sensor with respect to the horizontal plane, that is, rotation angle (θ) around the Y-axis, earth's gravity G (1G = 9.8 m / s), stationary output (Vo) And

Gx = G × sin θ Va = Gx × As−Vo θ = sin ⁻¹ {(Va−Vo) / As} (4) The inclination sensor can be obtained by Expression (4).

However, at this time, if the acceleration sensor is in a motion state, information obtained by adding the gravitational acceleration and the motion acceleration is obtained as the acceleration information. By performing low-pass filtering, a gravitational acceleration component can be obtained.

Therefore, it can be used to correct the drift component of the angle information of the gyro data based on this information. Therefore, although the inclination information obtained by the acceleration sensor is statically absolute reference data, it cannot be used for detecting a moving object in a moving state or a vehicle.

At this time, the gyro information that can be corrected is only the rotation around the X axis and the Y axis, and the information around the Z axis, which is the rotation direction with respect to the gravity axis, cannot be corrected. For the second motion acceleration sensor function, (5)
Consider the case where the acceleration sensor in the X-axis direction is inclined (θ) with respect to the horizontal plane in the same manner as described above for the acceleration information (α) due to the translational motion by the calculation of the expression.

Α = {(Va−Vo) / (As−Gsin θ)} / cos θ (5) Regarding the angle information (θ) at this time (without using the calculation information from the tilt sensor function), the Y It can be obtained by using the angle information obtained from the angular velocity sensor of the shaft.

In order to detect the position information, it is sufficient to integrate the motion acceleration information twice, thereby obtaining the position information. The above calculations are performed by calculating the motion acceleration components (α _x , α _y , α _z ) and the gravitational acceleration components (G _x ,
G _y , G _z ), it is possible to obtain acceleration information, velocity information, and position information in the translation direction in a three-dimensional space.

However, since the integral operation is performed in the same manner as in the case of the gyro, the position information includes an error. By the way, since no motion acceleration is applied in the stopped state, only the angle information obtained by using the gravitational acceleration is used, but in the non-stop state, the value of the motion acceleration is treated as significant.

In the present invention, the posture position information indicates one or both of angle information and translation position information. Next, returning to FIG. 1, the operation, function, and processing of each unit will be described.

[Image input means 17] Image input means 17
Converts an optical projection image obtained through a camera optical system into a dimensional image video signal by a CCD serving as an image sensor, and stores the video signal in a frame memory.

The image input means 17 performs preprocessing such as contour extraction on the video signal of the two-dimensional image, and transfers the time-series image data to the characteristic checking means 18. Thus, the image input unit 17 inputs time-series image data obtained by capturing an image around the moving object accompanying the movement of the moving object as the measurement target with the camera.

In this case, it is important to photograph a fixed object having many feature points within the angle of view of the camera as the peripheral image taken by the camera. In addition, it is important to point the camera toward a feature point such as a ceiling or a wall so that a moving object existing in the vicinity or an outside scenery when a moving object such as a vehicle is not photographed.

As described above, by taking the image input direction into account, reliable image processing can be performed, and motion information from the acceleration sensor and the angular velocity sensor can be corrected. "Feature point detecting means 18" The feature point detecting means 18
The time-series image data input from the image input unit 17 can be easily distinguished from other regions by image processing in the feature point extracting unit 18a, and it can be specified that the same point on the object has been photographed. Extract a plurality of feature points.

Next, the characteristic point detecting means 18 stores and stores the characteristic point data and coordinate values extracted by the characteristic point extracting section 18a in the characteristic point storing section 18b, and at the same time, stores the characteristic point / posture information storing section 18c. The feature points memorized last time are tracked,
The position is detected, and coordinate information on the two-dimensional image is obtained.

[Image Correlation Processing Based on Gyro / Acceleration Data Information] Generally referred to as two-dimensional correlation processing or template matching processing, the feature point data as a template is scanned in two-dimensional directions with respect to input image data. Perform correlation processing.

At this time, one of the factors causing the problem is the rotation of the camera around the Z axis. For example, when the movement of the operator's head is measured by the HMD 60 shown in FIG. 7, if the input direction of the camera 62 for measuring the movement is attached in the forward direction of the operator's head, the head is moved left and right. Roll, which is a rotation around the X-axis when tilted to an angle, causes rotation of a two-dimensional plane in the image captured by the camera 62.

Similarly, as shown in FIG. 9, even when the motion measuring camera 62 is mounted in the upward direction, the rotation around the Z axis when the neck is moved left and right. The image is rotated by the movement of Yaw.

When performing an inter-field correlation of an image, a correlation process must be performed after rotating a pattern which is a grayscale value on a screen, and an affine transformation process for performing a rotation is performed using a pattern of each feature point data. Each time, the angle is repeatedly changed and the correlation process is performed, so that a very large number of calculations are required.

Therefore, this process is one of the processes with the heaviest loads in image processing. Therefore, based on the attitude angle information from the angular velocity sensor and the translation information from the acceleration sensor, the image vector estimating means 10 described later
Since the two-dimensional movement of the feature points on the image data can be estimated, the search range can be narrowed based on this information, and the correlation processing can be performed more efficiently.

That is, by using the angle information calculated from the gyro, the affine transformation of a smaller angle range and the correlation processing of the range of the two-dimensional predicted position of the feature point by adding the acceleration information can be performed. Get better.

As described above, by using the gyro sensor information and the acceleration sensor information, it is possible to greatly reduce the load of the arithmetic processing. "Feature point / posture storage unit 18c"
8, the feature point storage unit 18b and the feature point / posture storage unit 18
In c, the position of the feature point with respect to the fixed space coordinates is stored.

[0088] During initialization, is exactly the same fixed coordinate space OXYZ the head coordinate system _{(O h X h Y h Z} h coordinate system). Here, since the feature points are the coordinates of a fixed object around the moving object, the feature points can be considered as absolute reference points fixed to the scene of the peripheral image.

Therefore, this feature point becomes an absolute reference point in the fixed coordinate space. FIG. 8 is a diagram showing the relationship between the position of the camera with respect to the head movement posture, and the relationship between the projection image plane that is the camera input image and the fixed point Po that is a feature point in space.

Here, the Po point viewed from the camera coordinates is represented by (X
c, Yc, Zc). Further, the projection relational expression between the three-dimensional space coordinates and the point Q (x, y) on the image plane is as shown in Expression (6).

X = fXc / Zc, y = fY / Zc (6) Further, since the translation vector of the head position is obtained from the translation information from the acceleration sensor, this information and the same feature point between the two images are obtained. A three-dimensional position in space can be estimated from two-dimensional coordinates.

The coordinate data on the two-dimensional coordinates of each feature point extracted by the feature point extraction unit 18a is stored in the feature point storage unit 18b. Further, feature points that are rich in changes such as gray values are stored in the next feature point / posture information storage unit 18c.

That is, the feature point / posture information storage unit 1
8c includes a unique number of the feature point and a fixed coordinate space OX.
Reference point reference information for storing feature point (Xo, Yo, Zo) coordinates for YZ in the form of internal information, time series data numbers of fixed points, and feature point positions of feature points estimated for each time series data number Information, attitude information from the angular velocity sensor at that time, and translation information from the acceleration sensor are stored.

Here, the unique number of the feature point is a unique number for each feature point extracted in the background image, and is managed so that at least eight points are always obtained within one angle of view of the input image. You.

The feature point time series data number is a number for managing feature point position information for each unique feature point. This feature point location information does not always store the data input in time series, but preferentially stores the location information at a certain threshold or more as the value of the translation information, which is the location information when the image was captured. I do.

For example, information of the same characteristic point Po from various directions other than the same point when the moving body moves is stored. Furthermore, when the value of the feature point position information is significantly different or cannot be detected, the sequence data at that time is discarded without being stored.

If this state continues for a certain period of time, it is determined that the feature point is not a fixed object, and all data are discarded. If a unit direction vector from the viewpoint Oc to the feature point Po is defined as m, the feature point position information includes a vector m viewed from the camera origin Oc, angle information from the angular velocity sensor, and translation information from the acceleration sensor. It is memorized.

[Third Attitude Calculation Means 20] Next, in the third attitude calculation means 20, when the feature point position information becomes two or more, the distance information to the feature points can be obtained by the principle of the stereo camera. The distance information is obtained and stored as reference reference information.

When the feature point position information reaches a score equal to or greater than a certain threshold, the third posture calculation means 20 obtains the position information from the origin O by the least square method and stores the information as reference reference information.

The reference reference information is managed so that the direction vector Mi viewed from the origin O and the distance Pi from the origin O to Po are calculated and stored, so that the processing of the feature point can be converted into the coordinates of the origin and used. Is done.

Equation (7) shows conditions for deriving an optimal solution from the feature point information of the two screens by the least squares method. Here, M represents a unit direction vector when the feature point is viewed from the origin O in the fixed coordinate space, and i (= 1... N) is a feature point unique number.

Σ | mi h Rmi | ² > min (7) mi is a unit direction vector at a certain point in time and is obtained by Expression (6). Rmi represents a vector after the rotation which has been rotated R with respect to the mi point, and is obtained from angle information from the angular velocity sensor.

H represents a translation vector from mi to Rmi, and is obtained from translation information from the acceleration sensor. Therefore, this reference reference information is approximated by the least squares method based on each attitude information including an error between the angle information from the angular velocity sensor and the translation information from the acceleration sensor, and information on a plurality of feature points on the image. This is the information that has been estimated.

Therefore, this processing is further performed for each time-series data, and averaging is performed on each unique reference reference information, thereby reducing errors due to the angle information from the angular velocity sensor and the translation information from the acceleration sensor. can do.

When the reference information is interpolated by a number equal to or larger than a certain threshold value, the storage of the time-series data of the feature point position information is stopped, and all the feature point position information is discarded. The above processing is also performed on newly detected feature points in the same procedure.

Further, as a condition for storing the feature point position information, the position information having a value of a translation information which is the position information at the time of capturing the image and a threshold value or more is preferentially stored. Under this condition, when the translation information becomes zero (actually, a certain threshold value or less), this feature point position information is directly used.

[0107] In this condition, since the fixed coordinate space OXYZ the head coordinate system with _{(O h X h Y h Z} h coordinate system) are the same, rotation and orientation information obtained by the two-dimensional image (6) Can be used as it is for the correction of the posture information from the angle sensor.

[Motion vector detecting means 19] The motion vector detecting means 19 detects motion information of an image for each field by the above-mentioned feature point matching method, and detects a motion vector from the information.

In the feature point matching method, a motion vector for each field is obtained from the inter-field correlation between some pixels whose gray scale values on the screen are selected as representative points and surrounding pixels.

That is, the motion vector detecting means 19
Based on the time-series data of the two-dimensional coordinate values of some feature points obtained by the feature point detecting means 18, the processing of the motion vector distribution on a two-dimensional image (xy coordinates) is performed.

Further, the motion vector detecting means 19 divides a vector area having the same direction and size from the distribution of the motion vector. "Stop determination control means 16" This stop determination control means 16
Is within a certain threshold value, based on the motion vector information from the motion vector detection means 19, and the first information from the LPF output of the gyro data is below a certain threshold value. If it is determined that the moving object to be measured is stopped, the information of the offset holding means 12 of the gyro output processing system is updated with the second information from the current LPF output.

The same determination condition is applied to the second attitude calculating means 27 for calculating the information from the acceleration detecting means 26. However, the difference from the gyro data is that when the data is determined to be stopped, the data is determined to be tilt data and arithmetic processing is performed.

Therefore, when the moving object to be measured is stopped, no error occurs due to drift, but when the state where the moving object is not stopped continues for a long time, the first posture calculating means 1
There is a problem that an error due to drift is added to the output of No. 5.

[Camera Attitude Parameter Holding Means 22] Next, the mounting position and direction of the camera 62 will be described. The result of the two-dimensional image analysis in the third attitude calculation means 20 is a coordinate system with respect to the camera coordinate system.

However, since the motion to be obtained is a spatial motion of the operator's head, it is necessary to obtain the relationship between the head motion coordinate system and the camera coordinate system. FIG. 7 shows an image in which the camera 62 is mounted on the HMD 60 in a case where the camera 62 is installed so that the shooting direction faces forward with respect to the head.

[0116] Figure 8 is a head coordinate system in the fixed coordinate system OXYZ space _{(O h X h Y h Z} h coordinate system) and the camera coordinate system (Oc
XcYcZc coordinate system). Fixed coordinate system OXY
The rotation posture of the head with respect to the Z coordinate system is rotated around the Z _h axis by y _h , rotated around the Y _h axis by p _{h, and} rotated by X _h.
Let R _{h be} a rotation matrix that determines the attitude when a rotation of r _h is given around the _h axis, and let P _{h be the} translation vector.

[0117] Moreover, given the rotation of the head O _h X _h Y _h Z _h coordinate system yc the camera rotation position around Zc axis relative, pc around Yc axis after rotation around Xc axis after rotation rc The rotation matrix for obtaining the posture at that time is defined as Rc, and the translation vector is defined as Pc.

The head coordinate system defines the neck portion, which is the base point of the movement with respect to the fixed coordinate system, as the origin O _h (X _h Y _h Z _h ).
Also, the position of the camera origin with respect to the head coordinate system is represented by Oc (X
c, Yc, Zc).

When the position of the point Po as viewed from the camera coordinate system is (Xc, Yc, Zc), the point Po in the space of the fixed coordinate system is expressed by the equation (8) with respect to the head and the attitude of the camera 62.
(Where Sx and Cx are sinX, co
sX).

[0120]

(Equation 1)

The first term on the right side of the relational expression (8) is a motion parameter of the head, and the second term is a camera posture parameter.
The camera attitude parameters are stored in the camera attitude parameter storage means 22 and are known in advance.

Therefore, it is possible to convert a motion vector obtained from a camera image from the above relational expression into a head motion in a fixed space. Conversely, the motion of the image feature point can be predicted from the acceleration information and the angular velocity information that are the motion information of the head.

In the “posture detection operation mode” 44 shown in FIG. 3, the movement parameter obtained by the calculation processing in the third posture calculation means 20 is substituted into the second term on the right side of the equation (8), thereby obtaining the moving object to be measured. Can be determined.

[Correction Calculation Means 21] The third posture calculation means 20 obtains the reference angle information including the posture angle converted into the vector from the origin O and the translation information by the calculation by the image processing.

However, since this processing requires more time than the processing by the gyro or the acceleration sensor, the translation position information from the first attitude calculating means 15 and the second
The rotation angle information from the posture calculating means 27 is used as posture position calculation information of the moving body posture position detecting device according to the present invention.

However, the correction calculation means 21 converts the angle information of the angular velocity sensor and the translation information of the acceleration sensor into a vector from the origin O, and converts the attitude angle information including the drift of the gyro data and the translation information from the acceleration sensor. The difference between the reference information at this time and the obtained vector is used as an error component between the angle information of the angular velocity sensor and the translation information of the acceleration sensor.

When an error (ε) between the information of the first attitude calculation means 15 and the information of the second attitude calculation means 27 and the reference information of the third attitude calculation means 20 occurs, the value of the third attitude calculation means 20 And the difference is added so as to be corrected slowly.

For example, regarding gyro data,
The width of the correction value at this time is a value obtained by multiplying the latest angular velocity (ωt) from the first attitude calculation means 15 by the coefficient (k) until the error (ε) becomes zero.

This is shown in equation (9). θt = θ _t−1 + ωt + ε · k (9) At this time, (k) is k = 0 if ωt = 0, k = 0.5 if 0>ωt> Tmax, and k = 1 if ωt> Tmax. Becomes

Therefore, even when an error occurs in the stopped state without any change in the posture, the error is not corrected. However, in the operating state, the width for canceling the error is increased according to the angular velocity.

By doing so, the image does not move without permission irrespective of the operator's intention while observing the VR-like display, and the error is canceled during the operation. Has the effect of making it difficult to understand the movement due to the correction.

In this case, the same processing is performed on the translation information. "Image vector estimating means 10" Image vector estimating means 1
0 substitutes the camera attitude parameter information of the camera attitude parameter holding means 22 and the gyro angular velocity information and the acceleration information from the acceleration sensor instead of the head movement parameter in the camera attitude calculation equation (8). To predict the movement of the camera 62,
The movement position of the feature point can be predicted from the movement information of the camera 62.

For the feature point detecting means 18, the affine transformation angle information by image rotation and the coordinate estimation information of the feature point matching area, and for the motion vector detecting means 19, the motion vector itself or the image area It can be used as determination information for removing movement and vector data that may cause an error such as a moving object existing in the vicinity or a moving object such as a vehicle when in a moving object such as a vehicle.

[0134] Other several processes such as the motion of a feature point or the motion of an image vector obtained by estimation based on the angular velocity information or angle information from the angular velocity detecting means 11 and the acceleration information or translation information from the acceleration detecting means 26. By calculating the information used in advance in advance,
The calculation load can be reduced.

[Measurement Undetermined and Warning Display] An image in which the characteristic point is not found is input from the image input unit 17 to the characteristic point detecting unit 18 by imaging the wall or ceiling with few characteristic points. The motion vector cannot be measured.

Further, when a moving object crosses right in front of the camera 62, or when the operator is in a moving object such as a vehicle and only the outside scene is shown, the motion by image processing is performed. The vector detecting means 19 detects a motion vector that is significantly different from the angle information based on the gyro data and the acceleration information from the acceleration sensor.

In addition, even when the past feature points disappear in the updated image data due to a rapid movement exceeding the processing capacity of the image processing system, detection becomes impossible in the same manner as described above.

In such a case, it is determined that measurement is impossible, and
The correction calculation means 21 calculates only the angle calculation information based on the gyro data and the acceleration information from the acceleration sensor,
An operation mode in which the correction based on the image processing operation information is not performed.

Therefore, according to the present invention, the posture can be always detected under any condition. After that, when the feature points stored last time are extracted,
The correction of the posture information is restarted based on the position information at that time.

However, if the measurement impossible state continues for a long time, an error due to drift of the gyro or integration of the acceleration sensor may be caused. Therefore, the present invention has a function of issuing a warning to the operator when such a state continues for a certain period.

For example, a method of issuing a warning by a buzzer sound or light such as an LED, a method of displaying the status in the HMD video information, and a status information to the host system which is inputting the information of the attitude / position detecting device. Can be notified to the operator as error information.

[Regarding the Camera Mounting Direction Changing Means 23] The mounting direction of the camera 62 is basically based on taking an image in front of the camera when the HMD 60 is mounted.

However, in order to prevent the device from being used under the condition where the above-mentioned unmeasurable state occurs, FIG.
The camera mounting direction changing means 23 which can change the camera mounting direction is provided as shown in FIG.

For example, the camera mounting direction changing means 23
By pointing the camera 62 in the direction of the vertex of the operator, or in the left or right direction, it is possible to set the camera 62 in a direction in which an image under the best conditions for measurement can be obtained in the used environment. .

Further, in response to the switching of the camera mounting direction, the camera posture parameter holding means 22 previously stores the posture parameter data for each mounting direction. The setting may be switched by a parameter setting command or the like.

The switching operation can be automatically performed by incorporating the camera attitude detecting means 24 in the camera 62. This is a method of detecting the mounting direction by the mounting position of the camera 62 on a joint surface between the camera mounting direction changing means 23 and the main body of the HMD 60 by a connection detector such as a microswitch or a Hall element.

Alternatively, by mounting the semiconductor acceleration sensor as the acceleration sensor inside the camera 62,
It can also be used as a tilt sensor that knows the mounting tilt angle of the camera 62.

Therefore, the camera attitude detecting means 24
It is possible to automatically detect that the mounting direction has been changed by the camera mounting direction changing unit 23 and automatically change the posture parameter data of the camera posture parameter holding unit 22, thereby reducing the burden on the user. be able to.

[Camera Parameter Automatic Calibration Function] The functions relating to the camera mounting direction described above can be extremely easily implemented by incorporating the measured camera posture parameters into the system in advance and using them. Changes can be made.

However, individual differences in the camera position with respect to the image input means 17 provided with the camera mounting direction changing means 23 having more flexibility or the base position of the neck as the origin of the movement of the operator (for example, Differences in head size, neck length, etc.) cause errors in the posture parameters.

Therefore, the camera posture parameter calculating means 2
Reference numeral 5 has a function of measuring the mounting posture and position, which are camera posture parameters, for each user, and storing the measured positions and positions in the camera posture parameter holding unit 22.

Expression (8) related to the camera parameter calculation
Then, by obtaining a motion vector obtained from image processing when known motion data is given, it is possible to automatically obtain a posture parameter of a camera mounting position.

The camera parameter automatic calibration function can be performed by a simple operation as follows. First, in the camera posture calibration, it is a condition that an image to be input is captured as far as possible.

Then, in a state where the user directs the camera 62 in a state where the apparatus is actually used, the head on which the apparatus is mounted does not perform a translational movement as much as possible, and it is easy to rotate the head such as up, down, left, right, and sideways. Calibration becomes possible only by performing an exercise and inputting an image.

In the “camera posture calibration mode” 35 shown in FIG. 3, the first posture calculation means 15 and the second posture
The attitude calculation means 27 performs normal operations and calculations. The camera attitude parameter calculating means 25 samples some points of the attitude data based on the gyro data, the translation information from the acceleration sensor, and the image processing data obtained by the third calculating means 20.

In the feature point matching at this time, since the camera posture parameter is not determined, the processing is performed in a mode in which the motion prediction of the feature point is not performed. By substituting these results into the equation (8) related to the camera posture parameter calculation, the camera posture parameters can be back calculated by the least square method.

When the calibration is completed, the posture parameters are stored in the camera posture parameter holding means 22 and thereafter used for calculation of the "posture detection operation mode" 44 of the normal operation shown in FIG.

Therefore, as for the camera mounting direction changing means 23, a method having a higher degree of freedom can be used, the degree of freedom in the camera direction is further increased, and even when the camera is oriented in any direction, the calibration as described above is performed. By executing the application, the apparatus can be used in a good condition, and errors such as individual differences can be eliminated.

In this case, the camera posture detecting means 24 and the like become unnecessary. "Feature point detection and storage means by preprocessing input" The present mobile object posture position detecting device has a mode for storing feature points by preprocessing as another feature.

That is, in the “feature point pre-processing mode” 37 shown in FIG. 3, first, the head is moved in a normally used range. The above-mentioned “posture detection operation mode” which is normally performed 4
In the same manner as in No. 4, a posture detection operation using a gyro sensor, an acceleration sensor and image processing is performed.

The feature points extracted at this time are stored in the feature point / posture information storage section 18c in the feature point detecting means 18 together with the posture information. Further, reference reference information is also calculated and stored in the feature point / posture information storage unit 18c.

Therefore, not only the characteristic points in the currently captured image but also the characteristic points and the posture information of the surrounding images in the environment are associated with each other, and the characteristic point / posture information storage unit 1
8c.

Then, the "posture detection operation mode"
In step 44, feature point information in the camera image is extracted from the feature point / posture information storage unit 18c from the pose information, and a motion vector is detected using the information.

Alternatively, the correction can be performed by extracting the reference reference information from the feature point / posture information storage section 18c. Therefore, the processing of the feature point extraction unit 18a of the feature point detection unit 18 can be simplified, so that the calculation load in image processing can be further reduced and the speed can be further increased.

Since the measurement of the feature point information in the “feature point pre-processing mode” 37 is completed in a short time, the occurrence of drift of the angular velocity sensor and the acceleration sensor can be reduced, and the information of the feature point can be reduced. Can be reduced.

This "feature point pre-processing mode" 37
Is an effective use method when used in a fixed environment such as, for example, sitting on a chair at a fixed place and performing work.

[Fixed Mark Detection Operation Mode] As another feature, there is a "fixed mark detection operation mode" 43 shown in FIG.

The “fixed mark detection operation mode” is the above-mentioned “feature point pre-processing storage mode” 37 in which the place always used by an individual is fixed or used in a room in an amusement facility. It is used when it is used in a more fixed environment than in the case of.

This is a mode in which an image pattern serving as a feature point is stored in advance, and a correlation process is performed only on this pattern. The fixed marks, which are image patterns, are arranged around the environment so that eight or more fixed marks can always be picked up within the camera angle of view.

In arranging the fixed marks, they are usually arranged as randomly as possible, instead of being arranged periodically. However, since information from a gyro sensor or an acceleration sensor is also taken into account, general fixed image processing is performed. This arrangement has a higher degree of freedom than the posture detection device using only the position.

Therefore, since the processing of the feature point extracting unit 18a of the feature point detecting means 18 can be omitted, the calculation load in the image processing can be further reduced and the speed can be increased.

Further, a "feature point pre-processing storage mode" is also provided.
Since a large storage memory such as 37 is not required, the system configuration can be simplified. "Fixed mark registration mode 42" As another feature, in the "fixed mark detection operation mode" 43, an image pattern serving as a feature point is stored in advance therein, as shown in FIG. In the "fixed mark detection mode" 42, prior to the "fixed mark detection operation mode" 43, the operator himself / herself registers a mark input from the image of the camera 62, and registers the mark in the "fixed mark detection operation mode". "43.

In the “fixed mark registration mode” 42,
The image input from the image input means 17 is converted into feature point data and stored internally. However, it is necessary to prevent an image other than the image registered as the fixed mark from appearing in the input image at this time.

Therefore, the “fixed mark registration mode” 4
In 2, the processing of the feature point extraction unit 18a of the feature point detection unit 18 can be omitted, as described above, so that the calculation load in image processing can be further reduced and the speed can be further increased. be able to.

The "fixed mark registration mode" 42
Then, when you want to obtain a stable signal by image processing,
By arranging easily identifiable markers and the like, the accuracy can be further improved.

Further, the "fixed mark registration mode" 42
In this case, since a large storage memory such as the “feature point preprocessing storage mode” 37 is not required, the system configuration can be simplified.

Further, the "fixed mark registration mode" 42
In this case, the mark can be produced by an individual or an image in the use environment can be registered as a mark, so that the degree of freedom in designing the surrounding environment in the use environment increases, and the convenience can be further improved.

[Infrared Light Emitting Means] As another feature, the present apparatus is provided with infrared light emitting means (not shown) such as an infrared LED arranged to irradiate the image input range of the image input means 17. ing.

The image input means 17 has no problem in the case of inputting an image whose surroundings are illuminated by natural light or lighting equipment, but it is desired to use the apparatus in an environment where lighting equipment is not used, such as at night or in a dark room. Occasionally, the device can be used in such an environment by causing the infrared LED provided on the device side to emit light.

In this case, the infrared LED emits light in synchronization with the image input processing of the image input means 17 and stops emitting light in other cases. In this way, by repeatedly turning on and off, unnecessary power consumption is avoided and low power consumption is achieved.

When a distant light pattern such as a night scene can be used, the infrared LED does not emit light. "Artificial retinal chip" As another feature, the image signal is stored as image data from an image pickup device for converting the projected image of the image input means 17 into a video signal in the above-described configuration, and the image data is further contoured. Functions such as extraction, processing of the feature extraction unit 18a, pattern matching processing, and adjustment of the gain level of a video input signal are configured by one artificial retinal chip.

This artificial retinal chip is an image sensor (for example, an artificial retinal IC manufactured by Mitsubishi Electric Corporation) having a built-in image calculation processing function and an analog adjustment function.
A gray image of 28 × 128 pixels can be processed and output.

The optical system of the camera 62 has an angle of view of 30 °, and the angle of rotation becomes 0.23 ° per pixel from 30/128 = 0.23 °. By doing so, higher resolution can be achieved.

As a result, the data can be sufficiently used as reference point data without drift. Therefore, the processing system for the subsequent arithmetic processing can be configured with a lower-cost general-purpose processing system.

By using this artificial retinal chip for the image input means 17, it is possible to reduce the size, cost, speed, and power consumption of the system. In the above description, various settings and mode changes relating to image processing can be automatically performed by setting internal switches (not shown) in the apparatus and apparatus status information.

Further, this can be changed by a setting command from a host computer using the information of the present apparatus. FIG. 3 shows a flowchart relating to the image processing mode described above.

This flow is started 31 by a reset input to the apparatus body or a power-on reset.
First, in the initialization mode 32, various data are initialized and, at the same time, data of the internal switch is read.

Next, if the camera calibration mode switch of the internal switch is set to “ON”, the camera mode determination section 33 executes “camera posture calibration mode” 35 to measure the camera posture parameters. Then, it is held in the camera posture parameter holding means 22.

If the camera calibration mode switch is "OFF", the process of "camera posture parameter change mode" 34 is performed. In the "camera posture parameter change mode" 34, the camera mounting direction information is checked from the status information of the camera posture detection means 24, and the camera posture parameter information stored therein corresponding to the information is read out, and the camera posture parameter holding is performed. It is held in the means 22.

At this time, if there is no status in the status information, that is, in the case of a camera apparatus without the camera attitude detecting means 24, the camera attitude parameter information is read out from the camera mounting direction setting switch information of the internal switch, and the camera attitude parameter information is read. It is held in the posture parameter holding means 22.

Next, various operation mode judging sections 36, 38,
At 41, the “posture detection operation mode” 44, “preprocessing data use / posture detection operation mode” 37, 44, “fixed mark detection operation mode with internal fixed mark” 43, “registration” Marked Fixed Mark Detection Operation Mode "
Select and execute one operation mode.

However, it is not necessary to support all the modes, such as the operation mode and the camera mode, as in this embodiment, and it is also possible to configure a device that operates only a certain mode alone.

Further, in the present embodiment, an example for measuring the movement of the operator's head has been described. However, the present invention is not limited to this, and the movement of the operator's other body part is detected. Can also be used to

FIG. 6 is a diagram exemplifying a system in which an HMD display device having a built-in head detecting device, which is a moving body posture position detecting device according to the present invention, is incorporated in a portable computer.

In FIG. 6, reference numeral 60 denotes an HMD display device with a built-in head posture detecting device, 61 denotes a mode switch,
70 is a microphone, 71 is a speaker, 72 is a signal cable, 73 is a graphic control system such as a PC card, and 74 is a portable computer.

FIG. 9 is a diagram exemplifying a case in which the camera 62, which is a means for capturing a peripheral image, is directed upward from the operator. The present invention described in the above-described embodiments includes the inventions described as the following supplementary notes in addition to claims 1 to 3 described in the claims.

Additional Notes 1. The attitude position determination means corrects the angle information from the angle information detection means using the angle information from the image analysis means, and translates the translation position information from the translation position information detection means from the image analysis means. The attitude position detecting device according to claim 1 or 2, wherein the angle information and the translation position information of the measurement object are determined by performing correction using the position information.

Appendix 2 A posture / position detection device that is attached to or built into the measurement target and moves with the movement of the measurement target to detect the posture / position information of the measurement target. Image information acquisition means, analyzing the image information, image analysis means for detecting angle information representing the inclination of the device, without using the image information, angle information detection means capable of detecting angle information, A posture position detecting device comprising: a posture position calculating unit that calculates angle information of a measurement target using the angle information from the image analyzing unit and the angle information from the angle information detecting unit. .

Appendix 3 A posture detection device that is attached to or incorporated in the measurement target, moves with the movement of the measurement target, and detects posture information of the measurement target. Image information acquisition means, image analysis means for analyzing the image information and detecting translational position information indicating the position movement of the apparatus, and translational position information capable of detecting translational position information without using the image information Detection means, and translation position information from the image analysis means, and attitude position calculation means for calculating the translation position information of the measurement object using the translation position information from the translation position information detection means. And a posture / position detecting device.

[0200] Appendix 4 Angle detecting means for calculating three-axis attitude angle information on the space of the moving body in a device for detecting the attitude of the moving body, and translation position detecting means for calculating three-axis translational position information on the space of the moving body Image processing attitude detecting means installed on the moving body itself and calculating six-axis image attitude information of translation and attitude of the moving body from an image around the moving body; and attitude angle information from the angle detecting means When,
Spatial attitude calculation means for calculating 6-axis attitude information of the moving body in space based on translational position information from the translational position detection means and image attitude information from the image processing attitude detection means. A moving body posture detecting device, comprising:

[0220] Appendix 4 Corresponds to the first embodiment. According to the present invention, the information of the angle detecting means and the translational position detecting means for detecting the relative posture change and the translational displacement information in the space without the reference point is provided in the space with respect to the moving object to be measured. By associating information with the image processing attitude detection means that can have an absolute reference point in
The attitude with respect to the absolute reference point in space can be obtained.

Appendix 5 The angle detecting means includes an angular velocity detecting means for detecting angular velocity information of a space on three orthogonal axes, and a first attitude calculating means for calculating attitude angle information of the moving body from the angle information. The position detecting means includes acceleration detecting means for detecting acceleration information in a translational direction of a space on three orthogonal axes, and second attitude calculating means for calculating tilt information and translation information of the moving body from the acceleration information. The image processing attitude detecting means includes an image inputting means for inputting an image around the device in time series, and a plurality of feature points extracted from the image input by the image inputting means. Feature point detecting means for searching for and detecting feature point information such as coordinates of feature points previously stored, and detecting a motion vector of the image from a displacement of the feature point in the two-dimensional image You A motion vector detecting unit, and a third posture calculating unit configured to calculate image posture information in a rotation direction and a translation direction of a moving body from the posture angle information, the translation information, and the feature point information, Means for changing information from the motion vector detecting means and information for changing the attitude calculating method of the first attitude calculating means and the attitude calculating method of the second attitude calculating means by the angular velocity detecting means and the acceleration detecting means. Supplementary note 4 characterized by comprising stop determination control means for sending, and correction calculation means for calculating information from each of the first attitude calculation means, the second attitude calculation means and the third attitude calculation means. 4. A moving body posture detection device according to claim 1.

According to the present invention, the angular velocity and the velocity data of the intermediate data when the angle is detected by the angular velocity detecting means and when the translation position is detected by the acceleration detecting means can be used as auxiliary data for image processing. It has the effect of reducing the calculation load of image processing, and also has the effect of reducing the processing load as compared with a device using only image processing, so that high-speed processing is possible and a low-speed processing system can be used. Further, by determining the stop state by comparison with a motion vector in image processing, gyro drift or the like can be corrected.

Further, as another effect, the relative attitude to the moving body can be detected even in the moving body. Appendix 6 The attitude angle information of the angular velocity detector and the translation information from the acceleration detector are used as six-axis attitude information of the moving body attitude detector, and the attitude angle information and the displacement information are used. 3. The image processing apparatus according to claim 3, wherein the image orientation information of the image processing orientation detection means, which is calculated at a lower speed, is used as correction information for reference reference of the 6-axis orientation information. Or 5. 4. A moving body posture detection device according to claim 1.

According to the present invention, information from the angular velocity detecting means and the acceleration detecting means, which is faster than the processing time of the attitude calculation by the image processing and has a small calculation load, is used as the six-axis attitude information of the mobile body attitude detection apparatus. The information from the image posture detecting means is used as the reference reference information obtained from the fixed space to correct the posture information, whereby the six-axis posture information can be detected at high speed, and the image posture detection can be performed. The error can be corrected by periodically correcting the reference point information by the means.

Appendix 7 The image processing apparatus according to claim 4, further comprising an angular velocity detecting unit, the acceleration detecting unit, and an image vector estimating unit that stores camera posture parameter information that is an attachment relationship between an image input direction of the image input unit and an attachment position coordinate. . Or 5. 4. A moving body posture detection device according to claim 1.

According to the present invention, other information such as the motion of a feature point or the motion of an image vector determined by vertebral determination based on the angular velocity information or angle information from the angular velocity detecting means and the acceleration information or translation information from the acceleration detecting means. By calculating information used in some processes in advance, it is possible to reduce the calculation load.

[Supplementary Note 8] The feature point detection means includes: a feature point information of the feature point detection means at the time when the image input means inputs an image; a posture angle information by the first posture calculation means; Is stored as a set of feature point posture information, and the feature point posture information at different positions with respect to the same feature point (the translation information having a value equal to or greater than a certain threshold value) is preferentially stored. The third posture calculating means has a feature point posture information storage function;
3. The characteristic point is obtained by approximating the coordinates of the characteristic point from the origin in the fixed coordinate space from the characteristic point posture information of the points or more. To 7. 4. A moving body posture detection device according to claim 1.

According to the present invention, when the coordinate information for the same feature point is obtained, the accuracy increases as the translation information increases, and the error can be reduced as the number of measurement data increases, and the fixed point space can be reduced. It is possible to reduce the error of the coordinate information obtained in the above. Further, since the feature point information obtained from the two-dimensional image processing includes the posture parameter information of the camera, the feature point information is converted to information from the fixed coordinate origin. By converting and storing, the process when comparing the attitude angle information of the angular velocity detecting means or the translation information from the speed detecting means with the reference reference information is simplified, and the processing speed is increased.

Supplementary Note 9 The correction calculating means is configured to execute the third
The image attitude information obtained by the approximate correction by the attitude calculation means is used as reference reference information, and the attitude angle information in the first attitude calculation means and the translation information in the second attitude calculation means match the reference reference information. The correction value is not corrected while the operation is stopped, and the correction range is adjusted by the value of the change data of the operation data during the operation. 4. A moving body posture detection device according to claim 1.

According to the present invention, it is possible to reduce errors due to image processing and errors due to information from angular velocity detecting means and acceleration detecting means by obtaining reference reference information from information obtained by approximate interpolation from a plurality of measurement points. The attitude angle information of the angular velocity detecting means or the translation information from the velocity detecting means, and the error correction is performed in such a manner as described above.
The image does not move freely regardless of the operator's intention when observing a typical display, and the error is canceled during operation, making it difficult for the operator to understand the movement due to the correction. This has the effect.

Supplementary Note 10. The feature point detecting means may detect the feature points at the same position (the translation information having a value equal to or less than a certain threshold value) at different angles (the translation information having a value equal to or greater than a certain threshold value) with respect to the same feature point. The attitude information is used as reference reference information of the angle, and the correction calculating means corrects only the attitude angle information so as to match the reference information of the angle. 4. A moving body posture detection device according to claim 1.

According to the present invention, in the VR-like virtual display system, since the movement of the rotation center is more than the movement by the translation, the data efficiency is deteriorated only by the data when the translation information is large, but the translation is zero. Since the angle information at the time becomes the reference information of the angle information of the gyro information as it is,
The arithmetic processing can be performed at high speed.

Supplementary Note 11. 6. A moving position and a rotation angle of a feature point of the captured image data are estimated based on information from the image vector estimating means. 11. A moving body posture detection device according to any one of claims 10 to 10.

According to the present invention, it is possible to predict a matching area for searching for a feature point even if the camera attachment state changes, to reduce the number of two-dimensional scans and image processing operations such as affine transformation, and to use a Kalman filter. The feature point processing can be easily speeded up as compared with the method of predicting the search area according to the above.

Supplementary Note 12 An image around the moving object is input in advance, feature point posture information of the feature point detection means at that time is stored, and thereafter, feature point extraction is performed from information of the first posture calculation means and the second posture calculation means. Without performing the processing of
In addition to using the stored information, the third attitude calculating means obtains reference reference information approximately corrected to the coordinates of the feature point from the origin in the fixed coordinate space based on the plurality of stored feature point attitude information and stores the information. Appendix 7 characterized in that:
12. A mobile body posture detection device according to any one of claims 11 to 11.

According to the present invention, the coordinates of the feature point in the fixed space are measured in advance in the mode for obtaining the coordinate information while the drift or error of the information from the angular velocity detecting means or the acceleration detecting means is small. Can be specified, the accuracy can be further improved, and the extraction process can be simplified, so that the calculation load in the image processing can be further reduced, and the speed can be further increased.

Supplementary Note 13. The motion vector detecting means,
Additional characteristic 7 is that only the motion vector based on the estimation with the information of the image vector estimating means is selectively output from the processed information. 13. A moving body posture detecting apparatus according to any one of claims to 12.

According to the present invention, the reliability is improved by using gyro data as a determination condition when a motion vector due to a moving object other than a fixed object exists in an image. Appendix 14 When a feature point is not detected by the feature point detection means, or when the information of the motion vector detection means is equal to or less than a threshold value of the similarity with the motion vector according to the information of the image vector determination means, it is determined that detection is impossible. 14. The moving body posture detection device according to claim 13, wherein the information is not used in the third posture calculation unit and the correction calculation unit in the post-processing.

According to the present invention, when a motion vector cannot be measured, for example, when a moving object crosses right in front of a camera or when an image in which a feature point such as a wall or a ceiling cannot be found is input, only gyro data By doing so, it is possible to reduce the conditions under which the measurement is not performed as compared with a device using only image processing, and it is possible to always continue to detect the posture under any conditions.

Supplementary Note 15. 15. The moving body posture detecting device according to claim 14, further comprising: a unit that notifies a user of the information when the motion vector detecting unit determines that the motion vector cannot be detected.

According to the present invention, when the detection becomes impossible for the above-mentioned reason, the user can be informed of the image processing state by notifying the user, and it is possible to reduce situations and factors that cause errors due to gyro drift and the like. it can.

[Supplementary Note 16] Supplementary note 4, further comprising a camera mounting direction changing means capable of changing an input direction of the image input means. 13. A moving body posture detecting apparatus according to any one of claims to 12.

According to the present invention, by directing the camera in a direction suitable for detection in the current environment, it is possible to direct the camera in a direction with good conditions by image embedding with a large amount of fixed image information with few moving objects.

Supplementary Note 17. 17. The moving body posture detection device according to claim 16, further comprising a camera posture parameter holding unit that notifies the changed camera mounting direction information to the image vector estimation unit.

According to the present invention, it is possible to set a posture parameter in a case where the image is oriented in a direction in which an image having good conditions for measurement can be obtained under the environment in use. Appendix 18 A camera attitude detecting means for detecting that the input direction of the image input means has been changed and transferring the information to a camera attitude holding means;
8. The moving body posture detecting device according to 7.

According to the present invention, by incorporating a tilt sensor or the like into the camera unit, the mounting direction can be automatically detected and the setting can be automatically performed, so that the burden on the user can be reduced.

Supplementary Note 19. A camera attitude parameter is calculated based on the information of the first attitude calculation means, the information of the second attitude calculation means, and the information of the third means attitude calculation means when the moving body is moved, and the information is calculated by the camera. 19. The moving body posture detecting apparatus according to claim 18, further comprising a camera posture parameter calculating unit that sends the posture information to the posture parameter holding unit.

According to the present invention, a method having a higher degree of freedom can be used for the camera mounting direction changing means, the degree of freedom for setting the camera direction can be increased, and the calibration can be performed in any direction. Then, it can be used, errors such as individual differences can be eliminated, and a camera posture detecting means is not required.

[Supplementary Note 20] Fixed mark information storage means for pre-stored characteristic points for a specific mark; and fixed mark instruction means for instructing to perform template matching only for the specific mark. Appendix 4. 20. A moving body posture detecting apparatus according to any one of claims to 19.

According to the present invention, since the feature point extraction processing can be omitted, it is possible to reduce the calculation load in the image processing, to achieve higher speed, and to increase the number of storage memories. Since it is not required, the system configuration can be simplified.

Appendix 21. 21. The moving body posture detecting device according to claim 20, wherein the image data input by the image input means is registered in the fixed mark information storage means.

According to the present invention, the feature point extraction processing can be omitted, so that the calculation load in image processing can be further reduced, the speed can be further increased, and the stability by image processing can be improved. When it is desired to obtain a specific signal, it is possible to improve accuracy by arranging easily identifiable markers, etc., and it is possible to simplify the system configuration because a large amount of storage memory is not required. Without having to create a special mark or installing special marks, images in the operating environment can be registered as marks, increasing the degree of freedom in designing the surrounding environment within the operating environment, making it more convenient Can be improved.

Appendix 22. 3. An infrared light emitting means arranged to irradiate an image input range of the image input means. To 21. 4. A moving body posture detection device according to claim 1.

According to the present invention, image input processing can be performed even at night or in a dark place.
The device can be used even in an environment where lighting equipment is not used.

Appendix 23. 3. An additional feature of the present invention is that the image input means to the feature point detecting means and the motion vector detecting means are constituted by artificial retinal chips. To 22. 4. A moving body posture detection device according to claim 1.

According to the present invention, the processing system for the arithmetic processing can be constituted by a low-cost and general-purpose processing system. It is possible to reduce the price, speed, and power consumption.

[0238]

Therefore, as described above, according to the present invention, it is possible to provide a posture / position detecting device which can be used favorably in any environment. Further, according to the present invention, it is possible to provide a posture / position detecting device which can be used even with a computer having a low processing capacity.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a configuration of a first exemplary embodiment of the present invention.

FIG. 2 is a block diagram illustrating a configuration during a camera posture correction operation in the first embodiment of the present invention.

FIG. 3 is a flowchart showing the flow of various modes relating to image processing in the first embodiment of the present invention.

FIG. 4 is a diagram illustrating, as an example of a VR system, a state in which a desktop image of a computer is virtually displayed in a space in front of an operator;

FIG. 5 is a diagram illustrating three-dimensional space coordinates in which the forward direction of the operator is a + X axis, the left direction is a + Y axis, and the upward direction is a + Z axis;

FIG. 6 is a diagram showing, as an example of implementation of the present invention, a system in which an HMD display device having a built-in head detecting device, which is a mobile posture position detecting device according to the present invention, is incorporated in a portable computer.

FIG. 7 is a diagram illustrating a case where the camera is installed so that a photographing direction is directed forward with respect to a head in the first embodiment according to the present invention, and a camera 62 is attached to the HMD 60; It is a figure which shows a case in an image.

FIG. 8 is a diagram illustrating a relationship between a position of a camera with respect to a head movement posture and a projection image plane that is a camera input image and a fixed point Po that is a feature point in space according to the first embodiment of the present invention. It is a figure showing the relationship.

FIG. 9 is a diagram exemplifying a case in which a camera 62, which is a means for imaging a peripheral image, is directed to an overhead direction of an operator in the first embodiment of the present invention.

FIG. 10 is a diagram showing an arrangement relationship between an angular velocity sensor and an acceleration sensor as the angular velocity detecting means 11 and the acceleration detecting means 26 in the first embodiment of the present invention.

FIG. 11 is a block diagram of a main part showing a configuration of the first exemplary embodiment of the present invention.

[Explanation of symbols]

 100: Angle detection means, 11: Angular velocity detection means, 15: First attitude calculation means, 101: Translation position detection means, 26: Acceleration detection means, 27: Second attitude calculation means, 102: Image processing attitude detection means, 17 ... image input means, 18 ... feature point detection means, 19 ... motion vector detection means, 20 ... third attitude calculation means, 103 ... spatial attitude calculation means, 16 ... stop determination control means, 21 ... correction calculation means, 60 ... HMD Reference numeral 62 62 Camera 10 Image vector estimating means 18a Feature point extraction unit 18b Feature point storage unit 18c Feature point / posture storage unit 23 Camera change direction hand, step 24 Camera orientation detection means , 25 ... Camera attitude parameter calculation means.

Claims (3)

[Claims] 1. A posture position detection device that is mounted on or built into a measurement object, moves along with the movement of the measurement object, and detects posture position information of the measurement object. Image information acquiring means capable of acquiring image information by means of: comparing and analyzing the image information at different points in time acquired by the image information acquiring means to obtain angle information representing the inclination of the apparatus and translation representing the position movement of the apparatus Image analysis means for detecting position information, angle information detection means capable of detecting angle information without using the image information, and translation position information detection means capable of detecting translational position information without using the image information And angle information and translational position information from the image analysis means,
Attitude position determining means for determining angle information and translational position information of the measuring object using the angle information from the angle information detecting means and the translational position information from the translational position information detecting means. A posture and position detection device characterized by the above-mentioned. 2. The method according to claim 1, wherein the angular information detecting means detects angular velocity information on three axes orthogonal to each other in space, and calculates first angle information of the apparatus from the angular velocity information.
The translational position information detecting means, wherein the translational position information detecting means detects acceleration information in the translation direction on three axes orthogonal to each other in space; and the angle information and the translational position information of the apparatus from the acceleration information. And a second posture / position calculating means for calculating the following. The image analyzing means includes: an image inputting means for inputting a captured image in a time series; a feature point extracted from the input image; A feature point detecting means for recording feature point information including the feature point information, and a third attitude position for calculating angle information and translational position information of the apparatus from the displacement of the position of the feature point; Calculating means for determining whether or not the apparatus is stopped based on the output of the angular velocity detecting means or the acceleration detecting means, and calculating the first attitude position based on the determination. A stop determining means for outputting information to modify the operation method of the operation method or the second orientation position calculation unit stages,
2. The attitude according to claim 1, further comprising: a correction arithmetic means that calculates information from the first attitude position calculating means, the second attitude position calculating means, and the third attitude position calculating means. 3. Position detection device. 3. A posture position detecting device which is attached to or built into a measurement object, moves along with the movement of the measurement object, and detects posture position information of the measurement object. Detecting means for detecting image information, image acquiring means capable of acquiring image information by imaging the periphery of the device, feature point extracting means for extracting a feature point from the image information acquired by the image acquiring means, and the image acquiring means Posture position calculation means for calculating second posture position information from a change in the position of the feature point between image information obtained at different points in time, by correcting the posture position information with the second posture position information And a posture position determining means for determining posture position information.