JP4422777B2 - Moving body posture detection device - Google Patents

Description

  The present invention relates to a mobile body posture detection device that detects the posture of a mobile body in a three-dimensional space, and in particular, detects the posture of a human head equipped with a head mounting display (HMD) and based on the information. In a virtual reality (VR) system that generates an image and displays the image as if there is a virtual space in front of the eyes, a system that can be used anywhere without limiting the location is provided. The present invention relates to a self-supporting mobile body posture detection device.

  Conventionally, the following devices are known as posture position detection devices.

  First, there is a reference position installation type posture position detection apparatus in which a transmitter serving as a reference point for light, ultrasonic waves, magnetic fields, or the like is provided at a location around an operator and a position corresponding to the transmitter is detected.

  On the other hand, a system using an angular velocity sensor such as a gyro has been developed as a prior art of a self-supporting posture position detection device.

  Since the gyro sensor outputs an angular velocity with respect to the rotation of the gyro itself, the displaced relative angle can be obtained by integrating this information.

  Therefore, by arranging the gyro sensors on three orthogonal spaces, the rotational posture position with respect to this space can be obtained.

  The output of these gyro sensors is output as a positive or negative signal with respect to the direction of rotation around a reference voltage, but the reference offset signal drifts due to changes in temperature and humidity, or externally The characteristics change due to the magnetic field, and the output signal itself fluctuates due to vibrations such as sound.

  However, since the difference of the output value with respect to the offset signal is always integrated and used to obtain the angle, if these drifts and noise are included, the angle information that is the calculation result has a large error. It will be accumulated.

  Therefore, conventionally, countermeasures for canceling these drift signals have been proposed.

  One of them is a method of canceling a gyro drift component by using information from a direction sensor as an absolute direction.

  As this direction sensor, a system using a GPS or a geomagnetic sensor has been developed.

  This is disclosed in, for example, Patent Documents 1 and 2 and the like.

  GPS is a system that identifies the position of GPS itself while catching signals from several satellites, but cannot be used in places where radio waves are difficult to reach, such as in buildings or tunnels.

  On the other hand, the geomagnetic sensor determines the magnetic strength with respect to the direction of the earth's generally north-facing geomagnetic vector based on the information output by the MR elements arranged in two orthogonal axes in the horizontal plane with respect to the earth's geomagnetic vector. Since it can be detected, the azimuth direction can be obtained from the two signals of each MR element.

  That is, assuming that the magnetic strengths are X and Y, the azimuth (θ) can be obtained from equation (1).

θ = arcTan (Y / X) (1)
Incidentally, the geomagnetic vector has an error called a declination with respect to a small geographical north that is unique to each region on the earth.

  In a posture detection device used for a car navigation system or the like, these must be corrected, but in a virtual display system for a VR system, it is only used as reference orientation information in a certain local space. Absolute direction accuracy is not required.

  Therefore, by comparing the displacement of the azimuth information and the displacement angle information of the gyro, an error due to a gyro drift or the like can be detected.

  Also, if the angle data, which is gyro accumulation information, is reset, the initial state is set, and if the geomagnetic vector at that time is measured, the absolute position can be corrected by comparison with the gyro accumulation result. .

  However, the problem with the geomagnetic sensor is that the compass by the geomagnetic sensor must be measured in a horizontal plane.

  Like the declination described above, the geomagnetic vector has an inclination called the elevation angle in the vertical plane that is unique to each region on the earth, and around Tokyo, it is about 48 degrees below the north. ing.

  This inclination is a factor that occurs as an orientation error when the compass is inclined from the horizontal plane.

  In systems like car navigation, the car is basically moving in a horizontal plane, so it can be used as it is.

  However, in the VR system, it is necessary to detect even a tilted state in order to measure a three-dimensional posture of a human.

  FIG. 4 shows an example of a VR system.

  This is an image of a virtual display of a computer desktop image in the space in front of the operator.

  Here, when the operator turns to the right, the screen on the right side of the desktop image appears on the HMD. When the operator turns to the opposite side, the screen on the left side appears. When the operator turns further up and down, the screen in that direction appears. You can feel as if there is a large desktop space in front of you.

  In FIG. 4, reference numeral 51 is a virtual screen, 52 is a real screen, 53 is an icon, 54 is a window, and 60 is an HMD.

  FIG. 5 shows the coordinate axes of the three-dimensional space at this time.

  That is, FIG. 5 shows three-dimensional space coordinates in which the forward direction of the operator is the + X axis, the left direction is the + Y axis, and the upward direction is the + Z axis.

  Also, the rotation direction of the right screw with respect to each axis is defined as + roll (Roll), + pitch (Pitch), and + yaw (Yaw).

  When the operator turns left and right to view the left and right screens, the Yaw direction changes. However, when the head is moved up and down to see the vertical direction of the screen, the Pitch direction changes.

  Further, as a function corresponding to the gimbal mechanism, there is a software gimbal mechanism method in which an inclination angle is measured by an acceleration sensor and a geomagnetic sensor azimuth output (θ) is corrected from the inclination information.

  As a device combining an acceleration sensor with a gyroscope and a geomagnetic sensor, there is a conventional example disclosed in Patent Document 3.

  However, in this case, three axes are required for the magnetic sensor, and the output for each axis is X.X. If Y and Z are assumed, and the inclination with respect to the horizontal plane at this time is Roll and Pitch, a general correction equation is expressed by equation (2).

  Here, Xh and Yh are values obtained by correcting the strength of the magnetic vector in the horizontal plane.

Xh = X * cos (Roll) + Y * sin (Roll) * sin (Pitch) -Zcos (Roll) * sin (Pilch)
Yh = Y * cos (Roll) + Z * sin (Roll)
θ = arcTan (Yh / Xh) (2)
The acceleration sensor has a gravitational acceleration component (G · cos (θg)) corresponding to the length projected perpendicularly to the axis of the detection direction when the sensor is inclined (θg) with respect to the gravity vector G, and In addition, a motion acceleration component due to acceleration / deceleration motion occurs.

  In order to cancel this motion acceleration component, it is necessary to perform a filtering process.

  By adopting such a configuration, it is theoretically possible to obtain a posture in a three-dimensional space without contradiction.

  The acceleration sensor is used as an inclination sensor that uses the output of the inclination information with reference to the gravity of the earth.

  Further, as a method for detecting the movement of a moving body by image processing, there is a conventional example disclosed in Patent Document 4.

Conventionally, a method of measuring the posture of a moving body by arranging a special mark or light source in a space around the moving body and detecting a motion vector by image processing is proposed.
JP-A-5-71964, JP-A-4-14934 JP-A-8-178687 JP-A-6-89342

  However, when the geomagnetic sensor is used for the posture position detection of FIG. 5 in order to realize the VR system of FIG. 4, the azimuth output is inclined from the horizontal direction, resulting in an error.

  Therefore, in general, geomagnetic sensors try to reduce errors by installing a mechanical gimbal mechanism for maintaining a horizontal state, but mounting a mechanical mechanism on a detection device is smaller and lighter. It becomes a difficult point in terms of conversion.

  In addition, the geomagnetic sensor is sensitive to external disturbances, is susceptible to magnetic field distortion by nearby irons, magnetized objects, buildings, etc., and has an absolute orientation such that the geomagnetic sensor itself has an error due to an external magnetic field. There are many problems with using it as a sensor.

  However, in applications that are used only in a certain local area, there is no problem even if the absolute orientation is different, but there is a motor, CRT, etc. nearby, and other magnetized objects only move. However, the magnetic field is disturbed and it is necessary to pay attention to the surrounding environment even if it is used only as a relative reference orientation.

  In addition, the geomagnetic sensor has problems such as a slow reaction speed.

  Furthermore, even a device in which an acceleration sensor is combined with a gyroscope and a geomagnetic sensor still has problems as the magnetic sensor as described above.

  In addition to the above problems, there are major problems in use.

  That is, the direction sensor indicates a relative value with respect to the absolute direction of the geography on the earth. For example, when the user is in a train or a car, the direction of the vehicle changes with respect to the earth even if the user does not move. In addition, the posture will change without permission.

  In a portable computer system that uses a display screen using VR spatial display technology, an image cannot be fixed in front of the user in such an environment, and the image can be displayed regardless of the user's will. Will move without permission.

  In addition, as described above, when the acceleration sensor is used as an inclination sensor on the earth, since humans always act against gravity, there is no problem in the horizontal state. If you try to use it in a space where gravity does not reach or if you try to use it in a space where gravity does not reach, you will not be able to use it.

  In the prior art as a self-supporting sensor for detecting the posture of a moving body as described above, in order to correct the drift of the gyro sensor, the orientation of the geomagnetic sensor serving as a relative or absolute reference is used as reference information. In this case, there is a problem that the reference information is shifted due to the influence of magnetic disturbance or an error caused by the inclination with respect to the horizontal plane.

  In addition, GPS cannot be used in places where radio waves are difficult to reach, such as in buildings or tunnels, and direction sensors and acceleration sensors are used only in places that are affected by gravity and the magnetic field of the earth because they are based on gravity and the earth's magnetic field. I can't.

  Furthermore, when trying to use the vehicle in a moving object such as a vehicle, the display moves freely regardless of the operator's will to detect the posture of the vehicle itself, not the posture of the operator. .

  Further, in the detection of the posture of the moving object by image processing, measurement cannot be performed if an operation that cannot be performed due to conditions such as feature points of the captured peripheral image occurs.

  In addition, in the conventional method of measuring the posture of a moving object by image processing, in order to always obtain a position in a space where a reference mark as a feature point is located by comparison with a feature point detected from a previous image In the state where there is no overlap between the previous image and the next image, the motion vector cannot be detected, and the feature point must be extracted again to move to the next image input process. Since the limit of the motion detection speed is determined by the speed performance, a high-speed image processing system is required to perform image processing corresponding to human movement.

  As described above, in image processing, conventionally, there are many so-called heavy calculation processes that require a device capable of high-speed calculation processing, and a computer with very high processing capacity is required. There are many problems to do.

  The present invention has been made paying attention to the above-described problems, and an object of the present invention is to provide a mobile body posture position detection device that can be used satisfactorily in any environment.

  Another object of the present invention is to provide a moving body posture position detecting device that can be used even with a computer having a low processing capacity.

According to the present invention, in order to solve the above problems,
In an apparatus that is attached to or built in a moving object as a measurement object, moves along with the movement of the moving object, and detects posture information of the moving object.
Angle detection means for calculating three-axis posture angle information on the space of the moving body;
Translation position detection means for calculating translation position information of three axes on the space of the moving body;
Image processing posture detection means installed on the moving body itself and calculating six-axis image posture information of translation and posture of the moving body from images around the moving body;
The six-axis posture of the moving object in space based on the posture angle information from the angle detection means, the translation position information from the translation position detection means, and the image posture information from the image processing posture detection means. A spatial attitude calculation means for calculating information,
The angle detection means includes
Angular velocity detection means for detecting angular velocity information of a space on three orthogonal axes;
A first posture calculating means for calculating posture angle information of the moving body from the angular velocity information;
The translation position detection means includes
Acceleration detecting means for detecting acceleration information in the translation direction of a space on three orthogonal axes;
A second posture calculating means for calculating translation position information including tilt information and translation information of the moving body from the acceleration information;
The image processing posture detection means is
Image input means for inputting images around the device in time series;
A feature that extracts a plurality of feature points from the image input by the image input means, stores the feature points, and searches for feature point information that is the coordinates of the feature points stored previously. Point detection means;
Calculating the angular velocity information from the angular velocity detecting means, and acceleration information of the translational direction from said acceleration detecting means, and a characteristic point information from the feature point detection unit, the image orientation information of the rotation direction and the translation direction of the moving body And a third posture calculating means.
Based on the angular velocity information from the angular velocity detecting means and the acceleration information in the translation direction from the acceleration detecting means, the two-dimensional motion of the feature point on the image data is estimated, and the search detection is performed based on this information. A moving body posture detecting device is provided which is characterized by narrowing the range.

  Therefore, as described above, according to the present invention, it is possible to provide a moving body posture detection device that can be used satisfactorily in any environment.

  Further, according to the present invention, it is possible to provide a moving body posture detecting apparatus that can be used even with a computer having a low processing capacity.

That is, according to the present invention, the information from the angular velocity sensor and the acceleration sensor, not only to use the posture detection of the moving body, by skillfully diverted this information, high-speed operation by narrowing the range of the search detection Can also be realized.

  Embodiments of the present invention will be described below with reference to the drawings.

  FIG. 1 and FIG. 11 are block diagrams of the whole and main parts showing the configuration of the first embodiment according to the present invention.

  FIG. 2 is an overall block diagram showing the configuration during the camera posture correction operation.

  FIG. 3 shows the flow of various modes related to image processing.

  That is, as shown in FIG. 1 and FIG. 11, the angle detection means 100 for calculating the three-axis posture angle information on the space of the moving body as the measurement object includes three pieces arranged in the three-axis space orthogonal to each other. An angular velocity detecting means 11 constituted by an angular velocity sensor and a first posture calculating means 15 for calculating posture angle information of the moving body from the angular velocity information from the angular velocity detecting means 11.

  Further, the translation position detecting means 101 for calculating the translational position information of the three axes in the space of the moving body includes the acceleration detection means 26 for detecting the acceleration information in the translational direction of the space on the three orthogonal axes, and the acceleration detection means. 26 includes a second posture calculation means 27 for calculating the tilt information and translation information of the moving body from the acceleration information from the reference numeral 26.

  Further, the image processing posture detection means 102 that is installed on the moving body itself and calculates six-axis image posture information of translation and posture of the moving body from images around the moving body projects the surrounding image onto the imaging surface. An image input unit 17 configured to include a camera unit and an image sensor (CCD or the like) for converting the projected image into a video signal, and inputs the image in time series, and is input from the image input unit 17. A plurality of feature points are extracted from the obtained image, the feature points are stored, and at the same time, feature point detection means 18 for searching for and detecting the coordinates of the previously stored feature points, and feature points by the feature point detection means 18 A motion vector detecting means 19 for detecting a motion vector of the image from the displacement in two dimensions, and calculating the posture of the moving body in the rotational direction and translational direction from the angular velocity information, the acceleration information, and the feature point information. 3rd figure Constituted by the calculating means 20.

  Further, movement in space is performed by information from each of the attitude angle information from the angle detection means 100, the translation position information from the translation position detection means 101, and the image attitude information from the image processing attitude detection means 102. The spatial posture calculation means 103 for calculating the six-axis posture information of the body is the posture of the first posture calculation means 15 based on the information from the motion vector detection means 19 and the information from the angular velocity detection means 11 and the acceleration detection means 26. The stop determination control means 16 for sending information for changing the calculation method and the attitude calculation method of the second attitude calculation means 27, the first attitude calculation means 15, the second attitude calculation means 27, and the third attitude calculation means 20, respectively. It is comprised with the correction calculating means 21 which calculates the information from.

  FIG. 10 shows the positional relationship between the angular velocity sensor 11 and the acceleration sensor as the angular velocity detector 11 and the acceleration detector 26 in the moving body posture position detector configured as described above.

  Here, as the angular velocity sensor, a piezoelectric vibration gyro sensor or the like is arranged and used as shown in FIG. 10 for each of the three axes of X, Y, and Z.

  The acceleration sensor is a small electromechanical sensor made by micromachine technology, etc., one is a biaxial device that detects acceleration in the X-axis and Y-axis directions, and the other is in the Y-axis direction. A single-axis type device capable of detecting the above is arranged and used as shown in FIG.

  And since the gyro sensor is arrange | positioned in the space on three orthogonal axes | shafts, it outputs the angular velocity with respect to the rotational motion around each arrange | positioned axis | shaft, and a code | symbol changes with the rotation directions around the axis | shaft.

  Expression (3) shows a relational expression for obtaining the angle (θ) from the gyro output (Vg).

  An offset, which is a reference output at rest, is held for each gyro, and an offset (Vo) and a differential signal of the gyro output become an angular velocity signal (ω), and an angle is obtained by integrating this.

  As a result of this calculation, a relative posture displacement in space is obtained.

ω = As · (Vg + Vo + Vt + Vn)
θ = Σ (ω) (3)
Here, As is a gyro scale factor including a time (Δt) factor, Vt is a fluctuation component, and Vn is a noise component.

  Further, the offset, which is the reference output, varies depending on a single unit, and needs to be stored and corrected for each single unit.

  Further, it fluctuates due to temperature change and the fluctuation component (Vt) and noise component (Vn) become error components.

  Therefore, since these error components are integrated into the gyro angle information, the value fluctuates even in the stopped state, and thus appears as an output drift error.

  Therefore, the differential output is subjected to low-pass filtering (LPF) to cut a high frequency (fh) such as a noise component (Vn) that is an error component or a vibration component applied to the apparatus, and the first attitude calculation means 15 performs the first operation. Use as information.

  At the same time, the second information cut at a lower frequency (fl) that is a drift component or the like is obtained as an offset value, and an error can be reduced by taking a difference from the second information.

  However, if the cut-off frequency (fl) of the low-pass filter (LPF) at this time is set high, the effect of reducing large drifts can be obtained, but the operation signal due to the original posture change is also cut off. , It stops responding to slow movements.

  Therefore, the low-pass filter information is stored as an offset only when it is determined by the stop determination control means 16 described later that the present apparatus is stopped.

  By doing so, the cut-off frequency (fl) of the low-pass filter can be set low, it is possible to detect even a slower movement, and fluctuation due to drift is reduced.

  The acceleration sensor can be used in two ways. One is a function as a tilt sensor using the gravitational acceleration of the earth's gravity, and the other is a motion acceleration sensor function generated by inertial motion.

  By integrating this motion acceleration information, velocity information is obtained, and by further integrating, position information is obtained.

  For example, when considering an acceleration sensor in the X-axis direction, the gravitational acceleration component (Gx) contributing to the X-axis acceleration sensor at this time, the X-axis acceleration sensor output (Va), the acceleration sensor scale factor (As), X The inclination of the axial acceleration sensor with respect to the horizontal plane, that is, the rotation angle (θ) around the Y axis, the gravity G of the earth (1G = 9.8 m / s), and the stationary output (Vo).

Gx = G × sin θ
Va = Gx × As−Vo
θ = sin ⁻¹ {(Va−Vo) / As} (4)
As an inclination sensor, it can obtain | require by Formula (4).

  However, at this time, if the acceleration sensor is in a motion state, information obtained by adding the gravitational acceleration and the motion acceleration is obtained as the acceleration information. Therefore, low-pass filtering is performed so that the resulting information passes through the low frequency component. By doing so, a gravitational acceleration component can be obtained.

  Therefore, it can be used to correct the drift component of the angle information of the gyro data based on this information.

  Therefore, although the tilt information by the acceleration sensor is statically absolute reference data, it cannot be used to detect a moving body in a motion state or a vehicle.

  Further, at this time, the gyro information that can be corrected is only the rotation around the X axis and the Y axis, and the information around the Z axis that is the rotation direction with respect to the gravity axis cannot be corrected.

  As for the second motion acceleration sensor function, the acceleration information (α) obtained by translational motion is tilted (θ) with respect to the horizontal plane in the same manner as described above by the calculation of equation (5). Think about the case.

α = {(Va−Vo) / (As−Gsin θ)} / cos θ (5)
The angle information (θ) at this time can be obtained by using angle information obtained from the Y-axis angular velocity sensor (without using the calculation information from the tilt sensor function).

  In order to detect the position information, the motion acceleration information may be integrated twice, whereby the position information can be obtained.

If the above calculation is decomposed into the motion acceleration components (α _x , α _y , α _z ) and the gravitational acceleration components (G _x , G _y , G _z ) with respect to the X, Y, and Z axes, the three-dimensional operation It is possible to obtain acceleration information, velocity information, and position information in the translation direction in space.

  However, since the integral type calculation is performed here as in the case of the gyroscope, the position information includes an error.

  By the way, since the motion acceleration is not applied in the stop state, only the angle information obtained by using the gravitational acceleration is used. However, when the stop state is not set, the value of the motion acceleration is treated as significant.

  In the present invention, posture position information represents one or both of angle information and translation position information.

  Next, returning to FIG. 1, the operation, function, and processing of each unit will be described.

"Image input means 17"
The image input means 17 converts an optical projection image obtained through the camera optical system into a video signal of a dimensional image by a CCD as an image sensor, and stores this video signal in a frame memory.

  The image input means 17 performs preprocessing such as contour extraction on the video signal of the two-dimensional image, and transfers the time-series image data to the feature inspection means 18.

  In this way, the image input means 17 inputs time-series image data obtained by taking an image of the periphery of the moving object with the movement of the moving object as the measurement object.

  In this case, as a peripheral image captured by the camera, it is important to capture a fixed object having many feature points within the camera imaging angle of view.

  In addition, it is important to point the camera toward a feature point such as a ceiling or a wall so that a moving object that exists in the vicinity or an outside scenery when in a moving body such as a vehicle is not photographed.

  Thus, by considering the image input direction, it is possible to perform reliable image processing, and it is possible to correct motion information by an acceleration sensor or an angular velocity sensor.

"Feature point detection means 18"
First, the feature point detection unit 18 can easily distinguish the time-series image data input from the image input unit 17 from other regions by image processing in the feature point extraction unit 18a, and the same on the object. A plurality of feature points that can specify that a point has been photographed are extracted.

  Next, the feature point detection means 18 retains and stores the feature point data and coordinate values extracted by the feature point extraction unit 18a in the feature point storage unit 18b and simultaneously stores them in the feature point / posture information storage unit 18c. The feature point is tracked, its position is detected, and coordinate information on the two-dimensional image is acquired.

"Image correlation processing using gyro / acceleration data information"
In general, this is called two-dimensional correlation processing or template matching processing, and correlation processing is performed while scanning feature point data as a template in two-dimensional directions with respect to input image data.

  At this time, one of the problematic factors is rotation around the Z axis of the camera.

  For example, when the movement of the operator's head is measured by the HMD 60 shown in FIG. 7, when the input direction of the camera 62 for measuring the movement is attached to the front of the operator's head, Due to the occurrence of Roll, which is the rotation around the X axis when tilted to, a two-dimensional plane rotation occurs in the image taken by the camera 62.

  Similarly, as shown in FIG. 9, even when the direction of the camera 62 for motion measurement is attached upward, the movement of Yaw, which is rotation around the Z axis when the neck is moved left and right Will cause the image to rotate.

  When inter-field correlation of images is performed, correlation processing must be performed after rotating the pattern that is the gray value on the screen, and affine transformation processing for rotation is processed for each pattern of each feature point data In addition, since the angle is repeatedly changed and correlation processing is performed, a very large number of calculations are required.

  Therefore, this processing is one of the most heavy processing in image processing.

  Therefore, based on the attitude angle information from the angular velocity sensor and the translation information from the acceleration sensor, the two-dimensional motion of the feature point on the image data can be estimated by the image vector estimation means 10 described later. Therefore, the search range can be narrowed based on this information, and the correlation process can be performed more efficiently.

  In other words, by using the angle information calculated from the gyro, it is only necessary to perform correlation processing between the affine transformation of a smaller angle range and the range of the predicted two-dimensional coordinate position of the feature point by adding the acceleration information.

  Thus, by using the gyro sensor information and the acceleration sensor information, it is possible to significantly reduce the processing load.

“Feature Point / Attitude Storage Unit 18c”
Next, the feature point storage unit 18b and the feature point / posture storage unit 18c in the feature point detection unit 18 store the positions of the feature points with respect to the fixed space coordinates.

During initialization, it is exactly the same fixed coordinate space OXYZ the head coordinate system _{(O h X h Y h Z} h coordinate system).

  Here, since the feature point is a coordinate of a fixed object around the moving body, it can be considered as an absolute reference point fixed in the scene of the surrounding image.

  Therefore, this feature point becomes an absolute reference point in the fixed coordinate space.

  FIG. 8 is a diagram illustrating the relationship between the position of the camera with respect to the head movement posture, the relationship between the projection image plane that is the camera input image, and the fixed point Po that is the feature point in the space.

  Here, the Po point viewed from the camera coordinates is (Xc, Yc, Zc).

  The projection relational expression between the three-dimensional space coordinates and the point Q (x, y) on the image plane is the expression (6).

x = fXc / Zc,
y = fY / Zc (6)
Furthermore, since the translation vector of the head position is obtained from the translation information from the acceleration sensor, the three-dimensional position in space can be estimated from this information and the two-dimensional coordinates of the same feature point between the two images.

  The coordinate data on the two-dimensional coordinates of each feature point extracted by the feature point extraction unit 18a is stored in the feature point storage unit 18b.

  Furthermore, feature points rich in changes such as gray values are stored in the next feature point / posture information storage unit 18c.

  That is, the feature point / posture information storage unit 18c includes reference point reference information for storing the feature point unique number and the feature point (Xo, Yo, Zo) coordinates for the fixed coordinate space OXYZ in the form of internal information. The fixed point time series data number, the feature point position information estimated for each time series data number, the posture information from the angular velocity sensor at that time, and the translation information from the acceleration sensor are stored.

  Here, the number unique to the feature point is a number unique to each feature point extracted from the background image, and is managed so that at least eight points are always obtained in one angle of view of the input image.

  The feature point time-series data number is a number for managing feature point position information for each unique feature point.

  This feature point position information does not store all time-series input data, but preferentially stores position information above a certain threshold value of translation information, which is position information when the image is captured. To do.

  For example, information on the same feature point Po from various directions that are not from the same point when the moving body moves is stored.

  Further, when the value of the feature point position information is greatly different or cannot be detected, the series data at that time is discarded without being stored.

  If this state continues for a certain period of time, it is determined that the feature point is not a fixed object, and all data is discarded.

  If a unit direction vector from the viewpoint Oc toward the feature point Po is defined as m, the feature point position information includes a vector m viewed from the camera origin Oc, angle information from the angular velocity sensor, and translation information from the acceleration sensor. Remembered.

"Third attitude calculation means 20"
Next, in the third posture calculation means 20, since the distance information to the feature point can be obtained by the principle of the stereo camera when the feature point position information becomes two points or more, the distance information is obtained as reference standard information. Remember.

  In addition, when the feature point position information reaches a certain threshold or more, the third posture calculation means 20 obtains the position information from the origin O by the least square method and stores it as reference standard information.

  This reference standard information is managed so that the feature point processing can be converted into the origin coordinates and used by calculating and storing the direction vector Mi viewed from the origin O and the distance Pi from the origin O to Po.

  Expression (7) represents a condition for deriving an optimal solution from the feature point information of the two screens by the least square method.

  Here, M represents a unit direction vector when the feature point is viewed from the origin O in the fixed coordinate space, and i (= 1... N) is a feature point unique number.

Σ | mi h Rmi | ² > min (7)
mi is a unit direction vector at a certain point in time, and is obtained by equation (6).

  Rmi represents a vector after movement that has been R-rotated with respect to the mi point, and is obtained from angle information from the angular velocity sensor.

  H represents a translation vector from mi to Rmi, and is obtained from translation information from the acceleration sensor.

  Therefore, this reference standard information is approximated by the least square method based on each posture information including errors between the angle information from the angular velocity sensor and the translation information from the acceleration sensor, and information on multiple feature points on the image. Information.

  Therefore, by further performing this process for each time series data and averaging each of the unique reference standard information, errors due to the angle information from the angular velocity sensor and the translation information from the acceleration sensor can be reduced. it can.

  When this reference standard information is interpolated with a number equal to or greater than a certain threshold, it is no longer stored in the time-series data of the feature point position information, and all the feature point position information is discarded.

  The above processing is also performed in the same procedure for newly detected feature points.

  Further, as a condition for storing the feature point position information, the position information of the translation information, which is the position information when the image is captured, is preferentially stored in a position equal to or greater than a certain threshold. This feature point position information is directly used when the translation information is zero (actually below a certain threshold).

Under this condition, since the fixed coordinate space OXYZ and the head coordinate system (O _h X _h Y _h Z _h coordinate system) match, the rotational posture information obtained by the equation (6) of the two-dimensional image remains as an angle. It can be used to correct posture information from the sensor.

"Motion vector detection means 19"
The motion vector detection means 19 detects motion information of an image for each field by the feature point matching method, and detects a motion vector from the information.

  The feature point matching method is to obtain a motion vector for each field from the inter-field correlation between some pixels selected with the gray value on the screen as a representative point and the surrounding pixels.

  That is, the motion vector detection means 19 is a motion vector distribution on a two-dimensional image (xy coordinates) based on time-series data of two-dimensional coordinate values of some feature points obtained by the feature point detection means 18. Perform the process.

  Further, the motion vector detecting means 19 divides vector regions having the same direction and size from the motion vector distribution.

"Stop determination control means 16"
The stop determination control means 16 is within a threshold value where there is motion of the surrounding image from the motion vector information from the motion vector detection means 19, and the first information from the LPF output of the gyro data is also below a certain threshold value. If it is determined that the state has continued for a certain period of time, it is determined that the measured moving body is stopped, and the information of the offset holding means 12 of the gyro output processing system is updated with the second information from the current LPF output. To do.

  The same determination condition is also applied to the second attitude calculation unit 27 that calculates information from the acceleration detection unit 26.

  However, the difference from the gyro data is that the data when it is determined that the vehicle is stopped is determined to be inclination data and is processed.

  Therefore, when the measured moving body is stopped, no error due to drift occurs. However, when the state where it is not stopped continues for a long time, the error due to drift is added to the output of the first attitude calculation means 15. There is a problem.

“Camera posture parameter holding means 22”
Next, the mounting position and direction of the camera 62 will be described.

  The two-dimensional image analysis result in the third posture calculation means 20 is a coordinate system with respect to the camera coordinate system.

  However, since the desired motion is a spatial motion of the operator's head, it is necessary to obtain the relationship between the head motion coordinate system and the camera coordinate system.

  FIG. 7 shows an image in which the camera 62 is attached to the HMD 60 in a case where the camera 62 is installed so that the photographing direction is directed forward with respect to the head.

Figure 8 shows the relationship between the head coordinate system with the fixed coordinate system OXYZ space _{(O h X h Y h Z} h coordinate system) and the camera coordinate system (OcXcYcZc coordinate system).

The rotation position of the head relative to a fixed coordinate system OXYZ coordinate system, then _{y h} rotated around _{Z h-axis,} then _{p h} rotated around _{Y h-axis} after rotation, the rotation of _{r h} around _{X h} axis after rotation the rotation matrix for obtaining the posture of when given as R _h, also defines the translational movement base vector as P _h.

Further, the posture of when given rotation rc head _{O h X} h _Y h _Z h coordinate yc the camera rotation position around Zc axis relative system, pc around Yc axis after rotation around Xc axis after rotation Is defined as Rc, and a translation vector is defined as Pc.

In the head coordinate system, the neck portion that is the base point of motion with respect to the fixed coordinate system is defined as the origin O _h (X _h Y _h Z _h ), and the position of the camera origin with respect to the head coordinate system is defined as Oc. It is defined as (Xc, Yc, Zc).

Further, if the position of the point Po as viewed from the camera coordinate system is (Xc, Yc, Zc), the point Po in the space of the fixed coordinate system is expressed as in the equation (8) with respect to the head and the posture of the camera 62. (However, Sx and Cx represent sinX and cosX).

  The first term on the right side of the relational expression (8) is the head movement parameter, and the second term is the camera posture parameter.

  The camera posture parameters are held in the camera posture parameter holding means 22 and are known in advance.

  Therefore, it is possible to convert the motion vector obtained from the camera image from the above relational expression into the motion of the head in the fixed space.

  Conversely, it is possible to predict the movement of the image feature point from the acceleration information and the angular velocity information, which are head movement information.

  In the “posture detection operation mode” 44 shown in FIG. 3, the posture of the moving body to be measured is changed by substituting the motion parameter obtained by the calculation processing in the third posture calculation means 20 into the second term on the right side of the equation (8). It can be obtained.

"Correction calculation means 21"
The third attitude calculation means 20 obtains reference standard information including an attitude angle converted into a vector from the origin O and translation information by calculation by image processing.

  However, since this process takes more time than a process using a gyroscope or an acceleration sensor, basically, the translation position information from the first attitude calculation means 15 and the rotation angle information from the second attitude calculation means 27 are included. It is used as posture position calculation information of the mobile body posture position detection apparatus according to the present invention.

  However, the correction calculation means 21 converts the angle information of the angular velocity sensor and the translation information of the acceleration sensor into a vector from the origin O, and converts the attitude angle information including the drift of the gyro data and the translation information from the acceleration sensor at this time. This is used to correct the difference between the reference standard information and the obtained vector as an error component between the angular information of the angular velocity sensor and the translation information of the acceleration sensor.

  When an error (ε) with respect to the reference reference information of the third attitude calculation means 20 occurs with respect to the information of the first attitude calculation means 15 and the second attitude calculation means 27, the value is corrected to the value of the third attitude calculation means 20. Therefore, the difference is added so as to be corrected again slowly.

  For example, regarding the gyro data, the width of the correction value at this time is obtained by multiplying the latest angular velocity (ωt) from the first attitude calculation means 15 by the coefficient (k) until the error (ε) becomes zero. Keep doing.

  This is shown in equation (9).

θt = θ _t−1 + ωt + ε · k (9)
At this time, if (k) is ωt = 0, then k = 0
If 0>ωt> Tmax, k = 0.5,
If ωt> Tmax, k = 1.

  Therefore, no error correction is performed even if an error occurs in the stop state without any change in posture.

  However, in the operating state, the width for canceling the error is increased according to the angular velocity.

  By doing so, the image is not moved freely regardless of the operator's intention while observing the VR-like display, and the error is canceled during the operation, so that the operator can correct the error. It has the effect of making it difficult to understand the movement caused by.

  The same processing is performed for translation information.

"Image vector estimation means 10"
The image vector estimation means 10 uses gyro angular velocity information and acceleration information from the acceleration sensor instead of the camera attitude parameter information of the camera attitude parameter holding means 22 and the head movement parameter in the camera attitude calculation formula (8). By substituting each into the equation, the movement of the camera 62 can be predicted, and the movement position of the feature point can be predicted from the movement information of the camera 62.

  For the feature point detection means 18, the affine transformation angle information by the image rotation or the coordinate estimation information of the feature point matching area, and for the motion vector detection means 19, either the motion vector itself or the image area. It can be used as determination information for removing motion data and vector data that cause an error factor such as a moving object such as a moving object such as a vehicle outside and a scenery outside the vehicle.

  Used in several other processes such as motions of feature points or image vectors obtained by estimation from angular velocity information or angular information from the angular velocity detection means 11 and acceleration information or translation information from the acceleration detection means 26 By calculating the information to be performed in advance, the calculation load can be reduced.

`` Unable to measure and warning display ''
If the image input unit 17 captures the feature point detection unit 18 such as a wall or a ceiling with few feature points, a motion vector cannot be measured when an image in which the feature point cannot be found is input.

  Also, when a moving object crosses just in front of the camera 62, or when an operator is in a moving body such as a vehicle and only an outside scene is captured, a motion vector detecting means by image processing is used. In 19, a motion vector greatly different from the angle information by the gyro data and the acceleration information from the acceleration sensor is detected.

  In addition, even when the past feature points are lost in the updated image data due to the rapid movement exceeding the processing capability of the image arithmetic processing system, the detection becomes impossible as described above.

  In such a case, it is determined that the measurement is impossible, and the correction calculation unit 21 calculates only the angle calculation information based on the gyro data and the acceleration information from the acceleration sensor, and sets the calculation mode in which the correction based on the image processing calculation information is not performed.

  Therefore, in the present invention, the posture can always be detected under any condition.

  Then, when the feature point memorize | stored last time is extracted, correction | amendment of attitude | position information is restarted with the positional information at that time.

  However, such a state incapable of being measured continues for a long time, which causes an error due to gyro drift or acceleration sensor integration.

  Therefore, the present invention has a function of issuing a warning to the operator when such a state continues for a certain period.

  For example, a method for issuing a warning by light such as a buzzer sound or LED, a method for displaying the status in the video information of the HMD, and status information for the host system that is inputting the information of the posture position detection device. The operator can be notified as error information.

"Camera mounting direction change means 23 related"
As for the mounting direction of the camera 62, it is basically based on taking a forward image when the HMD 60 is mounted.

  However, as shown in FIG. 2, the camera mounting direction changing means 23 capable of changing the camera mounting direction is provided so that the camera cannot be used under the condition that the measurement impossible state as described above occurs. Yes.

  For example, by directing the camera 62 to the top of the operator's head or to the left or right by the camera mounting direction changing means 23, an image having the best measurement conditions can be obtained in the environment in which the camera 62 is used. It becomes possible to set the direction.

  Further, for the switching of the camera mounting direction, since the camera posture parameter holding means 22 stores posture parameter data for each mounting direction in advance, parameter setting from a changeover switch (not shown) or a host system is performed. The setting may be switched by a command or the like.

  Further, by incorporating the camera posture detection means 24 in the camera 62, the switching operation can be automatically performed.

  This is a method of detecting the mounting direction depending on the mounting position of the camera 62 on the coupling surface between the camera mounting direction changing means 23 and the main body of the HMD 60 by means of a junction detector such as a micro switch or a Hall element.

  Alternatively, by mounting the semiconductor acceleration sensor, which is the acceleration sensor, on the inside of the camera 62, it can be used as an inclination sensor that knows the mounting inclination angle of the camera 62.

  Therefore, the camera posture detection unit 24 can automatically detect that the mounting direction has been changed by the camera mounting direction changing unit 23 and can automatically change the posture parameter data of the camera posture parameter holding unit 22. Thus, the burden on the user can be reduced.

"Camera parameter automatic calibration function"
The functions related to the camera mounting direction up to the above can be used to change the camera direction and its posture parameters very simply by incorporating the measured camera posture parameters into the system in advance.

  However, an individual difference of the camera position with respect to the image input means 17 having the camera mounting direction changing means 23 having a higher degree of freedom or the base position of the neck that is the origin of the movement of the operator (for example, the size of the head) Error in the posture parameter due to the difference in the length of the neck).

  Therefore, the camera posture parameter calculation unit 25 has a function of measuring the mounting posture and position, which are camera posture parameters, for each user and storing them in the camera posture parameter holding unit 22.

  By obtaining the motion vector obtained from the image processing when the known motion data is given to the relational expression (8) of the camera parameter calculation, the posture parameter of the camera mounting position can be obtained automatically.

  The camera parameter automatic calibration function can be performed by a simple operation as follows.

  First, in camera posture calibration, it is a condition that an image as far as possible is captured as far as an input image or the like.

  Then, with the camera 62 mounted in a state in which the device is actually used, the user performs a rotational motion such as swinging up and down, left and right and sideways without performing translational motion as much as possible. Calibration is possible simply by inputting an image.

  In the “camera posture calibration mode” 35 shown in FIG. 3, the first posture calculation means 15 and the second posture calculation means 27 perform normal operations and calculations.

  The camera attitude parameter calculation means 25 samples several points of attitude data based on gyro data, translation information from the acceleration sensor, and image processing data obtained by the third calculation means 20.

  The feature point matching at this time is processed in a mode in which the motion prediction of the feature point is not performed because the camera posture parameter is not determined.

  By substituting these results into the relational expression (8) of the camera posture parameter calculation, it is possible to reversely calculate the camera posture parameter by the least square method.

  When the calibration is completed, this posture parameter is stored in the camera posture parameter holding means 22 and thereafter used for calculation such as “posture detection operation mode” 44 of the normal operation shown in FIG.

  Therefore, a more flexible method can be used for the camera mounting direction changing means 23, and the degree of freedom of the camera direction is also increased, and the calibration as described above is performed in any direction. By doing so, the apparatus can be used in a good state, and errors such as individual differences can be eliminated.

  In this case, the camera posture detection means 24 and the like are not necessary.

"Feature point detection and storage means by pre-processing input"
The moving body posture position detection apparatus includes a mode for storing feature points by preprocessing as another feature.

  That is, in the “feature point preprocessing mode” 37 shown in FIG. 3, first, the head movement in the range normally used is performed.

  Similar to the “posture detection operation mode” 44 that is normally performed, the posture detection operation by the gyro sensor, the acceleration sensor, and the image processing is performed.

  The feature points extracted at this time are also stored in the feature point / posture information storage unit 18c in the feature point detection means 18 together with the posture information.

  Further, reference standard information is also calculated and stored in the feature point / posture information storage unit 18c.

  Therefore, not only the feature points in the currently captured image but also the feature points and posture information of the surrounding images in the environment are stored and held in the feature point / posture information storage unit 18c in association with each other.

  Then, in the “posture detection operation mode” 44 thereafter, the feature point information in the camera image is extracted from the feature point / posture information storage unit 18c from the posture information, and the motion vector is detected using the information.

  Alternatively, correction can be performed by extracting the reference standard information from the feature point / posture information storage unit 18c.

  Therefore, since the processing of the feature point extraction unit 18a of the feature point detection means 18 can be simplified, it is possible to reduce the calculation load in image processing and to increase the speed.

  In addition, since the measurement regarding the feature point information in the “feature point preprocessing mode” 37 is completed in a short time, the occurrence of drift of the angular velocity sensor and the acceleration sensor can be reduced, and the error of the feature point information can be reduced. Can be reduced.

  Further, the “feature point preprocessing mode” 37 is an effective usage method when used in a fixed environment where, for example, the user sits on a chair in a certain place and works.

"Fixed mark detection operation mode"
As another feature, there is a “fixed mark detection operation mode” 43 shown in FIG.

  This “fixed mark detection operation mode” is more than the case of the “feature point preprocessing storage mode” 37 described above, in which the place where the individual uses is always fixed or used in a room in the amusement facility. Is also used when used in a fixed environment.

  This is a mode in which an image pattern serving as a feature point is stored in advance, and correlation processing is performed only on this pattern.

  Fixed marks, which are image patterns, are arranged around the environment so that eight or more images can always be captured within the camera angle of view.

  Normally, this fixed mark is placed at random, avoiding periodic placement, but it also includes information from the gyro sensor and acceleration sensor. This arrangement is more flexible than the detection device.

  Therefore, since the processing of the feature point extraction unit 18a of the feature point detection means 18 can be omitted, it is possible to reduce the calculation load in image processing and to increase the speed.

  Further, since a large amount of storage memory such as the “feature point preprocessing storage mode” 37 is not required, the system configuration can be further simplified.

"Fixed mark registration mode 42"
Further, as another feature, in the “fixed mark detection operation mode” 43, an image pattern serving as a feature point is stored in advance, but this “fixed mark registration mode” shown in FIG. 42 is different from the aforementioned “fixed mark detection operation mode” 43 in that the operator inputs a mark input from the image of the camera 62 and performs the “fixed mark detection operation mode” 43 on the mark. ing.

  In the “fixed mark registration mode” 42, the image input from the image input means 17 is converted into feature point data and stored internally.

  However, it is necessary to prevent images other than those registered as fixed marks from appearing in the input image at this time.

  Therefore, in the “fixed mark registration mode” 42, since the processing of the feature point extraction unit 18a of the feature point detection means 18 can be omitted as described above, it is possible to reduce the calculation load in image processing. As a result, higher speed can be achieved.

  In the “fixed mark registration mode” 42, when a stable signal is obtained by image processing, it is possible to improve the accuracy by arranging a marker that is easy to identify.

  Furthermore, in this “fixed mark registration mode” 42, since a large amount of storage memory is not required as in the “feature point preprocessing storage mode” 37, the system configuration can be simplified.

  Furthermore, in this “fixed mark registration mode” 42, it is possible to produce a mark by an individual or register an image in the use environment as a mark, which increases the degree of freedom in designing the surrounding environment in the use environment. Convenience can be improved.

"Infrared light emitting means"
Furthermore, as another feature, this apparatus includes infrared light emitting means (not shown) such as an infrared LED arranged to irradiate the image input range of the image input means 17.

  The image input means 17 is not a problem when inputting an image in which the surroundings are illuminated by natural light or lighting equipment. However, when the user wants to use the apparatus in an environment where no lighting equipment is used, such as at night or in a dark room, By making the infrared LED provided on the apparatus side emit light, it can be used even in such an environment.

  In this case, the infrared LED is made to emit light in synchronization with the image input processing of the image input means 17, and the light emission is stopped otherwise.

  In this way, by repeatedly turning on and off, unnecessary power consumption is avoided and low power consumption is achieved.

  Further, when a distant light pattern such as a night view can be used, the infrared LED does not emit light.

"Artificial retina chip"
Further, as another feature, from the image sensor for converting the projection image of the image input means 17 into a video signal in the above configuration, the video signal is accumulated as image data, and the image data is further subjected to contour extraction, Functions such as processing of the feature extraction unit 18a, pattern matching processing, and adjustment of the gain level of the video input signal are configured by one artificial retina chip.

  This artificial retina chip is an image sensor (for example, an artificial retina IC manufactured by Mitsubishi Electric Corporation) with a built-in image arithmetic processing function and an analog adjustment function, and currently processes and outputs a grayscale image of 128 × 128 pixels. Can do.

  The optical system of the camera 62 has an angle of view of 30 °, and 30/128 = 0.23 ° with respect to the direction of the rotation angle, so that 0.23 ° per pixel, and the analog grayscale image signal is interpolated. Further, higher resolution can be achieved.

  As a result, it can be sufficiently used as reference point data without drift.

  Therefore, the processing system for the subsequent arithmetic processing can be configured with a general-purpose processing system at a lower price.

  By using this artificial retina chip for the image input means 17, it is possible to reduce the size, cost, speed, and power consumption of the system.

  In the above description, various settings relating to image processing and mode changes can be automatically performed by setting internal switches (not shown) in the apparatus and apparatus status information.

  This can also be changed by a setting command from a host computer that uses the information of the apparatus.

  FIG. 3 is a flowchart related to the image processing mode described above.

  This flow is started 31 by a reset input or power-on reset to the apparatus main body.

  First, in the initialization mode 32, initialization of various data is performed, and at the same time, data of the internal switch is read.

  Next, if the camera calibration mode switch of the internal switch is set to “ON”, the camera mode determination unit 33 executes “camera posture calibration mode” 35, measures the camera posture parameter, This is held in the camera posture parameter holding means 22.

  If the camera calibration mode switch is “OFF”, the processing of “camera posture parameter change mode” 34 is performed.

  In this “camera posture parameter change mode” 34, the camera mounting direction information is checked from the status information of the camera posture detection means 24, the camera posture parameter information stored therein corresponding to the information is read, and the camera posture parameter is held. Hold in the means 22.

  If the status information does not have a status at this time, that is, if the camera device does not have the camera posture detection means 24, the camera posture parameter information is read by the camera mounting direction setting switch information of the internal switch, and the camera posture parameter is held. Hold in the means 22.

  Next, in the various operation mode determination units 36, 38, 41, the “attitude detection operation mode” 44, “preprocessing data use / attitude detection operation mode” 37, 44, “internally fixed” are set by the operation mode setting switch of the internal switch. The above-mentioned five operation modes are selected and executed: “Fixed mark detection operation mode using marks” 43 and “Fixed mark detection operation mode using registered marks” 42 and 43.

  However, the operation mode, the camera mode, and the like do not need to support all the modes as in this embodiment, and can be configured as a device that operates only in a certain mode.

  Further, in the present embodiment, the example for measuring the head movement of the operator has been described, but not only the use limited to this, but also for detecting the movement of other body parts of the operator. Can also be used.

  FIG. 6 is a diagram exemplifying a system in which an HMD display device incorporating a head detecting device which is a mobile body posture position detecting device according to the present invention is incorporated in a portable computer.

  In FIG. 6, reference numeral 60 is an HMD display device with a built-in head posture detection device, 61 is a mode switch, 70 is a microphone, 71 is a speaker, 72 is a signal cable, 73 is a graphic control system such as a PC card, and 74 is portable. Computer.

  FIG. 9 is a diagram exemplifying a case where the camera 62 which is a peripheral image capturing unit is directed in the overhead direction of the operator.

  The present invention shown in the embodiment as described above includes the invention shown as the following supplementary notes in addition to the claim 1 shown in the claims.

  Appendix 1. The posture position determination means corrects the angle information from the angle information detection means using the angle information from the image analysis means, and translates the translation position information from the translation position information detection means from the image analysis means. The posture position detection apparatus according to claim 1, wherein angle information and translational position information of the measurement object are determined by correcting using position information.

Appendix 2. In a posture position detection device that is attached to or built in a measurement object, moves with the movement of the measurement object, and detects posture position information of the measurement object.
Image information acquisition means capable of acquiring image information by imaging the periphery of the device;
Image analysis means for analyzing the image information and detecting angle information representing the inclination of the apparatus;
Angle information detection means capable of detecting angle information without using the image information;
Posture position calculation means for calculating angle information of the measurement object using angle information from the image analysis means and angle information from the angle information detection means;
A posture position detecting device comprising:

Appendix 3. In a posture detection device that is attached to or built in a measurement object, moves with the movement of the measurement object, and detects posture information of the measurement object.
Image information acquisition means capable of acquiring image information by imaging the periphery of the device;
Image analysis means for analyzing the image information and detecting translational position information representing position movement of the device;
Translation position information detecting means capable of detecting translation position information without using the image information;
Posture position calculation means for calculating the translation position information of the measurement object using the translation position information from the image analysis means and the translation position information from the translation position information detection means;
A posture position detecting device comprising:

Appendix 4. In an apparatus for detecting the posture of a moving body, angle detection means for calculating three-axis posture angle information in the space of the moving body;
Translation position detection means for calculating translation position information of three axes on the space of the moving body;
Image processing posture detection means installed on the moving body itself and calculating six-axis image posture information of translation and posture of the moving body from images around the moving body;
6-axis attitude information of the moving body in space based on information of the attitude angle information from the angle detection means, the translation position information from the translation position detection means, and the image attitude information from the image processing attitude detection means. A spatial attitude calculation means for calculating
A moving body posture detection apparatus comprising:

  Appendix 4 Corresponds to the first embodiment.

  According to this invention, there is no reference point, and the information on the angle detection means and the translation position detection means for detecting relative posture change and translational displacement information in the space is measured on the space to be measured. By associating the information with the image processing posture detecting means capable of having an absolute reference point at the position, the posture with respect to the absolute reference point in space can be obtained.

Appendix 5. The angle detection means includes
Angular velocity detection means for detecting angular velocity information of a space on three orthogonal axes;
A first posture calculating means for calculating posture angle information of the moving body from the angle information;
The translation position detection means includes
Acceleration detecting means for detecting acceleration information in the translation direction of a space on three orthogonal axes;
A second posture calculating means for calculating tilt information and translation information of the moving body from the acceleration information;
The image processing posture detection means is
Image input means for inputting images around the device in time series;
A feature that extracts a plurality of feature points from the image input by the image input means, stores the feature points, and searches and detects feature point information such as the coordinates of the feature points stored previously. Point detection means;
Motion vector detection means for detecting a motion vector of the image from a displacement in the two-dimensional image of the feature point;
The posture angle information, the translation information, and third feature calculation means for calculating the image posture information of the rotation direction and translation direction of the moving body from the information of the feature points,
The space posture calculation means includes
Stop determination that sends information for changing the attitude calculation method of the first attitude calculation means and the attitude calculation method of the second attitude calculation means by the information from the motion vector detection means, the angular velocity detection means, and the acceleration detection means Control means;
Correction calculation means for calculating information from each of the first attitude calculation means, the second attitude calculation means, and the third attitude calculation means;
Supplementary note 4 characterized by comprising The moving body attitude | position detection apparatus of description.

  According to the present invention, the angular velocity and intermediate data velocity data when the angle is detected by the angular velocity detecting means and when the translation position is detected by the acceleration detecting means can be used as auxiliary data for image processing. It has the effect of reducing the processing load of processing, and the load of arithmetic processing is reduced compared to a device that only uses image processing. Therefore, high-speed processing is possible, and it can be configured with a low-speed processing system. Gyro drift or the like can be corrected by determining the stop state by comparison with a motion vector in image processing.

  In addition, as another effect, it is possible to detect the relative posture with respect to the moving body even in the moving body.

  Appendix 6. The attitude angle information of the angular velocity detection means (translated and processed at high speed) and the translation information from the acceleration detection means are used as 6-axis attitude information of the mobile body attitude detection device, and the attitude angle information and displacement information The image posture information of the image processing posture detection means that is processed at a lower speed is used as reference information correction information for the six-axis posture information. Or 5. The moving body attitude | position detection apparatus of description.

  According to this invention, information from the angular velocity detection means and the acceleration detection means, which is faster than the posture calculation processing time by image processing and has a small calculation load, is used as the six-axis posture information of the moving body posture detection device, The information from the posture detection means can be used as reference standard information obtained from the fixed space to correct the posture information, so that the six-axis posture information can be detected at high speed, and the reference by the image posture detection means. The error can be corrected by periodically correcting the point information.

  Appendix 7. Supplementary note 4 comprising image angle estimation means, acceleration detection means, and image vector estimation means for storing camera posture parameter information which is an attachment relationship between an image input direction of the image input means and attachment position coordinates. . Or 5. The moving body attitude | position detection apparatus of description.

  According to the present invention, the angular velocity information or angle information from the angular velocity detection means and the motion of feature points or the motion of image vectors obtained by the spine based on the acceleration information or translation information from the acceleration detection means. By calculating information used in the process in advance, it is possible to reduce the calculation load.

Appendix 8. The feature point detecting means includes
The feature point information of the feature point detection unit at the time when the image input unit inputs an image includes a set of posture angle information in the first posture calculation unit and translation information in the second posture calculation unit. A feature point orientation information storage function for preferentially storing feature point orientation information of different positions (the translation information having a value equal to or greater than a certain threshold) with respect to the same feature point Have
The third posture calculation means includes
3. Additional remarks obtained by approximating the coordinates of the feature points from the origin in the fixed coordinate space from the feature point orientation information of two or more points. To 7. The moving body attitude | position detection apparatus of description.

  According to the present invention, when obtaining coordinate information for the same feature point, the accuracy increases as the translation information increases, and the error can be reduced as the number of measurement data increases, and the feature point is obtained in a fixed space. It is possible to reduce the error of the coordinate information. Furthermore, since the feature point information obtained from the two-dimensional image processing includes the camera posture parameter information, the feature point information is converted into information from the fixed coordinate origin. By storing the information, the processing when comparing the attitude angle information of the angular velocity detection means or the translation information from the speed detection means with the reference standard information is simplified, and the processing becomes faster.

Appendix 9. The correction calculation means includes
The image posture information obtained by the approximate correction by the third posture calculation means is used as reference standard information, and the posture angle information in the first posture calculation means and the translation information in the second posture calculation means are the reference standards. Supplementary note 8, wherein the correction value is corrected so as to coincide with the information, and the correction value is not corrected when stopped, and the correction width is adjusted according to the value of the change data of the operation data during operation. The moving body attitude | position detection apparatus of description.

  According to this invention, it is possible to reduce errors due to image processing and errors due to information from the angular velocity detection means and the acceleration detection means by obtaining the reference standard information from information obtained by approximate interpolation from a plurality of measurement points. By detecting the attitude angle information of the detecting means and the translation information from the speed detecting means, and when correcting the error, the image moves freely regardless of the will of the operator when observing a VR-like display. In addition, since the error is canceled during the operation, there is an effect that it is difficult for the operator to understand the movement due to the correction.

Appendix 10. The feature point detecting means includes
Feature point posture information of different angles (those whose translation information has a value greater than or equal to a certain threshold) at the same position (those whose translation information has a certain threshold value or less) with respect to the same feature point Use as information,
The correction calculation means includes
Supplementary note 8. Only posture angle information is corrected so as to match the reference reference information of this angle. The moving body attitude | position detection apparatus of description.

  According to the present invention, in the VR virtual display system, the movement of the center of rotation is more than the operation by translation, so that the data efficiency is deteriorated only with the data when the translation information is large, but the angle when the translation is zero. Since the information becomes the reference standard information of the angle information of the gyro information as it is, the calculation process can be speeded up.

  Appendix 11. Note 7: The movement position and the rotation angle of the feature point of the captured image data are estimated based on the information from the image vector estimation means. The moving body attitude | position detection apparatus of thru | or 10.

  According to the present invention, it is possible to predict a matching region for feature point search even when the camera mounting state changes, reduce the number of two-dimensional scanning operations and image processing operations such as affine transformation, and search using a Kalman filter or the like. Feature point processing can be easily accelerated by the region prediction method.

Appendix 12. Input an image around the moving body in advance, store the feature point posture information of the feature point detection means at that time,
From the next time, using the stored information without performing feature point extraction processing from the information of the first posture calculation means and the second posture calculation means,
Based on a plurality of stored feature point posture information, the third posture calculation means obtains and stores reference standard information approximated to the coordinates of the feature point from the origin of the fixed coordinate space. Appendix 7. The moving body attitude | position detection apparatus of thru | or 11.

  According to the present invention, coordinate information can be specified while there is little occurrence of information drift and error from the angular velocity detecting means and the acceleration detecting means by measuring the coordinates of the fixed points of the feature points in a mode in advance. Therefore, the accuracy can be further improved and the extraction process can be simplified, so that the calculation load can be further reduced in the image processing, and the speed can be further increased.

Appendix 13. The motion vector detecting means includes
6. Only the motion vector obtained by estimation with the information of the image vector estimation means is selectively output from the processed information. The moving body attitude | position detection apparatus of thru | or 12.

  According to the present invention, reliability is improved by using gyro data as a determination condition when a motion vector due to a moving object other than a fixed object exists in an image.

  Appendix 14. When the feature point is not detected by the feature point detection unit, or when the information of the motion vector detection unit is equal to or less than a certain threshold with the motion vector based on the information of the image vector determination unit, it is determined that detection is impossible. 14. The mobile body posture detection apparatus according to appendix 13, wherein the information is not used for the third posture calculation unit and the correction calculation unit in post-processing.

  According to the present invention, when a motion object cannot be measured, such as when a moving object crosses immediately in front of the camera or an image in which a feature point such as a wall or ceiling cannot be found is input, only gyro data is used. As a result, it is possible to reduce the number of conditions that cannot be measured as compared with an apparatus using only image processing, and it is possible to always detect the posture under any conditions.

  Appendix 15. The moving body posture detecting apparatus according to claim 14, further comprising means for notifying a user of the information when the motion vector detecting means determines that the detection is impossible.

  According to the present invention, when detection becomes impossible due to the above-described reason, the user can be notified of the image processing state by notifying the user, and the situation and factors that cause errors due to gyro drift or the like can be reduced.

  Appendix 16. 3. Additional remarks comprising camera attachment direction changing means capable of changing the input direction of the image input means. The moving body attitude | position detection apparatus of thru | or 12.

  According to the present invention, by directing the camera in a direction suitable for detection in the current environment, the camera can be directed in a good condition by image processing with a large amount of fixed image information with few moving objects.

  Appendix 17. The moving body posture detection apparatus according to appendix 16, further comprising camera posture parameter holding means for notifying the image vector estimation means of the changed camera mounting direction information.

  According to the present invention, it is possible to set the posture parameter when the image is directed in a direction in which an image having good conditions for measurement can be obtained in the environment in use.

  Appendix 18. 18. The moving body posture detection apparatus according to appendix 17, further comprising camera posture detection means for detecting that the input direction of the image input means has been changed and transferring the information to the camera posture holding means.

  According to the present invention, by incorporating an inclination sensor or the like into the camera unit, the mounting direction can be automatically detected and the setting can be performed automatically, so that the burden on the user can be reduced.

  Appendix 19. A camera posture parameter is calculated from the information of the first posture calculation means, the information of the second posture calculation means, and the information of the third means posture calculation means when the moving body is moved, and the information is obtained from the camera. Item 19. The moving body posture detection apparatus according to appendix 18, further comprising camera posture parameter calculation means for sending to the posture parameter holding means.

  According to the present invention, a method with a higher degree of freedom can be used for the camera mounting direction changing means, and the setting of the camera direction can be further increased, and can be used if calibration is performed in any direction. In addition, it is possible to eliminate errors such as individual differences, and camera posture detection means is not required.

Appendix 20. Fixed mark information storage means for storing in advance a feature point for a specific mark;
Fixed mark instruction means for instructing to perform template matching only for the specific mark;
Additional note 4 characterized by comprising The moving body attitude | position detection apparatus of thru | or 19.

  According to the present invention, since the feature point extraction process can be omitted, it is possible to further reduce the calculation load in the image processing, and it is possible to increase the speed and not to require a large amount of storage memory. Therefore, the system configuration can be simplified.

  Appendix 21. The moving body posture detection apparatus according to appendix 20, wherein the image data input by the image input unit is registered in the fixed mark information storage unit.

  According to the present invention, since the feature point extraction process can be omitted, it is possible to reduce the calculation load in the image processing, and it is possible to further increase the speed and to provide a stable signal by the image processing. If you want to obtain a mark, it is possible to improve the accuracy by arranging easy-to-identify markers, etc., and because it does not require a lot of storage memory, the system configuration can be simplified. In addition, it is possible to register an image in the usage environment as a mark without installing a special mark, increasing the degree of freedom in designing the surrounding environment in the usage environment and improving convenience. be able to.

  Appendix 22. 3. An infrared light emitting unit arranged to irradiate an image input range by the image input unit. Thru 21. The moving body attitude | position detection apparatus of description.

  According to the present invention, image input processing can be performed even at night or in a dark place, so that the apparatus can be used even in an environment where no lighting equipment is used at night or in a dark place.

  Appendix 23. Additional remark 4 characterized in that the image input means to the feature point detection means and the motion vector detection means are constituted by an artificial retina chip. Thru 22. The moving body attitude | position detection apparatus of description.

  According to the present invention, it is possible to configure a processing system for arithmetic processing at a lower price and a general-purpose processing system. Furthermore, by adopting such a configuration, the system can be reduced in size, reduced in price, High speed and low power consumption can be achieved.

FIG. 1 is a block diagram showing the configuration of the first embodiment of the present invention. FIG. 2 is a block diagram illustrating the configuration during the camera posture correction operation in the first embodiment of the present invention. FIG. 3 is a flowchart showing the flow of various modes related to image processing in the first embodiment of the present invention. FIG. 4 is a diagram showing an image of a virtual display of a computer desktop image in a space in front of an operator as an example of a VR system. FIG. 5 is a diagram showing three-dimensional space coordinates in which the forward direction of the operator is the + X axis, the left direction is the + Y axis, and the upward direction is the + Z axis. FIG. 6 is a diagram showing a system in which an HMD display device incorporating a head detection device, which is a mobile body posture position detection device according to the present invention, is incorporated in a portable computer as an implementation example of the present invention. FIG. 7 is an image of the case where the camera 62 is attached to the HMD 60 when the camera is installed in the first embodiment according to the present invention so that the photographing direction is directed forward with respect to the head. FIG. FIG. 8 shows the relationship between the position of the camera with respect to the head movement posture and the relationship between the projected image plane as the camera input image and the fixed point Po as the feature point in the space in the first embodiment of the present invention. FIG. FIG. 9 is a diagram exemplifying a case where the camera 62 as the peripheral image capturing unit is directed in the overhead direction of the operator in the first embodiment of the present invention. FIG. 10 is a diagram showing the positional relationship between the angular velocity sensor and the acceleration sensor as the angular velocity detector 11 and the acceleration detector 26 in the first embodiment of the present invention. FIG. 11 is a block diagram of the main part showing the configuration of the first embodiment of the present invention.

Explanation of symbols

100: Angle detection means,
11: Angular velocity detection means,
15 ... 1st attitude | position calculation means,
101 ... Translation position detecting means,
26: Acceleration detecting means,
27. Second posture calculation means,
102: Image processing posture detection means,
17 ... image input means,
18 ... feature point detection means,
19 ... motion vector detection means,
20 ... Third posture calculation means,
103 ... spatial posture calculation means,
16 ... Stop determination control means,
21: Correction calculation means,
60 ... HMD,
62 ... Camera,
10: Image vector estimation means,
18a ... feature point extraction unit,
18b: Feature point storage unit,
18c: Feature point / posture storage unit,
23 ... Camera mounting direction change hand, Step 24 ... Camera posture detection means,
25: Camera posture parameter calculation means.

Claims (1)

In an apparatus that is attached to or built in a moving object as a measurement object, moves along with the movement of the moving object, and detects posture information of the moving object.
Angle detection means for calculating three-axis posture angle information on the space of the moving body;
Translation position detection means for calculating translation position information of three axes on the space of the moving body;
Image processing posture detection means installed on the moving body itself and calculating six-axis image posture information of translation and posture of the moving body from images around the moving body;
The six-axis posture of the moving object in space based on the posture angle information from the angle detection means, the translation position information from the translation position detection means, and the image posture information from the image processing posture detection means. A spatial attitude calculation means for calculating information,
The angle detection means includes
Angular velocity detection means for detecting angular velocity information of a space on three orthogonal axes;
A first posture calculating means for calculating posture angle information of the moving body from the angular velocity information;
The translation position detection means includes
Acceleration detecting means for detecting acceleration information in the translation direction of a space on three orthogonal axes;
A second posture calculating means for calculating translation position information including tilt information and translation information of the moving body from the acceleration information;
The image processing posture detection means is
Image input means for inputting images around the device in time series;
A feature that extracts a plurality of feature points from the image input by the image input means, stores the feature points, and searches for feature point information that is the coordinates of the feature points stored previously. Point detection means;
From the angular velocity information from the angular velocity detection means, the acceleration information in the translation direction from the acceleration detection means, and the feature point information from the feature point detection means, the image orientation information in the rotation direction and translation direction of the moving object is calculated. And a third posture calculating means.
Based on the angular velocity information from the angular velocity detecting means and the acceleration information in the translation direction from the acceleration detecting means, the two-dimensional motion of the feature point on the image data is estimated, and the search detection is performed based on this information. A moving body posture detecting device characterized by narrowing the range of the moving body.

Images