JP2011113561A - Methods for rendering graphical images - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide methods for defining the complexity and priority of graphics rendering in mobile apparatuses. <P>SOLUTION: The methods for defining the complexity and priority of graphics rendering in mobile equipment based on various physical states and factors related to the mobile system including those measured and sensed by the mobile equipment, such as position, pointing direction and vibration rate are disclosed and described. In particular, a handheld computing system with an image type user interface includes graphical images, generated in response to instantaneous position and orientation of a handheld device for improving the value of the presented image and the overall speed of the system. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  This application is a new application that does not depend on a previously filed provisional application. This application claims benefit and priority from a US provisional application having serial number 61 / 199,922, filed by British inventor Thomas Ellenby, San Francisco, Calif., On November 21, 2008.

  The disclosure of the invention below generally relates to computer graphic images rendered in dependence upon the physical state of a combined portable device.

  Modern computer systems are often used to generate highly dynamic images in response to user requests. Computerized device users can identify the parameters of interest, for example, in an interactive user interface manner, and indicate a desire for more information or specifically a desire for more certain types of information. Show. A great computer application known as “Google Earth” that gives images (a mix of photos and computer generated) in response to parameters that are fairly specified by the user's viewer is well known to many people. In the example, if a user wants to know the preferred location for scuba diving, he can simply re-draw the map by clicking on the appropriate user interface checkbox, Adjusted to include markers related to previously recorded preferred scuba diving locations. The level of detail of a map type graphical image is said to be responsive to the user's identification of various parameters.

  Military systems have been supplied for a long time to respond to desired targets within the gaze detection area. Certain radar systems, such as the Phalanx missile intercept system developed by General Dynamics, classify incoming targets by taking into account factors such as changes in target orientation. Targets with constant azimuth and closed range are consequently classified with respect to their range and approach speed. The target that can reach the ship earliest is classified as “most important” and is dealt with preferentially.

  Of course, nearly all of these technologies first appear in the world of computer games, which tend to lead all other fields, using new tricks and techniques for graphic rendering. In one important invention, the “real-time shape” method was developed by Dr. Alexander Migdal of MetaTools Inc., Carpenteria, California. Real-time shapes modify the details of a graphic object according to its apparent distance from the viewer of the image. For example, a race car in a computer game increases the details that are almost alive as it roars toward a composite image foreground game player, but it goes back into the background of a similar scene As you lose resolution.

  However, each of these systems is limited to the ability to render an image in terms of the instantaneous state of its surrounding environment and the state of the device associated with the scene. The devices of the art do not take into account the mobile device's dynamic parameters in their image rendering scheme. However, it is quite valuable to give a computer-generated graphic image dynamic with respect to the physical state of the system relative to the scene being rendered, for example a mobile device on which the image is displayed. right. In particular, the position and orientation of the mobile device may suggest preferences to an image rendering system in which the level of detail of the rendered image is affected by specific values corresponding to the position and orientation (direction).

  While the technology systems and inventions are designed to achieve specific goals and objectives, some of them are superior, these inventions of the technology are nevertheless possible at present Include limits to prevent use in methods. The invention of the art is not used and cannot be used here to realize the advantages and objectives of the teaching set forth below.

U.S. Pat.No. 5,815,411

  Here, Thomas Ellenby has invented a method for rendering graphic images, including a method for prioritizing the details of a graphic image according to the physical state of the mobile device associated with the environment being displayed.

  The main object of the present invention is to provide a method for rendering graphics depending on the physical state of the associated equipment.

  It is an object of the present invention to provide a portable system that reacts to the shape characteristics of the device.

  A further object is to provide computer graphic rendering with selective and variable details.

  A further understanding can be obtained by reference to the detailed description of the preferred embodiments and the accompanying diagrams. The illustrated embodiments are specific ways to implement the invention and do not include all possible ways. Thus, there may exist embodiments that do not depart from the spirit and scope of the disclosure as described by the appended claims, but do not appear here as specific examples. It will also be appreciated that many alternative types are possible.

  The concept of “importance” or “priority” as used to control the graphic rendering described in this document is a graphic image, for example, the position and orientation or orientation of the mobile system. Is different from systems common in the prior art in that the sensing state determined by the device subsystem of the portable device associated with the scene displayed by is taken into account in the classification of the importance of each object. Further disclosed is a method of generating a usage profile based on a particular user's habits and desires that is used to modify the importance factor of a selected graphic element. In addition, limits on the complexity of graphic objects to be generated based on the sensed state of the device (eg, portable system position and / or point direction relative to the scene being rendered) are disclosed.

  A method for defining and controlling the complexity of graphics and the priority of graphic rendering or generation by other portable devices with increased realities and known performance characteristics, and various sensing states of portable devices and other inputs Based on. These methods are particularly useful in aviation, sea and land navigation, gaming and tourism (increased reality and others), local search, sports viewing, and other fields. Portable devices further incorporate sensors such as GPS, compass, and accelerometer for various uses such as map display and game play. By using the sensed physical state of the device, such as position, point direction, position change speed, slew velocity, vibration velocity, etc., the method of the present invention makes the graphic most important for portable devices. Can be displayed first or given priority in generation, allowing the mobile device to also display graphics at a level of complexity appropriate to their sensing state.

1 is a block schematic diagram of a preferred embodiment of the present invention showing a vision system with associated subsystems and sensors. FIG. 2 is a flow chart showing the general operation of the described system. FIG. 3 is a flowchart showing the operation of the graphic limit subsystem by unit operation, Part 1. FIG. FIG. 3 is a flowchart showing the operation of the graphic limit subsystem by unit operation, Part 2. FIG. It is a flowchart and part 1 which show operation of a use profile subsystem. It is Part 2A which is a flowchart showing the operation of the usage profile subsystem. It is a flowchart and Part 2B which show operation of a use profile subsystem. Flowchart showing operation of usage profile subsystem, part 2C. It is Part 2D, a flowchart showing the operation of the usage profile subsystem. Part 2E is a flowchart showing the operation of the usage profile subsystem. Flowchart showing operation of usage profile subsystem, Part 2F. FIG. 3 is a flowchart showing the operation of the graphic controller subsystem, part 1. FIG. 7 is a flowchart showing the operation of the graphics controller subsystem, Graphics Hierarchy, Part 2. It is a flowchart which shows operation of a display utilization subsystem. It is a flowchart which shows operation of a sleep subsystem. It is the schematic which draws the production | generation of an object area. It is the schematic which draws the view area / address of a target area and a portable device. 3 is a flowchart showing operation in a snap mode, Part 1. 12 is a flowchart showing operation in a snap mode, Part 2.

Glossary of Special Terms Throughout this disclosure, some terms that are not defined in the general dictionary as defined in this document are mentioned. In order to provide a more accurate disclosure, the following term definitions are provided in a clarifying perspective so that the true breadth and scope can be more easily understood. Every attempt has been made to be accurate and elaborate, but it is a requirement that not all the meanings associated with each term can be fully explained. Accordingly, each term is intended to include common meanings that may also be derived from common usage within the related art or dictionary meaning. Where the definitions shown conflict with dictionaries or technical definitions, one of ordinary skill in the art must consider the context of use and provide sufficient discretion to understand the intended meaning. Those skilled in the art will be aware of errors in giving broader meaning to the terms used to fully understand the overall depth of the present teachings and to understand all the intended variations. .

Portable Device “Portable device” means any device that has a location, location and orientation that can be changed or changed by a user, such as a handheld calculator.

Importance “Importance” or “importance coefficient” refers to values associated with various graphic elements. The importance factor controls the order and level of detail of the graphic to be rendered.

Graphic Object A “graphic object” or “graphic” is used to refer to any partial subset of the entire image system that can contain multiple elements.

Preferred Embodiments of the Invention Rendering computer graphics requires a certain amount of processing power depending on the complexity of the graphic elements being rendered. In the real world, very simple shapes and colors may be used to represent certain objects. The White House may be represented as a simple white polygon in a very simple depiction (graphic object). Alternatively, a fairly detailed image with 1.6 million colors and complex shading and lighting effects may be used as a graphic object representing the White House.

The system described herein associates a complexity factor or “complexity number” with a graphic that can be rendered to represent an actual object. Some considerations on how complexity numbers may be generated include those listed in the following list.

Complexity number (CN)
The complexity of the graphics generated by the mobile device, and thus the complexity number (CN) associated therewith, may be modified by a system based on various situations, the situation being:
● The location of the device relative to the geographic location of the graphic (ie, the range from the device to the graphic location)

  ● Graphical device-to-device orientation with respect to the geolocation orientation.

  ● The slew rate of the device.

● Vibration of the device ● Change rate of the position of the device ● Change rate of the orientation of the object related to the graphic with respect to the device.

● Thread level of objects related to graphics (ie, do not generate detailed images of high-voltage cables, but instead generate graphics with a bright red area that is 10 times wider than the actual object).
● Graphical limits set by the user for all or some of the object classes.

  ● Graphical limits imposed by the software / application for all or some of the objects.

  ● Limit graphics complexity to reduce power consumption (eg, generate lower resolution graphics to save processing time and hence power consumption due to lower device power).

  For wireless links, limit the graphics complexity to be downloaded if the data transmission rate or throughput is low.

  ● A user-defined graphic level for a class of objects (for example, showing San Francisco Giants at a higher complexity level than Los Angeles Dodgers). Note that the user can define the maximum or minimum CN for a class of objects.

  ● The area of interest. If the object is in a predetermined target area, the CN is changed. The change in CN may be positive or negative.

  ● Wireless or other data link or media portable or actual latency.

  ● Positioning errors (for example, if the GPS determines that there is an error of +/- 100m, set a particular class of graphics, such as navigation markers, to the lowest CN and increase the size of the graphic by a fixed percentage Is done.

In any given image, various graphic elements may be more important than certain other elements. In each graphic element, an importance factor or importance number is associated.

Importance (IN)
Further, the priority for the generation of a particular graphic that should be generated by the device (ie, the “importance” level and hence the associated importance number (IN));
● Importance as defined by the software / application.

  ● User-defined importance ranking (for example, San Francisco Giants with a higher level of complexity than Dodgers).

  ● The type of object associated with the graphic. Under maritime navigation conditions, underwater obstacles will be more important than restaurants on the ground. This type of ranking may be modified by the user of the device.

  The location of the device relative to the graphic or the geographic location of the object associated with the graphic (ie, the range from the device to the location of the object).

● Danger level or emergency level of the object related to the graphic with respect to the device (for example, a cargo ship approaching with a narrow range in a certain direction)
● The speed of the object relative to the graphic.

  ● The speed of the object relative to the graphic for the device.

  ● Direction of movement of objects related to graphics with respect to the device.

  ● Environmental conditions (eg temperature, tide level, wind speed, ocean current, wave height, visibility reduced by fog and / or rain). Increase IN for areas of stormy weather in the area, such as squall, which can be the object itself.

  ● Date and time (for example, navigation markers are more important for navigators at night).

  ● The unlit object is first. At night, unlit obstacles, such as underwater or unmarked rocks, have a higher IN than obstacles illuminated with normal lighting, such as navigation markers.

  ● The area of interest. If the object is in a predetermined target area, IN is changed. The change in IN may be positive or negative.

It may be modified by a system based on various situations such as.

General description of the active graphics control system:
1) The graphics controller (GC) recalls all basic graphics elements as defined by application and / or user interaction. The geographical position of the basic graphic element is within the influence area, and the influence area is defined with respect to the unit position. The graphic controller then determines which graphic elements are in the unit view / address and attitude data based on unit attitude. The GC also determines what the future unit view / address can be.

  2) Each graphic element is a base defined by the application and has an “importance” (IN) assigned by the GC. For example, the application may define the area within 1 mile of Whale Rock as very important and may therefore be assigned a relatively high IN. Alternatively, the user may select a route composed of a set of waypoints. When these waypoints come into the affected area, they are assigned a relatively high IN. This is because in that case the unit gives priority to the navigation marker. Graphic elements such as hazardous areas that are defined as very important or main objects by the application / user are assigned a very high IN.

  3) If the user profile (UP) is active, the GC increases the IN of the graphic defined as the target by UP, for example by a percentage defined by the application, eg 100%. The increase may vary depending on the type of object / area of interest.

  4) The system determines if any graphic position is within the target area defined by UP, and the IN defined by either the user or the system is increased by the rate defined by the application .

5) The set of IN reduction thresholds defined by the application is 2D or 3D depending on the application and is centered on the unit position and decreases the IN when the graphic element is further away from the unit position. . For example,
0-500m range = 100% IN
501-1000m range = 80% IN
1001-2000m range = 60% IN.

6) The set of IN reduction thresholds defined by the application is 2D or 3D depending on the application and reduces the graphic IN based on the orientation from the unit line of sight. For example,
0 ° -15 ° = 100% IN from unit line of sight
16 ° -30 ° = 80% IN.

  7) The GC generates a graphic hierarchy (GH) ordered from the highest IN to the lowest IN.

  8) In the case of graphic complexity reduction due to unit vibration, the rate of slew rate or position change is indicated by the graphic limit subsystem due to unit operation, and the GC reduces the graphic complexity by the number of levels indicated.

  9) If the UP that is active indicates any basic graphic element modifications, such as alternative default settings or reduced complexity levels, the GC modifies that basic element as shown.

  10) GC calculates all graphic complexity numbers.

  11) If the total CN resource allocated is greater than the application that can be run, the graphic complexity of the graphic element that has the lowest IN and not at the lowest complexity level is reduced by one level (the application is very The rate of system resources available for image generation is limited, eg leaving a rate for data recall).

  12) The GC repeats steps 11 and 12 until the CN total is less than the system resource limit (proceed to step 17) or until all graphic elements are reduced to the lowest complexity level.

  13) The GC calculates the sum of all graphic CNs.

  14) If all graphic elements are reduced to the most basic complexity level and the CN total still exceeds the system resource limit, the unit will exclude the graphic with the lowest IN from the GH and remove the CN total Calculate again.

  15) The GC repeats steps 14 and 15 until the CN total is below the system resource limit (proceed to step 17) or until all graphics are deleted from GH.

  16) The GC informs the user that none of the requested graphic elements can be displayed in real time mode and switches to snapshot mode.

  17) The GC transmits the selected basic graphic element and the associated complexity level to the device's rendering engine.

An option in starting the device is to instantly create an object first with the lowest CN within a defined IN threshold so that the user can see it immediately, and then as described above It may be a simple iterative process to improve the graphic to be displayed.

Preferred embodiment of a graphics controller system in operation:
A vision system is used to depict the method described in this disclosure. This vision system may be a traditional optical coupler type device such as a trend display, or preferably a US patent issued entitled "Electro-Optic Vision Systems Which Exploit Position and Attitude" It may be a Bichon system of the type disclosed in US Pat. No. 5,815,411, which includes an internal positioning (GPS), attitude detection (head direction sensor) and vibration (accelerometer) device. Including. The disclosure of this vision system is incorporated herein by reference. It should be noted that the importance (IN) system and / or the graphic complexity (GC) system is a totally independent, single system with its own dedicated sensor. This disclosure is also used to specifically explain concepts related to the development of user-specific usage profiles, the phenomenon of graphic complexity due to detected unit behavior, the recall and control of basic graphic elements, and the allocation of system resources. The

Graphic limit subsystem 108 by unit operation;
While the operation of the equipment, in particular the vibration rate and slew rate, is used in this preferred embodiment to explain the modification of the complexity to be generated, in the section entitled “Complexity Number (CN)” above As described, other factors may be used in other embodiments in the same manner as described.

3 and 4 illustrate the operation of the graphics limit subsystem 108 by unit operation. The unit operation includes: a) vibration detected by the accelerometer 106, b) slew rate (pitch, roll and yaw) detected by the attitude sensor 105 (usually the nose heading sensor and / or the accelerometer 106). (Yaw)), c) one or more of the rate of change of position detected by the position sensor 104 (usually GPS). The method of modifying the complexity level of the graphic to be generated based on unit behavior is very similar for each of those three sensors, and vibration rate and slew rate are used here as examples. . FIG. 3 is a flow chart 300 illustrating how the unit-based graphics limit subsystem 108 operates on the detected vision system 107. In step 301, the system defines an application specific vibration limit H. This limit is the level of vibration at which the vision system begins to reduce the complexity of all graphics. In step 302, the system receives the operating signal from the accelerometer 106 and the time signal from the clock 101 and calculates the vibration rate. Accelerometers are also commonly associated with vision system deformable prism image stabilization systems. In step 303, the system checks whether the calculated vibration rate exceeds H. If the calculated vibration rate does not exceed H, the process proceeds to step 401 in FIG. 4 in the flowchart. When the calculated vibration rate exceeds H, the process proceeds to step 304 in the flowchart. In step 304, the system checks whether the calculated vibration rate exceeds the vision system stabilization system's ability to stabilize the image. If the calculated vibration rate does not exceed the vision system stabilization system's ability to stabilize the image, the flowchart proceeds to step 306. If the calculated vibration rate exceeds the vision system stabilization system's ability to stabilize the image, proceed to step 306 in the flowchart. In step 306, the system reduces the complexity level of all graphics by one level. This is because even though the vision system stabilization system can respond to the detected vibration, the user is also almost certainly vibrating / operating. In step 305, the system reduces the complexity level of all graphics by two or more levels, the amount being defined by an application-specific complexity reduction vibration threshold. Next, the process proceeds from step 305 and step 306 in the flowchart to step 401 in FIG. FIG. 4 is a flowchart 400 illustrating how the unit-based graphics limit subsystem 108 operates for a detected vision system 107 slew rate. In step 401, the system defines an application specific slew rate J. This limit is the slew rate at which the system begins to reduce the complexity of the entire graphic. In step 402, the system receives signals from attitude detector 105 and / or accelerometer 106 and clock 101 and calculates slew rate K. In step 403, the system checks whether the calculated slew rate K exceeds J. If K does not exceed J, go to step 501 of FIG. 5 in the flowchart to check the usage profile subsystem 10. If K exceeds J, the process proceeds to step 404 in the flowchart. In step 404, the system reduces the complexity level of the entire graphic by one or more levels, the amount being defined by an application specific complexity reduction slew rate threshold. Next, the flow proceeds to step 501 of FIG. 5 in the flowchart to check the usage profile subsystem 109.

Usage profile subsystem 109;
FIG. 5 is a flowchart 500 illustrating the basic operation of the usage profile subsystem 109. In step 501, the system checks whether the usage profile (UP) is active. If UP is active, go to step 502 in the flowchart. If UP is not active, go to step 503 in the flowchart. In step 502, the system checks whether the user has changed to a different UP. If the user has not changed the UP, the process proceeds to step 508 in the flowchart. If the user changes UP, the process proceeds to step 503 in the flowchart. In step 503, the system checks whether the user has selected an existing UP. If the user recalls an existing UP, in the flowchart, the usage profile subsystem 109 proceeds to step 507 where the selected UP is loaded, and then proceeds to step 508. If the user has not selected an existing UP, then in the flowchart, the system proceeds to step 504 where the system queries the user regarding the desire to save the newly created “virgin” UP. If the user indicates that he wants to save a newly created UP, in the flowchart the system will prompt the user to name the new UP and save it with that name (if the user does not name the UP) , The system assigns a default name) to step 505 and then to step 508. The default UP may be predefined by the application to include a basic set of parameters or may be blank. If the user does not wish to save the newly created UP, the flow proceeds to step 1201 of FIG. 12, which is the beginning of the graphic controller subsystem flowchart 1200 in the flowchart. In step 508, the system monitors system usage and updates the active UP accordingly. Step 508 is expanded in FIGS. 6-11.

  6-11 are flowcharts 600, 700, 800, 900, 1000, and 1100 illustrating the operation of step 508 of flowchart 500. FIG. In step 601, the system checks whether the user has defined a 2D or 3D target area for the system location. In other words, is the user defining an area that is always in the same position relative to the target unit? An example of such an area is a warning zone ring with the system in the center. If the user has defined a 2D or 3D target area for the unit, in the flowchart, the system proceeds to step 602 where it adds an active UP to this target area, and then proceeds to step 603. If the user has not defined a 2D or 3D target area for the unit, the process proceeds to step 603 in the flowchart. In step 603, the system checks whether the user has defined the point of interest. This point has a real-world position and is not fixed relative to the unit position. If the user has not defined the target point, the process proceeds to step 701 in the flowchart. If the user indicates a point of interest, in the flowchart, the system proceeds to step 604 where the user determines whether the user has defined an associated threshold for that point of interest. This threshold defines an area for points that are also objects. One example defines Whale Rock as the target and associates the threshold of 500m with Whale Rock, indicating that the user targets Whale Rock and all objects within 500m. If the user has not defined an associated threshold, the flow proceeds to step 605 where the system updates the active UP accordingly, and then proceeds to step 701. If the user has defined an associated threshold, then in the flowchart, go to step 605 where the system updates the active UP accordingly, then go to step 701.

  In step 701, the system checks whether the user has defined a line / route such as a target route. A line may be a straight line or a curve, and a route is defined as consisting of several line segments connected end to end. If the user does not define the target line / route, the process proceeds to step 704 in the flowchart. If the user has defined a line / route, the flow proceeds to step 702 where the system checks to see if the user has defined an associated threshold for that target line / route. This threshold defines the area for the line / route that is also the target. One example is that by defining a route from Auckland, New Zealand to Bay of Islands, and defining an associated threshold of 200m, the user is stationary and operating within 200m of the defined route Indicates that all objects that are currently being processed are targeted. If the user has not defined an associated threshold, the flow proceeds to step 703 where the system updates the active UP accordingly, and then proceeds to step 704. If the user has defined an associated threshold, the flow proceeds to step 703 where the system updates the active UP accordingly, and then proceeds to step 704. In step 704, the system checks whether the user has defined the target 2D or 3D area. If the user has not defined the target 2D or 3D area, the process proceeds to step 801 in the flowchart. If the user has not defined the target 2D or 3D area, then in the flowchart, go to step 706 where the system updates the active UP accordingly, then go to step 801. If the user has defined the target 2D or 3D area, then in the flowchart, go to step 706 where the system updates the active UP accordingly, then go to step 801.

  In step 801, the system checks whether the user has defined a specific type of graphic object. If the user has defined a specific type of graphic as a target, then in the flowchart, go to step 802 where the system updates the active UP accordingly, and then go to step 803. If the user has not defined a specific type of graphic, the process proceeds to step 803 in the flowchart. In step 803, the system checks to see if it has specified a new default setting for the type of graphic user interface (GUI). This allows the user to change the default setting for the GUI type, and that setting becomes part of the user's UP. In the future, that setting will be used as the default setting for the type of GUI for that user. In other words, the user modifies the GUI once and saves it as a new default, and the system automatically brings all GUIs of that type to that setting. If the user has changed the GUI default settings, proceed to step 804 where UP is updated accordingly in the flowchart, and then proceed to step 805. If the user has not entered the GUI type default setting, go to step 805 in the flowchart. In step 805, the system determines whether the user has reduced the complexity level of a particular type of graphic by one or more levels and indicated this reduction as a preference. If the user has reduced the complexity level of a particular type of graphic and indicated this as a preference, then in the flowchart, go to step 806 where the system updates the active UP accordingly, then step Proceed to 901. If the user has reduced the complexity level of a particular type of graphic and has not indicated this as a preference, the flow proceeds to step 901 in the flowchart.

  FIG. 9 is a flow chart 900 illustrating how the system generates a target area for a unit by monitoring the system's attitude reading. These areas are not defined by the user. In step 901, usage profile subsystem 109 receives a posture reading from posture detector 105. In step 902, the system checks whether its attitude is stationary (ie, not changing) within the tolerances defined by the application at the time defined by the application. If the system posture is not stationary, the process proceeds to step 1001 in the flowchart. If the system pose is stationary, the system proceeds to step 903 where the system calculates the vision system 107 field of view, and then proceeds to step 904 in the flowchart. In step 904, the system reads the calculated field of view and associated pose and stores it at the top of the list of such data. The number of entries in this list is defined by the application. If the list is full, the bottom entry is removed from the list and later saved for use in higher capacity, lower speed memory such as disk or flash that may be external to the device May be. Next, in the flowchart, the system proceeds to step 905 where it checks to see if there are more than two entries in the list. If there are no more than two entries in the list, go to step 1001 in the flowchart. If there are more than two entries in the list, in the flowchart, the system proceeds to step 906 where the attitude readings of the entries in the list are compared, and then proceeds to step 907. In step 907, the system checks to see if any of the entry's attitude reads match within the tolerance defined by the application. If some or all of these posture readings do not match the system, go to step 1001. If some or all of those posture readings match the system, go to step 908. In step 908, the system calculates the average value of the matching pose readings and calculates the field of view (FOV) centered on the average pose reading. The field of view spans all fields of view associated with average posture reading. The system then removes the average attitude reading and associated FOV from the list of such data. In the flowchart, the system then adds the newly calculated target FOV for the unit to the active UP. Next, the flowchart proceeds to step 1001.

  FIG. 10 is a flowchart 1000 illustrating how the system generates target areas that are not associated with a unit by monitoring system attitude and position readings. These areas have a stationary real world location and are not defined by the user. In step 1001, the usage profile subsystem 109 receives the posture reading from the posture detection device 105 and the position reading from the position detection device 304. In step 1002, the system checks whether the system is stationary (ie, has not changed) within the application-defined tolerance for a period defined by the application. If the system is not stationary, the process proceeds to step 1101 in the flowchart. If the system is stationary, in the flowchart, the system proceeds to step 1003 where the field of view (FOV) of the vision system 107 is calculated, and then proceeds to step 1004 in the flowchart. In step 1004, the system stores the calculated FOV and associated position and orientation readings at the top of the list of such data. The number of entries in this list is defined by the application. If the list is full, the bottom entry may be removed from the list and saved for later use in higher capacity, lower speed memory such as disk or flash that may be external to the device. Good. Next, in the flowchart, the system proceeds to step 1005 where the system checks to see if there are more than two entries in the list. If there are no more than two entries in the list, go to step 1101 in the flowchart. If there are more than two entries in the list, in the flowchart, the system proceeds to step 1006 where the system calculates the intersection of FOVs on the list, and then proceeds to step 1007. In step 1007, the system intersects the application defined number of FOVs in a common 3D area. FIG. 16 shows in a plane the intersection 1607 of three fields of view 1604, 1605, 1606 from different positions 1601, 1602, 1603, respectively. The shaded area is a common intersecting area. Note that there may be more than one common intersection area at step 1007. If the system confirms that enough listed FOVs do not intersect in the common area, it proceeds to step 1101 in the flowchart. If the system has enough listed FOVs intersecting in the common area, then in the flowchart, go to step 1008 where the boundary of the common intersection area is calculated by the unit, and then the calculated common intersection Proceed to step 1009 where the area is added to the active UP as the target 3D area. The flowchart then branches to step 1101.

FIG. 11 illustrates how the system excludes the system-generated target area or the related target FOV generated in the system, as shown in FIGS. 9 and 10, from active UPs. It is a flowchart 1100. In step 1100, the system checks to see if the vision system 107 field of view intersects all target areas defined in the system and all target FOVs defined in the system within a period defined by the application. . If all such areas have been crossed within the application defined period by the vision system 107, the flow proceeds to step 1201, which is the beginning of the graphics controller subsystem flowchart 1200 in the flowchart. If all such areas have not been crossed by the vision system 107 within the defined period, go to step 1102 where the non-crossed area or FOV is excluded from active UP in the flowchart. The process then proceeds to step 1201, which is the beginning of the graphics controller subsystem flowchart 1200.

Graphic controller subsystem 110;
The graphics controller subsystem 110 controls how the system recalls basic graphics elements and at what level of complexity each graphic is generated. Each basic graphic element consists of 1) a position that defines the location of the graphic with respect to an arbitrary reference coordinate system, 2) an orientation that defines the direction of the graphic with respect to an arbitrary reference coordinate system, and 3) each graphic complexity It consists of a model for the degree level and a complexity number (CN) The model is sufficient to define the shape, scale and content of the graphic. Note that the graphic may be 2D or 3D, or even animated, as defined by the model. Each graphic has a number of graphic complexity levels associated with it. These can range from, for example, more complex, full-fledged raster images to the minimum complexity required to share the meaning of the graphic, for example, simple vector images or icons related to the type of object of interest. It reaches. Some images may have only one complexity level, while other images have many complexity levels. The complexity number associated with each graphic complexity level defines the number of calculations required to generate a graphic at that level. These different complexity levels are used by the graphics controller subsystem 110 when allocating system resources for graphics generation, as described below.

  Each graphic is assigned an importance level (IN), which is defined by the application. For example, in shipping navigation applications, graphics associated with navigation markers have a relatively high IN, but in tourism applications that cover the same area, those navigation markers have lower importance and therefore The graphics associated with them have a lower IN. The IN assigned by the application can change when the application switches from one mode of operation to another. Using the above example, the application may have two modes of operation, navigation mode and tourism mode for the area. The IN is used by the graphics controller subsystem 110 when allocating system resources for graphics generation as described below.

  An affected area with unit location or real world location may be defined by software / application / microcode / user. This influence area may be a two-dimensional or three-dimensional shape, for example circular or spherical. The affected area may be a visible horizontal line, for example. The affected area defines the area where the location of the basic graphic element must be recalled by the graphic controller subsystem 110.

  FIG. 12 is a flowchart 1200 illustrating the operation of the graphics controller (GC) subsystem 110. In step 1201, the system checks whether the vision system has previously generated graphics. If a graphic has been generated, in the flowchart, the system proceeds to step 1202 where it checks to see if the position or orientation of the vision system has changed since the graphic was generated. If the position or orientation has changed, the process proceeds to step 1204 in the flowchart. If the position or orientation has not changed, in the flowchart the system proceeds to step 1203 where the user or the vision system itself checks to see if the graphic has been added or modified. If the graphic has not been modified or added, the flow proceeds to step 1214 where the previously generated graphic is transmitted to the display of the vision system 107 and then to step 1401 of FIG. Check system 111. If the graphic has been modified or added, go to step 1204 in the flowchart. In step 1204, the GC generates a graphic hierarchy (GH) as shown in FIG. 13, and the flowchart then branches to step 1205. In step 1205, the GC checks whether the graphics limit subsystem 108 due to unit operation has shown a decrease in overall graphics complexity. If such a decrease is indicated, then in the flowchart, go to step 1206 where the GC modifies GH accordingly, and then go to step 1207. If no such decrease is indicated, the process proceeds to step 1207 in the flowchart. In step 1207, the GC checks whether the active UP indicates a modification such as an alternate default setting or complexity level reduction for any basic graphic element in GH. If such a modification is indicated, the flow proceeds to step 1208 where the GC modifies GH accordingly, and then proceeds to step 1209. If no such correction is indicated, go to step 1209 in the flowchart. In step 1209, the GC calculates the sum of all complexity numbers for the basic graphic elements in GH with their current complexity numbers, and in the flowchart the system checks whether the CN sum is zero step 1216. Proceed to If the CN sum is equal to zero, i.e. all basic graphic elements have been excluded from its graphic hierarchy, for example by excessive action, in the flow chart tell the user that the system will switch to snapshot mode and The process proceeds to step 1217 to capture the image and capture the system position and orientation at the time to capture the still image, and then proceeds to step 1301 in FIG. 13 to generate a new GH. If the CN sum is not equal to zero, in the flowchart, the resources needed by the system to generate all graphics in GH at their current complexity level do not exceed the available system resources It proceeds to step 1210 to check whether or not. Resources available for graphics generation include processor speed, available memory, battery status / available power, power usage, data transmission rate, predefined graphics and a pre-loaded set of icons. May be. If the necessary resources exceed the allocated system resources, the process proceeds to step 1211 in the flowchart. If the required resources do not exceed the allocated system resources, in the flowchart, go to step 1215 where the GC generates graphics, then go to step 1401 in FIG. To check. In step 1211, the GC checks to see if all basic graphic elements in GH are at their lowest complexity level setting. If all the basic graphic elements in GH are not at their lowest complexity level, the GC will reduce the complexity level in the flow chart so that one level of the basic element with GH and hence the lowest IN Proceed to step 1212 to modify the CN of the base element that is not at the lowest complexity level, then proceed to step 1209 to calculate the CN total again. If all the basic graphic elements in GH are at their lowest complexity level, then in the flowchart go to step 1213 where the GC has the lowest IN with the lowest IN and go to step 1213, then go to step 1209, where CN Calculate the total again.

FIG. 13 is a flowchart showing how the GC generates a graphic hierarchy (GH). In step 1201, the GC recalls all basic graphic elements as defined by the application and user interaction, their location is within the affected area, and the affected area is at the system location at the application or user. And then determine which graphics are in the vision system 107 field of view based on the system pose and field of view data. FIG. 17 shows a plan view of the location 1701 of the vision system 107, the limits of the defined influence area 1702, the position of the basic graphic element 1703, the line of sight 1704, and the field of view / address 1705. If the device is not a vision system but a mobile phone, for example, the device may have an address field similar to the vision system field of view. This address field may be, for example, a cone extending at 30 ° with the longest dimension axis of the mobile phone. FIG. 17 also shows a “display buffer” area 1706 in which the basic graphic elements that have just left the view 1705 or that will soon enter the view are tracked. The flowchart then proceeds to step 1302 where each graphic has a basic “importance number” (IN) defined in the application assigned by the GC. For example, the application may define the area within a mile of Whale Rock as very important and may therefore assign a relatively high IN to it. Alternatively, the user may select a route composed of a set of intermediate points. As these midpoints enter the affected area, they are assigned a relatively high IN. This is because the unit gives priority to the navigation marker. Graphics that are defined as very important or main objects by the application / user, such as hazardous areas, are assigned a very high IN. The flowchart then proceeds to step 1303 where a check is made to see if any of the GC recalled graphics are defined as targets by the active UP. If graphics are defined as targets, then in the flowchart, the IN of those graphics is incremented by the rate defined by the application, proceeding to step 1304, and then proceeding to step 1305. If the graphic is not defined as a target, the process proceeds to step 1305 in the flowchart. In step 1305, the GC checks whether any of the graphic locations are within the defined area of interest, either by the user or the system. If any of the graphic positions is within the target area, then in the flowchart, the GC proceeds to step 1306 where the IN of the graphic in the target area is increased by a rate defined by the application, and then proceeds to step 1307. In step 1307, the GC checks whether the range to the position of any graphic exceeds the first range IN reduction threshold. Examples of such threshold systems may be 0-500m = 100% IN, 501-1000m = 80% IN, 1001m-1500m = 60% IN, etc. If the range to the location of any graphic exceeds the first range IN reduction threshold, proceed to step 1308 in the flowchart where those graphic INs are reduced according to the range IN reduction threshold set in the application. Then, the process proceeds to Step 1309. If the range up to any graphic position does not exceed the first range IN reduction threshold value, the process proceeds to step 1309 in the flowchart. In step 1309, the GC checks whether the orientation from the vision system 107 line of sight to the position of any graphic exceeds a first range IN reduction threshold. If the orientation to the position of any graphic does not exceed the first range IN reduction threshold value, the process proceeds to step 1311 in the flowchart. In step 1311, the GC generates a graphic hierarchy (GH) composed of basic graphic elements described in a priority order from the highest IN to the lowest IN. The flowchart then proceeds to step 1312. In step 1312, the system checks whether the snapshot mode is active. If the snapshot mode is active, the process proceeds to step 1801 in FIG. 18 in the flowchart. If the snapshot mode is not active, the process proceeds to step 1205 in FIG. 12 in the flowchart.

Snapshot mode Snapshot mode allows the system to display information to the user even in situations such as excessive equipment operation and does not allow real-time generation of images. This is done by capturing still images and associated positions and orientations, generating a graphic hierarchy, reducing the complexity level of the lowest set of primitives in that GH, and generating a composite image. The generation of the composite image may take more time than is normally possible to generate an image in real-time operation. This mode may typically be automatically activated as shown in steps 1216 and 1217 of FIG. 12, but may also be activated by the user upon request.

  FIG. 18 is a flowchart 1800 showing a portion of the operation of vision system 107 in snapshot mode. In step 1801, the system reduces the graphics complexity of the underlying elements at a lower rate of GH to the lowest level. For example, this percentage of 80% is defined by the application. This is done to speed up the generation of composite images by generating only the most important graphics with GH, and at their maximum complexity level. The flowchart then renders the composite image that the system requires the most time. This is because in a “real time” time frame, that frame is not necessary. Next, the process proceeds to step 1401 in FIG. 14, and the display usage subsystem 111 is checked.

FIG. 19 is a flowchart 1900 illustrating additional operation of the vision system 107 in snapshot mode. In step 1901, the system checks whether the user has activated the monitoring mode. When the monitoring mode is activated, the process proceeds to step 702 in FIG. 7 in the flowchart, and monitoring of the time trigger 206 is started. If the monitoring mode is not activated, in the flowchart, the system proceeds to step 1302 where it is determined whether the vision system 107 is turned off overall. If the vision system 107 is turned off, proceed to step 1903 in the flowchart where all systems are deactivated and the vision system 107 is shut down. If the vision system is not turned off, in the flowchart, the system proceeds to step 1904 where the user determines whether the user has activated the snapshot mode. If the snapshot mode is inactive, proceed to step 702 of FIG. 7 in the flowchart to check the graphic limit subsystem 108 by unit operation. If the snapshot mode has not been deactivated, the flow proceeds to step 1905 where the system checks to see if the user has indicated that the user will capture a new still image. If a new image is shown, in the flowchart, the system captures the new still image and proceeds to step 1906 where the system's position and orientation at the time the image was captured is followed by step 1301 in FIG. Generate new GH. If no new image is shown, the flow proceeds to step 1401 of FIG. 14 to check the display usage subsystem 111 in the flowchart.

Display usage subsystem 111;
FIG. 14 is a flowchart 1400 illustrating the operation of the display usage subsystem 111. This subsystem is operable to detect whether the user is actually looking at the vision system display and to activate or deactivate the display accordingly. Note that some activities, such as warming up the screen backlighting, never become inactive, but instead remain active while the vision system is active overall. In step 1401, the system checks whether one / multiple displays are active. If the / their displays are active, go to step 1402 in the flowchart. If the / their displays are not active, go to step 1403 in the flowchart. In step 1402, the system checks if the object is within the display activation range threshold defined by the application / user. This is done through the use of low power sonic, light emitting lasers or other similar range devices. The user may modify this activation threshold so that, for example, a corrector or sunglasses wearer can use the subsystem and activate the display. This may go a step further in that the desired display activation range threshold is part of the user usage strategy. If the object is detected within the display activation range threshold, in the flowchart, proceed to step 1405 where the display remains active, and then proceed to 1407. If the object is not detected within the display activation range threshold, proceed to step 1404 where the display is deactivated in the flowchart, and then proceed to step 1407. In step 1403, the system checks if the object is within the display activation range threshold defined by the application / user. If the object is detected within the display activation range premises, proceed to step 1406 where the display is activated in the flowchart, and then proceed to step 1407. If the object is not detected within the display activation range threshold, go to step 1407 in the flowchart. If the snapshot mode is active, the process proceeds to step 1901 in FIG. 19 in the flowchart. If the snapshot mode is not active, the flowchart proceeds to step 1501 of FIG. 15 to check the sleep subsystem 112.

Sleep subsystem 112;
FIG. 15 is a flowchart 1500 illustrating the operation of the sleep subsystem 112. This subsystem is enabled to return the system to the monitoring mode if the position or orientation of the vision system does not change in a user or application defined period.

  The initial graphic complexity reduction may also be performed by the CPU or application processor based on readings indicating high vibration rates from sensors such as accelerometers, for example. For example, a table of CN reduction at a given vibration rate for each class of objects may be used by the CPU to limit the CN before the GC system starts computing, thereby saving time and power Is done.

  Detected device acceleration is also defined by the location, application, user, etc., by monitoring the determined direction of the bore-site of a device such as a vision system in a plane perpendicular to the device acceleration It can also be used to refine by supplementing based on a pre-configured set of rules. An example is a vision system used in a car on a side street. Assuming normal vehicle suspension operation, the car's upward normal acceleration is much more sudden than the downward acceleration, so the system is averaged and stabilized equipment boresight Will only read at a rate of upward movement when determining.

  The above examples are directed to specific embodiments illustrating the desired type of devices and methods of these inventions. For completeness, a more general description of the apparatus and the elements and methods it contains and the steps it contains is given below.

Those skilled in the art will know how the graphic system is formed and the graphic elements of the composite image are generated with preferences regarding the importance and complexity of the graphic in terms of the instantaneous state of the associated handheld system of the generated image. You will understand completely. Although the present invention has been described in considerable detail in a clear and concise language and with reference to certain desirable types, including the optimal mode that the inventor expects, other types are possible. Therefore, the spirit and scope of the present invention should not be limited by the descriptions of the preferred types contained therein, but rather by the claims appended hereto.

101 ... clock
104 ... Position sensor
105 ... Attitude sensor
106 ... Accelerometer
107 ... Vision System
108 ... Graphic limit subsystem by unit operation
1601 ... Position
1602 ... Position
1603 ... Position
1604 ... Field of view
1605 ... Field of view
1606 ... Field of view
1607 ... intersection of three fields of view
1701… Location
1702… Affected area
1703: Position of basic graphic elements
1704 ... Gaze
1705 ... Field of view / address field
1706 “Display Buffer” area

Claims (3)

  A method for rendering a graphic image, the method being responsive to a physical state of a freely operable portable unit, including one determined by an inertial measurement device.   The method of claim 1, wherein the method is responsive to a position and pointing direction of the freely movable portable unit. The method of claim 1, wherein:
Determining the position of the mobile device;
Determining a pointing posture of the mobile device;
Generating an image including a plurality of graphic elements, wherein the order and details of the graphic elements are rendered by values measured as position and orientation;
A method further comprising:

Images