CA2953186A1 - Means, apparatus, and method for humanistic intelligence, undigital cyborg craft, and sousveillant systems for machine integrity - Google Patents

Abstract

Advances in sensing technology, combined with AI (Artificial Intelligence) have created devices and systems that sense our every movement in intricate detail. Society is evolving toward a world of surveillant systems - - machines that sense us more intimately, yet reveal less about their internal states and functions. The modern "user-friendly" trend is to hide functionality and machine state variables to "simplify" operation for end-users. While this "dumbed down" (and "dumbing down") technology trend is supported by most of society, there is a risk that it undermines the natural scientific curiosity and comprehensibility of some end-users, leading them away from trying to understand our technological world. Surveillant systems work against users of intellect, excluding them from particpating fully in technological progress, and possibly driving some users away from logical thinking and toward technopaganist "witchcraft" and insanity. To reverse this harmful trend, Minsky, Kurzweil, and Mann proposed the use of HI (Humanistic Intelligence) to create a "Society of Intelligent Veillance". Accordingly, Sousveillant Systems are systems that are designed to reveal their internal states to end-users intelligently, to complete an HI feedback loop. This paper is in-part a kind of "manifesto" to (or at least inspiration for) product designers, system designers, creators, manufacturers, etc., on how to create better technology that makes us smarter rather than stupider. Here is disclosed a number of embodiments of an invention related to the new emerging field of Sousveillant Systems. The invention is based on technology that makes its internal state apparent to the end user, without appreciable delay or omission ("Feedback delayed is feedback denied"). In one embodiment, a Haptic Augmented Reality Computer Aided Design system allows the user to create content using a special kind of lock-in amplifier. In another embodiment, the user's body, especially their core (e.g. transversus abdominis, obliques, rectus abdominis, erector spinae, etc.) facilitates a pointing device or cursor, in the feedback loop of an interactive computational process such as a game. This "PlankPoint .TM." or "CorePoint .TM." technology gives rise to a new form of fitness based on Integral Kinesiology such as absement or absangle with respect to a target path or goal trajectory through a multidimensional virtual reality or augmented reality space.

Description

t`-"AtTPTA`tfll '1 &ire-cit.() ;=1 == =
=: =..r , DEC 2 9 21)16 Means, apparatus, and method for Humanistic Intelligerrue-,--tindigita Cyborg Craft, and Sousveillant Systems for Machine Integrity Steve Mann WearTechTm Humanistic Intelligence Institute; Veillance Foundation http://www.weartech.org http://www_eyetap.org 2016 December Abstract ate content using a special kind of lock-in amplifier. In another embodiment, the user's body, especially their Advances in sensing technology, combined with AI
core (e.g. transversus abdorninis, obliques, rectus abdo-(Artificial Intelligence) have created devices and sys-minis, erector virtue, etc.) facilitates a pointing device terns that sense our every movement in intricate de-or cursor, in the feedback loop of an interactive compu-tail. Society is evolving toward a world of surveillant tational process such as a game. This µPlankPointni"
systems -- -- machines that sense us more intimately, or "CorePointm" technology gives rise to a new form yet reveal less about their internal states and func-of fitness based on Integral Kinesiology such as abse-lions. The modern "mser-friendly" trend is to hide ment or absangle with respect to a target path or goal functionality and machine state variables to "simplify"
trajectory through a multidimensional virtual reality or operation for end-users. While this `dumbed down"
augmented reality space.
(and "dumbing down") technology trend is supported Priority Claim: The author wishes to claim priority by most of society, there is a risk that it undermines in regards to a United States provisional patent appli-the natural scientific curiosity and comprehensibility of cation entitled "Undigital Cyborg Craft: A Manifesto some end-users, leading them away from trying to 217/- on Sousveillant Systems for Machine Integrity" filed derstand our technological world. Surveillant systems 2016 December 01.
work against users of intellect, excluding them from particpating fully in technological progress, and possi- 1. Humanistic Intelligence: Surveillance bly driving some users away from logical thinking and AND Sousveillance together toward technopaganist "witchcraft' and insanity.
To reverse this harmful trend, Minsky, Kurzweil, 1.1. Surveillance and Sousveillance and Mann proposed the use of HI (Humanistic Intel-The word "surveillance" is a word coined during ligence) to create a "Society of Intelligent Veillance7 the French Revolution from the prefix "sur" (meaning Accordingly, Sousveillant Systems are systems that are "over" or "from above") and the postfix "veillance"
designed to reveal their internal states to end-users in-(meaning "sight" or "watching") [481 The closest pure telligently, to complete an HI feedback loop. This paper English word is "oversight". See Table 1.
is in-part a kind of "manifesto" to (or at least inspira-tion for) product designers, system designers, creators, English French manufacturers, etc., on how to create better technology to watch veiller that makes us smarter rather than stupider. Here is watching (sensing in general) veillance disclosed a number of embodiments of an invention re- watching over (oversight) surveillance lated to the new emerging field of Sousveillant Systems. over (from above) Stir The invention is based on technology that 'makes its under (from below) sous internal state apparent to the end user, without appre- undersight (to watch from below) sousveillance cMble delay or omission ("Feedback delayed is feedback Table 1: English words with French translations.
denied"). Surveillance [70, 44, 22]
(oversight) is not the only In one embodiment, a Haptic Augmented Reality kind of veillance (sight).
Sousveillance (undersight) has Computer Aided Design systern allows the 'user to cre- also recently emerged as a new discipline [47, 59, 29, 74, 7, 6, 27, 73, 15, 5, 3, 83, 68, 72, 45, 79. 32].
These veillances (sur and sous veillance) are broad 1 'Senses Effectorst 2 concepts that go beyond visual (camera-based) sensing, Human to include audio sur/sousveillance, dataveillance, and =
.o c many other forms of sensing. When we say "we're being 1 = Al watched" we often mean it in a broader sense than just =
=

the visual. For example, when police are listening in on achin ; 1111' our phone conversations, we still call that surveillance, 5 Sensors Actuators 6 even though it involves more of their ears than their 1, Humanistic Intelligence eyes. So, more generally, "surveillance" refers to the (HI) condition of being sensed. We often don't know who or ----------- - - S.
Mann, 1998 what is doing the sensing. When a machine learning al- Figure 11. The Six Signal Flow Paths of HI: A human gorithm is sensing us, that is also surveillance. AI (Ar- (denoted symbolically by the circle) has senses and effec-tors (informatic inputs and outputs). A machine (denoted tificial Intelligence) often involves surveillance. Some surveillance can be harmful ("malveillance"). Some by the square) has sensors and actuators as its informatic inputs and outputs. But most importantly, HI involves can be beneficial ("bienveillance"), like when a machine in-tertwining of human and machine by the signal flow paths senses our presence to automate a task (flushing a toi- that they share in common. Therefore, these two special let, turning on a light, or adjusting the position of a paths of information flow are separated out, giving video game avatar). a total of six signal flow paths.
Systems that sense us using detailed sensory appara-tus are called surveillant systems. Systems that reveal themselves to us, and allow us to sense them and their internal state variables are called Sousveillant Systems. within the single `cyborg' being?' [65]
1.2. Winning at AI is losing HI involves an intertwining of human and machine in A common goal among AI (Artificial Intelligence) a way that the human can sense the machine and vice-researchers is to replicate human intelligence through versa, as illustrated in Fig. 11.
computation, and ultimately create another species having human rights and responsibilities. This creates HI is based on modern control-theory and cyber-a possible danger to humanity, through what many re- netics, and as such, requires both controllability (be-searchers refer to as the singularity. ing watched) and observability (watching), in order to There is a race to see who will be first to create complete the feedback loop. In this way, surveillance a truly intelligent machine. This highly competitive (being watched) and sousveillance (watching) are both research is, in many ways, like a game. But what will required in proper balance for the effective functioning be the prize? of the feedback between human and machine. Thus It is very possible that we can only win the AI Veillance (Surveillance AND
Sousveillance) is at the game by losing (our humanity). core of III. (See Fig. 12.) 1.3. Humanistic Intelligence Poorly designed human-computer interaction sys-terns often fail to provide transparency and immedi-HI (Humanistic Intelligence) is a new form of intelli- acy of user-feedback, i.e. they fail to provide sousveil-gence that harnesses beneficial veillance in both direc- lance. As an example of such a "Machine of Malice", tions (surveillance and sousveillance, not just surveil- an art installation was created by author S. Mann to lance). HI is defined by Minsky, Kurzweil, and Mann exemplify this common problem. The piece, entitled as. follows: "Digital Lightswitch" consists of a single pushbutton "Humanistic Intelligence [HI] is intelligence that lightswitch with push-on/push-off functionality. The arises because of a human being in the feedback button is pressed once to turn the light on, and again loop of a computational process, where the hu- to turn the light off (each press toggles its state). A
man and computer are inextricably intertwined. random 3 to 5 second delay is added, along with a ran-When a wearable computer embodies HI and be- dom packet loss of about ten percent. Thus the button comes so technologically advanced that its intel- only works 90 percent of the time, and, combined with ligence matches our own biological brain, some- the delay, users would often press it once, see no im-thing much more powerful enierges from this syn- mediate effect, and then press it again (e.g. turning it ergy that gives rise to superhuman intelligence back off before it had time to come on). See Fig. 13.

Veillance is the : .)i-s of Humanistic Intelligence In the context of htunan-human interaction, the - -transition from surveillance to veillance represents a , s =
. "fair" (French "Juste") sight and, more generally, fair Senses -411% ' Effectors and balanced sensing.
7 Huma But our society is embracing a new kind of entity, brought on by AI (Artificial Intelligence) and machine learning. Whether we consider an "AI" as a social en-tity, e.g. through Actor Network Theory [69, 41, 9, 84], Machin:
Sensors Actuators or simply as a device to interact with, there arises the >
question "Are smart things making us stupid?"[67].
,Humanistic Intelligence, Past technologies were transparent, e.g. electronic .(Hl) , .., valves ("vacuum tubes") were typically housed in ... , ------------------------- , transparent glass envelopes, into which we could look Figure 12. HI requires both Veillances: machines to see all of their internals revealed. And early devices must be able to sense us, and we must be able to sense them! Thus veillance is at the core of HI. In this sense included schematic diagrams - an effort by the manu-Surveillance is a half-truth without sousveillance. Surveil-facturer to help people undertand how things worked.
lance alone does not serve humanity. Humans have senses In the present day of computer chips and closed-and effectors at their informatic inputs and outputs. Ma-source software, manufacturers take extra effort not to chines have sensors and actuators at their informatic inputs help people undertand how things work, but to con-and outputs. The signal flow paths that connect them are ceaI functionality: (1) for secrecy; and (2) because they surveillance (when we're being watched or sensed by the (sometimes incorrectly) assume that their users do not machine) and sousveillance (when we're watching or sens-want to be bothered by detail, i.e. that their users are ing the machine). HI (Humanistic Intelligence) requires all six of these signal flow paths to be present [65]. When one of looking for and abstraction and actually want "both-ersome" details hidden [52].
the six is blocked (most commonly. Sousveillance), we have a breakdown in the feedback loop that allows for a true syn-At the same time these technologies are being more ergy ("cyborg" state). To prevent this breakdown, Sousveil-concealing and secretive, they are also being equipped lant Systems mandate Observability (Sousveillance).
with sensory capacity, so that (in the ANT sense) -these devices are evolving toward knowing more about . - -,, , us while revealing less about theniselves (i.e. toward 1 'Senses Effectors %
to z.
..3 surveillance).
Our inability to understand our technological world, o c --r-wJ
rrrr i .
o --------------------- ,7-, t in part through secrecy actions taken by manufactur-ers, and in part through a general apathy, leads to the use of modern devices through magic, witchcraft-like o =
Machin: t.i ¨in rituals rather than science [43]. This technopagan-Sensors Actuators 6 ism [77] leads people to strange rituals rather than , Machines of Malice trying to understand how things work. General wis-Digital Lightswitch") ,' dom from our experts tell us to "reboot" and try again, -- -------------------- -5. Mann, 2013 rather than understand what went wrong when some-Figure 13. Systems that fail to facilitate sousveillance are thing failed [75]. But this very act of doing the same machines of malice. Example: an art installation by au-thing (e.g. rebooting) over and over again, expecting thor S. Mann, consists of a pushbutton (push-on/push-off) light switch where a random 3 to 5 second delay is inserted a different result is the very definition of insanity:
along with a 10 percent packet loss. The parameters were adjusted to maximizeh frustration in order to show a neg-"Insanity is doing the same thing, over and -ative example of what happens when we fail to properly over again, but expecting different results."
implement a balanced veillance. Narcotics Anonymous, 1981.

2. Why we need HI to undo the insanity In this sense, not only do modern technologies drive of AI
us insane, they- actually require us to be insane in order to function properly in the technopagan world that is Much has been written about equiveillance, i.e. the being forced upon us by manufacturers who conceal its right to record while being recorded [81, 82, 45, 54], workings.
but here our focus is on the right to simply understand I propose as a solution, a prosthetic apparatus that machines that understand us.
embodies the insanity for us, so that we don't have to.

All call this app "LUNATIC". LUNATIC is a virtual STA N DI N
WAVES S IT TING WAVES
personal assistant. The user places a request to LU-NATIC and it then "tries the same thing over and over __ again..." on behalf of the user so that the user does not need to himself or herself become insane. For exam-Me t49 Kam -4574 ple, when downloading files, LUNATIC starts multi-Figure 14. Sitting waves as compared with standing waves.
ple downloads of the same file, repeatedly, and notifies the user when the result is obtained. LUNATIC de-termines the optimum number of simultaneous down- propagatory effects of waves by sampling them in phys-loads. Typically this number works out to 2 or 3. A ical space with an apparatus to which there is affixed single download often stalls, and the second one of- an augmented reality display. Whereas standing waves, ten completes before the first. If too many downloads as proposed by Melde in 1860, are well-known, and of the same file are initiated, the system slows down. can be modeled as a sum of waves traveling in oppo-So LUNATIC uses machine learning to detect slowed site directions, we shall now come to understand a new connections and makes a best guess as to the optimum concept that the author calls "sitting waves", arising number of times to repeat the same tasks over and over from a product of waves traveling in the same direc-again. This number is called the "optimum insanity", tion (Fig. 14), as observed through a phenomenological and is the level of insanity (number of repetitions) that augmented reality amplifier, in a time-integrated yet leads to the most likely successful outcome. sparsely-sampled spacetirne continuum. See Fig 15.
At times the optimum insanity increases without bound, typically when websites or servers are unreli- 3.1. Metawaves:
Veillance Wave Functions able or erratic. LUNATIUC is not performing a denial In quantum mechanics, a wavefunction is a complex-of service attack, but, rather, a "demand for service". A
valued function whose magnitude indicates the proba-side effect is that when large numbers of people use LU-bility of an observable. Although the function itself can NATIC, erratic websites will experience massive down-depict negative energy or negative probability, we ac-load traffic, such that LUNATIC disincentivises insan-cept this as a conceptual framework for understanding ity.
the observables (magnitude of the wavefunction).
In this sense, LUNATIC is a temporary solution In veillance theory, consider a metawavefunction, to technopagan insanity, and ultimately will hopefully 'Ow as a complex-valued function whose magnitude in-become unnecessary, as we transition to the age of dicates the probability of being observed. For example, Sousveillant Systems.
(0/4-10p) = Ou0p*dt, (1)

3. Early example of Sousveillant Systems (where * indicates complex-conjugation) grows from 1974: Sequential Wave Imprint-stronger when we get closer to a camera or micro-ing Machine phone or other sensor that is sensing (e.g. watching This section describes some unpublished aspects of or listening to) us.
Note that the complex metawave-a wearable computing and augmented reality invention function itself can be negative and it can even be by author S. Mann for making visible various other- (and usually- is) complex! This is different from the wise invisible physical phenomena, and displaying the veillance flux concept we reported elsewhere in the phenomena in near-perfect alignment with the reality literature [39, 38, 37], which is a real-valued vector to which they pertain. As an embodiment of HI (Hu- quantity, indicating the capacity to sense.
manistic Intelligence), the alignment between displayed At first, the metawavefunction may seem like a content and physical reality occurs in the feedback loop strange entity, because it is not directly measureable, of a computational or electric process. In this way, nor is its amplitude, i.e. it does not depict a quantum alignment errors approach zero as the feedforward gain field, or any kind of energy field for that matter.
increases without bound. In practice, extremely high Cameras and microphones and other sensors don't gain is possible with a special kind of phenomenologi- EMIT energy, but, rather, they sense energy. Cameras cal amplifier (ALIA = Alethioscopic/Arbitrary Lock-In sense light energy (photons). Microphones sense sound Amplifier / "PHENOMENAmp/zfierTm") designed and energy.
built by the author to visualize veillance.
Thus (z/iiilzi,,) does not correspond to any real or An example use-case is for measuring the speed of actual measurement of any energy like sound or light, wave propagation (e.g. the speed of light, speed of but, rather, it is a inetaquantity, i.e. a sensing of a sound, etc.), and, more importantly, for canceling the sensor, or a sensing of the capacity of a sensor to sense!

A standing wave A Sitting wave and its photographs A
---------------------------------------- 117-\
- -.411F8- - 40-______________________________________________________________ 1,11-11 _ 10*-G)FT ---------------------- FT r-\\,\"( ----------------------------------------- F- -- \\_) /=¨=.õ Slopefi/c \\I .
X 0 Xi X2 Xo X2 Xo Xi Space Space Space Figure 15. Left: a standing wave at four points in time. Middle and Right: a sitting wave at four points in time. Whereas the standing wave stands still only at the nodal points, (e.g. elsewhere varying in amplitude between -1 and +1), the sitting wave remains approximately fixed throughout its entire spatial dimension, due to a sheared spacetime continuum with time-axis at slope 1/c. The effect is as if we're moving along at the speed, c, of the wave propagation, causing the wave to, in effect. "sit" still in our moving reference frame. Right: four frames, F ... F4 from a 36-exposure film strip of a 35-lamp Sequential Wave Imprinting Machine, S. Mann, 1974. Each of these frames arose from sparse sampling of the spacetime continuum after it was averaged over millions of periods of a periodic electromagnetic wave.
The word "meta" is a Greek word that means "be- ing through the camera remotely, or a machine learning yond", and, by way of examples, a meta-conversation algorithm, may not be able to recognize the subject or is a conversation about conversations. A meta joke is perhaps even identify it is human. As we get closer a joke about jokes. Metadata (like the size of an image to the extent that we occupy a few pixels, the camera or the date and time at which it was taken) is data may begin to recognize that there is a human present, about data. Likewise metaveillance (metasensing) is and as we get even closer still, there rnay be a point the seeing of sight, or, more generally, the sensing of where the camera can identify the subject, and aspects sensing (e.g. sensing sensors and sensing their capacity of the subject's activities.
to sense).
Likewise with a microphone. From far away it might Thus the space around a video surveillance (or not be able to "hear" us. By this I mean that a remote sousveillance) camera, or a hidden microphone, can person or AI listening to a recording or live feed from have, associated with it, a metawavefunction, in the microphone might not be able to hear us through which (01,11/;,) increases as we get closer to the camera the microphone.
or microphone, and, for a fixed distance from the cam- Thus (0/,10,) gives us a probability of being recog-era or microphone, (01,101,) typically increases when nized or heard, or the like.
we're right in front of it, and falls off toward the edges Let's begin with the simplest example of a rnetaveil-(e.g. as many cameras have lens aberrations near the lance wave function, namely that from a microphone, edges of their fields of view, and microphones "hear" in the one-dimensional case, where we move further best when facing directly toward their subject). from, or closer to, the microphone along one degree-of-If we are a long way away from a camera, our face freedom.
may occupy less than 1 pixel of its resolution, and be The subscript, itt, is dropped when it is clear by unrecognized by it. By this, I mean that a person look- context that we are referring to a metawavefunction Receiver/Receive antenna i / (rabbit ears") Linear array of i /
Transmitter computer- /
Photographic imprint, or imprint on controlled light =
. µ , . retina by Persistence of Exposure Transmit antenna =ir ______________________________ ¨...-----i .. .011.- -......'"
11E __ . .
N
.:*.,* ..
,. ],1 ,= . ¨T
,.
-, - -.
i4A--- --¨li1 111.=("rabbit ears") i.\1i-t .sr i Wearabl-compute , , * .
¨ 4 ___________________________________________________________ I-- ___________ Fi t1 E- _____ 4IE1-f ...,. .
# .....
' -........-_ Oscilloscope=. --.
to monitor ........- i , -.........
-- _¨ _ I.F. POE ..
,......m.__. ¨
, .
......._ (Persistence of Exposure) 1 .
Figure 16. Photo illustration of SWIM, visualization of electromagnetic radio waves as "sitting waves".
rather than an ordinary wavefunction as we might find More generally, waves may travel to the left, or to in quantum mechanics, or the like. the right, so we have:
Consider an arbitrary traveling metawave function IN CiPz = 0. (6) iP(x,t) whose shape remains constant as it travels to the right or left in one spatial dimension (analogous to Multiplying these solutions together, we have:
the BCCE of optical flow[351). The constancy-of-shape simply means that at some future time, t +At, the wavea a (¨ a ¨ (¨ + c¨) ¨ o, (7) has moved some distance along, say, to x + Ax. Thus: at ax Ot ax tly(x, t) --= 1,b(x + Ax, t + At). (2) which gives:
a2115 ,2 a27b Expanding the right hand side in a Taylor series, we at2 =- ax2 -(8) have:
This is the wave equation in one spatial dimension, I,b(x+Ax,t+At) = 0(x, t)+OsAx+OtAt+h.o.t., (3) as discovered by Jean-Baptiste le Rond d'Alembert in 1746, due to his fascination with stringed musical where h.o.t. denotes (higher order terms). Putting the instruments such as the harpsichord[20], which Euler above two equations together, we have: generalized to multiple dimensions:

OxAx +//,,tAt + h.o.t. = __ _ _ _ 0. (4) _ v2ip 0, C Ot2 (9) If we neglect higher order terms, we have: where V2 is the Laplacian (Laplace operator, named af-ter Pierre-Simon de Laplace, who applied it to studing Ax gravitational potential, much like earlier work by Eu-+ ipt = 0, VAt i 0, (5) At ler on velocity potentials of fluids[261). This further-generalizes to the Klein-Gordon generalization of the where the change in distance, divided by the change in time, -/,162-1 is the speed, c of the traveling wave. Sehrodinger wave equation equation:
In the case of a surveillance camera, or a microwave 1 820 -- ¨ v2ip + (-7")2 ip = o, (10) motion sensor (microwave burglar alarm), c is the speed C2 8t2h of light. In the case of a microphone (or hydrophone), for a particle of mass m, where h --,--- h/27r is Planck's c is the speed of sound in air (or water). constant.

More generally. we can apply a wide range of wave rier, the situation is very simple, and we can visualize theories, wave mechanics, wave analysis, and other the carrier as if "sitting" still, i.e. as if we're moving contemporary mathematical tools, to metawaves and at the speed of light, in our coordinate frame of refer-veillance, and in particular, to understanding veillance ence, and the wave becomes a function of only space, through phenomenological augmented reality [50].
not time. The wave begins as a function of spacetime:
3.2. Broken timebase leads to spacebase t) = cos(wt ¨ kx): wavenumber k = w lc. (11) Waves in electrical systems are commonly viewed In this case the received signal, r(x,t) is given by:
on a device called an "oscillograph"[33, 40]. The word cos(wt ¨ kx)cos(wt) = ¨1 cos(2cet ¨ kx) + ¨1 cos(kx).

originates from the Latin word "oscillare" which means (12) "to swing" (oscillate), and the Creek word "graph"
Half the received signal, r, comes out at about twice the which means drawing or painting. A more modern carrier frequency, and the other half comes out in the word for such an apparatus is "osci11oscope"[42, 31]
neighbourhood of DC (near zero frequency). The signal from the Latin word "scopium" which derives from the we're interested in is the one that is not a function of Greek word "skopion" which means "to look at or view time, i.e. the "sitting wave", which we can recover by carefully" (as in the English word "skeptic" or "skep-lowpass filtering the received signal to get:
tical"). The oscillograph or oscilloscope is a device for displaying electric waves such as periodic electrical al- s(x) ¨1 cos(kx).
(13) ternating current signals.
This operation of multiplication by a wave function In 1974 author S. Mann came into possession of was performed at audio frequencies using a General Ra-an RCA Cathode Ray Oscillograph, Type TMV-122, dio GR736A wave analyzer, and at other times, using which was, at the time, approximately 40 years old, and a lock-in amplifier, and at radio frequencies using four had a defective sweep generator (timebase oscillator). diodes in a ring configuration, arid two center-tapped Since it had no timebase, the dot on the screen only transformers, as is commonly done, and at other times moved up-and-down, not left-to-right, thus it could not using modified superheterodyne radio receiving equip-draw a graph of any electrical signal, but for the fact ment.
that Mann decided to wave the oscillograph back and A drawback of some of these methods is their inabil-forth left-to-right to be able to see a two-dimensional ity to visualize more than one frequency component of graph. In certain situations, this proved to be a very the transmitted wave.
useful way of viewing certain kinds of physical phenom-3.3. Metawaves ena, when the phenomena could be associated with the position of the oscilloscope. This was done by mount- When the transmitter is stationary (whether it be an ing a sensor or effector to the oscilloscope. In one such antenna, or a speaker, or the like) and the receiver (e.g.
experiment, a microphone was mounted to the oscil- a receiving anteima, or a microphone) is attached to the loscope while it was waved back and forth in front oscilloscope, the device merely makes visible the oth-of a speaker, or vice-versa. In another experiment, erwise invisible sound waves or radio waves. But when an antenna was mounted to the oscilloscope while it these two roles are reversed, something very interesting was waved back and forth toward and away from an- happens: the apparatus becomes a device that senses other antenna. With the appropriate electrical circuit, sensors, and makes visible their sensory receptive fields.
something very interesting happened: traveling elec- In the audio case, this functions like a bug sweeper, in tric waves appeared to "sit still". The circuit, sketched which a speaker is moved through the space to sense out in Fig. 17, is very simple: a simple superhetero- microphones, but unlike other bug sweepers the appa-dyne receiver is implemented by frequency mixing with ratus returns the actual underlying veillance wavefunc-the carrier wave, e.g. cos(Lot) of the transmitter. In tion, as a form of augmented reality sensory field, and one embodiment the frequency mixer comprises four not just an indication that a bug is present.
diodes in a ring configuration, and two center-tapped Now consider the case in which the transmitted sig-transformers, as is commonly used in frequency mixers. nal is being modulated, or is otherwise a signal other When one of the two antennae (either one) is attached than a pure wave cos(wt). As an example, let's consider to an oscilloscope with no sweep (no timebase), while '(x, t) = cos(wt ¨ x) + cos(5(wt ¨ x)), so the received the other remains stationary, the oscilloscope traces signal is:
out the radio wave as a function of space rather than r(x, t) = ¨cos(x) +
¨1 cos(x ¨ 2wt) of time.

If the transmitted wave is a pure umnodulated car- ¨2cos(5x ¨ let) + ¨2 cos(5x ¨ 6c,./t), (14) --.r14, . =
=
f'`=
= =
t %111.===0# 4/11~
Figure 17. Chalkboard sketch of a simple experiment: a transmitter Tx is shown at the left, transmitting a wave, cos(wt¨kx) in the spacetime continuum. A receiver, Rx, shown further to the right, picks up the wave and feeds it to one or more mixers. In this case, two are shown, one for the in-phase component, i.e. the real part of the signal, and the other for the quadrature component, i.e. the imaginary part of the signal. A local oscillator supplies cos (wt) to the mixer for the real part, and sin(cot) to the mixer for the imaginary part. Let us consider the real part: The result (by simple trigonometric identity) is shown as: cos(cat ¨ kx)cos(wt) =;1. (cos(2.4.at kx) + .-2-cos(kx)). This output from the mixer is then fed to a lowpass filter, LPF, from which emerges -Icos(kx). Transmitter Tx and receiver Rx may be antennae, or they may be transducers such as a speaker and microphone. For example, transmitter Tx may be a loudspeaker transmitting a periodic waveform, such as a tone at 5000 CPS (Cycles Per Second), with receiver Rx being a microphone. In other embodiments, transmitter and receiver Tx and Rx are hydrophones, for an underwater SWIM, in which case underwater sound waves, sonar, or the like, is visualized using a waterproof underwater SWIM wand.
which when lowpa.ss filtered, only gives us the funda-mented reality visualizations of waves and metawaves.
mental. Thus a wave analyzer or modern lock-in am-plifier such as Stanford Research Systems SR510 can-not be used to visualize such a wave. A more tradi-Whereas a conamon ideal of lock-in amplifier design tional lock-in amplifier, such as Princeton Applied Re-is the ability to ignore harmonics, in our application search PAR124A, will visualize harmonics, but in the we wish to not only embrace harmonics, but to em-wrong proportion, i.e. since the reference signal is a brace them equally. If we were to turn on our sensing square wave, higher harmonics are under-represented of harmonics, one at a time, we would be witnessing (note that the Fourier series of a square wave falls off a buildup of the Fourier series of our reference signal.
as 1/n, e.g. the fifth harmonic comes in at only 20 For the square wave, each harmonic we add to our ref-percent of its proper strength).
erence signal, allows more and more of the measured Thus existing lock-in amplifiers are not ideal for this signal harmonics through, but colored by the coeffi-kind of visualization in general.
cients of the Fourier series representation of the square wave. Figure 18 illustrates a comparison of the the ref-3.4. A Lock-in amplifier designed for Metaveillance erence signal waveforms of the PAR124A lock-in ampli-The approach of Mann was therefore to invent a new fier with the modified lock-in amplifier of the Sequential kind of lock-in amplifier specifically designed for aug- Wave Imprinting Machine (SWIM) [46}.

________________________________________________________ Si&
qyS.
S40,, 1:11 SIGNAL REFERENCE 'SWIM
ACTOR SENSOR -str_ LI G HTS
SWIM
rfv WAND
SENSOR
I WAVEFORM =
>1.= MOVEMENT
________________________________________________________ 7"
PATH
SIG. REF. SIG.
GEN. AMP. AMP.
" =
LIA SWIM SW
R OS OX
COMP
(Sequential Wave _____________________ A ___________ Imprinting Machine) Figure 19. Mann's SWIM (Sequential Wave Imprinting Machine) for making radio waves, sound waves, etc., visible by shearing the spacetime continuum at the exact angle that presents the speed of light or speed of sound at exactly zero: SWIM works with a wide variety of phenomena, to provide a phenomenological augmented reality. A signal source might comprise some kind of signal generator, which produces signal directly, or the signal generator might drive a signal actor such as an actuator (e.g. loudspeaker) or signal transducer (light source) or signal conveyor like an electrode or antenna. Such a signal source, such as a radio transmitter, sound source such as a musical instrument, or other wave source that emits a periodic waveform or periodic disturbance of some kind, sends waves outwards at a speed of wave propagation, c, which is the speed of light in the case of radio waves, or the speed of sound in the case of sound waves (e.g. speed of sound in saltwater if we're visualizing underwater sound waves from a hydraulophone in the ocean, or speed of sound in air if we're visualizing the sound waves from a trupet or clarinet). A reference sensor or reference signal is derived from the sound source. If the sound source is a radio transmitter or loudspeaker, we might connect to it directly. But if we don't have access to it (e.g. if it is a surveillance system that is sealed against our access, or if it is an instrument that doesn't have an electrical connection), we simply place a reference sensor near it. The SWIM apparatus typically or often uses two inputs, one being a reference input from a stationary reference sensor, and the other being a signal input from a moving signal sensor. The signal sensor is moved back and forth together with a linear array of light sources.
For example, the signal sensor may be a microphone or an antenna attached to a linear array of lights. The reference sensor supplies the reference input of a LIA (Lock-In Amplifier or Lock-In Analyzer). In some embodiments, the individual sensors may also have associated or built-in amplifiers. The LIA has one or more outputs such as "X" representing the real part of a demodulated homodyne wave, "Y" representing the imaginary part (i.e.
in "quadrature" with "X") of the wave, "R" representing -VX2 + Y2, and e representing arctan Y/X. One or more of these outputs is connected to a SWIM comp (computer) or system that drives a sequence of lights such as a long row of LEDs (Light Emitting Diodes). A typical number of such LEDs is 11000 or 2000, driven sequentially in proportion to the voltage at "X" or the like. A simple embodiment of the SWIM comp is a ladder network of comparators, such as one or more LM3914 chips set to "dot" mode. Typically the bottom LED comes on when X = 0 volts, and the top LED comes on when X = 5 volts, and for 1000 LEDs we have 5 millivolts per next LED up the ladder. For each 1000 LEDs, 100 LM3914 chips are used. Preferably, though, SWIM comp is FPGA-based or ASIC based or a general-purpose computer that can also generate axis labels, alphanumerics, and the like, so that a WAVEFORM can be plotted together with tick-marks, and indicia overlaid thereupon. The row of lights is waved back-and forth along a MOVEMENT PATH, either by hand, or by robot, so that people can see the WAVEFORM with the naked eye, or photograph it with a camera, onto film or a computer image, or capture it into a VR (Virtual Reality) world for viewing and exploring therein.

i .+=== .
. .+
141 ____________________________ A:- 11 lY 11li I v ' v 1r e fin _.I.I th .õ, CO A AL9c4or ,-- T L__ N\---)14 -4z-,,-,, 1-r1/4A-=;. (4 =
, i Cr.A1V\--(c...---1-:",'=,6--.
:4.= -"la "?-24.134,-'' -4- ,N1 z1,== .-(,',,,-c,_,:.:,.c,,i 1,-1- - - . .,,, -Tt , f,. .
71 .c,

4..-..-.9.... /-----1 -7 , ,). ... 2,...
I'r,...
-- ---nr.---4--____z _________ , , 1._211k-,...:- -...
., _ - , , .
2. .
111.10,11101111 .. _ Figure 21. More typically the output of the mixer, homodyne receiver, lock-in amplifier, or the like, is supplied to an ladder network of LM3914 chips connected to hundreds of LEDs are attached to the receiver, and waved back and forth as a unit.
Typically the SWIM wand is modular, each module comprising ten LM3914 chips, each chip driving 10 LEDs for a total of 100 LEDs per module. The modules are joined together with connectors on each end, so that, for example, 10 modules joined together provide 1000 LEDs, giving an augmented reality display of 1000 by infinity pixels.
4. A new kind of Lock-in Amplifier sense field sensors, and sense their capacity to sense.
Figure 24 shows SWIM as a bug sweeper or the like, 4.1. System architecture doing metasensing. Other bug sweepers of the prior The system architecture for SWIM displaying sound art can find hidden microphones, hidden cameras, hid-waves, radio waves etc., is illustrated in Figure 19 den sensors, etc, but do not let us see their soundfields , , where the SWIM COMP (SWIM computer) shown in or lightfields or otherwise show the intricate nature of Figure 19 is implemented by way of a ladder of com-their sensory capacity. SWIM provides an augmented parators as illusrated in Fig. 20 for up to 10 light reality overlay of the capacity of a sensor to sense.
sources, and Fig. 21 beyond (e.g. more typically on Another innovation of SWIM is the capacity to vi-the order of 1000 light sources).
sualize not just a wave, but an entire Fourier series of An embodiment specific to radio waves is illustrated a wave, i.e. to see and visualize the harmnonic nature in Figure 22. An alterntive embodiment is illustrated and structure of a wave. For example, SWIM can trace in Figure 23.
out the waveform of a trumpet playing a note such as This system depicted in Figs. 14, 15, 16, 17, 18, 19, A440, and then trace out the waveform of a flute play-20, 21, 22, and 23, allows us to see with the naked ing the same note. These two waveforms will appear different due to their different harmonic structure. The eye, or on film or video or capture on a sensor array, ability of SWIM to do this is based on the invention sound waveforms coming from musical instruments, ra-of a new kind of lock-in amplifier. This new kind of dio waves coming from cellphones, motion sensors, and various other things that produce fields such as eletro-lock-in amplifier is illustrated in Figure 25.
magnetic radiation fields, soundfields, etc..
The embodiment shown here is functionally equiv-SWIM can also be used as a new kind of bug- alent to the Mann-modified PAR124A. There are sweeper, e.g. to see not just fields, but to see also three INStrumentation AMPlifiers, indicated as "INS.
the capacity to sense fields. In this way SWIM can AMP.". Each of these is an instrumentation op amp c6, 5 6,4....1()) kt)( CO 5 OE
c s kx) co(t,kr6) Figure 22. SWIM used to visualize radio waves from a transmitter Tx emitting a radio wave of the form cos(wt ¨ kx), at frequency w and wavenumber k as a function of space x and time t. A receiver Rx feeds a mixer such as a four-diode ring modulator, which is also supplied by a local oscillator L.O. that generates a good strong signal cos(wt) strong enough to keep the diodes conducting even in the presence of a weak signal at Rx. The output of the mixer is fed to the linear array of light sources in the SWIM, which includes a lowpass filter to filter out the high sum frequency, and admit only the low difference frequency cos(-- kx) = cos(kx). This difference frequency is a function only of space, not time.
with selectable input impedance: 109 ohms to match 4.2. NARLIA Circuit Components the Hi Z" on the PAR124A which is 1 Gigohm input impedance, and a Lo Z' setting suitable for use with Again, with reference to Figure 25, the EQUAL-a standard 600 ohm microphone (e.g. to be able to feel IZER helps to clean up the signal by allowing the user the capacity of a microphone to listen). The PAR124A
to filter out any 60 CPS or 120 CPS powerline hum or input impedance was selected by switches and by re-buzz, as well as introduce further low-cut and high-cut placeable modules. Four input impedances of the em- filters.
bodiment shown here are: 1 Gigohm: 1 Megohm; 600 The PURIFIER helps with the generation of a refer-ohms; and 50 ohms.
ence signal from the reference input ("REF. INPUT.").
The reference input often comes from a noisy signal.
The reference input has one amplifier for a gain that The PURIFIER is not merely a lowpass filter, but, is adjustable from 1 to *10,000 in a 1, 2, 5 sequence, rather, it determines the pitch period of the input, and *
with a calibrated vernier for gains in between. The sig-provides a pure sine wave having the same frequency nal input has two amplifiers, cascaded (on either side and strength (e.g. amplitude) and relative phase as the of an equalizer) for a gain that is adjustable from *1 to input.
*100,000,000, almost matching the original 23-position The REFERATOR (reference waveform generator) rotary switch of the PAR124A adjustability from nano-generates a reference waveform in which all of the har-volts to millivolts. The AD8429 from Analog Devices, monies are exactly equal in strength. It has an 11-with a noise level of 1nV*sqrt(sec) was found to per-position rotary switch on the front of it, and when the form quite well for this purpose. Each amplifier has switch is set to 1, it operates in the identity mode, i.e.
rudimentary equalization: a switchable gentle high-cut its input is the same as its output. When the switch filter, by way of a 3 position switch: wide-open (to 200k is set to 2, its output is the superposition of two sine CPS), 50k CPS, and 5k CPS.
waves, or cosine waves, one that is at twice the fre-47k1 111 fob, 525 G tie 1(\-e ry o vh-eiAcki ww/A)tedµ
RA( =
=

6(1 4 ]
4 Tcyhekts -----> Spa 5-1 Figure 23. Doppler-shift reflection bounce embodiment: Here the transmitter Tx and receiver Rx are both stationary.
The SWIM itself (including the wearer's body in a wearable embodiment of the invention) act as an antenna of sorts, in the sense that waves from the transmitter Tx are emitted in various directions, including directions toward the SWIM wand of lights. Those waves bounce off the SWIM wand and scatter in various directions, including directions back toward the receiver Rx. Some of the transmitted wave from the transmitter (in this example transmitting at 10.525 gigahertz), is used as the reference signal in a form of homodyne detector, lock-in amplifier, analyzer, mixer, or the like, so as to produce a baseband signal that is a function of space rather than time. In this case the spatial frequency is twice that of the setup depicted in Fig 22.
quency of the other. When it is set to a number, it monies. In the mode of receiving and displaying radio outputs a sum of sine waves or cosine waves at each waves, the reference signal is sensed and derived from frequency. When it is set to INF. (infinity), what we a carrier frequency of interest, through a PLL (phase-get is just a stream of short pulses, one pulse for each locked loop). In the metaveillance mode, the reference period. signal is generated rather than derived. A generated reference signal is transmitted, and a highly sensitive The REFERATOR must produce two outputs, one scanner is used to detect minute changes that are due which is the Hilbert transform of the other. This to that reference signal.
The most sensitive appara-means, of course, that one output will be a sum of tus for doing so is a lock-in amplifier. Such amplifiers sines, and the other output will be a sum of cosines. can typically amplify up to a billion times gain. The By convention, let us say that the first one, call it g(t), PAR124A
amplifier has a full-scale range from 1 volt is for the cosines and is the one that is fed to the upper down to 1 nanovolt, and with a further transimpedance mixer, and the second one, call it h(t), is for the sum stage can sense down to the femtoamp range.
of sines and is the one fed to the lower mixer. The first one is buffered output "REF. MON. I." and the sec- 43. Historical context ond one is buffered output "REF. MON Q.". Some of these connections are shown in blue, but for simplicity This work is based on work by S. Mann, referred in the diagram, not all are shown. A reference signal to as Phenomenological Augmented Reality [50], and is derived or generated, and ideally riot merely a pure the NARLIA
(Naturally Augmented Reality through sine wave, but, rather, a signal that contains all the a modified Lock-In Amplifier) project, conducted in frequencies of interest, up to a certain number of har- the 1970s, based on a specially modified PAR124A/126 "!*
Sq>
,111-SIGNAL
SENSORSWIM
WAVEFORM -;2.,=1' LIGHTS
((c 01 411( REFERENCE
ACTOR
SWIM
WAND
-)10.-- MOVEMENT
r ____________________________________________________ INN
PATH
BUG SIG. REF.
AMP. GEN.
= .=
LIA SWIM
OR QS OX __,. COMP.
A A ___________ SWIM
_____________________________________________________________________________ SWEEPER
Figure 24. Metaveillance: SWIM as a bug sweeper or other device to sense sensing, i.e. to sense sensors and sense their capacity to sense. In this case, the reference signal comes from something attached to the SWIM wand, such as a loudspeaker or a transmitter. This "Reference actor" can be an actuator (e.g. speaker), or a transducer that has no moving parts, or a direct conveyance like an antenna that simply conveys electricity directly without transduction. A SWIM
wand comprises the reference actor together with the SWIM lights, a typically a 2 dimensional or 1 dimensional array of lights such as LEDs. The SWIM wand may also bear various other forms of user-interface such as keyer, tactile actuator, etc., so the user can feel veillance waves pressing against the hand, for example, while approaching a hidden microphone.
The reference actor is either self-oscillating, or is driven by a reference generator. The reference generator (or a signal from the self-oscillating reference actor) is connected to the reference input of the LIA. The signal input from the LIA comes from the bug itself, or from a stationary signal sensor placed near the bug's suspected vicinity, its location being refined as we learn more about where the bug might be. In some situations we have access to the bug, e.g. we've found it, or it might be of our own doing (e.g. simply our own microphone we wish to visualize). In other cases we might not want to touch it, e.g. if it needs to be dusted for fingerprints and we don't wish to disturb it. In either case, the bug itself or the signal sensor is connected to the signal input of the LIA.
LIA (Lock In Amplifier). The standard PAR124A am- covery (formerly Princeton Applied Research) plifier was originally manufactured by Princeton Ap- introduces the 7124.
The only lock-in ampli-plied Research (PAR) in the early 1960s. fier that has an all analog front end separated, The PAR124A is widely regarded as the best lock-in via fiber, from the DSP
main unit." [Signal-amplifier ever made, and others have attempted (with Recovery.com main page of website, accessed various degrees of success) to imitate it. June 2016]
'Since the invention of the lock-in amplifier, Signal Recovery states that the PAR124A from more none has been more revered and trusted than than 50 years ago is still the most revered and trusted the PAR124A by Princeton Applied Research.
amplifier, and they go on to claim that their new 7124 With over 50 years of experience, Signal Re-product matches the performance of the 124A. The NARLIA (Naturally Augmented Reality Lock-In Amplifier) ___________________________________________________________ D.). SIG. MON.
rovERDRIVE INDICATOR 1>--10-REF. MON.
I.
¨0-30-REF. MON. Q.
1-10k; 1,2,5 seq -0=-=10.-X
7*500hm PARAMETRIC buff.
out.
INS. EQUALIZER selectable DC or AC
A1 C.-310-R to BNC+BBP
SIG. INPUT ' MP. slow/fast : i?k, 2 1>-01-PHASE
*MODE = ¨ X
and switch 111110. = *FREQ. LPF ¨0 _____ DISPLAY for to select A, - *Q
11\1)1;;7-1111' B, or A-B 1-10,000 :
AMP. . _ X, Y, R, and y PHASE ANGLE
in a 1,2,5 LPF __T...
sequence HARMONIC
selectable SELECTOR
DC or AC, (11-pos'n dill POSTAMP.
g(t) , (t) is the Hilbert slow or fast switch.) Gainbias ________________________________________________ transform of g(t) PITCH ______________________________________________________________ and R
REF. INPUT..Ø Ø PERIOD --).- 4 j 5 6 7 Analog INS.R derived voltages, Amp. PURIFIER: 2 10 il EVEN I
1 from X 0 to 5 Generates INF. ODD and Y, 1 volts a pure REFERATOR: affects lamp sinewave Generates reference brightness, with period waveform with the and amplitude and is also selected harmonics.
matching input. available for other SEE "MODEL 124A FUNCTIONAL BLOCK DIAGRAM", purposes.
FIG. 3-1 on page 12 of the PAR124A instruction manual, PRINCETION APPLIED RESEARCH CORPORATION, 1970.
S.W.I.M. i RF ADAPTER Linear array of LEDs and RF FREQ. T.-S.W.I.M., MIXER tactile actuator.
RF INPUT Voltages from OUTPUT TO ABOVE NARLIA INPUT Lock-In Amplifier drive the DC
servomotor inputs.
RF SIG. GEN.

Figure 25. A functional equivalent to the NARLIA (Naturally Augmented Reality through a modified Lock-In Amplifier) based on Mann's modifications to the PAR124A lock-in amplifier.
7124 presently sells for approximately $13,000 US. by long-gone manufacturer for decades. The In a paper published just last year, a comparison paper uses 124A and 7124 lock-in amplifier was made between the older and newer amplifier, which system to test noise and response signal of found that the older amplifier performed better. The several photoconductive detectors...* [80]
paper begins:
Whereas modern lock-in amplifiers (including the "The noise of photoconductive detector is so SR124 and SR7124) work by sine wave multiplication, weak that the PAR 124A lock-amplifier is the PAR124A worked by rapidly reversing the polarity main test facility despite of discontinuation of the input signal. This is equivalent to multiplying 1 Fourier coefficient 1 SW1Sitnef coefficient .
i ' 2 Fourier coefficients 2 SWIllaef. coefficients IV \ li \ filki \II
-1o.
1 2 -2 3 4 0 1 2 3 4 rI 11,1firr 3 Fourier coefficients 3 SWISAref. coefficients .
_ 1. fln 3 4 ____________ V o -WV . )11 Aiv. A flA 1,,,,,vIrn ;
-1o 1 2 3 4 -30 1 2 3 4 LiA3, 4 ' 8 Fourier coefficients 8 SWISket. coefficients5 .
6 ___________________________________ ,1411 _1 0 Unlyr4LLILit, .
Space Space¨ 7,õ

Figure 18. Left: A modern LIA (Lock In Amplifier) ignores , all but the fundamental. Older LIAs use polarity reversal343k .,,,,IN ,.
and are thus sensitive to increasing harmonics on a 1/n =,=
basis where n --- 1, 3, 5, .... This is why older LIAs often ,. . - ,- -= ¨= = = ===-= =,-- -.. = : -:-.4.;awiatwomaajoi7::!...-v=.="...::::;.=.'...,....L.....titr:
work better with the SWIM (Sequential Wave Imprinting . . - ..- ' r"';'7_---ir,, , _ ..
Machine)[461, as long as they're modified to compensate .
for weaker higher frequency components of the waveform 1 being visualized. Right: Reference waveforms of Mann's Mi. e..x.
"Alethioscope" have equal weightings of all harmonics. As we include more harmonics, instead of approaching a square ., wave, we approach a pulse train. Early SWIM used a pulse Ji .-:, . . , ,,,, t,`,=-'-' :',. 3µ..=44;, train as its reference signal. This made time "sit still" (like ,.. . . ...
.., , ,,c, a strobe light on a fan blade) for a true and accurate AR. Figure 20. In one embodiment of the SWIM, the output of (Augmented Reality) visualization without distortion.
a lock-in amplifier is supplied to an LM3914 chip connected to 10 LEDs which are attached to the receiver, and waved back and forth as a unit. In one embodiment, a baseband the input signal by a square wave. This allows odd liar- output of a frequency mixer in a homodyne Doppler radar set is connected to the LED barg-raph formed using the monies through, thus creating a kind of comb-filter in LM3914 chip.
the frequency domain. The PAR124A has a symmetry adjustment to make the square wave perfectly symmet-rical. In the Mann modification to the PAR124A, the output. The phase-coherent detection includes an in-symmetry is deliberately offset to one extreme or the phase ("real') output and a quadrature ("imaginary") other, so that the square wave is highly asymmetrical, output, one of which drives the comparators, and both thus allowing even harmonics to come through as well.
of which control the overall brightness of the lights for This modification unfortunately also creates a strong best visualization of the phenomenon being studied, i.e.
DC offset that must be corrected immediately after the the light brightness varies in proportion to the magni-mixer stage.
tude of an electromagnetic radio wave. In this way, Once performed, the result is a lock-in amplifier in the lights get brighter when the signal gets stronger.
which the reference signal is essentially a stream of There is a control for this effect, i.e. when set to zero short pulses, thus allowing all harmonics (even and the lights stay at full brightness always, and when set odd) of a signal to be represented in the output, which to maximum the lights vary from full to zero.
is connected to a linear array of light sources, or a hap-tic actuator (or both) that is/are waved back and forth. 4.4. A simple illustrative example using sound waves Each light source is connected to a comparator respon-sive to the output of the specially modified PAR124A The PAR124A (including the custom modifications) amplifier, and the comparators of all the light sources covers a frequency range up to 200,000 cycles per sec-are fed by a ladder network of voltage dividers so that ond, which works well for grasping, touching, holding, each light represents a consecutive voltage range of the and exploring sound waves (See Figure 26). For radio There are 21 cycles of this 411( _____________________________________ sound wave over its _______ >MP
1.5 metre distance of travel.
This moving speaker emits a 5000 CPS --) .Each cycle is (Cycles Per Second) 150cm/21 = 7cm long.
=
tone, which this microphone hears.
11.
f , *
, _1:µ
; ;
= -Row of green lights moves with speaker and Ø0111111 ! displays output of lock -in amplifier from microphone.
= .
r. .
**U=
Lock-in amplifiers.
Measured speed of sound = 0.07 metres/cycle *5000 cycles/second = 350 mis. (True value at 27deg. C is 347m/sec) õ
Figure 26. Sensing sensors and their capacity to sense: SWIM (Sequential Wave Imprinting Machine) used to show the capacity of a microphone (hand-held here) to hear, by way of a speaker affixed to a Linear array of lights that waves back and forth on a robotic arm. The waveform is visible by persistence-of-exposure, either by the human eye, or by photographic exposure. Here we can see and determine the speed of sound, since we have a 'sitting wave [511, as distinct from the concept of a standing wave. Since the wave is spatialized at a true and accurate physical scale, we can simply count the cycles, and divide the total distance by this number, then multiply by the frequency (cycles per second) to compute the speed of sound.
The wave appears "frozen' in spacetime, as if the speed of sound were zero, so we can see it.
waves, which go beyond this frequency, a radio fre- sweeper', in which the results of the bug sweeper are quency signal generator and frequency mixer are used experienced and spatialized (perfect spatial alignment to bring the radio signals down into the audio range. of the virtual overlay with reality) in real time. The speed of sound has been canceled, in effect, by the het-Figure 26 shows a simple teaching example with erodyne nature of the lock-in amplifier, resulting in a NARLIA, in which the speed of sound can be calcu- 'frozen' wave that appears to sit still rather than travel.
lated. Here the goal is to make tangible a microphone's We call this a 'sitting wave' [51] (distinct from the con-capacity to hear. An excitation source (in this case, cept of a standing wave). This example shows simply at a frequency of 5000 cycles per second) is generated a single frequency component of the wave, which is all by a speaker that travels together with the SWIM or that a conventional lock-in amplifier can do. But with T-SWIM. This functions as a tangible embodied 'bug ri7tin ite?
r, 4Wie, A
C..j.":7õ.. =
Figure 27. We're often being watched by motion sensors like the microwave sensor of Fig. ??. Left: When we try to look at the received baseband signal from the sensor, as a function of time (artificially spatialized), it is difficult to Figure 29. Miniaturized wristworn SWIM: Metaveillance understand and has little meaning other than a jumble of for everyday life.
Left: Invented and worn by author lines on the screen. Center: When we shut off the timebase S. Mann.
Wristworn SWIM makes visible the otherwise of the oscilloscope, and wave it back and forth, we see the invisible electromagnetic radio waves from a smartphone very sante waveform but displayed naturally as a function (heterodyned 4x/8x as if 20,000MCPS). Right: array of of space rather than time. Right: Stephanie, Age 9, builds LEDs on circuitboard made in collaboration with Sarang a robot to move SWIM hark-and forth in front of the sen- Nerkar. We find a listening device concealed inside a toy sor. As a function of space the displayed overlay is now in stuffed animal (Okapi). Visualized quantities are the real perfect alignment with the reality that generated it. This part of measured veillance wave functions. Magnitudes of alignment makes physical phenomena like electromagnetic these indicate relative veillance probability functions.
fields more comprehensible and easy to see and understand.
400404414,' - ----..... t 6, "fr =
Figure 30. Left: Sliderail SWIM to teach veillance wave principles. A speaker emittins a 10050 cycles/sec. tone.
=
The microphone's metawave has 11 cycles in a 15 inch run.
I ;4? 1 Teaching speed-of-sound calculation:
15 in. * 10050 cy-Nt cies/sec / 11 cyles = 13704.54...
348.09... m/s.
I At 25deg. C. theoretical speed of sound = 346.23 m/s (0.5%
measurement err.). The real part of the veillance wave-Figure 28. The SWIM's multicomponent/arbitrary wave- function is shown but SWIM can also display magnidude form lock-in amplifier. Square wave visualized using multi- (steady increase toward microphone). Right: "Bugbot"
component reference signal cos(cot) + cos(3wt) making vis- (bug-sweeping robot) finds live microphone hidden in book-ible the first two terms of its Fourier series expansion, shelf and visualizes its veillance waves in a 7-dimensional resulting in phenomenological augmented reality display (3 spatial + RGB
color +time) spacetime continuum.
of cos(cat) + 1/3 cos(3wt) on 600 LEDs in a linear array (Green=strongest;
redshift=toward; blueshift-=-away).
rapidly swept back and forth on the railcar of an optical ta-ble. This suggests expanding the principles of compressed sensingt10, 25] to metaveillance. Inset image: use beyond pounds or approx.
18kg). So in 1974, Mann in-veillance, e.g. atlas of musical instruments (trumpet pie- vented the SWIM
(Sequential Wave Imprinting Ma-tured) and their waveforms. Amplifiers in picture: SR865 chine). The SWIM, waved back-and-forth quickly (drifty with screen malfunction); SR510; and PAR124, not by hand or robot, visualizes waves, wave packets used at this time. (wavelets), chirps, chirplets, and inetawaves, through PoE (Persistence of Exposure) [46]. See Fig. 29 and NARLIA we can touch and hold and experience waves http://wearcarn.org/swim of various shapes and compositions.

5. Phenomenologial AR bots and drones 4.5. From timebase to spacebase Constrained to linear travel, SWIM is useful as a A more recent re-production of this early experiment measurement instrument (Fig. 30). Over the years the author built a variety of systems for phenomenologi-is illusrated in Figure 27, with an oscilloscope-based cal augmented reality, including some complex-valued implementation. An LED-implementation is shown in wave visualizers using X-Y oscilloscope plots as well Fig. 28. An Android-based version was also created.
as X-Y plotters (X-Y recorders) replacing the pens 4.6. Wearable SWIM with light sources that move through space. In one Oscillographs were too heavy to swing back- embodiment an X-Y plotter is connected to the real and forth quickly (RCA Type TMV-122 weighs 40 and imaginary (in-phase and quadrature) components Arat Wit, dpõ
44µ
go!?
gra ; Aft Alip*
Figure 31. Complex-valued "gravlet" wavefunction visu-alized on a robotic SWIM that spins while moving back and forth. Data[1] from LIGO[2] was used with its Hilbert transform, noting the result is a chirplet[55, 56, 57, 58, 11, Figure 33. (Top image) Wearable camera system with 66, 28, 12] ("gravitational signal" rather than "gravitational augmented reality eyeglass meets video surveil-wave"). SWIM explores periodic realtime data at any scale lance. (Bottom row) Drone meets video surveil-from atomic to cosmic, as well as displays arbitrary data.
lance. Surveillurninescent light sources glow brightly when within a surveillance camera's field-of-view, resulting in of the author's special flatband lock-in amplifier and augmented reality overlays that display surveillance cam-pushed through space to trace out a complex wave-era sightfields [49]. The overlays occurs in a feedback loop, form in 3D while a light bulb is attached where the so alignment is near perfect and instantaneous, because it pen normally- would go on the plotter.
is driven by fundamental physics rather than by compu-More recently, Mann and his students reproduced tation. Metawavefunction sampling is random and sparse, but recoverable [14].
this result using a spinning SWIM on a sliderail to reproduce gravitational waves - making visible an oth-erwise hidden world of physics. See Fig. 31.
creates an illusion that the speed of light, or sound, Camera metaveillance: Another variation on etc. is equal to zero so that otherwise imperceptible phenomenological augmented reality is to map out veil-waves and metawaves can be explored both visually lance from a surveillance camera, as metaveillance (see-and haptically. We discuss the design of TSWIM and ing sight). For this, a light source is connected to an its potential in creating an embodied understanding of amplifier to receive television signals, and indicate the the transmission and surveillance phenomena around metaveillance by way of video feedback. In one em-us. The result is a Natural Augmented Tactile Real-bodiment the light source is a television display (see ity, a new human augmentation framework for sensory, Fig. 32 (top row)). In another embodiment, the light computational, and actuatorial augmentation that lets source is a light bulb (see Fig. 32 (bottom row)).
us experience important phenomena that surround us ARbotics (AR robotics) can also be applied to vi-yet are otherwise invisible. Examples include the abil-sion (Fig. 33). Here we map out the magnitude of the ity to physically grasp and touch and feel electromag-metawave function, where the phase can be estimated netic radio waves, gravitational waves, sound waves, using Phase Retrieval via Wirtinger Flow [14 and metaw-aves (e.g. the sensing of sensors).

6. Tactile-SWIM
T-SWIM combines visual and haptic feedback to al-low users to both see and touch/grasp/feel waves in A variation of the SWIM is T-SWIM (Tactile - Se-their environment giving rise to a new form of tactile quential Wave Imprinting Machine), a naturally aug-Augmented Reality where our sense of touch is physi-mented tactile reality system for making otherwise cally realized in perfect alignment with the phenomena intangible electromagnetic radio waves, sound waves, of the reality around us.
metawaves, etc. graspable. TSWIM uses a haptic ac- See explanation in tuator driven by a special type of lock-in amplifier that http://wearcam.org/kineveillance.pdf as well as [46, ?[.
synchronizes with traveling waves to transform them Alignment is automatic and immediate due to natural into coordinates in which they are sitting still. This physics, without any need to sense or implement -LTransmitter levision camera ,\/IIII,k>4jNp Sightfield "Rabbit ear , receive antenna\
' .

Television r , _ receiver , /Transmitter ' ,ittli.;440-Tlevision camera 414*
f ; Sightfield Iv IP AI e otod "Rabbit ears" \ = . .::-, '..=-=::';',I.N. - -_,'-----.
receive antenna / .,......
. . ..
PHENOMENAmp ,\.-_'----------, Figure 32. Metaveillance by video feedback: Phenomenologial augmented reality overlay. Top row: a TV
(TeleVision) receiver is moved around in a dark room. A back-and-forth sweeping motion works best. As the TV receiver moves in and out of the camera's sightfield, the TV glows brightly when in view of the camera, due to video feedback, leaving an integrated exposure on photographic film. Here we see multiple black and while recordings presented on a pseudocolor scale for HDR (High Dynamic Range) of the overlay. Bottom row: the TV receiver can be replaced with a single light bulb (center image) or a linear array of light bulbs that sweep out an overlay of the camera's sightfield on top of visual reality (rightmost) giving a phenomenological augmented reality.
registration between the real and virtual worlds.
change with varying distances from the source in all TSWIM consists of two primary components: A
dimensions. The device itself can be held in a number modified lock-in amplifier and a graspable linear actua-of ways resulting in distinct haptic/tactile sensations:
tor with a combined receive antenna or other sensor at-by using the finger loops, by placing the thumb on the tached to the actuator (Figure 34). The actuator along rail and feeling the pressure of the motor as it attempts with the sensor borne by it (e.g. the receive antenna) to follow the wave, or by holding the actuator in the is held and moved throughout the space surrounding hand and feeling the inertial forces on the whole device a wave source. As the actuator pa_sses through the a.s the actuated components move. Further, users can various crests and troughs of a sitting wave [51], the interact with the source itself, placing hands or objects actuator pulls the user's fingers apart and together.
in between the path to the antenna to dampen or reflect Furthermore, an LED mounted on the device travels the wave (Figure 35).
with the user's finger to provide a corresponding vi-6.1. Metaveillance sual representation of the wave pattern, when viewed by way of a camera or the human eye. When the actua-Users can also observe metawaves, i.e. T-SWIM can tor is slid back and forth along the wave with sufficient sense sensors and sense their capacity to sense (Figure speed, persistence of exposure [51] results in the user's 36). By recording the response of a surveillance device seeing a complete sitting wave (see Figure ??).
to a known sig-nal, the metawavefunction of the device The user can explore various wave patterns as they is rendered. In this way, TSWIM becomes a form of =,,-.
. '''' 't : ' '=
e mr. t . (4k t ..
Figure 35. Demonstrating the attenuation of radio wave propagation through various materials. From left to right: unob-structed, thin wood, thick wood, human hand, copper foil.
prevent its movement. In this way the user can grasp INA and hold and feel and touch and explore radio waves, t sound waves, etc., and especially sitting waves.
-.T-SWIM uses a "feedback linear actuator" such as LED Llneill. the DC servo mechanism salvaged from or adapted . liki=

- .
# potentlooleter from a Hewlett Packard XY plotter, XY
recorder, = !a stripchart recorder, or the like.
...
i-. , ., . == '' [ . 11¨ ' '. . = ' ..
. A more modern satisfactory linear actuator is the Progressive Automations PA-14P that comes in stroke 07i = F i ng c.:17,=olloir.c: :p- to fropnol2its, d speeds an o atinsephes, writhgfionrg from aboutcesorangingfr1 6 1 p 35 s o 50 uid . Alisiiii0,,. = -i I S-ei-Warnotor to 2 ips (inches per second).
A faster-responding, and gentler (e.g. in terms of ,==.
user comfort) feedback linear actuator is the "montor-ized potentiometers" such as the Sparkfun COM-10976 '.--...'.:' (https://www.sparkfim.com/products/10976), or the Till11111) 1001) original "ALPS Fader motorized RSAON11M9 touch sensitive 10K linear slide potentiometer" which many others have immitated.
In an alternative embodiment, a hobby servo is used Figure 34. T-SWIM's tactile actuator (closeup).
as the linear actuator (or substituted as a rotary actu-ator with a long arm). The hobby servo is driven by a converter that converts analog voltage to pulses of augmented reality overlay, allowing users to see and pulse width varying in proportion to the analog volt-feel sur/sous/veillance fields/flux around them. age. Alternatively, an Arduino with analog input can be used with a PWI\l/Servo Shield, thus converting 6.2. Technical Implementation analog input to linear or rotary actuation that can be Embodiments of the early T-SWIM prototypes used touched, felt, grasped, etc., plus also move an LED
various X-Y recorders, X-Y pen plotters, or the like. attached to it for visuals. Alternatively the linear ac-In one embodiment, an HP (Hewlett Packard) X-Y tuator of Sean Follman's inForm system 1301 (based on plotter has the X-axis disconnected so that the pen the Soundwell clone of the ALPS fader) can be used.
or another device in its place (a hapticitactile handle A receive antenna is attached to the linear actua-or grip) can move freely and effortlessly back-and forth tor and picks up the target signal and feeds the sig-in the X direction, while the Y-axis was driven. In this nal to the lock-in amplifier (see Figure ??. An AT-way, a user can grab the pen or grip of the X-Y plot- MEGA2560 microcontroller provides PID 181 control ter and move it back and forth left-to-right to freely using the actuator's built--in linear slide potentiome-explore a tactile or haptic augmented reality. To the ter.The actuator is fitted with adjustable loops for the left of the X-Y plotter is mounted the SIGNAL AC- user's thumb and index finger, as shown in Figure 34.
TOR, SIG. GEN., or the like of Fig ??, or the BUG
There are two fundamental operational modes of T-of Fig 24. The output of the LIA, as marked "X" on SWIM:
the LIA of Fig. ?? or Fig. 24 is connected to the "Y"
input of the X-Y plotter, causing the plotter to move = touching and grasping electromagnetic radio the grip up-and-down or to apply a force to the grip of waves, i.e. a haptic/tactile version of the embodi-the user stalls it or graps it firmly enough to slow or ment shown in Figs ?? and 24;

=
feeling veillance flux, e.g. feeling the effects of a 3. inertial feel of the actuator and its LED or other surveillance camera, i.e. a haptic/tactile version load, as it flutters up and down quickly or more of the embodiment shown in Fig 32. slowly.
The invention alows a user's finger to be moved or The lock-in amplifier is the same design as described by Mann in the original SWIM device (see [51] for tech-influenced (in a tactile or haptic sense) by a wave.
This makes otherwise invisible waves graspable (tan-nical details). It amplifies a target signal with a moving time scale equal to the wave's speed, resulting in a sta-gible). For observation of transmitted waves, the user is physically pushed and pulled by various wireless de-tionary frame of reference with respect to the wave. In vices around them (i.e. devices that emit radio waves), the case of rendering waves from a transmitting source, and in the case of metaveillance, users can literally the amplifier is connected to the actuator's receive an-feel surveillance cameras pressing against their bod-tenna, while performing phase-coherent detection of ies. This effect grows with proximity, so that users the signal. The amplifier typically has a gain that is ad-can locate the source of a transmission or a reception justable from one to 109 with a dynamic reserve on the (surveillance) by feeling the point or regions of greatest order of 105 (e.g. capable of picking up signals buried tactility.
in noise that is about 100,000 times stronger than the signal of interest). In the case of metaveillance, the amplifier receives input from a sensor while stimulat-ing the surveillance device with a known probing signal (the "reference" signal of a specialized lock-in amplifier as described in [51]). )111i, ;
63. T-SWIM Discussion Various users were recruited to try TSWIM and learn differences between purely visual and visuo-haptic feedback. Users explored radio waves from a 10.525 GHz radio transmitter and the veillance flux of Figure 36. Measuring veillance flux of a CCTV surveillance a CCTV camera.
camera, and rendering that veillance flux as tactile interac-tion in a realtime feedback loop. A black cloth was placed In the metaveillance case (seeing the capacity of a behind the camera at the left side of the frame, to make the CCTV camera to see), users remarked that in contrast green LED on the tactile actuator more visible, and show to simply observing visual changes in the LED bright-its degree of exertion. Zooming into this picture you can ness, being physically acted on by the surveillance of see the faint trails the LED makes on either side of the cen-the device caused an innately emotional response in tral sightlines, as faint dotted lines formed by the pulsating them. Interestingly, while some interpreted the effect (oscillating) of the electrically modulated LED.
as being intrusive, one user remarked that he felt com-forted and protected by the tug while in the camera's 6.4. Going further with T-SWIM
field of view. Another found that the haptic sensa-tion allowed her to detect the surveillance in a discreet While the TSWIM device was designed to displace way, such that others were not notified of the aware-the finger along wave, other tactile/haptic representa-ness. She liked the idea of knowing that she was being tions of phenomenological augmented reality are pos-watched, without others knowing that she knew that sible. In general, vibrotactile stimuli can be modu-she was being watched.
lated by the amplitude (real or imaginary), frequency, T-SWIM allows the user to move their finger or phase of the observed wave, as applied as forces to through electromagetic radio waves in their environ-a static finger, or to provide movement to the finger, ment. It is possible with the T-SWIM invention to feel proportional to the wave amplitude, frequency, phase, and grasp wave representation, signals, etc., experienc-or the like.. Alternatively, the force or movement can ing:
be proportional to the amplitude of a wave (e.g. to the square root of the sum of the squares of an in-phase 1. motion of the finger along a wave; and a quadrature component).
Multiple TSWIM tactuators can be used together, 2. forces applied to the finger as a servomotor acts so that a user can simultaneously experience various even if it is stalled (when a user deliberately resists locations along a wave by running all of their fingers movement); over it.

Other examples of such CorepointTmtechnology in-clude a planking board in which the user's core mus-cles become a joystick, pointing device, or the like. The Inom board's shape is a circle, or alternatively, a point shape, such as like a cursor or directional element, such as the front of a surfboard. A satisfactory shape is designed = using SWIM as a haptic augmented reality computer Figure 37. Study of lens aberrations using Veillance Waves.
aided design tool, as shown in Fig. 38 This shape is Left: Cartesian Veillance Waves; Right: Polar Veillance suggestive of a cursor or pointer.
Waves. Lens aberration visible near lower left of Veillance The plankpoint board is mounted on a pivot so it Wave. Mann's apparatus (invention) attached to robot can tilt and turn. There are up to 4 degrees of free-built by Marc de Niverville.
dom which operate a gaming console in the following manner:
In some embodiments, the SWIM or T-SWIM are wireless devices that allow a user to walk around un-= tilt left-right (port-starboard), which moves the tethered, while grasping and holding electromagnetic cursor left-right;
waves or other signals.
TSWIM is a system which allows for unique visuo-= tilt fore-aft, i.e. tilting the board forward so the haptic exploration of waves and metawaves in the real front goes down and the back goes up, results world. As a natural augmented tactile reality system, in the cursor going down, and tilting back (stern it bridges the gap between the virtual and the phe-down and bow up) results in the cursor going up;
nomenological, and allows us to explore signals with continuous (i.e. non-discrete or "undigital") feedback.
= rotate, while keeping the board level or at the same overall tilt angle results in an advanced game func-tion;

7. Sparsity in the Spacetime Continuum = push-pull, i.e. putting more or less weight on the In conclusion, Metaveillance and Veillance Wave-board results in another advanced game function.
functions show great promise as new tools for under-standing the complex world where surveillance meets A srnartphone is placed on the plankpoint and its ac-moving cameras (wearables, drones, etc.). Further re-celerometer senses tilt angle, for the cursor left-right search is required in the area of Compressed Sens-and up-down functions. A game is written that works ing [10, 25] to fully utilize this work, e.g. to build in this way, so a player goes into a planking position to completely filled-in high-dimensional spacetime maps control the game.
of Veillance ("Compressed Metasensing").
In this embodiment, the invention comprises a Moreover, just as oscillography was the predeces-planking board with a tilt sensor that moves a cursor.
sor to modern television, the alethioscope and robotic A score is provided that is proportional to the absement SWIM ("ARbotics") could be the future that replaces or absangle (integrated absolute angle) of the board, as television, VR, and AR.
computed by the square root of the surn of the squares Finally, surveillance, AI, and security are half-truths of the two cursor dimensions away from a target point.
without sousveillance, HI, and suicurity (self care). To The target point moves throughout the game to create write the veillance/cyborg code-of-ethics we need to a gaming environment for core fitness training. The fully understand all veillances and how they interplay.
objective of the game is to get the lowest absement or Metaveillance gives us the tools to accomplish this absangle. A score can be presented as a reciprocal of understanding, in a multi,cross,inter/intra,trans,meta-this, so that the goal becomes highest score. Alterna-disciplinary/passionary mix of design, art, science, tively score can be based on presement, i.e. integral of technology, engineering, and mathematics.
reciprocal displacement or reciprocal tilt angle. Typi-cally a target moves and the user tries to follow it. See

8. Corepointm4 Fig. 39. When the target sits still, the invention is also still useful, as the integral is the absement, and we can One embodiment of the invention involves wrestling score by minimum absement or maximum presement.
with a robot to achieve a high degree of simultaneous There are various forms of the corepoint technology dexterity and strength. With this "Wrobot" (wrestling which include:
robot), the body can become a joystick or pointing device. = plankpointTm;

' ,*
opt, =
Figure 38. The planking pointer shape is made using SWIM to create two waveforms overlapping that define the shape.
This process of computer-aided design gives a shape that resembles a cursor, but with nice smooth edges.
TARGET
An important special case of Sousveillant Systems POINTER
is that of scientific exploration: not only is (big/little) !" =
Ask / ERROR AREA
data considered, but also due consideration must be given to how data is captured, understood, explored, -.101 and discovered, and in particular, to the use of scien-tific instruments to collect data and to make important Figure 39. Corepoint uses a typically time-varying tar-new discoveries, and learn about the world. Science is a get function, and the core muscles control a cursor. The domain where bottom-up transparency is of the utmost accumulated area between the two curves forms the time integral that the user tries to minimize. When the target importance, and scientists have the right and respon-is not moving, this area is the absement.
sibility to be able to understand the instruments that they use to make their discoveries. Such instruments must be sonsveilkznt systems!
=
pullpointTm; A simple example is a ShowGlowTmor Flow-GlowTmelectrical outlet which has built in LED (Light =
JUMPointim. Emitting Diode) indicators showing how much voltage is present, as well as how much amperage is flowing.
Pullpoint is a pullup bar similiary equipped, but in The outlet glows to indicate what's happening, namely typical embodiments with only one degree of freedom the current flowing through devices plugged into it, not counting in the score. See Fig. 40. The bar may also just the open-circuit voltage. Color indicates phase an-be robotically actuated, as shown rightmist in Fig. 41.
gle, so one can see at-a-glance not just the load, but also the power factor (e.g. how inductive or capacitive

9. Big Data is a Big Lie without little the load is). The simple concept of phase-coherent col-data: Humanistic Intelligence as a ormapping is useful for electrical power, as well as for Human Right the output of a lock-in amplifier in augmented reality wave visualization, such as back-to-back red and green I now introduce an important concept: Trans- LEDs on the in-phase output of a lock-in amplifier and parency by way of Humanistic Intelligence (HI) as a back-to-back yellow and blue LEDs on the quadrature human right, and in particular, Big/Little Data and output of the lock-in amplifier, thus indicating phase Sur/Sous Veillance, where "Little Data" is to sousveil- as color.
lance (undersight) as "Big Data" is to surveillance (oversight).

10. Surveillance, Sousveillance, and just Veillance (Sur- and Sous-veillance) is a core con-plain Veillance cept not just in human-human interaction (e.g. peo-ple watching other people) but also in terms of HCI
Surveillancel (oversight, i.e. being watched) and (Human-Computer Interaction). In this sense, veil-sousveillance2 (undersight, i.e. doing the watching) can lance is the core of III, leading us to the concept both be thought of in the context of control theory and of "Sousveillant Systems" which are forms of HCI in which internal computational states are made visible 1[70, 44]
to end users, when and if they wish. 2[47, 59, 29, 74, 7, 6, 27, 73, 15, 5, 3, 83, 68, 72, 45, 79, 32]

, , -. ..
õ. ;..

-11 , . It IIIPI Ell a" ,õ .._¨

, . ===. ¨ .
- ... V = ¨ _ t.,,..
..
, -.41, . 4.0 ,,,. =
._ =
tor .
..., = ,,õ. ....õ. ,... õ,õ., ,....
:
,, , µI' i )f -¨
% . -Figure -10. Corepointyltechnologies include Plankpoint planking board that allows a user's core muscles to function as a pointing device. The concept of integral kinesiology is also applied to ankle exercises (center picture). Another embodiment is Pullpoinfrm(rightmost). In this embodiment, a pullup bar is suspended from a pivot or swivel, or the like, and the angle of the swivel is sensed by a tilt sensor that forms the input device (pointing device) for a video game. Here in this example shown. the bar is the steering mechanism. In some embodiments, rings are attached to opposite ends of the ()Whip bar to engage the steering mechanism with the user's core muscle groups in a combined requirement of strength and dexterity.
feedback loops3. of -Personal Big Data{?] which might more properly ln particular. McStay considers surveillance in this be called "little data". Both of these concepts embody way, i.e. in regards to the form of privacy that is in- Big Data's sensory counterpart that corresponds more herently violated by profiling, and related closed-loop to sousveillance than surveillance ]51].
feedback systems that manipulate us while monitor-ing us [611. Ruppert considers the interplay between 10.1. The Veillance Divide is Justice Denied surveillance and public space, through a case study of A good number of recent neologisms like: "Big 'Toronto's Dundas Square [?1, where security guards Data". "IoT" (-Internet of Things"), -Al- ("Artificial prohibit the use of cameras while keeping the space Intelligence"), etc., describe technologies that airn to under heavy camera surveillance. This surveillance grant the gift of sight. or other sensory intelligence, without sousveillance [?] creates a lack of integrity, i.e.
to inanimate objects. But at the same time these surveillance is a half-truth without sousveillance [?].
inanimate objects are being bestowed with sight, that The intersection of Sousveillance and Media was very same sight (ability to see, understand. remem-pioneered by Bakir, i.e. sousveillance 118 not merely ber, and share what we see) is being taken away from a capture or memory right, but also sousveillance as humans. People are being forbidden from having the a disseminational (free-speech) right. This gave rise same sensory intelligence bestowed upoi. the things to two important concepts: sousveillance cultures and around them.
sousveillant assemblage [7], analogous to the -surveil-Indeed, we surrounded by a rapidly increasing lant assemblage" of i'34].
number of sensors [16] feeding often closed and secre-Surveillance has strong comwctions to big data, tive "Big Data repositories [?]. Entire cities are built where states and other large organizations, especially with cameras in ever.N..- streetlight [76. 631. Automatic in law enforcement. collect data secretly, or at least doors, luindwash faucets, an e h n d flush toilts tat oce maintain some degree of exclusivity in their access to used "single-pixel" sensors now use higher-resolution the data ',?].
cameras and computer vision [36].
Two important concepts have been proposed to help Surveillance 'is also widely used without regard to mitigate this one-sided nature of big data: (I) "giving genuine privacy, i.e. with only regard to Panoptic-Big Data a social intelligence"[?] and (2) the concept privacy. In Alberta. for example, the Privacy Coin-3[53, 511 missioner condones the use of sureillance cameras in CA 02953186 2016-12-29 ky c . , . S.
. . .
.., _ A
A .
4141. ' rr I
,1; ...,. =
It' , ,r / = / i i ,/, , . , (a)t' . . = =
-- ,-J -IP
110 = =7'=
,.,..._, ,.._....
i -= = = - =%,.õ.
'..?:!t1`,.':",:,,- =
==,:,,, :,='.;:?' ' = '"'"' = ' ., = ¨ ,. = ,. , -0 ... ,:i...,,-;:',.;=¨=:-.,-. - -==:,!!:=;.',.,=: ...I
.. ...
C

7 y -,x ....
Figure 41. Corepointm bar with rings, and sensor-only (at left) as compared with actuator+sensor (right). Here the S.
Mann modified an automobile alternator to function as both a sensor and an actuator. This provides simultaneous sening and haptic actuation, so a person playing the game can feel virtual bumps in a road, or the like.
the men's locker rooms of Calgary's Talisman Centre school lunches around the world. She also raised con-where people are naked [21] as long as only the police siderable money for charity, as a result of her documen-(or other "good people") can see the video. VVestside tary photography. But, in part due to the notoriety of receration Centre also in Calgary, Alberta, also uses her photo essays, she was suddenly banned from bring-surveillance cameras in their men's (but not women's) ing a camera to school, and barred from photographing locker rooms {?] (Fig 42). the lunches she was served by her school.
While surveillance (oversight) is increasing at an While schools begin the process of installing surveil-alarming rate, we're also swing a prohibition on lance cameras, students are increasingly being forbid-sousveillance (undersight).
den from having their own cameras. And for many Martha Payne, a 9-year old student at a school in people on special diets, or with special health needs, Scotland, was served disgusting school lunches that using photography to monitor their dietary intake is lacked nutritional value. So in 2012 she began pho-a medical necessity. I proposed the use of wear-tographing the food she was served [?]. When she be-able sensors (including wearable cameras) for auto-gan sharing these photographs with others, she gener-mated dietary intake measurement in 2002 (US Pat.
ated a tremendous amount of online discussion on the App. 20020198685 [60]). This concept is now gain-importance of good school nutrition. And she brought ing widespread use for self-sensing and health monitor-about massive improvements in the nutritional value of ing [24, 23].

Why No Cell Phones in _ Changing Areas?
Wain you be allay if a nude photo et you wags 0 , ir . iii =,,,i , .
1, _ ponied on the Internet or dicinaidel rya *an ' :1-- - =
How about a undo *ale of yew *mat 44kt 4 . ' Or, maybe yenir chile?
Inn >ma way be tow*, video 1 '4* surininance to ensure the safety& loZt : * users by deems acts cidieft.tiolence .
...e , ;r?
tand other ail** inning , , .11 =
'T .
, =.: ..= .., Can *oars an capstan of capturing mars ' 4' 1:: . ......
=:/ink . . . .
Vdenisidn's 'on mil pbvin" poicy 1 n protect tie primacy &our manibors and *mg& - -. -% "
..
Including gown ' ;.: . ' 14.*' , , . , ., Zi..-.H.*.' ' , *
jidigkin. .7-r Figure 42. Westside Receration Centre in Alberta, Canada, . .
uses surveillance cameras in their men's (but not women's) ..
shower/locker/changerooms [?], while they also have a no cell phone" policy allegedly to "protect the,-privacy" of their -,. , -guests. li Figure 43. Harbisson and Mann with passports depict-So when people suffer from acute effects like food ing the physical reality of their bodies as partly computa-poisoning or allergic reactions, or from longer-term tional, both examples of people who are part technological, chronic effects of poor nutrition, like obesity, being for- through the use of camera-based computer vision as a see bidden from keeping an accurate diary of what they ing aid. The Veillance Divide (e.g. when surveillance is the have eaten is not just an affront to their free speach. only allowable veillance) renders such people under attack It is also a direct attack on their health and safety.
as "existential contraband" -- contraband by their mere ex-Neil Harbisson, a colorblind artist and musician, has istence.
a wearable computer vision system that allows him to hear colors as musical tones. And he wears his cam- to continue using photography to monitor her dietary era and computer constantly. Wearable computing and intake, not only for the journalistic freedom (in the Personal Imaging (wearable cameras) are established Bakir sense), but also for the personal safety that such fields of research [Mann 1997,] dating back to my aug- self-monitoring systems can provide.
mented reality vision systems of the 1970s. I also wear 10.2. Surveillance is a half-truth, without sousveil-a computer vision system and visual memory aid. Har- lance bisson and Mann both have cameras attached in such a way as to be regarded as part of their bodies, and thus Surveillance is the veillance of hypocrisy, in their passports both include the apparatus, as it is a the sense that, as often practiced by security guards, part of their true selves and likenesses. And we are not closely monitoring surveillance cameras, these guards alone: many people now are beginning to use technol- tend to observe and object to individuals taking pic-ogy to assist them in their daily lives, and in soine ways, tures in the surveilled spaces. The opposite of the transition from a surveillance society to a veillance hypocrisy is integrity. [?]
society (i.e. one that includes both surveillance and Surveillance typically tells a story from the side sousveillance) is inevitable [4].
of the security forces. When stories told from other Referring back to Martha Payne, discussed earlier points-of-view are prohibited from capturing evidence in this chapter, there was a hypocrisy of the school of- to support these other points-of-view, what we have is ficials wanting to collect more and more data about us, something less than the full truth [?]. In this sense, while forbidding us from collecting data about them or surveillance often gives rise to a lialf-truth.
about ourselves (like monitoring our own dietary in-103. Justeveillance (fair sight) in AI and Machine take, monitoring our exercise, or helping us see and Learning remember what we see). We need to be critical of this hypocrisy because (1) it is a direct threat to our Much has been written about equiveillance, i.e. the health, wellness, and personal safety, and (2) data ob- right to record while being recorded [81, 82, 45, 54], tained in this manner lacks integrity (integrity is the and Martha's case is like so many others.
opposite of hypocrisy). Thus it is with great relief that In the context of human-human interaction, the recently Martha Payne fought back and won the right transition from surveillance to veillance represents a "fair" (French "Juste") sight and, more generally, fair All call this app "LUNATIC". LUNATIC is a virtual and balanced sensing. personal assistant. The user places a request to LU-But our society is embracing a new kind of entity, NATIC and it then "tries the same thing over and over brought on by AI (Artificial Intelligence) and machine again..." on behalf of the user so that the user does not learning. Whether we consider an "AI" as a social en- need to himself or herself become insane. For exam-tity, e.g. through Actor Network Theory 169, 41, 9, 841, ple, when downloading files, LUNATIC starts multi-or simply as a device to interact with, there arises the ple downloads of the same file, repeatedly, and notifies question "Are smart things making us stupid?"[67]. the user when the result is obtained. LUNATIC de-Past technologies were transparent, e.g. electronic termines the optimum number of simultaneous down-valves ("vacuum tubes") were typically housed in loads. Typically this number works out to 2 or 3. A
transparent glass envelopes, into which we could look single download often stalls, and the second one of-to see all of their internals revealed. And early devices ten completes before the first. If too many downloads included schematic diagrams - an effort by the manu- of the same file are initiated, the system slows down.
facturer to help people undertand how things worked. So LUNATIC uses machine learning to detect slowed In the present day of computer chips and closed- connections and makes a best guess as to the optimum source software, manufacturers take extra effort not to number of times to repeat the same tasks over and over help people undertand how things work, but to con- again. This number is called the "optimum insanity", ceal functionality: (1) for secrecy; and (2) because they and is the level of insanity (number of repetitions) that (sometimes incorrectly) assume that their users do not leads to the most likely successful outcome.
want to be bothered by detail, i.e. that their users are At times the optimum insanity increases without looking for an abstraction and actually want "bother- bound, typically when websites or servers are unreli-some" details hidden [52, ?]. able or erratic. LUNATIC is not performing a denial At the same time these technologies are being more of service attack, but, rather, a "demand for service". A
concealing and secretive, they are also being equipped side effect is that when large numbers of people use LU-with sensory capacity, so that (in the ANT sense) NATIC, erratic websites will experience massive down-these devices are evolving toward knowing more about load traffic, such that LUNATIC disincentivises insan-us while revealing less about themselves (i.e. toward ity.
surveillance).
In this sense, LUNATIC is a temporary solution Our inability to understand our technological world, to technopagan insanity, and ultimately will hopefully in part through secrecy actions taken by manufactur- become unnecessary, as we transition to the age of ers, and in part through a general apathy, leads to the Sousveillant Systems.
use of modern devices through magic, witchcraft-like rituals rather than science [43]. This technopagan- 10.4. Sousveillant Systems ism [77] leads people to strange rituals rather than Humanistic Intelligence is defined as intelligence trying to understand how things work. General wis- that arises by having the human being in the feedback dom from our experts tell us to "reboot" and try again, loop of a computational process [65].
rather than understand what went wrong when some- Sousveillant Systems are systems that are designed thing failed [75]. But this very act of doing the same to facilitate Humanistic Intelligence by making their thing (e.g. rebooting) over and over again, expecting state variables observable. In this way xnachines are a different result is the very definition of insanity: "watching" us while allowing us to also "watch" them.
"Insanity is doing the same thing, over and See Fig 12 (a detailed description is provided in [51]).
over again, but expecting different results." - 10.5. Undigital Cyborg Craft Narcotics Anonymous, 1981.
Sousveillant Systems give rise to a new form of In this sense, not only do modern technologies drive Human-Computer interaction in which a machine can us insane, they actually require us to be insane in order function as a true extension of the human mind and to function properly in the technopagan world that is body. Manfred Clynes coined the term "cyborg" to de-being forced upon us by manufacturers who conceal its note such an interaction [18], his favorite example being workings. a person riding a bicycle [17]. A
bicycle is a machine I propose4 as a solution, a prosthetic apparatus that that responds to us, and we respond to it in an Mune-embodies the insanity for us, so that we don't have to. diate way. Unlike modern computers where feedback 4This began as an interventionist/awareness-raising effort, is delayed, a bicycle provides immediate feedback of its but could evolve into a useful field of research. state variables all which can be sensed by the user.

Consider the CNC (Computer Numerical Control) milling machine that extends our capacity to make things. We begin with CAD (Computer Aided Design) and draw something, then send it to the machine, and let the machine work at it. There's not much real feed-, back happening here between the human and machine.
The feedback is delayed by several minutes or several hours.
David Pye defines "craft" ("workmanship") as that which constantly puts the work at risk [71]. As such, modern CNC machine work is not craft in the Pye sense. Nor is anything designed on a computer in which there is an "undo" or "edit revision history" function-ality.
Imagine a CNC machine that gave the kind of feed-back a bicycle does. Could we take an intimate expe-rience in craft, like a potter's wheel, and make a CNC
=
machine that works at that kind of continous ("undig-ital") feedback timescale?
HARCAD (Haptic Augmented Reality Computer Aided Design) is a system to explore "Cyborg Craft", :111 a new kind of undigital craft: being undigital with dig-ital computers. One such embodiment is the T-SWIM
(Fig 47) as a haptic augmented reality interaction de-vice and system. In this system, one or more users can grasp and touch and feel virtual objects in cyborgspa.ce, and share the resulting haptic augmented reality expe-rience, as a way of designing and making things like cars, buildings, furniture, etc., through a collaborative' design and realization process.
. = =.
Let's consider a simple example. We wish to make a table with a nice curvy shape to it. A good place to begin is with waves. As a basis for synthesis, we have a shape synthesizer. In some embodiments the shape Figure 44. Furniture design with Haptic Augmented synthesizer is made in software, from simple shapes like Reality Computer Aided Design. A shape generator or rectangles, circles, ellipses, etc., as one might find in a shape synthesizer, such as a PASCO SCIENTIFIC MODEL
computer program like Interviews idraw, Inkscape, or WA-9307 FOURIER SYNTHESIZER, generates shape in-Autodesk Fusion 360. In other embodiments the shape formation which is visualized in alignment with haptic sen-synthesizer exists as a piece of hardware, as illusrated in sation, so that shapes can be seen and felt in perfect align-Fig 44. Here is a multiple exposure, showing three ex- ment. A 3D (3-dimensional) wireless position sensor com-posures while moving back-and-forth, which align, ap- prises transducers with phase-coherent detection. The syn-proximately, with each other, while admitting a small thesizer is connected to a fixed transducer (loudspeaker), variation. The shape is sculpted by press-pull opera- and the reference input of a NARLIA (Natural Augmented Hons. Additionally, harmonic variations are made to Reality Lock-In Amplifier). A moving transducer (micro-phone) on a Haptic Augmented Reality CAD (HARCAD) the shapes, to design furniture. Here a wavetable is wand is connected to the signal input of the NARLIA. An designed in which the table is the fundamental of the output from the NARLIA is connected to a T-SWIM, here, waveform from a musical note, and the shelf above it 600 LEDs and a vibrotactile actuator, both part of the is the fundamental and first overtone (i.e. the first two IIARCAD wand. As the user moves the wand back-and-harmonics). Here the amplitude of the waveforms de- forth, a multisensory experience results in which the user creases from left-to-right as we move further from the can see, feel, and hear the virtual shapes (e.g. waves and sound source. their harmonics). Arbitrary shapes can be generated by Another embodiment comprises a robot that a user Fourier synthesis, and these shapes can be touched, felt, can wrestle with. One embodiment of the wrestling and manipulated.

robot involves a core-strenght development system, ,....:.õ.-- !.gii õ... J.. õ, ..
based on integral kinesiology. The user exercises their "" j,. . _ .
core by playing a video game or a real game or an in- = ¨
_.
teraction in which there is an error signal that must be = : ....."-.._ = ---:::_.
overcome. One version of this uses an interactive spray .
jet that takes evasive action while a user tries to dead- , head the jet from some distance. Deadheading a water jet is a good form of fitness exercise that is inherently absement based, or stability based, because it requires covering the jet from some distance, and then com-ing close to the base of the jet to block it completely.
=== = = . 6, PL. -- -. 5,,, : , =
The jet is preferably too strong to be blocked at the Ifbase, except by gradual approach. Coming straight to- . =, .
ward the jet, a user can carefully cover it, but a little wandering off-course causes the user's fingers or hand to be thrown wildly off course. Thus the exercise re- .
-quires combined simultaneous dexterity and strength. =
, To make this more challenging and more fun, a game . _ is made of it. The robot or a mobile phone app, or .
the process in general, has a moving cursor or pointer, and the user's objective is to exert a combination of strength and dexterity to track or follow the moving .6 , ..,, r pointer. An example with a water jet on a robot is .,,,i- (-shown in Fig. 45.
' VII =
Moreover, by attaching grip handles to the robot,. _ ___ .
.7-N,46..,,; - _ li a new form of interaction arises. In one embodiment, the grip handles are made from laser cut wood (See .
Fig 46). See 48 for the finished wrestling robot.
This gives rise to:
= fir,.;,, 41*-.
........,,,&.%
= WrescillatoryTmUser Interfaces;
i, :=., . =,,;V,..-=i:..,.--,, ',. '.t "S: ';' - ' -4, = The WrescillographTm;
*
-4 .=
.
= The Wrescilloscopelm...; and ;,:ii.. A. iii- - . ¨
= MAPatliTm(e.g. like a GPS says "recalculating"/
but instead of a map this os for a toolpatli. Tools. ,.
, , , like CNC have grab handles so you can wrestle .
with them and then have them go ... "recalculat-ing" the tool path.
/
Here we have a synergy between human and machine in which the feedback loop is essentially instantaneous.
This brings us full circle back to the topic of ..
Sousveillant Systems. If we're going to be able to do m _ things like wrestle with machines, we need machines = , to be more vulnerable, more expressive, and in many ways, more human. To make machines more expres-sive, let us equip them with premonitional user inter-.:.
., faces, an example of which is the concept of a Photool-pathTm, i.e. a photographic representation of a tool- Figure 45. "Wrobot", the wrestling robot. A nozzle, de-path, as shown in Fig. ??. signed in Fusion 360, is printed on the Autodesk Ember A set of special Photoolbits are provided for use with printer, and then attached to a robot. The robot then takes a CNC machine like for example, the DIVIS (Diversified evasive action as a user tries to deadhead the water jet.

'L4 Sok 1 1 ti\ill,1 -i :.=
, . ...
Air -,... it.
0 . ,?
lit' .. ..-4' I, , 1, '' 4,11,1,k; '''."`= ' - , -.11/4, /
, - = r , . . .1.-,,. ,A.

-,-1. -=-=,:,=,..,õ Illt .
,.. * .
....., '= -,. -0, . = .. k/A' = * '' ) ... , , '' -.f. 'µ,-,, , = .', 4 ' '' ii.--.-- : ''"-- ' z'''-','"' '-µ--- ...r.'". -:.T.'==c:''' :7 ':',=% ., µ==== , = '... ===,, -= ' - , 4111k/- ".' i . -,''' =:, : Or *
µ. ' ..--,54='=,.1 , - 1- ..
,,,.t;.-:, , 4 'A = = i ..,,,,,,L = , t 0..
-AD.
=
,f '''' ' , 4, = * , , , _ = . ' ' '1 .. ' , i! . = - . ' 1 ,,_'-' = ' ' _ ".- .-2-..-, ' .... ,-; ; / ;
7' ,-õ,. .-=`.;.. = '.,..,.. ir ''--:. . ?`-' .... '' .si ' =. , .,,,, . ".... .

=,.."-!-,, '',-,=' . ': -= s _õ,,, . ` f' ''''i '''. , 4.'"' \i, = ''. =
f-1 -;iti :_ iii* ', ' :',"õ'",t = : , v , :.
4õ,77,17 Y -1. µI'iltb. .',, -;;:.:4=1 r µ=
,.
-,! --, . rp ia 2:1 4'44 ' ' ':::t=Z ;:i.it':'' : -' . --- , ' .a.lir Li ' : ' *e ..1.,:;ii.i.',:, ,. : 1,,;...-' .
07 flj):- = 'sitJ ' -,-.. 0 f strõ, k k,........., Figure 18. µVrestlirig with robots as a means for achieving Cyborg Craft:
Robotic-inspired abakographic liglitpainting (top row). Stephanie. age 10, wrestles with the robot while controlling the spinning of a Potterycraft wheel with her left foot.

=
=
76rer''7- -4r o-õ.
=
Figure 39. Scott creating an artwork at Pier 9. pictured with Mann's photographic toolpath visualizer. Toolpath previsu-aliztion using Augmented Reality. The Photoolbit 'is inserted, or alterntively, the Photoolpath technology is built into the device. An exposure is niade as the toolpath is run at full speed, allowing the user to see the toolpath overlaid on reality.
Slight changes in perspective are facilitated through simple 31) modeling, while at the same time retaining the photo-realistic rendering. At any time during the work, the cutting may he paused and the portion cut is compared with the Photoolpath.
After the cut is complete it can again be checked against the AR overlay.
Figure 50. fig:scott6 Machine Systems) 5-Axis Enclosed Router. It has an kited phase coherently with a phase coherent detector existing tool library in which 12 items can be loaded. array so it can ignore other light sources other than the In an embodiment of the invention, some of the 12 tool Photoolbit.
items loaded are Photoolbits. One or more Photoolbits Alternatively a photogTaphic toolbit (Photoolbit) are included in the library. In 011C embodiment there subsitute is intergrated right into the print head. In is simply an LED light source in a housing with the this way the overlay can happen during printing.
same dimensions as typical tools used in the DMS tool In another embodiment. a laser cutter is fitted with library. Part of the tooling sequence can include selec-a photographic print head that prints on photographic tion of a Photoolbit to trace ont with during a long ex-material at the same time as it (cuts or etches) a work-posure photograph which can then be re-rendered from piece. In this way we can continuously view the tool-any viewpoint and fed to DEG (Digital Eye Glass) such path as it. is evolving. A photodetector is aimed at the as the EyeTap AR (Augmented Reality) system.
laser so as to pick up its light and amplify its lig,ht and Photoolbits include small batteries that charge wire-then pass that along to one or more LEDs borne by lessly when in proximity to their tool crib storage loca- the head.
Preferably a simple attachment is provided.
tion, and turn 011 automatically when grasped by the Preferably the simple attachment takes a form similar toolbit holder.
to the focus piece of an Epilog Legend EXT36, so it In another embodiment. the light source is modu- can simply be inserted and stuck on with the magnets "
===
41.4 r Ark 6A=...' i __ Iry Er .
Figure 51. Example of Sousveillant Systems with Epilog laser cutter. A special AR head attachment is clipped onto the head of the machine. Here we see an example AR visualization while making a piece entitled "Crowdftmded Justice" (2016 October 31st art installation about justice, based on coin-operated gallows spool for the back of a bubblegum machine).
The shape of the gallows rope spool is clearly visible as an AR overlay, and we can previsualize it, postvisualize it, or see it during evolution of the toolpath.
the Epilog and the sousveillant print head attachment.
This allows sousveillance to be added to a laser cuttor , Legerr... or other computer controlled tool, without requiring collaboration with the manufacturer. See Fig. 51 01, 10.6. Feedback delayed is feedback denied Let us conclude with a hypothetical anecdote set in the future of musical instruments, which parallels . - the author's personal experience with scientific instru-ments. This points to a possible dystopia not so much . .
of government surveillance, but of machine inequiveil-. lance.
, . =
The year is 2067. Ken is the world's foremost con-cert pianist, bringing his own Steinway grand piano to . . , = . , each concert he performs in, now that concert halls no longer have real pianos. Software synthesis has ad-Figure 46. Laser cutting of grip handles. Each cut is 1/4 vanced to the point where none of the audience mem-inch thick plywood, and there are a total of 3 layers for each hers can hear the difference. Steinway stopped making handle. The layers are glued together and then attached to real pianos i th b n 2020. Yamaha a,nd others also switched e ro ot.
to digital-only production the following year.
Even Ken has trouble telling the difference, when someone else is playing, but when he plays a real piano himself, he can feel the difference in the vibrations. In already present. The insert has a small lightweight essence, the performance of the Steinway Digital is as battery and just snaps in place, being of lightweight good as the original.
but the vibrotactile user-interface construction to minimize payload. A small ferrous is delayed. The tactuators installed in each key shim-piece is for being held by the Epilog's magnets, and late the player's feeling of a real piano, but there is a the rest of it is made of aluminium or lightweight plas- slight but noticeable delay that only the musician can tic, and houses a batter holder, photodiode, amplifier, feel. And user-interface is everything to a great musical and LED. A small microprocessor or microcontroller performance.
is used in some embodiments and it distinguishes be-Ken no longer has access to a real piano now that tween etching and cutting modes of the laser cutter. It his Steinway grand piano was water-damaged by a roof thus drives one or more LEDs or at least one multicolor leak while he was away last March. He tried to buy LED with a unique color for etching, different than the a new piano but could not find one. Tucker Music color for cutting. A third color indicates traverses or had one in their catalog, for $6,000,000, but when Ken movement of the head in which no cutting or etch- called Jim Tucker, Jim said there were no more left.
ing takes place. This is all done with no requirement Jim sold about 50 of them at that price, over the past that there by any communication or sensing between few years, as he collects and restores the world's last #
10.7. The need for antiques to obtain truth in science The above story depicts a true event, except for a few small changes. My (not Ken's) instrument that was damaged was not a musical instrument, 1.1 but, rather, a scientific instrument called a Lock-In Amplifier[62, 64, 78, 19] made by Princeton Applied Research in the early 1960s. It was easy to under-' stand and modify. I actually did some modifications and parts-swapping among several amplifiers to get W1ANWIl 1\ _ .
f I-441 I H
some special capabilities for AR (augmented reality) L; veillance visualizations, such as bug-sweeping and be-ing able to see sound waves, radio waves, and sense Figure 47. Haptic,s, Augmented Reality, and Com- sensors and their capacity to sense [50].
puter Aided Design. A Tactile Sequential Wave Imprint-The roof leak occurred in March 2016, while the ing Machine [Mann 1974, 1979, 1991, 20151 is connected to amplifier was running (it takes a few hours to warm an antenna moved in front of a radio transmitter. The ac-up, and since it uses very little electricity it is best to tuator is fed by a signal from a lock-in amplifier comiected leave it running continuously).
to the moving antenna plus another stationary antenna.
The PAR124A is no longer available, and large orga-In this way the user can grasp, touch, hold, and feel oth-nizations like research universities and government labs erwise invisible electromagnetic radio waves. In an early are hanging on to the few that remain in operation.
embodiment (Mann, 1979), radio waves are picked up (or It should be a simple matter of purchasing a new reflected) by the moving metal bar of a pen plotter, and the user grasps the pen to feel the radio waves. A light amplifier, but none of the manufacturers are willing to bulb is also attached to the pen so that the user can also make the modifications I require, nor are they willing see the radio waves, through PoE (Persistence-of-Exposure) to disclose their principles of operation to allow me to of the human eye, or photographic film. In more modern do so. Neither Princeton Applied Research, nor Stan-versions of the apparatus, a linear actuator drives an LED
ford Research Systems (nor any other modern maker light attached to the finger. By wrestling with this robotic of lock-in amplifiers) is able to supply me with an in-device, the user and the device together trace out the ra- strument I can understand.
dio wave into the Autodesk Fusion 360 Cloud-Based 3D
One company claims to have equalled the perfor-CAD Platform for instant collaboration with others (e.g.
mance of the PAR124A, at a selling price of $17,000:
multiple people at different geographical locations remotely wrestling with earh other and with nature in order to col- Since the invention of the lock-in amplifier, laboratively create new artistic designs). Together with the none has been more revered and trusted than Metavision Augmented Reality glasses. multiple people use the PAR124A by Princeton Applied Research.
aluminium foil brushes to collaboratively sculpt and shape With over 50 years of experience, Signal Re-cloud-based waveforms to design buildings, furniture, auto-mobiles,covery (formerly Princeton Applied Research) or other curvaceous products.
introduces the 7124. The only lock-in ampli-fier that has an all analog front end separated, via fiber, from the DSP main unit.
remaining real pianos, but no more are coming up for Recent research findings, however, show that the sale.
PAR124A from the early 1960s still outperforms any scientific instrument currently manufactured [80].
Ken has felt that his musical performances have de-And performance alone is not the only criterion.
de-clined now that he no longer has access to a real piano.
With a scientific instrument we must know the truth Software, AI, and machine learning make better music about the world around us. The instrument must func-anyway, so there's no longer a need for human musi-tion as a true extension of our mind and body, and hide cians, anyway.
nothing from us. Modern instruments conceal certain But there has been no great advancement in mu-aspects of their functionality, thus requiring a certain sic in recent years, now that there are no longer any degree of technopaganism [77] to operate.
great musicians still passionate about music for music's Thus we currently live in a world where we can't do sake. Today's musician spends most of the time writing good science without access to antiques.
grant proposals and configuring software license servers Imagine a world in which there are no Steinway rather than playing music.
grand painos anymore, a world bereft of quality, ex-cpet old ones in need of restoration. A musician would simply objects already present in the environment. For have to be or hire a restorer and repair technician, and exa,mple, radio waves or sound waves (or computer vi-hope for access to one of the few working specimens sion) emitted from a transducer of position sensor 122 that remain.
bounce off things in the environment, are reflected, and Is this the world we want to live in?
then received by sensor 122, thus enabling the deter-Science demands integrity.
mination of the 3D position of wand 120. A location Only through Sousveillant Sy-steins and "little data"
sensing field 150 comprises a sound field, electromag-can we preserve the tradition of science, in the face of netic field, lightfield, or the like. For example, if sensor technopaganism, surveillance, and "Big-only Data". A
122 is a 40,000cps (cycles per second) sonar, the sens-goal of our research is to produce devices that embody ing field is the sound waves bouncing around in a pool sousveillance-by-design, starting with scientific test in-or room or other space. Location sensing is relative struments like lock-in amplifiers, and progressing to-to other fixed or moving objects such as desks, floor, ward concepts like "little data". To that end, let us ceiling, the walls or bottom of a pool, the bottom of hope that we can build sousveillance into our networked the sea or a lake, fish swimming in the sea or lake, or world, starting with instruments. the like.
The wand optionally includes one or more visual el-11. Detailed description of the drawings ements, real or virtual, such as LEDs 121 or virtual LEDs synthesized in an eyeglass display as if emenat-Fig. 1 illustrates an embodiment of a sousveillant ing from the wand 120. The LEDs (real or virtual) system in the form of a haptic/tactile augmented real-are sequenced by a SWIM (Sequential Wave Imprint-manufacture computer-aided design (CAD) and computer-aided ing Machine) computer 115, by an output signal, X, manufacture (CAM) system, i.e. a system that allows a from the multisensory processor.
person to design and build something, by using a hap-tic wand 120 to sculpt and shape something. A shape Signal X may be real or complex, or multidimen-generator 110 generates initial shape information. This sional in nature. Computer 115 drives LEDs 121 to initial shape information comes from another user or make visible a CAD (Computer Aided Design) shape collaborator, or it is pulled from a shape library or it 130. Wand 120 includes a bidirectional tactor 123 which both senses and affects tactility. Tactor 123 cre-is drawn by other means, such as by traditional CAD, ates a vibrotactile sensation in wand 120 so that when for being entered as data, into the shape generator 110.
grasped by the hand of a user, the user can feel shape Alternatively, input comes from nature, such as from a 130.
phenomenological process, or from some visual or other element found in nature itself, and forms a basis upon Moreover, tactor 123 is also a sensor, e.g. a trans-which shape generator 110 derives its input.
ducer that can operate both ways (transmit and re-Shape generator 110 exists as software within a pro-ceive). During a refractory period (i.e. during which it cessing system, or as hardware, either separate from, is unresponsive to stimulus) it emits tactile signals. In or as part of a general processing system. A multi-between emissions it senses how it is being squeezed. A
sensory processor 111 receives input from the shape pressure sensor function in processor 111 records this generator 110 by way of a shape information reference squeeze force, and modifies a pertion of shape 130 at signal 112. Reference signal 112 fiinctions as an initial which wand 120 is located. So a user reshapes shape starting point for a user to work with.
130 by waving the wand 120 back and forth while A user of the system grasps a device such as wand squeezing the wand, to alter shape 130. A shape ta-120 and moves it through 3D (3-dimensional) space.
ble in processor 111 is indexed by a position of wand The wand 120 includes a position sensor 122. A sat-120, while the value at the index corresponding to the isfactory position sensor 122 is a transducer connected position of wand 120 is adjusted in proportion to the through a signal amplifier 116, producing spatial sen- squeeze force on tactor 123.
sory signal 113 to a lock-in amplifier also fed by a sta-The updated shape table in processor 111 is con-tionary transducer as reference. A location sensor 140 tinuously played out on the LEDs and output (during is either disposed in the environment around the user refractory period) phase of tactor 123, so the user can (e.g. a stationary transducer) or is computed from touch and feel and see the updated shape 130, in a the environment. In some embodiments sensor 140 continuously fluid manner.
is merely virtual, comprising the natural unprepared In some embodiments tactor 123 is a pressure sensor environment, as first encountered by a user. Thus no and hydrophone and wand 120 is waved back and forth advance preparation is required, and the user can enter underwater while a small jet of water conveys shape 130 any space. In this case location sensor 140 comprises information to a hand of a user, and also senses, hy-draulophonically, force given by the hand of the user.
sional complex-valued manifolds in the spacetime con-In air, this can also work, with simply a change in fluid tinuum of quantum gravity waves, as users U201 and for water to air. thus providing a fluid user interface U202 collaborate on preparing online course materials.
for use in air, where the tactor 123 is a fluid (air or wa-Shared objects S200 are not limited to CAD/CAM
ter) sensor and transmit transducer in one, or contains but also include the making of virtual content that is separate send and receive transducers.
never physically realized, such as course material for Fig. 2 illustrates an embodiment of a collabora-teaching quantum field theory or other abstract con-tive cloud-based Haptic Augmented Reality Computer- cepts.
Aided Design system based on the embodiment of User U201 grasps wand 120 and moves it along ob-Fig. 1.
ject S200 to select a neighbourhood of points around Here wand 120 is held by user U201 running a cloud-point P221, and feels that part of S200, while seeing based CAD software instance 201. A wide variety of all of S200.
cloud-based CAD software packages will work with this User U202 grasps wand 220 and moves it along ob-system. A good choice of cloud-based CAD software is ject S200 to select a neighbourhood of points around Fusion 360 by Autodesk. Another user U202, or the point P222, and feels that part of S200, while seeing same user at a different point-in-time (e.g. with U202 all of S200.
being a user-present, and U201 being the same user at Within instance 200, P201 and P202 are updated a past point in time) is running the same cloud-based so that both users U201 and U202 see and touch and CAD sofware, such as, but not limited to, Autodesk feel and grasp the same shared object S200 and observe Fusion 360, e.g. different users (or the same users at (see and feel) each others' edits.
different points-in-time) can use different software as In this simple example, the shared object is an es-long as all the software follows the same HARCAD
sentially one-dimensional manifold in multdimensional (Haptic Augmented Reality Computer Aided Design) (3D or 4D or higher dimensional) space, so that it ex-protocol.
ists simply as a list of numbers, preferably as a list User U202 is running instance 202 of CAD software of double precision floating point numbers as thus an such as Autodesk Fusion 360. In situations where user "undigital" (essentially continuous) representation of U202 is the same user at a different (future) point- the object S200.
in-time, user U202 is running the satne or a different User U201 is trying to pull up on a portion of the instance 202 as instance 201.
beet a bit toward the right of center, whereas user Within the instances 201 and 202 of CAD software, U202 is trying to pull down on a portion of the object a shared object S200 exists within the cloud software near the left side of the object S200.
instance 200, and resides on cloud storage 290 by way The object representation in this case comprises a of network 280. Storage 290 may be a disk, solid state table of numbers, with enough sampling to losslessly drive, magnetic core, punched paper tape, or any past, describe it. There are several cycles (about seven cy-present, or future, e.g. newly yet-to-be discovered form cies) of waveform of which the first several harmonics of computer ntemory storage device.
are relevant, so by Nyquist's theorem, we need maybe Users U201 and U202 interact by manipulating var-7*7=49 cycles per second, times two is about 98 sam-ions shared objects in a virtual or augmented reality ples, i.e. about 100 or so samples would suffice for the space, such as shared object S200.
specific object shown in Fig. 2, but to allow for further Shared object, S200 exists as a 3D spatial object in edits, let's say there's 1000 samples in object S200, i.e.
3D space, but objects in higher dimensional spaces are tht S200 is represented by 1000 floating point numbers, also edited and manipulated, since virtual objects are thus having 1000 undigital degrees of freedom.
not limited by dimensions. Some of the shared objects More generally, more complicated objects like an au-are four dimensional spimes (spacetimes), visible in the tomobile or jet engine, will include even more degrees axes of x,y,z,t (time), whereas others are of higher di-of freedom or data points. For example, the curve of mensions. Some objects are real whereas others are an automobile's body is represented by a Fourier ex-complex, e.g. the users share and manipulate corn-pansion of the surface in 3D space, and is manipulated plex objects like functions in complex space, such as by grasping the surface, using the "press pull" function waveforms made of a spatiotemporal complex-valued of Fusion 360.
Fourier series.
Therefore we require a simple way to manipulate the Thus shared objects S200 range from simple things shared objects in multidimensional spaces. In our sim-like circles, rectangles, cubes, spheres, cylinders, cones, pie example, user U201 grabs point P221 by hovering and the like, to more complex things like multidimen-wand 120 over point P221 and squeezing the tactor of the wand, so as to select this point. A software interface also large objects and even have deliberately added reads this squeeze and requests a block of object S200 mass. A nice weighted wand feels good in the hand, from the cloud server instance 200 and updates the nu-and especially in pursuit of physical fitness, the wands merical values, in this case by increasing the numbers may be weighted and used also in a competitive kind in the neighbourhood of P221 region, i.e. as indicated of sport.
on scale S1000, with finer scale divisions just before in-This sets forth a way to revolutionize the workplace, dex i200 and i600. Index markers i0, i200, i400, i600, and rather than being fat and lazy sitting on a chair and i1000 are shown on scale S1000, and indicate 1001 all day, we can be fit while desinging things.
indices running from 0 to 1000.
Thus there is a new method of fitness, using the Under this programming, point P201 falls at sample MannFitTmSystem in which players or competitors, or 587 of the sample array indicated by scale S1000, thus workers, collaboratively design objects like furniture, the 587th sampe of object S200 is increased, as well as or other objects for sale, by playing fitness games that the sample points in its neighbourhood.
develop a combination of strength and dexterity. I call This is done by adjusting an object representation, this "dexstrength", i.e. the capacity to act with pre-such as a spline or a series representation such as a cision+accuracy while at the same time exerting one's Fourier series representation, resulting in a general up- self.
ward shift of the portion of object S200 in the high five Abakography (computational lightpainting) itself hundreds area of indices, e.g. samples at indices 580 can be made into a game or sport, as outlined in "In-all the way up to 600 also increase substantially.
telligent image Processing", author Steve Mann, pub-Likewise user U202 is tugging down on the left side fisher John Wiley and Sons, year 2001.
of object S200 in a similar fashion, around sample 117 Fig. 3 illustrates an embodiment of a collaborative of the list of munbers that represent object S200. Our cloud-based fitness game designed to create strength algorithm thus needs to decrease the 117th number in and stability in a user's core muscles. Such a game the list of numbers that represent object S200. For is aimed at providing strength, dexterity, endurance, smoothness and continuity, we filter this movement stability, and control over the following nmscle groups similarly, e.g. by modifying the represetnation of the generally known as the "core":
data, e.g. by taking a Fourier transform of the desired = rectus abdominis;
change, selecting only the first several principal com-ponents, (typically the lowest frequency coefficients), = transverse abdominals zeroing out the higher (less principal) coefficients, and then performing the inverse Fourier tranform to get the = oblique musicles (internal and external new modified object S200. "obliques");
In this way user U202 can tug down on the left side = pelvic floor muscles;
of object S200 while user U201 tugs up on another part of the object just right of center, and the two changes = multifidus;
are made and experienced by both users who can each see the whole object and feel the part of the object they = lumbar spine stabilizers;
are exploring.
= sacrospinalis;
When the two users run on top of each other (e.g.
if both try to edit the same part of the object) they = diaphragm;
are essentially "wrestling" with each other, and the ac-tion and general feel of the apparatus is much like the = latissimus dorsi;
Wrobot (wrestling robot) in which they feel each oth-ers' tugging and know that they are both trying to edit = gluteus maximus;
and the same piece of object S200. ln this way it feels a bit = trapezius.
like a ouija board, when multiple people try to move a shared planchette, but these multiple people here are Fitness rings 310 hang in pairs each at the end of in separate geographic locations.
a destabilizer bar 311 which hangs from its center by Wands 120 and 220 are ideally lightweight and easy cable 312 from ceiling 313. Cable 312 connects to a to carry around or put in a pocket, or wear as a neck-swivel 316 which forms a pivot point about which the lace, or even exist as virtual objects of zero mass or bar can rotate.
exist as small things like rings worn on a finger.
Each player hangs from a pair of rings hanging from But wands 120 and 220, in some embodiments are a destabilizer bar hanging by a single cord. The cord may be a rope, chain, cable, wire, or swivel bar, or / Coreboardlmso both players can see a time-varying pivot point affixed without a cord.
graph of their absement overlaid upon each other, and Each pivot point is connected to a rotary poten- this provides motivation to hold out for longer.
tionieter. The counterclockwise most side of the po-For simplicity on the drawing, only the circuit for tentiometer is connected to +15 volts from a power player 2 is shown, but in reality both players have such supply and the clockwise most side is connected to -circuits or software running, and, more generally, more 15 volts from the power supply. This causes a voltage than 2 players are involved in many embodiments of change proportional to the angle of the bar. When the the invention. Typically any number of players can bar is horizontal straight across, the voltage is zero. A
logon and join the game over the cloud-based server on standard 270 degree rotary potentiometer is used with network 280.
a +/- 15 volt (i.e. 30 volt center-tapped) DC (Direct Players achieve "Corepoints" by getting minimum Current) power supply. Thus there is about 1 volt in-absement. A Corepoint is defined as a thousand times crease or decrease for each 9 degrees of tilt. When the reciprocal of the absement, i.e. 1000 divided by the bar tilts to the left the voltage goes up, and vice the time-integral of the absolute value of the deviation versa. So if the bar tilts 9 degrees left of horizontal, from horizontal of the bar. Corepoints are displayed the voltage is +1 volts. When the bar tilts 9 degrees on the Coreboard, for both players to see.
to the right, the voltage is -1 volts. In Fig. 3 the bar Players next advance to level 1 of the game. In level is shown at about a +15 degree angle (positive angles 1 of the game the additional feat of raising the feet are counter clockwise), so the voltage present at the is provided, i.e. leg-raises, into L-seat position while wiper terminal of the potentiometer is about plus one also minimizing absement. Additional corepoints are and two thirds of a volt.
computed based on height of legs, and good form, as This system forms the basis for the Core- determined by a 3D vision system.
Pointntsystem in which the user's core muscles form Players then advance to level 2 of the game. Level 2 a pointing device. The pointing device is, in this ex-of the game involves generalized absement in which a ample, one-dimensional in the sense that there is just target function is defined and displayed as object S200 one degree of freedom which is the voltage at the center to both players. This object is a surface, typically on arm of the potentiometer.
one-dimensional manifold in two or three-dimensional The Corepoint system functions as a simple game space.
starting with level 0 of the gatne to warmup, and then The gaol of the game is to move a cursor along the advancing.
curve of the object S200 from left-to-right. The target In level 0, the objective is simply to hold the bar hor-curve of object S200 will have two instances, instance izontal. For each player, there is a separate output of 301 traced out by user U201 ("player 1"), and instance the central arm of the potentiometer which is a voltage.
302 traced out by user U202 ("player 2"). Instances For user U301, the voltage is vi and for user U302 the 301 and 302 shows level 2 of the game, where we see voltage is v2. For each player, this voltage goes through rough jagged traces corresponding to the actual angles an absolute value circuit or absolute value taker, absval as a function of time, and the smooth trace of instance 350, the output of whith is fed to integrator 351. A S200 as the desired angle as a function of time.
satisfactory integrator is formed by an op a,mp such as The curves resemble traces on a CRO (Cathode Ray a Fairchild 741 or Signetics 5532, with capacitor 353, Oscillograph), and are virtualized as such in the game.
in its feedback loop, and resistor 352, in series with Players both see an accurate rendition of a 1935 RCA
its negative input, the positive input being grounded.
Cathode Ray Oscillograph, Type TMV-122, displayed Note that signal 355 is the negative of the integral, so it on the screen. With its round screen, it is reminiscent requires further inversion, this being part of the abse-of a radar scope and provides a nice retro aesthetic for ment tracker 360's job. The integration is implemented the game. The characteristic green glow of object S200 in hardware or software or firmware. Preferably the has a cursor that sweeps from left-to-right as a function integration is performed computationally rather than of time. Instead of keeping the bar horizontal, players requiring a separate op amp, since there are other com-must tilt the bar left to increase the voltage on the putations to also be performed.
TMV-122 plot (i.e. move its cursor up), or right to The integral of the absolute value of the voltage drop the cursor down. For this action, the waveform from potentiometer 322 is referred to as the absement, of object S200 is added to the operational amplifier and exists as signal 355 which is fed to the absement of integrator 351, or, in the software embodiment, the tracker 360. The absement tracker displays continu-1000 or 1001 samples of object S200 are sequentially ously the absement on a shared cloud SCOREboard subtracted from a sampled voltage of v1 for player 1 and v2 for player 2. This is done as follows:
The TMV-122 Cathod Ray Oscillograph emulation is just one example of a game that can be played with = An initial input voltage, v(t) (either v1 or v2 de- this invention. It is a good game for the novice, because pending on which player) is received, and used to there is very little distraction, and the game is very update instance 301 for m and instance 302 for v, simple and easy to understand.
as traces evolving along with time (e.g. getting Level 3 of the game advances to a driving game.
traced from left to right. Thus initially all that Level 3 uses a video game, for steering a car, for will exist of instance 301 and 302 is the leftmost example. In level 3, the bar functions as the steering point. A difference is computed between the input mechanism for racing the car.
voltage and the first element of the list of 1000 or The next level, level 4, is the same driving game but 1001 numbers corresponding to object S200. This using the leg-raise function for control of the speed of difference is called eo and corresponds to the error the vehicle. To go faster, players raise their legs higher, signal between the desired and actual position of in an L-seat position.
the bar 311 or 321;
In addition to the angle sensor formed by poten-= At the next point in time, a new voltage sample is tiometers 321 and 322, another sensor senses leg level, taken, and this sample of voltage v(t) at this later and the object of the game is to do an L-seat position on time, for each player, is used to update instance the rings, raising the legs, as indicated in Fig. 41 (left), 301 and 302, and is used to compute a new dif- keeping the legs as high as possible. A suitable sensor ference voltage against the next sample of object is a 3D camera such as a Kinect camera, aimed at the S200, for each player, and this difference is called body, namely the legs, of each user U201 and U202.
el (each player has a growing list of error terms, Users may be together facing each other, in which case i.e. there's an co for each player and an el for each one vision system can monitor both users, or they can player, and so on);
be in different places, such as in different countries, connected by network 280, over a cloud server, where = Continuing, at ea.cah next point in time, a new each user is monitored by a separate 3D camera based voltage sample is taken, and this sample of voltage vision system.
v(t) at each later point in time, is used to compute In the video game, steering is by the angle of po-new difference voltages against each next sample tentiometer 321 and 322, which function as steering of object S200, and these difference are called e2, wheels for the players such as users U201 and U202 re-e3 e4, and so on, all the way up to emoo;
spectively, and the accelerator of the virtual car (one for each player) is controlled by the leg raise height.
= At each point in time, error voltages and their cor- The result is to develop solid core muscles by us-responding error angles (i.e. as calcualted at 9 de- ing fitness and fun, pointing with the core muscles, as grees per volt) are displayed on the Coreboard for core function and stability and control maps directly the two or more players, along with the running to performance in the game.
total error, thus far, as well as the Corepoints cal-Fig. 4 illustrates an embodiment of the invention us-culated as 1000 divided by the total error:
ing the tilt sensor 421 (gyroscopic sensor) built into a smartphone 471 rather than that the separate poten-= At each iteration, the winning player thus far, is tiometer 321 as a tilt or angle sensor.
identified, based on the player with minimum error (maximum Corepoints).
The display of smartphone 471 shows an initial "splash screen" screen display that has wavy water-At the conclusion of the game, the player with the low- like lines on it, along with also some branding and a est error (maximum number of Corepoints) is identified corporate product slogan or aphorism such as "Abs of as the winner, and the result is entered into a perma- cement with absement" or the like. Upon the screen is nent record in cloud storage 290. This record is made also the HORIZONETmindication, i.e. the zone of the for every sample so that a player in the future can play horizon, as indicated by waves 481 symbolizing water against a player from the past.
waves as seen on an ocean view that indicate horizon.
Players can compete against each other in real time, The waves are synthetic in some embodiments whereas or they can compete against other players from the in others there is provided a view of a nice beach, ro-past, or against themselves from the past, e.g. each tated to indicate either the horizon as target position shape S200 defines a "course" that can be played, and or reference position.
any player can compete against someone who did, in Coordinates are selectable as "Normal" (Forward) the past, that same course of shape S200.
mode or "Corrective" (Backward) mode, i.e. as either showing the horizon as it should be, or as it currently It either rocks around on the floor directly, or rocks is.
on a surface 560, or mates with a socket attached to The waves 481 are modulated as per their degree of surface 560 to keep it from slipping side-to-side or fore-correspondence with the correct angle of tilt. to-aft.
In level 0 of the game, the waves simply show where Near the bow of surface 567 is a display region 571 the horizon should be, and the player keeps level, and comprised of an array of addressable picture elements, the waves indicate tilt, e.g. as going "stormy" when or simply a place for putting an external display such upset, or "calm" when closer to target.
as an external srnartphone or other computing display In level 2 of the game, the target angle is built into device, i.e. region 571 may simply be a rubber mat or a pre-tilt or pre-rotation of the waves 481 as displayed, a recess or indentation or framing.
and this is the means for showing the target angle.
For a better neck position, the display region 571 is, Instance 401 shows level 2 of the game, where we see in some embodiments of the invention, on the ground a rough jagged trace corresponding to the actual angle in front of the apparatus, or the surface 567 is extended as a function of time, and the smooth trace of instance outward more, just for display 571, or in some embod-S200 as the desired angle as a function of time.
iments there is a small wireframe extension attached ln one embodiment, the waves 481 swing around to the front of surface 567 that slides in and out, to to various angles, and the player must counteract this extend forward (adjustable) and hold display 571.
tendency.
Alternatively an eyeglass 509 is used to display the In one embodiment, the absement is presented as a material as instance 501 of a game scenario. The ma-seasickness, and the goal is to minimize seasickness.
terial of instance 501 is displayed on display 571 or This is done with a seasickness metaphor which is eyeglass 509 or both or a mixture of these (e.g. some how the whole system works to present absement as content as an augmented reality overlay).
something comprehensible to the user, since teaching A typical game is a game of buzzwire or buzzwave in of principles of absement is usually based on water ac-which the user U201 needs to follow along a complex-cumulated in a bucket, for example (as a metaphor for valued waveform, such as wave 540, with pointer 539.
the process of integration).
Wave 540 is a function of spime (space or time) on a Fig. 5 illustrates an embodiment of the invention stime (space or time) axis 541, and in particular, its for planking or pushups, upon a surface that is shaped real part as plotted on a real axis 542, and its imagi-like a foreshortened surfboard or "boogie board". The nary part as plotted on an imaginary axis 543. Wave board is about 18 inches wide, and about 28 inches 540 may be generated by any of a wide range of meth-long, and there is a ball joint or swivel on the bottoin ods. A satisfactory method is to slide a microphone of it, about 13 inches back from the front. This point along a rail, at one end of which there is a loudspeaker forms a pivot 562, around which the board can rock connected to an oscillator output of a lock-in ampli-side-to-side, fore and aft, rotate clockwise or counter-fier, wherein the microphone is connected to the signal clockwise with respect to the ground, or push a little input of the lock-in amplifier, and the "X" and "Y' out-bit closer to the ground or a little further away (i.e. to puts of the lock-in amplifier are recorded while doing sense total weight upon the apparatus). Thus there are this. The result is an object that can be visualized in 4 degrees of freedom, of which three are motion based 3D space, resembling a corkscrew kind of shape, with and the fourth is pressure or force based.
some irregularities due to sound reflections off walls, Pivot 562 defines centerline 565 which is about etc., and this is shown here for level 2 of the game.
where the center-of-gravity for the upper body con-The game is a 3D maze of sorts which must be navi-tact region of user U201 falls along. Thus for doing gated by keeping the pointer 539 as close to wave 540 as pushups or doing planks in pushup position, user U201 possible. The distance between pointer 539 and wave places the palms along centerline 565, whereas for do-540 is captured as a function of space or time, along the ing planks in forearm position, user U201 places his wave 540. The time integral of this distance is the ab-or her elbows approximately along centerline 565, with sement, and is displayed dynamically on a scoreboard forearms 570 resting on the board surface 567.
551, e.g. here shown as 3.25 degree seconds (degrees Pivot 562 is a ball or partial ball, such as a rounded times seconds). (For small angles, absement and absan-end of a pipe end cap, or other rounded shape, in gle are approximately equivalent.) The score of 1000 /
the range of 1.5 to 6 inches in diameter, or so (can absement is also displayed. The absement increases or be smaller or larger depending on desired degree-of-stays the same during the following of the wave, while difficulty when it sits directly on a flat floor or flat the user U201 tilts surface 567 left-to-right to move base surface 560).
along the real axis and fore-to-aft to move along the imaginary axis. Holding surface 567 level keeps the design, with a water therned pattern 640. The water pointer 539 on the spime axis 541, whereas the more themed pattern 640 is made of LEDs (light emitting tilted surface 567 is, the further from the spime axis diodes) that are addressed or sequenced with waves the pointer 539 goes. A tilt sensor senses the left-right that either create the game experience, or augment it.
tilt of surface 567 and converts that tilt to a position Preferably surface 667 is translucent and carries light along the real axis 542, and senses the fore-aft tilt of throughout it, so that a relatively small number of pic-surface 567 and converts tht to a position along the ture elements ("pixels") can be used to create an in-imaginary axis 543.
teresting set of wave patterns for gaming experience In level 0 of the game, the wave 540 is just the spline design.
axis itself, and the goal is simply to keep surface 567 as Surface 667 has a port side that faces to the user's level as possible at all times, to stay on this axis. The left when the user is facing forward toward the bow, absement is a record of how unlevel surface 567 is over and a starboard side that faces to the right when the time. Tilting 1 degree for 10 seconds costs the same user is facing forward toward the bow. The bow is a against one's performance record as tilting 10 degrees pointed end intended to indicate a user's forward-facing for 1 second. The goal here is simply to get the small-direction in the game. There is a stern side opposite est area under that curve. The absernent is computed the bow side. Base 660 is radially synunetric and the in Euclidian geometry or in other embodiments it is controller 600 is preferably designed so that the whole computed in any other metric space or even in spaces board surface 667 can pivot and point in any head-that are not metric spaces. Thus more generally an ing. The heading in which the board 667 is facing, accumulated deviation from straight is computed, as within the game, is indicated by a compass 620 dis-desired.
played as a virtual compass display on a screen display In level 2 of the game (what's shown in the figure), 671, in some embodhnents that have the display 671.
the goal is to navigate the path as close as possible.
In embodiments without display 671, the heading is in-Again the absement is computed in Euclidian georn-dicated in a simpler (lower cost) fashion by way of a etry or in other embodiments it is computed in any small number of pixels of wave patterns in pattern 640.
other metric space or even in spaces that are not metric Display 671, when present, exists in a housing that also spaces. Thus more generally an accumulated deviation houses a processor 602 that runs the score keeping or from the course is computed, as desired. gaining functions of the apparatus.
As such there are three degrees of freedom in the To up the ante, the user U201 places the feet on a orientation of the board of surface 667. A player or fitness ball or other unstable surface such as surface 575, which is part of the game. Surface 575 includes user:
a gas pedal 576 and brake pedal 577 allowing the user 1. tilts the board along the port-starbord axis;
U201 some control over navigating the course of wave 540. 2. tilts the board along the bow-stern axis; and In level 2 of the game, surface 575 is grounded and simply serves as a pedal interface.
3. changes its compass heading, i.e. the way that it In level 3 of the game, surface 575 becomes unstable is pointing.
and rocks or swivels to make it harder for the user U201 The controller 600 outputs three signals for each to keep level and stable. This more quicly develops core of these three axes, and these three signals are read muscles.
by processor 602. Processor 602 generates a course Fig. 6 illustrates an embodiment of the invention and displays it on screen display 671. The course is, that does not require the use of a separate smartplione in one euabodiment, a complex-valued waveform dis-or other external device. Surface 667 rests upon a joy-played as looking down the spirne axis, with the real stick type controller 600, that supports surface 667 and axis aimed toward STARBOARD side, and the imagi-allows it to swivel as a whole. Controller 600 is a game nary axis aimed toward BOW end of surface 667. The controller. A satisfactory game controller is the Log-real axis 642 runs from PORT to STARBOARD side itech Extreme 3D Pro, cut off shorter so that the handle and passes through the center of rotation of surface part is essentially replaced by the board of surface 667.
667 afforded by controller 600. The imaginary axis 643 However, in a preferred embodiment, a game controller runs from STERN to BOW and also passes through the 600 is built directly from first principles, as part of the center of rotation of surface 667 afforded by controller device of the invention. The controller 600 sits on base 600.
660.
A player must navigate in a somewhat circular Rio-Surface 667 is fashioned after a surfboard in style of tion to follow the waveform. In a simple embodiment the waveform is of the form ei2"Ct where i = \/T ¨ 1) or craft, but with outstretched arms in pushup posi-and fc is a carrier frequency of oscillation, and t is time.
tion, or the user planks upon the wheel in forearm po-This circular motion develops abdominal nniscles sition. A hub 700 for the wheel sits on the bottom while resulting in a fun game. As the player progresses of the wheel facing down (hence leftmost, shown as a to higher levels of the game, higher harmonics are in-dotted or hidden line because it is under the solid disk troduced into the waveform, resulting in greater chal-and we can't normally see it from above). The hub 700 lenges.
is a round end or ball or half ball or fraction of a ball Additionally, a fourth degree of freedom is provided that faces a floor or ground or tarmac or the like, or in some embodiments, and this fourth degree of free-in other embodiments, faces a socketplate that sits on dom is the total weight pressing down on surface 667, the ground or earth or tarmac or floor, or the like. The allowing a user to "bump", press, thrust, etc., with ball of hub 700 and the socketplate together form a floorward and ceilingward (or groundward and sky-ball and socket joint in some embodiments whereas in ward) forces. For example, when there is no gravi-other embodiments the ball of hub 700 sits directly on tyward (floorward or ground.ward) force, a process in the ground, making the unit more compact and easier processor 602 goes into a standby mode, until there to carry around. In some embodiments the hub 700 is some such force that "wakes up" processor 602 and detaches and stows at the edge of the disk or ring, so begins the game or training session.
that it is easier to carry the disk or ring under the arm A heavy player is also sensed and distinguished from while walking or jogging.
a player weighing less, and also the apparatus tracks In another embodiment, a portion of the hub de-and monitors weight gain and loss over time.
taches while the rest of it remains. The portion that Thumb control 601 is for being pressed by a thumb detaches is hemispherical in shape, and there are pro-of a user, and it faces upward on surface 667. Index vided a variety of different hemispheres (i.e. different ball diameters) that a user can use in order to adjust finger control 602 faces downward and is thus shown as a dotted (hidden) line in the drawing of Fig. 6. Fin-the degree of difficulty of keeping the apparatus level or ger controls 602, 603, 604, and 605 face downwards so at the desired or required angle during a fitness train-a user can grasp them together and controls 601 to ing session such as planking or doing pushups on the 605 form a left-hand port-side control. A right handed board or wheel 767. The detachable and/or split hub starbord side control is also provided as a mirror image, works also with surface 667 of Fig. 6 (surfboard shape) mirrored along the imaginary axis 643. Additionally el- or other embodiments of the invention.
bow controllers such as control 606 for the left elbow Preferably the split hub embodiment has an upper and another controller for the right elbow, allow the part attached to the wheel that has a transparent win-user to control some functions with the elbows while dow facing up, and a lower part that has a cavity or reseting the elbows near the real axis 642.
recess into which a smartphone can be placed. In this embodiment the user can look through the window and In another embodiment, surface 667 is a semitrans-see the screen of a smartphone inside the hollow or par-parent mirror, and another surface mirror is below it, and together the two mirrors form an infinity mirror.
tially hollow lower portion of hub 700. The two halves are held together by magnets, and the magnetic clip In level 0 of the game, base 660 can be the bottom opens and closes to accept the smartphone. The smart-mirror, and LEDs around the space between the in -phone typically displays a heading indicator that shows finity mirrors provide an image of an infinity tunnel.
North, and runs a driving game that is played with an The player simply attempts to steer straight down the infinity tunnel.
additional degree of freedom, namely the position of the hands upon the wheel of surface 767. Now the user For level 2 of the game, an actuated bottom mirror moves in a certain pattern and the user attempts to can:
follow that pattern.
1. tilt along a port-starboard axis;
Fig. 7 shows an alternate embodiment of the inven-tion in which surface 767 is a wheel that comprises, 2. tilt along a bow-stern axis;
includes, or is a disk 768 or a ring 769 with spokes 797.
The ring 769 with spokes 797 is constructed like a steer-3. turn the wheel clockwise or counterclockwise with ing wheel, i.e. with the spokes below (behind) the ring, respect to his or her body and the ground;
angled down toward the floor, so as not to obstruct its topside. The spokes meet at hub 700. A player grasps 4. and also change his or her body heading with re-the wheel like one might grasp the steering wheel of a spect to the ground, e.g. with the user's head car or boat or airplane or other vehicle or conveyance facing North, then change a little bit to the

Claims

WHAT I CLAIM AS MY INVENTION IS:

1. A core fitness training system for planking or pushups, said system including a user-interface surface for the hands or arms of a user of said surface, a pivot for said surface, and a video game for being played with said system, a position on a cursor of said video game varying in proportion to the tilt of said surface.
2. The core fitness training system of claim 1, where the horizontal position of said cursor is controlled by a left-right tilt of said surface, and the vertical position of 14. The system of any of claims 1 to 14, said system said cursor is controlled by a fore-aft tilt of said surface. including an audio or visual feedback, said feedback 3. The core fitness training system of claim 2, where indicating an error between a desired and an actual a score of said video game increases in proportion to a tilt of said board or surface.
decrease in the time-integral of an error signal between 15. A fitness device for holding a planking position on, a desired cursor position and an actual cursor position. said fitness device comprised of two boards that are 4. The core fitness training system of claim 2, where a rotatably connected, a first board for being placed on score of said video game increases in proportion to the the ground or a floor, and a second board for bearing square root of the sum of the squares of a horizontal human weight from the hands or arms of a user of said error and a vertical error, said horizontal error being fitness device, said fitness device also including a sensor the difference between an actual left-right tilt of said for sensing tilt of said second board.
surface, and a desired left-right tilt, said vertical error 15. A fitness device for holding a planking position being the difference between an actual fore-aft tilt of on, said fitness device comprised of two boards that said surface, and a desired fore-aft tilt. are rotatably connected, one of said board for bearing 5. A fitness system for training of a user's core mus- human weight from the hands or arms of a user of said cies, said fitness system including a board and a game fitness device and having a receptacle for receiving a activity, said board having a pivot of sufficient strength tilt-sensing device.
to bear downward-facing load of human body weight, 16. The fitness system or device of any of claims 1 and said game having a game controller, said game to 16, further including an apparatus for accepting the controller responsive to tilting of said board. feet of said user.
6. A fitness system comprising a joystick type game 17. The fitness system or device of any of claims 1 to control, said game control incuding a large fiat surface 16, further including a pedipulatory input device for for accepting the hands or arms of a player, said joy- accepting input from one or both feet of a user of said stick operable by tilting said surface. fitness system or device.
7. A fitness system comprising two flat surfaces, a first 18. A hands-and-feet planking system, said system in-surface for placement on a floor or ground, and a second eluding a wobbleboard for the hands, said wobbleboard surface for accepting the hands or arms of a user of for controlling a pointing device of a game console or said fitness system, said surfaces joined with a pivot or game device, said system also including a separate foot-swivel joint having at least two degrees-of-freedom, operated controller to which said game console or de-8. The system of claim 7 where the degroes-of-freedom vice is also responsive.
are left-right tilt of the second surface with respect to 19. A principally flat game console, said game con-the first surface, and fore-aft tilt of the second surface sole including means for sensing tilt of said game con-with respect to the first surface. sole, as well as pivotal support means, said pivotal sup-9. The system of claim 8 further including a third port means of sufficient strength to bear human body degree of freedom, the third degree of freedom being a weight.
rotation of said second surface with respect to said first 20. A method of fitness, said method comprising the surface, along an axis of rotation that passes through steps of: providing a surface upon which a particiant both of said surfaces.
can perform a planking operation or pushups, said sur-10. The system of any of claims 7 to 9, further includ- face upon a pivot which keeps it from falling down un-ing a sensor for sensing movement along said degrees- der the load of the user's body weight, but allows the of-freedom.
surface to swivel or tilt or rotate; providing a game for 11. The system of claim 10, where said sensor is a the user to play;
providing a visual or audible cursor for joystick control, said joystick control being attached said game; varying a position or sound of said cursor between said first surface and said second surface, in response to a tilt of said surface.
wherein relative movements of the two surfaces oper- 21. The method of claim 20 where said cursor is a ates said joystick.
visual cursor, said cursor moving left-to-right in pro-12. The system of any of claims 7 to 10, further includ- portion of a left-to-right tilting angle of said surface, ing a recessed area in said second surface, said recessed and said cursor moving up-to-down in proportion to a area for receiving a smartphone.
fore-aft angle of said surface.
13. The system of any of claims 7 to 9, further includ- 22. The method of claim 20 where said cursor is an au-ing a recessed area in said second surface, said recessed dible cursor comprised of a musical sound, said musical area for receiving a smartphone, said smartphone for sound being played at musically accurate or pleasant sensing movement along said degrees-of-freedom. note pitches when said board is at a desired tilt an-gle, and said musical sound being played at a musi- 36. The fitness device or application of claim 32, where cally bent, warped, or unpleasant note pitch when said said course is a virtual wire in a buzzwire game.
board is at an undesired tilt angle. 37. The fitness device or application of claim 32, where 23. The method of claim 22 where said musical sound said course is a glowing one dimensional manifold in a is a song played at normal pitch when said surface is at two or more dimensional space.
a desired tilt angle, and at a wavering or warped pitch 38. The fitness device or application of claim 32, 36, or when said surface is at an undesired tilt angle. 37, where said game includes a virtual ring around said 24. The method of claim 23 where the degree of wa- virtual pathways, and a sound effect when said virtual vering or warping of pitch is proportional to the square ring touches said virtual pathways.
root of the sum of the squares of the difference between 39. The fitness device or application of claim 28, or the actual and desired left-right tilt of the surface and any of claims 30-38, where said game includes music, the difference between the actual and desired fore-aft and where said music plays at a pleasant or normal tilt of the surface.
pitch when said user follows said course, and where 25. A means for fitness based on a competitive gaming said music plays at a wavering or warped pitch when situation in which one or more players at the same or said user deviates from said course.
different points in time compete against each other or 40. The fitness device or application of claim 39, where themselves, each player putting a substantial portion of the degree of wavering or warping of pitch is propor-his or her body weight upon a pivotable gaming inter- tional to the square root of the sum of the squares of face, while being presented with a course to navigate, the difference in each dimension between a user's posi-and a navigational cursor that is controlled by tilting tion in the course, and the nearest part of the course.
said interface.
41. The fitness device or application of claim 28, or 26. A pointing device for being operated by both hands any of claims 30 to 40, where said fitness device or or both forearms of a user, the pointing device includ- application provides a score to said user, said score ing a surface for facing upward, and a ball or partial derived from a reciprocal of the time integral of a time-ball for facing downward.
varying error function, said error function equal to the 27. The pointing device of claim 26, further including difference between the user's navigation of the course a socket for said ball.
and the actual course.
28. A fitness device or application program for the 42. A collaborative gaming system, using a plurality pointing device of claim 26 or 27, said application in- of fitness devices or applications of claim 28 to 41, said eluding a course to be followed by a user of said pro- collaborative gaming system computing relative scores grain by operating the pointing device, of multiple players by integrating a distance of devia-29. A fitness device or application program for the tion from a course to be followed by said players.
pointing device of claim 26 or 27, said application in- 43. The system of claim 42 where said course is a eluding a cursor made visible to a user of said device or complex course having a real part derived from the in-program, said cursor responsive to said pointing device, phase component of a lock-in amplifier, and an imagi-30. The fitness device or application of claim 29, said nary part derived from a quadrature component of the device or application providing a course to be followed lock-in amplifier.
by a said user.
44. A system for generating the course of any of claims 31. The fitness device or application of claim 30 where 1 to 43, said system including a stationary element and said course includes one or more virtual pathways that a moving element, and a lock-in amplifier device, wave the user must stay within.
analyzer device, phase-coherent detector device, or ho-32. The fitness device or application of claim 30 where modyne device, said device having a reference input said course includes one or more virtual pathways that responsive to said statinary element, and a signal in-the user must stay without.
put responsive to said moving element, or vice-versa.
33. The fitness device or application of claims 28, 30, 45. The system of claim 44 where said elements are 31, or 32, where said course is a path in a virtual maze antennae.
game.
46. The system of claim 44 where said elements are 34. The fitness device or application of claims 28, 30, transducers.
31, or 32, where said course is a road or path in a 47. The system of claim 46 where one of said ele-virtual driving game.
ments is a loudspeaker, transmit transducer, transmit 35. The fitness device or application of claims 28, 30, hydrophone, or transmit geophone, and the other of 31, or 32, where said course is a path in a virtual space- said elements is a microphone, receive transducer, re-ship driving game.
ceive hydrophone, or receive geophone.

48. A portable or mobile computer application, said tasks, said task being to design a virtual object, said application for being used in a portable or mobile corn- object being constructed by moving said virtual cursor.
puting device, by placing said device on a flat surface 60. The system of claim 59, where multiple users each that can tilt or pivot at a plurality of different angles, have one of the user-interfaces of claim 51, where a said device including a tilt sensor, said tilt sensor sens-collaborative CAD task is presented, and where each ing said plurality of different angles, said application user of said system is presented with their own cursor providing a course to be navigated by a user of said ap- and the cursors of other collaborators.
plication, said application for integrating a tilt of said 61. A
workplace environment based on the system of surface and generating a score based on a reciprocal of any of claims 51 to 60, where points, cash, or other an absement of said tilt, incentives are provided to users creating the best CAD
49. A fitness device comprising a wobbleboard with designs using the system.
a holder for a smart phone, said holder comprising a Additional drawings (in addition to the drawings, di-recessed region in which to place a smartphone while agrarns, and illustrations throughout the description):
exercising on said wobbleboard.
50. A steering device for operating a virtual game while bearing weight of a human user hanging from or resting a portion of his or her body weight on said steering device, said steering device including a swivel, said steering device for including a computing device, said computing device including a tilt sensor, said com-puting device displaying a course to be followed by a user of said steering device, said computing device also displaying a cursor varying in proportion to a tilt of said steering device, said computing device providing a score based on a reciprocal of an accumulated deviation between said cursor and said course.
51. A core fitness training system for planking or pushups or pullups or L-seat exercises, or the like, said system including a user-interface for the hands or arms of a user of said user-interface, a pivot for said user-interface, and a video game for being played with said system, a position on a cursor of said video game vary-ing in proportion to the tilt of said user-interface.
52. The system of claim 51, where said game includes a course to be followed by a player of said game, and where a score of said game is derived from a reciprocal of an accumulated deviation of said cursor from said course.
53. The system of claim 52, where said course is a waveform from a lock-in amplifier.
54. The system of claim 52, where said game includes levels, and where a level of said game includes where said course is a straight line path.
55. The system of claim 52, where said course is a road.
56. The system of claim 52, where said course is a maze.
57. The system of claim 52, where said course is a flight path.
58. The system of claim 54, where said cursor is au-dible, and said cursor is the pitch stability of a sound track.
59. The system of any of claims 51 to 58, where said system includes a CAD (Computer-Aided Design) for gravitational-waves from binary black-hole merg-[84] D. Wood and S. Graham. Permeable boundaries in ers. In Journal of Physics: Conference Series, volume the software-sorted society: Surveillance and the dif-363, page 012031. IOP Publishing, 2012.
ferentiation of mobility. Mobile technologies of the city, [67] F. Morozov. Is smart making us dumb. The Wall pages 177-191, 2006.

Street Journal, 2013.
[68] M. Mortensen. Who is surveilling whom? negotia-tions of surveillance and sousveillance in relation to wikileaks' release of the gun camera tape collateral murder. Photogmphies, 7(1):23-37, 2014.
[69] R. Munro. Actor-network theory. The SAGE handbook of power. London: Sage Publications Ltd, pages 125--39, 2009.
[70] C. Norris, M. McCabill, and D. Wood. The growth of CCTV: a global perspective on the international diffu-sion of video surveillance in publicly accessible space.
Surveillance F Society, 2(2/3), 2002.
[71] D. Pye. The nature and art of workmanship. Cam-bridge UP, 1968.
[72] D. Quessada. De Ia sousveillance. Multitudes, (1):54-59, 2010.
[73] P. Reilly. Every little helps? youtube, sousveilkmce and the `anti-tesco'riot in stokes croft. New Media Society, 17(5):755-771, 2015.
[74] C. Reynolds. Negative sousveillance. First Interna-tional Conference of the International Association for Computing and Philosophy (IA CAP11), pages 306 ¨
309, 2011.
[75] V. Robertson. Deus ex machina? witchcraft, and the techno-world. Literature 6' Aesthetics, 19(2), 2011.
[76] F. Spielman. Infrastructure Trust launches plan to overhaul Chicago's outdoor lights. Chicago Sun-Times, 2015 September 17.
[77] R. Stivers and P. Stirk. Technology as magic: The triumph of the irrational. AkC Black, 2001.
[78] C. A. Stutt. Low-frequency spectrum of lock-in ampli-fiers. 1949.
[79] R. Vertegaal and J. S. Shell. Attentive user interfaces:
the surveillance and sousveillance of gaze-aware ob-jects. Social Science Information, 47(3):275-298, 2008.
[80] Y. Wang, Y. Zhang, X. He, G. Fang, and H. Gong.
The signal detection technology of photoconductive detector with lock-in amplifier. In Selected Pro-ceedings of the Photoelectronic Technology Committee Conferences held August-October 2014, pages 95220F
95220F. International Society for Optics and Photon-ics, 2015.
[81] K. Weber. Mobile devices and a new understanding of presence. In Workshop paper from SISSI2010 at the 12th annual ACM international conference on ubiqui-tous computing, page 101, 2010.
[82] K. Weber. Google glasses: Surveillance, sousveillance, equiveillance. In 5th International Conference on In-formation Law and Ethics, Corfu/Greece. download-able as paper 2095355 from papers.ssrn.com, 2012.
[83] K. Weber. Surveillance, sousveillance, equiveil-lance: Google glasses. Social Science Research Net-work, Research Network Working Paper, pp. 1-3 -http://tinyurl. corn/ 6nh74;j1, 2012.