The last two posts¹ were focused on the lens. They dis­cussed its var­i­ous modes of oper­a­tion, the dichotomy between the out­ward-fac­ing “causal esti­mate” and the inward-fac­ing “eval­u­a­tion”, and the prob­lems of for­eign con­text and overgeneralization.

But it’s impor­tant to note that the lens is, at heart, a sta­tis­ti­cal machine. It is only con­cerned with how accu­rately it can per­form its tasks (casual pre­dic­tion, event inter­pre­ta­tion etc.), and does not inten­tion­ally dis­tort itself to sat­isfy any agenda. In other words, the lens lacks agency and a hypo­thet­i­cal “lens-only” generic is only good for mak­ing unin­ter­ested pre­dic­tions or clas­si­fi­ca­tions².

To breathe more life into the generic mind I intro­duced the agent, a cog­ni­tive process with an inher­ent goal to cause the pro­duc­tion of cer­tain eval­u­a­tions. The con­cept behind its oper­a­tion is as follows:

• Use the lens to per­form value pre­dic­tion. In other words, given some known prior infor­ma­tion and the desired eval­u­a­tions, fig­ure out what must be in the pos­te­rior infor­ma­tion for the lens to pro­duce such an evaluation.
• Try to engi­neer the actions and cir­cum­stances of the generic to increase the odds that the right events hap­pen. If this is done suc­cess­fully then the lens will pro­duce the eval­u­a­tions that the agent desires.

The Textbook Artificial Intelligence

An arti­fi­cial intel­li­gence as clas­si­cally defined (here­after called “the AI”) can be thought of as a generic with the fol­low­ing qualities:

• It has an inter­nal sense which esti­mates some kind of objec­tive func­tion, i.e. how close the AI thinks it is to achiev­ing a cer­tain goal.
• It has an eval­u­a­tion called Improvement which mea­sures dif­fer­ences in the objec­tive func­tion between the prior and the pos­te­rior information.
• It has an agent which moti­vates the AI to choose actions that result in Improvement.

Some argue that this setup is already enough to pro­duce arbi­trar­ily com­plex behav­iors, and indeed thought exper­i­ments like the Paperclip Maximizer³ sug­gest that a suf­fi­ciently advanced AI can be very cre­ative in its approach, devel­op­ing many instru­men­tal goals such as Stay Alive, Maximize Raw Materials, Maximize Production Capacity, and even­tu­ally Eliminate Rivals (incl. Humans) in its pur­suit for Improvement. But such thought exper­i­ments sweep a huge amount of com­plex­ity and numer­ous hid­den assump­tions under the phrase “suf­fi­ciently advanced AI”. For example:

Who designed the inter­nal sense that is the AI’s objec­tive function?
A scary super­hu­man AI is often assumed to have an arbi­trar­ily com­plex, yet per­fectly pre­cise method of mea­sur­ing its objec­tive. In real­ity, our goals are vague and can be dif­fi­cult to define pre­cisely on a grand scale.

What is the space of actions the AI can take?
A scary super­hu­man AI is assumed to be able to do any­thing, or arbi­trar­ily expand its capa­bil­i­ties. But it is very dif­fi­cult to rea­son about actions that were not known to be pos­si­ble. A paper­clip max­i­mizer capa­ble of doing this arbi­trar­ily well may as well sim­u­late entire uni­verses as sand­box envi­ron­ments, which kind of make its paper­clip max­i­miza­tion skills seem underwhelming.

How are the instru­men­tal goals being made, and how are they sched­uled or balanced?
This hypo­thet­i­cal max­i­miza­tion AI needs to cre­ate instru­men­tal goals, and some­how not get side­tracked by these goals. It must have a remark­able amount of ded­i­ca­tion and meta-AI knowl­edge to be able to cre­ate these goals with­out miss­ing the for­est for the trees. And arguably, intel­li­gent enti­ties that don’t feel a need to be wholly focused on a sin­gu­lar goal are not nec­es­sar­ily less advanced than intel­li­gent enti­ties that do feel such a need.

This last point is impor­tant – the text­book AI def­i­n­i­tion, as broad as it is, excludes intel­li­gent agents that act on shift­ing, poorly defined, or inter­nal goals. Humans do exhibit this level of rich­ness, and so should generic sen­tient beings.

Multiple Agents in a Tug of War

We can get a much richer (and more chaotic) set of behav­iors if we allow gener­ics to house more than one agent. How can a generic act in the inter­est of mul­ti­ple, pos­si­bly con­flict­ing goals? We can con­sider each agent kind of like a force, where the action of the generic is the over­all out­come of each of these forces. Each agent has an intrin­sic pool of energy which is expended when the agent exerts its influ­ence on a generic’s behav­ior. When the lens per­forms event inter­pre­ta­tion of the result­ing actions, each agent may or may not replen­ish energy depend­ing on the eval­u­a­tions that are pro­duced. The agent can choose to use the energy imme­di­ately, save it for spe­cific sce­nar­ios, or oth­er­wise spend its energy judi­ciously accord­ing to some strategy.

In prin­ci­ple, the eval­u­a­tions that replen­ish energy (the source eval­u­a­tions) may or may not match the eval­u­a­tions that an agent tries to gen­er­ate (the sink eval­u­a­tions). But clearly, an agent with no energy may as well not exist. It is usu­ally more use­ful to con­sider cases where there is con­sid­er­able align­ment between source and sink eval­u­a­tions, for the sim­ple rea­son that such agents act to pre­vent them­selves from run­ning out of energy.

It is impor­tant to note that many events are out­side a generic’s con­trol, so even if these agents take part in pos­i­tive feed­back loops to increase their own energy, ran­dom exter­nal events will gen­er­ate a wide vari­ety of eval­u­a­tions, which may pre­vent any one agent from dom­i­nat­ing. Also don’t for­get that inter­nal senses can par­tic­i­pate in the inter­ac­tion, and that the inter­nal senses may depend on any part of the generic mind, includ­ing the behav­iors of the agents. This gives a recur­sive fla­vor to the whole sys­tem and explains why actions can be entirely inter­nally moti­vated. Agents can have strong pref­er­ences for or against actions even when the exter­nal effects of the actions are neg­li­gi­ble, sim­ply because the agents are tuned into changes picked up by the inter­nal senses (e.g. in self-image, per­ceived social face, etc.).

Lens to Agent, Agent to Lens

All agents depend on the lens for value pre­dic­tion and event inter­pre­ta­tion, so changes in the lens can con­sid­er­ably affect the bal­ance of power between the agents. Particularly of note is the effect of over-gen­er­al­iza­tion. If a lens devel­ops a ten­dency to pro­duce a cer­tain eval­u­a­tion more often than usual, then energy can be redis­trib­uted in a skewed man­ner. The generic will act in unusual ways reflect­ing the new bal­ance of power between the agents, which could result in a self-ful­fill­ing prophecy effect as the generic puts them­selves into sit­u­a­tions where the mis­tak­enly over­pro­duced eval­u­a­tion is fur­ther enhanced.

But no mat­ter how skewed it can some­times be, the lens is an unin­ter­ested sta­tis­ti­cal machine. If the lens is put into sce­nar­ios where its over-gen­er­al­iza­tion ten­den­cies are revealed through a series of incor­rect value pre­dic­tions, the lens will cor­rect itself and bal­ance between the agents would be restored. Closely related is the com­plex behav­ior spe­cial­iza­tion effect, where gener­ics engag­ing in more com­plex (but not wholly unpre­dictable) behav­iors tend to dis­trib­ute energy between more agents. This is because the lens tends to use more var­ied eval­u­a­tions to more accu­rately explain the behav­ior of the generic, result­ing in more vari­ety in how agents replen­ish and expend energy. Said more sim­ply, instead of explain­ing all behav­iors under the eval­u­a­tion Uh, Just Because?, the accu­racy-seek­ing behav­ior of the lens encour­ages a generic to pro­duce finer-grained eval­u­a­tions for explain­ing their behav­ior, which in turn allows energy to be shared across more agents.

1. Which were posted nearly 3 years ago! I am com­ing back from a hia­tus caused by pri­or­i­tiz­ing other work. You may notice that I have changed writ­ing styles, mak­ing my posts more curt and tech­ni­cal. This is because I felt that the pre­vi­ous style was hard to main­tain and diluted the con­tent too much.
2. This sounds an awful lot like the cur­rent state of AI. Regardless of how much mean­ing humans put into an AI agent’s deci­sions, to the AI itself the deci­sions are sim­ply the ones observed to max­i­mize some goal func­tion, which is usu­ally some mea­sure of pre­dic­tive accuracy.
3. A super­hu­man AI that attempts to max­i­mize a factory’s pro­duc­tion of paper­clips, which even­tu­ally grows beyond the fac­tory and threat­ens human­ity in try­ing to cre­ate more paper­clips than humans will ever need. See the linked arti­cle for more information.