The Scientific Method in Proficiology

Physics equipment and ruler

In a sense, study­ing generic sen­tient beings is kind of like study­ing a fic­tional uni­verse. There’s no obvi­ous notion of accu­racy or cor­rect­ness – after all, a generic mind is not nec­es­sar­ily an accu­rate depic­tion of human psy­chol­ogy. The whole field seems kind of self-con­tained and any state­ment about the world of the gener­ics frankly appears unfal­si­fi­able, in the same way that the­o­ries about a fic­tional uni­verse are impos­si­ble to dis­prove through experiment.

Why should we care about such fic­tional domains? When there are high stakes to explain or under­stand a phe­nom­e­non, we look toward sci­ence for answers. It is tempt­ing to treat sci­en­tific con­cepts as con­crete things that objec­tively exist, mak­ing sci­en­tific the­o­ries seem much more tightly grounded in real­ity than the afore­men­tioned the­o­ries around fic­tional uni­verses. But I argue that in many cases, the two are not all that dif­fer­ent. Of course, I am not say­ing that sci­en­tific work is purely fic­tional. All bod­ies of sci­ence agree that state­ments must be tested against real-world obser­va­tions in order to be taken seri­ously. However, there are many dif­fer­ent ways to do sci­en­tific work, and by group­ing these meth­ods under a com­mon name we are desen­si­tiz­ing our­selves to their dif­fer­ences. In point­ing out these dif­fer­ences, I will show you how sci­en­tific the­ory can be more fan­tas­ti­cal than we ini­tially expect.

Science-p: The Study of Evidence-based Procedure

Let’s start with the dri­est, least fan­tas­ti­cal way in which sci­ence can be done. Suppose that we expect to fre­quently encounter some famil­iar sce­nario, and that we have an explicit goal to achieve in that sce­nario. In other words, we have a prob­lem state­ment – each encounter with this sce­nario is an instance of our prob­lem, and we must try to achieve our goal for each prob­lem instance to the best of our abil­ity. We gen­er­ally do what I will call sci­ence-p when we research into the prob­lem hop­ing to improve our abil­ity to solve it. To explain sci­ence-p, we should first under­stand the main chal­lenges of try­ing to solve our problem:

  • The meth­ods used to achieve our goal can be quite involved, requir­ing many com­pli­cated steps. We need to some­how learn about such meth­ods in the first place.
  • There can be many hid­den ways in which the prob­lem instances can dif­fer. We need to reli­ably detect these dif­fer­ences so that we use the cor­rect meth­ods to solve them.
  • Our meth­ods do not always work every time, and can dif­fer in the extent and dimen­sion in which they solve our prob­lems. We likely need to try our meth­ods many times in order to assess their performance.

In sci­ence-p, obser­va­tion is king and above all else our focus is on try­ing to solve our prob­lem. In a research paper on a sci­ence-p sub­ject, the exper­i­men­tal setup / results / analy­sis sec­tions are the most impor­tant. Exactly why the result is achieved is of sec­ond-class impor­tance – for sci­ence-p, a method that demon­stra­bly and con­sis­tently achieves great results across a vari­ety of prob­lem instances is a good one, even if researchers are not very sure how it works.

This type of work is com­mon in med­i­cine1, so let’s use that field as an exam­ple. Suppose we know about one treat­ment pro­ce­dure for a dis­ease – per­haps some plant was com­monly used as a folk rem­edy for the dis­ease, and early exper­i­ments show that it does indeed improve the dis­ease out­come in a mea­sur­able way2. It is by far not per­fect though – the treat­ment effect is pretty mild and some peo­ple do not respond to the med­ica­tion at all. Science-p research can be done to iden­tify whether there are dif­fer­ences in the symp­toms or demo­graph­ics of peo­ple who are affected less by the med­ica­tion. Researchers can try chang­ing how we admin­is­ter the med­ica­tion to increase its potency. They can try to extract com­pounds from the plant to see which ones are respon­si­ble for the improve­ment, and try cre­at­ing other meth­ods of admin­is­ter­ing the same com­pound. Throughout the process, the com­mu­nity cre­ates new meth­ods, detailed mea­sure­ment pro­ce­dures, and an exten­sive tax­on­omy of dis­ease instances / treat­ment out­comes / side effects. But this research doesn’t help pre­dict what hap­pens when you do some­thing sub­stan­tially dif­fer­ent from the stan­dard pro­ce­dure, so sci­ence-p researchers must rely on other forms of sci­ence in order to come up with cre­ative new approaches for the problem.

Science-u: The Quest for a Universal Model

In sci­ence-u we no longer have an explicit prob­lem to solve. A the­ory from sci­ence-u is designed so it can be applied on an arbi­trary setup that has never been stud­ied before. One of the main goals of sci­ence-u is to cre­ate a model that can pre­dict to high pre­ci­sion the out­come of any setup from a uni­verse of pos­si­bil­i­ties (which can con­tain hypo­thet­i­cal sce­nar­ios that are dif­fi­cult if not impos­si­ble to engi­neer with cur­rent technology).

The cre­ation of a model here is cru­cial – gen­er­ally there are infi­nitely many setups in our uni­verse and we need a way to pre­dict an out­come in each case. To use sci­ence-u to pre­dict the out­come of a setup, we first trans­late the setup into our model in a way that makes the observ­able prop­er­ties of the setup match our spec­i­fi­ca­tions. Then we apply the model and try to find other prop­er­ties that can be mea­sured by obser­va­tion. The sci­ence-u pre­dic­tions are pre­cisely the out­comes of those obser­va­tions. We just need to run an exper­i­ment where we cre­ate the appro­pri­ate setup, make our obser­va­tions, and com­pare against the pre­dicted results.

In sci­ence-u, it is impor­tant that the setups are defined as pre­cisely as pos­si­ble. Problem instances from sci­ence-p are expected to vary con­sid­er­ably, but we don’t want a given setup from sci­ence-u to con­tain hid­den vari­a­tions. Experiments must try to engi­neer these setups with very pre­cise equip­ment in order to prop­erly test the pre­dic­tive accu­racy of a sci­ence-u the­ory. Science-u work is com­mon in physics3, a field which uses some of our most advanced equip­ment for exper­i­men­tal work.

A sec­ond impor­tant goal of sci­ence-u is to cre­ate mod­els that are as sim­ple as pos­si­ble. It’s pos­si­ble to fit obser­va­tional data with com­plex curves built out of thou­sands of para­me­ters, and build a the­ory say­ing that real­ity fits our com­plex curve. But it would be highly unsat­is­fy­ing to sci­ence-u research, even if the com­plex curve does cre­ate accu­rate pre­dic­tions. I believe part of the rea­son is because we have a lim­ited bud­get for design­ing and run­ning sci­ence-u exper­i­ments, so we need to be smart in what exper­i­ments we choose to test. We need to come up with setups that are dif­fer­ent from the setups of past exper­i­ments but are still pos­si­ble to repli­cate with our tech­nol­ogy, and we use our knowl­edge of the the­o­ries them­selves to come up with such exper­i­ments. This is hard to do with a com­plex the­ory which only works due to the fine tun­ing of thou­sands of parameters.

Science-a: Using Models for Approximation

Sometimes, a field of sci­en­tific research attempts to study an exist­ing real-world envi­ron­ment that is so com­pli­cated that we can­not really hope for a uni­ver­sal model. Similar to sci­ence-u, the pri­mary goal of sci­ence-a is to make accu­rate pre­dic­tions about setups from the envi­ron­ment. However, we can’t even observe all the nuances of the setups we care about, let alone faith­fully repli­cate these setups in exper­i­ments. Instead, sci­ence-a focuses on the cre­ation of many dif­fer­ent mod­els, all of which work on a sim­pli­fi­ca­tion of the envi­ron­ment and can be used to make impre­cise pre­dic­tions for a given setup.

Science-a mod­els are much more fal­li­ble than their sci­ence-u coun­ter­parts, so it is impor­tant for researchers to rec­og­nize the lim­i­ta­tions of these mod­els and know when to choose one model over another. Even with such pre­cau­tion­ary steps, sci­ence-a work can be much less reli­able than sci­ence-p or sci­ence-u work. For exam­ple, sci­ence-a is com­monly used in eco­nom­ics and cog­ni­tive / social psy­chol­ogy4, and we con­stantly find econ­o­mists dis­agree­ing on what effects a gov­ern­ment pol­icy will have, or psy­chol­o­gists dis­agree­ing on why we see cer­tain ten­den­cies in human behav­ior. By no means is sci­ence-a use­less though. Having mul­ti­ple sci­ence-a mod­els can help sci­en­tists study an envi­ron­ment from many dif­fer­ent per­spec­tives, and by hav­ing dis­cus­sions over these mod­els the depth of the community’s under­stand­ing of the envi­ron­ment improves over time. Doing sci­ence-a research not only helps cre­ate more accu­rate sci­ence-a mod­els, it also reveals pre­vi­ously unstud­ied nuances of the setups, thereby help­ing researchers cre­ate more pre­cise research ques­tions in the future.

That said, I think there is a ten­dency for sci­ence-a to per­form delib­er­ate under­fit­ting, where the com­mu­nity believes they will never fully under­stand the nuances of their setups and decides to focus on the most com­mon behav­iors or par­ti­tion their setups across fuzzy bound­aries. For exam­ple, psy­cho­log­i­cal and eco­nomic mod­els have an implicit under­stand­ing that some peo­ple or groups behave dif­fer­ently from what the model pre­dicts, but are gen­er­ally happy with suc­cess­fully mod­el­ing the behav­ior of the major­ity. They may also choose to focus on a spe­cific cul­ture or gov­ern­ment style, even though peo­ple / gov­ern­ments from dif­fer­ent cat­e­gories may be sim­i­lar and peo­ple / gov­ern­ments can vary con­sid­er­ably within one cat­e­gory. These kinds of sim­pli­fi­ca­tions help advance the field as a whole, but leave us unable to make sense of the excep­tional setups within the envi­ron­ments we study5.

Scientific Models as Fantasy

In sci­ence-p, we don’t really try to make mod­els. We might bor­row mod­els from sci­ence-u or sci­ence-a to help us come up with bet­ter meth­ods for our prob­lem, but we have no need to pre­dict what hap­pens in any sce­nario other than the one that occurs in our prob­lem instances. We can talk purely in con­crete terms – our meth­ods, our prob­lem instances, and how well we are achiev­ing our goal.

In sci­ence-u, we have a uni­ver­sal model that tries to pre­dict every­thing cor­rectly. Typically when we have mul­ti­ple mod­els, they do not agree and sooner or later we will run an exper­i­ment that dis­proves one model in favor of another. It is tempt­ing to believe that there is only The One True Model that is basi­cally syn­ony­mous with objec­tive real­ity, but even with our harsh require­ments we can end up mul­ti­ple mod­els which pre­dict the exact same behav­iors. For exam­ple, there are sev­eral equiv­a­lent inter­pre­ta­tions of quan­tum physics, which appear wildly dif­fer­ent but ulti­mately pro­duce the same pre­dic­tions6. Gravity is usu­ally described as a “force act­ing at a dis­tance”, but it is equally if not more accu­rate to describe grav­ity as the cur­va­ture of a 4D sur­face. Both the invis­i­ble force and the 4D sur­face are fan­tasies – they exist only within the model, not as actual “things” in the universe.

In sci­ence-a, we are aware of the fact that we are using mod­els, and that they are mere sim­pli­fi­ca­tions of a com­pli­cated envi­ron­ment. Each model describes a fan­tasy world of sim­pli­fied setups and out­comes, and we only ask for these mod­els to give fairly accu­rate pre­dic­tions in some sit­u­a­tions, i.e. the major­ity of setups within a cer­tain cat­e­gory. There is less of a ten­dency to ele­vate the model to objec­tive exis­tence, but it still hap­pens all the time. For exam­ple, to some the exis­tence of a person’s sub­con­scious mind / mem­o­ries can seem just as objec­tive as the exis­tence of their eyes / brain.

Science-s: Spanning all Possibilities with Customized Models

I con­sider profi­ci­ol­ogy to be apply­ing a dif­fer­ent kind of sci­en­tific approach, which I will call sci­ence-s. In sci­ence-s, we have mul­ti­ple pos­si­ble envi­ron­ments of real-world setups. Each envi­ron­ment fits inside a uni­verse which con­tains both nat­u­rally occur­ing and hypo­thet­i­cal setups. The out­come of a given setup can depend on the envi­ron­ment we are work­ing under, so it doesn’t make sense to talk about pre­dict­ing the out­come of the setup. But sci­ence-s is not inter­ested in mak­ing this kind of pre­dic­tion any­way. Instead of eval­u­at­ing based on pre­dic­tive accu­racy, sci­ence-s uses a plau­si­bil­ity fil­ter, a func­tion which deter­mines whether a given the­ory pro­duces “accept­able” out­comes for all setups within a uni­verse. We assume the plau­si­bil­ity fil­ter is already given to us, and that we can use the fil­ter to judge the plau­si­bil­ity of a the­ory even on hypo­thet­i­cal scenarios.

The cri­te­rion of “accept­abil­ity” is more lenient than that of pre­dic­tive accu­racy. In fact, there can be infi­nitely many the­o­ries which are deemed plau­si­ble. Science-s focuses on cre­at­ing not just one plau­si­ble the­ory, but a cus­tomiz­able the­ory tem­plate which can pro­duce many plau­si­ble the­o­ries. Similar to sci­ence-u, this the­ory tem­plate is based on a model, albeit one with con­fig­urable para­me­ters. The pri­mary goal of sci­ence-s is to cre­ate a the­ory tem­plate which can gen­er­ate a plau­si­ble the­ory con­sis­tent with the observed out­comes of real-world setups from one envi­ron­ment; the sec­ondary goal is to make the mod­els as sim­ple as pos­si­ble. We can eval­u­ate the qual­ity of a the­ory tem­plate with the fol­low­ing procedure:

  • Choose some envi­ron­ment of inter­est. It should already con­tain a his­tory of past setups with their cor­re­spond­ing final outcomes.
  • Generate a the­ory from the the­ory tem­plate which fits as closely as pos­si­ble with the out­comes of past setups. In other words, use the his­tor­i­cal out­comes as “train­ing data” for a cus­tomized model.
  • Run the the­ory through the plau­si­bil­ity fil­ter to get a plau­si­bil­ity score.
  • Repeat the above process sev­eral times with many dif­fer­ent envi­ron­ments, and judge the qual­ity of the the­ory tem­plate by the plau­si­bil­ity scores & the close­ness to which the gen­er­ated the­o­ries fit the past setups.

In the case of profi­ci­ol­ogy, the envi­ron­ments are the lives of real peo­ple or fic­tional char­ac­ters, and the setups are the events that have hap­pened to them & their responses. A uni­verse may con­tain both actual events and fic­tional / coun­ter­fac­tual events, even though the lat­ter do not appear as “train­ing data”. Our the­o­ries are descrip­tions of how a char­ac­ter will respond to arbi­trary events, and we cre­ate a plau­si­bil­ity fil­ter by hav­ing judges eval­u­ate whether a the­ory describes the behav­ior of an intel­li­gent / sen­tient being, i.e. whether we think it is plau­si­ble that a sen­tient being will behave in the way described by the the­ory. We use a the­ory tem­plate to gen­er­ate cus­tom mod­els and fit the mod­els to a character’s known actions. We sub­ject these mod­els through a myr­iad of pos­si­bly fic­tional events to see how they respond, and we show these results to judges ask­ing whether the mod­els are behav­ing appro­pri­ately as sen­tient beings.

An Argument for Science-s Research

Despite its strong empha­sis on hypo­thet­i­cal setups and cus­tomized mod­els that are demon­stra­bly not objec­tive / uni­ver­sal, I believe sci­ence-s should be adopted as a legit­i­mate research method­ol­ogy because sci­ence-s research is a pro­duc­tive way of study­ing the full span of pos­si­bil­i­ties for poorly under­stood and highly var­ied phe­nom­ena such as sentience:

  • A sci­ence-p approach is overly prag­matic; focus­ing purely on solv­ing famil­iar prob­lem instances does lit­tle to reveal the broader pat­terns of the phe­nom­e­non of inter­est, and leaves us unpre­pared for future / less famil­iar issues.
  • A sci­ence-u approach is impos­si­ble if we can­not make repro­ducible obser­va­tions or pre­cisely engi­neer spe­cific setups. We would require in-depth under­stand­ing and con­trol over the fac­tors that are caus­ing the phe­nom­e­non of inter­est to vary.
  • A sci­ence-a approach tends to over­sim­plify and makes no attempt to explore vari­a­tions of the phe­nom­e­non beyond what is observed in real­ity. Without a the­ory tem­plate, we are forced to shoe­horn the full range of pos­si­bil­i­ties into a finite set of the­o­ries, and even if we legit­i­mately didn’t care about uncom­mon cases we would have trou­ble under­stand­ing non-sta­tic envi­ron­ments that slowly drift toward unex­plored possibilities.

Science-s rec­og­nizes that there are many vari­a­tions of the phe­nom­e­non we want to study, and gives us a frame­work to cre­ate cus­tomized the­o­ries cov­er­ing any vari­a­tion, even hypo­thet­i­cal ones that poten­tially could appear in a future envi­ron­ment. But of course, sci­ence-s has its lim­i­ta­tions too. It may be hard to find a good plau­si­bil­ity fil­ter for the phe­nom­e­non – judges may dis­agree in their assess­ment of plau­si­bil­ity, and over time peo­ple may develop dif­fer­ent opin­ions about the topic. In addi­tion, even if we can cre­ate highly con­sis­tent and plau­si­ble the­o­ries, we won’t nec­es­sar­ily be able to pre­dict the out­comes of future setups from the envi­ron­ment. This can hap­pen if the phe­nom­e­non con­tains so much vari­a­tion that we need an unre­al­is­tic amount of “train­ing data” in order to pre­dict future out­comes. But much like how sci­ence-a can be use­ful even if its pre­dic­tions can be unre­li­able, I believe that even with imper­fect plau­si­bil­ity fil­ters and an inabil­ity to make pre­dic­tions, sci­ence-s work can advance our under­stand­ing of real-world envi­ron­ments and the phe­nom­e­non of inter­est as a whole.

Proficiology takes a sci­ence-s approach to study the full span of sen­tient thought, includ­ing the cog­ni­tion of peo­ple who are dif­fer­ent from the major­ity in many ways. By using generic minds con­tain­ing expe­ri­ence foun­da­tions / lenses / agents to model sen­tient thought, we can pro­duce cus­tom mod­els that are con­sis­tent with a person’s past behav­ior and can respond plau­si­bly to future events. Studying this model can deepen our under­stand­ing of the person’s behav­ior, even if the model doesn’t accu­rately pre­dict their future behav­ior. Creating, eval­u­at­ing, and attempt­ing to sim­plify entire the­ory tem­plates helps us under­stand sen­tience as a whole, which is great con­sid­er­ing that we may be on the cusp of devel­op­ing arti­fi­cial sen­tience with AI technology.

Footnotes

  1. It is also com­mon in applied sci­ences, abnor­mal psy­chol­ogy, and deep learn­ing research.
  2. As for how our ances­tors dis­cov­ered this method in an era that pre­dates evi­dence-based med­i­cine, well peo­ple will try ran­dom things and find some truths (and a lot of myths) through trial and error.
  3. In fact most sci­ence-u work is physics, which to be fair is a very broad field of study. But I think there are exam­ples of non-physics sci­ence-u work in bio­chem­istry and the Earth sciences.
  4. It is also used in ecol­ogy, evo­lu­tion­ary biol­ogy, lin­guis­tics, soci­ol­ogy etc.
  5. Note that delib­er­ate under­fit­ting is unac­cept­able in sci­ence-u. A sci­ence-u the­ory is expected to be uni­ver­sal, and any excep­tion is a weak­ness of the theory.
  6. In essence the the­o­ries are:
    • There is only one uni­verse which is fun­da­men­tally built out of prob­a­bil­ity dis­tri­b­u­tions and has no objec­tive prop­er­ties (Copenhagen interpretation)
    • There is a vast mul­ti­verse of branch­ing pos­si­bil­i­ties of which we only observe one branch (Multiverse interpretation)
    • There is only one uni­verse which objec­tively con­tains both mat­ter and waves, where mat­ter moves by rid­ing on top of the waves (Pilot wave interpretation)

    Like the grav­ity exam­ple in the next sen­tence, the prob­a­bil­ity dis­tri­b­u­tions / mul­ti­verse / pilot waves don’t actu­ally exist as con­crete “things”; they are only used as parts of the model.

Leave a Reply

Your email address will not be published. Required fields are marked *