You found my old blog. Thanks for visiting! For my new writing, visit mikesententia.com.
From years of experience, I know how to test new models. But I’ve never expressed it in words. Here’s my first attempt.
Ananael left some great comments on my last post, which made me realized I hadn’t fully developed my ideas on testing magick. It’s not a simple matter of explaining myself poorly. It’s that I hadn’t ever explained my intuitions, even to myself.
So I spent this week developing those intuitions into something I can explain. It’s hard, but that’s how you grow as a thinker, and it’s one of the best parts of blogging.
In most posts, I’m explaining, teaching, and giving exercises. Here, I’m figuring out new ideas, and they aren’t quite a cohesive story yet. So think of each section as its own post: A separate idea about the common theme. The order isn’t terribly important, so feel free to skip around.
Summary
When you evaluate a new model (of anything: magick, physics, biology, whatever), you want to evaluate that model’s predictions. Some models will predict a refinement to a procedure right away. If that’s the model you’ve got, then A/B testing* makes sense.
*A/B testing = Comparing two procedures to determine which has a better outcome. Most useful when the difference is not obvious — a 10% improvement, rather than 10x better.
But many models — most of them, in my experience — don’t predict refinements right away. Instead, they predict something interesting but useless, and only become useful when you develop them further or combine them with other models. If your standard is “Models must produce a refinement of a procedure I use,” you’ll discard a lot of accurate models before they ever grow into usefulness.
Developing, Not Convincing
My focus here is on developing a model of magick, not convincing others that it’s useful. If you want to show someone that something’s useful, you’d better compare it to their current solution on some problem they care about solving. That’s A/B testing. But those results come from mature models. When you’re developing the model that will eventually produce that better technique, A/B testing is premature. This post is about explaining out why.
Also premature: Trying to convince people that a model is useful before it makes useful predictions. So, in the larger series about my model of magick, I’ll also show you useful techniques that either have, or clearly would, pass A/B testing. (I’ll clarify “clearly would pass” later in this post).
1. The Catapult-Maker
My thinking started with this allegory:
Imagine you build catapults. That’s your job. You live in ancient Rome, and you’ve never seen a coil spring. Your model of the world is “Only flexible wood can throw rocks.” Then Mr. Spring comes along with a metal coil spring. He explains it to you, and you decide to test it.
Sure, it’s a bit contrived, but work with me here.
You have two options. One is to run A/B tests on flinging rocks into walls. (You’re a catapult-maker, so that’s your ultimate use). You’ll find that the metal spring isn’t as good, and discard it.
The other is to look at what each model predicts. Your wood-only model predicts the non-wood spring can’t throw a rock at all. Mr. Spring’s model predicts the spring will throw a rock a short distance. Both models are predicting what will happen if you load a rock into a spring, so there’s no need to run situations A and B, you just run that one situation and see which model’s prediction is accurate. Then you see the spring throw the rock, and you learn something about the world.
Is the spring useful on its own? Probably not, at least for knocking down castle walls. But maybe you can use it to make a spring-assisted catapult. Or maybe your machinist friend can use it to make a wristwatch, or a toaster, or a door that closes behind you. (Maybe not in ancient Rome, but you get the idea).
Often, a new model won’t immediately improve your solution to a problem. Especially if it’s a problem you’ve been solving for a long time, and your current solution is optimized with A/B tests. (That is, tests not based on a model, but tests by guessing at possible improvements, trying them, and refining the procedure better than your model could predict on its own).
2. Quantum Physics Doesn’t Optimize*
The double-slit experiment (in 1909) was one of quantum phisics’ first experiments. It goes roughly like this:
Procedure: Fire a single photon at a wall with two small slits. Use a photon detector on the other side of the wall to determine which path it took.
Quantum physics prediction: The single photon behaves like a wave, so it will cancel itself out, giving the same wave-pattern as a stream of photons would.
Newtonian physics prediction: Only one photon won’t cancel itself out, so there will be no wave pattern.
Don’t worry if the physics details didn’t make sense, you won’t need them.
There aren’t two situations. Just two predictions on one scenario. It’s an experiment, but not an A/B test. It didn’t optimize anything, and it didn’t produce anything directly useful. It’s 100 years later, and aside from physics tests, we have precisely zero machines that rely on shooting a single photons through two slits.
The early predictions of a model often cover essentially useless corner-cases. The reason no one had tested how single photons move through two slits is that it hadn’t impacted any useful problem.
Quantum physics didn’t optimize anything from Newtonian physics, as far as I know. Once it matured, it gave us new, useful things, like semiconductors for computer chips. But it didn’t do anything that lends itself to A/B testing, and certainly not in the first stages of the model.
If we insist on a procedural refinement before exploring a model, we’ll throw out anything complicated that takes time to develop into maturity.
*The title was probably inaccurate. I expect that quantum physics optimizes some things. But “Quantum physics sometimes optimizes, but not in these examples” isn’t very snappy. And it doesn’t affect my larger point.
3. Stages of Science
The classic model of science is: Observe, hypothesis, test. But that’s really a simplification.
Most of the time, I find this pattern: Observe, basic model, non-useful hypothesis, test (to verify the model), develop the model further, useful hypothesis (like a new technique), test.
That’s really what we saw in the quantum physics example: The first predictions of a model aren’t techniques, they’re just validations that you’re on the right track. The useful techniques that lend themselves to A/B testing don’t come until a model matures. That’s probably the source of my feeling that A/B testing isn’t the right tool when you’re initially making a model.
4. Controls in Medical Tests
What about control conditions in medical tests, with control and treatment groups? Aren’t those comparing two procedures?
Well, there are two conditions, but it’s not really what I think of as A/B testing. When I talk about A/B testing, I don’t simply mean any comparison of two conditions. I mean a situation where you’re genuinely unsure which procedure is better, so you test them to find out.
That’s not how medical trials work. Doctors don’t simply go out and try it. They don’t even take a good model, see what it predicts, and run the tests. There’s a 10 to 20-year process of animal testing, review boards, tests on other species of animals, then finally a series of tests on humans, culminating in a full RCT. If you ran A/B tests as you developed your model of medicine, you’d get thrown out of the profession. It’s the last step of developing a technique for mass consumption, not something you’d do early on.
Even if the thing you’re testing can’t hurt people, an RCT is a very slow test. It’s great for verifying a treatment, particularly when you’re getting ready to sell it on a large scale. I’d love to be at that point, but I’m not.
When you’re making your model, though, you need fast tests that are pretty good. Case studies and small trials, with reasonable effort put in to avoid coincidence and placebo. That’s really all one person can do.
5. Local vs Global Optimization
Those terms come from computer science, but they apply any time you’re trying to find the best answer.
Local optimization means small refinements. If you ride horses, then better horse shoes, a padded saddle, and maybe a rubber saddle for a better grip are local optimizations. You’re still using a horse.
Going from a horse to a car is a global optimization. It’s a new way of solving the problem, and you don’t get there by improving a horse.
Two things to notice so far:
- You can propose local optimizations without understanding how the underlying solution works. You could easily try several options for horse shoes without being a vet.
- No matter how much you improve your horse, you don’t get a car.
The first cars were awful. They’d stall, couldn’t brake well or handle rough terrain, and were wildly unsafe. They would have lost A/B tests with horses. In fact, they did: People kept using horses for decades, until the product matured*.
*Also until prices dropped. Which maybe suggests another thing models need to become popular: An easy entry point. I’ll write about that later.
If you insist on a successful A/B test at the start of a model, you’ll wind up with incremental improvements, not a new global optimization. Improved horses, not cars.
6. Downhill at First
Let’s shift to the specifics of modelling magick. Most styles of operate by sending instructions to a system (my term for the forces mages channel). When you do a ritual, a system turns those symbolic actions into changes in the world. You can think of the system as the implementer of those natural laws that mages reference.
Like using a calculator for arithmetic, systems make magick faster and easier. It insulates you from needing to understand the details of magick’s implementation. But that insulation keeps you from exploring. So most of my magick happens without systems*, even though that makes it harder. And my first goal isn’t to improve on the system’s results, it’s to understand the system’s procedure, even if the results are (at first) less good.
*Remember, “system” = the forces mages channel. My magick definitely uses procedures, patterns, and systems-in-the-normal-english-sense.
Imagine you’re learning to program computers. One of your exercises might be programming a web browser, just to learn how they work. You wouldn’t expect it to out-perform Firefox. That’s the wrong metric. But that learning will eventually lead to programs that solve problems in new, better ways.
That’s one reason why I don’t advocate abandoning systems until you’re already experienced with direct magick. Continue to use what you’re good at when you need reliable results, and only use direct magick when you’re exploring magick’s implementation. I expect most ritual mages to use direct magick for problems their ritual style doesn’t address, not as a replacement for rituals.
7. Solving New Problems
Most of the time, I don’t explore a new part of magick to simply improve a technique. I do it to solve a new problem, one that doesn’t have a solution yet.
Let me give you an example. When I first started learning to awaken and strengthen my mental muscles, I could wake them up, but they’d draw so much power from the rest of my mind that I’d get very tired for several hours to a day afterward. I’d nap in the afternoon, then go to sleep at 7PM, and couldn’t do any creative work on the days I was activating. We’ll call that the “Do Nothing” condition.
Then I developed a better model of the power flow, which lead to better techniques, with steps I never would have thought of without those better models. Using those techniques to activate the same amount of mental muscles, I’d get tired for a few minutes at most.
Minutes vs hours, with results that happen every time. You don’t need side-by-side tests to know which one works better. It’s a big enough effect that it’s obvious.
I’ve replicated these results with 2 other mages, so it’s not something unique to me. (I wasn’t just being mean and not teaching them the better technique. You need to practice the basic version before learning the advanced one).
If a technique produces an improvement that could be mistaken for “Do Nothing,” I consider that a failure. A 5% improvement, or anything you’d need a suite of tests and statistics to detect, just doesn’t cut it for me. I’ll have more examples in the upcoming series on my model of magick and the results that make me confident in it.
8. Achieving Confidence
Predicting one new technique wouldn’t give me a lot of confidence in a model as a whole. Maybe it just got lucky.
But this year’s models build on last year’s ones. So when I create a technique this year, it doesn’t just support this year’s models, it supports all the models those build on. Almost everything I talk about on this blog is a few years old, so each model has at least a few techniques supporting it.
Summary
Thanks for reading all that. It covered a lot of ideas on testing, in a lot of topics: The limits of A/B testing, how different types of testing work better for different points in modelling, and how to become reasonably confident even without control conditions. Not everything was 100% developed, but hopefully it sparked some new ideas for you. If it did, please share in the comments.
If you liked this post, consider visiting my current blog at mikesententia.com.Tags: Testing
From this article I can see that your definition of “A/B testing” is much narrower than I previously thought. As I see it, the quantum slit experiment is A/B because it compares one condition in which the photon passing through the slit has a detector before the slit with a second condition in which the detector is placed after the slit (which is how the experiment was actually set up). All medical trials are also A/B at each phase of experimentation, because drugs are tested in relation to control groups and other similar drugs in petrie dishes, animals, and finally humans. And so forth. This is probably the source of my confusion about your usage of the term, because with my scientific background that was what I thought you were talking about.
The thing is that then, according to your definition, almost none of the testing I can do on magical techniques can be referred to as “A/B” according to your definition.
Hey Ananael, A/B testing is a marketing term. It’s for when you have some ideas, plausible things that might work (so there is some level of model or expertise behind it), but you’re not sure which option is best, so you test it. You might set up two phone numbers, and put one in flyer A and one in flyer B, and see which one gets the most calls, and refine your ads that way. And that’s really what I mean by A/B testing: Where you have a decent reason to suspect that B might be a refinement, but you’re not entirely sure, so you try it and see. Now that we have the terms straightened out, it sounds like you and I are on the same page as far as developing and testing magick models, do you agree?
Well, that shows what I know about marketing – that is, absolutely nothing.
I think that we’re basically in agreement that magical techniques should be tested as scientifically as possible within the constraints imposed by the nature of spiritual and magical work, as far as generating hypotheses, developing and testing models, and so forth. I will say, though, that I’ve never really doubted your commitment to that basic methodology. Most of the time my disagreements have been related to very specific aspects of the ideas you’re putting forth, and as in this case I’m sure some of those have been related to differences in our terminology.
Others are related to differences in our experiences, and those are going to be harder to work out if you’re trying to put together a general model that is reasonably applicable to most practitioners. An example – it takes effort for you to turn on your “mental muscles?” That statement has be wondering whether we’ve been talking about the same thing all along, because my biggest initial problem with magick was just the opposite – getting them to shut off. I was a worrier as a kid, and whenever I worried about something it would happen. I had to get my magical abilities under control before I could stop being the unluckiest person ever.
Also, I would like to make one more comment regarding you not being interested in a 5% improvement in an existing technique. Sometimes even a small improvement can lead to a significant refinement of a model. For example, when Copernicus first came out with the heliocentric solar system, it was only a marginal improvement over the Ptolemaic Earth-centric model that it replaced. But it was the Copernican model that allowed Kepler to make his later refinements and solve the laws of planetary motion, which finally produced dramatically more successful results. I will agree, though, that a 5% improvement over the “doing nothing” category is not very useful, since traditional magical techniques work substantially better than that.
Good point about the small gains from Copernicus’s model. I’ll have to remember that for the future.
On turning off event-causing magick: This is something I’ve seen before. Typically, this type of magick is caused by a system connecting to your mind and reading your thoughts as instructions. (When I see this, I can trace a connection from the person to the system, issue commands to the system, etc). The system only requires a few mental muscles to be active, then it does most of the work, and the mage’s main task is learning to instruct is properly (and in your case, not instruct it accidentally). The mental muscles for working with a system are turned on in a lot of people, particularly if you’ve been working with systems as a kid.
Then there are other mental muscles that you need to do magick without a system. They control connections in a certain domain, like the body’s energy, or the mind, or connections to spirits. If you’re building a technique without a system, then controlling connections is a big part of it, and you’ll need those mental muscles. Those (and others like them) are the ones that most mages need to work to activate.
I haven’t explained enough in this comment to really explain mental muscles. That will probably take a post. But I’m glad you brought this up, because it’s not something I normally think about, and now that I am, it’s obvious this would be a big question for a lot of mages who mostly do rituals. I’ll be sure to talk about it in the series going through all the different components of my model.
I think I have to make sure I’ve got my mind around your definition of “system” before I can really comment on your assessment of my childhood experiences.
I will say that the “thought to reality” effect has nothing to do with anything that I personally would define as a system – that is, any sort of formal method (but again, not sure if that’s the definition you’re using). It’s more like what New Agers talk about in “The Secret” – I would just focus my attention on things and they would happen, and at that age most of what I kept my attention on was stuff I worried about. I still use this “intuitive” method when I need a spell and don’t have an opportunity to work the forms, but if I can I like using the forms better because they create bigger probability shifts.
The other thing that I find odd about straining to activate magical powers is that the form of energy work I’ve studied the most and personally found the most effective is Qigong, in which the key to making it work well is to keep my mind and body relaxed enough that the energy can flow freely. For me, tension clamps it down and gets in my way, so straining is profoundly counterproductive.
Eureka. 2 things I just realized.
First, my term “system” is throwing you off. Maybe I should pick a different term. I initially chose the term because I was thinking about “systems of magick,” meaning a formal set of procedures, and thinking about what drove them. And the thing that drove them kind of blended with the formal system in my thinking, so I wound up using the term “system” for it, and it wound up fitting fairly well so I kept it.
Think “computer system.” Or maybe “thought activated computer system.” So, the thing itself is systematic, and if you’re using it properly, you should be systematic about your approach. But it’s thought activated, so if you’re connected to it and don’t know what you’re doing, it will still respond.
How do you get it in the first place? Well, my best guess is that if a kid grows up with people who are connected to a system (even if the adult doesn’t realize it), the system will connect to the kid too. When one person is working with a system, I often see it connect to everyone else in the room too, so this is an educated guess, though it’s still a guess.
On straining to activate magickal powers: Agreed, if you are straining, you’re doing it wrong. I think there’s one item I haven’t explained enough yet that will make it all clear:
There are 2 things you do. First you awaken / activate your mental muscles. Then, once they’re awake, you do practical magick.
Imagine your body is atrophied, and you want to practice martial arts. The first step is to slowly use your muscles for simple things like walking, until they’re not atrophied anymore. That’s very tiring. But once it’s done, you can learn whatever you want, and it’s much smoother.
That’s what mental activation is about: Awakening your atrophied mental muscles. Sure, some mental muscles you’ve been using for years (like for communicating with the systems you use, and for controlling energy in your body, and maybe some other things). But others you haven’t used yet, because you haven’t explored that part of magick yet. (Again, just guessing, but everyone I’ve ever worked with who hadn’t consciously activated their mental muscles had at least some that weren’t awake yet). The difficulty and fatigue is from exercising through the atrophy.
If you’re getting that same level of fatigue in your daily practice (as opposed to while you’re activating new mental muscles), you’re doing something wrong. (Usually, it’s that you’re trying to use an atrophied mental muscle without activating it first).
A lot of this stuff, I’ve only explained in person, with a demonstration covering a lot of the gaps. So this conversation is really helpful for me. Thanks for the comments.
So would you say that in your terminology “system” means the same thing that chaos magicians describe as an “egregore?” If so, it seems to me that the latter is a fairly well-established term in modern esotericism, and since most people use “system” to instead describe a formal method or set of related techniques you might want to consider adopting it.
If that’s the case, though, I still have no idea how I would have been exposed to one as a child. Nobody I grew up with practiced magick, and while my great-grandmother was a member of the Brotherhood of Light so my family was never outright opposed to magical practice, she was the last real practitioner before me and she died years before I was born. Not only that, but the Brotherhood of Light magical system works very poorly for me. Golden Dawn-style Hermeticism and Thelema resonate much better, and as far as I know nobody in my family ever practiced either of those.
Here’s a thought, though. Could someone who grows up with a family that’s hostile to magical practice have a radically different experience connecting with the “tech” of magical work than I did? Peter Carroll talks about the “psychic censor” which he imagines as some hidden part of the mind that’s hostile to magical operations. That’s never made any sense to me, just like the idea of having to “force” your magical powers to turn on seems alien and well beyond the realm of my own experience. But maybe both come from something rooted in the conditioning that most children in our society receive surrounding this sort of work.
I think that Egrigores and Systems both started with the same observation: Mages channel forces, and we need a name for those forces. But I don’t like “egrigore” because the definitions and usages I see don’t really align to the actual forces I work with. You might be interested in this post:
https://magickofthought.com/2011/10/does-egrigore-equal-system/
After the confusion you and some other readers have had about “system,” I would like to find a better term. Any ideas?
On getting a system as a child: In that case, I don’t know how it would have happened. This is still an open question for me: How do people who don’t intend to practice magick get connected to systems? Because it happens a lot, mostly with psychics, and I’ve really only guessed on the mechanism.
After reading over that post, I don’t necessarily think that the term “egregore” has to be tied into the belief-powers-magick idea. Often the people who use it are coming from that perspective, but in those cases it’s connotational, not definitional. “System” is so general that in my opinion it lends itself more easily to all sorts of misunderstandings. “Egregore” may only represent 90% of what you’re trying to convey, but “system” represents what you’re trying to convey along with a million other things, from computer science concepts to political and economic schemas.
When drawing a distinction between external and internal magical forces I usually just talk about macrocosmic versus microcosmic forces. If I’m discussing particular entities, I talk about those entities as spirits or godforms or what have you. There’s magick you can do using your own internal “energy” and magick that you can do by harnessing that of other spirits and beings in addition to your own. And I guess I see all of those variables as hard to sum up into a single overarching term.