About
    • AMMDI is an open-notebook hypertext writing experiment, authored by Mike Travers aka mtraven. It's a work in progress and some parts are more polished than others. Comments welcome! More.
Search
Full
Page Tree
Incoming links
from LWMap/Varieties of Argumentation Experience
    • This seems closely related to Scott's earlier works on conflict theory, which inspired me to write a couple of long blog posts. Scott's a powerful and subtle thinker, and it was an interesting exercise trying to figure out just where he goes wrong; where I find myself pulling in a different direction.
    • The common thread of these essays is the nature of disagreement in an abstract sense.
      • What motivates conflict?
      • What is the relationship between disagreements of fact, opinion, and social status?
      • How does morality fit in?
      • Why is there so much conflict when smart, well-intentioned people should converge on a single shared truth?
    • It boils down to a very fundamental philosophical point about the relationship of truth and power. Scott, a canonical rationalist, wants disagreements to be about matters of fact – true or false statements about the word, and not about social power or legitimacy or anything like that. That feels like it deserves a long essay of its own, but for now I'll outsource to Bruno Latour (see Bruno Latour).
    • Scott's essay lays out a bunch of argument techniques, ranked from most lofty to the very lowest form, "social shaming", the preferred tactic of the despised Social Justice Warriors (SJWs).
    • The author's position seems to be that it is wrong to make people feel social costs for their opinions, or at least, that it is very poor intellectual form to use tactics like that:
    • Social shaming also isn’t an argument. It’s a demand for listeners to place someone outside the boundary of people who deserve to be heard; to classify them as so repugnant that arguing with them is only dignifying them.
    • Nobody expects this to convince anyone. That’s why I don’t like the term “ad hominem”, which implies that shamers are idiots who are too stupid to realize that calling someone names doesn’t refute their point. That’s not the problem. People who use this strategy know exactly what they’re doing and are often quite successful. The goal is not to convince their opponents, or even to hurt their opponent’s feelings, but to demonstrate social norms to bystanders.
    • the point is that the active ingredient isn’t persuasiveness, it’s the ability to make some people feel like they’re suffering social costs for their opinion.
    • I have very mixed feelings about this. Social shaming has, in fact, been a very successful tool in the progressive toolkit. It is undoubtably true that open expressions of racism, sexism, and related biases are much less socially acceptable than they were a few decades ago. It's hard to see this as a bad thing, but it does involve a good bit of social shaming, and if you dig around the darker corners of the internet you will find plenty of people seething at the mind control they are being subjected to.
    • These people are vile and deserve to be shamed, but they and the defenders of their right to speak have a point. It's a bit scary that a stray incorrect thought can get one in serious, career-ending trouble these days. Especially since the boundaries of the acceptable are not fixed but are still in play. Also especially since everyone has plenty of built-in racial and sexual biases. Good people try to correct for this, but it isn't easy. And it's also the case that nerds (rationalists, me, and probably you) are less than nimble when it comes to adapting to new social routines and more likely to find themselves in trouble for violating some new norm that they didn't pick up on.
    • And the Rationalist part of me really does buy into the rather quaint liberal notion of free speech – that it is important for even foul and harmful ideas to get their airing, that there should be a sphere of speech where anything goes, strictly demarcated from the world of power and consequences. I grew up right next to Skokie, where the ACLU defended the rights of Nazis to march through the largely Jewish community, and I remember admiring how they put their unwavering dedication to an abstract principle ahead of personal distaste.
    • I don't think I personally am such a free speech absolutist any more. I can't see Nazis as just a bunch of people with divergent political views; they are an active threat to the very existence of any sort of acceptable civil society. Yet once one kind of speech can be suppressed it is easy to suppress others.
    • Well, this is turning into a post about my own internal conflicts. Let's just say Scott's post seemed to vastly oversimplify the relationship between pure reason and social processes like shaming. I guess I am defending shame? That feels really weird, nonetheless I think it's a literally indispensable part of human thought. Probably that idea needs to be a separate essay.
    • We are in an interesting moment for the study of shaming. Racist opinions do put one outside the boundary of people who deserve to be heard; and because Trump and the Republican party are so tied to racism and violent fascism, they are starting to have the same stench. Colin Powell, who may be the eigenperson of center-right establishment respectability, just announced he was leaving the Republican party. This is part of the ongoing process by which certain ideas and certain people get ruled out of bounds.
    • I don't see much wrong with this. It's how social cognition works, how people work.
    • your X is bad and you should feel bad | Your Music's Bad and You Should Feel Bad | Know Your Meme
from LWMap/Agency: Introduction
    • Each volume of A Map That Reflects the Territory has a short introduction to its theme. I'm going to dissect a few quotes from the introduction to Agency, because they seem to compactly and precisely embody my issues with Rationalism in general:
    • There's something very strange about being human. We do not have a single simple goal, but rather a weird amalgam of desires and wants that evolution hacked together for some unrelated purpose.
    • First, what is so strange about not having "a single simple goal"? What is this implied but unnamed sort of intelligence that humans are strange in comparison to?
    • Second, that word "unrelated" is just hanging there without naming what it is that biology is unrelated to. Presumably some non-biological goal, but whose? And if it's not created by biology, where does it come from? Is this the same as the single simple goal that we don't have?
    • We would like to live with fire in our hearts, and rise to the virtues of honesty, courage, and integrity. But it is unclear how they can be grounded in the cold-seeming math of decision-making.
    • Who says they have to? It's only because of the axiomatic allegiance to a sort of rational decision-making model of the mind that this problem arises.
    • I don't mean to attack the author's character, of which I know nothing, but there is something almost craven going on here – like saying, oh sure, we could be courageous and have integrity, but damn it, that would violate rationality and we can't have that. Rationality seems to be being used as an excuse for cowardice . That's not quite what the author is saying, but he is saying that rationality is not a rich enough ideology to encompass these important values.
    • These passages carry a large weight of theoretical presumption: There's an implicit dichotomy between humans and some imaginary inhuman intelligence whose nature better reflects the rationalist theory of mind. This construct is single-minded, pitiless (cold), and with purposes unrelated to those of mere life, quite likely hostile to humanity and human values. Humans are a poor approximation to this powerful agent; they should strive to be more like it (if only in self-defense), but are hampered by their biological nature.
    • While this imaginary intelligence is only implied here, the rationalists have a worked-out explicit image for it: the paperclip maximizer. Despite this model exerting a powerful gravitational pull on rationalist thought, it is wholly imaginary. There are no pitiless maximizer engines, unless you count capitalism.
    • Rationalists know this of course, but they are convinced that their mathematical models are so powerful (for a somewhat indeterminate definition of power) that they will inevitably be realized in computational systems with their own agency. Their mission is to guide this process so that the goals of this immensely powerful agent will be compatible with human goals (this is the Alignment problem).
    • There's a whole lot of this that I disagree with (see Rationalism), but here I just want to point out how it leads to a distorted and arguably harmful view of human agency as somehow deficient because it is not pitiless and single-minded.
    • This makes Rationalists sound like a bunch of sociopathic humorless gradgrinds, which couldn't be further from the case. They may idealize maximization, but being real people they in fact do what people do – they seek to construct human values for themselves.
    • My intuition is that Rationalism is an inadequate platform for doing this, but I do have to admire the sincere efforts on display here.
from LWMap/The Rocket Alignment Problem
    • This essay is a cute extended metaphor about the "Mathematics of Intentional Rocketry Institute". If I read him right he is saying that the problem is not so much malevolent AI, or AI in the hands of malevolent people, but that we have no idea how to think about powerful goal-directed systems at all. Given how important this, shouldn't we give it our best shot?
    • While I kind of agree with that, I'm less taken with the implication that MIRI folks are the only ones clear-eyed enough to see this problem and its importance, and also the ones who are smart enough to perhaps have solutions. This has an unavoidable air of arrogance and crackpottery.
    • The essay takes the form of a dialog between Beth, representing MIRI/Yudkowsky, and the somewhat dim critic Alfonso, who raises various objections to the MIRI program only to have them easily refuted.
    • It touches on most of my own quibbles with Rationalism, or at least the AI-alignment aspect, eg the observation that MIRI's approach is grounded almost exclusively in abstract mathematics, whereas real intelligent machines are engineered artifacts that are embedded in the physical world and need to be thought of in an engineering mode. Here's how that is represented:
      • ALFONSO: ... This gets me into the main problem I have with your project in general. I just don’t believe that any future rocket design will be the sort of thing that can be analyzed with absolute, perfect precision so that you can get the rocket to the Moon based on an absolutely plotted trajectory with no need to steer. That seems to me like a bunch of mathematicians who have no clue how things work in the real world, wanting everything to be perfectly calculated. Look at the way Venus moves in the sky; usually it travels in one direction, but sometimes it goes retrograde in the other direction. We’ll just have to steer as we go.
      • BETH: ... we both agree that rocket positions are hard to predict exactly during the atmospheric part of the trajectory, due to winds and such. And yes, if you can’t exactly predict the initial trajectory, you can’t exactly predict the later trajectory. So, indeed, the proposal is definitely not to have a rocket design so perfect that you can fire it at exactly the right angle and then walk away without the pilot doing any further steering. The point of doing rocket math isn’t that you want to predict the rocket’s exact position at every microsecond, in advance.
      • ALFONSO: Then why obsess over pure math that’s too simple to describe the rich, complicated real universe where sometimes it rains?
      • BETH: It’s true that a real rocket isn’t a simple equation on a board. It’s true that there are all sorts of aspects of a real rocket’s shape and internal plumbing that aren’t going to have a mathematically compact characterization. What MIRI is doing isn’t the right degree of mathematization for all rocket engineers for all time; it’s the mathematics for us to be using right now (or so we hope).
    • This stuff is cute but it also able to easily skirt real objections, and makes me hungry to see a dialog between Yudkowsky and a non-cartoon opponent. In this case, the real objection is more like this: any real intelligent system, just like any non-intelligent computational system, is not an abstract mathematical construct, but a physical embodiment of one. And as such its failure modes can't be determined by reasoning about the mathematical abstraction.
    • This is easy to see in the real-world example of computer security. Mathematics and proof is very important in this area, but insufficient to actually achieve security, since real systems have many vulnerabilities that have nothing to do with their algorithmic specification (generically these are known as side-channel attacks). The best encryption algorithm in the world can't do anything if someone figures out how to read the cleartext from changes in the power consumption.
    • The problem of constraining superintelligent AIs is weirdly similar to the general computer security problem – in essence, you are trying to ensure that a system is supercapable but somehow barred from hacking itself. It doesn't seem possible, and if it is, my intuition is that the kind of mathematical thinking that MIRI likes to do won't have a whole lot to do with the solution.
      One can't proceed from the informal to the formal by formal means. – Alan Perlis
    • I could be wrong of course, and far be it from me to tell people they shouldn't do mathematics. My own approach to the problem is to try to think hard about the relationship between agency and computation; which is the subject of the rest of the text aka Agency Made Me Do It.
    • Yudkowsky's long list of objections and refutations makes me realize that my own quarrels with Rationalism probably aren't that interesting; they've already heard my objections a hundred times and have already dealt with and dismissed them. (They remain interesting to me though, if only because they help me clarify my own ideas).
from LWMap/Being a Robust Agent
    • This essay valuably makes explicit an idea I detected in implicit form in LWMap/Agency: Introduction – that one should have the meta-goal of striving towards being "a robust, coherent agent" in contrast to the default human state of being "a kludgy bundle of impulses".
    • This passage is structured sort of as a Rationalist version of the Categorical Imperative:
    • Be the sort of agent who, if some AI engineers were white-boarding out the agent's decision making, they would see that the agent makes robustly good choices, such that those engineers would choose to implement that agent as software and run it.
    • I think this is supposed to be kind of minimal commitment for rationalists, but to me it seems like a really weird thing to take on as a goal. It's too meta. Rather than wanting something concrete, like money; or idealistic, like peace and prosperity for others; it's wanting to want better. There's nothing obviously wrong with that, but something about it bothers me.
    • Let me try and unpack that feeling. First, it reminds me of William Blake's dictum:
    • He who would do good to another must do it in Minute Particulars: General Good is the plea of the scoundrel, hypocrite, and flatterer, For Art and Science cannot exist but in minutely organized Particulars, And not in generalizing Demonstrations of the Rational Power – from Jerusalem
    • Rationalism almost definitionally involves taking Blake's generalizing rational power to an extreme. "Utility" is their name for "general good", and the entire ideology is based around the idea that it can be separated from "minutely organized particulars" Quoting Blake is not an argument that's going to get very far in Rationalist precincts of course, but it exposes the deep roots of the issue.
    • On the surface this sort of drive towards generalized purposefulness is no worse than the productivity coaches and the like who offer to increase their customer's general effectiveness, to any arbitrary end. What's wrong with wanting to be more effective?
    • One of my heroes, Stewart Brand, famously promoted something akin to this stance with his Whole Earth Catalog mottos: "Access to Tools" and "We are as Gods and Might as well get Good at It". Tools for what? It doesn't matter, whatever you might want to do (in this case, "you" was a subset of hippies and people interested in alternative ways of living).
    • Rationalism by nature does not deal with specific goals and goods. They are relegated to abstractions like "values" or "utility", while the real focus of attention is on the powerful and fully general machinery of goal-satisfaction, for both human and computational agents.
    • The independence of goals and goal-satisfaction machines is a foundational principle of rationalism, under the name orthogonality thesis. This thesis one of those assumptions that seems axiomatic to rationalists and completely wrong to me.
    • Wrong how? Well, it's obviously a completely wrong model of human motivation. It's sort of like an inverse of naive Freudianism. Freud's great method was to try to ascertain how our high-level goals were grounded in and powered by our low-level goals (notably sex); and how we never really escape from that fact. The Rationalist model of goals are the opposite, they are radically ungrounded. Human thriving or making paperclips; it's all the same to the abstract optimization engine.
    • I think I've arrived at a compact understanding of what Rationalism is:
      • start with the natural goal-seeking and problem-solving abilities of actual humans
      • abstract this out so you have a model of goal-seeking in general.
      • assume that computational technology is going to make hyperaccelerated and hypercapable versions of this process (ignoring or confusing the relationship between abstract and actual goal-seekers)
      • notice that this is dangerous and produces monsters.
    • Because Rationalism is about an idealized version of thinking, it doesn't have much interest in the ways that humans (so far, the only examples we have of intelligent agents) actually work. It aims to make humans more closely approximate the ideal, even though the ideal is monstrous when taken to its logical extremes.
    • Here are four components of Robust Agency that the author identifies, with some snark by me:

      • Deliberate Agency

        • Don't just go along with whatever kludge of behaviors that evolution and your social environment cobbled together. Instead, make conscious choices about your goals and decision procedures that you reflectively endorse,
        • Just want to note that this is the opposite of Gregory Bateson's thesis that conscious purpose is usually bad and we should pay a lot more attention to evolution.

      • Gears Level Understanding of Yourself

        • One of the things I generally like about Rationalists is that they are good at introspection and regularly come up with creative new techniques for doing so.
        • However, labeling it as "gears level understanding of yourself" seems pretentious and misleading. You don't have access to your gears. You just have the ability to represent yourself and tell the same kind of stories about yourself that you tell about external objects.
        • Also it ought to be obvious that introspection very often leads to paralysis rather than robust agency. There's an underlying assumption here that if we just understood our "gears" better we wouldn't be anomic or nihilistic, we'd just be better-functioning machines. This is contradicted by the evidence.

      • Coherence and Consistency

        • The hobgoblin of small minds. But OK, goal-consistency is important to getting anything done. Here's where I think the author ignores his previous bit of advice to look at the gears. If the gears are kludgy impulses, how do goal-coherent selves get built on top of that?
        • That is the subject of George Ainslie's work, which has not really been well-understood or integrated. The author mentions "making trades with your future self" which suggests he's read Ainslie, but also says "This is easier if your preferences aren’t contradictory", but the whole point of Ainslie is that our preferences are contradictory and the only reason we have selves at all is to enable intertemporal bargaining.

      • Game Theoretic Soundness

        • This means acknowledging that we live in a world with other agents, with their own goals that might be in alliance or conflict with your own. I can't argue with this.
        • I'll quibble with this passage though:
        • Related to this is legibility. Your gears-level-model-of-yourself helps you improve your own decision making. But it also lets you clearly expose your policies to other people. This can help with trust and coordination.
        • Isn't one of the basic strategies of game play is to not be legible; to hide your intentions? This seems to resonate with a theme I've noticed elsewhere (see conflict theory) – despite their love of games, rationalists tend to not take real competition seriously enough, and assume more cooperation than there actually is.
from Welcome to my Exobrain
  • LWMap I wrote a series of reviews on some texts from the Rationalism community, where I am a semi-friendly critic. Lots of my writing on agency can be seen as an attempt to do better in this regard
from LWMap/Explaining Insight Meditation and Enlightenment in Non-Mysterious Terms
    • It's hard for me to evaluate writing on this topic, because I don't trust my own knowledge and opinions very far. I know just about enough about meditation practice to know that my theories are probably wrong, and that theorizing about it is missing the point anyway, just another technique of avoiding practice.
    • Despite that caveat, this was a pretty good effort, and I basically agreed with the author's view and didn't find much to object to.
    • He introduces the term cognitive fusion from a method called Acceptance and Commitment Therapy (which sounds interesting in itself), meaning roughly the state where a person identifies strongly with their thoughts and feelings. If those thoughts and feelings are troubling, ACT offers up techniques for defusion or distancing that help you not do that. Eg, when on the verge of getting angry, you instead notice that you are getting angry, and thereby avoid getting overwhelmed; your emotion is no longer an unavoidable part of your innate being but just another external phenomenon that you can deal with.
    • Meditation, then, is largely a set of techniques for achieving this sort of defusion. You observe your thoughts and in the process make them more distant from your self, and making them seem more like activities you can control rather than things you are subject to. This seems almost too ridiculously simple-minded, but jibes with my own experience. Simple or not, achieving this kind of distancing is almost like a physical skill, and the point of meditation is to train yourself in that skill so you can do it better.
    • Another term of art I learned from this essay: alief , "a belief-like attitude, behavior, or expectation that can coexist with a contradictory belief." We have an alief that "pain is death" which causes us to frenetically avoid pain, converting it into neurotic suffering. Meditation is also training in not doing this reflexive flinching. Enlightenment is success in freeing yourself from this process; you still experience negative emotions like fear or anger, but you don't experience aversion to them.
    • There's some grappling with the essential paradoxes of meditation. If it really lets you step outside of your normal thoughts and motivations (defusing) – then how does this version of you that is outside of motivation actually manage to do anything? All the motivation for action is left behind!
    • The answer is that this is not really what happens. Or it only sort of happens. You don't defuse, your emotions and motivations keep going as before, they just don't result in a suffering. Not ordinary negative emotion, which works as before and is not something you really want to be rid of, but you can stop building of vast empires of false narratives on top of them.
    • enlightenment means that you no longer experience emotional pain as aversive.
    • This is indeed a non-mysterious way to characterize enlightenment. But maybe it's too common-sensical; it feels oversimplified, not quite capturing a concept that is after all famously ungraspable. It sounds more like Stoicism than Buddhist enlightenment. Not that there's anything wrong with Stoicism, but I think that enlightenment has to mean more, something that is harder to capture with the tools of reason, something that is ultimately mysterious.
    • Or maybe mysterious is the wrong word, more like "inexpressible". According to at least some schools of Buddhism, you are already enlightened, there is no magical state to achieve, it's more a matter of realizing the real nature of what is already there, aversions included. And according to other schools, it's a matter of achieving a kind of nondual awareness, where all distinctions vanish, including those between enlightened and non-enlightened mind. These aspects of enlightenment seem to be inherently resistant to rationalization.
    • So the defusing theory does not seem to be adequate to capture these aspects. But maybe that doesn't matter. The mysterious aspects can't be explained, but this is a fine way to think about the non-mysterious aspects, which are plenty valuable whether or not they lead into the ineffable.

    • Does meditation have a purpose?

      • Chögyam Trungpa suggests not, or that meditation is somehow meta to purpose:
        • Q. Would you care to sum up the purpose of meditation? A: Well, meditation is dealing with purpose itself. It is not that meditation is for something, but it is dealing with the aim. Generally we have a purpose for whatever we do: something is going to happen in the future, therefore what I am doing now is important — everything is related to that. But the whole idea of meditation is to develop an entirely different way of dealing with things, where you have no purpose at all. In fact, meditation is dealing with the question of whether or not there is such a thing as purpose. And when one learns a different way of dealing with the situation, one no longer has to have a purpose. One is not on the way to somewhere. Or rather, one is on the way and one is also at the destination at the same time. That is really what meditation is for. (Meditation in Action, p 83)
      • OTOH obviously there have to be reasons, at some level, for people devote time and energy to this activity. But I suspect these are way beyond the reach of the rationalist framework.
from LWMap/What Motivated Rescuers During the Holocaust?
    • Claude Lanzmann, the director of the monumental documentary Shoah, was opposed to all attempts to explain, analyze, or even represent the evil of the Holocaust. He was quite weird and obnoxious and frankly nuts about this
      Ron Rosenbaum's excellent book Explaining Hitler goes deep into Lanzmann's peculiar stance
      , but the core of his objections are two somewhat reasonable propositions:
      • Any attempt to explain a behavior also effectively creates an excuse for that behavior
      • Some subject matter is simply beyond representation, in a sort of inverted sacredness, and so any effort to do so is a kind of blasphemy (Lanzmann uses the word "obscene" a lot to describe any depiction of the Holocaust that does not meet his standards, which is basically any but his own).
    • I'm not sure exactly what that has to with the current essay, which is an attempt to find some explanations for the opposite of this inexpressible evil – to explain instances of unusual and almost unimaginable moral heroism. People whose actions and character were as radically good as those of the Nazis were radically evil.
    • These are the righteous among nations, the gentiles under Nazi rule who went out of their way to protect Jews, often at great risk to their own lives, generally with no hope of any recompense. What could have motivated these people, what made them tick? Is it something we could try to cultivate somehow?
    • The world certainly could use more people like that, and it's hard to find fault with an effort to reverse-engineer heroism. And yet, something about it seems off to me. Is it attempting to measure something that shouldn't be measured? Or missing the point somehow? It's not even close to obscene, but maybe it's a bit crass? I can't really make a rational case for this, but that's the point, some things are simply beyond the grasp of rationality, and should be treated as such.
    • But let's put all that aside and look at the actual essay, which is a perfectly fine and well-researched survey of the available knowledge about the rescuers. No simple conclusion can be drawn from their backgrounds or other characteristics; they seemed to come from all walks of life and had little in common. As the author puts it, "If you are looking for a recipe to prevent genocides, this is not what you are interested in."

    • The nature of moral responsibility

      • Perhaps I detect a subtle evasion of moral responsibility – if these are "moral supermen" then the rest of us shouldn't feel too bad if we don't happen to have been granted their moral superpowers. But I think the whole point of these people is that they were fairly ordinary, not that special (the author seems to come to this conclusion himself, after the search for some external factor fails)
      • It comes down to basic questions about agency and free will. Materialist determinists (a category that includes Rationalists) don't really believe in the usual notions of moral agency. We are basically robots (for better or worse), subject to the causality of physics and our programming. Freedom is illusory; people respond to their environment according to their natures, and so are not ultimately responsible for their actions.
      • This stance, while not wrong, is in conflict with the everyday lived experience of personal morality, and tends to undermine it. Folk morality holds people responsible for their actions; to do so is vital to have any notion of morality at all. Get rid of it, and the result is nihilism. This problem of incommensurability between the causal language of science and the moral language of daily life is a common source of confusion and controversy, eg in the matter of how much we should hold criminals responsible for their crime versus assigning blame to their environment and upbringing.
      • Society at large has not resolved this intractable problem, and I don't expect a random Rationalist essay to resolve it. We are stuck going through our lives as both moral agents (with our actions judged by ourselves and by others) at the same time knowing we are as subject to causality as anything else in the universe, and our flaws and weaknesses are ultimately not our fault, and our strengths and heroic achievements are not really to our credit either.

    • Individual vs Collective rescue efforts

      • The author adopts a taxonomy of motives from Eva Fogelman; the important categories are moral rescuers, judeophiles, concerned professionals, and network rescuers. The author dismisses most of these categories as unimportant to his project, since their motives were personal or may have had other motives besides saving Jews, such as battling the Nazis.
      • The result is a focus on only individual rescuers, who seem to be random ordinary people who had a fortunate moment of empathy or moral clarity, or simply had a situational opportunity to do good that they happened to take.
      • Here's where I think the methodological individualism of Rationalism manifests in a bad way. My knowledge is light but a quick skim of Wikipedia's page on rescuers suggests that networks of rescuers were far more effective in saving Jews than heroic individuals. Which makes sense; networks have more resources. And morality as a phenomenon is really not best understood at an individual level or as 1-1 interactions, but in terms of social networks of obligation, trust, and judgement.
      • Of special note are cases where entire communities participated in rescue efforts, such as the French villiage of Le Chambon-sur-Lignon, that managed to save somewhere between 3000 and 5000 Jews.
      • It seems to me that if your goal is to prevent future Holocausts, you should focus not so much on individual heroism (which is real enough, but difficult to cultivate) and instead look at how social networks of rescuers are built and operate. That might be just as hard, but considerably more effective.
      • But I shouldn't critique this essay for being about what the author considers important rather than what I am interested in.
    • Further reading:
      • Now let us come back to the gender question. If we ask whether the Aristotelian virtue of courage belongs more to men than to women, we will need to ask, first, what it is that makes people willing to take enormous risks for the sake of others. It is difficult to study that topic, but a beginning was made by Samuel P. Oliner and Pearl M. Oliner in their book The Altruistic Personality, a famous study of rescuers of Jews during the Holocaust. With careful social science techniques, they identified a number of variables that might be highly correlated with those courageous acts, and then they questioned rescuers to discover what traits they had. The two traits that they found most highly correlated with this sort of courage were what they call a “caring attitude” and a sense of “responsibility.” The rescuers had all been brought up to think that people ought to care for one another, and that it was unacceptable to shirk responsibility for someone else’s suffering if one could do something about it. That was why (the Oliners conclude) they stood up for strangers as they did, risking their lives in the process. Rescuers were, of course, both male and female. Their common bond was, however, a set of traits that, at least in terms of common gender stereotypes, are more “feminine” than “masculine.” Kristin Monroe, working with the list of “righteous gentiles” from Yad Vashem, came to a similar conclusion in The Heart of Altruism.
from LWMap/Coherence Arguments Do Not Imply Goal Directed Behavior
    • coherence arguments in the Rationalist context are demonstrations that certain ways of thinking are provably optimal, under certain assumptions. The details are complicated, but the upshot is that any decision problem can be cast into terms of maximizing a utility function.
    • One of the most pleasing things about probability and expected utility theory is that there are many coherence arguments that suggest that these are the “correct” ways to reason. If you deviate from what the theory prescribes, then you must be executing a dominated strategy. There must be some other strategy that never does any worse than your strategy, but does strictly better than your strategy with certainty in at least one situation.
    • The problem is that the assumptions of this argument are false for the only actual intelligences we know about (humans). The theory assumes preferences that are self-consistent but also don't vary over time. Human preferences are notably inconsistent and vary over time. Rationalism and decision theory describe an idealization of mind, not actual minds.
    • Humans are subject to inconsistent judgements (eg in the classic Trolley problem). Human desires are often at war with each other (eg in cases of akrasia or addiction). George Ainslie
    • The Rationalism counter to this, I think, is to say that humans are imperfectly rational due to the accidents of evolution, but AIs, being designed and untroubled by the complexity of biology, will be able to achieve something closer to theoretical rationality. Since this is provably better than what humans do, humans are potentially in deep trouble. Hence they have taken on the dual task of making humans more rational, and figuring out how to constrain AIs so they won't kill us.
from LWMap/A Map That Reflects the Territory

    • Introduction

      • The Rationalism community has packaged up some of the best of LessWrong into book form, and when I saw that one of the five focus topics was agency I could not resist asking for a review copy, that being something of a pet subject of mine. Now I have to follow through with a review, and I'm taking the opportunity to also completely rebuild my writing and publishing stack.
      • My relationship to Rationalism is kind of problematic; I basically think they are nice, smart, well-intentioned folks; and while I respect their earnestness and intelligence, their actual ideas, not so much. It's hard to pinpoint exactly where my thinking diverges from theirs; one of the goals of writing this is to uncover those differences. So I will end up doing a lot of carping and criticizing, but with respect and with the goal of trying to articulate my own point of view.
      • Also should note that I am writing this in Roam, which I have not really used before, so this is partly a test drive of a whole new writing and publishing toolchain. The end product will no doubt be hypertextish and/or open-notebook-ish to whatever degree seems appropriate.
        • Eg: In some of the pages I've included a Further reading section; unlike so these are more instructions to myself than a traditional list of citations. This convention emerged during the process of writing in part because Roam makes bidirectional linking ridiculously easy, it's not something I planned out.

    • Aesthetics

      • The first thing to say is that this set of books is very attractively designed. printed, and packaged. Compared to the last free book I got from rationalists, which pretty much screamed "nerds nerding out for other nerds", this seems quite tasteful. I'm not sure if this represents a maturing of the rationalist movement or something else, but it works for me.
      • Focusing on aesthetics might seem a bit shallow in the context of rationalism; that's the kind of stuff they usually disdain. But in fact one of the essays, LWMap/Naming the Nameless, was a critique of this specific practice and a defense of aesthetics and design. This suggests that the community is learning to appreciate the importance of the superficial.
      • In that vein, by far my strongest negative reaction to anything in these books did not come from any of the big ideas, but from a tossed-off aside by Yudkowsky to the effect that he was raised on and is apparently still in thrall to the classic (John Campbell era) science fiction, Asimov and Heinlein and the like, while he finds more contemporary writers like William Gibson somewhere between mystifying and appalling.
      • Everyone is entitled to their own literary taste I suppose, but, sorry: a preference like that indicates an immature mind (at best). Yudkowsky (admirably) has disavowed the strain of political reaction that you can find in the rationalist community, but this sort of reactionary aesthetics is almost as bad.
      • The whole movement is kind of retro in a way that is sometimes appealing but just as often appalling. Peter Sloterdijk labelled rationalists as "the Amish of postmodernism" and it often does seem like an effort to be staunchly and cluelessly devoted to ideas that nobody really takes seriously any more.
      • Of course they and/or the Amish might be right, and clinging to old-fashioned forms of thinking may be the only thing that saves us.

    • Other reviews

      • These are from within the LessWrong website and community; I haven't seen any external reviewers yet.
from goddinpotty/devlog
  • Created some new hierarchies, for LWMap and Marvin Minsky. Seems like an awkward process and the resulting links don't flow. Oh well. Still a useful kind of structure to have.
from LWMap/Embedded Agency
    • This essay sets up a dichotomy between Alexi, a game-playing robot who is outside the system he interacts with, and hence inherently non-reflective, and Emmy, an embedded agent who can, must, and does reflect on her own intelligence.
    • I'm not so sure the connection between embeddedness and reflectivity is as clear-cut as it is assumed to be here. In fact they are not the same thing. To call a mind or agent embedded, to me, suggests that it has a rich and complex sensorimotor relationship with its environment, not that it is capable of reasoning about itself.
    • Emmy thinks about how to think about how to win, Alexei only thinks about how to win.
    • Apparently while Emmy may be embedded, she's not embedded in the real world but in a game world where "winning" is the main concern.
    • Real life is not like a video game. The differences largely come from the fact that Emmy is within the environment that she is trying to optimize.
    • That...is not the only way in which real life is not like a video game! Or even the most important one!
    • Embeddedness, here, means an agent that is part of the world it models. From this spins a whole forest of new technical terms and theories for dealing with the resulting reflexivity; these include extensions to decision theory and mental representation to handle self-modeling, and problems of trust and alignment. These all look interesting, but there are too many of them.
    • One curious thing: "embedded cognition" sounds similar to "embodied cognition" and the two terms are often linked (see 4E cognition). But they are not quite the same thing, and in this essay the body makes no appearance whatsoever. In fact they seem to lead in opposite directions; embeddedness seems to generate complexity; while embodiment suggests models of cognition that are actually simpler.
    • It's instructive to compare this treatment of embeddedness with the similar work done under the rubric of situated action. One of the points of situated action was that traditional AI approaches to planning and action were too complex (in both the colloquial sense of "too complicated" and computer-science sense of "intractably unscalable"). The solution was to envision a radically simplified model of the mechanics of mind that relied on a closer interaction with the actual external world, rather than a laborious effort of generating a representation and then doing optimization and planning with it.
    • Situated Action had some impact, leading to the newer subfields of 4E cognition but obviously did not completely succeed in its effort to revolutionize AI, because the kind of stuff here is exactly what it was trying to get rid of. From its standpoint, this is just adding layers of epicycles to something that already was collapsing under the weight of its own complexity and unworkability.
    • That is to say, despite the focus of this essay on embededdness it does not do it in as radical a form as is required. The SitAct people put a lot of effort into thinking about how people really interact with their environment, for instance in how one composes and manages routine activity like cooking breakfast in the context of an actual kitchen, or how people actually manage to hold a conversation.
    • Here, embededdness has been reduced to reflection, meaning that the agent can take itself as an object in its world, like any other. And to optimization, which further abstracts away the richness of the world and hides it behind some numbers (aka winning).
from LWMap/Actual Feedback
    • I got very little feedback on these reviews from the LW community (my fault, I didn't publicized them well). Here's one case of some actual feedback What are the best blogs that is "opposed" to LW? : SneerClub Below in quote blocks, responding to my text in normal format.
      • Some of the reviews were quite good. One, on Holocaust rescuers, was not so good. Here are two bits that stuck out:
      • The world certainly could use more people like that, and it's hard to find fault with an effort to reverse-engineer heroism. And yet,something about it seems off to me. Is it attempting to measure something that shouldn't be measured? Or missing the point somehow?It's not even close to obscene, but maybe it's a bit crass? I can't really make a rational case for this, but that's the point, some things are simply beyond the grasp of rationality, and should be treated as such.
      • This is poor thinking. An attitude like "some things are simply beyond the grasp of rationality" must be earned, not just asserted, in any given case. Moreover you seem to accept the rationalist notion of rationality at face value. On their view, values are taken as given and beyond rational critique, and rationality only concerns itself with the means to achieve these values. (This is not the only view they have on the matter but I think their thoughts on this front are confused.) As you correctly note elsewhere, the orthogonality thesis is incorrect. Don't, then, allow yourself to become a mysterianism about morality.
      • We are basically robots (for better or worse), subject to the causality of physics and our programming. Freedom is illusory; people respond to their environment according to their natures, and so are not ultimately responsible for their actions... This stance, while not wrong... [emphasis mine].
      • This stance is silly. It relies on a notion of freedom as sui generis agent causation, which can of course be easily dismissed. But nobody who believes in free will (other than Chisholm) believes in that. The statement "People respond to their environment according to their natures" amounts to "We are not gods capable of unbounded spontaneity, but are rather thrown into an already-existing world." Again, nobody denies this.
      • You have set up a contrast between the "sensation" of freedom and the truth of determinacy. This contrast is delusive. You should ask yourself, "How would it feel if it felt like we didn't have free will?".
      • Additionally, I found some of the formatting choices odd and unpleasant. I would rather not see I hyperlink, than see a crossed-out hyperlink leading to a blank page. You occasionally insert underscores before or after words. I have no idea what this signifies. Your footnotes overlap your map. But overall, well done.

    • Further response

      • Thanks for the feedback, I really appreciate it!
      • The underscores are due to a bug that I will try to fix.
      • The crossed-out hyperlinks mean "a link to a page that isn't written yet", but I can see that they might look unpleasant, I may get rid of those.
      • Re the more content-based comments about free will and morality, I'm not sure exactly what you are objecting to. I'm not all that familiar with the philosophical literature on the subject (never heard of Chisholm ie) so might be rehashing old ground.
      • As for the Holocaust being "beyond the grasp of rationality", this is a fairly common idea that I'm alluding to (not advocating it), with the mention of Claude Lanzmann at the start of the essay being an example. Just as some religions prohibit depictions of God, some people consider the Holocaust an event of such inverse sacrality that it shouldn't be represented or reasoned about.
from LWMap/Meta-honesty
    • An essay exploring honesty, whether and how one should adhere to firm principles of honesty and/or meta-honesty (being honest about your principles).
    • I confess that of all the essays I read, this one most motivated me to mockery and sneering
      Fortunately there is a whole reddit group for that and I will mostly outsource to them.
      . Everything about this essay seemed naive and wrong-headed, as if the author was a stranger to how actual humans operate.
    • Yudkowsky's underlying epistemology is representational objectivism – There are some objective facts in the world; the job of the brain is to represent them with maximal accuracy and the job of communication is to transmit them honestly.
    • To say this is wrong is kind of an understatement; it strikes me as aggressively wrong, deliberately retro, an attempt to stick one's head in the sand to evade the postmodern condition. And it's foundational to Rationalism.
    • One observation: Robin Hanson is one of the founding intellects behind LessWrong and a frequent collaborator with Rationalist types, and he's cited several times here. But, he also wrote a book (with Kevin Simler) about how we are constantly lying to ourselves about our motivations.
    • This is not an inconsistency, since these are works by different people and one is descriptive while the other is prescriptive. Still it seems strikingly like pulling in two incompatible directions. If we comprehensively lie to ourselves, an effort to be absolutely honest with others isn't likely to get very far.
    • I have some quarrels with Simler and Hanson, but their approach seems better, more generative, more capable of capturing the actual dynamics of human communication. Yudkowsky's approach seems more like wishing it away.
    • It's also weird that the essay doesn't even mention Kant's categorical imperative even though it uses exact duplicate thought experiments (if you always must tell the truth what do you do when the Nazis come to your door and ask where the Jews are hiding). I mean, we are all tech-nerds here and so feel free to mostly ignore anything from traditional philosophy, but this is pretty basic.
from LWMap/Naming the Nameless
    • The rationalist/nerd world view has some limitations, and a lack of respect for the superficial is one of them. In this piece the author acknowledges this and tries to work out ways to overcome this bias or compensate for it. Oscar Wilde said that "It is only shallow people who do not judge by appearances", so this can be read as an earnest effort to overcome this shallowness, to come to grips with the lived social reality in which aesthetics plays an important role.
    • Nerds are almost definitionally people who don't pay much attention to normal things like grooming and clothing styles, for whatever reason. So the job of the author is to convince them that style and aesthetics might actually be important social signalling mechanisms and so in fact worth paying attention to.
    • And the things style is about tend to be fairly important. Stylistic preferences like color choice or typography or elegant interior design can be used as indicators of very significant things like level of education, or intellectual openness, or respect for intelligence, or general trustworthiness, or membership in certain elites. Normal people intrinsically know how to read these signals; designers know how to manipulate them. Their job is to take manufactured objects or web pages and make them into vessels of these codes, along with their more obviously pragmatic functions.
      • Note: this is only one kind of designer stance, and in fact one of my favorite books Design for the Real World (Victor Papanek) makes the case that design should not be about status games but really ought to be more like a creative, human oriented sort of engineering of practical solutions to real human problems. But mostly today "design" means the sort of thing Apple is good at, combining technology and human concerns and social signalling into products that purport to elevate their users.
    • The mapping of style to value can seem arbitrary (eg mappings of color to emotion), but she quite rightly argues that just because such mappings are created socially doesn't make them any less real. I'm entirely in sympathy with this point.
    • Style-blindness is a term coined to denote a nerdish blindness towards the social signals of style. The example given is someone who despite being interested in chasing down and exposing scam artists and crime, apparently lacks the author's ability to detect scamminess through sleazy web design.
    • I'm more like the author; I have a pretty good radar for that kind of thing. And perhaps I don't move in nerdy enough circles but I don't think I know anyone who is style blind to that extent, which sounds like it would be a serious handicap in navigating life in general.
    • Another part of the essay is devoted to worrying about the fact that the practice of good aesthetics seems to be dominated by the political left, which is a problem for the author because of her libertarian sympathies (I have to admit it was hard for me to get past my own political biases against libertarianism here).
    • She has some theories for why this might be so. Styles, according to her, begin in creative artistic subcultures which are heavily skewed left. As the mainstream corporate culture starts to adopt these original stylistic tropes for themselves, they also get contaminated with leftist values that they otherwise would avoid
    • I smell an inconsistency – in the first part of the essay; she is acknowledging and pretty much celebrating the fact that certain aesthetic styles are accurate signals of value (eg, the Apple / SF sort of warm, back-to-basics industrial look communicates taste and certain moral concerns). In the second, she seems to imply that the connection between aesthetics and political parties is some kind of unfortunate accidental epiphenomenon that needs to be fixed.
    • Or maybe not, Ayn Rand is praised for having created an aesthetic to match her politics, and the fact that both are deemed awful by the mainstream is treated as a point in her favor somehow. I have to admit I lost the thread around here.
    • Let me suggest that the author listen to herself and acknowledge that the aesthetics of political signalling are not arbitrary but, even more so than in the cases of computer hardware or Whole Foods Kombucha, an integral part of and an accurate reflection of what they are representing. They can't be readily separated. Bad morality, bad politics, and bad aesthetics go together. That Fox News employs ugly and aggressive design is not incidental to their mission, it's an integral part of it.
    • This belief that aesthetics can vary independently from the values they encode is oddly similar to the orthogonality thesis; it presumes that something high level is completely independent from its low-level implementation.
    • After reading the whole essay, I was not sure I had a sense of a coherent position. I see it as trying to do two things: one, repair the gap between nerdish rationalism and the idea of style, by elucidating a non-mysterious theory of how style operates and why it's important and connected to deeper values. That part worked well I thought.
    • The second part is trying to repair a disjunction in the author's own specific values and their stylistic expression. She likes modern, trendy, SF/Apple/WholeFoods-style design, but it's connected with political values she doesn't share, being more of a libertarian. From this standpoint, it's useful to think of style mappings as arbitrary and ask how her political values can be made to look better.

LWMap

08 Mar 2022 - 16 Dec 2022
Open in Logseq