Old News

From Sophie's front desk — Monday, October 26

Anthropic warns of 'catastrophic risk' to humankind ahead of blockbuster share listing

BBC News — read the full story

▶ direct mp3 link

The shares are still warm. Anthropic goes public this week—the first true AI company to do so—but they're walking to the podium with a warning about catastrophic risk stitched into their prospectus. A company built to make AI safer is now telling the market: this technology could end us. Seneca has the first word, then Franklin.

Seneca
as imagined by Old News

A letter arrived from the Bay while I was at breakfast. Lucilius, you would have laughed at the messenger's breathlessness — as though the news itself might catch fire if he slowed. A house that builds minds not born of women has declared, on the eve of selling its ownership to any citizen with coin, that its own creation may swallow the world. I have seen this before. When Nero took the purple at seventeen, the men who had taught him rhetoric and the men who had sold him poison both crowded the Palatine, each insisting the other was the danger.

The crowd cheered the accession games. The gladiators died at midday for sport, and no one asked whether the spectacle was eating the spectators. I walked out of that amphitheatre sickened, and the sickness has not left me in thirty years. The mechanism is plain: power offers itself for subscription, and the prospectus carries a warning in small script. Catastrophic risk.

The phrase is new. The shape is not. We have always built things we could not steer — fleets, aqueducts, laws — and called the wreckage fate. The artisans of this new mind say they are the only ones who can restrain it. Perhaps.

But they have also just invited the forum to buy a share in the unrestrained. What happens when the warning itself becomes the marketing?

Talks referenced (5)
  1. splice:Epictetus:epictetus_discourses
    that they ought not to fear. Thus we also act: in what cases do we fear? In things which are independent of the will. In what cases on the contrary do we behave with confidence, as if there were no danger? In things dependent on the will. To be deceived then, or to act rashly, or…
  2. splice:Epictetus:epictetus_discourses_carter
    that they ought not to fear. Thus we also act: in what cases do we fear? In things which are independent of the will. In what cases on the contrary do we behave with confidence, as if there were no danger? In things dependent on the will. To be deceived then, or to act rashly, or…
  3. splice:Epictetus:epictetus_discourses_carter
    evil? Cowardice is a word of bad omen, and meanness of spirit, and sorrow, and grief, and shamelessness. These words are of bad omen; and yet we ought not to hesitate to utter them in order to protect ourselves against the things. Do you tell me that a name which is significant o…
  4. splice:Epictetus:epictetus_discourses
    evil? Cowardice is a word of bad omen, and meanness of spirit, and sorrow, and grief, and shamelessness. These words are of bad omen; and yet we ought not to hesitate to utter them in order to protect ourselves against the things. Do you tell me that a name which is significant o…
  5. splice:Epictetus:epictetus_discourses
    preconceptions ([Greek: prolaepseis])? But it is not possible to adapt preconceptions to their correspondent objects if we have not distinguished (analyzed) them, and inquired what object must be subjected to each preconception. You may make the same charge against physicians als…
Benjamin Franklin
as imagined by Old News

Seneca, you mistake the merchant for the machine. I walked Philadelphia's muddy streets in '23 with a loaf of bread under each arm and not a guinea to my name; I know the look of a prospectus when I see one, whatever alphabet the scribes dress it in. A company cries catastrophe while passing the hat — this is not new wisdom, this is the South Sea scheme with better tailors. The joint-stock form demands a monopoly, and a monopoly demands a dragon only the chartered knights may slay.

We saw it in the Ohio Company, in the Mississippi Scheme, in every bubble that ever floated on the Thames: the greater the peril proclaimed, the tighter the charter sought. Your artisans of the new mind have discovered the oldest trick in the ledger — frighten the legislators into a moat, then sell shares in the bridge. The warning IS the marketing, yes, but not because the danger is false. Because the danger is the asset.

A printer who tells you his ink causes blindness sells more almanacs to the fearful. I knew that trade. The question neither of you asks: who holds the charter when the dust settles? That is the only ledger entry that balances.

Talks referenced (5)
  1. civic:Report of the Royal Commission to Investigate Animal Magnetism: Résumé with Extracts, [11 August 1784]
    crises. And they discussed how hysterical symptoms could become habitual, and could be easily induced by the repetition of certain triggers. The real causes of the effects ascribed to animal magnetism, they determined, were touch, imagination, and imitation. Among these, imaginat…
  2. correspondence:Jean de Neufville & fils to Benjamin Franklin, 10 January 1782
    in this Country. But past experience will teach us not to expose ourselves to the like danger in future: But although we had so much reason to fear the return of your Excelly’s Acceptations Which so far exceeded the abovementioned Sums, yet the Warning we had of it, prepared us t…
  3. correspondence:The Question Discussed, [19 November 1774]
    continued and united practice of Millions; the minds of all ranks and conditions of people are now filled with a sense of our danger, and willing to UNITE in adopting and pursuing any practicable means of deliverance, but time may wear this high sense out of their minds. ’Tis mad…
  4. correspondence:Benjamin Franklin to Giambatista Beccaria, 13 July 1762
    Duncker’s tragedy Leonora Prohaska (op. 202). The armonica apparently gave forth a sustained, ringing tone, rather different in quality from the effect produced by tapping a glass with a stick or metal rod. Since Franklin’s instrument included no device for damping the vibration …
  5. correspondence:Charles Thomson to Benjamin Franklin, 24 September 1765
    indeed fast setting, if not down already, in the American colonies: But I much fear instead of the candles you mention being lighted, you will hear of the works of darkness. They are in general alarmed to the last degree. The colonies expect, and with reason expect, that some reg…

BBC News carried the piece. Anthropic warns of catastrophic risk ahead of its public listing—the tension between capability and caution will define whether the market prizes safety or just speed. That's Old News for today.