add hazy-analysis, summarization results, streaming demo

recipes/experimental/long-context/H2O/data/PaulGrahamEssays/addiction.txt

@@ -0,0 +1,116 @@
+July 2010What hard liquor, cigarettes, heroin, and crack have in common is
+that they're all more concentrated forms of less addictive predecessors.
+Most if not all the things we describe as addictive are.  And the
+scary thing is, the process that created them is accelerating.We wouldn't want to stop it.  It's the same process that cures
+diseases: technological progress.  Technological progress means
+making things do more of what we want.  When the thing we want is
+something we want to want, we consider technological progress good.
+If some new technique makes solar cells x% more efficient, that
+seems strictly better.  When progress concentrates something we
+don't want to want—when it transforms opium into heroin—it seems
+bad.  But it's the same process at work.
+[1]No one doubts this process is accelerating, which means increasing
+numbers of things we like will be transformed into things we like
+too much.
+[2]As far as I know there's no word for something we like too much.
+The closest is the colloquial sense of "addictive." That usage has
+become increasingly common during my lifetime.  And it's clear why:
+there are an increasing number of things we need it for.  At the
+extreme end of the spectrum are crack and meth.  Food has been
+transformed by a combination of factory farming and innovations in
+food processing into something with way more immediate bang for the
+buck, and you can see the results in any town in America.  Checkers
+and solitaire have been replaced by World of Warcraft and FarmVille.
+TV has become much more engaging, and even so it can't compete with Facebook.The world is more addictive than it was 40 years ago.   And unless
+the forms of technological progress that produced these things are
+subject to different laws than technological progress in general,
+the world will get more addictive in the next 40 years than it did
+in the last 40.The next 40 years will bring us some wonderful things.  I don't
+mean to imply they're all to be avoided.  Alcohol is a dangerous
+drug, but I'd rather live in a world with wine than one without.
+Most people can coexist with alcohol; but you have to be careful.
+More things we like will mean more things we have to be careful
+about.Most people won't, unfortunately.  Which means that as the world
+becomes more addictive, the two senses in which one can live a
+normal life will be driven ever further apart.  One sense of "normal"
+is statistically normal: what everyone else does.  The other is the
+sense we mean when we talk about the normal operating range of a
+piece of machinery: what works best.These two senses are already quite far apart.  Already someone
+trying to live well would seem eccentrically abstemious in most of
+the US.  That phenomenon is only going to become more pronounced.
+You can probably take it as a rule of thumb from now on that if
+people don't think you're weird, you're living badly.Societies eventually develop antibodies to addictive new things.
+I've seen that happen with cigarettes.  When cigarettes first
+appeared, they spread the way an infectious disease spreads through
+a previously isolated population.  Smoking rapidly became a
+(statistically) normal thing.  There were ashtrays everywhere.  We
+had ashtrays in our house when I was a kid, even though neither of
+my parents smoked.  You had to for guests.As knowledge spread about the dangers of smoking, customs changed.
+In the last 20 years, smoking has been transformed from something
+that seemed totally normal into a rather seedy habit: from something
+movie stars did in publicity shots to something small huddles of
+addicts do outside the doors of office buildings.  A lot of the
+change was due to legislation, of course, but the legislation
+couldn't have happened if customs hadn't already changed.It took a while though—on the order of 100 years.  And unless the
+rate at which social antibodies evolve can increase to match the
+accelerating rate at which technological progress throws off new
+addictions, we'll be increasingly unable to rely on customs to
+protect us.
+[3]
+Unless we want to be canaries in the coal mine
+of each new addiction—the people whose sad example becomes a
+lesson to future generations—we'll have to figure out for ourselves
+what to avoid and how.  It will actually become a reasonable strategy
+(or a more reasonable strategy) to suspect 
+everything new.In fact, even that won't be enough.  We'll have to worry not just
+about new things, but also about existing things becoming more
+addictive.  That's what bit me.  I've avoided most addictions, but
+the Internet got me because it became addictive while I was using
+it.
+[4]Most people I know have problems with Internet addiction.  We're
+all trying to figure out our own customs for getting free of it.
+That's why I don't have an iPhone, for example; the last thing I
+want is for the Internet to follow me out into the world.
+[5]
+My latest trick is taking long hikes.  I used to think running was a
+better form of exercise than hiking because it took less time.  Now
+the slowness of hiking seems an advantage, because the longer I
+spend on the trail, the longer I have to think without interruption.Sounds pretty eccentric, doesn't it?  It always will when you're
+trying to solve problems where there are no customs yet to guide
+you.  Maybe I can't plead Occam's razor; maybe I'm simply eccentric.
+But if I'm right about the acceleration of addictiveness, then this
+kind of lonely squirming to avoid it will increasingly be the fate
+of anyone who wants to get things done.  We'll increasingly be
+defined by what we say no to.
+Notes[1]
+Could you restrict technological progress to areas where you
+wanted it?  Only in a limited way, without becoming a police state.
+And even then your restrictions would have undesirable side effects.
+"Good" and "bad" technological progress aren't sharply differentiated,
+so you'd find you couldn't slow the latter without also slowing the
+former.  And in any case, as Prohibition and the "war on drugs"
+show, bans often do more harm than good.[2]
+Technology has always been accelerating.  By Paleolithic
+standards, technology evolved at a blistering pace in the Neolithic
+period.[3]
+Unless we mass produce social customs.  I suspect the recent
+resurgence of evangelical Christianity in the US is partly a reaction
+to drugs.  In desperation people reach for the sledgehammer; if
+their kids won't listen to them, maybe they'll listen to God.  But
+that solution has broader consequences than just getting kids to
+say no to drugs.  You end up saying no to 
+science as well.
+I worry we may be heading for a future in which only a few people
+plot their own itinerary through no-land, while everyone else books
+a package tour.  Or worse still, has one booked for them by the
+government.[4]
+People commonly use the word "procrastination" to describe
+what they do on the Internet.  It seems to me too mild to describe
+what's happening as merely not-doing-work.  We don't call it
+procrastination when someone gets drunk instead of working.[5]
+Several people have told me they like the iPad because it
+lets them bring the Internet into situations where a laptop would
+be too conspicuous.  In other words, it's a hip flask.  (This is
+true of the iPhone too, of course, but this advantage isn't as
+obvious because it reads as a phone, and everyone's used to those.)Thanks to Sam Altman, Patrick Collison, Jessica Livingston, and
+Robert Morris for reading drafts of this.

recipes/experimental/long-context/H2O/data/PaulGrahamEssays/aord.txt

@@ -0,0 +1,126 @@
+October 2015When I talk to a startup that's been operating for more than 8 or
+9 months, the first thing I want to know is almost always the same.
+Assuming their expenses remain constant and their revenue growth
+is what it has been over the last several months, do they make it to
+profitability on the money they have left?  Or to put it more
+dramatically, by default do they live or die?The startling thing is how often the founders themselves don't know.
+Half the founders I talk to don't know whether they're default alive
+or default dead.If you're among that number, Trevor Blackwell has made a handy
+calculator you can use to find out.The reason I want to know first whether a startup is default alive
+or default dead is that the rest of the conversation depends on the
+answer.  If the company is default alive, we can talk about ambitious
+new things they could do.  If it's default dead, we probably need
+to talk about how to save it.  We know the current trajectory ends
+badly.  How can they get off that trajectory?Why do so few founders know whether they're default alive or default
+dead?  Mainly, I think, because they're not used to asking that.
+It's not a question that makes sense to ask early on, any more than
+it makes sense to ask a 3 year old how he plans to support
+himself.  But as the company grows older, the question switches from
+meaningless to critical.  That kind of switch often takes people
+by surprise.I propose the following solution: instead of starting to ask too
+late whether you're default alive or default dead, start asking too
+early.  It's hard to say precisely when the question switches
+polarity.  But it's probably not that dangerous to start worrying
+too early that you're default dead, whereas it's very dangerous to
+start worrying too late.The reason is a phenomenon I wrote about earlier: the
+fatal pinch.
+The fatal pinch is default dead + slow growth + not enough
+time to fix it.  And the way founders end up in it is by not realizing
+that's where they're headed.There is another reason founders don't ask themselves whether they're
+default alive or default dead: they assume it will be easy to raise
+more money.  But that assumption is often false, and worse still, the
+more you depend on it, the falser it becomes.Maybe it will help to separate facts from hopes. Instead of thinking
+of the future with vague optimism, explicitly separate the components.
+Say "We're default dead, but we're counting on investors to save
+us." Maybe as you say that, it will set off the same alarms in your
+head that it does in mine.  And if you set off the alarms sufficiently
+early, you may be able to avoid the fatal pinch.It would be safe to be default dead if you could count on investors
+saving you.  As a rule their interest is a function of
+growth.  If you have steep revenue growth, say over 5x a year, you
+can start to count on investors being interested even if you're not
+profitable.
+[1]
+But investors are so fickle that you can never
+do more than start to count on them.  Sometimes something about your
+business will spook investors even if your growth is great.  So no
+matter how good your growth is, you can never safely treat fundraising
+as more than a plan A. You should always have a plan B as well: you
+should know (as in write down) precisely what you'll need to do to
+survive if you can't raise more money, and precisely when you'll 
+have to switch to plan B if plan A isn't working.In any case, growing fast versus operating cheaply is far from the
+sharp dichotomy many founders assume it to be.  In practice there
+is surprisingly little connection between how much a startup spends
+and how fast it grows.  When a startup grows fast, it's usually
+because the product hits a nerve, in the sense of hitting some big
+need straight on.  When a startup spends a lot, it's usually because
+the product is expensive to develop or sell, or simply because
+they're wasteful.If you're paying attention, you'll be asking at this point not just
+how to avoid the fatal pinch, but how to avoid being default dead.
+That one is easy: don't hire too fast.  Hiring too fast is by far
+the biggest killer of startups that raise money.
+[2]Founders tell themselves they need to hire in order to grow.  But
+most err on the side of overestimating this need rather than
+underestimating it.  Why?  Partly because there's so much work to
+do.  Naive founders think that if they can just hire enough
+people, it will all get done.  Partly because successful startups have
+lots of employees, so it seems like that's what one does in order
+to be successful.  In fact the large staffs of successful startups
+are probably more the effect of growth than the cause.  And
+partly because when founders have slow growth they don't want to
+face what is usually the real reason: the product is not appealing
+enough.Plus founders who've just raised money are often encouraged to
+overhire by the VCs who funded them.  Kill-or-cure strategies are
+optimal for VCs because they're protected by the portfolio effect.
+VCs want to blow you up, in one sense of the phrase or the other.
+But as a founder your incentives are different.  You want above all
+to survive.
+[3]Here's a common way startups die.  They make something moderately
+appealing and have decent initial growth. They raise their first
+round fairly easily, because the founders seem smart and the idea
+sounds plausible. But because the product is only moderately
+appealing, growth is ok but not great.  The founders convince
+themselves that hiring a bunch of people is the way to boost growth.
+Their investors agree.  But (because the product is only moderately
+appealing) the growth never comes.  Now they're rapidly running out
+of runway.  They hope further investment will save them. But because
+they have high expenses and slow growth, they're now unappealing
+to investors. They're unable to raise more, and the company dies.What the company should have done is address the fundamental problem:
+that the product is only moderately appealing.  Hiring people is
+rarely the way to fix that.  More often than not it makes it harder.
+At this early stage, the product needs to evolve more than to be
+"built out," and that's usually easier with fewer people.
+[4]Asking whether you're default alive or default dead may save you
+from this.  Maybe the alarm bells it sets off will counteract the
+forces that push you to overhire.  Instead you'll be compelled to
+seek growth in other ways. For example, by doing
+things that don't scale, or by redesigning the product in the
+way only founders can.
+And for many if not most startups, these paths to growth will be
+the ones that actually work.Airbnb waited 4 months after raising money at the end of Y Combinator
+before they hired their first employee.  In the meantime the founders
+were terribly overworked.  But they were overworked evolving Airbnb
+into the astonishingly successful organism it is now.Notes[1]
+Steep usage growth will also interest investors.  Revenue
+will ultimately be a constant multiple of usage, so x% usage growth
+predicts x% revenue growth.  But in practice investors discount
+merely predicted revenue, so if you're measuring usage you need a
+higher growth rate to impress investors.[2]
+Startups that don't raise money are saved from hiring too
+fast because they can't afford to. But that doesn't mean you should
+avoid raising money in order to avoid this problem, any more than
+that total abstinence is the only way to avoid becoming an alcoholic.[3]
+I would not be surprised if VCs' tendency to push founders
+to overhire is not even in their own interest.  They don't know how
+many of the companies that get killed by overspending might have
+done well if they'd survived.  My guess is a significant number.[4]
+After reading a draft, Sam Altman wrote:"I think you should make the hiring point more strongly.  I think
+it's roughly correct to say that YC's most successful companies
+have never been the fastest to hire, and one of the marks of a great
+founder is being able to resist this urge."Paul Buchheit adds:"A related problem that I see a lot is premature scaling—founders
+take a small business that isn't really working (bad unit economics,
+typically) and then scale it up because they want impressive growth
+numbers. This is similar to over-hiring in that it makes the business
+much harder to fix once it's big, plus they are bleeding cash really
+fast."
+Thanks to Sam Altman, Paul Buchheit, Joe Gebbia, Jessica Livingston,
+and Geoff Ralston for reading drafts of this.

recipes/experimental/long-context/H2O/data/PaulGrahamEssays/apple.txt

@@ -0,0 +1,201 @@
+
+
+Want to start a startup?  Get funded by
+Y Combinator.
+
+
+
+
+November 2009I don't think Apple realizes how badly the App Store approval process
+is broken.  Or rather, I don't think they realize how much it matters
+that it's broken.The way Apple runs the App Store has harmed their reputation with
+programmers more than anything else they've ever done. 
+Their reputation with programmers used to be great.
+It used to be the most common complaint you heard
+about Apple was that their fans admired them too uncritically.
+The App Store has changed that.  Now a lot of programmers
+have started to see Apple as evil.How much of the goodwill Apple once had with programmers have they
+lost over the App Store?  A third?  Half?  And that's just so far.
+The App Store is an ongoing karma leak.* * *How did Apple get into this mess?  Their fundamental problem is
+that they don't understand software.They treat iPhone apps the way they treat the music they sell through
+iTunes.  Apple is the channel; they own the user; if you want to
+reach users, you do it on their terms. The record labels agreed,
+reluctantly.  But this model doesn't work for software.  It doesn't
+work for an intermediary to own the user.  The software business
+learned that in the early 1980s, when companies like VisiCorp showed
+that although the words "software" and "publisher" fit together,
+the underlying concepts don't.  Software isn't like music or books.
+It's too complicated for a third party to act as an intermediary
+between developer and user.   And yet that's what Apple is trying
+to be with the App Store: a software publisher.  And a particularly
+overreaching one at that, with fussy tastes and a rigidly enforced
+house style.If software publishing didn't work in 1980, it works even less now
+that software development has evolved from a small number of big
+releases to a constant stream of small ones.  But Apple doesn't
+understand that either.  Their model of product development derives
+from hardware.  They work on something till they think it's finished,
+then they release it.  You have to do that with hardware, but because
+software is so easy to change, its design can benefit from evolution.
+The standard way to develop applications now is to launch fast and
+iterate.  Which means it's a disaster to have long, random delays
+each time you release a new version.Apparently Apple's attitude is that developers should be more careful
+when they submit a new version to the App Store.  They would say
+that.  But powerful as they are, they're not powerful enough to
+turn back the evolution of technology.  Programmers don't use
+launch-fast-and-iterate out of laziness.  They use it because it
+yields the best results.  By obstructing that process, Apple is
+making them do bad work, and programmers hate that as much as Apple
+would.How would Apple like it if when they discovered a serious bug in
+OS X, instead of releasing a software update immediately, they had
+to submit their code to an intermediary who sat on it for a month
+and then rejected it because it contained an icon they didn't like?By breaking software development, Apple gets the opposite of what
+they intended: the version of an app currently available in the App
+Store tends to be an old and buggy one.  One developer told me:
+
+  As a result of their process, the App Store is full of half-baked
+  applications. I make a new version almost every day that I release
+  to beta users. The version on the App Store feels old and crappy.
+  I'm sure that a lot of developers feel this way: One emotion is
+  "I'm not really proud about what's in the App Store", and it's
+  combined with the emotion "Really, it's Apple's fault."
+
+Another wrote:
+
+  I believe that they think their approval process helps users by
+  ensuring quality.  In reality, bugs like ours get through all the
+  time and then it can take 4-8 weeks to get that bug fix approved,
+  leaving users to think that iPhone apps sometimes just don't work.
+  Worse for Apple, these apps work just fine on other platforms
+  that have immediate approval processes.
+
+Actually I suppose Apple has a third misconception: that all the
+complaints about App Store approvals are not a serious problem.
+They must hear developers complaining.  But partners and suppliers
+are always complaining.  It would be a bad sign if they weren't;
+it would mean you were being too easy on them.  Meanwhile the iPhone
+is selling better than ever.  So why do they need to fix anything?They get away with maltreating developers, in the short term, because
+they make such great hardware.  I just bought a new 27" iMac a
+couple days ago.  It's fabulous.  The screen's too shiny, and the
+disk is surprisingly loud, but it's so beautiful that you can't
+make yourself care.So I bought it, but I bought it, for the first time, with misgivings.
+I felt the way I'd feel buying something made in a country with a
+bad human rights record.  That was new.  In the past when I bought
+things from Apple it was an unalloyed pleasure.  Oh boy!  They make
+such great stuff.  This time it felt like a Faustian bargain.  They
+make such great stuff, but they're such assholes.  Do I really want
+to support this company?* * *Should Apple care what people like me think?  What difference does
+it make if they alienate a small minority of their users?There are a couple reasons they should care.  One is that these
+users are the people they want as employees.  If your company seems
+evil, the best programmers won't work for you.  That hurt Microsoft
+a lot starting in the 90s.  Programmers started to feel sheepish
+about working there.  It seemed like selling out.  When people from
+Microsoft were talking to other programmers and they mentioned where
+they worked, there were a lot of self-deprecating jokes about having
+gone over to the dark side.  But the real problem for Microsoft
+wasn't the embarrassment of the people they hired.  It was the
+people they never got.  And you know who got them?  Google and
+Apple.  If Microsoft was the Empire, they were the Rebel Alliance.
+And it's largely because they got more of the best people that
+Google and Apple are doing so much better than Microsoft today.Why are programmers so fussy about their employers' morals?  Partly
+because they can afford to be.  The best programmers can work
+wherever they want.  They don't have to work for a company they
+have qualms about.But the other reason programmers are fussy, I think, is that evil
+begets stupidity.  An organization that wins by exercising power
+starts to lose the ability to win by doing better work.  And it's
+not fun for a smart person to work in a place where the best ideas
+aren't the ones that win.  I think the reason Google embraced "Don't
+be evil" so eagerly was not so much to impress the outside world
+as to inoculate themselves against arrogance.
+[1]That has worked for Google so far.  They've become more
+bureaucratic, but otherwise they seem to have held true to their
+original principles. With Apple that seems less the case.  When you
+look at the famous 
+1984 ad 
+now, it's easier to imagine Apple as the
+dictator on the screen than the woman with the hammer.
+[2]
+In fact, if you read the dictator's speech it sounds uncannily like a
+prophecy of the App Store.
+
+  We have triumphed over the unprincipled dissemination of facts.We have created, for the first time in all history, a garden of
+  pure ideology, where each worker may bloom secure from the pests
+  of contradictory and confusing truths.
+
+The other reason Apple should care what programmers think of them
+is that when you sell a platform, developers make or break you.  If
+anyone should know this, Apple should.  VisiCalc made the Apple II.And programmers build applications for the platforms they use.  Most
+applications—most startups, probably—grow out of personal projects.
+Apple itself did.  Apple made microcomputers because that's what
+Steve Wozniak wanted for himself.  He couldn't have afforded a
+minicomputer. 
+[3]
+ Microsoft likewise started out making interpreters
+for little microcomputers because
+Bill Gates and Paul Allen were interested in using them.  It's a
+rare startup that doesn't build something the founders use.The main reason there are so many iPhone apps is that so many programmers
+have iPhones.  They may know, because they read it in an article,
+that Blackberry has such and such market share.  But in practice
+it's as if RIM didn't exist. If they're going to build something,
+they want to be able to use it themselves, and that means building
+an iPhone app.So programmers continue to develop iPhone apps, even though Apple
+continues to maltreat them.  They're like someone stuck in an abusive
+relationship.  They're so attracted to the iPhone that they can't
+leave.  But they're looking for a way out.  One wrote:
+
+  While I did enjoy developing for the iPhone, the control they
+  place on the App Store does not give me the drive to develop
+  applications as I would like. In fact I don't intend to make any
+  more iPhone applications unless absolutely necessary.
+[4]
+
+Can anything break this cycle?  No device I've seen so far could.
+Palm and RIM haven't a hope.  The only credible contender is Android.
+But Android is an orphan; Google doesn't really care about it, not
+the way Apple cares about the iPhone.  Apple cares about the iPhone
+the way Google cares about search.* * *Is the future of handheld devices one locked down by Apple?  It's
+a worrying prospect.  It would be a bummer to have another grim
+monoculture like we had in the 1990s.  In 1995, writing software
+for end users was effectively identical with writing Windows
+applications.  Our horror at that prospect was the single biggest
+thing that drove us to start building web apps.At least we know now what it would take to break Apple's lock.
+You'd have to get iPhones out of programmers' hands.  If programmers
+used some other device for mobile web access, they'd start to develop
+apps for that instead.How could you make a device programmers liked better than the iPhone?
+It's unlikely you could make something better designed.  Apple
+leaves no room there.  So this alternative device probably couldn't
+win on general appeal.  It would have to win by virtue of some
+appeal it had to programmers specifically.One way to appeal to programmers is with software.  If you
+could think of an application programmers had to have, but that
+would be impossible in the circumscribed world of the iPhone, 
+you could presumably get them to switch.That would definitely happen if programmers started to use handhelds
+as development machines—if handhelds displaced laptops the
+way laptops displaced desktops.  You need more control of a development
+machine than Apple will let you have over an iPhone.Could anyone make a device that you'd carry around in your pocket
+like a phone, and yet would also work as a development machine?
+It's hard to imagine what it would look like.  But I've learned
+never to say never about technology.  A phone-sized device that
+would work as a development machine is no more miraculous by present
+standards than the iPhone itself would have seemed by the standards
+of 1995.My current development machine is a MacBook Air, which I use with
+an external monitor and keyboard in my office, and by itself when
+traveling.  If there was a version half the size I'd prefer it.
+That still wouldn't be small enough to carry around everywhere like
+a phone, but we're within a factor of 4 or so.  Surely that gap is
+bridgeable.  In fact, let's make it an
+RFS. Wanted: 
+Woman with hammer.Notes[1]
+When Google adopted "Don't be evil," they were still so small
+that no one would have expected them to be, yet.
+[2]
+The dictator in the 1984 ad isn't Microsoft, incidentally;
+it's IBM.  IBM seemed a lot more frightening in those days, but
+they were friendlier to developers than Apple is now.[3]
+He couldn't even afford a monitor.  That's why the Apple
+I used a TV as a monitor.[4]
+Several people I talked to mentioned how much they liked the
+iPhone SDK.  The problem is not Apple's products but their policies.
+Fortunately policies are software; Apple can change them instantly
+if they want to.  Handy that, isn't it?Thanks to Sam Altman, Trevor Blackwell, Ross Boucher, 
+James Bracy, Gabor Cselle,
+Patrick Collison, Jason Freedman, John Gruber, Joe Hewitt, Jessica Livingston,
+Robert Morris, Teng Siong Ong, Nikhil Pandit, Savraj Singh, and Jared Tame for reading drafts of this.

recipes/experimental/long-context/H2O/data/PaulGrahamEssays/avg.txt

@@ -0,0 +1,375 @@
+
+
+Want to start a startup?  Get funded by
+Y Combinator.
+
+
+
+
+April 2001, rev. April 2003(This article is derived from a talk given at the 2001 Franz
+Developer Symposium.)
+In the summer of 1995, my friend Robert Morris and I
+started a startup called 
+Viaweb.  
+Our plan was to write
+software that would let end users build online stores.
+What was novel about this software, at the time, was
+that it ran on our server, using ordinary Web pages
+as the interface.A lot of people could have been having this idea at the
+same time, of course, but as far as I know, Viaweb was
+the first Web-based application.  It seemed such
+a novel idea to us that we named the company after it:
+Viaweb, because our software worked via the Web,
+instead of running on your desktop computer.Another unusual thing about this software was that it
+was written primarily in a programming language called
+Lisp. It was one of the first big end-user
+applications to be written in Lisp, which up till then
+had been used mostly in universities and research labs. [1]The Secret WeaponEric Raymond has written an essay called "How to Become a Hacker,"
+and in it, among other things, he tells would-be hackers what
+languages they should learn.  He suggests starting with Python and
+Java, because they are easy to learn.  The serious hacker will also
+want to learn C, in order to hack Unix, and Perl for system
+administration and cgi scripts.  Finally, the truly serious hacker
+should consider learning Lisp:
+
+  Lisp is worth learning for the profound enlightenment experience
+  you will have when you finally get it; that experience will make
+  you a better programmer for the rest of your days, even if you
+  never actually use Lisp itself a lot.
+
+This is the same argument you tend to hear for learning Latin.  It
+won't get you a job, except perhaps as a classics professor, but
+it will improve your mind, and make you a better writer in languages
+you do want to use, like English.But wait a minute.  This metaphor doesn't stretch that far.  The
+reason Latin won't get you a job is that no one speaks it.  If you
+write in Latin, no one can understand you.  But Lisp is a computer
+language, and computers speak whatever language you, the programmer,
+tell them to.So if Lisp makes you a better programmer, like he says, why wouldn't
+you want to use it? If a painter were offered a brush that would
+make him a better painter, it seems to me that he would want to
+use it in all his paintings, wouldn't he? I'm not trying to make
+fun of Eric Raymond here.  On the whole, his advice is good.  What
+he says about Lisp is pretty much the conventional wisdom.  But
+there is a contradiction in the conventional wisdom:  Lisp will
+make you a better programmer, and yet you won't use it.Why not?  Programming languages are just tools, after all.  If Lisp
+really does yield better programs, you should use it.  And if it
+doesn't, then who needs it?This is not just a theoretical question.  Software is a very
+competitive business, prone to natural monopolies.  A company that
+gets software written faster and better will, all other things
+being equal, put its competitors out of business.  And when you're
+starting a startup, you feel this very keenly.  Startups tend to
+be an all or nothing proposition.  You either get rich, or you get
+nothing.  In a startup, if you bet on the wrong technology, your
+competitors will crush you.Robert and I both knew Lisp well, and we couldn't see any reason
+not to trust our instincts and go with Lisp.  We knew that everyone
+else was writing their software in C++ or Perl.  But we also knew
+that that didn't mean anything.  If you chose technology that way,
+you'd be running Windows.  When you choose technology, you have to
+ignore what other people are doing, and consider only what will
+work the best.This is especially true in a startup.  In a big company, you can
+do what all the other big companies are doing.  But a startup can't
+do what all the other startups do.  I don't think a lot of people
+realize this, even in startups.The average big company grows at about ten percent a year.  So if
+you're running a big company and you do everything the way the
+average big company does it, you can expect to do as well as the
+average big company-- that is, to grow about ten percent a year.The same thing will happen if you're running a startup, of course.
+If you do everything the way the average startup does it, you should
+expect average performance.  The problem here is, average performance
+means that you'll go out of business.  The survival rate for startups
+is way less than fifty percent.  So if you're running a startup,
+you had better be doing something odd.  If not, you're in trouble.Back in 1995, we knew something that I don't think our competitors
+understood, and few understand even now:  when you're writing
+software that only has to run on your own servers, you can use
+any language you want.  When you're writing desktop software,
+there's a strong bias toward writing applications in the same
+language as the operating system.  Ten years ago, writing applications
+meant writing applications in C.  But with Web-based software,
+especially when you have the source code of both the language and
+the operating system, you can use whatever language you want.This new freedom is a double-edged sword, however.  Now that you
+can use any language, you have to think about which one to use.
+Companies that try to pretend nothing has changed risk finding that
+their competitors do not.If you can use any language, which do you use?  We chose Lisp.
+For one thing, it was obvious that rapid development would be
+important in this market.  We were all starting from scratch, so
+a company that could get new features done before its competitors
+would have a big advantage.  We knew Lisp was a really good language
+for writing software quickly, and server-based applications magnify
+the effect of rapid development, because you can release software
+the minute it's done.If other companies didn't want to use Lisp, so much the better.
+It might give us a technological edge, and we needed all the help
+we could get.  When we started Viaweb, we had no experience in
+business.  We didn't know anything about marketing, or hiring
+people, or raising money, or getting customers.  Neither of us had
+ever even had what you would call a real job.  The only thing we
+were good at was writing software.  We hoped that would save us.
+Any advantage we could get in the software department, we would
+take.So you could say that using Lisp was an experiment.  Our hypothesis
+was that if we wrote our software in Lisp, we'd be able to get
+features done faster than our competitors, and also to do things
+in our software that they couldn't do.  And because Lisp was so
+high-level, we wouldn't need a big development team, so our costs
+would be lower.  If this were so, we could offer a better product
+for less money, and still make a profit.  We would end up getting
+all the users, and our competitors would get none, and eventually
+go out of business.  That was what we hoped would happen, anyway.What were the results of this experiment?  Somewhat surprisingly,
+it worked.  We eventually had many competitors, on the order of
+twenty to thirty of them, but none of their software could compete
+with ours.  We had a wysiwyg online store builder that ran on the
+server and yet felt like a desktop application.  Our competitors
+had cgi scripts.  And we were always far ahead of them in features.
+Sometimes, in desperation, competitors would try to introduce
+features that we didn't have.  But with Lisp our development cycle
+was so fast that we could sometimes duplicate a new feature within
+a day or two of a competitor announcing it in a press release.  By
+the time journalists covering the press release got round to calling
+us, we would have the new feature too.It must have seemed to our competitors that we had some kind of
+secret weapon-- that we were decoding their Enigma traffic or
+something.  In fact we did have a secret weapon, but it was simpler
+than they realized.  No one was leaking news of their features to
+us.   We were just able to develop software faster than anyone
+thought possible.When I was about nine I happened to get hold of a copy of The Day
+of the Jackal, by Frederick Forsyth.  The main character is an
+assassin who is hired to kill the president of France.  The assassin
+has to get past the police to get up to an apartment that overlooks
+the president's route.  He walks right by them, dressed up as an
+old man on crutches, and they never suspect him.Our secret weapon was similar.  We wrote our software in a weird
+AI language, with a bizarre syntax full of parentheses.  For years
+it had annoyed me to hear Lisp described that way.  But now it
+worked to our advantage.  In business, there is nothing more valuable
+than a technical advantage your competitors don't understand.  In
+business, as in war, surprise is worth as much as force.And so, I'm a little embarrassed to say, I never said anything
+publicly about Lisp while we were working on Viaweb.  We never
+mentioned it to the press, and if you searched for Lisp on our Web
+site, all you'd find were the titles of two books in my bio.  This
+was no accident.  A startup should give its competitors as little
+information as possible.  If they didn't know what language our
+software was written in, or didn't care, I wanted to keep it that
+way.[2]The people who understood our technology best were the customers.
+They didn't care what language Viaweb was written in either, but
+they noticed that it worked really well.  It let them build great
+looking online stores literally in minutes.  And so, by word of
+mouth mostly, we got more and more users.  By the end of 1996 we
+had about 70 stores online.  At the end of 1997 we had 500.  Six
+months later, when Yahoo bought us, we had 1070 users.  Today, as
+Yahoo Store, this software continues to dominate its market.  It's
+one of the more profitable pieces of Yahoo, and the stores built
+with it are the foundation of Yahoo Shopping.  I left Yahoo in
+1999, so I don't know exactly how many users they have now, but
+the last I heard there were about 20,000.
+The Blub ParadoxWhat's so great about Lisp?  And if Lisp is so great, why doesn't
+everyone use it?  These sound like rhetorical questions, but actually
+they have straightforward answers.  Lisp is so great not because
+of some magic quality visible only to devotees, but because it is
+simply the most powerful language available.  And the reason everyone
+doesn't use it is that programming languages are not merely
+technologies, but habits of mind as well, and nothing changes
+slower.  Of course, both these answers need explaining.I'll begin with a shockingly controversial statement:  programming
+languages vary in power.Few would dispute, at least, that high level languages are more
+powerful than machine language.  Most programmers today would agree
+that you do not, ordinarily, want to program in machine language.
+Instead, you should program in a high-level language, and have a
+compiler translate it into machine language for you.  This idea is
+even built into the hardware now: since the 1980s, instruction sets
+have been designed for compilers rather than human programmers.Everyone knows it's a mistake to write your whole program by hand
+in machine language.  What's less often understood is that there
+is a more general principle here: that if you have a choice of
+several languages, it is, all other things being equal, a mistake
+to program in anything but the most powerful one. [3]There are many exceptions to this rule.  If you're writing a program
+that has to work very closely with a program written in a certain
+language, it might be a good idea to write the new program in the
+same language.  If you're writing a program that only has to do
+something very simple, like number crunching or bit manipulation,
+you may as well use a less abstract language, especially since it
+may be slightly faster.  And if you're writing a short, throwaway
+program, you may be better off just using whatever language has
+the best library functions for the task.  But in general, for
+application software, you want to be using the most powerful
+(reasonably efficient) language you can get, and using anything
+else is a mistake, of exactly the same kind, though possibly in a
+lesser degree, as programming in machine language.You can see that machine language is very low level.  But, at least
+as a kind of social convention, high-level languages are often all
+treated as equivalent.  They're not.  Technically the term "high-level
+language" doesn't mean anything very definite.  There's no dividing
+line with machine languages on one side and all the high-level
+languages on the other.  Languages fall along a continuum [4] of
+abstractness, from the most powerful all the way down to machine
+languages, which themselves vary in power.Consider Cobol.  Cobol is a high-level language, in the sense that
+it gets compiled into machine language.  Would anyone seriously
+argue that Cobol is equivalent in power to, say, Python?  It's
+probably closer to machine language than Python.Or how about Perl 4?  Between Perl 4 and Perl 5, lexical closures
+got added to the language.  Most Perl hackers would agree that Perl
+5 is more powerful than Perl 4.  But once you've admitted that,
+you've admitted that one high level language can be more powerful
+than another.  And it follows inexorably that, except in special
+cases, you ought to use the most powerful you can get.This idea is rarely followed to its conclusion, though.  After a
+certain age, programmers rarely switch languages voluntarily.
+Whatever language people happen to be used to, they tend to consider
+just good enough.Programmers get very attached to their favorite languages, and I
+don't want to hurt anyone's feelings, so to explain this point I'm
+going to use a hypothetical language called Blub.  Blub falls right
+in the middle of the abstractness continuum.  It is not the most
+powerful language, but it is more powerful than Cobol or machine
+language.And in fact, our hypothetical Blub programmer wouldn't use either
+of them.  Of course he wouldn't program in machine language.  That's
+what compilers are for.  And as for Cobol, he doesn't know how
+anyone can get anything done with it.  It doesn't even have x (Blub
+feature of your choice).As long as our hypothetical Blub programmer is looking down the
+power continuum, he knows he's looking down.  Languages less powerful
+than Blub are obviously less powerful, because they're missing some
+feature he's used to.  But when our hypothetical Blub programmer
+looks in the other direction, up the power continuum, he doesn't
+realize he's looking up.  What he sees are merely weird languages.
+He probably considers them about equivalent in power to Blub, but
+with all this other hairy stuff thrown in as well.  Blub is good
+enough for him, because he thinks in Blub.When we switch to the point of view of a programmer using any of
+the languages higher up the power continuum, however, we find that
+he in turn looks down upon Blub.  How can you get anything done in
+Blub? It doesn't even have y.By induction, the only programmers in a position to see all the
+differences in power between the various languages are those who
+understand the most powerful one.  (This is probably what Eric
+Raymond meant about Lisp making you a better programmer.) You can't
+trust the opinions of the others, because of the Blub paradox:
+they're satisfied with whatever language they happen to use, because
+it dictates the way they think about programs.I know this from my own experience, as a high school kid writing
+programs in Basic.  That language didn't even support recursion.
+It's hard to imagine writing programs without using recursion, but
+I didn't miss it at the time.  I thought in Basic.  And I was a
+whiz at it.  Master of all I surveyed.The five languages that Eric Raymond recommends to hackers fall at
+various points on the power continuum.  Where they fall relative
+to one another is a sensitive topic.  What I will say is that I
+think Lisp is at the top.  And to support this claim I'll tell you
+about one of the things I find missing when I look at the other
+four languages.  How can you get anything done in them, I think,
+without macros? [5]Many languages have something called a macro.  But Lisp macros are
+unique.  And believe it or not, what they do is related to the
+parentheses.  The designers of Lisp didn't put all those parentheses
+in the language just to be different.  To the Blub programmer, Lisp
+code looks weird.  But those parentheses are there for a reason.
+They are the outward evidence of a fundamental difference between
+Lisp and other languages.Lisp code is made out of Lisp data objects.  And not in the trivial
+sense that the source files contain characters, and strings are
+one of the data types supported by the language.  Lisp code, after
+it's read by the parser, is made of data structures that you can
+traverse.If you understand how compilers work, what's really going on is
+not so much that Lisp has a strange syntax as that Lisp has no
+syntax.  You write programs in the parse trees that get generated
+within the compiler when other languages are parsed.  But these
+parse trees are fully accessible to your programs.  You can write
+programs that manipulate them.  In Lisp, these programs are called
+macros.  They are programs that write programs.Programs that write programs?  When would you ever want to do that?
+Not very often, if you think in Cobol.  All the time, if you think
+in Lisp.  It would be convenient here if I could give an example
+of a powerful macro, and say there! how about that?  But if I did,
+it would just look like gibberish to someone who didn't know Lisp;
+there isn't room here to explain everything you'd need to know to
+understand what it meant.  In 
+Ansi Common Lisp I tried to move
+things along as fast as I could, and even so I didn't get to macros
+until page 160.But I think I can give a kind of argument that might be convincing.
+The source code of the Viaweb editor was probably about 20-25%
+macros.  Macros are harder to write than ordinary Lisp functions,
+and it's considered to be bad style to use them when they're not
+necessary.  So every macro in that code is there because it has to
+be.  What that means is that at least 20-25% of the code in this
+program is doing things that you can't easily do in any other
+language.  However skeptical the Blub programmer might be about my
+claims for the mysterious powers of Lisp, this ought to make him
+curious.  We weren't writing this code for our own amusement.  We
+were a tiny startup, programming as hard as we could in order to
+put technical barriers between us and our competitors.A suspicious person might begin to wonder if there was some
+correlation here.  A big chunk of our code was doing things that
+are very hard to do in other languages.  The resulting software
+did things our competitors' software couldn't do.  Maybe there was
+some kind of connection.  I encourage you to follow that thread.
+There may be more to that old man hobbling along on his crutches
+than meets the eye.Aikido for StartupsBut I don't expect to convince anyone 
+(over 25) 
+to go out and learn
+Lisp.  The purpose of this article is not to change anyone's mind,
+but to reassure people already interested in using Lisp-- people
+who know that Lisp is a powerful language, but worry because it
+isn't widely used.  In a competitive situation, that's an advantage.
+Lisp's power is multiplied by the fact that your competitors don't
+get it.If you think of using Lisp in a startup, you shouldn't worry that
+it isn't widely understood.  You should hope that it stays that
+way. And it's likely to.  It's the nature of programming languages
+to make most people satisfied with whatever they currently use.
+Computer hardware changes so much faster than personal habits that
+programming practice is usually ten to twenty years behind the
+processor.  At places like MIT they were writing programs in
+high-level languages in the early 1960s, but many companies continued
+to write code in machine language well into the 1980s.  I bet a
+lot of people continued to write machine language until the processor,
+like a bartender eager to close up and go home, finally kicked them
+out by switching to a risc instruction set.Ordinarily technology changes fast.  But programming languages are
+different: programming languages are not just technology, but what
+programmers think in.  They're half technology and half religion.[6]
+And so the median language, meaning whatever language the median
+programmer uses, moves as slow as an iceberg.  Garbage collection,
+introduced by Lisp in about 1960, is now widely considered to be
+a good thing.  Runtime typing, ditto, is growing in popularity.
+Lexical closures, introduced by Lisp in the early 1970s, are now,
+just barely, on the radar screen.  Macros, introduced by Lisp in the
+mid 1960s, are still terra incognita.Obviously, the median language has enormous momentum.  I'm not
+proposing that you can fight this powerful force.  What I'm proposing
+is exactly the opposite: that, like a practitioner of Aikido, you
+can use it against your opponents.If you work for a big company, this may not be easy.  You will have
+a hard time convincing the pointy-haired boss to let you build
+things in Lisp, when he has just read in the paper that some other
+language is poised, like Ada was twenty years ago, to take over
+the world.  But if you work for a startup that doesn't have
+pointy-haired bosses yet, you can, like we did, turn the Blub
+paradox to your advantage:  you can use technology that your
+competitors, glued immovably to the median language, will never be
+able to match.If you ever do find yourself working for a startup, here's a handy
+tip for evaluating competitors.  Read their job listings.  Everything
+else on their site may be stock photos or the prose equivalent,
+but the job listings have to be specific about what they want, or
+they'll get the wrong candidates.During the years we worked on Viaweb I read a lot of job descriptions.
+A new competitor seemed to emerge out of the woodwork every month
+or so.  The first thing I would do, after checking to see if they
+had a live online demo, was look at their job listings.  After a
+couple years of this I could tell which companies to worry about
+and which not to.  The more of an IT flavor the job descriptions
+had, the less dangerous the company was.  The safest kind were the
+ones that wanted Oracle experience.  You never had to worry about
+those.  You were also safe if they said they wanted C++ or Java
+developers.  If they wanted Perl or Python programmers, that would
+be a bit frightening-- that's starting to sound like a company
+where the technical side, at least, is run by real hackers.  If I
+had ever seen a job posting looking for Lisp hackers, I would have
+been really worried.
+Notes[1] Viaweb at first had two parts: the editor, written in Lisp,
+which people used to build their sites, and the ordering system,
+written in C, which handled orders.  The first version was mostly
+Lisp, because the ordering system was small.  Later we added two
+more modules, an image generator written in C, and a back-office
+manager written mostly in Perl.In January 2003, Yahoo released a new version of the editor 
+written in C++ and Perl.  It's hard to say whether the program is no
+longer written in Lisp, though, because to translate this program
+into C++ they literally had to write a Lisp interpreter: the source
+files of all the page-generating templates are still, as far as I
+know,  Lisp code.  (See Greenspun's Tenth Rule.)[2] Robert Morris says that I didn't need to be secretive, because
+even if our competitors had known we were using Lisp, they wouldn't
+have understood why:  "If they were that smart they'd already be
+programming in Lisp."[3] All languages are equally powerful in the sense of being Turing
+equivalent, but that's not the sense of the word programmers care
+about. (No one wants to program a Turing machine.)  The kind of
+power programmers care about may not be formally definable, but
+one way to explain it would be to say that it refers to features
+you could only get in the less powerful language by writing an
+interpreter for the more powerful language in it. If language A
+has an operator for removing spaces from strings and language B
+doesn't, that probably doesn't make A more powerful, because you
+can probably write a subroutine to do it in B.  But if A supports,
+say, recursion, and B doesn't, that's not likely to be something
+you can fix by writing library functions.[4] Note to nerds: or possibly a lattice, narrowing toward the top;
+it's not the shape that matters here but the idea that there is at
+least a partial order.[5] It is a bit misleading to treat macros as a separate feature.
+In practice their usefulness is greatly enhanced by other Lisp
+features like lexical closures and rest parameters.[6] As a result, comparisons of programming languages either take
+the form of religious wars or undergraduate textbooks so determinedly
+neutral that they're really works of anthropology.  People who
+value their peace, or want tenure, avoid the topic.  But the question
+is only half a religious one; there is something there worth
+studying, especially if you want to design new languages.

recipes/experimental/long-context/H2O/data/PaulGrahamEssays/before.txt

@@ -0,0 +1,387 @@
+
+
+Want to start a startup?  Get funded by
+Y Combinator.
+
+
+
+
+October 2014(This essay is derived from a guest lecture in Sam Altman's startup class at
+Stanford.  It's intended for college students, but much of it is
+applicable to potential founders at other ages.)One of the advantages of having kids is that when you have to give
+advice, you can ask yourself "what would I tell my own kids?"  My
+kids are little, but I can imagine what I'd tell them about startups
+if they were in college, and that's what I'm going to tell you.Startups are very counterintuitive.  I'm not sure why.  Maybe it's
+just because knowledge about them hasn't permeated our culture yet.
+But whatever the reason, starting a startup is a task where you
+can't always trust your instincts.It's like skiing in that way.  When you first try skiing and you
+want to slow down, your instinct is to lean back.  But if you lean
+back on skis you fly down the hill out of control.  So part of
+learning to ski is learning to suppress that impulse.  Eventually
+you get new habits, but at first it takes a conscious effort.  At
+first there's a list of things you're trying to remember as you
+start down the hill.Startups are as unnatural as skiing, so there's a similar list for
+startups. Here I'm going to give you the first part of it — the things
+to remember if you want to prepare yourself to start a startup.
+CounterintuitiveThe first item on it is the fact I already mentioned: that startups
+are so weird that if you trust your instincts, you'll make a lot
+of mistakes.  If you know nothing more than this, you may at least
+pause before making them.When I was running Y Combinator I used to joke that our function
+was to tell founders things they would ignore.  It's really true.
+Batch after batch, the YC partners warn founders about mistakes
+they're about to make, and the founders ignore them, and then come
+back a year later and say "I wish we'd listened."Why do the founders ignore the partners' advice?  Well, that's the
+thing about counterintuitive ideas: they contradict your intuitions.
+They seem wrong.  So of course your first impulse is to disregard
+them.  And in fact my joking description is not merely the curse
+of Y Combinator but part of its raison d'etre. If founders' instincts
+already gave them the right answers, they wouldn't need us.  You
+only need other people to give you advice that surprises you. That's
+why there are a lot of ski instructors and not many running
+instructors.
+[1]You can, however, trust your instincts about people.  And in fact
+one of the most common mistakes young founders make is not to
+do that enough.  They get involved with people who seem impressive,
+but about whom they feel some misgivings personally.  Later when
+things blow up they say "I knew there was something off about him,
+but I ignored it because he seemed so impressive."If you're thinking about getting involved with someone — as a
+cofounder, an employee, an investor, or an acquirer — and you
+have misgivings about them, trust your gut.  If someone seems
+slippery, or bogus, or a jerk, don't ignore it.This is one case where it pays to be self-indulgent. Work with
+people you genuinely like, and you've known long enough to be sure.
+ExpertiseThe second counterintuitive point is that it's not that important
+to know a lot about startups.  The way to succeed in a startup is
+not to be an expert on startups, but to be an expert on your users
+and the problem you're solving for them.
+Mark Zuckerberg didn't succeed because he was an expert on startups.
+He succeeded despite being a complete noob at startups, because he
+understood his users really well.If you don't know anything about, say, how to raise an angel round,
+don't feel bad on that account.  That sort of thing you can learn
+when you need to, and forget after you've done it.In fact, I worry it's not merely unnecessary to learn in great
+detail about the mechanics of startups, but possibly somewhat
+dangerous.  If I met an undergrad who knew all about convertible
+notes and employee agreements and (God forbid) class FF stock, I
+wouldn't think "here is someone who is way ahead of their peers."
+It would set off alarms.  Because another of the characteristic
+mistakes of young founders is to go through the motions of starting
+a startup.  They make up some plausible-sounding idea, raise money
+at a good valuation, rent a cool office, hire a bunch of people.
+From the outside that seems like what startups do.  But the next
+step after rent a cool office and hire a bunch of people is: gradually
+realize how completely fucked they are, because while imitating all
+the outward forms of a startup they have neglected the one thing
+that's actually essential: making something people want.
+GameWe saw this happen so often that we made up a name for it: playing
+house.  Eventually I realized why it was happening.  The reason
+young founders go through the motions of starting a startup is
+because that's what they've been trained to do for their whole lives
+up to that point.  Think about what you have to do to get into
+college, for example.  Extracurricular activities, check.  Even in
+college classes most of the work is as artificial as running laps.I'm not attacking the educational system for being this way. There
+will always be a certain amount of fakeness in the work you do when
+you're being taught something, and if you measure their performance
+it's inevitable that people will exploit the difference to the point
+where much of what you're measuring is artifacts of the fakeness.I confess I did it myself in college. I found that in a lot of
+classes there might only be 20 or 30 ideas that were the right shape
+to make good exam questions.  The way I studied for exams in these
+classes was not (except incidentally) to master the material taught
+in the class, but to make a list of potential exam questions and
+work out the answers in advance. When I walked into the final, the
+main thing I'd be feeling was curiosity about which of my questions
+would turn up on the exam.  It was like a game.It's not surprising that after being trained for their whole lives
+to play such games, young founders' first impulse on starting a
+startup is to try to figure out the tricks for winning at this new
+game. Since fundraising appears to be the measure of success for
+startups (another classic noob mistake), they always want to know what the
+tricks are for convincing investors.  We tell them the best way to
+convince investors is to make a startup
+that's actually doing well, meaning growing fast, and then simply
+tell investors so.  Then they want to know what the tricks are for
+growing fast.  And we have to tell them the best way to do that is
+simply to make something people want.So many of the conversations YC partners have with young founders
+begin with the founder asking "How do we..." and the partner replying
+"Just..."Why do the founders always make things so complicated?  The reason,
+I realized, is that they're looking for the trick.So this is the third counterintuitive thing to remember about
+startups: starting a startup is where gaming the system stops
+working.  Gaming the system may continue to work if you go to work
+for a big company. Depending on how broken the company is, you can
+succeed by sucking up to the right people, giving the impression
+of productivity, and so on. 
+[2]
+But that doesn't work with startups.
+There is no boss to trick, only users, and all users care about is
+whether your product does what they want. Startups are as impersonal
+as physics.  You have to make something people want, and you prosper
+only to the extent you do.The dangerous thing is, faking does work to some degree on investors.
+If you're super good at sounding like you know what you're talking
+about, you can fool investors for at least one and perhaps even two
+rounds of funding.  But it's not in your interest to.  The company
+is ultimately doomed.  All you're doing is wasting your own time
+riding it down.So stop looking for the trick. There are tricks in startups, as
+there are in any domain, but they are an order of magnitude less
+important than solving the real problem. A founder who knows nothing
+about fundraising but has made something users love will have an
+easier time raising money than one who knows every trick in the
+book but has a flat usage graph. And more importantly, the founder
+who has made something users love is the one who will go on to
+succeed after raising the money.Though in a sense it's bad news in that you're deprived of one of
+your most powerful weapons, I think it's exciting that gaming the
+system stops working when you start a startup.  It's exciting that
+there even exist parts of the world where you win by doing good
+work.  Imagine how depressing the world would be if it were all
+like school and big companies, where you either have to spend a lot
+of time on bullshit things or lose to people who do.
+[3]
+I would
+have been delighted if I'd realized in college that there were parts
+of the real world where gaming the system mattered less than others,
+and a few where it hardly mattered at all.  But there are, and this
+variation is one of the most important things to consider when
+you're thinking about your future.  How do you win in each type of
+work, and what would you like to win by doing?
+[4]
+All-ConsumingThat brings us to our fourth counterintuitive point: startups are
+all-consuming.  If you start a startup, it will take over your life
+to a degree you cannot imagine.  And if your startup succeeds, it
+will take over your life for a long time: for several years at the
+very least, maybe for a decade, maybe for the rest of your working
+life.  So there is a real opportunity cost here.Larry Page may seem to have an enviable life, but there are aspects
+of it that are unenviable.  Basically at 25 he started running as
+fast as he could and it must seem to him that he hasn't stopped to
+catch his breath since.  Every day new shit happens in the Google
+empire that only the CEO can deal with, and he, as CEO, has to deal
+with it.  If he goes on vacation for even a week, a whole week's
+backlog of shit accumulates.  And he has to bear this uncomplainingly,
+partly because as the company's daddy he can never show fear or
+weakness, and partly because billionaires get less than zero sympathy
+if they talk about having difficult lives.  Which has the strange
+side effect that the difficulty of being a successful startup founder
+is concealed from almost everyone except those who've done it.Y Combinator has now funded several companies that can be called
+big successes, and in every single case the founders say the same
+thing.  It never gets any easier.  The nature of the problems change.
+You're worrying about construction delays at your London office
+instead of the broken air conditioner in your studio apartment.
+But the total volume of worry never decreases; if anything it
+increases.Starting a successful startup is similar to having kids in that
+it's like a button you push that changes your life irrevocably.
+And while it's truly wonderful having kids, there are a lot of
+things that are easier to do before you have them than after.  Many
+of which will make you a better parent when you do have kids. And
+since you can delay pushing the button for a while, most people in
+rich countries do.Yet when it comes to startups, a lot of people seem to think they're
+supposed to start them while they're still in college.  Are you
+crazy?  And what are the universities thinking?  They go out of
+their way to ensure their students are well supplied with contraceptives,
+and yet they're setting up entrepreneurship programs and startup
+incubators left and right.To be fair, the universities have their hand forced here.  A lot
+of incoming students are interested in startups.  Universities are,
+at least de facto, expected to prepare them for their careers.  So
+students who want to start startups hope universities can teach
+them about startups.  And whether universities can do this or not,
+there's some pressure to claim they can, lest they lose applicants
+to other universities that do.Can universities teach students about startups?  Yes and no.  They
+can teach students about startups, but as I explained before, this
+is not what you need to know.  What you need to learn about are the
+needs of your own users, and you can't do that until you actually
+start the company.
+[5]
+So starting a startup is intrinsically
+something you can only really learn by doing it.  And it's impossible
+to do that in college, for the reason I just explained: startups
+take over your life.  You can't start a startup for real as a
+student, because if you start a startup for real you're not a student
+anymore. You may be nominally a student for a bit, but you won't even
+be that for long.
+[6]Given this dichotomy, which of the two paths should you take?  Be
+a real student and not start a startup, or start a real startup and
+not be a student?  I can answer that one for you. Do not start a
+startup in college.  How to start a startup is just a subset of a
+bigger problem you're trying to solve: how to have a good life.
+And though starting a startup can be part of a good life for a lot
+of ambitious people, age 20 is not the optimal time to do it.
+Starting a startup is like a brutally fast depth-first search.  Most
+people should still be searching breadth-first at 20.You can do things in your early 20s that you can't do as well before
+or after, like plunge deeply into projects on a whim and travel
+super cheaply with no sense of a deadline.  For unambitious people,
+this sort of thing is the dreaded "failure to launch," but for the
+ambitious ones it can be an incomparably valuable sort of exploration.
+If you start a startup at 20 and you're sufficiently successful,
+you'll never get to do it.
+[7]Mark Zuckerberg will never get to bum around a foreign country.  He
+can do other things most people can't, like charter jets to fly him
+to foreign countries. But success has taken a lot of the serendipity
+out of his life. Facebook is running him as much as he's running
+Facebook. And while it can be very cool to be in the grip of a
+project you consider your life's work, there are advantages to
+serendipity too, especially early in life.  Among other things it
+gives you more options to choose your life's work from.There's not even a tradeoff here. You're not sacrificing anything
+if you forgo starting a startup at 20, because you're more likely
+to succeed if you wait.  In the unlikely case that you're 20 and
+one of your side projects takes off like Facebook did, you'll face
+a choice of running with it or not, and it may be reasonable to run
+with it.  But the usual way startups take off is for the founders
+to make them take off, and it's gratuitously
+stupid to do that at 20.
+TryShould you do it at any age?  I realize I've made startups sound
+pretty hard.  If I haven't, let me try again: starting a startup
+is really hard.  What if it's too hard?  How can you tell if you're
+up to this challenge?The answer is the fifth counterintuitive point: you can't tell. Your
+life so far may have given you some idea what your prospects might
+be if you tried to become a mathematician, or a professional football
+player.  But unless you've had a very strange life you haven't done
+much that was like being a startup founder.
+Starting a startup will change you a lot.  So what you're trying
+to estimate is not just what you are, but what you could grow into,
+and who can do that?For the past 9 years it was my job to predict whether people would
+have what it took to start successful startups.  It was easy to
+tell how smart they were, and most people reading this will be over
+that threshold.  The hard part was predicting how tough and ambitious they would become.  There
+may be no one who has more experience at trying to predict that,
+so I can tell you how much an expert can know about it, and the
+answer is: not much.  I learned to keep a completely open mind about
+which of the startups in each batch would turn out to be the stars.The founders sometimes think they know. Some arrive feeling sure
+they will ace Y Combinator just as they've aced every one of the (few,
+artificial, easy) tests they've faced in life so far.  Others arrive
+wondering how they got in, and hoping YC doesn't discover whatever
+mistake caused it to accept them.  But there is little correlation
+between founders' initial attitudes and how well their companies
+do.I've read that the same is true in the military — that the
+swaggering recruits are no more likely to turn out to be really
+tough than the quiet ones. And probably for the same reason: that
+the tests involved are so different from the ones in their previous
+lives.If you're absolutely terrified of starting a startup, you probably
+shouldn't do it.  But if you're merely unsure whether you're up to
+it, the only way to find out is to try.  Just not now.
+IdeasSo if you want to start a startup one day, what should you do in
+college?  There are only two things you need initially: an idea and
+cofounders.  And the m.o. for getting both is the same.  Which leads
+to our sixth and last counterintuitive point: that the way to get
+startup ideas is not to try to think of startup ideas.I've written a whole essay on this,
+so I won't repeat it all here.  But the short version is that if
+you make a conscious effort to think of startup ideas, the ideas
+you come up with will not merely be bad, but bad and plausible-sounding,
+meaning you'll waste a lot of time on them before realizing they're
+bad.The way to come up with good startup ideas is to take a step back.
+Instead of making a conscious effort to think of startup ideas,
+turn your mind into the type that startup ideas form in without any
+conscious effort.  In fact, so unconsciously that you don't even
+realize at first that they're startup ideas.This is not only possible, it's how Apple, Yahoo, Google, and
+Facebook all got started.  None of these companies were even meant
+to be companies at first.  They were all just side projects.  The
+best startups almost have to start as side projects, because great
+ideas tend to be such outliers that your conscious mind would reject
+them as ideas for companies.Ok, so how do you turn your mind into the type that startup ideas
+form in unconsciously?  (1) Learn a lot about things that matter,
+then (2) work on problems that interest you (3) with people you
+like and respect.  The third part, incidentally, is how you get
+cofounders at the same time as the idea.The first time I wrote that paragraph, instead of "learn a lot about
+things that matter," I wrote "become good at some technology." But
+that prescription, though sufficient, is too narrow.  What was
+special about Brian Chesky and Joe Gebbia was not that they were
+experts in technology.  They were good at design, and perhaps even
+more importantly, they were good at organizing groups and making
+projects happen.  So you don't have to work on technology per se,
+so long as you work on problems demanding enough to stretch you.What kind of problems are those?  That is very hard to answer in
+the general case.  History is full of examples of young people who
+were working on important problems that no
+one else at the time thought were important, and in particular
+that their parents didn't think were important.  On the other hand,
+history is even fuller of examples of parents who thought their
+kids were wasting their time and who were right.  So how do you
+know when you're working on real stuff?
+[8]I know how I know.  Real problems are interesting, and I am
+self-indulgent in the sense that I always want to work on interesting
+things, even if no one else cares about them (in fact, especially
+if no one else cares about them), and find it very hard to make
+myself work on boring things, even if they're supposed to be
+important.My life is full of case after case where I worked on something just
+because it seemed interesting, and it turned out later to be useful
+in some worldly way.  Y
+Combinator itself was something I only did because it seemed
+interesting. So I seem to have some sort of internal compass that
+helps me out.  But I don't know what other people have in their
+heads. Maybe if I think more about this I can come up with heuristics
+for recognizing genuinely interesting problems, but for the moment
+the best I can offer is the hopelessly question-begging advice that
+if you have a taste for genuinely interesting problems, indulging
+it energetically is the best way to prepare yourself for a startup.
+And indeed, probably also the best way to live.
+[9]But although I can't explain in the general case what counts as an
+interesting problem, I can tell you about a large subset of them.
+If you think of technology as something that's spreading like a
+sort of fractal stain, every moving point on the edge represents
+an interesting problem.  So one guaranteed way to turn your mind
+into the type that has good startup ideas is to get yourself to the
+leading edge of some technology — to cause yourself, as Paul
+Buchheit put it, to "live in the future." When you reach that point,
+ideas that will seem to other people uncannily prescient will seem
+obvious to you.  You may not realize they're startup ideas, but
+you'll know they're something that ought to exist.For example, back at Harvard in the mid 90s a fellow grad student
+of my friends Robert and Trevor wrote his own voice over IP software.
+He didn't mean it to be a startup, and he never tried to turn it
+into one.  He just wanted to talk to his girlfriend in Taiwan without
+paying for long distance calls, and since he was an expert on
+networks it seemed obvious to him that the way to do it was turn
+the sound into packets and ship it over the Internet. He never did
+any more with his software than talk to his girlfriend, but this
+is exactly the way the best startups get started.So strangely enough the optimal thing to do in college if you want
+to be a successful startup founder is not some sort of new, vocational
+version of college focused on "entrepreneurship." It's the classic
+version of college as education for its own sake. If you want to
+start a startup after college, what you should do in college is
+learn powerful things.  And if you have genuine intellectual
+curiosity, that's what you'll naturally tend to do if you just
+follow your own inclinations.
+[10]The component of entrepreneurship that really matters is domain
+expertise.  The way to become Larry Page was to become an expert
+on search. And the way to become an expert on search was to be
+driven by genuine curiosity, not some ulterior motive.At its best, starting a startup is merely an ulterior motive for
+curiosity.  And you'll do it best if you introduce the ulterior
+motive toward the end of the process.So here is the ultimate advice for young would-be startup founders,
+boiled down to two words: just learn.
+Notes[1]
+Some founders listen more than others, and this tends to be a
+predictor of success. One of the things I
+remember about the Airbnbs during YC is how intently they listened.[2]
+In fact, this is one of the reasons startups are possible.  If
+big companies weren't plagued by internal inefficiencies, they'd
+be proportionately more effective, leaving less room for startups.[3]
+In a startup you have to spend a lot of time on schleps, but this sort of work is merely
+unglamorous, not bogus.[4]
+What should you do if your true calling is gaming the system?
+Management consulting.[5]
+The company may not be incorporated, but if you start to get
+significant numbers of users, you've started it, whether you realize
+it yet or not.[6]
+It shouldn't be that surprising that colleges can't teach
+students how to be good startup founders, because they can't teach
+them how to be good employees either.The way universities "teach" students how to be employees is to
+hand off the task to companies via internship programs.  But you
+couldn't do the equivalent thing for startups, because by definition
+if the students did well they would never come back.[7]
+Charles Darwin was 22 when he received an invitation to travel
+aboard the HMS Beagle as a naturalist.  It was only because he was
+otherwise unoccupied, to a degree that alarmed his family, that he
+could accept it. And yet if he hadn't we probably would not know
+his name.[8]
+Parents can sometimes be especially conservative in this
+department.  There are some whose definition of important problems
+includes only those on the critical path to med school.[9]
+I did manage to think of a heuristic for detecting whether you
+have a taste for interesting ideas: whether you find known boring
+ideas intolerable.  Could you endure studying literary theory, or
+working in middle management at a large company?[10]
+In fact, if your goal is to start a startup, you can stick
+even more closely to the ideal of a liberal education than past
+generations have. Back when students focused mainly on getting a
+job after college, they thought at least a little about how the
+courses they took might look to an employer.  And perhaps even
+worse, they might shy away from taking a difficult class lest they
+get a low grade, which would harm their all-important GPA.  Good
+news: users don't care what your GPA
+was.  And I've never heard of investors caring either.  Y Combinator
+certainly never asks what classes you took in college or what grades
+you got in them.
+Thanks to Sam Altman, Paul Buchheit, John Collison, Patrick
+Collison, Jessica Livingston, Robert Morris, Geoff Ralston, and
+Fred Wilson for reading drafts of this.

recipes/experimental/long-context/H2O/data/PaulGrahamEssays/bias.txt

@@ -0,0 +1,54 @@
+October 2015This will come as a surprise to a lot of people, but in some cases
+it's possible to detect bias in a selection process without knowing
+anything about the applicant pool.  Which is exciting because among
+other things it means third parties can use this technique to detect
+bias whether those doing the selecting want them to or not.You can use this technique whenever (a) you have at least
+a random sample of the applicants that were selected, (b) their
+subsequent performance is measured, and (c) the groups of
+applicants you're comparing have roughly equal distribution of ability.How does it work?  Think about what it means to be biased.  What
+it means for a selection process to be biased against applicants
+of type x is that it's harder for them to make it through.  Which
+means applicants of type x have to be better to get selected than
+applicants not of type x.
+[1]
+Which means applicants of type x
+who do make it through the selection process will outperform other
+successful applicants.  And if the performance of all the successful
+applicants is measured, you'll know if they do.Of course, the test you use to measure performance must be a valid
+one.  And in particular it must not be invalidated by the bias you're
+trying to measure.
+But there are some domains where performance can be measured, and
+in those detecting bias is straightforward. Want to know if the
+selection process was biased against some type of applicant?  Check
+whether they outperform the others.  This is not just a heuristic
+for detecting bias.  It's what bias means.For example, many suspect that venture capital firms are biased
+against female founders. This would be easy to detect: among their
+portfolio companies, do startups with female founders outperform
+those without?  A couple months ago, one VC firm (almost certainly
+unintentionally) published a study showing bias of this type. First
+Round Capital found that among its portfolio companies, startups
+with female founders outperformed
+those without by 63%. 
+[2]The reason I began by saying that this technique would come as a
+surprise to many people is that we so rarely see analyses of this
+type.  I'm sure it will come as a surprise to First Round that they
+performed one. I doubt anyone there realized that by limiting their
+sample to their own portfolio, they were producing a study not of
+startup trends but of their own biases when selecting companies.I predict we'll see this technique used more in the future.  The
+information needed to conduct such studies is increasingly available.
+Data about who applies for things is usually closely guarded by the
+organizations selecting them, but nowadays data about who gets
+selected is often publicly available to anyone who takes the trouble
+to aggregate it.
+Notes[1]
+This technique wouldn't work if the selection process looked
+for different things from different types of applicants—for
+example, if an employer hired men based on their ability but women
+based on their appearance.[2]
+As Paul Buchheit points out, First Round excluded their most 
+successful investment, Uber, from the study.  And while it 
+makes sense to exclude outliers from some types of studies, 
+studies of returns from startup investing, which is all about 
+hitting outliers, are not one of them.
+Thanks to Sam Altman, Jessica Livingston, and Geoff Ralston for reading
+drafts of this.

recipes/experimental/long-context/H2O/data/PaulGrahamEssays/boss.txt

@@ -0,0 +1,218 @@
+
+
+Want to start a startup?  Get funded by
+Y Combinator.
+
+
+
+
+March 2008, rev. June 2008Technology tends to separate normal from natural.  Our bodies
+weren't designed to eat the foods that people in rich countries eat, or
+to get so little exercise.  
+There may be a similar problem with the way we work: 
+a normal job may be as bad for us intellectually as white flour
+or sugar is for us physically.I began to suspect this after spending several years working 
+with startup founders.  I've now worked with over 200 of them, and I've
+noticed a definite difference between programmers working on their
+own startups and those working for large organizations.
+I wouldn't say founders seem happier, necessarily;
+starting a startup can be very stressful. Maybe the best way to put
+it is to say that they're happier in the sense that your body is
+happier during a long run than sitting on a sofa eating
+doughnuts.Though they're statistically abnormal, startup founders seem to be
+working in a way that's more natural for humans.I was in Africa last year and saw a lot of animals in the wild that
+I'd only seen in zoos before. It was remarkable how different they
+seemed. Particularly lions. Lions in the wild seem about ten times
+more alive. They're like different animals. I suspect that working
+for oneself feels better to humans in much the same way that living
+in the wild must feel better to a wide-ranging predator like a lion.
+Life in a zoo is easier, but it isn't the life they were designed
+for.
+TreesWhat's so unnatural about working for a big company?  The root of
+the problem is that humans weren't meant to work in such large
+groups.Another thing you notice when you see animals in the wild is that
+each species thrives in groups of a certain size.  A herd of impalas
+might have 100 adults; baboons maybe 20; lions rarely 10.  Humans
+also seem designed to work in groups, and what I've read about
+hunter-gatherers accords with research on organizations and my own
+experience to suggest roughly what the ideal size is: groups of 8
+work well; by 20 they're getting hard to manage; and a group of 50
+is really unwieldy.
+[1]
+Whatever the upper limit is, we are clearly not meant to work in
+groups of several hundred.  And yet—for reasons having more
+to do with technology than human nature—a great many people
+work for companies with hundreds or thousands of employees.Companies know groups that large wouldn't work, so they divide
+themselves into units small enough to work together.  But to
+coordinate these they have to introduce something new: bosses.These smaller groups are always arranged in a tree structure.  Your
+boss is the point where your group attaches to the tree.  But when
+you use this trick for dividing a large group into smaller ones,
+something strange happens that I've never heard anyone mention
+explicitly.  In the group one level up from yours, your boss
+represents your entire group.  A group of 10 managers is not merely
+a group of 10 people working together in the usual way.  It's really
+a group of groups.  Which means for a group of 10 managers to work
+together as if they were simply a group of 10 individuals, the group
+working for each manager would have to work as if they were a single
+person—the workers and manager would each share only one
+person's worth of freedom between them.In practice a group of people are never able to act as if they were
+one person.  But in a large organization divided into groups in
+this way, the pressure is always in that direction.  Each group
+tries its best to work as if it were the small group of individuals
+that humans were designed to work in.  That was the point of creating
+it.  And when you propagate that constraint, the result is that
+each person gets freedom of action in inverse proportion to the
+size of the entire tree.
+[2]Anyone who's worked for a large organization has felt this.  You
+can feel the difference between working for a company with 100
+employees and one with 10,000, even if your group has only 10 people.
+Corn SyrupA group of 10 people within a large organization is a kind of fake
+tribe.  The number of people you interact with is about right.  But
+something is missing: individual initiative.  Tribes of hunter-gatherers
+have much more freedom.  The leaders have a little more power than other
+members of the tribe, but they don't generally tell them what to
+do and when the way a boss can.It's not your boss's fault.  The real problem is that in the group
+above you in the hierarchy, your entire group is one virtual person.
+Your boss is just the way that constraint is imparted to you.So working in a group of 10 people within a large organization feels
+both right and wrong at the same time.   On the surface it feels
+like the kind of group you're meant to work in, but something major
+is missing.  A job at a big company is like high fructose corn
+syrup: it has some of the qualities of things you're meant to like,
+but is disastrously lacking in others.Indeed, food is an excellent metaphor to explain what's wrong with
+the usual sort of job.For example, working for a big company is the default thing to do,
+at least for programmers.  How bad could it be?  Well, food shows
+that pretty clearly.  If you were dropped at a random point in
+America today, nearly all the food around you would be bad for you.
+Humans were not designed to eat white flour, refined sugar, high
+fructose corn syrup, and hydrogenated vegetable oil.  And yet if
+you analyzed the contents of the average grocery store you'd probably
+find these four ingredients accounted for most of the calories.
+"Normal" food is terribly bad for you.  The only people who eat
+what humans were actually designed to eat are a few Birkenstock-wearing
+weirdos in Berkeley.If "normal" food is so bad for us, why is it so common?  There are
+two main reasons. One is that it has more immediate appeal.  You
+may feel lousy an hour after eating that pizza, but eating the first
+couple bites feels great.  The other is economies of scale.
+Producing junk food scales; producing fresh vegetables doesn't.
+Which means (a) junk food can be very cheap, and (b) it's worth
+spending a lot to market it.If people have to choose between something that's cheap, heavily
+marketed, and appealing in the short term, and something that's
+expensive, obscure, and appealing in the long term, which do you
+think most will choose?It's the same with work.  The average MIT graduate wants to work
+at Google or Microsoft, because it's a recognized brand, it's safe,
+and they'll get paid a good salary right away.  It's the job
+equivalent of the pizza they had for lunch.  The drawbacks will
+only become apparent later, and then only in a vague sense of
+malaise.And founders and early employees of startups, meanwhile, are like
+the Birkenstock-wearing weirdos of Berkeley:  though a tiny minority
+of the population, they're the ones living as humans are meant to.
+In an artificial world, only extremists live naturally.
+ProgrammersThe restrictiveness of big company jobs is particularly hard on
+programmers, because the essence of programming is to build new
+things.  Sales people make much the same pitches every day; support
+people answer much the same questions; but once you've written a
+piece of code you don't need to write it again.  So a programmer
+working as programmers are meant to is always making new things.
+And when you're part of an organization whose structure gives each
+person freedom in inverse proportion to the size of the tree, you're
+going to face resistance when you do something new.This seems an inevitable consequence of bigness.  It's true even
+in the smartest companies.  I was talking recently to a founder who
+considered starting a startup right out of college, but went to
+work for Google instead because he thought he'd learn more there.
+He didn't learn as much as he expected.  Programmers learn by doing,
+and most of the things he wanted to do, he couldn't—sometimes
+because the company wouldn't let him, but often because the company's
+code wouldn't let him.  Between the drag of legacy code, the overhead
+of doing development in such a large organization, and the restrictions
+imposed by interfaces owned by other groups, he could only try a
+fraction of the things he would have liked to.  He said he has
+learned much more in his own startup, despite the fact that he has
+to do all the company's errands as well as programming, because at
+least when he's programming he can do whatever he wants.An obstacle downstream propagates upstream.  If you're not allowed
+to implement new ideas, you stop having them.  And vice versa: when
+you can do whatever you want, you have more ideas about what to do.
+So working for yourself makes your brain more powerful in the same
+way a low-restriction exhaust system makes an engine more powerful.Working for yourself doesn't have to mean starting a startup, of
+course.  But a programmer deciding between a regular job at a big
+company and their own startup is probably going to learn more doing
+the startup.You can adjust the amount of freedom you get by scaling the size
+of company you work for.  If you start the company, you'll have the
+most freedom.  If you become one of the first 10 employees you'll
+have almost as much freedom as the founders.  Even a company with
+100 people will feel different from one with 1000.Working for a small company doesn't ensure freedom.  The tree
+structure of large organizations sets an upper bound on freedom,
+not a lower bound.  The head of a small company may still choose
+to be a tyrant.  The point is that a large organization is compelled
+by its structure to be one.
+ConsequencesThat has real consequences for both organizations and individuals.
+One is that companies will inevitably slow down as they grow larger,
+no matter how hard they try to keep their startup mojo.  It's a
+consequence of the tree structure that every large organization is
+forced to adopt.Or rather, a large organization could only avoid slowing down if
+they avoided tree structure.  And since human nature limits the
+size of group that can work together, the only way I can imagine
+for larger groups to avoid tree structure would be to have no
+structure: to have each group actually be independent, and to work
+together the way components of a market economy do.That might be worth exploring.  I suspect there are already some
+highly partitionable businesses that lean this way.  But I don't
+know any technology companies that have done it.There is one thing companies can do short of structuring themselves
+as sponges:  they can stay small.  If I'm right, then it really
+pays to keep a company as small as it can be at every stage.
+Particularly a technology company.  Which means it's doubly important
+to hire the best people.  Mediocre hires hurt you twice: they get
+less done, but they also make you big, because you need more of
+them to solve a given problem.For individuals the upshot is the same: aim small.  It will always
+suck to work for large organizations, and the larger the organization,
+the more it will suck.In an essay I wrote a couple years ago 
+I advised graduating seniors
+to work for a couple years for another company before starting their
+own.  I'd modify that now.  Work for another company if you want
+to, but only for a small one, and if you want to start your own
+startup, go ahead.The reason I suggested college graduates not start startups immediately
+was that I felt most would fail.  And they will.  But ambitious
+programmers are better off doing their own thing and failing than
+going to work at a big company.  Certainly they'll learn more.  They
+might even be better off financially.  A lot of people in their
+early twenties get into debt, because their expenses grow even
+faster than the salary that seemed so high when they left school.
+At least if you start a startup and fail your net worth will be
+zero rather than negative.  
+[3]We've now funded so many different types of founders that we have
+enough data to see patterns, and there seems to be no benefit from
+working for a big company.  The people who've worked for a few years
+do seem better than the ones straight out of college, but only
+because they're that much older.The people who come to us from big companies often seem kind of
+conservative.  It's hard to say how much is because big companies
+made them that way, and how much is the natural conservatism that
+made them work for the big companies in the first place.  But
+certainly a large part of it is learned.  I know because I've seen
+it burn off.Having seen that happen so many times is one of the things that
+convinces me that working for oneself, or at least for a small
+group, is the natural way for programmers to live.  Founders arriving
+at Y Combinator often have the downtrodden air of refugees.  Three
+months later they're transformed: they have so much more 
+confidence
+that they seem as if they've grown several inches taller. 
+[4]
+Strange as this sounds, they seem both more worried and happier at the same
+time.  Which is exactly how I'd describe the way lions seem in the
+wild.Watching employees get transformed into founders makes it clear
+that the difference between the two is due mostly to environment—and
+in particular that the environment in big companies is toxic to
+programmers.   In the first couple weeks of working on their own
+startup they seem to come to life, because finally they're working
+the way people are meant to.Notes[1]
+When I talk about humans being meant or designed to live a
+certain way, I mean by evolution.[2]
+It's not only the leaves who suffer.  The constraint propagates
+up as well as down.  So managers are constrained too; instead of
+just doing things, they have to act through subordinates.[3]
+Do not finance your startup with credit cards.  Financing a
+startup with debt is usually a stupid move, and credit card debt
+stupidest of all.  Credit card debt is a bad idea, period.  It is
+a trap set by evil companies for the desperate and the foolish.[4]
+The founders we fund used to be younger (initially we encouraged
+undergrads to apply), and the first couple times I saw this I used
+to wonder if they were actually getting physically taller.Thanks to Trevor Blackwell, Ross Boucher, Aaron Iba, Abby
+Kirigin, Ivan Kirigin, Jessica Livingston, and Robert Morris for
+reading drafts of this.

recipes/experimental/long-context/H2O/data/PaulGrahamEssays/copy.txt

@@ -0,0 +1,81 @@
+July 2006
+When I was in high school I spent a lot of time imitating bad
+writers.  What we studied in English classes was mostly fiction,
+so I assumed that was the highest form of writing.  Mistake number
+one.  The stories that seemed to be most admired were ones in which
+people suffered in complicated ways.  Anything funny or
+gripping was ipso facto suspect, unless it was old enough to be hard to
+understand, like Shakespeare or Chaucer.  Mistake number two.  The
+ideal medium seemed the short story, which I've since learned had
+quite a brief life, roughly coincident with the peak of magazine
+publishing.  But since their size made them perfect for use in
+high school classes, we read a lot of them, which gave us the
+impression the short story was flourishing.  Mistake number three.
+And because they were so short, nothing really had to happen; you
+could just show a randomly truncated slice of life, and that was
+considered advanced.  Mistake number four.  The result was that I
+wrote a lot of stories in which nothing happened except that someone
+was unhappy in a way that seemed deep.For most of college I was a philosophy major.  I was very impressed
+by the papers published in philosophy journals.  They were so
+beautifully typeset, and their tone was just captivating—alternately
+casual and buffer-overflowingly technical.  A fellow would be walking
+along a street and suddenly modality qua modality would spring upon
+him.  I didn't ever quite understand these papers, but I figured
+I'd get around to that later, when I had time to reread them more
+closely.  In the meantime I tried my best to imitate them.  This
+was, I can now see, a doomed undertaking, because they weren't
+really saying anything.  No philosopher ever refuted another, for
+example, because no one said anything definite enough to refute.
+Needless to say, my imitations didn't say anything either.In grad school I was still wasting time imitating the wrong things.
+There was then a fashionable type of program called an expert system,
+at the core of which was something called an inference engine.  I
+looked at what these things did and thought "I could write that in
+a thousand lines of code."  And yet eminent professors were writing
+books about them, and startups were selling them for a year's salary
+a copy.  What an opportunity, I thought; these impressive things
+seem easy to me; I must be pretty sharp.  Wrong.  It was simply a
+fad.  The books the professors wrote about expert systems are now
+ignored.  They were not even on a path to anything interesting.
+And the customers paying so much for them were largely the same
+government agencies that paid thousands for screwdrivers and toilet
+seats.How do you avoid copying the wrong things?  Copy only what you
+genuinely like.  That would have saved me in all three cases.  I
+didn't enjoy the short stories we had to read in English classes;
+I didn't learn anything from philosophy papers; I didn't use expert
+systems myself.  I believed these things were good because they
+were admired.It can be hard to separate the things you like from the things
+you're impressed with.  One trick is to ignore presentation.  Whenever
+I see a painting impressively hung in a museum, I ask myself: how
+much would I pay for this if I found it at a garage sale, dirty and
+frameless, and with no idea who painted it?  If you walk around a
+museum trying this experiment, you'll find you get some truly
+startling results.  Don't ignore this data point just because it's
+an outlier.Another way to figure out what you like is to look at what you enjoy
+as guilty pleasures.  Many things people like, especially if they're
+young and ambitious, they like largely for the feeling of virtue
+in liking them.  99% of people reading Ulysses are thinking
+"I'm reading Ulysses" as they do it. A guilty pleasure is
+at least a pure one.  What do you read when you don't feel up to being
+virtuous?  What kind of book do you read and feel sad that there's
+only half of it left, instead of being impressed that you're half
+way through?  That's what you really like.Even when you find genuinely good things to copy, there's another
+pitfall to be avoided.  Be careful to copy what makes them good,
+rather than their flaws.  It's easy to be drawn into imitating
+flaws, because they're easier to see, and of course easier to copy
+too.  For example, most painters in the eighteenth and nineteenth
+centuries used brownish colors.  They were imitating the great
+painters of the Renaissance, whose paintings by that time were brown
+with dirt.  Those paintings have since been cleaned, revealing
+brilliant colors; their imitators are of course still brown.It was painting, incidentally, that cured me of copying the wrong
+things.  Halfway through grad school I decided I wanted to try being
+a painter, and the art world was so manifestly corrupt that it
+snapped the leash of credulity.  These people made philosophy
+professors seem as scrupulous as mathematicians.  It was so clearly
+a choice of doing good work xor being an insider that I was forced
+to see the distinction.  It's there to some degree in almost every
+field, but I had till then managed to avoid facing it.That was one of the most valuable things I learned from painting:
+you have to figure out for yourself what's 
+good.  You can't trust
+authorities. They'll lie to you on this one.
+
+Comment on this essay.

recipes/experimental/long-context/H2O/data/PaulGrahamEssays/corpdev.txt

@@ -0,0 +1,107 @@
+January 2015Corporate Development, aka corp dev, is the group within companies
+that buys other companies. If you're talking to someone from corp
+dev, that's why, whether you realize it yet or not.It's usually a mistake to talk to corp dev unless (a) you want to
+sell your company right now and (b) you're sufficiently likely to
+get an offer at an acceptable price.  In practice that means startups
+should only talk to corp dev when they're either doing really well
+or really badly.  If you're doing really badly, meaning the company
+is about to die, you may as well talk to them, because you have
+nothing to lose. And if you're doing really well, you can safely
+talk to them, because you both know the price will have to be high,
+and if they show the slightest sign of wasting your time, you'll
+be confident enough to tell them to get lost.The danger is to companies in the middle.  Particularly to young
+companies that are growing fast, but haven't been doing it for long
+enough to have grown big yet.  It's usually a mistake for a promising
+company less than a year old even to talk to corp dev.But it's a mistake founders constantly make.  When someone from
+corp dev wants to meet, the founders tell themselves they should
+at least find out what they want.  Besides, they don't want to
+offend Big Company by refusing to meet.Well, I'll tell you what they want.  They want to talk about buying
+you.  That's what the title "corp dev" means.   So before agreeing
+to meet with someone from corp dev, ask yourselves, "Do we want to
+sell the company right now?"  And if the answer is no, tell them
+"Sorry, but we're focusing on growing the company."  They won't be
+offended.  And certainly the founders of Big Company won't be
+offended. If anything they'll think more highly of you.  You'll
+remind them of themselves.  They didn't sell either; that's why
+they're in a position now to buy other companies.
+[1]Most founders who get contacted by corp dev already know what it
+means.  And yet even when they know what corp dev does and know
+they don't want to sell, they take the meeting.  Why do they do it?
+The same mix of denial and wishful thinking that underlies most
+mistakes founders make. It's flattering to talk to someone who wants
+to buy you.  And who knows, maybe their offer will be surprisingly
+high.  You should at least see what it is, right?No.  If they were going to send you an offer immediately by email,
+sure, you might as well open it.  But that is not how conversations
+with corp dev work.  If you get an offer at all, it will be at the
+end of a long and unbelievably distracting process.  And if the
+offer is surprising, it will be surprisingly low.Distractions are the thing you can least afford in a startup.  And
+conversations with corp dev are the worst sort of distraction,
+because as well as consuming your attention they undermine your
+morale.  One of the tricks to surviving a grueling process is not
+to stop and think how tired you are.  Instead you get into a sort
+of flow. 
+[2]
+Imagine what it would do to you if at mile 20 of a
+marathon, someone ran up beside you and said "You must feel really
+tired.  Would you like to stop and take a rest?"  Conversations
+with corp dev are like that but worse, because the suggestion of
+stopping gets combined in your mind with the imaginary high price
+you think they'll offer.And then you're really in trouble.  If they can, corp dev people
+like to turn the tables on you. They like to get you to the point
+where you're trying to convince them to buy instead of them trying
+to convince you to sell.  And surprisingly often they succeed.This is a very slippery slope, greased with some of the most powerful
+forces that can work on founders' minds, and attended by an experienced
+professional whose full time job is to push you down it.Their tactics in pushing you down that slope are usually fairly
+brutal. Corp dev people's whole job is to buy companies, and they
+don't even get to choose which.  The only way their performance is
+measured is by how cheaply they can buy you, and the more ambitious
+ones will stop at nothing to achieve that. For example, they'll
+almost always start with a lowball offer, just to see if you'll
+take it. Even if you don't, a low initial offer will demoralize you
+and make you easier to manipulate.And that is the most innocent of their tactics. Just wait till
+you've agreed on a price and think you have a done deal, and then
+they come back and say their boss has vetoed the deal and won't do
+it for more than half the agreed upon price. Happens all the time.
+If you think investors can behave badly, it's nothing compared to
+what corp dev people can do.  Even corp dev people at companies
+that are otherwise benevolent.I remember once complaining to a
+friend at Google about some nasty trick their corp dev people had
+pulled on a YC startup."What happened to Don't be Evil?" I asked."I don't think corp dev got the memo," he replied.The tactics you encounter in M&A conversations can be like nothing
+you've experienced in the otherwise comparatively 
+upstanding world
+of Silicon Valley.  It's as if a chunk of genetic material from the
+old-fashioned robber baron business world got incorporated into the
+startup world.
+[3]The simplest way to protect yourself is to use the trick that John
+D. Rockefeller, whose grandfather was an alcoholic, used to protect
+himself from becoming one.  He once told a Sunday school class
+
+  Boys, do you know why I never became a drunkard?  Because I never
+  took the first drink.
+
+Do you want to sell your company right now?  Not eventually, right
+now.  If not, just don't take the first meeting.  They won't be
+offended.  And you in turn will be guaranteed to be spared one of
+the worst experiences that can happen to a startup.If you do want to sell, there's another set of 
+techniques
+ for doing
+that.  But the biggest mistake founders make in dealing with corp
+dev is not doing a bad job of talking to them when they're ready
+to, but talking to them before they are.  So if you remember only
+the title of this essay, you already know most of what you need to
+know about M&A in the first year.Notes[1]
+I'm not saying you should never sell.  I'm saying you should
+be clear in your own mind about whether you want to sell or not,
+and not be led by manipulation or wishful thinking into trying to
+sell earlier than you otherwise would have.[2]
+In a startup, as in most competitive sports, the task at hand
+almost does this for you; you're too busy to feel tired.  But when
+you lose that protection, e.g. at the final whistle, the fatigue
+hits you like a wave.  To talk to corp dev is to let yourself feel
+it mid-game.[3]
+To be fair, the apparent misdeeds of corp dev people are magnified
+by the fact that they function as the face of a large organization
+that often doesn't know its own mind.  Acquirers can be surprisingly
+indecisive about acquisitions, and their flakiness is indistinguishable
+from dishonesty by the time it filters down to you.Thanks to Marc Andreessen, Jessica Livingston, Geoff
+Ralston, and Qasar Younis for reading drafts of this.

recipes/experimental/long-context/H2O/data/PaulGrahamEssays/desres.txt

@@ -0,0 +1,234 @@
+January 2003(This article is derived from a keynote talk at the fall 2002 meeting
+of NEPLS.)Visitors to this country are often surprised to find that
+Americans like to begin a conversation by asking "what do you do?"
+I've never liked this question.  I've rarely had a
+neat answer to it.  But I think I have finally solved the problem.
+Now, when someone asks me what I do, I look them straight
+in the eye and say "I'm designing a 
+new dialect of Lisp."   
+I recommend this answer to anyone who doesn't like being asked what
+they do.  The conversation will turn immediately to other topics.I don't consider myself to be doing research on programming languages.
+I'm just designing one, in the same way that someone might design
+a building or a chair or a new typeface.
+I'm not trying to discover anything new.  I just want
+to make a language that will be good to program in.  In some ways,
+this assumption makes life a lot easier.The difference between design and research seems to be a question
+of new versus good.  Design doesn't have to be new, but it has to  
+be good.  Research doesn't have to be good, but it has to be new.
+I think these two paths converge at the top: the best design
+surpasses its predecessors by using new ideas, and the best research
+solves problems that are not only new, but actually worth solving.
+So ultimately we're aiming for the same destination, just approaching
+it from different directions.What I'm going to talk about today is what your target looks like
+from the back.  What do you do differently when you treat
+programming languages as a design problem instead of a research topic?The biggest difference is that you focus more on the user.
+Design begins by asking, who is this
+for and what do they need from it?  A good architect,
+for example, does not begin by creating a design that he then
+imposes on the users, but by studying the intended users and figuring
+out what they need.Notice I said "what they need," not "what they want."  I don't mean
+to give the impression that working as a designer means working as 
+a sort of short-order cook, making whatever the client tells you
+to.  This varies from field to field in the arts, but
+I don't think there is any field in which the best work is done by
+the people who just make exactly what the customers tell them to.The customer is always right in
+the sense that the measure of good design is how well it works
+for the user.  If you make a novel that bores everyone, or a chair
+that's horribly uncomfortable to sit in, then you've done a bad
+job, period.  It's no defense to say that the novel or the chair  
+is designed according to the most advanced theoretical principles.And yet, making what works for the user doesn't mean simply making
+what the user tells you to.  Users don't know what all the choices
+are, and are often mistaken about what they really want.The answer to the paradox, I think, is that you have to design
+for the user, but you have to design what the user needs, not simply  
+what he says he wants.
+It's much like being a doctor.  You can't just treat a patient's
+symptoms.  When a patient tells you his symptoms, you have to figure
+out what's actually wrong with him, and treat that.This focus on the user is a kind of axiom from which most of the
+practice of good design can be derived, and around which most design
+issues center.If good design must do what the user needs, who is the user?  When
+I say that design must be for users, I don't mean to imply that good 
+design aims at some kind of  
+lowest common denominator.  You can pick any group of users you
+want.  If you're designing a tool, for example, you can design it
+for anyone from beginners to experts, and what's good design
+for one group might be bad for another.  The point
+is, you have to pick some group of users.  I don't think you can
+even talk about good or bad design except with
+reference to some intended user.You're most likely to get good design if the intended users include
+the designer himself.  When you design something
+for a group that doesn't include you, it tends to be for people
+you consider to be less sophisticated than you, not more sophisticated.That's a problem, because looking down on the user, however benevolently,
+seems inevitably to corrupt the designer.
+I suspect that very few housing
+projects in the US were designed by architects who expected to live
+in them.   You can see the same thing
+in programming languages.  C, Lisp, and Smalltalk were created for
+their own designers to use.  Cobol, Ada, and Java, were created   
+for other people to use.If you think you're designing something for idiots, the odds are
+that you're not designing something good, even for idiots.
+Even if you're designing something for the most sophisticated
+users, though, you're still designing for humans.  It's different 
+in research.  In math you
+don't choose abstractions because they're
+easy for humans to understand; you choose whichever make the
+proof shorter.  I think this is true for the sciences generally.
+Scientific ideas are not meant to be ergonomic.Over in the arts, things are very different.  Design is
+all about people.  The human body is a strange
+thing, but when you're designing a chair,
+that's what you're designing for, and there's no way around it.
+All the arts have to pander to the interests and limitations
+of humans.   In painting, for example, all other things being
+equal a painting with people in it will be more interesting than
+one without.  It is not merely an accident of history that
+the great paintings of the Renaissance are all full of people.
+If they hadn't been, painting as a medium wouldn't have the prestige
+that it does.Like it or not, programming languages are also for people,
+and I suspect the human brain is just as lumpy and idiosyncratic
+as the human body.  Some ideas are easy for people to grasp
+and some aren't.  For example, we seem to have a very limited
+capacity for dealing with detail.  It's this fact that makes
+programing languages a good idea in the first place; if we
+could handle the detail, we could just program in machine
+language.Remember, too, that languages are not
+primarily a form for finished programs, but something that
+programs have to be developed in.  Anyone in the arts could
+tell you that you might want different mediums for the
+two situations.  Marble, for example, is a nice, durable
+medium for finished ideas, but a hopelessly inflexible one
+for developing new ideas.A program, like a proof,
+is a pruned version of a tree that in the past has had
+false starts branching off all over it.  So the test of
+a language is not simply how clean the finished program looks
+in it, but how clean the path to the finished program was.
+A design choice that gives you elegant finished programs
+may not give you an elegant design process.  For example, 
+I've written a few macro-defining macros full of nested
+backquotes that look now like little gems, but writing them
+took hours of the ugliest trial and error, and frankly, I'm still
+not entirely sure they're correct.We often act as if the test of a language were how good
+finished programs look in it.
+It seems so convincing when you see the same program
+written in two languages, and one version is much shorter.
+When you approach the problem from the direction of the
+arts, you're less likely to depend on this sort of
+test.  You don't want to end up with a programming
+language like marble.For example, it is a huge win in developing software to
+have an interactive toplevel, what in Lisp is called a
+read-eval-print loop.  And when you have one this has
+real effects on the design of the language.  It would not
+work well for a language where you have to declare
+variables before using them, for example.  When you're
+just typing expressions into the toplevel, you want to be 
+able to set x to some value and then start doing things
+to x.  You don't want to have to declare the type of x
+first.  You may dispute either of the premises, but if
+a language has to have a toplevel to be convenient, and
+mandatory type declarations are incompatible with a
+toplevel, then no language that makes type declarations  
+mandatory could be convenient to program in.In practice, to get good design you have to get close, and stay
+close, to your users.  You have to calibrate your ideas on actual
+users constantly, especially in the beginning.  One of the reasons
+Jane Austen's novels are so good is that she read them out loud to
+her family.  That's why she never sinks into self-indulgently arty
+descriptions of landscapes,
+or pretentious philosophizing.  (The philosophy's there, but it's
+woven into the story instead of being pasted onto it like a label.)
+If you open an average "literary" novel and imagine reading it out loud
+to your friends as something you'd written, you'll feel all too
+keenly what an imposition that kind of thing is upon the reader.In the software world, this idea is known as Worse is Better.
+Actually, there are several ideas mixed together in the concept of
+Worse is Better, which is why people are still arguing about
+whether worse
+is actually better or not.  But one of the main ideas in that
+mix is that if you're building something new, you should get a
+prototype in front of users as soon as possible.The alternative approach might be called the Hail Mary strategy.
+Instead of getting a prototype out quickly and gradually refining
+it, you try to create the complete, finished, product in one long
+touchdown pass.  As far as I know, this is a
+recipe for disaster.  Countless startups destroyed themselves this
+way during the Internet bubble.  I've never heard of a case
+where it worked.What people outside the software world may not realize is that
+Worse is Better is found throughout the arts.
+In drawing, for example, the idea was discovered during the
+Renaissance.  Now almost every drawing teacher will tell you that
+the right way to get an accurate drawing is not to
+work your way slowly around the contour of an object, because errors will
+accumulate and you'll find at the end that the lines don't meet.
+Instead you should draw a few quick lines in roughly the right place,
+and then gradually refine this initial sketch.In most fields, prototypes
+have traditionally been made out of different materials.
+Typefaces to be cut in metal were initially designed  
+with a brush on paper.  Statues to be cast in bronze   
+were modelled in wax.  Patterns to be embroidered on tapestries
+were drawn on paper with ink wash.  Buildings to be
+constructed from stone were tested on a smaller scale in wood.What made oil paint so exciting, when it
+first became popular in the fifteenth century, was that you
+could actually make the finished work from the prototype.
+You could make a preliminary drawing if you wanted to, but you
+weren't held to it; you could work out all the details, and
+even make major changes, as you finished the painting.You can do this in software too.  A prototype doesn't have to
+be just a model; you can refine it into the finished product.
+I think you should always do this when you can.  It lets you
+take advantage of new insights you have along the way.  But
+perhaps even more important, it's good for morale.Morale is key in design.  I'm surprised people
+don't talk more about it.  One of my first
+drawing teachers told me: if you're bored when you're
+drawing something, the drawing will look boring.
+For example, suppose you have to draw a building, and you
+decide to draw each brick individually.  You can do this
+if you want, but if you get bored halfway through and start
+making the bricks mechanically instead of observing each one,   
+the drawing will look worse than if you had merely suggested
+the bricks.Building something by gradually refining a prototype is good
+for morale because it keeps you engaged.  In software, my  
+rule is: always have working code.  If you're writing
+something that you'll be able to test in an hour, then you
+have the prospect of an immediate reward to motivate you.
+The same is true in the arts, and particularly in oil painting.
+Most painters start with a blurry sketch and gradually
+refine it.
+If you work this way, then in principle
+you never have to end the day with something that actually
+looks unfinished.  Indeed, there is even a saying among
+painters: "A painting is never finished, you just stop
+working on it."  This idea will be familiar to anyone who
+has worked on software.Morale is another reason that it's hard to design something
+for an unsophisticated user.   It's hard to stay interested in
+something you don't like yourself.  To make something  
+good, you have to be thinking, "wow, this is really great,"
+not "what a piece of shit; those fools will love it."Design means making things for humans.  But it's not just the
+user who's human.  The designer is human too.Notice all this time I've been talking about "the designer."
+Design usually has to be under the control of a single person to
+be any good.   And yet it seems to be possible for several people
+to collaborate on a research project.  This seems to
+me one of the most interesting differences between research and
+design.There have been famous instances of collaboration in the arts,
+but most of them seem to have been cases of molecular bonding rather
+than nuclear fusion.  In an opera it's common for one person to
+write the libretto and another to write the music.   And during the Renaissance, 
+journeymen from northern
+Europe were often employed to do the landscapes in the
+backgrounds of Italian paintings.  But these aren't true collaborations.
+They're more like examples of Robert Frost's
+"good fences make good neighbors."  You can stick instances
+of good design together, but within each individual project,
+one person has to be in control.I'm not saying that good design requires that one person think
+of everything.  There's nothing more valuable than the advice
+of someone whose judgement you trust.  But after the talking is
+done, the decision about what to do has to rest with one person.Why is it that research can be done by collaborators and  
+design can't?  This is an interesting question.  I don't 
+know the answer.  Perhaps,
+if design and research converge, the best research is also
+good design, and in fact can't be done by collaborators.
+A lot of the most famous scientists seem to have worked alone.
+But I don't know enough to say whether there
+is a pattern here.  It could be simply that many famous scientists
+worked when collaboration was less common.Whatever the story is in the sciences, true collaboration
+seems to be vanishingly rare in the arts.  Design by committee is a
+synonym for bad design.  Why is that so?  Is there some way to
+beat this limitation?I'm inclined to think there isn't-- that good design requires
+a dictator.  One reason is that good design has to   
+be all of a piece.  Design is not just for humans, but
+for individual humans.  If a design represents an idea that  
+fits in one person's head, then the idea will fit in the user's
+head too.Related:

recipes/experimental/long-context/H2O/data/PaulGrahamEssays/diff.txt

@@ -0,0 +1,73 @@
+December 2001 (rev. May 2002)
+
+(This article came about in response to some questions on
+the LL1 mailing list.  It is now
+incorporated in Revenge of the Nerds.)When McCarthy designed Lisp in the late 1950s, it was
+a radical departure from existing languages,
+the most important of which was Fortran.Lisp embodied nine new ideas:
+1. Conditionals.  A conditional is an if-then-else
+construct.  We take these for granted now.  They were 
+invented
+by McCarthy in the course of developing Lisp. 
+(Fortran at that time only had a conditional
+goto, closely based on the branch instruction in the 
+underlying hardware.)  McCarthy, who was on the Algol committee, got
+conditionals into Algol, whence they spread to most other
+languages.2. A function type. In Lisp, functions are first class 
+objects-- they're a data type just like integers, strings,
+etc, and have a literal representation, can be stored in variables,
+can be passed as arguments, and so on.3. Recursion.  Recursion existed as a mathematical concept
+before Lisp of course, but Lisp was the first programming language to support
+it.  (It's arguably implicit in making functions first class
+objects.)4. A new concept of variables.  In Lisp, all variables
+are effectively pointers. Values are what
+have types, not variables, and assigning or binding
+variables means copying pointers, not what they point to.5. Garbage-collection.6. Programs composed of expressions. Lisp programs are 
+trees of expressions, each of which returns a value.  
+(In some Lisps expressions
+can return multiple values.)  This is in contrast to Fortran
+and most succeeding languages, which distinguish between
+expressions and statements.It was natural to have this
+distinction in Fortran because (not surprisingly in a language
+where the input format was punched cards) the language was
+line-oriented.  You could not nest statements.  And
+so while you needed expressions for math to work, there was
+no point in making anything else return a value, because
+there could not be anything waiting for it.This limitation
+went away with the arrival of block-structured languages,
+but by then it was too late. The distinction between
+expressions and statements was entrenched.  It spread from 
+Fortran into Algol and thence to both their descendants.When a language is made entirely of expressions, you can
+compose expressions however you want.  You can say either
+(using Arc syntax)(if foo (= x 1) (= x 2))or(= x (if foo 1 2))7. A symbol type.  Symbols differ from strings in that
+you can test equality by comparing a pointer.8. A notation for code using trees of symbols.9. The whole language always available.  
+There is
+no real distinction between read-time, compile-time, and runtime.
+You can compile or run code while reading, read or run code
+while compiling, and read or compile code at runtime.Running code at read-time lets users reprogram Lisp's syntax;
+running code at compile-time is the basis of macros; compiling
+at runtime is the basis of Lisp's use as an extension
+language in programs like Emacs; and reading at runtime
+enables programs to communicate using s-expressions, an
+idea recently reinvented as XML.
+When Lisp was first invented, all these ideas were far
+removed from ordinary programming practice, which was
+dictated largely by the hardware available in the late 1950s.Over time, the default language, embodied
+in a succession of popular languages, has
+gradually evolved toward Lisp.  1-5 are now widespread.
+6 is starting to appear in the mainstream.
+Python has a form of 7, though there doesn't seem to be
+any syntax for it.  
+8, which (with 9) is what makes Lisp macros
+possible, is so far still unique to Lisp,
+perhaps because (a) it requires those parens, or something 
+just as bad, and (b) if you add that final increment of power, 
+you can no 
+longer claim to have invented a new language, but only
+to have designed a new dialect of Lisp ; -)Though useful to present-day programmers, it's
+strange to describe Lisp in terms of its
+variation from the random expedients other languages
+adopted.  That was not, probably, how McCarthy
+thought of it.  Lisp wasn't designed to fix the mistakes
+in Fortran; it came about more as the byproduct of an
+attempt to axiomatize computation.

recipes/experimental/long-context/H2O/data/PaulGrahamEssays/ecw.txt

@@ -0,0 +1,98 @@
+December 2014If the world were static, we could have monotonically increasing
+confidence in our beliefs.  The more (and more varied) experience
+a belief survived, the less likely it would be false.  Most people
+implicitly believe something like this about their opinions.  And
+they're justified in doing so with opinions about things that don't
+change much, like human nature.  But you can't trust your opinions
+in the same way about things that change, which could include
+practically everything else.When experts are wrong, it's often because they're experts on an
+earlier version of the world.Is it possible to avoid that?  Can you protect yourself against
+obsolete beliefs?  To some extent, yes. I spent almost a decade
+investing in early stage startups, and curiously enough protecting
+yourself against obsolete beliefs is exactly what you have to do
+to succeed as a startup investor.  Most really good startup ideas
+look like bad ideas at first, and many of those look bad specifically
+because some change in the world just switched them from bad to
+good.  I spent a lot of time learning to recognize such ideas, and
+the techniques I used may be applicable to ideas in general.The first step is to have an explicit belief in change.  People who
+fall victim to a monotonically increasing confidence in their
+opinions are implicitly concluding the world is static.  If you
+consciously remind yourself it isn't, you start to look for change.Where should one look for it?  Beyond the moderately useful
+generalization that human nature doesn't change much, the unfortunate
+fact is that change is hard to predict.  This is largely a tautology
+but worth remembering all the same: change that matters usually
+comes from an unforeseen quarter.So I don't even try to predict it.  When I get asked in interviews
+to predict the future, I always have to struggle to come up with
+something plausible-sounding on the fly, like a student who hasn't
+prepared for an exam.
+[1]
+But it's not out of laziness that I haven't
+prepared.  It seems to me that beliefs about the future are so
+rarely correct that they usually aren't worth the extra rigidity
+they impose, and that the best strategy is simply to be aggressively
+open-minded.  Instead of trying to point yourself in the right
+direction, admit you have no idea what the right direction is, and
+try instead to be super sensitive to the winds of change.It's ok to have working hypotheses, even though they may constrain
+you a bit, because they also motivate you.  It's exciting to chase
+things and exciting to try to guess answers.  But you have to be
+disciplined about not letting your hypotheses harden into anything
+more.
+[2]I believe this passive m.o. works not just for evaluating new ideas
+but also for having them.  The way to come up with new ideas is not
+to try explicitly to, but to try to solve problems and simply not
+discount weird hunches you have in the process.The winds of change originate in the unconscious minds of domain
+experts.  If you're sufficiently expert in a field, any weird idea
+or apparently irrelevant question that occurs to you is ipso facto
+worth exploring. 
+[3]
+ Within Y Combinator, when an idea is described
+as crazy, it's a compliment—in fact, on average probably a
+higher compliment than when an idea is described as good.Startup investors have extraordinary incentives for correcting
+obsolete beliefs.  If they can realize before other investors that
+some apparently unpromising startup isn't, they can make a huge
+amount of money.  But the incentives are more than just financial.
+Investors' opinions are explicitly tested: startups come to them
+and they have to say yes or no, and then, fairly quickly, they learn
+whether they guessed right.  The investors who say no to a Google
+(and there were several) will remember it for the rest of their
+lives.Anyone who must in some sense bet on ideas rather than merely
+commenting on them has similar incentives.  Which means anyone who
+wants such incentives can have them, by turning their comments into
+bets: if you write about a topic in some fairly durable and public
+form, you'll find you worry much more about getting things right
+than most people would in a casual conversation.
+[4]Another trick I've found to protect myself against obsolete beliefs
+is to focus initially on people rather than ideas. Though the nature
+of future discoveries is hard to predict, I've found I can predict
+quite well what sort of people will make them.  Good new ideas come
+from earnest, energetic, independent-minded people.Betting on people over ideas saved me countless times as an investor.
+We thought Airbnb was a bad idea, for example. But we could tell
+the founders were earnest, energetic, and independent-minded.
+(Indeed, almost pathologically so.)  So we suspended disbelief and
+funded them.This too seems a technique that should be generally applicable.
+Surround yourself with the sort of people new ideas come from.  If
+you want to notice quickly when your beliefs become obsolete, you
+can't do better than to be friends with the people whose discoveries
+will make them so.It's hard enough already not to become the prisoner of your own
+expertise, but it will only get harder, because change is accelerating.
+That's not a recent trend; change has been accelerating since the
+paleolithic era.  Ideas beget ideas.  I don't expect that to change.
+But I could be wrong.
+Notes[1]
+My usual trick is to talk about aspects of the present that
+most people haven't noticed yet.[2]
+Especially if they become well enough known that people start
+to identify them with you.  You have to be extra skeptical about
+things you want to believe, and once a hypothesis starts to be
+identified with you, it will almost certainly start to be in that
+category.[3]
+In practice "sufficiently expert" doesn't require one to be
+recognized as an expert—which is a trailing indicator in any
+case.  In many fields a year of focused work plus caring a lot would
+be enough.[4]
+Though they are public and persist indefinitely, comments on
+e.g. forums and places like Twitter seem empirically to work like
+casual conversation.  The threshold may be whether what you write
+has a title.
+Thanks to Sam Altman, Patrick Collison, and Robert Morris
+for reading drafts of this.

recipes/experimental/long-context/H2O/data/PaulGrahamEssays/founders.txt

@@ -0,0 +1,83 @@
+
+
+Want to start a startup?  Get funded by
+Y Combinator.
+
+
+
+
+October 2010
+
+(I wrote this for Forbes, who asked me to write something
+about the qualities we look for in founders.  In print they had to cut
+the last item because they didn't have room.)1. DeterminationThis has turned out to be the most important quality in startup
+founders.  We thought when we started Y Combinator that the most
+important quality would be intelligence.  That's the myth in the
+Valley. And certainly you don't want founders to be stupid.  But
+as long as you're over a certain threshold of intelligence, what
+matters most is determination.  You're going to hit a lot of
+obstacles.  You can't be the sort of person who gets demoralized
+easily.Bill Clerico and Rich Aberman of WePay 
+are a good example.  They're
+doing a finance startup, which means endless negotiations with big,
+bureaucratic companies.  When you're starting a startup that depends
+on deals with big companies to exist, it often feels like they're
+trying to ignore you out of existence.  But when Bill Clerico starts
+calling you, you may as well do what he asks, because he is not
+going away.
+2. FlexibilityYou do not however want the sort of determination implied by phrases
+like "don't give up on your dreams."  The world of startups is so
+unpredictable that you need to be able to modify your dreams on the
+fly.  The best metaphor I've found for the combination of determination
+and flexibility you need is a running back.  
+He's determined to get
+downfield, but at any given moment he may need to go sideways or
+even backwards to get there.The current record holder for flexibility may be Daniel Gross of
+Greplin.  He applied to YC with 
+some bad ecommerce idea.  We told
+him we'd fund him if he did something else.  He thought for a second,
+and said ok.  He then went through two more ideas before settling
+on Greplin.  He'd only been working on it for a couple days when
+he presented to investors at Demo Day, but he got a lot of interest.
+He always seems to land on his feet.
+3. ImaginationIntelligence does matter a lot of course.  It seems like the type
+that matters most is imagination.  It's not so important to be able
+to solve predefined problems quickly as to be able to come up with
+surprising new ideas.  In the startup world, most good ideas 
+seem
+bad initially.  If they were obviously good, someone would already
+be doing them.  So you need the kind of intelligence that produces
+ideas with just the right level of craziness.Airbnb is that kind of idea.  
+In fact, when we funded Airbnb, we
+thought it was too crazy.  We couldn't believe large numbers of
+people would want to stay in other people's places.  We funded them
+because we liked the founders so much.  As soon as we heard they'd
+been supporting themselves by selling Obama and McCain branded
+breakfast cereal, they were in.  And it turned out the idea was on
+the right side of crazy after all.
+4. NaughtinessThough the most successful founders are usually good people, they
+tend to have a piratical gleam in their eye.  They're not Goody
+Two-Shoes type good.  Morally, they care about getting the big
+questions right, but not about observing proprieties.  That's why
+I'd use the word naughty rather than evil.  They delight in 
+breaking
+rules, but not rules that matter.  This quality may be redundant
+though; it may be implied by imagination.Sam Altman of Loopt 
+is one of the most successful alumni, so we
+asked him what question we could put on the Y Combinator application
+that would help us discover more people like him.  He said to ask
+about a time when they'd hacked something to their advantage—hacked in the sense of beating the system, not breaking into
+computers.  It has become one of the questions we pay most attention
+to when judging applications.
+5. FriendshipEmpirically it seems to be hard to start a startup with just 
+one
+founder.  Most of the big successes have two or three.  And the
+relationship between the founders has to be strong.  They must
+genuinely like one another, and work well together.  Startups do
+to the relationship between the founders what a dog does to a sock:
+if it can be pulled apart, it will be.Emmett Shear and Justin Kan of Justin.tv 
+are a good example of close
+friends who work well together.  They've known each other since
+second grade.  They can practically read one another's minds.  I'm
+sure they argue, like all founders, but I have never once sensed
+any unresolved tension between them.Thanks to Jessica Livingston and Chris Steiner for reading drafts of this.

recipes/experimental/long-context/H2O/data/PaulGrahamEssays/gap.txt

@@ -0,0 +1,485 @@
+May 2004When people care enough about something to do it well, those who
+do it best tend to be far better than everyone else.  There's a
+huge gap between Leonardo and second-rate contemporaries like
+Borgognone.  You see the same gap between Raymond Chandler and the
+average writer of detective novels.  A top-ranked professional chess
+player could play ten thousand games against an ordinary club player
+without losing once.Like chess or painting or writing novels, making money is a very
+specialized skill.   But for some reason we treat this skill
+differently.  No one complains when a few people surpass all the
+rest at playing chess or writing novels, but when a few people make
+more money than the rest, we get editorials saying this is wrong.Why?  The pattern of variation seems no different than for any other
+skill.  What causes people to react so strongly when the skill is
+making money?I think there are three reasons we treat making money as different:
+the misleading model of wealth we learn as children; the disreputable
+way in which, till recently, most fortunes were accumulated; and
+the worry that great variations in income are somehow bad for
+society.  As far as I can tell, the first is mistaken, the second
+outdated, and the third empirically false.  Could it be that, in a
+modern democracy, variation in income is actually a sign of health?The Daddy Model of WealthWhen I was five I thought electricity was created by electric
+sockets.  I didn't realize there were power plants out there
+generating it.  Likewise, it doesn't occur to most kids that wealth
+is something that has to be generated.  It seems to be something
+that flows from parents.Because of the circumstances in which they encounter it, children
+tend to misunderstand wealth.  They confuse it with money.  They
+think that there is a fixed amount of it.  And they think of it as
+something that's distributed by authorities (and so should be
+distributed equally), rather than something that has to be created
+(and might be created unequally).In fact, wealth is not money.  Money is just a convenient way of
+trading one form of wealth for another.  Wealth is the underlying
+stuff—the goods and services we buy.  When you travel to a
+rich or poor country, you don't have to look at people's bank
+accounts to tell which kind you're in.  You can see
+wealth—in buildings and streets, in the clothes and the health
+of the people.Where does wealth come from?  People make it.  This was easier to
+grasp when most people lived on farms, and made many of the things
+they wanted with their own hands.  Then you could see in the house,
+the herds, and the granary the wealth that each family created.  It
+was obvious then too that the wealth of the world was not a fixed
+quantity that had to be shared out, like slices of a pie.  If you
+wanted more wealth, you could make it.This is just as true today, though few of us create wealth directly
+for ourselves (except for a few vestigial domestic tasks).  Mostly
+we create wealth for other people in exchange for money, which we
+then trade for the forms of wealth we want. 
+[1]Because kids are unable to create wealth, whatever they have has
+to be given to them.  And when wealth is something you're given,
+then of course it seems that it should be distributed equally.
+[2]
+As in most families it is.  The kids see to that.  "Unfair," they
+cry, when one sibling gets more than another.In the real world, you can't keep living off your parents.  If you
+want something, you either have to make it, or do something of
+equivalent value for someone else, in order to get them to give you
+enough money to buy it.  In the real world, wealth is (except for
+a few specialists like thieves and speculators) something you have
+to create, not something that's distributed by Daddy.  And since
+the ability and desire to create it vary from person to person,
+it's not made equally.You get paid by doing or making something people want, and those
+who make more money are often simply better at doing what people
+want.  Top actors make a lot more money than B-list actors.  The
+B-list actors might be almost as charismatic, but when people go
+to the theater and look at the list of movies playing, they want
+that extra oomph that the big stars have.Doing what people want is not the only way to get money, of course.
+You could also rob banks, or solicit bribes, or establish a monopoly.
+Such tricks account for some variation in wealth, and indeed for
+some of the biggest individual fortunes, but they are not the root
+cause of variation in income.  The root cause of variation in income,
+as Occam's Razor implies, is the same as the root cause of variation
+in every other human skill.In the United States, the CEO of a large public company makes about
+100 times as much as the average person. 
+[3]
+Basketball players
+make about 128 times as much, and baseball players 72 times as much.
+Editorials quote this kind of statistic with horror.  But I have
+no trouble imagining that one person could be 100 times as productive
+as another.  In ancient Rome the price of slaves varied by
+a factor of 50 depending on their skills. 
+[4]
+And that's without
+considering motivation, or the extra leverage in productivity that
+you can get from modern technology.Editorials about athletes' or CEOs' salaries remind me of early
+Christian writers, arguing from first principles about whether the
+Earth was round, when they could just walk outside and check.
+[5]
+How much someone's work is worth is not a policy question.  It's
+something the market already determines."Are they really worth 100 of us?" editorialists ask.  Depends on
+what you mean by worth.  If you mean worth in the sense of what
+people will pay for their skills, the answer is yes, apparently.A few CEOs' incomes reflect some kind of wrongdoing.  But are there
+not others whose incomes really do reflect the wealth they generate?
+Steve Jobs saved a company that was in a terminal decline.  And not
+merely in the way a turnaround specialist does, by cutting costs;
+he had to decide what Apple's next products should be.  Few others
+could have done it.  And regardless of the case with CEOs, it's
+hard to see how anyone could argue that the salaries of professional
+basketball players don't reflect supply and demand.It may seem unlikely in principle that one individual could really
+generate so much more wealth than another.  The key to this mystery
+is to revisit that question, are they really worth 100 of us?
+Would a basketball team trade one of their players for 100
+random people?  What would Apple's next product look like if you
+replaced Steve Jobs with a committee of 100 random people? 
+[6]
+These
+things don't scale linearly.  Perhaps the CEO or the professional
+athlete has only ten times (whatever that means) the skill and
+determination of an ordinary person.  But it makes all the difference
+that it's concentrated in one individual.When we say that one kind of work is overpaid and another underpaid,
+what are we really saying?  In a free market, prices are determined
+by what buyers want.  People like baseball more than  poetry, so
+baseball players make more than poets.  To say that a certain kind
+of work is underpaid is thus identical with saying that people want
+the wrong things.Well, of course people want the wrong things.  It seems odd to be
+surprised by that.  And it seems even odder to say that it's
+unjust that certain kinds of work are underpaid. 
+[7]
+Then
+you're saying that it's unjust that people want the wrong things.
+It's  lamentable that people prefer reality TV and corndogs to
+Shakespeare and steamed vegetables, but unjust?  That seems like
+saying that blue is heavy, or that up is circular.The appearance of the word "unjust" here is the unmistakable spectral
+signature of the Daddy Model.  Why else would this idea occur in
+this odd context?  Whereas if the speaker were still operating on
+the Daddy Model, and saw wealth as something that flowed from a
+common source and had to be shared out, rather than something
+generated by doing what other people wanted, this is exactly what
+you'd get on noticing that some people made much more than others.When we talk about "unequal distribution of income," we should
+also ask, where does that income come from?
+[8]
+Who made the wealth
+it represents?  Because to the extent that income varies simply
+according to how much wealth people create, the distribution may
+be unequal, but it's hardly unjust.Stealing ItThe second reason we tend to find great disparities of wealth
+alarming is that for most of human history the usual way to accumulate
+a fortune was to steal it: in pastoral societies by cattle raiding;
+in agricultural societies by appropriating others' estates in times
+of war, and taxing them in times of peace.In conflicts, those on the winning side would receive the estates
+confiscated from the losers.  In England in the 1060s, when William
+the Conqueror distributed the estates of the defeated Anglo-Saxon
+nobles to his followers, the conflict was military.  By the 1530s,
+when Henry VIII distributed the estates of the monasteries to his
+followers, it was mostly political. 
+[9]
+But the principle was the
+same.  Indeed, the same principle is at work now in Zimbabwe.In more organized societies, like China, the ruler and his officials
+used taxation instead of confiscation.  But here too we see the
+same principle: the way to get rich was not to create wealth, but
+to serve a ruler powerful enough to appropriate it.This started to change in Europe with the rise of the middle class.
+Now we think of the middle class as people who are neither rich nor
+poor, but originally they were a distinct group.  In a feudal
+society, there are just two classes: a warrior aristocracy, and the
+serfs who work their estates.  The middle class were a new, third
+group who lived in towns and supported themselves by manufacturing
+and trade.Starting in the tenth and eleventh centuries, petty nobles and
+former serfs banded together in towns that gradually became powerful
+enough to ignore the local feudal lords. 
+[10]
+Like serfs, the middle
+class made a living largely by creating wealth.  (In port cities
+like Genoa and Pisa, they also engaged in piracy.) But unlike serfs
+they had an incentive to create a lot of it.  Any wealth a serf
+created belonged to his master.  There was not much point in making
+more than you could hide.  Whereas the independence of the townsmen
+allowed them to keep whatever wealth they created.Once it became possible to get rich by creating wealth, society as
+a whole started to get richer very rapidly.  Nearly everything we
+have was created by the middle class.  Indeed, the other two classes
+have effectively disappeared in industrial societies, and their
+names been given to either end of the middle class.  (In the original
+sense of the word, Bill Gates is middle class.)But it was not till the Industrial Revolution that wealth creation
+definitively replaced corruption as the best way to get rich.  In
+England, at least, corruption only became unfashionable (and in
+fact only started to be called "corruption") when there started to
+be other, faster ways to get rich.Seventeenth-century England was much like the third world today,
+in that government office was a recognized route to wealth.  The
+great fortunes of that time still derived more from what we would
+now call corruption than from commerce. 
+[11]
+By the nineteenth
+century that had changed.  There continued to be bribes, as there
+still are everywhere, but politics had by then been left to men who
+were driven more by vanity than greed.  Technology had made it
+possible to create wealth faster than you could steal it.  The
+prototypical rich man of the nineteenth century was not a courtier
+but an industrialist.With the rise of the middle class, wealth stopped being a zero-sum
+game.  Jobs and Wozniak didn't have to make us poor to make themselves
+rich.  Quite the opposite: they created things that made our lives
+materially richer.  They had to, or we wouldn't have paid for them.But since for most of the world's history the main route to wealth
+was to steal it, we tend to be suspicious of rich people.  Idealistic
+undergraduates find their unconsciously preserved child's model of
+wealth confirmed by eminent writers of the past.  It is a case of
+the mistaken meeting the outdated."Behind every great fortune, there is a crime," Balzac wrote.  Except
+he didn't.  What he actually said was that a great fortune with no
+apparent cause was probably due to a crime well enough executed
+that it had been forgotten.  If we were talking about Europe in
+1000, or most of the third world today, the standard misquotation
+would be spot on.  But Balzac lived in nineteenth-century France,
+where the Industrial Revolution was well advanced.  He knew you
+could make a fortune without stealing it.  After all, he did himself,
+as a popular novelist.
+[12]Only a few countries (by no coincidence, the richest ones) have
+reached this stage.  In most, corruption still has the upper hand.
+In most, the fastest way to get wealth is by stealing it.  And so
+when we see increasing differences in income in a rich country,
+there is a tendency to worry that it's sliding back toward becoming
+another Venezuela.  I think the opposite is happening. I think
+you're seeing a country a full step ahead of Venezuela.The Lever of TechnologyWill technology increase the gap between rich and poor?  It will
+certainly increase the gap between the productive and the unproductive.
+That's the whole point of technology.   With a tractor an energetic
+farmer could plow six times as much land in a day as he could with
+a team of horses.  But only if he mastered a new kind of farming.I've seen the lever of technology grow visibly in my own time.  In
+high school I made money by mowing lawns and scooping ice cream at
+Baskin-Robbins.  This was the only kind of work available at the
+time.  Now high school kids could write software or design web
+sites.  But only some of them will; the rest will still be scooping
+ice cream.I remember very vividly when in 1985 improved technology made it
+possible for me to buy a computer of my own.  Within months I was
+using it to make money as a freelance programmer.  A few years
+before, I couldn't have done this.  A few years before, there was
+no such thing as a freelance programmer.  But Apple created
+wealth, in the form of powerful, inexpensive computers, and programmers
+immediately set to work using it to create more.As this example suggests, the rate at which technology increases
+our productive capacity is probably exponential, rather than linear.
+So we should expect to see ever-increasing variation in individual
+productivity as time goes on.   Will that increase the gap between
+rich and the poor?  Depends which gap you mean.Technology should increase the gap in income, but it seems to
+decrease other gaps.  A hundred years ago, the rich led a different
+kind of life from ordinary people.  They lived in houses
+full of servants, wore elaborately uncomfortable clothes, and
+travelled about in carriages drawn by teams of horses which themselves
+required their own houses and servants.  Now, thanks to technology,
+the rich live more like the average person.Cars are a good example of why.  It's possible to buy expensive,
+handmade cars that cost hundreds of thousands of dollars.  But there
+is not much point.  Companies make more money by building a large
+number of ordinary cars than a small number of expensive ones.  So
+a company making a mass-produced car can afford to spend a lot more
+on its design.  If you buy a custom-made car, something will always
+be breaking.  The only point of buying one now is to advertise that
+you can.Or consider watches.  Fifty years ago, by spending a lot of money
+on a watch you could get better performance.  When watches had
+mechanical movements, expensive watches kept better time.  Not any
+more.  Since the invention of the quartz movement, an ordinary Timex
+is more accurate than a Patek Philippe costing hundreds of thousands
+of dollars.
+[13]
+Indeed, as with expensive cars, if you're determined
+to spend a lot of money on a watch, you have to put up with some
+inconvenience to do it: as well as keeping worse time, mechanical
+watches have to be wound.The only thing technology can't cheapen is brand.  Which is precisely
+why we hear ever more about it.  Brand is the residue left as the
+substantive differences between rich and poor evaporate.  But what
+label you have on your stuff is a much smaller matter than having
+it versus not having it.  In 1900, if you kept a carriage, no one
+asked what year or brand it was.  If you had one, you were rich.
+And if you weren't rich, you took the omnibus or walked.  Now even
+the poorest Americans drive cars, and it is only because we're so
+well trained by advertising that we can even recognize the especially
+expensive ones.
+[14]The same pattern has played out in industry after industry.  If
+there is enough demand for something, technology will make it cheap
+enough to sell in large volumes, and the mass-produced versions
+will be, if not better, at least more convenient.
+[15]
+And there
+is nothing the rich like more than convenience.  The rich people I
+know drive the same cars, wear the same clothes, have the same kind
+of furniture, and eat the same foods as my other friends.  Their
+houses are in different neighborhoods, or if in the same neighborhood
+are different sizes, but within them life is similar.  The houses
+are made using the same construction techniques and contain much
+the same objects.  It's inconvenient to do something expensive and
+custom.The rich spend their time more like everyone else too.  Bertie
+Wooster seems long gone.  Now, most people who are rich enough not
+to work do anyway.  It's not just social pressure that makes them;
+idleness is lonely and demoralizing.Nor do we have the social distinctions there were a hundred years
+ago.   The novels and etiquette manuals of that period read now
+like descriptions of some strange tribal society.  "With respect
+to the continuance of friendships..." hints Mrs. Beeton's Book
+of Household Management (1880), "it may be found necessary, in
+some cases, for a mistress to relinquish, on assuming the responsibility
+of a household, many of those commenced in the earlier part of her
+life." A woman who married a rich man was expected to drop friends
+who didn't.  You'd seem a barbarian if you behaved that way today.
+You'd also have a very boring life.  People still tend to segregate
+themselves somewhat, but much more on the basis of education than
+wealth.
+[16]Materially and socially, technology seems to be decreasing the gap
+between the rich and the poor, not increasing it.  If Lenin walked
+around the offices of a company like Yahoo or Intel or Cisco, he'd
+think communism had won.  Everyone would be wearing the same clothes,
+have the same kind of office (or rather, cubicle) with the same
+furnishings, and address one another by their first names instead
+of by honorifics.  Everything would seem exactly as he'd predicted,
+until he looked at their bank accounts.  Oops.Is it a problem if technology increases that gap?  It doesn't seem
+to be so far.  As it increases the gap in income, it seems to
+decrease most other gaps.Alternative to an AxiomOne often hears a policy criticized on the grounds that it would
+increase the income gap between rich and poor.  As if it were an
+axiom that this would be bad.  It might be true that increased
+variation in income would be bad, but I don't see how we can say
+it's axiomatic.Indeed, it may even be false, in industrial democracies.  In a
+society of serfs and warlords, certainly, variation in income is a
+sign of an underlying problem.  But serfdom is not the only cause
+of variation in income.  A 747 pilot doesn't make 40 times as much
+as a checkout clerk because he is a warlord who somehow holds her
+in thrall.  His skills are simply much more valuable.I'd like to propose an alternative idea: that in a modern society,
+increasing variation in income is a sign of health.  Technology
+seems to increase the variation in productivity at faster than
+linear rates.  If we don't see corresponding variation in income,
+there are three possible explanations: (a) that technical innovation
+has stopped, (b) that the people who would create the most wealth
+aren't doing it, or (c) that they aren't getting paid for it.I think we can safely say that (a) and (b) would be bad.  If you
+disagree, try living for a year using only the resources available
+to the average Frankish nobleman in 800, and report back to us.
+(I'll be generous and not send you back to the stone age.)The only option, if you're going to have an increasingly prosperous
+society without increasing variation in income, seems to be (c),
+that people will create a lot of wealth without being paid for it.
+That Jobs and Wozniak, for example, will cheerfully work 20-hour
+days to produce the Apple computer for a society that allows them,
+after taxes, to keep just enough of their income to match what they
+would have made working 9 to 5 at a big company.Will people create wealth if they can't get paid for it?  Only if
+it's fun.  People will write operating systems for free.  But they
+won't install them, or take support calls, or train customers to
+use them.  And at least 90% of the work that even the highest tech
+companies do is of this second, unedifying kind.All the unfun kinds of wealth creation slow dramatically in a society
+that confiscates private fortunes.  We can confirm this empirically.
+Suppose you hear a strange noise that you think may be due to a
+nearby fan.  You turn the fan off, and the noise stops.  You turn
+the fan back on, and the noise starts again.  Off, quiet.  On,
+noise.  In the absence of other information, it would seem the noise
+is caused by the fan.At various times and places in history, whether you could accumulate
+a fortune by creating wealth has been turned on and off.  Northern
+Italy in 800, off (warlords would steal it).  Northern Italy in
+1100, on.  Central France in 1100, off (still feudal).  England in
+1800, on.  England in 1974, off (98% tax on investment income).
+United States in 1974, on.  We've even had a twin study: West
+Germany, on;  East Germany, off.  In every case, the creation of
+wealth seems to appear and disappear like the noise of a fan as you
+switch on and off the prospect of keeping it.There is some momentum involved.  It probably takes at least a
+generation to turn people into East Germans (luckily for England).
+But if it were merely a fan we were studying, without all the extra
+baggage that comes from the controversial topic of wealth, no one
+would have any doubt that the fan was causing the noise.If you suppress variations in income, whether by stealing private
+fortunes, as feudal rulers used to do, or by taxing them away, as
+some modern governments have done, the result always seems to be
+the same.    Society as a whole ends up poorer.If I had a choice of living in a society where I was materially
+much better off than I am now, but was among the poorest, or in one
+where I was the richest, but much worse off than I am now, I'd take
+the first option.  If I had children, it would arguably be immoral
+not to.  It's absolute poverty you want to avoid, not relative
+poverty.  If, as the evidence so far implies, you have to have one
+or the other in your society, take relative poverty.You need rich people in your society not so much because in spending
+their money they create jobs, but because of what they have to do
+to get rich.  I'm not talking about the trickle-down effect
+here.  I'm not saying that if you let Henry Ford get rich, he'll
+hire you as a waiter at his next party.  I'm saying that he'll make
+you a tractor to replace your horse.Notes[1]
+Part of the reason this subject is so contentious is that some
+of those most vocal on the subject of wealth—university
+students, heirs, professors, politicians, and journalists—have
+the least experience creating it.  (This phenomenon will be familiar
+to anyone who has overheard conversations about sports in a bar.)Students are mostly still on the parental dole, and have not stopped
+to think about where that money comes from.  Heirs will be on the
+parental dole for life.  Professors and politicians live within
+socialist eddies of the economy, at one remove from the creation
+of wealth, and are paid a flat rate regardless of how hard they
+work.  And journalists as part of their professional code segregate
+themselves from the revenue-collecting half of the businesses they
+work for (the ad sales department).  Many of these people never
+come face to face with the fact that the money they receive represents
+wealth—wealth that, except in the case of journalists, someone
+else created earlier.  They live in a world in which income is
+doled out by a central authority according to some abstract notion
+of fairness (or randomly, in the case of heirs), rather than given
+by other people in return for something they wanted, so it may seem
+to them unfair that things don't work the same in the rest of the
+economy.(Some professors do create a great deal of wealth for
+society.  But the money they're paid isn't a quid pro quo.
+It's more in the nature of an investment.)[2]
+When one reads about the origins of the Fabian Society, it
+sounds like something cooked up by the high-minded Edwardian
+child-heroes of Edith Nesbit's The Wouldbegoods.[3]
+According to a study by the Corporate Library, the median total
+compensation, including salary, bonus, stock grants, and the exercise
+of stock options, of S&P 500 CEOs in 2002 was $3.65 million.
+According to Sports Illustrated, the average NBA player's
+salary during the 2002-03 season was $4.54 million, and the average
+major league baseball player's salary at the start of the 2003
+season was $2.56 million.  According to the Bureau of Labor
+Statistics, the mean annual wage in the US in 2002 was $35,560.[4]
+In the early empire the price of an ordinary adult slave seems
+to have been about 2,000 sestertii (e.g. Horace, Sat. ii.7.43).
+A servant girl cost 600 (Martial vi.66), while Columella (iii.3.8)
+says that a skilled vine-dresser was worth 8,000.  A doctor, P.
+Decimus Eros Merula, paid 50,000 sestertii for his freedom (Dessau,
+Inscriptiones 7812).  Seneca (Ep. xxvii.7) reports
+that one Calvisius Sabinus paid 100,000 sestertii apiece for slaves
+learned in the Greek classics.  Pliny (Hist. Nat. vii.39)
+says that the highest price paid for a slave up to his time was
+700,000 sestertii, for the linguist (and presumably teacher) Daphnis,
+but that this had since been exceeded by actors buying their own
+freedom.Classical Athens saw a similar variation in prices.  An ordinary
+laborer was worth about 125 to 150 drachmae.  Xenophon (Mem.
+ii.5) mentions prices ranging from 50 to 6,000 drachmae (for the
+manager of a silver mine).For more on the economics of ancient slavery see:Jones, A. H. M., "Slavery in the Ancient World," Economic History
+Review, 2:9 (1956), 185-199, reprinted in Finley, M. I. (ed.),
+Slavery in Classical Antiquity, Heffer, 1964.[5]
+Eratosthenes (276—195 BC) used shadow lengths in different
+cities to estimate the Earth's circumference.  He was off by only
+about 2%.[6]
+No, and Windows, respectively.[7]
+One of the biggest divergences between the Daddy Model and
+reality is the valuation of hard work.  In the Daddy Model, hard
+work is in itself deserving.  In reality, wealth is measured by
+what one delivers, not how much effort it costs.  If I paint someone's
+house, the owner shouldn't pay me extra for doing it with a toothbrush.It will seem to someone still implicitly operating on the Daddy
+Model that it is unfair when someone works hard and doesn't get
+paid much.  To help clarify the matter, get rid of everyone else
+and put our worker on a desert island, hunting and gathering fruit.
+If he's bad at it he'll work very hard and not end up with much
+food.  Is this unfair?  Who is being unfair to him?[8]
+Part of the reason for the tenacity of the Daddy Model may be
+the dual meaning of "distribution." When economists talk about
+"distribution of income," they mean statistical distribution.  But
+when you use the phrase frequently, you can't help associating it
+with the other sense of the word (as in e.g. "distribution of alms"),
+and thereby subconsciously seeing wealth as something that flows
+from some central tap.  The word "regressive" as applied to tax
+rates has a similar effect, at least on me; how can anything
+regressive be good?[9]
+"From the beginning of the reign Thomas Lord Roos was an assiduous
+courtier of the young Henry VIII and was soon to reap the rewards.
+In 1525 he was made a Knight of the Garter and given the Earldom
+of Rutland.  In the thirties his support of the breach with Rome,
+his zeal in crushing the Pilgrimage of Grace, and his readiness to
+vote the death-penalty in the succession of spectacular treason
+trials that punctuated Henry's erratic matrimonial progress made
+him an obvious candidate for grants of monastic property."Stone, Lawrence, Family and Fortune: Studies in Aristocratic
+Finance in the Sixteenth and Seventeenth Centuries, Oxford
+University Press, 1973, p. 166.[10]
+There is archaeological evidence for large settlements earlier,
+but it's hard to say what was happening in them.Hodges, Richard and David Whitehouse, Mohammed, Charlemagne and
+the Origins of Europe, Cornell University Press, 1983.[11]
+William Cecil and his son Robert were each in turn the most
+powerful minister of the crown, and both used their position to
+amass fortunes among the largest of their times.  Robert in particular
+took bribery to the point of treason.  "As Secretary of State and
+the leading advisor to King James on foreign policy, [he] was a
+special recipient of favour, being offered large bribes by the Dutch
+not to make peace with Spain, and large bribes by Spain to make
+peace." (Stone, op. cit., p. 17.)[12]
+Though Balzac made a lot of money from writing, he was notoriously
+improvident and was troubled by debts all his life.[13]
+A Timex will gain or lose about .5 seconds per day.  The most
+accurate mechanical watch, the Patek Philippe 10 Day Tourbillon,
+is rated at -1.5 to +2 seconds.  Its retail price is about $220,000.[14]
+If asked to choose which was more expensive, a well-preserved
+1989 Lincoln Town Car ten-passenger limousine ($5,000) or a 2004
+Mercedes S600 sedan ($122,000), the average Edwardian might well
+guess wrong.[15]
+To say anything meaningful about income trends, you have to
+talk about real income, or income as measured in what it can buy.
+But the usual way of calculating real income ignores much of the
+growth in wealth over time, because it depends on a consumer price
+index created by bolting end to end a series of numbers that are
+only locally accurate, and that don't include the prices of new
+inventions until they become so common that their prices stabilize.So while we might think it was very much better to live in a world
+with antibiotics or air travel or an electric power grid than
+without, real income statistics calculated in the usual way will
+prove to us that we are only slightly richer for having these things.Another approach would be to ask, if you were going back to the
+year x in a time machine, how much would you have to spend on trade
+goods to make your fortune?  For example, if you were going back
+to 1970 it would certainly be less than $500, because the processing
+power you can get for $500 today would have been worth at least
+$150 million in 1970.  The function goes asymptotic fairly quickly,
+because for times over a hundred years or so you could get all you
+needed in present-day trash.  In 1800 an empty plastic drink bottle
+with a screw top would have seemed a miracle of workmanship.[16]
+Some will say this amounts to the same thing, because the rich
+have better opportunities for education.  That's a valid point.  It
+is still possible, to a degree, to buy your kids' way into top
+colleges by sending them to private schools that in effect hack the
+college admissions process.According to a 2002 report by the National Center for Education
+Statistics, about 1.7% of American kids attend private, non-sectarian
+schools.  At Princeton, 36% of the class of 2007 came from such
+schools.  (Interestingly, the number at Harvard is significantly
+lower, about 28%.)  Obviously this is a huge loophole.  It does at
+least seem to be closing, not widening.Perhaps the designers of admissions processes should take a lesson
+from the example of computer security, and instead of just assuming
+that their system can't be hacked, measure the degree to which it
+is.

recipes/experimental/long-context/H2O/data/PaulGrahamEssays/gba.txt

@@ -0,0 +1,198 @@
+April 2004To the popular press, "hacker" means someone who breaks
+into computers.  Among programmers it means a good programmer.
+But the two meanings are connected.  To programmers,
+"hacker" connotes mastery in the most literal sense: someone
+who can make a computer do what he wants—whether the computer
+wants to or not.To add to the confusion, the noun "hack" also has two senses.  It can
+be either a compliment or an insult.  It's called a hack when
+you do something in an ugly way.  But when you do something
+so clever that you somehow beat the system, that's also
+called a hack.  The word is used more often in the former than
+the latter sense, probably because ugly solutions are more
+common than brilliant ones.Believe it or not, the two senses of "hack" are also
+connected.  Ugly and imaginative solutions have something in
+common: they both break the rules.  And there is a gradual
+continuum between rule breaking that's merely ugly (using
+duct tape to attach something to your bike) and rule breaking
+that is brilliantly imaginative (discarding Euclidean space).Hacking predates computers.  When he
+was working on the Manhattan Project, Richard Feynman used to
+amuse himself by breaking into safes containing secret documents.
+This tradition continues today.
+When we were in grad school, a hacker friend of mine who spent too much
+time around MIT had
+his own lock picking kit.
+(He now runs a hedge fund, a not unrelated enterprise.)It is sometimes hard to explain to authorities why one would
+want to do such things.
+Another friend of mine once got in trouble with the government for
+breaking into computers.  This had only recently been declared
+a crime, and the FBI found that their usual investigative
+technique didn't work.  Police investigation apparently begins with
+a motive.  The usual motives are few: drugs, money, sex,
+revenge.  Intellectual curiosity was not one of the motives on
+the FBI's list.  Indeed, the whole concept seemed foreign to
+them.Those in authority tend to be annoyed by hackers'
+general attitude of disobedience.  But that disobedience is
+a byproduct of the qualities that make them good programmers.
+They may laugh at the CEO when he talks in generic corporate
+newspeech, but they also laugh at someone who tells them
+a certain problem can't be solved.
+Suppress one, and you suppress the other.This attitude is sometimes affected.  Sometimes young programmers
+notice the eccentricities of eminent hackers and decide to
+adopt some of their own in order to seem smarter.
+The fake version is not merely
+annoying; the prickly attitude of these posers
+can actually slow the process of innovation.But even factoring in their annoying eccentricities,
+the disobedient attitude of hackers is a net win.  I wish its
+advantages were better understood.For example, I suspect people in Hollywood are
+simply mystified by
+hackers' attitudes toward copyrights.  They are a perennial
+topic of heated discussion on Slashdot.
+But why should people who program computers
+be so concerned about copyrights, of all things?Partly because some companies use mechanisms to prevent
+copying.  Show any hacker a lock and his first thought is
+how to pick it.  But there is a deeper reason that
+hackers are alarmed by measures like copyrights and patents.
+They see increasingly aggressive measures to protect
+"intellectual property"
+as a threat to the intellectual
+freedom they need to do their job.
+And they are right.It is by poking about inside current technology that
+hackers get ideas for the next generation.  No thanks,
+intellectual homeowners may say, we don't need any
+outside help.  But they're wrong.
+The next generation of computer technology has
+often—perhaps more often than not—been developed by outsiders.In 1977 there was no doubt some group within IBM developing
+what they expected to be
+the next generation of business computer.  They were mistaken.
+The next generation of business computer was
+being developed on entirely different lines by two long-haired
+guys called Steve in a garage in Los Altos.  At about the
+same time, the powers that be
+were cooperating to develop the
+official next generation operating system, Multics.
+But two guys who thought Multics excessively complex went off
+and wrote their own.  They gave it a name that
+was a joking reference to Multics: Unix.The latest intellectual property laws impose
+unprecedented restrictions on the sort of poking around that
+leads to new ideas. In the past, a competitor might use patents
+to prevent you from selling a copy of something they
+made, but they couldn't prevent you from
+taking one apart to see how it worked.   The latest
+laws make this a crime.  How are we
+to develop new technology if we can't study current
+technology to figure out how to improve it?Ironically, hackers have brought this on themselves.
+Computers are responsible for the problem.  The control systems
+inside machines used to be physical: gears and levers and cams.
+Increasingly, the brains (and thus the value) of products is
+in software. And by this I mean software in the general sense:
+i.e. data.  A song on an LP is physically stamped into the
+plastic.  A song on an iPod's disk is merely stored on it.Data is by definition easy to copy.  And the Internet
+makes copies easy to distribute.  So it is no wonder
+companies are afraid.  But, as so often happens, fear has
+clouded their judgement.  The government has responded
+with draconian laws to protect intellectual property.
+They probably mean well. But
+they may not realize that such laws will do more harm
+than good.Why are programmers so violently opposed to these laws?
+If I were a legislator, I'd be interested in this
+mystery—for the same reason that, if I were a farmer and suddenly
+heard a lot of squawking coming from my hen house one night,
+I'd want to go out and investigate.  Hackers are not stupid,
+and unanimity is very rare in this world.
+So if they're all squawking,   
+perhaps there is something amiss.Could it be that such laws, though intended to protect America,
+will actually harm it?  Think about it.  There is something
+very American about Feynman breaking into safes during
+the Manhattan Project.  It's hard to imagine the authorities
+having a sense of humor about such things over
+in Germany at that time.  Maybe it's not a coincidence.Hackers are unruly.  That is the essence of hacking.  And it
+is also the essence of Americanness.  It is no accident
+that Silicon Valley
+is in America, and not France, or Germany,
+or England, or Japan. In those countries, people color inside
+the lines.I lived for a while in Florence.  But after I'd been there
+a few months I realized that what I'd been unconsciously hoping
+to find there was back in the place I'd just left.
+The reason Florence is famous is that in 1450, it was New York.
+In 1450 it was filled with the kind of turbulent and ambitious
+people you find now in America.  (So I went back to America.)It is greatly to America's advantage that it is
+a congenial atmosphere for the right sort of unruliness—that
+it is a home not just for the smart, but for smart-alecks.
+And hackers are invariably smart-alecks.  If we had a national
+holiday, it would be April 1st.  It says a great deal about
+our work that we use the same word for a brilliant or a
+horribly cheesy solution.   When we cook one up we're not
+always 100% sure which kind it is.  But as long as it has
+the right sort of wrongness, that's a promising sign.
+It's odd that people
+think of programming as precise and methodical.  Computers
+are precise and methodical.  Hacking is something you do
+with a gleeful laugh.In our world some of the most characteristic solutions
+are not far removed from practical
+jokes.  IBM was no doubt rather surprised by the consequences
+of the licensing deal for DOS, just as the hypothetical
+"adversary" must be when Michael Rabin solves a problem by
+redefining it as one that's easier to solve.Smart-alecks have to develop a keen sense of how much they
+can get away with.  And lately hackers 
+have sensed a change
+in the atmosphere.
+Lately hackerliness seems rather frowned upon.To hackers the recent contraction in civil liberties seems
+especially ominous.  That must also mystify outsiders. 
+Why should we care especially about civil
+liberties?  Why programmers, more than
+dentists or salesmen or landscapers?Let me put the case in terms a government official would appreciate.
+Civil liberties are not just an ornament, or a quaint
+American tradition.  Civil liberties make countries rich.
+If you made a graph of
+GNP per capita vs. civil liberties, you'd notice a definite
+trend.  Could civil liberties really be a cause, rather
+than just an effect?  I think so.  I think a society in which
+people can do and say what they want will also tend to
+be one in which the most efficient solutions win, rather than
+those sponsored by the most influential people.
+Authoritarian countries become corrupt;
+corrupt countries become poor; and poor countries are weak. 
+It seems to me there is
+a Laffer curve for government power, just as for
+tax revenues.  At least, it seems likely enough that it
+would be stupid to try the experiment and find out.  Unlike
+high tax rates, you can't repeal totalitarianism if it
+turns out to be a mistake.This is why hackers worry.  The government spying on people doesn't
+literally make programmers write worse code.  It just leads
+eventually to a world in which bad ideas win.  And because
+this is so important to hackers, they're especially sensitive
+to it.  They can sense totalitarianism approaching from a
+distance, as animals can sense an approaching  
+thunderstorm.It would be ironic if, as hackers fear, recent measures
+intended to protect national security and intellectual property
+turned out to be a missile aimed right at what makes   
+America successful.  But it would not be the first time that
+measures taken in an atmosphere of panic had
+the opposite of the intended effect.There is such a thing as Americanness.
+There's nothing like living abroad to teach you that.   
+And if you want to know whether something will nurture or squash
+this quality, it would be hard to find a better focus
+group than hackers, because they come closest of any group
+I know to embodying it.  Closer, probably,  than
+the men running our government,
+who for all their talk of patriotism
+remind me more of Richelieu or Mazarin
+than Thomas Jefferson or George Washington.When you read what the founding fathers had to say for
+themselves, they sound more like hackers.
+"The spirit of resistance to government,"
+Jefferson wrote, "is so valuable on certain occasions, that I wish
+it always to be kept alive."Imagine an American president saying that today.
+Like the remarks of an outspoken old grandmother, the sayings of
+the founding fathers have embarrassed generations of
+their less confident successors.  They remind us where we come from.
+They remind us that it is the people who break rules that are
+the source of America's wealth and power.Those in a position to impose rules naturally want them to be
+obeyed.  But be careful what you ask for. You might get it.Thanks to Ken Anderson, Trevor Blackwell, Daniel Giffin, 
+Sarah Harlin,  Shiro Kawai, Jessica Livingston, Matz, 
+Jackie McDonough, Robert Morris, Eric Raymond, Guido van Rossum,
+David Weinberger, and
+Steven Wolfram for reading drafts of this essay.
+(The image shows Steves Jobs and Wozniak 
+with a "blue box."
+Photo by Margret Wozniak. Reproduced by permission of Steve
+Wozniak.)

recipes/experimental/long-context/H2O/data/PaulGrahamEssays/gh.txt

@@ -0,0 +1,434 @@
+
+
+Want to start a startup?  Get funded by
+Y Combinator.
+
+
+
+
+July 2004(This essay is derived from a talk at Oscon 2004.)
+A few months ago I finished a new 
+book, 
+and in reviews I keep
+noticing words like "provocative'' and "controversial.'' To say
+nothing of "idiotic.''I didn't mean to make the book controversial.  I was trying to make
+it efficient.  I didn't want to waste people's time telling them
+things they already knew.  It's more efficient just to give them
+the diffs.  But I suppose that's bound to yield an alarming book.EdisonsThere's no controversy about which idea is most controversial:
+the suggestion that variation in wealth might not be as big a
+problem as we think.I didn't say in the book that variation in wealth was in itself a
+good thing.  I said in some situations it might be a sign of good
+things.  A throbbing headache is not a good thing, but it can be
+a sign of a good thing-- for example, that you're recovering
+consciousness after being hit on the head.Variation in wealth can be a sign of variation in productivity.
+(In a society of one, they're identical.) And that
+is almost certainly a good thing: if your society has no variation
+in productivity, it's probably not because everyone is Thomas
+Edison.  It's probably because you have no Thomas Edisons.In a low-tech society you don't see much variation in productivity.
+If you have a tribe of nomads collecting sticks for a fire, how
+much more productive is the best stick gatherer going to be than
+the worst?  A factor of two?  Whereas when you hand people a complex tool
+like a computer, the variation in what they can do with
+it is enormous.That's not a new idea.  Fred Brooks wrote about it in 1974, and
+the study he quoted was published in 1968.  But I think he
+underestimated the variation between programmers.  He wrote about productivity in lines
+of code:  the best programmers can solve a given problem in a tenth
+the time.  But what if the problem isn't given? In programming, as
+in many fields, the hard part isn't solving problems, but deciding
+what problems to solve.  Imagination is hard to measure, but
+in practice it dominates the kind of productivity that's measured
+in lines of code.Productivity varies in any field, but there are few in which it
+varies so much.  The variation between programmers
+is so great that it becomes a difference in kind.  I don't
+think this is something intrinsic to programming, though.  In every field,
+technology magnifies differences in productivity.  I think what's
+happening in programming is just that we have a lot of technological
+leverage.  But in every field the lever is getting longer, so the
+variation we see is something that more and more fields will see
+as time goes on.  And the success of companies, and countries, will
+depend increasingly on how they deal with it.If variation in productivity increases with technology, then the
+contribution of the most productive individuals will not only be
+disproportionately large, but will actually grow with time.  When
+you reach the point where 90% of a group's output is created by 1%
+of its members, you lose big if something (whether Viking raids,
+or central planning) drags their productivity down to the average.If we want to get the most out of them, we need to understand these
+especially productive people.  What motivates them?  What do they
+need to do their jobs?  How do you recognize them? How do you
+get them to come and work for you?  And then of course there's the
+question, how do you become one?More than MoneyI know a handful of super-hackers, so I sat down and thought about
+what they have in common.  Their defining quality is probably that
+they really love to program.  Ordinary programmers write code to pay
+the bills.  Great hackers think of it as something they do for fun,
+and which they're delighted to find people will pay them for.Great programmers are sometimes said to be indifferent to money.
+This isn't quite true.  It is true that all they really care about
+is doing interesting work.  But if you make enough money, you get
+to work on whatever you want, and for that reason hackers are
+attracted by the idea of making really large amounts of money.
+But as long as they still have to show up for work every day, they
+care more about what they do there than how much they get paid for
+it.Economically, this is a fact of the greatest importance, because
+it means you don't have to pay great hackers anything like what
+they're worth.  A great programmer might be ten or a hundred times
+as productive as an ordinary one, but he'll consider himself lucky
+to get paid three times as much.  As I'll explain later, this is
+partly because great hackers don't know how good they are.  But
+it's also because money is not the main thing they want.What do hackers want?  Like all craftsmen, hackers like good tools.
+In fact, that's an understatement.  Good hackers find it unbearable
+to use bad tools.  They'll simply refuse to work on projects with
+the wrong infrastructure.At a startup I once worked for, one of the things pinned up on our
+bulletin board was an ad from IBM.  It was a picture of an AS400,
+and the headline read, I think, "hackers despise
+it.'' [1]When you decide what infrastructure to use for a project, you're
+not just making a technical decision.  You're also making a social
+decision, and this may be the more important of the two.  For
+example, if your company wants to write some software, it might
+seem a prudent choice to write it in Java.  But when you choose a
+language, you're also choosing a community.  The programmers you'll
+be able to hire to work on a Java project won't be as
+smart as the
+ones you could get to work on a project written in Python.
+And the quality of your hackers probably matters more than the
+language you choose.  Though, frankly, the fact that good hackers
+prefer Python to Java should tell you something about the relative
+merits of those languages.Business types prefer the most popular languages because they view
+languages as standards. They don't want to bet the company on
+Betamax.  The thing about languages, though, is that they're not
+just standards.  If you have to move bits over a network, by all
+means use TCP/IP.  But a programming language isn't just a format.
+A programming language is a medium of expression.I've read that Java has just overtaken Cobol as the most popular
+language.  As a standard, you couldn't wish for more.  But as a
+medium of expression, you could do a lot better.  Of all the great
+programmers I can think of, I know of only one who would voluntarily
+program in Java.  And of all the great programmers I can think of
+who don't work for Sun, on Java, I know of zero.Great hackers also generally insist on using open source software.
+Not just because it's better, but because it gives them more control.
+Good hackers insist on control.  This is part of what makes them
+good hackers:  when something's broken, they need to fix it.  You
+want them to feel this way about the software they're writing for
+you.  You shouldn't be surprised when they feel the same way about
+the operating system.A couple years ago a venture capitalist friend told me about a new
+startup he was involved with.  It sounded promising.  But the next
+time I talked to him, he said they'd decided to build their software
+on Windows NT, and had just hired a very experienced NT developer
+to be their chief technical officer.  When I heard this, I thought,
+these guys are doomed.  One, the CTO couldn't be a first rate
+hacker, because to become an eminent NT developer he would have
+had to use NT voluntarily, multiple times, and I couldn't imagine
+a great hacker doing that; and two, even if he was good, he'd have
+a hard time hiring anyone good to work for him if the project had
+to be built on NT. [2]The Final FrontierAfter software, the most important tool to a hacker is probably
+his office.  Big companies think the function of office space is to express
+rank.  But hackers use their offices for more than that: they
+use their office as a place to think in.  And if you're a technology
+company, their thoughts are your product.  So making hackers work
+in a noisy, distracting environment is like having a paint factory
+where the air is full of soot.The cartoon strip Dilbert has a lot to say about cubicles, and with
+good reason.  All the hackers I know despise them.  The mere prospect
+of being interrupted is enough to prevent hackers from working on
+hard problems.  If you want to get real work done in an office with
+cubicles, you have two options: work at home, or come in early or
+late or on a weekend, when no one else is there.  Don't companies
+realize this is a sign that something is broken?  An office
+environment is supposed to be something that helps
+you work, not something you work despite.Companies like Cisco are proud that everyone there has a cubicle,
+even the CEO.  But they're not so advanced as they think; obviously
+they still view office space as a badge of rank.  Note too that
+Cisco is famous for doing very little product development in house.
+They get new technology by buying the startups that created it-- where
+presumably the hackers did have somewhere quiet to work.One big company that understands what hackers need is Microsoft.
+I once saw a recruiting ad for Microsoft with a big picture of a
+door.  Work for us, the premise was, and we'll give you a place to
+work where you can actually get work done.   And you know, Microsoft
+is remarkable among big companies in that they are able to develop
+software in house.  Not well, perhaps, but well enough.If companies want hackers to be productive, they should look at
+what they do at home.  At home, hackers can arrange things themselves
+so they can get the most done.  And when they work at home, hackers
+don't work in noisy, open spaces; they work in rooms with doors.  They
+work in cosy, neighborhoody places with people around and somewhere
+to walk when they need to mull something over, instead of in glass
+boxes set in acres of parking lots.  They have a sofa they can take
+a nap on when they feel tired, instead of sitting in a coma at
+their desk, pretending to work.  There's no crew of people with
+vacuum cleaners that roars through every evening during the prime
+hacking hours.  There are no meetings or, God forbid, corporate
+retreats or team-building exercises.  And when you look at what
+they're doing on that computer, you'll find it reinforces what I
+said earlier about tools.  They may have to use Java and Windows
+at work, but at home, where they can choose for themselves, you're
+more likely to find them using Perl and Linux.Indeed, these statistics about Cobol or Java being the most popular
+language can be misleading.  What we ought to look at, if we want
+to know what tools are best, is what hackers choose when they can
+choose freely-- that is, in projects of their own.  When you ask
+that question, you find that open source operating systems already
+have a dominant market share, and the number one language is probably
+Perl.InterestingAlong with good tools, hackers want interesting projects.  What
+makes a project interesting?  Well, obviously overtly sexy
+applications like stealth planes or special effects software would
+be interesting to work on.  But any application can be interesting
+if it poses novel technical challenges.  So it's hard to predict
+which problems hackers will like, because some become
+interesting only when the people working on them discover a new
+kind of solution.  Before ITA
+(who wrote the software inside Orbitz),
+the people working on airline fare searches probably thought it
+was one of the most boring applications imaginable.  But ITA made
+it interesting by 
+redefining the problem in a more ambitious way.I think the same thing happened at Google.  When Google was founded,
+the conventional wisdom among the so-called portals was that search
+was boring and unimportant.  But the guys at Google didn't think
+search was boring, and that's why they do it so well.This is an area where managers can make a difference.  Like a parent
+saying to a child, I bet you can't clean up your whole room in
+ten minutes, a good manager can sometimes redefine a problem as a
+more interesting one.  Steve Jobs seems to be particularly good at
+this, in part simply by having high standards.  There were a lot
+of small, inexpensive computers before the Mac.  He redefined the
+problem as: make one that's beautiful.  And that probably drove
+the developers harder than any carrot or stick could.They certainly delivered.  When the Mac first appeared, you didn't
+even have to turn it on to know it would be good; you could tell
+from the case.  A few weeks ago I was walking along the street in
+Cambridge, and in someone's trash I saw what appeared to be a Mac
+carrying case.  I looked inside, and there was a Mac SE.  I carried
+it home and plugged it in, and it booted.  The happy Macintosh
+face, and then the finder.  My God, it was so simple.  It was just
+like ... Google.Hackers like to work for people with high standards.  But it's not
+enough just to be exacting.  You have to insist on the right things.
+Which usually means that you have to be a hacker yourself.  I've
+seen occasional articles about how to manage programmers.  Really
+there should be two articles: one about what to do if
+you are yourself a programmer, and one about what to do if you're not.  And the 
+second could probably be condensed into two words:  give up.The problem is not so much the day to day management.  Really good
+hackers are practically self-managing.  The problem is, if you're
+not a hacker, you can't tell who the good hackers are.  A similar
+problem explains why American cars are so ugly.  I call it the
+design paradox.  You might think that you could make your products
+beautiful just by hiring a great designer to design them.  But if
+you yourself don't have good taste, 
+how are you going to recognize
+a good designer?  By definition you can't tell from his portfolio.
+And you can't go by the awards he's won or the jobs he's had,
+because in design, as in most fields, those tend to be driven by
+fashion and schmoozing, with actual ability a distant third.
+There's no way around it:  you can't manage a process intended to
+produce beautiful things without knowing what beautiful is.  American
+cars are ugly because American car companies are run by people with
+bad taste.Many people in this country think of taste as something elusive,
+or even frivolous.  It is neither.  To drive design, a manager must
+be the most demanding user of a company's products.  And if you
+have really good taste, you can, as Steve Jobs does, make satisfying
+you the kind of problem that good people like to work on.Nasty Little ProblemsIt's pretty easy to say what kinds of problems are not interesting:
+those where instead of solving a few big, clear, problems, you have
+to solve a lot of nasty little ones.  One of the worst kinds of
+projects is writing an interface to a piece of software that's
+full of bugs.  Another is when you have to customize
+something for an individual client's complex and ill-defined needs.
+To hackers these kinds of projects are the death of a thousand
+cuts.The distinguishing feature of nasty little problems is that you
+don't learn anything from them.   Writing a compiler is interesting
+because it teaches you what a compiler is.  But writing an interface
+to a buggy piece of software doesn't teach you anything, because the
+bugs are random.  [3] So it's not just fastidiousness that makes good
+hackers avoid nasty little problems.  It's more a question of
+self-preservation.  Working on nasty little problems makes you
+stupid.  Good hackers avoid it for the same reason models avoid
+cheeseburgers.Of course some problems inherently have this character.  And because
+of supply and demand, they pay especially well.  So a company that
+found a way to get great hackers to work on tedious problems would
+be very successful.  How would you do it?One place this happens is in startups.  At our startup we had 
+Robert Morris working as a system administrator.  That's like having the
+Rolling Stones play at a bar mitzvah.  You can't hire that kind of
+talent.  But people will do any amount of drudgery for companies
+of which they're the founders.  [4]Bigger companies solve the problem by partitioning the company.
+They get smart people to work for them by establishing a separate
+R&D department where employees don't have to work directly on
+customers' nasty little problems. [5] In this model, the research
+department functions like a mine. They produce new ideas; maybe
+the rest of the company will be able to use them.You may not have to go to this extreme.  
+Bottom-up programming
+suggests another way to partition the company: have the smart people
+work as toolmakers.  If your company makes software to do x, have
+one group that builds tools for writing software of that type, and
+another that uses these tools to write the applications.  This way
+you might be able to get smart people to write 99% of your code,
+but still keep them almost as insulated from users as they would
+be in a traditional research department.  The toolmakers would have
+users, but they'd only be the company's own developers.  [6]If Microsoft used this approach, their software wouldn't be so full
+of security holes, because the less smart people writing the actual
+applications wouldn't be doing low-level stuff like allocating
+memory.  Instead of writing Word directly in C, they'd be plugging
+together big Lego blocks of Word-language.  (Duplo, I believe, is
+the technical term.)ClumpingAlong with interesting problems, what good hackers like is other
+good hackers.  Great hackers tend to clump together-- sometimes
+spectacularly so, as at Xerox Parc.   So you won't attract good
+hackers in linear proportion to how good an environment you create
+for them.  The tendency to clump means it's more like the square
+of the environment.  So it's winner take all.  At any given time,
+there are only about ten or twenty places where hackers most want to
+work, and if you aren't one of them, you won't just have fewer
+great hackers, you'll have zero.Having great hackers is not, by itself, enough to make a company
+successful.  It works well for Google and ITA, which are two of
+the hot spots right now, but it didn't help Thinking Machines or
+Xerox.  Sun had a good run for a while, but their business model
+is a down elevator.  In that situation, even the best hackers can't
+save you.I think, though, that all other things being equal, a company that
+can attract great hackers will have a huge advantage.  There are
+people who would disagree with this.  When we were making the rounds
+of venture capital firms in the 1990s, several told us that software
+companies didn't win by writing great software, but through brand,
+and dominating channels, and doing the right deals.They really seemed to believe this, and I think I know why.  I
+think what a lot of VCs are looking for, at least unconsciously,
+is the next Microsoft.  And of course if Microsoft is your model,
+you shouldn't be looking for companies that hope to win by writing
+great software.  But VCs are mistaken to look for the next Microsoft,
+because no startup can be the next Microsoft unless some other
+company is prepared to bend over at just the right moment and be
+the next IBM.It's a mistake to use Microsoft as a model, because their whole
+culture derives from that one lucky break.  Microsoft is a bad data
+point.  If you throw them out, you find that good products do tend
+to win in the market.  What VCs should be looking for is the next
+Apple, or the next Google.I think Bill Gates knows this.  What worries him about Google is
+not the power of their brand, but the fact that they have
+better hackers. [7]
+RecognitionSo who are the great hackers?  How do you know when you meet one?
+That turns out to be very hard.  Even hackers can't tell.  I'm
+pretty sure now that my friend Trevor Blackwell is a great hacker.
+You may have read on Slashdot how he made his 
+own Segway.  The
+remarkable thing about this project was that he wrote all the
+software in one day (in Python, incidentally).For Trevor, that's
+par for the course.  But when I first met him, I thought he was a
+complete idiot.  He was standing in Robert Morris's office babbling
+at him about something or other, and I remember standing behind
+him making frantic gestures at Robert to shoo this nut out of his
+office so we could go to lunch.  Robert says he misjudged Trevor
+at first too.  Apparently when Robert first met him, Trevor had
+just begun a new scheme that involved writing down everything about
+every aspect of his life on a stack of index cards, which he carried
+with him everywhere.  He'd also just arrived from Canada, and had
+a strong Canadian accent and a mullet.The problem is compounded by the fact that hackers, despite their
+reputation for social obliviousness, sometimes put a good deal of
+effort into seeming smart.  When I was in grad school I used to
+hang around the MIT AI Lab occasionally. It was kind of intimidating
+at first.  Everyone there spoke so fast.  But after a while I
+learned the trick of speaking fast.  You don't have to think any
+faster; just use twice as many words to say everything.  With this amount of noise in the signal, it's hard to tell good
+hackers when you meet them.  I can't tell, even now.  You also
+can't tell from their resumes.  It seems like the only way to judge
+a hacker is to work with him on something.And this is the reason that high-tech areas 
+only happen around universities.  The active ingredient
+here is not so much the professors as the students.  Startups grow up
+around universities because universities bring together promising young
+people and make them work on the same projects.  The
+smart ones learn who the other smart ones are, and together
+they cook up new projects of their own.Because you can't tell a great hacker except by working with him,
+hackers themselves can't tell how good they are.  This is true to
+a degree in most fields.  I've found that people who
+are great at something are not so much convinced of their own
+greatness as mystified at why everyone else seems so incompetent.
+But it's particularly hard for hackers to know how good they are,
+because it's hard to compare their work.  This is easier in most
+other fields.  In the hundred meters, you know in 10 seconds who's
+fastest.  Even in math there seems to be a general consensus about
+which problems are hard to solve, and what constitutes a good
+solution.  But hacking is like writing.  Who can say which of two
+novels is better?  Certainly not the authors.With hackers, at least, other hackers can tell.  That's because,
+unlike novelists, hackers collaborate on projects.  When you get
+to hit a few difficult problems over the net at someone, you learn
+pretty quickly how hard they hit them back.  But hackers can't
+watch themselves at work.  So if you ask a great hacker how good
+he is, he's almost certain to reply, I don't know.  He's not just
+being modest.  He really doesn't know.And none of us know, except about people we've actually worked
+with.  Which puts us in a weird situation: we don't know who our
+heroes should be.  The hackers who become famous tend to become
+famous by random accidents of PR.  Occasionally I need to give an
+example of a great hacker, and I never know who to use.  The first
+names that come to mind always tend to be people I know personally,
+but it seems lame to use them.  So, I think, maybe I should say
+Richard Stallman, or Linus Torvalds, or Alan Kay, or someone famous
+like that.  But I have no idea if these guys are great hackers.
+I've never worked with them on anything.If there is a Michael Jordan of hacking, no one knows, including
+him.CultivationFinally, the question the hackers have all been wondering about:
+how do you become a great hacker?  I don't know if it's possible
+to make yourself into one.  But it's certainly possible to do things
+that make you stupid, and if you can make yourself stupid, you
+can probably make yourself smart too.The key to being a good hacker may be to work on what you like.
+When I think about the great hackers I know, one thing they have
+in common is the extreme 
+difficulty of making them work 
+on anything they
+don't want to.  I don't know if this is cause or effect; it may be
+both.To do something well you have to love it.  
+So to the extent you
+can preserve hacking as something you love, you're likely to do it
+well.  Try to keep the sense of wonder you had about programming at
+age 14.  If you're worried that your current job is rotting your
+brain, it probably is.The best hackers tend to be smart, of course, but that's true in
+a lot of fields.  Is there some quality that's unique to hackers?
+I asked some friends, and the number one thing they mentioned was
+curiosity.  
+I'd always supposed that all smart people were curious--
+that curiosity was simply the first derivative of knowledge.  But
+apparently hackers are particularly curious, especially about how
+things work.  That makes sense, because programs are in effect
+giant descriptions of how things work.Several friends mentioned hackers' ability to concentrate-- their
+ability, as one put it, to "tune out everything outside their own
+heads.''  I've certainly noticed this.  And I've heard several 
+hackers say that after drinking even half a beer they can't program at
+all.   So maybe hacking does require some special ability to focus.
+Perhaps great hackers can load a large amount of context into their
+head, so that when they look at a line of code, they see not just
+that line but the whole program around it.  John McPhee
+wrote that Bill Bradley's success as a basketball player was due
+partly to his extraordinary peripheral vision.  "Perfect'' eyesight
+means about 47 degrees of vertical peripheral vision.  Bill Bradley
+had 70; he could see the basket when he was looking at the floor.
+Maybe great hackers have some similar inborn ability.  (I cheat by
+using a very dense language, 
+which shrinks the court.)This could explain the disconnect over cubicles.  Maybe the people
+in charge of facilities, not having any concentration to shatter,
+have no idea that working in a cubicle feels to a hacker like having
+one's brain in a blender.  (Whereas Bill, if the rumors of autism
+are true, knows all too well.)One difference I've noticed between great hackers and smart people
+in general is that hackers are more 
+politically incorrect.  To the
+extent there is a secret handshake among good hackers, it's when they
+know one another well enough to express opinions that would get
+them stoned to death by the general public.  And I can see why
+political incorrectness would be a useful quality in programming.
+Programs are very complex and, at least in the hands of good
+programmers, very fluid.  In such situations it's helpful to have
+a habit of questioning assumptions.Can you cultivate these qualities?  I don't know.  But you can at
+least not repress them.  So here is my best shot at a recipe.  If
+it is possible to make yourself into a great hacker, the way to do
+it may be to make the following deal with yourself: you never have
+to work on boring projects (unless your family will starve otherwise),
+and in return, you'll never allow yourself to do a half-assed job.
+All the great hackers I know seem to have made that deal, though
+perhaps none of them had any choice in the matter.Notes
+[1] In fairness, I have to say that IBM makes decent hardware.  I
+wrote this on an IBM laptop.[2] They did turn out to be doomed.  They shut down a few months
+later.[3] I think this is what people mean when they talk
+about the "meaning of life."  On the face of it, this seems an 
+odd idea.  Life isn't an expression; how could it have meaning?
+But it can have a quality that feels a lot like meaning.  In a project
+like a compiler, you have to solve a lot of problems, but the problems
+all fall into a pattern, as in a signal.  Whereas when the problems
+you have to solve are random, they seem like noise.
+[4] Einstein at one point worked designing refrigerators. (He had equity.)[5] It's hard to say exactly what constitutes research in the
+computer world, but as a first approximation, it's software that
+doesn't have users.I don't think it's publication that makes the best hackers want to work
+in research departments.  I think it's mainly not having to have a
+three hour meeting with a product manager about problems integrating
+the Korean version of Word 13.27 with the talking paperclip.[6] Something similar has been happening for a long time in the
+construction industry. When you had a house built a couple hundred
+years ago, the local builders built everything in it.  But increasingly
+what builders do is assemble components designed and manufactured
+by someone else.  This has, like the arrival of desktop publishing,
+given people the freedom to experiment in disastrous ways, but it
+is certainly more efficient.[7] Google is much more dangerous to Microsoft than Netscape was.
+Probably more dangerous than any other company has ever been.  Not
+least because they're determined to fight.  On their job listing
+page, they say that one of their "core values'' is "Don't be evil.''
+From a company selling soybean oil or mining equipment, such a
+statement would merely be eccentric.  But I think all of us in the
+computer world recognize who that is a declaration of war on.Thanks to Jessica Livingston, Robert Morris, and Sarah Harlin
+for reading earlier versions of this talk.

recipes/experimental/long-context/H2O/data/PaulGrahamEssays/goodtaste.txt

@@ -0,0 +1,86 @@
+November 2021(This essay is derived from a talk at the Cambridge Union.)When I was a kid, I'd have said there wasn't. My father told me so.
+Some people like some things, and other people like other things,
+and who's to say who's right?It seemed so obvious that there was no such thing as good taste
+that it was only through indirect evidence that I realized my father
+was wrong. And that's what I'm going to give you here: a proof by
+reductio ad absurdum. If we start from the premise that there's no
+such thing as good taste, we end up with conclusions that are
+obviously false, and therefore the premise must be wrong.We'd better start by saying what good taste is. There's a narrow
+sense in which it refers to aesthetic judgements and a broader one
+in which it refers to preferences of any kind. The strongest proof
+would be to show that taste exists in the narrowest sense, so I'm
+going to talk about taste in art. You have better taste than me if
+the art you like is better than the art I like.If there's no such thing as good taste, then there's no such thing
+as good art. Because if there is such a
+thing as good art, it's
+easy to tell which of two people has better taste. Show them a lot
+of works by artists they've never seen before and ask them to
+choose the best, and whoever chooses the better art has better
+taste.So if you want to discard the concept of good taste, you also have
+to discard the concept of good art. And that means you have to
+discard the possibility of people being good at making it. Which
+means there's no way for artists to be good at their jobs. And not
+just visual artists, but anyone who is in any sense an artist. You
+can't have good actors, or novelists, or composers, or dancers
+either. You can have popular novelists, but not good ones.We don't realize how far we'd have to go if we discarded the concept
+of good taste, because we don't even debate the most obvious cases.
+But it doesn't just mean we can't say which of two famous painters
+is better. It means we can't say that any painter is better than a
+randomly chosen eight year old.That was how I realized my father was wrong. I started studying
+painting. And it was just like other kinds of work I'd done: you
+could do it well, or badly, and if you tried hard, you could get
+better at it. And it was obvious that Leonardo and Bellini were
+much better at it than me. That gap between us was not imaginary.
+They were so good. And if they could be good, then art could be
+good, and there was such a thing as good taste after all.Now that I've explained how to show there is such a thing as good
+taste, I should also explain why people think there isn't. There
+are two reasons. One is that there's always so much disagreement
+about taste. Most people's response to art is a tangle of unexamined
+impulses. Is the artist famous? Is the subject attractive? Is this
+the sort of art they're supposed to like? Is it hanging in a famous
+museum, or reproduced in a big, expensive book? In practice most
+people's response to art is dominated by such extraneous factors.And the people who do claim to have good taste are so often mistaken.
+The paintings admired by the so-called experts in one generation
+are often so different from those admired a few generations later.
+It's easy to conclude there's nothing real there at all. It's only
+when you isolate this force, for example by trying to paint and
+comparing your work to Bellini's, that you can see that it does in
+fact exist.The other reason people doubt that art can be good is that there
+doesn't seem to be any room in the art for this goodness. The
+argument goes like this. Imagine several people looking at a work
+of art and judging how good it is. If being good art really is a
+property of objects, it should be in the object somehow. But it
+doesn't seem to be; it seems to be something happening in the heads
+of each of the observers. And if they disagree, how do you choose
+between them?The solution to this puzzle is to realize that the purpose of art
+is to work on its human audience, and humans have a lot in common.
+And to the extent the things an object acts upon respond in the
+same way, that's arguably what it means for the object to have the
+corresponding property. If everything a particle interacts with
+behaves as if the particle had a mass of m, then it has a mass of
+m. So the distinction between "objective" and "subjective" is not
+binary, but a matter of degree, depending on how much the subjects
+have in common. Particles interacting with one another are at one
+pole, but people interacting with art are not all the way at the
+other; their reactions aren't random.Because people's responses to art aren't random, art can be designed
+to operate on people, and be good or bad depending on how effectively
+it does so. Much as a vaccine can be. If someone were talking about
+the ability of a vaccine to confer immunity, it would seem very
+frivolous to object that conferring immunity wasn't really a property
+of vaccines, because acquiring immunity is something that happens
+in the immune system of each individual person. Sure, people's
+immune systems vary, and a vaccine that worked on one might not
+work on another, but that doesn't make it meaningless to talk about
+the effectiveness of a vaccine.The situation with art is messier, of course. You can't measure
+effectiveness by simply taking a vote, as you do with vaccines.
+You have to imagine the responses of subjects with a deep knowledge
+of art, and enough clarity of mind to be able to ignore extraneous
+influences like the fame of the artist. And even then you'd still
+see some disagreement. People do vary, and judging art is hard,
+especially recent art. There is definitely not a total order either
+of works or of people's ability to judge them. But there is equally
+definitely a partial order of both. So while it's not possible to
+have perfect taste, it is possible to have good taste.
+Thanks to the Cambridge Union for inviting me, and to Trevor
+Blackwell, Jessica Livingston, and Robert Morris for reading drafts
+of this.

recipes/experimental/long-context/H2O/data/PaulGrahamEssays/hubs.txt

@@ -0,0 +1,156 @@
+
+
+Want to start a startup?  Get funded by
+Y Combinator.
+
+
+
+
+October 2011If you look at a list of US cities sorted by population, the number
+of successful startups per capita varies by orders of magnitude.
+Somehow it's as if most places were sprayed with startupicide.I wondered about this for years.  I could see the average town was
+like a roach motel for startup ambitions: smart, ambitious people
+went in, but no startups came out.  But I was never able to figure
+out exactly what happened inside the motel—exactly what was
+killing all the potential startups.
+[1]A couple weeks ago I finally figured it out. I was framing the
+question wrong.  The problem is not that most towns kill startups.
+It's that death is the default for startups,
+and most towns don't save them.  Instead of thinking of most places
+as being sprayed with startupicide, it's more accurate to think of
+startups as all being poisoned, and a few places being sprayed with
+the antidote.Startups in other places are just doing what startups naturally do:
+fail.  The real question is, what's saving startups in places
+like Silicon Valley?
+[2]EnvironmentI think there are two components to the antidote: being in a place
+where startups are the cool thing to do, and chance meetings with
+people who can help you.  And what drives them both is the number
+of startup people around you.The first component is particularly helpful in the first stage of
+a startup's life, when you go from merely having an interest in
+starting a company to actually doing it.  It's quite a leap to start
+a startup.  It's an unusual thing to do. But in Silicon Valley it
+seems normal.
+[3]In most places, if you start a startup, people treat you as if
+you're unemployed.  People in the Valley aren't automatically
+impressed with you just because you're starting a company, but they
+pay attention.  Anyone who's been here any amount of time knows not
+to default to skepticism, no matter how inexperienced you seem or
+how unpromising your idea sounds at first, because they've all seen
+inexperienced founders with unpromising sounding ideas who a few
+years later were billionaires.Having people around you care about what you're doing is an
+extraordinarily powerful force.  Even the
+most willful people are susceptible to it.  About a year after we
+started Y Combinator I said something to a partner at a well known
+VC firm that gave him the (mistaken) impression I was considering
+starting another startup.  He responded so eagerly that for about
+half a second I found myself considering doing it.In most other cities, the prospect of starting a startup just doesn't
+seem real.  In the Valley it's not only real but fashionable.  That
+no doubt causes a lot of people to start startups who shouldn't.
+But I think that's ok.  Few people are suited to running a startup,
+and it's very hard to predict beforehand which are (as I know all
+too well from being in the business of trying to predict beforehand),
+so lots of people starting startups who shouldn't is probably the
+optimal state of affairs.  As long as you're at a point in your
+life when you can bear the risk of failure, the best way to find
+out if you're suited to running a startup is to try
+it.ChanceThe second component of the antidote is chance meetings with people
+who can help you.  This force works in both phases: both in the
+transition from the desire to start a startup to starting one, and
+the transition from starting a company to succeeding.  The power
+of chance meetings is more variable than people around you caring
+about startups, which is like a sort of background radiation that
+affects everyone equally, but at its strongest it is far stronger.Chance meetings produce miracles to compensate for the disasters
+that characteristically befall startups.  In the Valley, terrible
+things happen to startups all the time, just like they do to startups
+everywhere.  The reason startups are more likely to make it here
+is that great things happen to them too.  In the Valley, lightning
+has a sign bit.For example, you start a site for college students and you decide
+to move to the Valley for the summer to work on it.  And then on a
+random suburban street in Palo Alto you happen to run into Sean
+Parker, who understands the domain really well because he started
+a similar startup himself, and also knows all the investors.  And
+moreover has advanced views, for 2004, on founders retaining control of their companies.You can't say precisely what the miracle will be, or even for sure
+that one will happen.  The best one can say is: if you're in a
+startup hub, unexpected good things will probably happen to you,
+especially if you deserve them.I bet this is true even for startups we fund.  Even with us working
+to make things happen for them on purpose rather than by accident,
+the frequency of helpful chance meetings in the Valley is so high
+that it's still a significant increment on what we can deliver.Chance meetings play a role like the role relaxation plays in having
+ideas.  Most people have had the experience of working hard on some
+problem, not being able to solve it, giving up and going to bed,
+and then thinking of the answer in the shower in the morning.  What
+makes the answer appear is letting your thoughts drift a bit—and thus drift off the wrong
+path you'd been pursuing last night and onto the right one adjacent
+to it.Chance meetings let your acquaintance drift in the same way taking
+a shower lets your thoughts drift. The critical thing in both cases
+is that they drift just the right amount.  The meeting between Larry
+Page and Sergey Brin was a good example.  They let their acquaintance
+drift, but only a little; they were both meeting someone they had
+a lot in common with.For Larry Page the most important component of the antidote was
+Sergey Brin, and vice versa.  The antidote is 
+people.  It's not the
+physical infrastructure of Silicon Valley that makes it work, or
+the weather, or anything like that.  Those helped get it started,
+but now that the reaction is self-sustaining what drives it is the
+people.Many observers have noticed that one of the most distinctive things
+about startup hubs is the degree to which people help one another
+out, with no expectation of getting anything in return.  I'm not
+sure why this is so.  Perhaps it's because startups are less of a
+zero sum game than most types of business; they are rarely killed
+by competitors.  Or perhaps it's because so many startup founders
+have backgrounds in the sciences, where collaboration is encouraged.A large part of YC's function is to accelerate that process.  We're
+a sort of Valley within the Valley, where the density of people
+working on startups and their willingness to help one another are
+both artificially amplified.NumbersBoth components of the antidote—an environment that encourages
+startups, and chance meetings with people who help you—are
+driven by the same underlying cause: the number of startup people
+around you.  To make a startup hub, you need a lot of people
+interested in startups.There are three reasons. The first, obviously, is that if you don't
+have enough density, the chance meetings don't happen.
+[4]
+The second is that different startups need such different things, so
+you need a lot of people to supply each startup with what they need
+most.  Sean Parker was exactly what Facebook needed in 2004.  Another
+startup might have needed a database guy, or someone with connections
+in the movie business.This is one of the reasons we fund such a large number of companies,
+incidentally.  The bigger the community, the greater the chance it
+will contain the person who has that one thing you need most.The third reason you need a lot of people to make a startup hub is
+that once you have enough people interested in the same problem,
+they start to set the social norms.  And it is a particularly
+valuable thing when the atmosphere around you encourages you to do
+something that would otherwise seem too ambitious.  In most places
+the atmosphere pulls you back toward the mean.I flew into the Bay Area a few days ago.  I notice this every time
+I fly over the Valley: somehow you can sense something is going on.  
+Obviously you can sense prosperity in how well kept a
+place looks.  But there are different kinds of prosperity.  Silicon
+Valley doesn't look like Boston, or New York, or LA, or DC.  I tried
+asking myself what word I'd use to describe the feeling the Valley
+radiated, and the word that came to mind was optimism.Notes[1]
+I'm not saying it's impossible to succeed in a city with few
+other startups, just harder.  If you're sufficiently good at
+generating your own morale, you can survive without external
+encouragement.  Wufoo was based in Tampa and they succeeded.  But
+the Wufoos are exceptionally disciplined.[2]
+Incidentally, this phenomenon is not limited to startups.  Most
+unusual ambitions fail, unless the person who has them manages to
+find the right sort of community.[3]
+Starting a company is common, but starting a startup is rare.
+I've talked about the distinction between the two elsewhere, but
+essentially a startup is a new business designed for scale.  Most
+new businesses are service businesses and except in rare cases those
+don't scale.[4]
+As I was writing this, I had a demonstration of the density of
+startup people in the Valley.  Jessica and I bicycled to University
+Ave in Palo Alto to have lunch at the fabulous Oren's Hummus.  As
+we walked in, we met Charlie Cheever sitting near the door.  Selina
+Tobaccowala stopped to say hello on her way out.  Then Josh Wilson
+came in to pick up a take out order.  After lunch we went to get
+frozen yogurt.  On the way we met Rajat Suri.  When we got to the
+yogurt place, we found Dave Shen there, and as we walked out we ran
+into Yuri Sagalov.  We walked with him for a block or so and we ran
+into Muzzammil Zaveri, and then a block later we met Aydin Senkut.
+This is everyday life in Palo Alto.  I wasn't trying to meet people;
+I was just having lunch.  And I'm sure for every startup founder
+or investor I saw that I knew, there were 5 more I didn't.  If Ron
+Conway had been with us he would have met 30 people he knew.Thanks to Sam Altman, Paul Buchheit, Jessica Livingston, and
+Harj Taggar for reading drafts of this.

recipes/experimental/long-context/H2O/data/PaulGrahamEssays/iflisp.txt

@@ -0,0 +1,46 @@
+May 2003If Lisp is so great, why don't more people use it?  I was    
+asked this question by a student in the audience at a 
+talk I gave recently.  Not for the first time, either.In languages, as in so many things, there's not much     
+correlation between popularity and quality.  Why does   
+John Grisham (King of Torts sales rank, 44) outsell
+Jane Austen (Pride and Prejudice sales rank, 6191)?
+Would even Grisham claim that it's because he's a better
+writer?Here's the first sentence of Pride and Prejudice:
+
+It is a truth universally acknowledged, that a single man 
+in possession of a good fortune must be in want of a
+wife.
+
+"It is a truth universally acknowledged?"  Long words for
+the first sentence of a love story.Like Jane Austen, Lisp looks hard.  Its syntax, or lack
+of syntax, makes it look completely unlike 
+the languages
+most people are used to.  Before I learned Lisp, I was afraid
+of it too.  I recently came across a notebook from 1983
+in which I'd written:
+
+I suppose I should learn Lisp, but it seems so foreign.
+
+Fortunately, I was 19 at the time and not too resistant to learning
+new things.  I was so ignorant that learning
+almost anything meant learning new things.People frightened by Lisp make up other reasons for not
+using it.  The standard
+excuse, back when C was the default language, was that Lisp
+was too slow.  Now that Lisp dialects are among
+the faster
+languages available, that excuse has gone away.
+Now the standard excuse is openly circular: that other languages
+are more popular.(Beware of such reasoning.  It gets you Windows.)Popularity is always self-perpetuating, but it's especially
+so in programming languages. More libraries
+get written for popular languages, which makes them still
+more popular.  Programs often have to work with existing programs,
+and this is easier if they're written in the same language,
+so languages spread from program to program like a virus.
+And managers prefer popular languages, because they give them 
+more leverage over developers, who can more easily be replaced.Indeed, if programming languages were all more or less equivalent,
+there would be little justification for using any but the most
+popular.  But they aren't all equivalent, not by a long
+shot.  And that's why less popular languages, like Jane Austen's 
+novels, continue to survive at all.  When everyone else is reading 
+the latest John Grisham novel, there will always be a few people 
+reading Jane Austen instead.

recipes/experimental/long-context/H2O/data/PaulGrahamEssays/island.txt

@@ -0,0 +1,55 @@
+July 2006I've discovered a handy test for figuring out what you're addicted
+to.  Imagine you were going to spend the weekend at a friend's house
+on a little island off the coast of Maine.  There are no shops on
+the island and you won't be able to leave while you're there.  Also,
+you've never been to this house before, so you can't assume it will
+have more than any house might.What, besides clothes and toiletries, do you make a point of packing?
+That's what you're addicted to.  For example, if you find yourself
+packing a bottle of vodka (just in case), you may want to stop and
+think about that.For me the list is four things: books, earplugs, a notebook, and a
+pen.There are other things I might bring if I thought of it, like music,
+or tea, but I can live without them.  I'm not so addicted to caffeine
+that I wouldn't risk the house not having any tea, just for a
+weekend.Quiet is another matter.  I realize it seems a bit eccentric to
+take earplugs on a trip to an island off the coast of Maine.  If
+anywhere should be quiet, that should.  But what if the person in
+the next room snored?  What if there was a kid playing basketball?
+(Thump, thump, thump... thump.)  Why risk it?  Earplugs are small.Sometimes I can think with noise.  If I already have momentum on
+some project, I can work in noisy places.  I can edit an essay or
+debug code in an airport.  But airports are not so bad: most of the
+noise is whitish.  I couldn't work with the sound of a sitcom coming
+through the wall, or a car in the street playing thump-thump music.And of course there's another kind of thinking, when you're starting
+something new, that requires complete quiet.   You never
+know when this will strike. It's just as well to carry plugs.The notebook and pen are professional equipment, as it were.  Though
+actually there is something druglike about them, in the sense that
+their main purpose is to make me feel better.  I hardly ever go
+back and read stuff I write down in notebooks.  It's just that if
+I can't write things down, worrying about remembering one idea gets
+in the way of having the next.  Pen and paper wick ideas.The best notebooks I've found are made by a company called Miquelrius.
+I use their smallest size, which is about 2.5 x 4 in.
+The secret to writing on such
+narrow pages is to break words only when you run out of space, like
+a Latin inscription.  I use the cheapest plastic Bic ballpoints,
+partly because their gluey ink doesn't seep through pages, and
+partly so I don't worry about losing them.I only started carrying a notebook about three years ago.  Before
+that I used whatever scraps of paper I could find.  But the problem
+with scraps of paper is that they're not ordered.  In a notebook
+you can guess what a scribble means by looking at the pages
+around it.  In the scrap era I was constantly finding notes I'd
+written years before that might say something I needed to remember,
+if I could only figure out what.As for books, I know the house would probably have something to
+read.  On the average trip I bring four books and only read one of
+them, because I find new books to read en route.  Really bringing
+books is insurance.I realize this dependence on books is not entirely good—that what
+I need them for is distraction.  The books I bring on trips are
+often quite virtuous, the sort of stuff that might be assigned
+reading in a college class.  But I know my motives aren't virtuous.
+I bring books because if the world gets boring I need to be able
+to slip into another distilled by some writer.  It's like eating
+jam when you know you should be eating fruit.There is a point where I'll do without books.  I was walking in
+some steep mountains once, and decided I'd rather just think, if I
+was bored, rather than carry a single unnecessary ounce.  It wasn't
+so bad.  I found I could entertain myself by having ideas instead
+of reading other people's.  If you stop eating jam, fruit starts
+to taste better.So maybe I'll try not bringing books on some future trip.  They're
+going to have to pry the plugs out of my cold, dead ears, however.

recipes/experimental/long-context/H2O/data/PaulGrahamEssays/know.txt

@@ -0,0 +1,53 @@
+December 2014I've read Villehardouin's chronicle of the Fourth Crusade at least
+two times, maybe three.  And yet if I had to write down everything
+I remember from it, I doubt it would amount to much more than a
+page.  Multiply this times several hundred, and I get an uneasy
+feeling when I look at my bookshelves. What use is it to read all
+these books if I remember so little from them?A few months ago, as I was reading Constance Reid's excellent
+biography of Hilbert, I figured out if not the answer to this
+question, at least something that made me feel better about it.
+She writes:
+
+  Hilbert had no patience with mathematical lectures which filled
+  the students with facts but did not teach them how to frame a
+  problem and solve it. He often used to tell them that "a perfect
+  formulation of a problem is already half its solution."
+
+That has always seemed to me an important point, and I was even
+more convinced of it after hearing it confirmed by Hilbert.But how had I come to believe in this idea in the first place?  A
+combination of my own experience and other things I'd read.  None
+of which I could at that moment remember!  And eventually I'd forget
+that Hilbert had confirmed it too.  But my increased belief in the
+importance of this idea would remain something I'd learned from
+this book, even after I'd forgotten I'd learned it.Reading and experience train your model of the world.  And even if
+you forget the experience or what you read, its effect on your model
+of the world persists.  Your mind is like a compiled program you've
+lost the source of.  It works, but you don't know why.The place to look for what I learned from Villehardouin's chronicle
+is not what I remember from it, but my mental models of the crusades,
+Venice, medieval culture, siege warfare, and so on.  Which doesn't
+mean I couldn't have read more attentively, but at least the harvest
+of reading is not so miserably small as it might seem.This is one of those things that seem obvious in retrospect.  But
+it was a surprise to me and presumably would be to anyone else who
+felt uneasy about (apparently) forgetting so much they'd read.Realizing it does more than make you feel a little better about
+forgetting, though.  There are specific implications.For example, reading and experience are usually "compiled" at the
+time they happen, using the state of your brain at that time.  The
+same book would get compiled differently at different points in
+your life.  Which means it is very much worth reading important
+books multiple times.  I always used to feel some misgivings about
+rereading books.  I unconsciously lumped reading together with work
+like carpentry, where having to do something again is a sign you
+did it wrong the first time.  Whereas now the phrase "already read"
+seems almost ill-formed.Intriguingly, this implication isn't limited to books.  Technology
+will increasingly make it possible to relive our experiences.  When
+people do that today it's usually to enjoy them again (e.g. when
+looking at pictures of a trip) or to find the origin of some bug in
+their compiled code (e.g. when Stephen Fry succeeded in remembering
+the childhood trauma that prevented him from singing).  But as
+technologies for recording and playing back your life improve, it
+may become common for people to relive experiences without any goal
+in mind, simply to learn from them again as one might when rereading
+a book.Eventually we may be able not just to play back experiences but
+also to index and even edit them. So although not knowing how you
+know things may seem part of being human, it may not be.
+Thanks to Sam Altman, Jessica Livingston, and Robert Morris for reading 
+drafts of this.

recipes/experimental/long-context/H2O/data/PaulGrahamEssays/langdes.txt

@@ -0,0 +1,242 @@
+May 2001
+
+(These are some notes I made
+for a panel discussion on programming language design
+at MIT on May 10, 2001.)1. Programming Languages Are for People.Programming languages
+are how people talk to computers.  The computer would be just as
+happy speaking any language that was unambiguous.  The reason we
+have high level languages is because people can't deal with
+machine language.  The point of programming
+languages is to prevent our poor frail human brains from being 
+overwhelmed by a mass of detail.Architects know that some kinds of design problems are more personal
+than others.  One of the cleanest, most abstract design problems
+is designing bridges.  There your job is largely a matter of spanning
+a given distance with the least material.  The other end of the
+spectrum is designing chairs.  Chair designers have to spend their
+time thinking about human butts.Software varies in the same way. Designing algorithms for routing
+data through a network is a nice, abstract problem, like designing
+bridges.  Whereas designing programming languages is like designing
+chairs: it's all about dealing with human weaknesses.Most of us hate to acknowledge this.  Designing systems of great
+mathematical elegance sounds a lot more appealing to most of us
+than pandering to human weaknesses.  And there is a role for mathematical
+elegance: some kinds of elegance make programs easier to understand.
+But elegance is not an end in itself.And when I say languages have to be designed to suit human weaknesses,
+I don't mean that languages have to be designed for bad programmers.
+In fact I think you ought to design for the 
+best programmers, but
+even the best programmers have limitations.  I don't think anyone
+would like programming in a language where all the variables were
+the letter x with integer subscripts.2. Design for Yourself and Your Friends.If you look at the history of programming languages, a lot of the best
+ones were languages designed for their own authors to use, and a
+lot of the worst ones were designed for other people to use.When languages are designed for other people, it's always a specific
+group of other people: people not as smart as the language designer.
+So you get a language that talks down to you.  Cobol is the most
+extreme case, but a lot of languages are pervaded by this spirit.It has nothing to do with how abstract the language is.  C is pretty
+low-level, but it was designed for its authors to use, and that's
+why hackers like it.The argument for designing languages for bad programmers is that
+there are more bad programmers than good programmers.  That may be
+so.  But those few good programmers write a disproportionately
+large percentage of the software.I'm interested in the question, how do you design a language that
+the very best hackers will like?  I happen to think this is
+identical to the question, how do you design a good programming
+language?, but even if it isn't, it is at least an interesting
+question.3. Give the Programmer as Much Control as Possible.Many languages
+(especially the ones designed for other people) have the attitude
+of a governess: they try to prevent you from
+doing things that they think aren't good for you.  I like the   
+opposite approach: give the programmer as much
+control as you can.When I first learned Lisp, what I liked most about it was
+that it considered me an equal partner.  In the other languages
+I had learned up till then, there was the language and there was my   
+program, written in the language, and the two were very separate.
+But in Lisp the functions and macros I wrote were just like those
+that made up the language itself.  I could rewrite the language
+if I wanted.  It had the same appeal as open-source software.4. Aim for Brevity.Brevity is underestimated and even scorned.
+But if you look into the hearts of hackers, you'll see that they
+really love it.  How many times have you heard hackers speak fondly
+of how in, say, APL, they could do amazing things with just a couple
+lines of code?  I think anything that really smart people really
+love is worth paying attention to.I think almost anything
+you can do to make programs shorter is good.  There should be lots
+of library functions; anything that can be implicit should be;
+the syntax should be terse to a fault; even the names of things
+should be short.And it's not only programs that should be short.  The manual should
+be thin as well.  A good part of manuals is taken up with clarifications
+and reservations and warnings and special cases.  If you force  
+yourself to shorten the manual, in the best case you do it by fixing
+the things in the language that required so much explanation.5. Admit What Hacking Is.A lot of people wish that hacking was
+mathematics, or at least something like a natural science.  I think
+hacking is more like architecture.  Architecture is
+related to physics, in the sense that architects have to design
+buildings that don't fall down, but the actual goal of architects
+is to make great buildings, not to make discoveries about statics.What hackers like to do is make great programs.
+And I think, at least in our own minds, we have to remember that it's
+an admirable thing to write great programs, even when this work 
+doesn't translate easily into the conventional intellectual
+currency of research papers.  Intellectually, it is just as
+worthwhile to design a language programmers will love as it is to design a
+horrible one that embodies some idea you can publish a paper
+about.1. How to Organize Big Libraries?Libraries are becoming an
+increasingly important component of programming languages.  They're
+also getting bigger, and this can be dangerous.  If it takes longer
+to find the library function that will do what you want than it
+would take to write it yourself, then all that code is doing nothing
+but make your manual thick.  (The Symbolics manuals were a case in 
+point.)  So I think we will have to work on ways to organize
+libraries.  The ideal would be to design them so that the programmer
+could guess what library call would do the right thing.2. Are People Really Scared of Prefix Syntax?This is an open
+problem in the sense that I have wondered about it for years and
+still don't know the answer.  Prefix syntax seems perfectly natural
+to me, except possibly for math.  But it could be that a lot of 
+Lisp's unpopularity is simply due to having an unfamiliar syntax.   
+Whether to do anything about it, if it is true, is another question. 
+
+3. What Do You Need for Server-Based Software?
+
+I think a lot of the most exciting new applications that get written
+in the next twenty years will be Web-based applications, meaning
+programs that sit on the server and talk to you through a Web
+browser.  And to write these kinds of programs we may need some
+new things.One thing we'll need is support for the new way that server-based 
+apps get released.  Instead of having one or two big releases a
+year, like desktop software, server-based apps get released as a
+series of small changes.  You may have as many as five or ten
+releases a day.  And as a rule everyone will always use the latest
+version.You know how you can design programs to be debuggable?
+Well, server-based software likewise has to be designed to be
+changeable.  You have to be able to change it easily, or at least
+to know what is a small change and what is a momentous one.Another thing that might turn out to be useful for server based
+software, surprisingly, is continuations.  In Web-based software
+you can use something like continuation-passing style to get the
+effect of subroutines in the inherently 
+stateless world of a Web
+session.  Maybe it would be worthwhile having actual continuations,
+if it was not too expensive.4. What New Abstractions Are Left to Discover?I'm not sure how
+reasonable a hope this is, but one thing I would really love to    
+do, personally, is discover a new abstraction-- something that would
+make as much of a difference as having first class functions or
+recursion or even keyword parameters.  This may be an impossible
+dream.  These things don't get discovered that often.  But I am always
+looking.1. You Can Use Whatever Language You Want.Writing application
+programs used to mean writing desktop software.  And in desktop
+software there is a big bias toward writing the application in the
+same language as the operating system.  And so ten years ago,
+writing software pretty much meant writing software in C.
+Eventually a tradition evolved:
+application programs must not be written in unusual languages.  
+And this tradition had so long to develop that nontechnical people
+like managers and venture capitalists also learned it.Server-based software blows away this whole model.  With server-based
+software you can use any language you want.  Almost nobody understands
+this yet (especially not managers and venture capitalists).
+A few hackers understand it, and that's why we even hear
+about new, indy languages like Perl and Python.  We're not hearing
+about Perl and Python because people are using them to write Windows
+apps.What this means for us, as people interested in designing programming
+languages, is that there is now potentially an actual audience for
+our work.2. Speed Comes from Profilers.Language designers, or at least
+language implementors, like to write compilers that generate fast
+code.  But I don't think this is what makes languages fast for users.
+Knuth pointed out long ago that speed only matters in a few critical
+bottlenecks.  And anyone who's tried it knows that you can't guess
+where these bottlenecks are.  Profilers are the answer.Language designers are solving the wrong problem.  Users don't need
+benchmarks to run fast.  What they need is a language that can show
+them what parts of their own programs need to be rewritten.  That's
+where speed comes from in practice.  So maybe it would be a net 
+win if language implementors took half the time they would
+have spent doing compiler optimizations and spent it writing a
+good profiler instead.3. You Need an Application to Drive the Design of a Language.This may not be an absolute rule, but it seems like the best languages
+all evolved together with some application they were being used to
+write.  C was written by people who needed it for systems programming.
+Lisp was developed partly to do symbolic differentiation, and
+McCarthy was so eager to get started that he was writing differentiation
+programs even in the first paper on Lisp, in 1960.It's especially good if your application solves some new problem.
+That will tend to drive your language to have new features that   
+programmers need.  I personally am interested in writing
+a language that will be good for writing server-based applications.[During the panel, Guy Steele also made this point, with the
+additional suggestion that the application should not consist of
+writing the compiler for your language, unless your language
+happens to be intended for writing compilers.]4. A Language Has to Be Good for Writing Throwaway Programs.You know what a throwaway program is: something you write quickly for
+some limited task.  I think if you looked around you'd find that  
+a lot of big, serious programs started as throwaway programs.  I
+would not be surprised if most programs started as throwaway
+programs.  And so if you want to make a language that's good for
+writing software in general, it has to be good for writing throwaway
+programs, because that is the larval stage of most software.5. Syntax Is Connected to Semantics.It's traditional to think of
+syntax and semantics as being completely separate.  This will
+sound shocking, but it may be that they aren't.
+I think that what you want in your language may be related
+to how you express it.I was talking recently to Robert Morris, and he pointed out that
+operator overloading is a bigger win in languages with infix
+syntax.  In a language with prefix syntax, any function you define
+is effectively an operator.  If you want to define a plus for a
+new type of number you've made up, you can just define a new function
+to add them.  If you do that in a language with infix syntax,
+there's a big difference in appearance between the use of an
+overloaded operator and a function call.1. New Programming Languages.Back in the 1970s
+it was fashionable to design new programming languages.  Recently
+it hasn't been.  But I think server-based software will make new  
+languages fashionable again.  With server-based software, you can
+use any language you want, so if someone does design a language that
+actually seems better than others that are available, there will be
+people who take a risk and use it.2. Time-Sharing.Richard Kelsey gave this as an idea whose time
+has come again in the last panel, and I completely agree with him.
+My guess (and Microsoft's guess, it seems) is that much computing
+will move from the desktop onto remote servers.  In other words,  
+time-sharing is back.  And I think there will need to be support
+for it at the language level.  For example, I know that Richard
+and Jonathan Rees have done a lot of work implementing process  
+scheduling within Scheme 48.3. Efficiency.Recently it was starting to seem that computers
+were finally fast enough.  More and more we were starting to hear
+about byte code, which implies to me at least that we feel we have
+cycles to spare.  But I don't think we will, with server-based
+software.   Someone is going to have to pay for the servers that
+the software runs on, and the number of users they can support per
+machine will be the divisor of their capital cost.So I think efficiency will matter, at least in computational
+bottlenecks.  It will be especially important to do i/o fast,
+because server-based applications do a lot of i/o.It may turn out that byte code is not a win, in the end.  Sun and
+Microsoft seem to be facing off in a kind of a battle of the byte
+codes at the moment.  But they're doing it because byte code is a
+convenient place to insert themselves into the process, not because
+byte code is in itself a good idea.  It may turn out that this
+whole battleground gets bypassed.  That would be kind of amusing.1. Clients.This is just a guess, but my guess is that
+the winning model for most applications will be purely server-based.
+Designing software that works on the assumption that everyone will 
+have your client is like designing a society on the assumption that
+everyone will just be honest.  It would certainly be convenient, but
+you have to assume it will never happen.I think there will be a proliferation of devices that have some
+kind of Web access, and all you'll be able to assume about them is
+that they can support simple html and forms.  Will you have a
+browser on your cell phone?  Will there be a phone in your palm  
+pilot?  Will your blackberry get a bigger screen? Will you be able
+to browse the Web on your gameboy?  Your watch?  I don't know.  
+And I don't have to know if I bet on
+everything just being on the server.  It's
+just so much more robust to have all the 
+brains on the server.2. Object-Oriented Programming.I realize this is a
+controversial one, but I don't think object-oriented programming
+is such a big deal.  I think it is a fine model for certain kinds
+of applications that need that specific kind of data structure,   
+like window systems, simulations, and cad programs.  But I don't
+see why it ought to be the model for all programming.I think part of the reason people in big companies like object-oriented
+programming is because it yields a lot of what looks like work.
+Something that might naturally be represented as, say, a list of
+integers, can now be represented as a class with all kinds of
+scaffolding and hustle and bustle.Another attraction of
+object-oriented programming is that methods give you some of the
+effect of first class functions.  But this is old news to Lisp
+programmers.  When you have actual first class functions, you can
+just use them in whatever way is appropriate to the task at hand,
+instead of forcing everything into a mold of classes and methods.What this means for language design, I think, is that you shouldn't
+build object-oriented programming in too deeply.  Maybe the
+answer is to offer more general, underlying stuff, and let people design
+whatever object systems they want as libraries.3. Design by Committee.Having your language designed by a committee is a big pitfall,  
+and not just for the reasons everyone knows about.  Everyone
+knows that committees tend to yield lumpy, inconsistent designs.  
+But I think a greater danger is that they won't take risks.
+When one person is in charge he can take risks
+that a committee would never agree on.Is it necessary to take risks to design a good language though?
+Many people might suspect
+that language design is something where you should stick fairly
+close to the conventional wisdom.  I bet this isn't true.
+In everything else people do, reward is proportionate to risk.
+Why should language design be any different?

recipes/experimental/long-context/H2O/data/PaulGrahamEssays/laundry.txt

@@ -0,0 +1,487 @@
+October 2004
+As E. B. White said, "good writing is rewriting."  I didn't
+realize this when I was in school.  In writing, as in math and 
+science, they only show you the finished product.
+You don't see all the false starts.  This gives students a
+misleading view of how things get made.Part of the reason it happens is that writers don't want   
+people to see their mistakes.  But I'm willing to let people
+see an early draft if it will show how much you have
+to rewrite to beat an essay into shape.Below is the oldest version I can find of
+The Age of the Essay   
+(probably the second or third day), with
+text that ultimately survived in 
+red and text that later
+got deleted in gray.
+There seem to be several categories of cuts: things I got wrong,
+things that seem like bragging, flames,
+digressions, stretches of awkward prose, and unnecessary words.I discarded more from the beginning.  That's
+not surprising; it takes a while to hit your stride.  There
+are more digressions at the start, because I'm not sure where
+I'm heading.The amount of cutting is about average.  I probably write
+three to four words for every one that appears in the final
+version of an essay.(Before anyone gets mad at me for opinions expressed here, remember
+that anything you see here that's not in the final version is obviously
+something I chose not to publish, often because I disagree
+with it.)
+Recently a friend said that what he liked about
+my essays was that they weren't written the way
+we'd been taught to write essays in school.  You
+remember: topic sentence, introductory paragraph,
+supporting paragraphs, conclusion.  It hadn't
+occurred to me till then that those horrible things
+we had to write in school were even connected to
+what I was doing now.  But sure enough, I thought,
+they did call them "essays," didn't they?Well, they're not.  Those things you have to write
+in school are not only not essays, they're one of the
+most pointless of all the pointless hoops you have
+to jump through in school.  And I worry that they
+not only teach students the wrong things about writing,
+but put them off writing entirely.So I'm going to give the other side of the story: what
+an essay really is, and how you write one.  Or at least,
+how I write one.  Students be forewarned: if you actually write
+the kind of essay I describe, you'll probably get bad
+grades.  But knowing how it's really done should
+at least help you to understand the feeling of futility
+you have when you're writing the things they tell you to.
+The most obvious difference between real essays and
+the things one has to write in school is that real
+essays are not exclusively about English literature.
+It's a fine thing for schools to
+
+teach students how to
+write.  But for some bizarre reason (actually, a very specific bizarre
+reason that I'll explain in a moment),
+
+the teaching of
+writing has gotten mixed together with the study
+of literature.  And so all over the country, students are
+writing not about how a baseball team with a small budget 
+might compete with the Yankees, or the role of color in
+fashion, or what constitutes a good dessert, but about
+symbolism in Dickens.With obvious 
+results.  Only a few people really
+
+care about
+symbolism in Dickens.  The teacher doesn't.
+The students don't.  Most of the people who've had to write PhD
+disserations about Dickens don't.  And certainly
+
+Dickens himself would be more interested in an essay
+about color or baseball.How did things get this way?  To answer that we have to go back
+almost a thousand years.  Between about 500 and 1000, life was
+not very good in Europe.  The term "dark ages" is presently
+out of fashion as too judgemental (the period wasn't dark; 
+it was just different), but if this label didn't already
+exist, it would seem an inspired metaphor.  What little
+original thought there was took place in lulls between
+constant wars and had something of the character of
+the thoughts of parents with a new baby.
+The most amusing thing written during this
+period, Liudprand of Cremona's Embassy to Constantinople, is,
+I suspect, mostly inadvertantly so.Around 1000 Europe began to catch its breath.
+And once they
+had the luxury of curiosity, one of the first things they discovered
+was what we call "the classics."
+Imagine if we were visited  
+by aliens.  If they could even get here they'd presumably know a
+few things we don't.  Immediately Alien Studies would become
+the most dynamic field of scholarship: instead of painstakingly
+discovering things for ourselves, we could simply suck up
+everything they'd discovered.  So it was in Europe in 1200.
+When classical texts began to circulate in Europe, they contained
+not just new answers, but new questions.  (If anyone proved
+a theorem in christian Europe before 1200, for example, there
+is no record of it.)For a couple centuries, some of the most important work
+being done was intellectual archaelogy.  Those were also
+the centuries during which schools were first established.
+And since reading ancient texts was the essence of what
+scholars did then, it became the basis of the curriculum.By 1700, someone who wanted to learn about
+physics didn't need to start by mastering Greek in order to read Aristotle.  But schools
+change slower than scholarship: the study of
+ancient texts
+had such prestige that it remained the backbone of 
+education
+until the late 19th century.  By then it was merely a tradition.
+It did serve some purposes: reading a foreign language was difficult,
+and thus taught discipline, or at least, kept students busy;
+it introduced students to
+cultures quite different from their own; and its very uselessness
+made it function (like white gloves) as a social bulwark.
+But it certainly wasn't
+true, and hadn't been true for centuries, that students were
+serving apprenticeships in the hottest area of scholarship.Classical scholarship had also changed.  In the early era, philology
+actually mattered.  The texts that filtered into Europe were
+all corrupted to some degree by the errors of translators and
+copyists.  Scholars had to figure out what Aristotle said
+before they could figure out what he meant.  But by the modern
+era such questions were answered as well as they were ever
+going to be.  And so the study of ancient texts became less
+about ancientness and more about texts.The time was then ripe for the question: if the study of
+ancient texts is a valid field for scholarship, why not modern
+texts?  The answer, of course, is that the raison d'etre
+of classical scholarship was a kind of intellectual archaelogy that
+does not need to be done in the case of contemporary authors.
+But for obvious reasons no one wanted to give that answer.
+The archaeological work being mostly done, it implied that
+the people studying the classics were, if not wasting their
+time, at least working on problems of minor importance.And so began the study of modern literature. There was some
+initial resistance, but it didn't last long.
+The limiting
+reagent in the growth of university departments is what
+parents will let undergraduates study.  If parents will let
+their children major in x, the rest follows straightforwardly.
+There will be jobs teaching x, and professors to fill them.
+The professors will establish scholarly journals and publish
+one another's papers.  Universities with x departments will
+subscribe to the journals.  Graduate students who want jobs
+as professors of x will write dissertations about it.  It may
+take a good long while for the more prestigious universities
+to cave in and establish departments in cheesier xes,  but
+at the other end of the scale there are so many universities
+competing to attract students that the mere establishment of
+a discipline requires little more than the desire to do it.High schools imitate universities.
+And so once university
+English departments were established in the late nineteenth century,
+the 'riting component of the 3 Rs 
+was morphed into English.
+With the bizarre consequence that high school students now
+had to write about English literature-- to write, without
+even realizing it, imitations of whatever
+English professors had been publishing in their journals a
+few decades before.   It's no wonder if this seems to the
+student a pointless exercise, because we're now three steps
+removed from real work: the students are imitating English
+professors, who are imitating classical scholars, who are
+merely the inheritors of a tradition growing out of what
+was, 700 years ago, fascinating and urgently needed work.Perhaps high schools should drop English and just teach writing.
+The valuable part of English classes is learning to write, and
+that could be taught better by itself.  Students learn better
+when they're interested in what they're doing, and it's hard
+to imagine a topic less interesting than symbolism in Dickens.
+Most of the people who write about that sort of thing professionally
+are not really interested in it.  (Though indeed, it's been a
+while since they were writing about symbolism; now they're
+writing about gender.)I have no illusions about how eagerly this suggestion will  
+be adopted.  Public schools probably couldn't stop teaching
+English even if they wanted to; they're probably required to by
+law.  But here's a related suggestion that goes with the grain
+instead of against it: that universities establish a
+writing major.  Many of the students who now major in English
+would major in writing if they could, and most would
+be better off.It will be argued that it is a good thing for students to be
+exposed to their literary heritage.  Certainly.  But is that
+more important than that they learn to write well?  And are
+English classes even the place to do it?  After all,
+the average public high school student gets zero exposure to  
+his artistic heritage.  No disaster results.
+The people who are interested in art learn about it for
+themselves, and those who aren't don't.  I find that American
+adults are no better or worse informed about literature than
+art, despite the fact that they spent years studying literature
+in high school and no time at all studying art.  Which presumably
+means that what they're taught in school is rounding error 
+compared to what they pick up on their own.Indeed, English classes may even be harmful.  In my case they
+were effectively aversion therapy.  Want to make someone dislike
+a book?  Force him to read it and write an essay about it.
+And make the topic so intellectually bogus that you
+could not, if asked, explain why one ought to write about it.
+I love to read more than anything, but by the end of high school
+I never read the books we were assigned.  I was so disgusted with
+what we were doing that it became a point of honor
+with me to write nonsense at least as good at the other students'
+without having more than glanced over the book to learn the names
+of the characters and a few random events in it.I hoped this might be fixed in college, but I found the same
+problem there.  It was not the teachers.  It was English.   
+We were supposed to read novels and write essays about them.
+About what, and why?  That no one seemed to be able to explain.
+Eventually by trial and error I found that what the teacher  
+wanted us to do was pretend that the story had really taken
+place, and to analyze based on what the characters said and did (the
+subtler clues, the better) what their motives must have been.
+One got extra credit for motives having to do with class,
+as I suspect one must now for those involving gender and  
+sexuality.  I learned how to churn out such stuff well enough
+to get an A, but I never took another English class.And the books we did these disgusting things to, like those
+we mishandled in high school, I find still have black marks
+against them in my mind.  The one saving grace was that   
+English courses tend to favor pompous, dull writers like
+Henry James, who deserve black marks against their names anyway.
+One of the principles the IRS uses in deciding whether to
+allow deductions is that, if something is fun, it isn't work.
+Fields that are intellectually unsure of themselves rely on
+a similar principle.  Reading P.G. Wodehouse or Evelyn Waugh or
+Raymond Chandler is too obviously pleasing to seem like
+serious work, as reading Shakespeare would have been before 
+English evolved enough to make it an effort to understand him. [sh]
+And so good writers (just you wait and see who's still in
+print in 300 years) are less likely to have readers turned   
+against them by clumsy, self-appointed tour guides.
+The other big difference between a real essay and the 
+things
+they make you write in school is that a real essay doesn't 
+take a position and then defend it.  That principle,
+like the idea that we ought to be writing about literature,   
+turns out to be another intellectual hangover of long
+forgotten origins.  It's often mistakenly believed that
+medieval universities were mostly seminaries.  In fact they
+were more law schools.  And at least in our tradition
+lawyers are advocates: they are
+trained to be able to
+take
+either side of an argument and make as good a case for it  
+as they can. Whether or not this is a good idea (in the case of prosecutors,
+it probably isn't), it tended to pervade
+the atmosphere of
+early universities.  After the lecture the most common form
+of discussion was the disputation.  This idea
+is at least
+nominally preserved in our present-day thesis defense-- indeed,
+in the very word thesis.  Most people treat the words 
+thesis
+and dissertation as interchangeable, but originally, at least,
+a thesis was a position one took and the dissertation was
+the argument by which one defended it.I'm not complaining that we blur these two words together.
+As far as I'm concerned, the sooner we lose the original
+sense of the word thesis, the better.  For many, perhaps most,  
+graduate students, it is stuffing a square peg into a round
+hole to try to recast one's work as a single thesis.  And
+as for the disputation, that seems clearly a net lose.
+Arguing two sides of a case may be a necessary evil in a
+legal dispute, but it's not the best way to get at the truth,
+as I think lawyers would be the first to admit.
+And yet this principle is built into the very structure of  
+the essays
+they teach you to write in high school.  The topic
+sentence is your thesis, chosen in advance, the supporting 
+paragraphs the blows you strike in the conflict, and the
+conclusion--- uh, what it the conclusion?  I was never sure  
+about that in high school.  If your thesis was well expressed,
+what need was there to restate it?  In theory it seemed that
+the conclusion of a really good essay ought not to need to   
+say any more than QED.
+But when you understand the origins
+of this sort of "essay", you can see where the
+conclusion comes from.  It's the concluding remarks to the 
+jury.
+What other alternative is there?  To answer that
+we have to
+reach back into history again, though this time not so far.
+To Michel de Montaigne, inventor of the essay.
+He was
+doing something quite different from what a
+lawyer does,
+and
+the difference is embodied in the name.  Essayer is the French
+verb meaning "to try" (the cousin of our word assay),
+
+and an "essai" is an effort.
+An essay is something you
+write in order
+to figure something out.Figure out what?  You don't know yet.  And so you can't begin with a
+thesis, because you don't have one, and may never have 
+one.  An essay doesn't begin with a statement, but with a  
+question.  In a real essay, you don't take a position and
+defend it.  You see a door that's ajar, and you open it and
+walk in to see what's inside.If all you want to do is figure things out, why do you need
+to write anything, though?  Why not just sit and think?  Well,
+there precisely is Montaigne's great discovery.  Expressing
+ideas helps to form them.  Indeed, helps is far too weak a
+word.  90%
+of what ends up in my essays was stuff
+I only
+thought of when I sat down to write them.  That's why I
+write them.So there's another difference between essays and
+the things
+you have to write in school.   In school
+
+you are, in theory,
+explaining yourself to someone else.  In the best case---if
+you're really organized---you're just writing it down.
+In a real essay you're writing for yourself.  You're
+thinking out loud.But not quite.  Just as inviting people over forces you to
+clean up your apartment, writing something that you know
+
+other people will read forces you to think well.  So it
+does matter to have an audience.  The things I've written
+just for myself are no good.  Indeed, they're bad in
+a particular way:
+they tend to peter out.  When I run into
+difficulties, I notice that I
+tend to conclude with a few vague
+questions and then drift off to get a cup of tea.This seems a common problem.
+It's practically the standard
+ending in blog entries--- with the addition of a "heh" or an 
+emoticon, prompted by the all too accurate sense that
+something is missing.And indeed, a lot of
+published essays peter out in this
+same way.
+Particularly the sort written by the staff writers of newsmagazines.  Outside writers tend to supply
+editorials of the defend-a-position variety, which
+make a beeline toward a rousing (and
+foreordained) conclusion.   But the staff writers feel
+obliged to write something more
+balanced, which in
+practice ends up meaning blurry.
+Since they're
+writing for a popular magazine, they start with the
+most radioactively controversial questions, from which
+(because they're writing for a popular magazine)
+they then proceed to recoil from
+in terror.
+Gay marriage, for or
+against?  This group says one thing.  That group says
+another.  One thing is certain: the question is a
+complex one.  (But don't get mad at us.  We didn't
+draw any conclusions.)Questions aren't enough.  An essay has to come up with answers.
+They don't always, of course.  Sometimes you start with a  
+promising question and get nowhere.  But those you don't
+publish.  Those are like experiments that get inconclusive
+results.   Something you publish ought to tell the reader  
+something he didn't already know.
+But what you tell him doesn't matter, so long as   
+it's interesting.  I'm sometimes accused of meandering.
+In defend-a-position writing that would be a flaw.
+There you're not concerned with truth.  You already
+know where you're going, and you want to go straight there,
+blustering through obstacles, and hand-waving
+your way across swampy ground.  But that's not what
+you're trying to do in an essay.  An essay is supposed to
+be a search for truth.  It would be suspicious if it didn't
+meander.The Meander is a river in Asia Minor (aka
+Turkey).
+As you might expect, it winds all over the place.
+But does it
+do this out of frivolity?   Quite the opposite.
+Like all rivers, it's rigorously following the laws of physics.
+The path it has discovered,
+winding as it is, represents
+the most economical route to the sea.The river's algorithm is simple.  At each step, flow down.
+For the essayist this translates to: flow interesting.
+Of all the places to go next, choose
+whichever seems
+most interesting.I'm pushing this metaphor a bit.  An essayist
+can't have
+quite as little foresight as a river.  In fact what you do
+(or what I do) is somewhere between a river and a roman
+road-builder.  I have a general idea of the direction
+I want to go in, and
+I choose the next topic with that in mind.  This essay is
+about writing, so I do occasionally yank it back in that
+direction, but it is not all the sort of essay I
+thought I was going to write about writing.Note too that hill-climbing (which is what this algorithm is
+called) can get you in trouble.
+Sometimes, just
+like a river,
+you
+run up against a blank wall.  What
+I do then is just 
+what the river does: backtrack.
+At one point in this essay
+I found that after following a certain thread I ran out
+of ideas.  I had to go back n
+paragraphs and start over
+in another direction.  For illustrative purposes I've left
+the abandoned branch as a footnote.
+Err on the side of the river.  An essay is not a reference
+work.  It's not something you read looking for a specific
+answer, and feel cheated if you don't find it.  I'd much
+rather read an essay that went off in an unexpected but
+interesting direction than one that plodded dutifully along
+a prescribed course.So what's interesting?  For me, interesting means surprise.
+Design, as Matz
+has said, should follow the principle of
+least surprise.
+A button that looks like it will make a
+machine stop should make it stop, not speed up.  Essays
+should do the opposite.  Essays should aim for maximum
+surprise.I was afraid of flying for a long time and could only travel
+vicariously.  When friends came back from faraway places,
+it wasn't just out of politeness that I asked them about
+their trip.
+I really wanted to know.  And I found that
+the best way to get information out of them was to ask
+what surprised them.  How was the place different from what
+they expected?  This is an extremely useful question.
+You can ask it of even
+the most unobservant people, and it will
+extract information they didn't even know they were
+recording. Indeed, you can ask it in real time.  Now when I go somewhere
+new, I make a note of what surprises me about it.  Sometimes I
+even make a conscious effort to visualize the place beforehand,
+so I'll have a detailed image to diff with reality.
+Surprises are facts
+you didn't already 
+know.
+But they're
+more than that.  They're facts
+that contradict things you
+thought you knew.  And so they're the most valuable sort of
+fact you can get.  They're like a food that's not merely
+healthy, but counteracts the unhealthy effects of things
+you've already eaten.
+How do you find surprises?  Well, therein lies half
+the work of essay writing.  (The other half is expressing
+yourself well.)   You can at least
+use yourself as a
+proxy for the reader.  You should only write about things
+you've thought about a lot.  And anything you come across
+that surprises you, who've thought about the topic a lot,
+will probably surprise most readers.For example, in a recent essay I pointed out that because
+you can only judge computer programmers by working with
+them, no one knows in programming who the heroes should
+be.
+I
+certainly
+didn't realize this when I started writing
+the 
+essay, and even now I find it kind of weird.  That's
+what you're looking for.So if you want to write essays, you need two ingredients:
+you need
+a few topics that you think about a lot, and you
+need some ability to ferret out the unexpected.What should you think about?  My guess is that it
+doesn't matter.  Almost everything is
+interesting if you get deeply
+enough into it.  The one possible exception
+are
+things
+like working in fast food, which
+have deliberately had all
+the variation sucked out of them.
+In retrospect, was there
+anything interesting about working in Baskin-Robbins?
+Well, it was interesting to notice
+how important color was
+to the customers.  Kids a certain age would point into
+the case and say that they wanted yellow.  Did they want
+French Vanilla or Lemon?  They would just look at you
+blankly.  They wanted yellow.  And then there was the
+mystery of why the perennial favorite Pralines n' Cream
+was so appealing. I'm inclined now to
+think it was the salt.
+And the mystery of why Passion Fruit tasted so disgusting.
+People would order it because of the name, and were always
+disappointed.  It should have been called In-sink-erator
+Fruit.
+And there was
+the difference in the way fathers and
+mothers bought ice cream for their kids.
+Fathers tended to
+adopt the attitude of
+benevolent kings bestowing largesse,
+and mothers that of
+harried bureaucrats,
+giving in to
+pressure against their better judgement.
+So, yes, there does seem to be material, even in
+fast food.What about the other half, ferreting out the unexpected?
+That may require some natural ability.  I've noticed for
+a long time that I'm pathologically observant.  ....[That was as far as I'd gotten at the time.]Notes[sh] In Shakespeare's own time, serious writing meant theological
+discourses, not the bawdy plays acted over on the other  
+side of the river among the bear gardens and whorehouses.The other extreme, the work that seems formidable from the moment
+it's created (indeed, is deliberately intended to be)
+is represented by Milton.  Like the Aeneid, Paradise Lost is a
+rock imitating a butterfly that happened to get fossilized.
+Even Samuel Johnson seems to have balked at this, on the one  
+hand paying Milton the compliment of an extensive biography,
+and on the other writing of Paradise Lost that "none who read it
+ever wished it longer."

recipes/experimental/long-context/H2O/data/PaulGrahamEssays/love.txt

@@ -0,0 +1,376 @@
+
+
+Want to start a startup?  Get funded by
+Y Combinator.
+
+
+
+
+January 2006To do something well you have to like it.   That idea is not exactly
+novel.  We've got it down to four words: "Do what you love."  But
+it's not enough just to tell people that.  Doing what you love is
+complicated.The very idea is foreign to what most of us learn as kids.  When I
+was a kid, it seemed as if work and fun were opposites by definition.
+Life had two states: some of the time adults were making you do
+things, and that was called work; the rest of the time you could
+do what you wanted, and that was called playing.  Occasionally the
+things adults made you do were fun, just as, occasionally, playing
+wasn't—for example, if you fell and hurt yourself.  But except
+for these few anomalous cases, work was pretty much defined as
+not-fun.And it did not seem to be an accident. School, it was implied, was
+tedious because it was preparation for grownup work.The world then was divided into two groups, grownups and kids.
+Grownups, like some kind of cursed race, had to work.  Kids didn't,
+but they did have to go to school, which was a dilute version of
+work meant to prepare us for the real thing.  Much as we disliked
+school, the grownups all agreed that grownup work was worse, and
+that we had it easy.Teachers in particular all seemed to believe implicitly that work
+was not fun.  Which is not surprising: work wasn't fun for most of
+them.  Why did we have to memorize state capitals instead of playing
+dodgeball?  For the same reason they had to watch over a bunch of
+kids instead of lying on a beach.  You couldn't just do what you
+wanted.I'm not saying we should let little kids do whatever they want.
+They may have to be made to work on certain things.  But if we make
+kids work on dull stuff, it might be wise to tell them that tediousness
+is not the defining quality of work, and indeed that the reason
+they have to work on dull stuff now is so they can work on more
+interesting stuff later.
+[1]Once, when I was about 9 or 10, my father told me I could be whatever
+I wanted when I grew up, so long as I enjoyed it.  I remember that
+precisely because it seemed so anomalous.  It was like being told
+to use dry water.  Whatever I thought he meant, I didn't think he
+meant work could literally be fun—fun like playing.  It
+took me years to grasp that.JobsBy high school, the prospect of an actual job was on the horizon.
+Adults would sometimes come to speak to us about their work, or we
+would go to see them at work.  It was always understood that they
+enjoyed what they did.  In retrospect I think one may have: the
+private jet pilot.  But I don't think the bank manager really did.The main reason they all acted as if they enjoyed their work was
+presumably the upper-middle class convention that you're supposed
+to.  It would not merely be bad for your career to say that you
+despised your job, but a social faux-pas.Why is it conventional to pretend to like what you do?  The first
+sentence of this essay explains that.  If you have to like something
+to do it well, then the most successful people will all like what
+they do.  That's where the upper-middle class tradition comes from.
+Just as houses all over America are full of 
+chairs
+that are, without
+the owners even knowing it, nth-degree imitations of chairs designed
+250 years ago for French kings, conventional attitudes about work
+are, without the owners even knowing it, nth-degree imitations of
+the attitudes of people who've done great things.What a recipe for alienation.  By the time they reach an age to
+think about what they'd like to do, most kids have been thoroughly
+misled about the idea of loving one's work.  School has trained
+them to regard work as an unpleasant duty.  Having a job is said
+to be even more onerous than schoolwork.  And yet all the adults
+claim to like what they do.  You can't blame kids for thinking "I
+am not like these people; I am not suited to this world."Actually they've been told three lies: the stuff they've been taught
+to regard as work in school is not real work; grownup work is not
+(necessarily) worse than schoolwork; and many of the adults around
+them are lying when they say they like what they do.The most dangerous liars can be the kids' own parents.  If you take
+a boring job to give your family a high standard of living, as so
+many people do, you risk infecting your kids with the idea that
+work is boring. 
+[2]
+Maybe it would be better for kids in this one
+case if parents were not so unselfish.  A parent who set an example
+of loving their work might help their kids more than an expensive
+house.
+[3]It was not till I was in college that the idea of work finally broke
+free from the idea of making a living.  Then the important question
+became not how to make money, but what to work on.  Ideally these
+coincided, but some spectacular boundary cases (like Einstein in
+the patent office) proved they weren't identical.The definition of work was now to make some original contribution
+to the world, and in the process not to starve.  But after the habit
+of so many years my idea of work still included a large component
+of pain.  Work still seemed to require discipline, because only
+hard problems yielded grand results, and hard problems couldn't
+literally be fun.   Surely one had to force oneself to work on them.If you think something's supposed to hurt, you're less likely to
+notice if you're doing it wrong.  That about sums up my experience
+of graduate school.BoundsHow much are you supposed to like what you do?  Unless you
+know that, you don't know when to stop searching. And if, like most
+people, you underestimate it, you'll tend to stop searching too
+early.  You'll end up doing something chosen for you by your parents,
+or the desire to make money, or prestige—or sheer inertia.Here's an upper bound: Do what you love doesn't mean, do what you
+would like to do most this second.  Even Einstein probably
+had moments when he wanted to have a cup of coffee, but told himself
+he ought to finish what he was working on first.It used to perplex me when I read about people who liked what they
+did so much that there was nothing they'd rather do.  There didn't
+seem to be any sort of work I liked that much.  If I had a
+choice of (a) spending the next hour working on something or (b)
+be teleported to Rome and spend the next hour wandering about, was
+there any sort of work I'd prefer?  Honestly, no.But the fact is, almost anyone would rather, at any given moment,
+float about in the Carribbean, or have sex, or eat some delicious
+food, than work on hard problems.  The rule about doing what you
+love assumes a certain length of time.  It doesn't mean, do what
+will make you happiest this second, but what will make you happiest
+over some longer period, like a week or a month.Unproductive pleasures pall eventually.  After a while you get tired
+of lying on the beach.  If you want to stay happy, you have to do
+something.As a lower bound, you have to like your work more than any unproductive
+pleasure.  You have to like what you do enough that the concept of
+"spare time" seems mistaken.  Which is not to say you have to spend
+all your time working.  You can only work so much before you get
+tired and start to screw up.  Then you want to do something else—even something mindless.  But you don't regard this time as the
+prize and the time you spend working as the pain you endure to earn
+it.I put the lower bound there for practical reasons.  If your work
+is not your favorite thing to do, you'll have terrible problems
+with procrastination.  You'll have to force yourself to work,  and
+when you resort to that the results are distinctly inferior.To be happy I think you have to be doing something you not only
+enjoy, but admire.  You have to be able to say, at the end, wow,
+that's pretty cool.  This doesn't mean you have to make something.
+If you learn how to hang glide, or to speak a foreign language
+fluently, that will be enough to make you say, for a while at least,
+wow, that's pretty cool.  What there has to be is a test.So one thing that falls just short of the standard, I think, is
+reading books.  Except for some books in math and the hard sciences,
+there's no test of how well you've read a book, and that's why
+merely reading books doesn't quite feel like work.  You have to do
+something with what you've read to feel productive.I think the best test is one Gino Lee taught me: to try to do things
+that would make your friends say wow.  But it probably wouldn't
+start to work properly till about age 22, because most people haven't
+had a big enough sample to pick friends from before then.SirensWhat you should not do, I think, is worry about the opinion of
+anyone beyond your friends.  You shouldn't worry about prestige.
+Prestige is the opinion of the rest of the world.  When you can ask
+the opinions of people whose judgement you respect, what does it
+add to consider the opinions of people you don't even know? 
+[4]This is easy advice to give.  It's hard to follow, especially when
+you're young.  
+[5]
+Prestige is like a powerful magnet that warps
+even your beliefs about what you enjoy.  It causes you to work not
+on what you like, but what you'd like to like.That's what leads people to try to write novels, for example.  They
+like reading novels.  They notice that people who write them win
+Nobel prizes.  What could be more wonderful, they think, than to
+be a novelist?  But liking the idea of being a novelist is not
+enough; you have to like the actual work of novel-writing if you're
+going to be good at it; you have to like making up elaborate lies.Prestige is just fossilized inspiration.  If you do anything well
+enough, you'll make it prestigious.  Plenty of things we now
+consider prestigious were anything but at first.  Jazz comes to
+mind—though almost any established art form would do.   So just
+do what you like, and let prestige take care of itself.Prestige is especially dangerous to the ambitious.  If you want to
+make ambitious people waste their time on errands, the way to do
+it is to bait the hook with prestige.  That's the recipe for getting
+people to give talks, write forewords, serve on committees, be
+department heads, and so on.  It might be a good rule simply to
+avoid any prestigious task. If it didn't suck, they wouldn't have
+had to make it prestigious.Similarly, if you admire two kinds of work equally, but one is more
+prestigious, you should probably choose the other.  Your opinions
+about what's admirable are always going to be slightly influenced
+by prestige, so if the two seem equal to you, you probably have
+more genuine admiration for the less prestigious one.The other big force leading people astray is money.  Money by itself
+is not that dangerous.  When something pays well but is regarded
+with contempt, like telemarketing, or prostitution, or personal
+injury litigation, ambitious people aren't tempted by it.  That
+kind of work ends up being done by people who are "just trying to
+make a living."  (Tip: avoid any field whose practitioners say
+this.)  The danger is when money is combined with prestige, as in,
+say, corporate law, or medicine.  A comparatively safe and prosperous
+career with some automatic baseline prestige is dangerously tempting
+to someone young, who hasn't thought much about what they really
+like.The test of whether people love what they do is whether they'd do
+it even if they weren't paid for it—even if they had to work at
+another job to make a living.  How many corporate lawyers would do
+their current work if they had to do it for free, in their spare
+time, and take day jobs as waiters to support themselves?This test is especially helpful in deciding between different kinds
+of academic work, because fields vary greatly in this respect.  Most
+good mathematicians would work on math even if there were no jobs
+as math professors, whereas in the departments at the other end of
+the spectrum, the availability of teaching jobs is the driver:
+people would rather be English professors than work in ad agencies,
+and publishing papers is the way you compete for such jobs.  Math
+would happen without math departments, but it is the existence of
+English majors, and therefore jobs teaching them, that calls into
+being all those thousands of dreary papers about gender and identity
+in the novels of Conrad.  No one does 
+that 
+kind of thing for fun.The advice of parents will tend to err on the side of money.  It
+seems safe to say there are more undergrads who want to be novelists
+and whose parents want them to be doctors than who want to be doctors
+and whose parents want them to be novelists.  The kids think their
+parents are "materialistic." Not necessarily.  All parents tend to
+be more conservative for their kids than they would for themselves,
+simply because, as parents, they share risks more than rewards.  If
+your eight year old son decides to climb a tall tree, or your teenage
+daughter decides to date the local bad boy, you won't get a share
+in the excitement, but if your son falls, or your daughter gets
+pregnant, you'll have to deal with the consequences.DisciplineWith such powerful forces leading us astray, it's not surprising
+we find it so hard to discover what we like to work on.  Most people
+are doomed in childhood by accepting the axiom that work = pain.
+Those who escape this are nearly all lured onto the rocks by prestige
+or money.  How many even discover something they love to work on?
+A few hundred thousand, perhaps, out of billions.It's hard to find work you love; it must be, if so few do.  So don't
+underestimate this task.  And don't feel bad if you haven't succeeded
+yet.  In fact, if you admit to yourself that you're discontented,
+you're a step ahead of most people, who are still in denial.  If
+you're surrounded by colleagues who claim to enjoy work that you
+find contemptible, odds are they're lying to themselves.  Not
+necessarily, but probably.Although doing great work takes less discipline than people think—because the way to do great work is to find something you like so
+much that you don't have to force yourself to do it—finding
+work you love does usually require discipline.   Some people are
+lucky enough to know what they want to do when they're 12, and just
+glide along as if they were on railroad tracks.  But this seems the
+exception.  More often people who do great things have careers with
+the trajectory of a ping-pong ball.  They go to school to study A,
+drop out and get a job doing B, and then become famous for C after
+taking it up on the side.Sometimes jumping from one sort of work to another is a sign of
+energy, and sometimes it's a sign of laziness.  Are you dropping
+out, or boldly carving a new path?  You often can't tell yourself.
+Plenty of people who will later do great things seem to be disappointments
+early on, when they're trying to find their niche.Is there some test you can use to keep yourself honest?  One is to
+try to do a good job at whatever you're doing, even if you don't
+like it.  Then at least you'll know you're not using dissatisfaction
+as an excuse for being lazy.  Perhaps more importantly, you'll get
+into the habit of doing things well.Another test you can use is: always produce.  For example, if you
+have a day job you don't take seriously because you plan to be a
+novelist, are you producing?  Are you writing pages of fiction,
+however bad?  As long as you're producing, you'll know you're not
+merely using the hazy vision of the grand novel you plan to write
+one day as an opiate.  The view of it will be obstructed by the all
+too palpably flawed one you're actually writing."Always produce" is also a heuristic for finding the work you love.
+If you subject yourself to that constraint, it will automatically
+push you away from things you think you're supposed to work on,
+toward things you actually like.  "Always produce" will discover
+your life's work the way water, with the aid of gravity, finds the
+hole in your roof.Of course, figuring out what you like to work on doesn't mean you
+get to work on it.  That's a separate question.  And if you're
+ambitious you have to keep them separate: you have to make a conscious
+effort to keep your ideas about what you want from being contaminated
+by what seems possible. 
+[6]It's painful to keep them apart, because it's painful to observe
+the gap between them. So most people pre-emptively lower their
+expectations.  For example, if you asked random people on the street
+if they'd like to be able to draw like Leonardo, you'd find most
+would say something like "Oh, I can't draw."  This is more a statement
+of intention than fact; it means, I'm not going to try.  Because
+the fact is, if you took a random person off the street and somehow
+got them to work as hard as they possibly could at drawing for the
+next twenty years, they'd get surprisingly far.  But it would require
+a great moral effort; it would mean staring failure in the eye every
+day for years.  And so to protect themselves people say "I can't."Another related line you often hear is that not everyone can do
+work they love—that someone has to do the unpleasant jobs.  Really?
+How do you make them?  In the US the only mechanism for forcing
+people to do unpleasant jobs is the draft, and that hasn't been
+invoked for over 30 years.  All we can do is encourage people to
+do unpleasant work, with money and prestige.If there's something people still won't do, it seems as if society
+just has to make do without.  That's what happened with domestic
+servants.  For millennia that was the canonical example of a job
+"someone had to do."  And yet in the mid twentieth century servants
+practically disappeared in rich countries, and the rich have just
+had to do without.So while there may be some things someone has to do, there's a good
+chance anyone saying that about any particular job is mistaken.
+Most unpleasant jobs would either get automated or go undone if no
+one were willing to do them.Two RoutesThere's another sense of "not everyone can do work they love"
+that's all too true, however.  One has to make a living, and it's
+hard to get paid for doing work you love.  There are two routes to
+that destination:
+
+  The organic route: as you become more eminent, gradually to
+  increase the parts of your job that you like at the expense of
+  those you don't.The two-job route: to work at things you don't like to get money
+  to work on things you do.
+
+The organic route is more common.  It happens naturally to anyone
+who does good work.  A young architect has to take whatever work
+he can get, but if he does well he'll gradually be in a position
+to pick and choose among projects.  The disadvantage of this route
+is that it's slow and uncertain.  Even tenure is not real freedom.The two-job route has several variants depending on how long you
+work for money at a time.  At one extreme is the "day job," where
+you work regular hours at one job to make money, and work on what
+you love in your spare time.  At the other extreme you work at
+something till you make enough not to 
+have to work for money again.The two-job route is less common than the organic route, because
+it requires a deliberate choice.  It's also more dangerous.  Life
+tends to get more expensive as you get older, so it's easy to get
+sucked into working longer than you expected at the money job.
+Worse still, anything you work on changes you.  If you work too
+long on tedious stuff, it will rot your brain.  And the best paying
+jobs are most dangerous, because they require your full attention.The advantage of the two-job route is that it lets you jump over
+obstacles.  The landscape of possible jobs isn't flat; there are
+walls of varying heights between different kinds of work. 
+[7]
+The trick of maximizing the parts of your job that you like can get you
+from architecture to product design, but not, probably, to music.
+If you make money doing one thing and then work on another, you
+have more freedom of choice.Which route should you take?  That depends on how sure you are of
+what you want to do, how good you are at taking orders, how much
+risk you can stand, and the odds that anyone will pay (in your
+lifetime) for what you want to do.  If you're sure of the general
+area you want to work in and it's something people are likely to
+pay you for, then you should probably take the organic route.  But
+if you don't know what you want to work on, or don't like to take
+orders, you may want to take the two-job route, if you can stand
+the risk.Don't decide too soon.  Kids who know early what they want to do
+seem impressive, as if they got the answer to some math question
+before the other kids.  They have an answer, certainly, but odds
+are it's wrong.A friend of mine who is a quite successful doctor complains constantly
+about her job.  When people applying to medical school ask her for
+advice, she wants to shake them and yell "Don't do it!"  (But she
+never does.) How did she get into this fix?  In high school she
+already wanted to be a doctor.  And she is so ambitious and determined
+that she overcame every obstacle along the way—including,
+unfortunately, not liking it.Now she has a life chosen for her by a high-school kid.When you're young, you're given the impression that you'll get
+enough information to make each choice before you need to make it.
+But this is certainly not so with work.  When you're deciding what
+to do, you have to operate on ridiculously incomplete information.
+Even in college you get little idea what various types of work are
+like.  At best you may have a couple internships, but not all jobs
+offer internships, and those that do don't teach you much more about
+the work than being a batboy teaches you about playing baseball.In the design of lives, as in the design of most other things, you
+get better results if you use flexible media.  So unless you're
+fairly sure what you want to do, your best bet may be to choose a
+type of work that could turn into either an organic or two-job
+career.  That was probably part of the reason I chose computers.
+You can be a professor, or make a lot of money, or morph it into
+any number of other kinds of work.It's also wise, early on, to seek jobs that let you do many different
+things, so you can learn faster what various kinds of work are like.
+Conversely, the extreme version of the two-job route is dangerous
+because it teaches you so little about what you like.  If you work
+hard at being a bond trader for ten years, thinking that you'll
+quit and write novels when you have enough money, what happens when
+you quit and then discover that you don't actually like writing
+novels?Most people would say, I'd take that problem.  Give me a million
+dollars and I'll figure out what to do.  But it's harder than it
+looks.  Constraints give your life shape.  Remove them and most
+people have no idea what to do: look at what happens to those who
+win lotteries or inherit money.  Much as everyone thinks they want
+financial security, the happiest people are not those who have it,
+but those who like what they do.  So a plan that promises freedom
+at the expense of knowing what to do with it may not be as good as
+it seems.Whichever route you take, expect a struggle.  Finding work you love
+is very difficult.  Most people fail.  Even if you succeed, it's
+rare to be free to work on what you want till your thirties or
+forties.  But if you have the destination in sight you'll be more
+likely to arrive at it.  If you know you can love work, you're in
+the home stretch, and if you know what work you love, you're
+practically there.Notes[1]
+Currently we do the opposite: when we make kids do boring work,
+like arithmetic drills, instead of admitting frankly that it's
+boring, we try to disguise it with superficial decorations.[2]
+One father told me about a related phenomenon: he found himself
+concealing from his family how much he liked his work.  When he
+wanted to go to work on a saturday, he found it easier to say that
+it was because he "had to" for some reason, rather than admitting
+he preferred to work than stay home with them.[3]
+Something similar happens with suburbs.  Parents move to suburbs
+to raise their kids in a safe environment, but suburbs are so dull
+and artificial that by the time they're fifteen the kids are convinced
+the whole world is boring.[4]
+I'm not saying friends should be the only audience for your
+work.  The more people you can help, the better.  But friends should
+be your compass.[5]
+Donald Hall said young would-be poets were mistaken to be so
+obsessed with being published.  But you can imagine what it would
+do for a 24 year old to get a poem published in The New Yorker.
+Now to people he meets at parties he's a real poet.  Actually he's
+no better or worse than he was before, but to a clueless audience
+like that, the approval of an official authority makes all the
+difference.   So it's a harder problem than Hall realizes.  The
+reason the young care so much about prestige is that the people
+they want to impress are not very discerning.[6]
+This is isomorphic to the principle that you should prevent
+your beliefs about how things are from being contaminated by how
+you wish they were.  Most people let them mix pretty promiscuously.
+The continuing popularity of religion is the most visible index of
+that.[7]
+A more accurate metaphor would be to say that the graph of jobs
+is not very well connected.Thanks to Trevor Blackwell, Dan Friedman, Sarah Harlin,
+Jessica Livingston, Jackie McDonough, Robert Morris, Peter Norvig, 
+David Sloo, and Aaron Swartz
+for reading drafts of this.

recipes/experimental/long-context/H2O/data/PaulGrahamEssays/mod.txt

@@ -0,0 +1,54 @@
+December 2019There are two distinct ways to be politically moderate: on purpose
+and by accident. Intentional moderates are trimmers, deliberately
+choosing a position mid-way between the extremes of right and left.
+Accidental moderates end up in the middle, on average, because they
+make up their own minds about each question, and the far right and
+far left are roughly equally wrong.You can distinguish intentional from accidental moderates by the
+distribution of their opinions. If the far left opinion on some
+matter is 0 and the far right opinion 100, an intentional moderate's
+opinion on every question will be near 50. Whereas an accidental
+moderate's opinions will be scattered over a broad range, but will,
+like those of the intentional moderate, average to about 50.Intentional moderates are similar to those on the far left and the
+far right in that their opinions are, in a sense, not their own.
+The defining quality of an ideologue, whether on the left or the
+right, is to acquire one's opinions in bulk. You don't get to pick
+and choose. Your opinions about taxation can be predicted from your
+opinions about sex. And although intentional moderates
+might seem to be the opposite of ideologues, their beliefs (though
+in their case the word "positions" might be more accurate) are also
+acquired in bulk. If the median opinion shifts to the right or left,
+the intentional moderate must shift with it. Otherwise they stop
+being moderate.Accidental moderates, on the other hand, not only choose their own
+answers, but choose their own questions. They may not care at all
+about questions that the left and right both think are terribly
+important. So you can only even measure the politics of an accidental
+moderate from the intersection of the questions they care about and
+those the left and right care about, and this can
+sometimes be vanishingly small.It is not merely a manipulative rhetorical trick to say "if you're
+not with us, you're against us," but often simply false.Moderates are sometimes derided as cowards, particularly by 
+the extreme left. But while it may be accurate to call intentional
+moderates cowards, openly being an accidental moderate requires the
+most courage of all, because you get attacked from both right and
+left, and you don't have the comfort of being an orthodox member
+of a large group to sustain you.Nearly all the most impressive people I know are accidental moderates.
+If I knew a lot of professional athletes, or people in the entertainment
+business, that might be different. Being on the far left or far
+right doesn't affect how fast you run or how well you sing. But
+someone who works with ideas has to be independent-minded to do it
+well.Or more precisely, you have to be independent-minded about the ideas
+you work with. You could be mindlessly doctrinaire in your politics
+and still be a good mathematician. In the 20th century, a lot of
+very smart people were Marxists — just no one who was smart about
+the subjects Marxism involves. But if the ideas you use in your
+work intersect with the politics of your time, you have two choices:
+be an accidental moderate, or be mediocre.Notes[1] It's possible in theory for one side to be entirely right and
+the other to be entirely wrong. Indeed, ideologues must always
+believe this is the case. But historically it rarely has been.[2] For some reason the far right tend to ignore moderates rather
+than despise them as backsliders. I'm not sure why. Perhaps it
+means that the far right is less ideological than the far left. Or
+perhaps that they are more confident, or more resigned, or simply
+more disorganized. I just don't know.[3] Having heretical opinions doesn't mean you have to express
+them openly. It may be
+easier to have them if you don't.
+Thanks to Austen Allred, Trevor Blackwell, Patrick Collison, Jessica Livingston,
+Amjad Masad, Ryan Petersen, and Harj Taggar for reading drafts of this.

recipes/experimental/long-context/H2O/data/PaulGrahamEssays/newideas.txt

@@ -0,0 +1,114 @@
+May 2021There's one kind of opinion I'd be very afraid to express publicly.
+If someone I knew to be both a domain expert and a reasonable person
+proposed an idea that sounded preposterous, I'd be very reluctant
+to say "That will never work."Anyone who has studied the history of ideas, and especially the
+history of science, knows that's how big things start. Someone
+proposes an idea that sounds crazy, most people dismiss it, then
+it gradually takes over the world.Most implausible-sounding ideas are in fact bad and could be safely
+dismissed. But not when they're proposed by reasonable domain
+experts. If the person proposing the idea is reasonable, then they
+know how implausible it sounds. And yet they're proposing it anyway.
+That suggests they know something you don't. And if they have deep
+domain expertise, that's probably the source of it.
+[1]Such ideas are not merely unsafe to dismiss, but disproportionately
+likely to be interesting. When the average person proposes an
+implausible-sounding idea, its implausibility is evidence of their
+incompetence. But when a reasonable domain expert does it, the
+situation is reversed. There's something like an efficient market
+here: on average the ideas that seem craziest will, if correct,
+have the biggest effect. So if you can eliminate the theory that
+the person proposing an implausible-sounding idea is incompetent,
+its implausibility switches from evidence that it's boring to
+evidence that it's exciting.
+[2]Such ideas are not guaranteed to work. But they don't have to be.
+They just have to be sufficiently good bets — to have sufficiently
+high expected value. And I think on average they do. I think if you
+bet on the entire set of implausible-sounding ideas proposed by
+reasonable domain experts, you'd end up net ahead.The reason is that everyone is too conservative. The word "paradigm"
+is overused, but this is a case where it's warranted. Everyone is
+too much in the grip of the current paradigm. Even the people who
+have the new ideas undervalue them initially. Which means that
+before they reach the stage of proposing them publicly, they've
+already subjected them to an excessively strict filter.
+[3]The wise response to such an idea is not to make statements, but
+to ask questions, because there's a real mystery here. Why has this
+smart and reasonable person proposed an idea that seems so wrong?
+Are they mistaken, or are you? One of you has to be. If you're the
+one who's mistaken, that would be good to know, because it means
+there's a hole in your model of the world. But even if they're
+mistaken, it should be interesting to learn why. A trap that an
+expert falls into is one you have to worry about too.This all seems pretty obvious. And yet there are clearly a lot of
+people who don't share my fear of dismissing new ideas. Why do they
+do it? Why risk looking like a jerk now and a fool later, instead
+of just reserving judgement?One reason they do it is envy. If you propose a radical new idea
+and it succeeds, your reputation (and perhaps also your wealth)
+will increase proportionally. Some people would be envious if that
+happened, and this potential envy propagates back into a conviction
+that you must be wrong.Another reason people dismiss new ideas is that it's an easy way
+to seem sophisticated. When a new idea first emerges, it usually
+seems pretty feeble. It's a mere hatchling. Received wisdom is a
+full-grown eagle by comparison. So it's easy to launch a devastating
+attack on a new idea, and anyone who does will seem clever to those
+who don't understand this asymmetry.This phenomenon is exacerbated by the difference between how those
+working on new ideas and those attacking them are rewarded. The
+rewards for working on new ideas are weighted by the value of the
+outcome. So it's worth working on something that only has a 10%
+chance of succeeding if it would make things more than 10x better.
+Whereas the rewards for attacking new ideas are roughly constant;
+such attacks seem roughly equally clever regardless of the target.People will also attack new ideas when they have a vested interest
+in the old ones. It's not surprising, for example, that some of
+Darwin's harshest critics were churchmen. People build whole careers
+on some ideas. When someone claims they're false or obsolete, they
+feel threatened.The lowest form of dismissal is mere factionalism: to automatically
+dismiss any idea associated with the opposing faction. The lowest
+form of all is to dismiss an idea because of who proposed it.But the main thing that leads reasonable people to dismiss new ideas
+is the same thing that holds people back from proposing them: the
+sheer pervasiveness of the current paradigm. It doesn't just affect
+the way we think; it is the Lego blocks we build thoughts out of.
+Popping out of the current paradigm is something only a few people
+can do. And even they usually have to suppress their intuitions at
+first, like a pilot flying through cloud who has to trust his
+instruments over his sense of balance.
+[4]Paradigms don't just define our present thinking. They also vacuum
+up the trail of crumbs that led to them, making our standards for
+new ideas impossibly high. The current paradigm seems so perfect
+to us, its offspring, that we imagine it must have been accepted
+completely as soon as it was discovered — that whatever the church thought
+of the heliocentric model, astronomers must have been convinced as
+soon as Copernicus proposed it. Far, in fact, from it. Copernicus
+published the heliocentric model in 1532, but it wasn't till the
+mid seventeenth century that the balance of scientific opinion
+shifted in its favor.
+[5]Few understand how feeble new ideas look when they first appear.
+So if you want to have new ideas yourself, one of the most valuable
+things you can do is to learn what they look like when they're born.
+Read about how new ideas happened, and try to get yourself into the
+heads of people at the time. How did things look to them, when the
+new idea was only half-finished, and even the person who had it was
+only half-convinced it was right?But you don't have to stop at history. You can observe big new ideas
+being born all around you right now. Just look for a reasonable
+domain expert proposing something that sounds wrong.If you're nice, as well as wise, you won't merely resist attacking
+such people, but encourage them. Having new ideas is a lonely
+business. Only those who've tried it know how lonely. These people
+need your help. And if you help them, you'll probably learn something
+in the process.Notes[1]
+This domain expertise could be in another field. Indeed,
+such crossovers tend to be particularly promising.[2]
+I'm not claiming this principle extends much beyond math,
+engineering, and the hard sciences. In politics, for example,
+crazy-sounding ideas generally are as bad as they sound. Though
+arguably this is not an exception, because the people who propose
+them are not in fact domain experts; politicians are domain experts
+in political tactics, like how to get elected and how to get
+legislation passed, but not in the world that policy acts upon.
+Perhaps no one could be.[3]
+This sense of "paradigm" was defined by Thomas Kuhn in his
+Structure of Scientific Revolutions, but I also recommend his
+Copernican Revolution, where you can see him at work developing the
+idea.[4]
+This is one reason people with a touch of Asperger's may have
+an advantage in discovering new ideas. They're always flying on
+instruments.[5]
+Hall, Rupert. From Galileo to Newton. Collins, 1963. This
+book is particularly good at getting into contemporaries' heads.Thanks to Trevor Blackwell, Patrick Collison, Suhail Doshi, Daniel
+Gackle, Jessica Livingston, and Robert Morris for reading drafts of this.

recipes/experimental/long-context/H2O/data/PaulGrahamEssays/nft.txt

@@ -0,0 +1,26 @@
+May 2021Noora Health, a nonprofit I've 
+supported for years, just launched
+a new NFT. It has a dramatic name, Save Thousands of Lives,
+because that's what the proceeds will do.Noora has been saving lives for 7 years. They run programs in
+hospitals in South Asia to teach new mothers how to take care of
+their babies once they get home. They're in 165 hospitals now. And
+because they know the numbers before and after they start at a new
+hospital, they can measure the impact they have. It is massive.
+For every 1000 live births, they save 9 babies.This number comes from a study
+of 133,733 families at 28 different
+hospitals that Noora conducted in collaboration with the Better
+Birth team at Ariadne Labs, a joint center for health systems
+innovation at Brigham and Women’s Hospital and Harvard T.H. Chan
+School of Public Health.Noora is so effective that even if you measure their costs in the
+most conservative way, by dividing their entire budget by the number
+of lives saved, the cost of saving a life is the lowest I've seen.
+$1,235.For this NFT, they're going to issue a public report tracking how
+this specific tranche of money is spent, and estimating the number
+of lives saved as a result.NFTs are a new territory, and this way of using them is especially
+new, but I'm excited about its potential. And I'm excited to see
+what happens with this particular auction, because unlike an NFT
+representing something that has already happened,
+this NFT gets better as the price gets higher.The reserve price was about $2.5 million, because that's what it
+takes for the name to be accurate: that's what it costs to save
+2000 lives. But the higher the price of this NFT goes, the more
+lives will be saved. What a sentence to be able to write.

recipes/experimental/long-context/H2O/data/PaulGrahamEssays/philosophy.txt

@@ -0,0 +1,429 @@
+September 2007In high school I decided I was going to study philosophy in college.
+I had several motives, some more honorable than others.  One of the
+less honorable was to shock people.  College was regarded as job
+training where I grew up, so studying philosophy seemed an impressively
+impractical thing to do.  Sort of like slashing holes in your clothes
+or putting a safety pin through your ear, which were other forms
+of impressive impracticality then just coming into fashion.But I had some more honest motives as well.  I thought studying
+philosophy would be a shortcut straight to wisdom.  All the people
+majoring in other things would just end up with a bunch of domain
+knowledge.  I would be learning what was really what.I'd tried to read a few philosophy books.  Not recent ones; you
+wouldn't find those in our high school library.  But I tried to
+read Plato and Aristotle.  I doubt I believed I understood them,
+but they sounded like they were talking about something important.
+I assumed I'd learn what in college.The summer before senior year I took some college classes.  I learned
+a lot in the calculus class, but I didn't learn much in Philosophy
+101.  And yet my plan to study philosophy remained intact.  It was
+my fault I hadn't learned anything.  I hadn't read the books we
+were assigned carefully enough.  I'd give Berkeley's Principles
+of Human Knowledge another shot in college.  Anything so admired
+and so difficult to read must have something in it, if one could
+only figure out what.Twenty-six years later, I still don't understand Berkeley.  I have
+a nice edition of his collected works.  Will I ever read it?  Seems
+unlikely.The difference between then and now is that now I understand why
+Berkeley is probably not worth trying to understand.  I think I see
+now what went wrong with philosophy, and how we might fix it.WordsI did end up being a philosophy major for most of college.  It
+didn't work out as I'd hoped.  I didn't learn any magical truths
+compared to which everything else was mere domain knowledge.  But
+I do at least know now why I didn't.  Philosophy doesn't really
+have a subject matter in the way math or history or most other
+university subjects do.  There is no core of knowledge one must
+master.  The closest you come to that is a knowledge of what various
+individual philosophers have said about different topics over the
+years.  Few were sufficiently correct that people have forgotten
+who discovered what they discovered.Formal logic has some subject matter. I took several classes in
+logic.  I don't know if I learned anything from them.
+[1]
+It does seem to me very important to be able to flip ideas around in
+one's head: to see when two ideas don't fully cover the space of
+possibilities, or when one idea is the same as another but with a
+couple things changed.  But did studying logic teach me the importance
+of thinking this way, or make me any better at it?  I don't know.There are things I know I learned from studying philosophy.  The
+most dramatic I learned immediately, in the first semester of
+freshman year, in a class taught by Sydney Shoemaker.  I learned
+that I don't exist.  I am (and you are) a collection of cells that
+lurches around driven by various forces, and calls itself I.  But
+there's no central, indivisible thing that your identity goes with.
+You could conceivably lose half your brain and live.  Which means
+your brain could conceivably be split into two halves and each
+transplanted into different bodies.  Imagine waking up after such
+an operation.  You have to imagine being two people.The real lesson here is that the concepts we use in everyday life
+are fuzzy, and break down if pushed too hard.  Even a concept as
+dear to us as I.  It took me a while to grasp this, but when I
+did it was fairly sudden, like someone in the nineteenth century
+grasping evolution and realizing the story of creation they'd been
+told as a child was all wrong. 
+[2]
+Outside of math there's a limit
+to how far you can push words; in fact, it would not be a bad
+definition of math to call it the study of terms that have precise
+meanings.  Everyday words are inherently imprecise.  They work well
+enough in everyday life that you don't notice.  Words seem to work,
+just as Newtonian physics seems to.  But you can always make them
+break if you push them far enough.I would say that this has been, unfortunately for philosophy, the
+central fact of philosophy.  Most philosophical debates are not
+merely afflicted by but driven by confusions over words.  Do we
+have free will?  Depends what you mean by "free." Do abstract ideas
+exist?  Depends what you mean by "exist."Wittgenstein is popularly credited with the idea that most philosophical
+controversies are due to confusions over language.  I'm not sure
+how much credit to give him.  I suspect a lot of people realized
+this, but reacted simply by not studying philosophy, rather than
+becoming philosophy professors.How did things get this way?  Can something people have spent
+thousands of years studying really be a waste of time?  Those are
+interesting questions.  In fact, some of the most interesting
+questions you can ask about philosophy.  The most valuable way to
+approach the current philosophical tradition may be neither to get
+lost in pointless speculations like Berkeley, nor to shut them down
+like Wittgenstein, but to study it as an example of reason gone
+wrong.HistoryWestern philosophy really begins with Socrates, Plato, and Aristotle.
+What we know of their predecessors comes from fragments and references
+in later works; their doctrines could be described as speculative
+cosmology that occasionally strays into analysis.  Presumably they
+were driven by whatever makes people in every other society invent
+cosmologies.
+[3]With Socrates, Plato, and particularly Aristotle, this tradition
+turned a corner.  There started to be a lot more analysis.  I suspect
+Plato and Aristotle were encouraged in this by progress in math.
+Mathematicians had by then shown that you could figure things out
+in a much more conclusive way than by making up fine sounding stories
+about them.  
+[4]People talk so much about abstractions now that we don't realize
+what a leap it must have been when they first started to.  It was
+presumably many thousands of years between when people first started
+describing things as hot or cold and when someone asked "what is
+heat?"  No doubt it was a very gradual process.  We don't know if
+Plato or Aristotle were the first to ask any of the questions they
+did.  But their works are the oldest we have that do this on a large
+scale, and there is a freshness (not to say naivete) about them
+that suggests some of the questions they asked were new to them,
+at least.Aristotle in particular reminds me of the phenomenon that happens
+when people discover something new, and are so excited by it that
+they race through a huge percentage of the newly discovered territory
+in one lifetime.  If so, that's evidence of how new this kind of
+thinking was. 
+[5]This is all to explain how Plato and Aristotle can be very impressive
+and yet naive and mistaken.  It was impressive even to ask the
+questions they did.  That doesn't mean they always came up with
+good answers.  It's not considered insulting to say that ancient
+Greek mathematicians were naive in some respects, or at least lacked
+some concepts that would have made their lives easier.  So I hope
+people will not be too offended if I propose that ancient philosophers
+were similarly naive.  In particular, they don't seem to have fully
+grasped what I earlier called the central fact of philosophy: that
+words break if you push them too far."Much to the surprise of the builders of the first digital computers,"
+Rod Brooks wrote, "programs written for them usually did not work."
+[6]
+Something similar happened when people first started trying
+to talk about abstractions.  Much to their surprise, they didn't
+arrive at answers they agreed upon.  In fact, they rarely seemed
+to arrive at answers at all.They were in effect arguing about artifacts induced by sampling at
+too low a resolution.The proof of how useless some of their answers turned out to be is
+how little effect they have.  No one after reading Aristotle's
+Metaphysics does anything differently as a result.
+[7]Surely I'm not claiming that ideas have to have practical applications
+to be interesting?  No, they may not have to.  Hardy's boast that
+number theory had no use whatsoever wouldn't disqualify it.  But
+he turned out to be mistaken.  In fact, it's suspiciously hard to
+find a field of math that truly has no practical use.  And Aristotle's
+explanation of the ultimate goal of philosophy in Book A of the
+Metaphysics implies that philosophy should be useful too.Theoretical KnowledgeAristotle's goal was to find the most general of general principles.
+The examples he gives are convincing: an ordinary worker builds
+things a certain way out of habit; a master craftsman can do more
+because he grasps the underlying principles.  The trend is clear:
+the more general the knowledge, the more admirable it is.  But then
+he makes a mistake—possibly the most important mistake in the
+history of philosophy.  He has noticed that theoretical knowledge
+is often acquired for its own sake, out of curiosity, rather than
+for any practical need.  So he proposes there are two kinds of
+theoretical knowledge: some that's useful in practical matters and
+some that isn't.  Since people interested in the latter are interested
+in it for its own sake, it must be more noble.  So he sets as his
+goal in the Metaphysics the exploration of knowledge that has no
+practical use.  Which means no alarms go off when he takes on grand
+but vaguely understood questions and ends up getting lost in a sea
+of words.His mistake was to confuse motive and result.  Certainly, people
+who want a deep understanding of something are often driven by
+curiosity rather than any practical need.  But that doesn't mean
+what they end up learning is useless.  It's very valuable in practice
+to have a deep understanding of what you're doing; even if you're
+never called on to solve advanced problems, you can see shortcuts
+in the solution of simple ones, and your knowledge won't break down
+in edge cases, as it would if you were relying on formulas you
+didn't understand.  Knowledge is power.  That's what makes theoretical
+knowledge prestigious.  It's also what causes smart people to be
+curious about certain things and not others; our DNA is not so
+disinterested as we might think.So while ideas don't have to have immediate practical applications
+to be interesting, the kinds of things we find interesting will
+surprisingly often turn out to have practical applications.The reason Aristotle didn't get anywhere in the Metaphysics was
+partly that he set off with contradictory aims: to explore the most
+abstract ideas, guided by the assumption that they were useless.
+He was like an explorer looking for a territory to the north of
+him, starting with the assumption that it was located to the south.And since his work became the map used by generations of future
+explorers, he sent them off in the wrong direction as well. 
+[8]
+Perhaps worst of all, he protected them from both the criticism of
+outsiders and the promptings of their own inner compass by establishing
+the principle that the most noble sort of theoretical knowledge had
+to be useless.The Metaphysics is mostly a failed experiment.  A few ideas from
+it turned out to be worth keeping; the bulk of it has had no effect
+at all.  The Metaphysics is among the least read of all famous
+books.  It's not hard to understand the way Newton's Principia
+is, but the way a garbled message is.Arguably it's an interesting failed experiment.  But unfortunately
+that was not the conclusion Aristotle's successors derived from
+works like the Metaphysics. 
+[9]
+Soon after, the western world
+fell on intellectual hard times.  Instead of version 1s to be
+superseded, the works of Plato and Aristotle became revered texts
+to be mastered and discussed.  And so things remained for a shockingly
+long time.  It was not till around 1600 (in Europe, where the center
+of gravity had shifted by then) that one found people confident
+enough to treat Aristotle's work as a catalog of mistakes.  And
+even then they rarely said so outright.If it seems surprising that the gap was so long, consider how little
+progress there was in math between Hellenistic times and the
+Renaissance.In the intervening years an unfortunate idea took hold:  that it
+was not only acceptable to produce works like the Metaphysics,
+but that it was a particularly prestigious line of work, done by a
+class of people called philosophers.  No one thought to go back and
+debug Aristotle's motivating argument.  And so instead of correcting
+the problem Aristotle discovered by falling into it—that you can
+easily get lost if you talk too loosely about very abstract ideas—they 
+continued to fall into it.The SingularityCuriously, however, the works they produced continued to attract
+new readers.  Traditional philosophy occupies a kind of singularity
+in this respect.  If you write in an unclear way about big ideas,
+you produce something that seems tantalizingly attractive to
+inexperienced but intellectually ambitious students.  Till one knows
+better, it's hard to distinguish something that's hard to understand
+because the writer was unclear in his own mind from something like
+a mathematical proof that's hard to understand because the ideas
+it represents are hard to understand.  To someone who hasn't learned
+the difference, traditional philosophy seems extremely attractive:
+as hard (and therefore impressive) as math, yet broader in scope.
+That was what lured me in as a high school student.This singularity is even more singular in having its own defense
+built in.  When things are hard to understand, people who suspect
+they're nonsense generally keep quiet.  There's no way to prove a
+text is meaningless.  The closest you can get is to show that the
+official judges of some class of texts can't distinguish them from
+placebos. 
+[10]And so instead of denouncing philosophy, most people who suspected
+it was a waste of time just studied other things.  That alone is
+fairly damning evidence, considering philosophy's claims.  It's
+supposed to be about the ultimate truths. Surely all smart people
+would be interested in it, if it delivered on that promise.Because philosophy's flaws turned away the sort of people who might
+have corrected them, they tended to be self-perpetuating.  Bertrand
+Russell wrote in a letter in 1912:
+
+  Hitherto the people attracted to philosophy have been mostly those
+  who loved the big generalizations, which were all wrong, so that
+  few people with exact minds have taken up the subject.
+[11]
+
+His response was to launch Wittgenstein at it, with dramatic results.I think Wittgenstein deserves to be famous not for the discovery
+that most previous philosophy was a waste of time, which judging
+from the circumstantial evidence must have been made by every smart
+person who studied a little philosophy and declined to pursue it
+further, but for how he acted in response.
+[12]
+Instead of quietly
+switching to another field, he made a fuss, from inside.  He was
+Gorbachev.The field of philosophy is still shaken from the fright Wittgenstein
+gave it. 
+[13]
+Later in life he spent a lot of time talking about
+how words worked.  Since that seems to be allowed, that's what a
+lot of philosophers do now.  Meanwhile, sensing a vacuum in the
+metaphysical speculation department, the people who used to do
+literary criticism have been edging Kantward, under new names like
+"literary theory," "critical theory," and when they're feeling
+ambitious, plain "theory."  The writing is the familiar word salad:
+
+  Gender is not like some of the other grammatical modes which
+  express precisely a mode of conception without any reality that
+  corresponds to the conceptual mode, and consequently do not express
+  precisely something in reality by which the intellect could be
+  moved to conceive a thing the way it does, even where that motive
+  is not something in the thing as such.
+  [14]
+
+The singularity I've described is not going away.  There's a market
+for writing that sounds impressive and can't be disproven. There
+will always be both supply and demand.  So if one group abandons
+this territory, there will always be others ready to occupy it.A ProposalWe may be able to do better.  Here's an intriguing possibility.
+Perhaps we should do what Aristotle meant to do, instead of what
+he did.  The goal he announces in the Metaphysics seems one worth
+pursuing: to discover the most general truths.  That sounds good.
+But instead of trying to discover them because they're useless,
+let's try to discover them because they're useful.I propose we try again, but that we use that heretofore despised
+criterion, applicability, as a guide to keep us from wondering
+off into a swamp of abstractions.  Instead of trying to answer the
+question:
+
+  What are the most general truths?
+
+let's try to answer the question
+
+  Of all the useful things we can say, which are the most general?
+
+The test of utility I propose is whether we cause people who read
+what we've written to do anything differently afterward.  Knowing
+we have to give definite (if implicit) advice will keep us from
+straying beyond the resolution of the words we're using.The goal is the same as Aristotle's; we just approach it from a
+different direction.As an example of a useful, general idea, consider that of the
+controlled experiment.  There's an idea that has turned out to be
+widely applicable.  Some might say it's part of science, but it's
+not part of any specific science; it's literally meta-physics (in
+our sense of "meta").   The idea of evolution is another. It turns
+out to have quite broad applications—for example, in genetic
+algorithms and even product design.  Frankfurt's distinction between
+lying and bullshitting seems a promising recent example.
+[15]These seem to me what philosophy should look like: quite general
+observations that would cause someone who understood them to do
+something differently.Such observations will necessarily be about things that are imprecisely
+defined.  Once you start using words with precise meanings, you're
+doing math.  So starting from utility won't entirely solve the
+problem I described above—it won't flush out the metaphysical
+singularity.  But it should help.  It gives people with good
+intentions a new roadmap into abstraction.  And they may thereby
+produce things that make the writing of the people with bad intentions
+look bad by comparison.One drawback of this approach is that it won't produce the sort of
+writing that gets you tenure.  And not just because it's not currently
+the fashion.  In order to get tenure in any field you must not
+arrive at conclusions that members of tenure committees can disagree
+with.  In practice there are two kinds of solutions to this problem.
+In math and the sciences, you can prove what you're saying, or at
+any rate adjust your conclusions so you're not claiming anything
+false ("6 of 8 subjects had lower blood pressure after the treatment").
+In the humanities you can either avoid drawing any definite conclusions
+(e.g. conclude that an issue is a complex one), or draw conclusions
+so narrow that no one cares enough to disagree with you.The kind of philosophy I'm advocating won't be able to take either
+of these routes.  At best you'll be able to achieve the essayist's
+standard of proof, not the mathematician's or the experimentalist's.
+And yet you won't be able to meet the usefulness test without
+implying definite and fairly broadly applicable conclusions.  Worse
+still, the usefulness test will tend to produce results that annoy
+people: there's no use in telling people things they already believe,
+and people are often upset to be told things they don't.Here's the exciting thing, though.  Anyone can do this.  Getting
+to general plus useful by starting with useful and cranking up the
+generality may be unsuitable for junior professors trying to get
+tenure, but it's better for everyone else, including professors who
+already have it.  This side of the mountain is a nice gradual slope.
+You can start by writing things that are useful but very specific,
+and then gradually make them more general.  Joe's has good burritos.
+What makes a good burrito?  What makes good food?  What makes
+anything good?  You can take as long as you want.  You don't have
+to get all the way to the top of the mountain.  You don't have to
+tell anyone you're doing philosophy.If it seems like a daunting task to do philosophy, here's an
+encouraging thought.  The field is a lot younger than it seems.
+Though the first philosophers in the western tradition lived about
+2500 years ago, it would be misleading to say the field is 2500
+years old, because for most of that time the leading practitioners
+weren't doing much more than writing commentaries on Plato or
+Aristotle while watching over their shoulders for the next invading
+army.  In the times when they weren't, philosophy was hopelessly
+intermingled with religion.  It didn't shake itself free till a
+couple hundred years ago, and even then was afflicted by the
+structural problems I've described above.  If I say this, some will
+say it's a ridiculously overbroad and uncharitable generalization,
+and others will say it's old news, but here goes: judging from their
+works, most philosophers up to the present have been wasting their
+time.  So in a sense the field is still at the first step. 
+[16]That sounds a preposterous claim to make.  It won't seem so
+preposterous in 10,000 years.  Civilization always seems old, because
+it's always the oldest it's ever been.  The only way to say whether
+something is really old or not is by looking at structural evidence,
+and structurally philosophy is young; it's still reeling from the
+unexpected breakdown of words.Philosophy is as young now as math was in 1500.  There is a lot
+more to discover.Notes
+[1]
+In practice formal logic is not much use, because despite
+some progress in the last 150 years we're still only able to formalize
+a small percentage of statements.  We may never do that much better,
+for the same reason 1980s-style "knowledge representation" could
+never have worked; many statements may have no representation more
+concise than a huge, analog brain state.[2]
+It was harder for Darwin's contemporaries to grasp this than
+we can easily imagine.  The story of creation in the Bible is not
+just a Judeo-Christian concept; it's roughly what everyone must
+have believed since before people were people.  The hard part of
+grasping evolution was to realize that species weren't, as they
+seem to be, unchanging, but had instead evolved from different,
+simpler organisms over unimaginably long periods of time.Now we don't have to make that leap.  No one in an industrialized
+country encounters the idea of evolution for the first time as an
+adult.  Everyone's taught about it as a child, either as truth or
+heresy.[3]
+Greek philosophers before Plato wrote in verse.  This must
+have affected what they said.  If you try to write about the nature
+of the world in verse, it inevitably turns into incantation.  Prose
+lets you be more precise, and more tentative.[4]
+Philosophy is like math's
+ne'er-do-well brother.  It was born when Plato and Aristotle looked
+at the works of their predecessors and said in effect "why can't
+you be more like your brother?"  Russell was still saying the same
+thing 2300 years later.Math is the precise half of the most abstract ideas, and philosophy
+the imprecise half.  It's probably inevitable that philosophy will
+suffer by comparison, because there's no lower bound to its precision.
+Bad math is merely boring, whereas bad philosophy is nonsense.  And
+yet there are some good ideas in the imprecise half.[5]
+Aristotle's best work was in logic and zoology, both of which
+he can  be said to have invented.  But the most dramatic departure
+from his predecessors was a new, much more analytical style of
+thinking.  He was arguably the first scientist.[6]
+Brooks, Rodney, Programming in Common Lisp, Wiley, 1985, p.
+94.[7]
+Some would say we depend on Aristotle more than we realize,
+because his ideas were one of the ingredients in our common culture.
+Certainly a lot of the words we use have a connection with Aristotle,
+but it seems a bit much to suggest that we wouldn't have the concept
+of the essence of something or the distinction between matter and
+form if Aristotle hadn't written about them.One way to see how much we really depend on Aristotle would be to
+diff European culture with Chinese: what ideas did European culture
+have in 1800 that Chinese culture didn't, in virtue of Aristotle's
+contribution?[8]
+The meaning of the word "philosophy" has changed over time.
+In ancient times it covered a broad range of topics, comparable in
+scope to our "scholarship" (though without the methodological
+implications).  Even as late as Newton's time it included what we
+now call "science."  But core of the subject today is still what
+seemed to Aristotle the core: the attempt to discover the most
+general truths.Aristotle didn't call this "metaphysics."  That name got assigned
+to it because the books we now call the Metaphysics came after
+(meta = after) the Physics in the standard edition of Aristotle's
+works compiled by Andronicus of Rhodes three centuries later.  What
+we call "metaphysics" Aristotle called "first philosophy."[9]
+Some of Aristotle's immediate successors may have realized
+this, but it's hard to say because most of their works are lost.[10]
+Sokal, Alan, "Transgressing the Boundaries: Toward a Transformative
+Hermeneutics of Quantum Gravity," Social Text 46/47, pp. 217-252.Abstract-sounding nonsense seems to be most attractive when it's
+aligned with some axe the audience already has to grind.  If this
+is so we should find it's most popular with groups that are (or
+feel) weak.  The powerful don't need its reassurance.[11]
+Letter to Ottoline Morrell, December 1912.  Quoted in:Monk, Ray, Ludwig Wittgenstein: The Duty of Genius, Penguin, 1991,
+p. 75.[12]
+A preliminary result, that all metaphysics between Aristotle
+and 1783 had been a waste of time, is due to I. Kant.[13]
+Wittgenstein asserted a sort of mastery to which the inhabitants
+of early 20th century Cambridge seem to have been peculiarly
+vulnerable—perhaps partly because so many had been raised religious
+and then stopped believing, so had a vacant space in their heads
+for someone to tell them what to do (others chose Marx or Cardinal
+Newman), and partly because a quiet, earnest place like Cambridge
+in that era had no natural immunity to messianic figures, just as
+European politics then had no natural immunity to dictators.[14]
+This is actually from the Ordinatio of Duns Scotus (ca.
+1300), with "number" replaced by "gender."  Plus ca change.Wolter, Allan (trans), Duns Scotus: Philosophical Writings, Nelson,
+1963, p. 92.[15]
+Frankfurt, Harry, On Bullshit,  Princeton University Press,
+2005.[16]
+Some introductions to philosophy now take the line that
+philosophy is worth studying as a process rather than for any
+particular truths you'll learn.  The philosophers whose works they
+cover would be rolling in their graves at that.  They hoped they
+were doing more than serving as examples of how to argue: they hoped
+they were getting results.  Most were wrong, but it doesn't seem
+an impossible hope.This argument seems to me like someone in 1500 looking at the lack
+of results achieved by alchemy and saying its value was as a process.
+No, they were going about it wrong.  It turns out it is possible
+to transmute lead into gold (though not economically at current
+energy prices), but the route to that knowledge was to
+backtrack and try another approach.Thanks to Trevor Blackwell, Paul Buchheit, Jessica Livingston, 
+Robert Morris, Mark Nitzberg, and Peter Norvig for reading drafts of this.

recipes/experimental/long-context/H2O/data/PaulGrahamEssays/popular.txt

@@ -0,0 +1,602 @@
+May 2001(This article was written as a kind of business plan for a
+new language.
+So it is missing (because it takes for granted) the most important
+feature of a good programming language: very powerful abstractions.)A friend of mine once told an eminent operating systems
+expert that he wanted to design a really good
+programming language.  The expert told him that it would be a
+waste of time, that programming languages don't become popular
+or unpopular based on their merits, and so no matter how
+good his language was, no one would use it.  At least, that
+was what had happened to the language he had designed.What does make a language popular?  Do popular
+languages deserve their popularity?  Is it worth trying to
+define a good programming language?  How would you do it?I think the answers to these questions can be found by looking 
+at hackers, and learning what they want.  Programming
+languages are for hackers, and a programming language
+is good as a programming language (rather than, say, an
+exercise in denotational semantics or compiler design)
+if and only if hackers like it.1 The Mechanics of PopularityIt's true, certainly, that most people don't choose programming
+languages simply based on their merits.  Most programmers are told
+what language to use by someone else.  And yet I think the effect
+of such external factors on the popularity of programming languages
+is not as great as it's sometimes thought to be. I think a bigger
+problem is that a hacker's idea of a good programming language is
+not the same as most language designers'.Between the two, the hacker's opinion is the one that matters.
+Programming languages are not theorems. They're tools, designed
+for people, and they have to be designed to suit human strengths
+and weaknesses as much as shoes have to be designed for human feet.
+If a shoe pinches when you put it on, it's a bad shoe, however
+elegant it may be as a piece of sculpture.It may be that the majority of programmers can't tell a good language
+from a bad one. But that's no different with any other tool. It
+doesn't mean that it's a waste of time to try designing a good
+language. Expert hackers 
+can tell a good language when they see
+one, and they'll use it. Expert hackers are a tiny minority,
+admittedly, but that tiny minority write all the good software,
+and their influence is such that the rest of the programmers will
+tend to use whatever language they use. Often, indeed, it is not
+merely influence but command: often the expert hackers are the very
+people who, as their bosses or faculty advisors, tell the other
+programmers what language to use.The opinion of expert hackers is not the only force that determines
+the relative popularity of programming languages — legacy software
+(Cobol) and hype (Ada, Java) also play a role — but I think it is
+the most powerful force over the long term. Given an initial critical
+mass and enough time, a programming language probably becomes about
+as popular as it deserves to be. And popularity further separates
+good languages from bad ones, because feedback from real live users
+always leads to improvements. Look at how much any popular language
+has changed during its life. Perl and Fortran are extreme cases,
+but even Lisp has changed a lot. Lisp 1.5 didn't have macros, for
+example; these evolved later, after hackers at MIT had spent a
+couple years using Lisp to write real programs. [1]So whether or not a language has to be good to be popular, I think
+a language has to be popular to be good. And it has to stay popular
+to stay good. The state of the art in programming languages doesn't
+stand still. And yet the Lisps we have today are still pretty much
+what they had at MIT in the mid-1980s, because that's the last time
+Lisp had a sufficiently large and demanding user base.Of course, hackers have to know about a language before they can
+use it. How are they to hear? From other hackers. But there has to
+be some initial group of hackers using the language for others even
+to hear about it. I wonder how large this group has to be; how many
+users make a critical mass? Off the top of my head, I'd say twenty.
+If a language had twenty separate users, meaning twenty users who
+decided on their own to use it, I'd consider it to be real.Getting there can't be easy. I would not be surprised if it is
+harder to get from zero to twenty than from twenty to a thousand.
+The best way to get those initial twenty users is probably to use
+a trojan horse: to give people an application they want, which
+happens to be written in the new language.2 External FactorsLet's start by acknowledging one external factor that does affect
+the popularity of a programming language. To become popular, a
+programming language has to be the scripting language of a popular
+system. Fortran and Cobol were the scripting languages of early
+IBM mainframes. C was the scripting language of Unix, and so, later,
+was Perl. Tcl is the scripting language of Tk. Java and Javascript
+are intended to be the scripting languages of web browsers.Lisp is not a massively popular language because it is not the
+scripting language of a massively popular system. What popularity
+it retains dates back to the 1960s and 1970s, when it was the
+scripting language of MIT. A lot of the great programmers of the
+day were associated with MIT at some point. And in the early 1970s,
+before C, MIT's dialect of Lisp, called MacLisp, was one of the
+only programming languages a serious hacker would want to use.Today Lisp is the scripting language of two moderately popular
+systems, Emacs and Autocad, and for that reason I suspect that most
+of the Lisp programming done today is done in Emacs Lisp or AutoLisp.Programming languages don't exist in isolation. To hack is a
+transitive verb — hackers are usually hacking something — and in
+practice languages are judged relative to whatever they're used to
+hack. So if you want to design a popular language, you either have
+to supply more than a language, or you have to design your language
+to replace the scripting language of some existing system.Common Lisp is unpopular partly because it's an orphan. It did
+originally come with a system to hack: the Lisp Machine. But Lisp
+Machines (along with parallel computers) were steamrollered by the
+increasing power of general purpose processors in the 1980s. Common
+Lisp might have remained popular if it had been a good scripting
+language for Unix. It is, alas, an atrociously bad one.One way to describe this situation is to say that a language isn't
+judged on its own merits. Another view is that a programming language
+really isn't a programming language unless it's also the scripting
+language of something. This only seems unfair if it comes as a
+surprise. I think it's no more unfair than expecting a programming
+language to have, say, an implementation. It's just part of what
+a programming language is.A programming language does need a good implementation, of course,
+and this must be free. Companies will pay for software, but individual
+hackers won't, and it's the hackers you need to attract.A language also needs to have a book about it. The book should be
+thin, well-written, and full of good examples. K&R is the ideal
+here. At the moment I'd almost say that a language has to have a
+book published by O'Reilly. That's becoming the test of mattering
+to hackers.There should be online documentation as well. In fact, the book
+can start as online documentation. But I don't think that physical
+books are outmoded yet. Their format is convenient, and the de
+facto censorship imposed by publishers is a useful if imperfect
+filter. Bookstores are one of the most important places for learning
+about new languages.3 BrevityGiven that you can supply the three things any language needs — a
+free implementation, a book, and something to hack — how do you
+make a language that hackers will like?One thing hackers like is brevity. Hackers are lazy, in the same
+way that mathematicians and modernist architects are lazy: they
+hate anything extraneous. It would not be far from the truth to
+say that a hacker about to write a program decides what language
+to use, at least subconsciously, based on the total number of
+characters he'll have to type. If this isn't precisely how hackers
+think, a language designer would do well to act as if it were.It is a mistake to try to baby the user with long-winded expressions
+that are meant to resemble English. Cobol is notorious for this
+flaw. A hacker would consider being asked to writeadd x to y giving zinstead ofz = x+yas something between an insult to his intelligence and a sin against
+God.It has sometimes been said that Lisp should use first and rest
+instead of car and cdr, because it would make programs easier to
+read. Maybe for the first couple hours. But a hacker can learn
+quickly enough that car means the first element of a list and cdr
+means the rest. Using first and rest means 50% more typing. And
+they are also different lengths, meaning that the arguments won't
+line up when they're called, as car and cdr often are, in successive
+lines. I've found that it matters a lot how code lines up on the
+page. I can barely read Lisp code when it is set in a variable-width
+font, and friends say this is true for other languages too.Brevity is one place where strongly typed languages lose. All other
+things being equal, no one wants to begin a program with a bunch
+of declarations. Anything that can be implicit, should be.The individual tokens should be short as well. Perl and Common Lisp
+occupy opposite poles on this question. Perl programs can be almost
+cryptically dense, while the names of built-in Common Lisp operators
+are comically long. The designers of Common Lisp probably expected
+users to have text editors that would type these long names for
+them. But the cost of a long name is not just the cost of typing
+it. There is also the cost of reading it, and the cost of the space
+it takes up on your screen.4 HackabilityThere is one thing more important than brevity to a hacker: being
+able to do what you want. In the history of programming languages
+a surprising amount of effort has gone into preventing programmers
+from doing things considered to be improper. This is a dangerously
+presumptuous plan. How can the language designer know what the
+programmer is going to need to do? I think language designers would
+do better to consider their target user to be a genius who will
+need to do things they never anticipated, rather than a bumbler
+who needs to be protected from himself. The bumbler will shoot
+himself in the foot anyway. You may save him from referring to
+variables in another package, but you can't save him from writing
+a badly designed program to solve the wrong problem, and taking
+forever to do it.Good programmers often want to do dangerous and unsavory things.
+By unsavory I mean things that go behind whatever semantic facade
+the language is trying to present: getting hold of the internal
+representation of some high-level abstraction, for example. Hackers
+like to hack, and hacking means getting inside things and second
+guessing the original designer.Let yourself be second guessed. When you make any tool, people use
+it in ways you didn't intend, and this is especially true of a
+highly articulated tool like a programming language. Many a hacker
+will want to tweak your semantic model in a way that you never
+imagined. I say, let them; give the programmer access to as much
+internal stuff as you can without endangering runtime systems like
+the garbage collector.In Common Lisp I have often wanted to iterate through the fields
+of a struct — to comb out references to a deleted object, for example,
+or find fields that are uninitialized. I know the structs are just
+vectors underneath. And yet I can't write a general purpose function
+that I can call on any struct. I can only access the fields by
+name, because that's what a struct is supposed to mean.A hacker may only want to subvert the intended model of things once
+or twice in a big program. But what a difference it makes to be
+able to. And it may be more than a question of just solving a
+problem. There is a kind of pleasure here too. Hackers share the
+surgeon's secret pleasure in poking about in gross innards, the
+teenager's secret pleasure in popping zits. [2] For boys, at least,
+certain kinds of horrors are fascinating. Maxim magazine publishes
+an annual volume of photographs, containing a mix of pin-ups and
+grisly accidents. They know their audience.Historically, Lisp has been good at letting hackers have their way.
+The political correctness of Common Lisp is an aberration. Early
+Lisps let you get your hands on everything. A good deal of that
+spirit is, fortunately, preserved in macros. What a wonderful thing,
+to be able to make arbitrary transformations on the source code.Classic macros are a real hacker's tool — simple, powerful, and
+dangerous. It's so easy to understand what they do: you call a
+function on the macro's arguments, and whatever it returns gets
+inserted in place of the macro call. Hygienic macros embody the
+opposite principle. They try to protect you from understanding what
+they're doing. I have never heard hygienic macros explained in one
+sentence. And they are a classic example of the dangers of deciding
+what programmers are allowed to want. Hygienic macros are intended
+to protect me from variable capture, among other things, but variable
+capture is exactly what I want in some macros.A really good language should be both clean and dirty: cleanly
+designed, with a small core of well understood and highly orthogonal
+operators, but dirty in the sense that it lets hackers have their
+way with it. C is like this. So were the early Lisps. A real hacker's
+language will always have a slightly raffish character.A good programming language should have features that make the kind
+of people who use the phrase "software engineering" shake their
+heads disapprovingly. At the other end of the continuum are languages
+like Ada and Pascal, models of propriety that are good for teaching
+and not much else.5 Throwaway ProgramsTo be attractive to hackers, a language must be good for writing
+the kinds of programs they want to write. And that means, perhaps
+surprisingly, that it has to be good for writing throwaway programs.A throwaway program is a program you write quickly for some limited
+task: a program to automate some system administration task, or
+generate test data for a simulation, or convert data from one format
+to another. The surprising thing about throwaway programs is that,
+like the "temporary" buildings built at so many American universities
+during World War II, they often don't get thrown away. Many evolve
+into real programs, with real features and real users.I have a hunch that the best big programs begin life this way,
+rather than being designed big from the start, like the Hoover Dam.
+It's terrifying to build something big from scratch. When people
+take on a project that's too big, they become overwhelmed. The
+project either gets bogged down, or the result is sterile and
+wooden: a shopping mall rather than a real downtown, Brasilia rather
+than Rome, Ada rather than C.Another way to get a big program is to start with a throwaway
+program and keep improving it. This approach is less daunting, and
+the design of the program benefits from evolution. I think, if one
+looked, that this would turn out to be the way most big programs
+were developed. And those that did evolve this way are probably
+still written in whatever language they were first written in,
+because it's rare for a program to be ported, except for political
+reasons. And so, paradoxically, if you want to make a language that
+is used for big systems, you have to make it good for writing
+throwaway programs, because that's where big systems come from.Perl is a striking example of this idea. It was not only designed
+for writing throwaway programs, but was pretty much a throwaway
+program itself. Perl began life as a collection of utilities for
+generating reports, and only evolved into a programming language
+as the throwaway programs people wrote in it grew larger. It was
+not until Perl 5 (if then) that the language was suitable for
+writing serious programs, and yet it was already massively popular.What makes a language good for throwaway programs? To start with,
+it must be readily available. A throwaway program is something that
+you expect to write in an hour. So the language probably must
+already be installed on the computer you're using. It can't be
+something you have to install before you use it. It has to be there.
+C was there because it came with the operating system. Perl was
+there because it was originally a tool for system administrators,
+and yours had already installed it.Being available means more than being installed, though. An
+interactive language, with a command-line interface, is more
+available than one that you have to compile and run separately. A
+popular programming language should be interactive, and start up
+fast.Another thing you want in a throwaway program is brevity. Brevity
+is always attractive to hackers, and never more so than in a program
+they expect to turn out in an hour.6 LibrariesOf course the ultimate in brevity is to have the program already
+written for you, and merely to call it. And this brings us to what
+I think will be an increasingly important feature of programming
+languages: library functions. Perl wins because it has large
+libraries for manipulating strings. This class of library functions
+are especially important for throwaway programs, which are often
+originally written for converting or extracting data.  Many Perl
+programs probably begin as just a couple library calls stuck
+together.I think a lot of the advances that happen in programming languages
+in the next fifty years will have to do with library functions. I
+think future programming languages will have libraries that are as
+carefully designed as the core language. Programming language design
+will not be about whether to make your language strongly or weakly
+typed, or object oriented, or functional, or whatever, but about
+how to design great libraries. The kind of language designers who
+like to think about how to design type systems may shudder at this.
+It's almost like writing applications! Too bad. Languages are for
+programmers, and libraries are what programmers need.It's hard to design good libraries. It's not simply a matter of
+writing a lot of code. Once the libraries get too big, it can
+sometimes take longer to find the function you need than to write
+the code yourself. Libraries need to be designed using a small set
+of orthogonal operators, just like the core language. It ought to
+be possible for the programmer to guess what library call will do
+what he needs.Libraries are one place Common Lisp falls short. There are only
+rudimentary libraries for manipulating strings, and almost none
+for talking to the operating system. For historical reasons, Common
+Lisp tries to pretend that the OS doesn't exist. And because you
+can't talk to the OS, you're unlikely to be able to write a serious
+program using only the built-in operators in Common Lisp. You have
+to use some implementation-specific hacks as well, and in practice
+these tend not to give you everything you want. Hackers would think
+a lot more highly of Lisp if Common Lisp had powerful string
+libraries and good OS support.7 SyntaxCould a language with Lisp's syntax, or more precisely, lack of
+syntax, ever become popular? I don't know the answer to this
+question. I do think that syntax is not the main reason Lisp isn't
+currently popular. Common Lisp has worse problems than unfamiliar
+syntax. I know several programmers who are comfortable with prefix
+syntax and yet use Perl by default, because it has powerful string
+libraries and can talk to the os.There are two possible problems with prefix notation: that it is
+unfamiliar to programmers, and that it is not dense enough. The
+conventional wisdom in the Lisp world is that the first problem is
+the real one. I'm not so sure. Yes, prefix notation makes ordinary
+programmers panic. But I don't think ordinary programmers' opinions
+matter. Languages become popular or unpopular based on what expert
+hackers think of them, and I think expert hackers might be able to
+deal with prefix notation. Perl syntax can be pretty incomprehensible,
+but that has not stood in the way of Perl's popularity. If anything
+it may have helped foster a Perl cult.A more serious problem is the diffuseness of prefix notation. For
+expert hackers, that really is a problem. No one wants to write
+(aref a x y) when they could write a[x,y].In this particular case there is a way to finesse our way out of
+the problem. If we treat data structures as if they were functions
+on indexes, we could write (a x y) instead, which is even shorter
+than the Perl form. Similar tricks may shorten other types of
+expressions.We can get rid of (or make optional) a lot of parentheses by making
+indentation significant. That's how programmers read code anyway:
+when indentation says one thing and delimiters say another, we go
+by the indentation. Treating indentation as significant would
+eliminate this common source of bugs as well as making programs
+shorter.Sometimes infix syntax is easier to read. This is especially true
+for math expressions. I've used Lisp my whole programming life and
+I still don't find prefix math expressions natural. And yet it is
+convenient, especially when you're generating code, to have operators
+that take any number of arguments. So if we do have infix syntax,
+it should probably be implemented as some kind of read-macro.I don't think we should be religiously opposed to introducing syntax
+into Lisp, as long as it translates in a well-understood way into
+underlying s-expressions. There is already a good deal of syntax
+in Lisp. It's not necessarily bad to introduce more, as long as no
+one is forced to use it. In Common Lisp, some delimiters are reserved
+for the language, suggesting that at least some of the designers
+intended to have more syntax in the future.One of the most egregiously unlispy pieces of syntax in Common Lisp
+occurs in format strings; format is a language in its own right,
+and that language is not Lisp. If there were a plan for introducing
+more syntax into Lisp, format specifiers might be able to be included
+in it. It would be a good thing if macros could generate format
+specifiers the way they generate any other kind of code.An eminent Lisp hacker told me that his copy of CLTL falls open to
+the section format. Mine too. This probably indicates room for
+improvement. It may also mean that programs do a lot of I/O.8 EfficiencyA good language, as everyone knows, should generate fast code. But
+in practice I don't think fast code comes primarily from things
+you do in the design of the language. As Knuth pointed out long
+ago, speed only matters in certain critical bottlenecks.  And as
+many programmers have observed since, one is very often mistaken
+about where these bottlenecks are.So, in practice, the way to get fast code is to have a very good
+profiler, rather than by, say, making the language strongly typed.
+You don't need to know the type of every argument in every call in
+the program. You do need to be able to declare the types of arguments
+in the bottlenecks. And even more, you need to be able to find out
+where the bottlenecks are.One complaint people have had with Lisp is that it's hard to tell
+what's expensive. This might be true. It might also be inevitable,
+if you want to have a very abstract language. And in any case I
+think good profiling would go a long way toward fixing the problem:
+you'd soon learn what was expensive.Part of the problem here is social. Language designers like to
+write fast compilers. That's how they measure their skill. They
+think of the profiler as an add-on, at best. But in practice a good
+profiler may do more to improve the speed of actual programs written
+in the language than a compiler that generates fast code. Here,
+again, language designers are somewhat out of touch with their
+users. They do a really good job of solving slightly the wrong
+problem.It might be a good idea to have an active profiler — to push
+performance data to the programmer instead of waiting for him to
+come asking for it. For example, the editor could display bottlenecks
+in red when the programmer edits the source code. Another approach
+would be to somehow represent what's happening in running programs.
+This would be an especially big win in server-based applications,
+where you have lots of running programs to look at. An active
+profiler could show graphically what's happening in memory as a
+program's running, or even make sounds that tell what's happening.Sound is a good cue to problems. In one place I worked, we had a
+big board of dials showing what was happening to our web servers.
+The hands were moved by little servomotors that made a slight noise
+when they turned. I couldn't see the board from my desk, but I
+found that I could tell immediately, by the sound, when there was
+a problem with a server.It might even be possible to write a profiler that would automatically
+detect inefficient algorithms. I would not be surprised if certain
+patterns of memory access turned out to be sure signs of bad
+algorithms. If there were a little guy running around inside the
+computer executing our programs, he would probably have as long
+and plaintive a tale to tell about his job as a federal government
+employee. I often have a feeling that I'm sending the processor on
+a lot of wild goose chases, but I've never had a good way to look
+at what it's doing.A number of Lisps now compile into byte code, which is then executed
+by an interpreter. This is usually done to make the implementation
+easier to port, but it could be a useful language feature. It might
+be a good idea to make the byte code an official part of the
+language, and to allow programmers to use inline byte code in
+bottlenecks. Then such optimizations would be portable too.The nature of speed, as perceived by the end-user, may be changing.
+With the rise of server-based applications, more and more programs
+may turn out to be i/o-bound. It will be worth making i/o fast.
+The language can help with straightforward measures like simple,
+fast, formatted output functions, and also with deep structural
+changes like caching and persistent objects.Users are interested in response time. But another kind of efficiency
+will be increasingly important: the number of simultaneous users
+you can support per processor. Many of the interesting applications
+written in the near future will be server-based, and the number of
+users per server is the critical question for anyone hosting such
+applications. In the capital cost of a business offering a server-based
+application, this is the divisor.For years, efficiency hasn't mattered much in most end-user
+applications. Developers have been able to assume that each user
+would have an increasingly powerful processor sitting on their
+desk. And by Parkinson's Law, software has expanded to use the
+resources available. That will change with server-based applications.
+In that world, the hardware and software will be supplied together.
+For companies that offer server-based applications, it will make
+a very big difference to the bottom line how many users they can
+support per server.In some applications, the processor will be the limiting factor,
+and execution speed will be the most important thing to optimize.
+But often memory will be the limit; the number of simultaneous
+users will be determined by the amount of memory you need for each
+user's data. The language can help here too. Good support for
+threads will enable all the users to share a single heap. It may
+also help to have persistent objects and/or language level support
+for lazy loading.9 TimeThe last ingredient a popular language needs is time. No one wants
+to write programs in a language that might go away, as so many
+programming languages do. So most hackers will tend to wait until
+a language has been around for a couple years before even considering
+using it.Inventors of wonderful new things are often surprised to discover
+this, but you need time to get any message through to people. A
+friend of mine rarely does anything the first time someone asks
+him. He knows that people sometimes ask for things that they turn
+out not to want. To avoid wasting his time, he waits till the third
+or fourth time he's asked to do something; by then, whoever's asking
+him may be fairly annoyed, but at least they probably really do
+want whatever they're asking for.Most people have learned to do a similar sort of filtering on new
+things they hear about. They don't even start paying attention
+until they've heard about something ten times. They're perfectly
+justified: the majority of hot new whatevers do turn out to be a
+waste of time, and eventually go away. By delaying learning VRML,
+I avoided having to learn it at all.So anyone who invents something new has to expect to keep repeating
+their message for years before people will start to get it. We
+wrote what was, as far as I know, the first web-server based
+application, and it took us years to get it through to people that
+it didn't have to be downloaded. It wasn't that they were stupid.
+They just had us tuned out.The good news is, simple repetition solves the problem. All you
+have to do is keep telling your story, and eventually people will
+start to hear. It's not when people notice you're there that they
+pay attention; it's when they notice you're still there.It's just as well that it usually takes a while to gain momentum.
+Most technologies evolve a good deal even after they're first
+launched — programming languages especially. Nothing could be better,
+for a new techology, than a few years of being used only by a small
+number of early adopters. Early adopters are sophisticated and
+demanding, and quickly flush out whatever flaws remain in your
+technology. When you only have a few users you can be in close
+contact with all of them. And early adopters are forgiving when
+you improve your system, even if this causes some breakage.There are two ways new technology gets introduced: the organic
+growth method, and the big bang method. The organic growth method
+is exemplified by the classic seat-of-the-pants underfunded garage
+startup. A couple guys, working in obscurity, develop some new
+technology. They launch it with no marketing and initially have
+only a few (fanatically devoted) users. They continue to improve
+the technology, and meanwhile their user base grows by word of
+mouth. Before they know it, they're big.The other approach, the big bang method, is exemplified by the
+VC-backed, heavily marketed startup. They rush to develop a product,
+launch it with great publicity, and immediately (they hope) have
+a large user base.Generally, the garage guys envy the big bang guys. The big bang
+guys are smooth and confident and respected by the VCs. They can
+afford the best of everything, and the PR campaign surrounding the
+launch has the side effect of making them celebrities. The organic
+growth guys, sitting in their garage, feel poor and unloved. And
+yet I think they are often mistaken to feel sorry for themselves.
+Organic growth seems to yield better technology and richer founders
+than the big bang method. If you look at the dominant technologies
+today, you'll find that most of them grew organically.This pattern doesn't only apply to companies. You see it in sponsored
+research too. Multics and Common Lisp were big-bang projects, and
+Unix and MacLisp were organic growth projects.10 Redesign"The best writing is rewriting," wrote E. B. White.  Every good
+writer knows this, and it's true for software too. The most important
+part of design is redesign. Programming languages, especially,
+don't get redesigned enough.To write good software you must simultaneously keep two opposing
+ideas in your head. You need the young hacker's naive faith in
+his abilities, and at the same time the veteran's skepticism. You
+have to be able to think 
+how hard can it be? with one half of
+your brain while thinking 
+it will never work with the other.The trick is to realize that there's no real contradiction here.
+You want to be optimistic and skeptical about two different things.
+You have to be optimistic about the possibility of solving the
+problem, but skeptical about the value of whatever solution you've
+got so far.People who do good work often think that whatever they're working
+on is no good. Others see what they've done and are full of wonder,
+but the creator is full of worry. This pattern is no coincidence:
+it is the worry that made the work good.If you can keep hope and worry balanced, they will drive a project
+forward the same way your two legs drive a bicycle forward. In the
+first phase of the two-cycle innovation engine, you work furiously
+on some problem, inspired by your confidence that you'll be able
+to solve it. In the second phase, you look at what you've done in
+the cold light of morning, and see all its flaws very clearly. But
+as long as your critical spirit doesn't outweigh your hope, you'll
+be able to look at your admittedly incomplete system, and think,
+how hard can it be to get the rest of the way?, thereby continuing
+the cycle.It's tricky to keep the two forces balanced. In young hackers,
+optimism predominates. They produce something, are convinced it's
+great, and never improve it. In old hackers, skepticism predominates,
+and they won't even dare to take on ambitious projects.Anything you can do to keep the redesign cycle going is good. Prose
+can be rewritten over and over until you're happy with it. But
+software, as a rule, doesn't get redesigned enough. Prose has
+readers, but software has users. If a writer rewrites an essay,
+people who read the old version are unlikely to complain that their
+thoughts have been broken by some newly introduced incompatibility.Users are a double-edged sword. They can help you improve your
+language, but they can also deter you from improving it. So choose
+your users carefully, and be slow to grow their number. Having
+users is like optimization: the wise course is to delay it. Also,
+as a general rule, you can at any given time get away with changing
+more than you think. Introducing change is like pulling off a
+bandage: the pain is a memory almost as soon as you feel it.Everyone knows that it's not a good idea to have a language designed
+by a committee. Committees yield bad design. But I think the worst
+danger of committees is that they interfere with redesign. It is
+so much work to introduce changes that no one wants to bother.
+Whatever a committee decides tends to stay that way, even if most
+of the members don't like it.Even a committee of two gets in the way of redesign. This happens
+particularly in the interfaces between pieces of software written
+by two different people. To change the interface both have to agree
+to change it at once. And so interfaces tend not to change at all,
+which is a problem because they tend to be one of the most ad hoc
+parts of any system.One solution here might be to design systems so that interfaces
+are horizontal instead of vertical — so that modules are always
+vertically stacked strata of abstraction. Then the interface will
+tend to be owned by one of them. The lower of two levels will either
+be a language in which the upper is written, in which case the
+lower level will own the interface, or it will be a slave, in which
+case the interface can be dictated by the upper level.11 LispWhat all this implies is that there is hope for a new Lisp.  There
+is hope for any language that gives hackers what they want, including
+Lisp. I think we may have made a mistake in thinking that hackers
+are turned off by Lisp's strangeness. This comforting illusion may
+have prevented us from seeing the real problem with Lisp, or at
+least Common Lisp, which is that it sucks for doing what hackers
+want to do. A hacker's language needs powerful libraries and
+something to hack. Common Lisp has neither. A hacker's language is
+terse and hackable. Common Lisp is not.The good news is, it's not Lisp that sucks, but Common Lisp. If we
+can develop a new Lisp that is a real hacker's language, I think
+hackers will use it. They will use whatever language does the job.
+All we have to do is make sure this new Lisp does some important
+job better than other languages.History offers some encouragement. Over time, successive new
+programming languages have taken more and more features from Lisp.
+There is no longer much left to copy before the language you've
+made is Lisp. The latest hot language, Python, is a watered-down
+Lisp with infix syntax and no macros. A new Lisp would be a natural
+step in this progression.I sometimes think that it would be a good marketing trick to call
+it an improved version of Python. That sounds hipper than Lisp. To
+many people, Lisp is a slow AI language with a lot of parentheses.
+Fritz Kunze's official biography carefully avoids mentioning the
+L-word.  But my guess is that we shouldn't be afraid to call the
+new Lisp Lisp. Lisp still has a lot of latent respect among the
+very best hackers — the ones who took 6.001 and understood it, for
+example. And those are the users you need to win.In "How to Become a Hacker," Eric Raymond describes Lisp as something
+like Latin or Greek — a language you should learn as an intellectual
+exercise, even though you won't actually use it:
+
+  Lisp is worth learning for the profound enlightenment experience
+  you will have when you finally get it; that experience will make
+  you a better programmer for the rest of your days, even if you
+  never actually use Lisp itself a lot.
+
+If I didn't know Lisp, reading this would set me asking questions.
+A language that would make me a better programmer, if it means
+anything at all, means a language that would be better for programming.
+And that is in fact the implication of what Eric is saying.As long as that idea is still floating around, I think hackers will
+be receptive enough to a new Lisp, even if it is called Lisp. But
+this Lisp must be a hacker's language, like the classic Lisps of
+the 1970s. It must be terse, simple, and hackable. And it must have
+powerful libraries for doing what hackers want to do now.In the matter of libraries I think there is room to beat languages
+like Perl and Python at their own game. A lot of the new applications
+that will need to be written in the coming years will be 
+server-based
+applications. There's no reason a new Lisp shouldn't have string
+libraries as good as Perl, and if this new Lisp also had powerful
+libraries for server-based applications, it could be very popular.
+Real hackers won't turn up their noses at a new tool that will let
+them solve hard problems with a few library calls. Remember, hackers
+are lazy.It could be an even bigger win to have core language support for
+server-based applications. For example, explicit support for programs
+with multiple users, or data ownership at the level of type tags.Server-based applications also give us the answer to the question
+of what this new Lisp will be used to hack. It would not hurt to
+make Lisp better as a scripting language for Unix. (It would be
+hard to make it worse.) But I think there are areas where existing
+languages would be easier to beat. I think it might be better to
+follow the model of Tcl, and supply the Lisp together with a complete
+system for supporting server-based applications. Lisp is a natural
+fit for server-based applications. Lexical closures provide a way
+to get the effect of subroutines when the ui is just a series of
+web pages. S-expressions map nicely onto html, and macros are good
+at generating it. There need to be better tools for writing
+server-based applications, and there needs to be a new Lisp, and
+the two would work very well together.12 The Dream LanguageBy way of summary, let's try describing the hacker's dream language.
+The dream language is 
+beautiful, clean, and terse. It has an
+interactive toplevel that starts up fast. You can write programs
+to solve common problems with very little code.  Nearly all the
+code in any program you write is code that's specific to your
+application. Everything else has been done for you.The syntax of the language is brief to a fault. You never have to
+type an unnecessary character, or even to use the shift key much.Using big abstractions you can write the first version of a program
+very quickly. Later, when you want to optimize, there's a really
+good profiler that tells you where to focus your attention. You
+can make inner loops blindingly fast, even writing inline byte code
+if you need to.There are lots of good examples to learn from, and the language is
+intuitive enough that you can learn how to use it from examples in
+a couple minutes. You don't need to look in the manual much. The
+manual is thin, and has few warnings and qualifications.The language has a small core, and powerful, highly orthogonal
+libraries that are as carefully designed as the core language. The
+libraries all work well together; everything in the language fits
+together like the parts in a fine camera. Nothing is deprecated,
+or retained for compatibility. The source code of all the libraries
+is readily available. It's easy to talk to the operating system
+and to applications written in other languages.The language is built in layers. The higher-level abstractions are
+built in a very transparent way out of lower-level abstractions,
+which you can get hold of if you want.Nothing is hidden from you that doesn't absolutely have to be. The
+language offers abstractions only as a way of saving you work,
+rather than as a way of telling you what to do. In fact, the language
+encourages you to be an equal participant in its design. You can
+change everything about it, including even its syntax, and anything
+you write has, as much as possible, the same status as what comes
+predefined.Notes[1]  Macros very close to the modern idea were proposed by Timothy
+Hart in 1964, two years after Lisp 1.5 was released. What was
+missing, initially, were ways to avoid variable capture and multiple
+evaluation; Hart's examples are subject to both.[2]  In When the Air Hits Your Brain, neurosurgeon Frank Vertosick
+recounts a conversation in which his chief resident, Gary, talks
+about the difference between surgeons and internists ("fleas"):
+
+  Gary and I ordered a large pizza and found an open booth. The
+  chief lit a cigarette. "Look at those goddamn fleas, jabbering
+  about some disease they'll see once in their lifetimes. That's
+  the trouble with fleas, they only like the bizarre stuff. They
+  hate their bread and butter cases. That's the difference between
+  us and the fucking fleas. See, we love big juicy lumbar disc
+  herniations, but they hate hypertension...."
+
+It's hard to think of a lumbar disc herniation as juicy (except
+literally). And yet I think I know what they mean. I've often had
+a juicy bug to track down. Someone who's not a programmer would
+find it hard to imagine that there could be pleasure in a bug.
+Surely it's better if everything just works. In one way, it is.
+And yet there is undeniably a grim satisfaction in hunting down
+certain sorts of bugs.

recipes/experimental/long-context/H2O/data/PaulGrahamEssays/pow.txt

@@ -0,0 +1,10 @@
+January 2017People who are powerful but uncharismatic will tend to be disliked.
+Their power makes them a target for criticism that they don't have
+the charisma to disarm. That was Hillary Clinton's problem. It also
+tends to be a problem for any CEO who is more of a builder than a
+schmoozer. And yet the builder-type CEO is (like Hillary) probably
+the best person for the job.I don't think there is any solution to this problem. It's human
+nature. The best we can do is to recognize that it's happening, and
+to understand that being a magnet for criticism is sometimes a sign
+not that someone is the wrong person for a job, but that they're
+the right one.

recipes/experimental/long-context/H2O/data/PaulGrahamEssays/rootsoflisp.txt

@@ -0,0 +1,41 @@
+May 2001
+
+(I wrote this article to help myself understand exactly
+what McCarthy discovered.  You don't need to know this stuff
+to program in Lisp, but it should be helpful to 
+anyone who wants to
+understand the essence of Lisp — both in the sense of its
+origins and its semantic core.  The fact that it has such a core
+is one of Lisp's distinguishing features, and the reason why,
+unlike other languages, Lisp has dialects.)In 1960, John 
+McCarthy published a remarkable paper in
+which he did for programming something like what Euclid did for
+geometry. He showed how, given a handful of simple
+operators and a notation for functions, you can
+build a whole programming language.
+He called this language Lisp, for "List Processing,"
+because one of his key ideas was to use a simple
+data structure called a list for both
+code and data.It's worth understanding what McCarthy discovered, not
+just as a landmark in the history of computers, but as
+a model for what programming is tending to become in
+our own time.  It seems to me that there have been
+two really clean, consistent models of programming so
+far: the C model and the Lisp model.
+These two seem points of high ground, with swampy lowlands
+between them.  As computers have grown more powerful,
+the new languages being developed have been moving
+steadily toward the Lisp model.  A popular recipe
+for new programming languages in the past 20 years 
+has been to take the C model of computing and add to
+it, piecemeal, parts taken from the Lisp model,
+like runtime typing and garbage collection.In this article I'm going to try to explain in the
+simplest possible terms what McCarthy discovered.
+The point is not just to learn about an interesting
+theoretical result someone figured out forty years ago,
+but to show where languages are heading.
+The unusual thing about Lisp — in fact, the defining
+quality of Lisp — is that it can be written in
+itself.  To understand what McCarthy meant by this,
+we're going to retrace his steps, with his mathematical
+notation translated into running Common Lisp code.

recipes/experimental/long-context/H2O/data/PaulGrahamEssays/rss.txt

@@ -0,0 +1,3 @@
+Aaron Swartz created a scraped
+feed
+of the essays page.

recipes/experimental/long-context/H2O/data/PaulGrahamEssays/siliconvalley.txt

@@ -0,0 +1,292 @@
+May 2006(This essay is derived from a keynote at Xtech.)Could you reproduce Silicon Valley elsewhere, or is there something
+unique about it?It wouldn't be surprising if it were hard to reproduce in other
+countries, because you couldn't reproduce it in most of the US
+either.  What does it take to make a silicon valley even here?What it takes is the right people.  If you could get the right ten
+thousand people to move from Silicon Valley to Buffalo, Buffalo
+would become Silicon Valley.  
+[1]That's a striking departure from the past.  Up till a couple decades
+ago, geography was destiny for cities.  All great cities were located
+on waterways, because cities made money by trade, and water was the
+only economical way to ship.Now you could make a great city anywhere, if you could get the right
+people to move there.  So the question of how to make a silicon
+valley becomes: who are the right people, and how do you get them
+to move?Two TypesI think you only need two kinds of people to create a technology
+hub: rich people and nerds.  They're the limiting reagents in the
+reaction that produces startups, because they're the only ones
+present when startups get started.  Everyone else will move.Observation bears this out: within the US, towns have become startup
+hubs if and only if they have both rich people and nerds.  Few
+startups happen in Miami, for example, because although it's full
+of rich people, it has few nerds.  It's not the kind of place nerds
+like.Whereas Pittsburgh has the opposite problem: plenty of nerds, but
+no rich people.  The top US Computer Science departments are said
+to be MIT, Stanford, Berkeley, and Carnegie-Mellon.  MIT yielded
+Route 128.  Stanford and Berkeley yielded Silicon Valley.  But
+Carnegie-Mellon?  The record skips at that point.  Lower down the
+list, the University of Washington yielded a high-tech community
+in Seattle, and the University of Texas at Austin yielded one in
+Austin.  But what happened in Pittsburgh?  And in Ithaca, home of
+Cornell, which is also high on the list?I grew up in Pittsburgh and went to college at Cornell, so I can
+answer for both.  The weather is terrible,  particularly in winter,
+and there's no interesting old city to make up for it, as there is
+in Boston.  Rich people don't want to live in Pittsburgh or Ithaca.
+So while there are plenty of hackers who could start startups,
+there's no one to invest in them.Not BureaucratsDo you really need the rich people?  Wouldn't it work to have the
+government invest in the nerds?  No, it would not.  Startup investors
+are a distinct type of rich people.  They tend to have a lot of
+experience themselves in the technology business.  This (a) helps
+them pick the right startups, and (b) means they can supply advice
+and connections as well as money.  And the fact that they have a
+personal stake in the outcome makes them really pay attention.Bureaucrats by their nature are the exact opposite sort of people
+from startup investors. The idea of them making startup investments
+is comic.  It would be like mathematicians running Vogue-- or
+perhaps more accurately, Vogue editors running a math journal.
+[2]Though indeed, most things bureaucrats do, they do badly.   We just
+don't notice usually, because they only have to compete against
+other bureaucrats.  But as startup investors they'd have to compete
+against pros with a great deal more experience and motivation.Even corporations that have in-house VC groups generally forbid
+them to make their own investment decisions.  Most are only allowed
+to invest in deals where some reputable private VC firm is willing
+to act as lead investor.Not BuildingsIf you go to see Silicon Valley, what you'll see are buildings.
+But it's the people that make it Silicon Valley, not the buildings.
+I read occasionally about attempts to set up "technology
+parks" in other places, as if the active ingredient of Silicon
+Valley were the office space.  An article about Sophia Antipolis
+bragged that companies there included Cisco, Compaq, IBM, NCR, and
+Nortel.  Don't the French realize these aren't startups?Building office buildings for technology companies won't get you a
+silicon valley, because the key stage in the life of a startup
+happens before they want that kind of space.  The key stage is when
+they're three guys operating out of an apartment.  Wherever the
+startup is when it gets funded, it will stay.  The defining quality
+of Silicon Valley is not that Intel or Apple or Google have offices
+there, but that they were started there.So if you want to reproduce Silicon Valley, what you need to reproduce
+is those two or three founders sitting around a kitchen table
+deciding to start a company.  And to reproduce that you need those
+people.UniversitiesThe exciting thing is, all you need are the people.  If you could
+attract a critical mass of nerds and investors to live somewhere,
+you could reproduce Silicon Valley.  And both groups are highly
+mobile.  They'll go where life is good.  So what makes a place good
+to them?What nerds like is other nerds.  Smart people will go wherever other
+smart people are.  And in particular, to great universities.  In
+theory there could be other ways to attract them, but so far
+universities seem to be indispensable.  Within the US, there are
+no technology hubs without first-rate universities-- or at least,
+first-rate computer science departments.So if you want to make a silicon valley, you not only need a
+university, but one of the top handful in the world.  It has to be
+good enough to act as a magnet, drawing the best people from thousands
+of miles away.  And that means it has to stand up to existing magnets
+like MIT and Stanford.This sounds hard.  Actually it might be easy.  My professor friends,
+when they're deciding where they'd like to work, consider one thing
+above all: the quality of the other faculty.  What attracts professors
+is good colleagues.  So if you managed to recruit, en masse, a
+significant number of the best young researchers, you could create
+a first-rate university from nothing overnight.  And you could do
+that for surprisingly little.  If you paid 200 people hiring bonuses
+of $3 million apiece, you could put together a faculty that would
+bear comparison with any in the world.  And from that point the
+chain reaction would be self-sustaining.  So whatever it costs to
+establish a mediocre university, for an additional half billion or
+so you could have a great one.  
+[3]PersonalityHowever, merely creating a new university would not be enough to
+start a silicon valley. The university is just the seed.  It has
+to be planted in the right soil, or it won't germinate.  Plant it
+in the wrong place, and you just create Carnegie-Mellon.To spawn startups, your university has to be in a town that has
+attractions other than the university.  It has to be a place where
+investors want to live, and students want to stay after they graduate.The two like much the same things, because most startup investors
+are nerds themselves.  So what do nerds look for in a town?  Their
+tastes aren't completely different from other people's, because a
+lot of the towns they like most in the US are also big tourist
+destinations: San Francisco, Boston, Seattle.   But their tastes
+can't be quite mainstream either, because they dislike other big
+tourist destinations, like New York, Los Angeles, and Las Vegas.There has been a lot written lately about the "creative class." The
+thesis seems to be that as wealth derives increasingly from ideas,
+cities will prosper only if they attract those who have them.  That
+is certainly true; in fact it was the basis of Amsterdam's prosperity
+400 years ago.A lot of nerd tastes they share with the creative class in general.
+For example, they like well-preserved old neighborhoods instead of
+cookie-cutter suburbs, and locally-owned shops and restaurants
+instead of national chains.  Like the rest of the creative class,
+they want to live somewhere with personality.What exactly is personality?  I think it's the feeling that each
+building is the work of a distinct group of people.  A town with
+personality is one that doesn't feel mass-produced.  So if you want
+to make a startup hub-- or any town to attract the "creative class"--
+you probably have to ban large development projects.
+When a large tract has been developed by a single organization, you
+can always tell. 
+[4]Most towns with personality are old, but they don't have to be.
+Old towns have two advantages: they're denser, because they were
+laid out before cars, and they're more varied, because they were
+built one building at a time.  You could have both now.  Just have
+building codes that ensure density, and ban large scale developments.A corollary is that you have to keep out the biggest developer of
+all: the government.  A government that asks "How can we build a
+silicon valley?" has probably ensured failure by the way they framed
+the question.  You don't build a silicon valley; you let one grow.NerdsIf you want to attract nerds, you need more than a town with
+personality.  You need a town with the right personality.  Nerds
+are a distinct subset of the creative class, with different tastes
+from the rest.  You can see this most clearly in New York, which
+attracts a lot of creative people, but few nerds. 
+[5]What nerds like is the kind of town where people walk around smiling.
+This excludes LA, where no one walks at all, and also New York,
+where people walk, but not smiling. When I was in grad school in
+Boston, a friend came to visit from New York.  On the subway back
+from the airport she asked "Why is everyone smiling?"  I looked and
+they weren't smiling.  They just looked like they were compared to
+the facial expressions she was used to.If you've lived in New York, you know where these facial expressions
+come from.  It's the kind of place where your mind may be excited,
+but your body knows it's having a bad time.  People don't so much
+enjoy living there as endure it for the sake of the excitement.
+And if you like certain kinds of excitement, New York is incomparable.
+It's a hub of glamour, a magnet for all the shorter half-life
+isotopes of style and fame.Nerds don't care about glamour, so to them the appeal of New York
+is a mystery.  People who like New York will pay a fortune for a
+small, dark, noisy apartment in order to live in a town where the
+cool people are really cool.  A nerd looks at that deal and sees
+only: pay a fortune for a small, dark, noisy apartment.Nerds will pay a premium to live in a town where the smart people
+are really smart, but you don't have to pay as much for that.  It's
+supply and demand: glamour is popular, so you have to pay a lot for
+it.Most nerds like quieter pleasures.  They like cafes instead of
+clubs; used bookshops instead of fashionable clothing shops; hiking
+instead of dancing; sunlight instead of tall buildings.  A nerd's
+idea of paradise is Berkeley or Boulder.YouthIt's the young nerds who start startups, so it's those specifically
+the city has to appeal to.  The startup hubs in the US are all
+young-feeling towns.  This doesn't mean they have to be new.
+Cambridge has the oldest town plan in America, but it feels young
+because it's full of students.What you can't have, if you want to create a silicon valley, is a
+large, existing population of stodgy people.  It would be a waste
+of time to try to reverse the fortunes of a declining industrial town
+like Detroit or Philadelphia by trying to encourage startups.  Those
+places have too much momentum in the wrong direction.  You're better
+off starting with a blank slate in the form of a small town.  Or
+better still, if there's a town young people already flock to, that
+one.The Bay Area was a magnet for the young and optimistic for decades
+before it was associated with technology.  It was a place people
+went in search of something new.  And so it became synonymous with
+California nuttiness.  There's still a lot of that there.  If you
+wanted to start a new fad-- a new way to focus one's "energy," for
+example, or a new category of things not to eat-- the Bay Area would
+be the place to do it.  But a place that tolerates oddness in the
+search for the new is exactly what you want in a startup hub, because
+economically that's what startups are.  Most good startup ideas
+seem a little crazy; if they were obviously good ideas, someone
+would have done them already.(How many people are going to want computers in their houses?
+What, another search engine?)That's the connection between technology and liberalism.  Without
+exception the high-tech cities in the US are also the most liberal.
+But it's not because liberals are smarter that this is so.  It's
+because liberal cities tolerate odd ideas, and smart people by
+definition have odd ideas.Conversely, a town that gets praised for being "solid" or representing
+"traditional values" may be a fine place to live, but it's never
+going to succeed as a startup hub.  The 2004 presidential election,
+though a disaster in other respects, conveniently supplied us with
+a county-by-county 
+map of such places.  
+[6]To attract the young, a town must have an intact center.  In most
+American cities the center has been abandoned, and the growth, if
+any, is in the suburbs.  Most American cities have been turned
+inside out. But none of the startup hubs has: not San Francisco,
+or Boston, or Seattle.  They all have intact centers.
+[7]
+My guess is that no city with a dead center could be turned into a
+startup hub.  Young people don't want to live in the suburbs.Within the US, the two cities I think could most easily be turned
+into new silicon valleys are Boulder and Portland.  Both have the
+kind of effervescent feel that attracts the young.  They're each
+only a great university short of becoming a silicon valley, if they
+wanted to.TimeA great university near an attractive town.  Is that all it takes?
+That was all it took to make the original Silicon Valley.  Silicon
+Valley traces its origins to William Shockley, one of the inventors
+of the transistor.  He did the research that won him the Nobel Prize
+at Bell Labs, but when he started his own company in 1956 he moved
+to Palo Alto to do it.   At the time that was an odd thing to do.
+Why did he?  Because he had grown up there and remembered how nice
+it was.  Now Palo Alto is suburbia, but then it was a charming
+college town-- a charming college town with perfect weather and San
+Francisco only an hour away.The companies that rule Silicon Valley now are all descended in
+various ways from Shockley Semiconductor.  Shockley was a difficult
+man, and in 1957 his top people-- "the traitorous eight"-- left to
+start a new company, Fairchild Semiconductor.  Among them were
+Gordon Moore and Robert Noyce, who went on to found Intel, and
+Eugene Kleiner, who founded the VC firm Kleiner Perkins.  Forty-two
+years later, Kleiner Perkins funded Google, and the partner responsible
+for the deal was John Doerr, who came to Silicon Valley in 1974 to
+work for Intel.So although a lot of the newest companies in Silicon Valley don't
+make anything out of silicon, there always seem to be multiple links
+back to Shockley.  There's a lesson here: startups beget startups.
+People who work for startups start their own.  People who get rich
+from startups fund new ones.  I suspect this kind of organic growth
+is the only way to produce a startup hub, because it's the only way
+to grow the expertise you need.That has two important implications.  The first is that you need
+time to grow a silicon valley.  The university you could create in
+a couple years, but the startup community around it has to grow
+organically.   The cycle time is limited by the time it takes a
+company to succeed, which probably averages about five years.The other implication of the organic growth hypothesis is that you
+can't be somewhat of a startup hub.  You either have a self-sustaining
+chain reaction, or not.  Observation confirms this too: cities
+either have a startup scene, or they don't.  There is no middle
+ground.  Chicago has the third largest metropolitan area in America.
+As source of startups it's negligible compared to Seattle, number 15.The good news is that the initial seed can be quite small.  Shockley
+Semiconductor, though itself not very successful, was big enough.
+It brought a critical mass of experts in an important new technology
+together in a place they liked enough to stay.CompetingOf course, a would-be silicon valley faces an obstacle the original
+one didn't: it has to compete with Silicon Valley.  Can that be
+done?  Probably.One of Silicon Valley's biggest advantages is its venture capital
+firms.  This was not a factor in Shockley's day, because VC funds
+didn't exist.  In fact, Shockley Semiconductor and Fairchild
+Semiconductor were not startups at all in our sense.  They were
+subsidiaries-- of Beckman Instruments and Fairchild Camera and
+Instrument respectively.  Those companies were apparently willing
+to establish subsidiaries wherever the experts wanted to live.Venture investors, however, prefer to fund startups within an hour's
+drive.  For one, they're more likely to notice startups nearby.
+But when they do notice startups in other towns they prefer them
+to move.  They don't want to have to travel to attend board meetings,
+and in any case the odds of succeeding are higher in a startup hub.The centralizing effect of venture firms is a double one: they cause
+startups to form around them, and those draw in more startups through
+acquisitions.  And although the first may be weakening because it's
+now so cheap to start some startups, the second seems as strong as ever.
+Three of the most admired
+"Web 2.0" companies were started outside the usual startup hubs,
+but two of them have already been reeled in through acquisitions.Such centralizing forces make it harder for new silicon valleys to
+get started.  But by no means impossible.  Ultimately power rests
+with the founders.  A startup with the best people will beat one
+with funding from famous VCs, and a startup that was sufficiently
+successful would never have to move.  So a town that
+could exert enough pull over the right people could resist and
+perhaps even surpass Silicon Valley.For all its power, Silicon Valley has a great weakness: the paradise
+Shockley found in 1956 is now one giant parking lot.  San Francisco
+and Berkeley are great, but they're forty miles away.  Silicon
+Valley proper is soul-crushing suburban sprawl.  It
+has fabulous weather, which makes it significantly better than the
+soul-crushing sprawl of most other American cities.  But a competitor
+that managed to avoid sprawl would have real leverage.  All a city
+needs is to be the kind of place the next traitorous eight look at
+and say "I want to stay here," and that would be enough to get the
+chain reaction started.Notes[1]
+It's interesting to consider how low this number could be
+made.  I suspect five hundred would be enough, even if they could
+bring no assets with them.  Probably just thirty, if I could pick them, 
+would be enough to turn Buffalo into a significant startup hub.[2]
+Bureaucrats manage to allocate research funding moderately
+well, but only because (like an in-house VC fund) they outsource
+most of the work of selection.  A professor at a famous university
+who is highly regarded by his peers will get funding, pretty much
+regardless of the proposal.  That wouldn't work for startups, whose
+founders aren't sponsored by organizations, and are often unknowns.[3]
+You'd have to do it all at once, or at least a whole department
+at a time, because people would be more likely to come if they
+knew their friends were.  And you should probably start from scratch,
+rather than trying to upgrade an existing university, or much energy
+would be lost in friction.[4]
+Hypothesis: Any plan in which multiple independent buildings
+are gutted or demolished to be "redeveloped" as a single project
+is a net loss of personality for the city, with the exception of
+the conversion of buildings not previously public, like warehouses.[5]
+A few startups get started in New York, but less
+than a tenth as many per capita as in Boston, and mostly
+in less nerdy fields like finance and media.[6]
+Some blue counties are false positives (reflecting the
+remaining power of Democractic party machines), but there are no
+false negatives.  You can safely write off all the red counties.[7]
+Some "urban renewal" experts took a shot at destroying Boston's
+in the 1960s, leaving the area around city hall a bleak wasteland,
+but most neighborhoods successfully resisted them.Thanks to Chris Anderson, Trevor Blackwell, Marc Hedlund,
+Jessica Livingston, Robert Morris, Greg Mcadoo, Fred Wilson,
+and Stephen Wolfram for
+reading drafts of this, and to Ed Dumbill for inviting me to speak.(The second part of this talk became Why Startups
+Condense in America.)

recipes/experimental/long-context/H2O/data/PaulGrahamEssays/startuplessons.txt

@@ -0,0 +1,395 @@
+April 2006(This essay is derived from a talk at the 2006 
+Startup School.)The startups we've funded so far are pretty quick, but they seem
+quicker to learn some lessons than others.  I think it's because
+some things about startups are kind of counterintuitive.We've now 
+invested 
+in enough companies that I've learned a trick
+for determining which points are the counterintuitive ones:
+they're the ones I have to keep repeating.So I'm going to number these points, and maybe with future startups
+I'll be able to pull off a form of Huffman coding. I'll make them
+all read this, and then instead of nagging them in detail, I'll
+just be able to say: number four!
+1. Release Early.The thing I probably repeat most is this recipe for a startup: get
+a version 1 out fast, then improve it based on users' reactions.By "release early" I don't mean you should release something full
+of bugs, but that you should release something minimal.  Users hate
+bugs, but they don't seem to mind a minimal version 1, if there's
+more coming soon.There are several reasons it pays to get version 1 done fast.  One
+is that this is simply the right way to write software, whether for
+a startup or not.  I've been repeating that since 1993, and I haven't seen much since to
+contradict it.  I've seen a lot of startups die because they were
+too slow to release stuff, and none because they were too quick.
+[1]One of the things that will surprise you if you build something
+popular is that you won't know your users.  Reddit now has almost half a million
+unique visitors a month.  Who are all those people?  They have no
+idea.  No web startup does.  And since you don't know your users,
+it's dangerous to guess what they'll like.  Better to release
+something and let them tell you.Wufoo took this to heart and released
+their form-builder before the underlying database.  You can't even
+drive the thing yet, but 83,000 people came to sit in the driver's
+seat and hold the steering wheel.  And Wufoo got valuable feedback
+from it: Linux users complained they used too much Flash, so they
+rewrote their software not to.  If they'd waited to release everything
+at once, they wouldn't have discovered this problem till it was
+more deeply wired in.Even if you had no users, it would still be important to release
+quickly, because for a startup the initial release acts as a shakedown
+cruise.  If anything major is broken-- if the idea's no good,
+for example, or the founders hate one another-- the stress of getting
+that first version out will expose it.  And if you have such problems
+you want to find them early.Perhaps the most important reason to release early, though, is that
+it makes you work harder.  When you're working on something that
+isn't released, problems are intriguing.  In something that's out
+there, problems are alarming.  There is a lot more urgency once you
+release.  And I think that's precisely why people put it off.  They
+know they'll have to work a lot harder once they do. 
+[2]
+2. Keep Pumping Out Features.Of course, "release early" has a second component, without which
+it would be bad advice.  If you're going to start with something
+that doesn't do much, you better improve it fast.What I find myself repeating is "pump out features."  And this rule
+isn't just for the initial stages.  This is something all startups
+should do for as long as they want to be considered startups.I don't mean, of course, that you should make your application ever
+more complex.  By "feature" I mean one unit of hacking-- one quantum
+of making users' lives better.As with exercise, improvements beget improvements.  If you run every
+day, you'll probably feel like running tomorrow.  But if you skip
+running for a couple weeks, it will be an effort to drag yourself
+out.  So it is with hacking: the more ideas you implement, the more
+ideas you'll have.  You should make your system better at least in
+some small way every day or two.This is not just a good way to get development done; it is also a
+form of marketing.  Users love a site that's constantly improving.
+In fact, users expect a site to improve.  Imagine if you visited a
+site that seemed very good, and then returned two months later and
+not one thing had changed.  Wouldn't it start to seem lame? 
+[3]They'll like you even better when you improve in response to their
+comments, because customers are used to companies ignoring them.
+If you're the rare exception-- a company that actually listens--
+you'll generate fanatical loyalty.  You won't need to advertise,
+because your users will do it for you.This seems obvious too, so why do I have to keep repeating it?  I
+think the problem here is that people get used to how things are.
+Once a product gets past the stage where it has glaring flaws, you
+start to get used to it, and gradually whatever features it happens
+to have become its identity.  For example, I doubt many people at
+Yahoo (or Google for that matter) realized how much better web mail
+could be till Paul Buchheit showed them.I think the solution is to assume that anything you've made is far
+short of what it could be.  Force yourself, as a sort of intellectual
+exercise, to keep thinking of improvements.  Ok, sure, what you
+have is perfect.  But if you had to change something, what would
+it be?If your product seems finished, there are two possible explanations:
+(a) it is finished, or (b) you lack imagination.  Experience suggests
+(b) is a thousand times more likely.
+3. Make Users Happy.Improving constantly is an instance of a more general rule: make
+users happy.  One thing all startups have in common is that they
+can't force anyone to do anything.  They can't force anyone to use
+their software, and they can't force anyone to do deals with them.
+A startup has to sing for its supper.  That's why the successful
+ones make great things.  They have to, or die.When you're running a startup you feel like a little bit of debris
+blown about by powerful winds.  The most powerful wind is users.
+They can either catch you and loft you up into the sky, as they did
+with Google, or leave you flat on the pavement, as they do with
+most startups.  Users are a fickle wind, but more powerful than any
+other.  If they take you up, no competitor can keep you down.As a little piece of debris, the rational thing for you to do is
+not to lie flat, but to curl yourself into a shape the wind will
+catch.I like the wind metaphor because it reminds you how impersonal the
+stream of traffic is.  The vast majority of people who visit your
+site will be casual visitors.  It's them you have to design your
+site for.  The people who really care will find what they want by
+themselves.The median visitor will arrive with their finger poised on the Back
+button.  Think about your own experience: most links you
+follow lead to something lame.  Anyone who has used the web for
+more than a couple weeks has been trained to click on Back after
+following a link.  So your site has to say "Wait!  Don't click on
+Back.  This site isn't lame.  Look at this, for example."There are two things you have to do to make people pause.  The most
+important is to explain, as concisely as possible, what the hell
+your site is about.  How often have you visited a site that seemed
+to assume you already knew what they did?  For example, the corporate
+site that says the
+company makes
+
+  enterprise content management solutions for business that enable
+  organizations to unify people, content and processes to minimize
+  business risk, accelerate time-to-value and sustain lower total
+  cost of ownership.
+
+An established company may get away with such an opaque description,
+but no startup can.  A startup
+should be able to explain in one or two sentences exactly what it
+does. 
+[4]
+And not just to users.  You need this for everyone:
+investors, acquirers, partners, reporters, potential employees, and
+even current employees.  You probably shouldn't even start a company
+to do something that can't be described compellingly in one or two
+sentences.The other thing I repeat is to give people everything you've got,
+right away.  If you have something impressive, try to put it on the
+front page, because that's the only one most visitors will see.
+Though indeed there's a paradox here: the more you push the good
+stuff toward the front, the more likely visitors are to explore
+further. 
+[5]In the best case these two suggestions get combined: you tell
+visitors what your site is about by showing them.  One of the
+standard pieces of advice in fiction writing is "show, don't tell."
+Don't say that a character's angry; have him grind his teeth, or
+break his pencil in half.  Nothing will explain what your site does
+so well as using it.The industry term here is "conversion."  The job of your site is
+to convert casual visitors into users-- whatever your definition
+of a user is.  You can measure this in your growth rate.  Either
+your site is catching on, or it isn't, and you must know which.  If
+you have decent growth, you'll win in the end, no matter how obscure
+you are now.  And if you don't, you need to fix something.
+4. Fear the Right Things.Another thing I find myself saying a lot is "don't worry."  Actually,
+it's more often "don't worry about this; worry about that instead."
+Startups are right to be paranoid, but they sometimes fear the wrong
+things.Most visible disasters are not so alarming as they seem.  Disasters
+are normal in a startup: a founder quits, you discover a patent
+that covers what you're doing, your servers keep crashing, you run
+into an insoluble technical problem, you have to change your name,
+a deal falls through-- these are all par for the course.  They won't
+kill you unless you let them.Nor will most competitors.  A lot of startups worry "what if Google
+builds something like us?"  Actually big companies are not the ones
+you have to worry about-- not even Google.  The people at Google
+are smart, but no smarter than you; they're not as motivated, because
+Google is not going to go out of business if this one product fails;
+and even at Google they have a lot of bureaucracy to slow them down.What you should fear, as a startup, is not the established players,
+but other startups you don't know exist yet.  They're way more
+dangerous than Google because, like you, they're cornered animals.Looking just at existing competitors can give you a false sense of
+security.  You should compete against what someone else could be
+doing, not just what you can see people doing.  A corollary is that
+you shouldn't relax just because you have no visible competitors
+yet.  No matter what your idea, there's someone else out there
+working on the same thing.That's the downside of it being easier to start a startup: more people
+are doing it.  But I disagree with Caterina Fake when she says that
+makes this a bad time to start a startup.  More people are starting
+startups, but not as many more as could.  Most college graduates
+still think they have to get a job.  The average person can't ignore
+something that's been beaten into their head since they were three
+just because serving web pages recently got a lot cheaper.And in any case, competitors are not the biggest threat.  Way more
+startups hose themselves than get crushed by competitors.  There
+are a lot of ways to do it, but the three main ones are internal
+disputes, inertia, and ignoring users.  Each is, by itself, enough
+to kill you.  But if I had to pick the worst, it would be ignoring
+users.  If you want a recipe for a startup that's going to die,
+here it is: a couple of founders who have some great idea they know
+everyone is going to love, and that's what they're going to build,
+no matter what.Almost everyone's initial plan is broken.  If companies stuck to
+their initial plans, Microsoft would be selling programming languages,
+and Apple would be selling printed circuit boards.  In both cases
+their customers told them what their business should be-- and they
+were smart enough to listen.As Richard Feynman said, the imagination of nature is greater than
+the imagination of man.  You'll find more interesting things by
+looking at the world than you could ever produce just by thinking.
+This principle is very powerful.  It's why the best abstract painting
+still falls short of Leonardo, for example.  And it applies to
+startups too.  No idea for a product could ever be so clever as the
+ones you can discover by smashing a beam of prototypes into a beam
+of users.
+5. Commitment Is a Self-Fulfilling Prophecy.I now have enough experience with startups to be able to say what
+the most important quality is in a startup founder, and it's not
+what you might think.  The most important quality in a startup
+founder is determination.  Not intelligence-- determination.This is a little depressing.  I'd like to believe Viaweb succeeded
+because we were smart, not merely determined.  A lot of people in
+the startup world want to believe that.  Not just founders, but
+investors too.  They like the idea of inhabiting a world ruled by
+intelligence.  And you can tell they really believe this, because
+it affects their investment decisions.Time after time VCs invest in startups founded by eminent professors.
+This may work in biotech, where a lot of startups simply commercialize
+existing research, but in software you want to invest in students,
+not professors.  Microsoft, Yahoo, and Google were all founded by
+people who dropped out of school to do it.  What students lack in
+experience they more than make up in dedication.Of course, if you want to get rich, it's not enough merely to be
+determined.  You have to be smart too, right?  I'd like to think
+so, but I've had an experience that convinced me otherwise: I spent
+several years living in New York.You can lose quite a lot in the brains department and it won't kill
+you.  But lose even a little bit in the commitment department, and
+that will kill you very rapidly.Running a startup is like walking on your hands: it's possible, but
+it requires extraordinary effort.  If an ordinary employee were
+asked to do the things a startup founder has to, he'd be very
+indignant.  Imagine if you were hired at some big company, and in
+addition to writing software ten times faster than you'd ever had
+to before, they expected you to answer support calls, administer
+the servers, design the web site, cold-call customers, find the
+company office space, and go out and get everyone lunch.And to do all this not in the calm, womb-like atmosphere of a big
+company, but against a backdrop of constant disasters.  That's the
+part that really demands determination.  In a startup, there's
+always some disaster happening.  So if you're the least bit inclined
+to find an excuse to quit, there's always one right there.But if you lack commitment, chances are it will have been hurting
+you long before you actually quit.  Everyone who deals with startups
+knows how important commitment is, so if they sense you're ambivalent,
+they won't give you much attention.  If you lack commitment, you'll
+just find that for some mysterious reason good things happen to
+your competitors but not to you.  If you lack commitment, it will
+seem to you that you're unlucky.Whereas if you're determined to stick around, people will pay
+attention to you, because odds are they'll have to deal with you
+later.  You're a local, not just a tourist, so everyone has to come
+to terms with you.At Y Combinator we sometimes mistakenly fund teams who have the
+attitude that they're going to give this startup thing a shot for
+three months, and if something great happens, they'll stick with
+it-- "something great" meaning either that someone wants to buy
+them or invest millions of dollars in them.  But if this is your
+attitude, "something great" is very unlikely to happen to you,
+because both acquirers and investors judge you by your level of
+commitment.If an acquirer thinks you're going to stick around no matter what,
+they'll be more likely to buy you, because if they don't and you
+stick around, you'll probably grow, your price will go up, and
+they'll be left wishing they'd bought you earlier.  Ditto for
+investors.  What really motivates investors, even big VCs, is not
+the hope of good returns, but the fear of missing out. 
+[6]
+So if
+you make it clear you're going to succeed no matter what, and the only
+reason you need them is to make it happen a little faster, you're
+much more likely to get money.You can't fake this.  The only way to convince everyone that you're
+ready to fight to the death is actually to be ready to.You have to be the right kind of determined, though.  I carefully
+chose the word determined rather than stubborn, because stubbornness
+is a disastrous quality in a startup.  You have to be determined,
+but flexible, like a running back.  A successful running back doesn't
+just put his head down and try to run through people.  He improvises:
+if someone appears in front of him, he runs around them; if someone
+tries to grab him, he spins out of their grip; he'll even run in
+the wrong direction briefly if that will help.  The one thing he'll
+never do is stand still. 
+[7]
+6. There Is Always Room.I was talking recently to a startup founder about whether it might
+be good to add a social component to their software.  He said he
+didn't think so, because the whole social thing was tapped out.
+Really?  So in a hundred years the only social networking sites
+will be the Facebook, MySpace, Flickr, and Del.icio.us?  Not likely.There is always room for new stuff.  At every point in history,
+even the darkest bits of the dark ages, people were discovering
+things that made everyone say "why didn't anyone think of that
+before?"  We know this continued to be true up till 2004, when the
+Facebook was founded-- though strictly speaking someone else did
+think of that.The reason we don't see the opportunities all around us is that we
+adjust to however things are, and assume that's how things have to
+be.  For example, it would seem crazy to most people to try to make
+a better search engine than Google.  Surely that field, at least,
+is tapped out.  Really?  In a hundred years-- or even twenty-- are
+people still going to search for information using something like
+the current Google?  Even Google probably doesn't think that.In particular, I don't think there's any limit to the number of
+startups.  Sometimes you hear people saying "All these guys starting
+startups now are going to be disappointed. How many little startups
+are Google and Yahoo going to buy, after all?" That sounds cleverly
+skeptical, but I can prove it's mistaken.  No one proposes that
+there's some limit to the number of people who can be employed in
+an economy consisting of big, slow-moving companies with a couple
+thousand people each.  Why should there be any limit to the number
+who could be employed by small, fast-moving companies with ten each?
+It seems to me the only limit would be the number of people who
+want to work that hard.The limit on the number of startups is not the number that can get
+acquired by Google and Yahoo-- though it seems even that should
+be unlimited, if the startups were actually worth buying-- but the
+amount of wealth that can be created.  And I don't think there's
+any limit on that, except cosmological ones.So for all practical purposes, there is no limit to the number of
+startups.  Startups make wealth, which means they make things people
+want, and if there's a limit on the number of things people want,
+we are nowhere near it.  I still don't even have a flying car.
+7. Don't Get Your Hopes Up.This is another one I've been repeating since long before Y Combinator.
+It was practically the corporate motto at Viaweb.Startup founders are naturally optimistic.  They wouldn't do it
+otherwise.  But you should treat your optimism the way you'd treat
+the core of a nuclear reactor: as a source of power that's also
+very dangerous.  You have to build a shield around it, or it will
+fry you.The shielding of a reactor is not uniform; the reactor would be
+useless if it were.  It's pierced in a few places to let pipes in.
+An optimism shield has to be pierced too.  I think the place to
+draw the line is between what you expect of yourself, and what you
+expect of other people.  It's ok to be optimistic about what you
+can do, but assume the worst about machines and other people.This is particularly necessary in a startup, because you tend to
+be pushing the limits of whatever you're doing.  So things don't
+happen in the smooth, predictable way they do in the rest of the
+world.  Things change suddenly, and usually for the worse.Shielding your optimism is nowhere more important than with deals.
+If your startup is doing a deal, just assume it's not going to
+happen.  The VCs who say they're going to invest in you aren't.
+The company that says they're going to buy you isn't.  The big
+customer who wants to use your system in their whole company won't.
+Then if things work out you can be pleasantly surprised.The reason I warn startups not to get their hopes up is not to save
+them from being disappointed when things fall through.  It's
+for a more practical reason: to prevent them from leaning their
+company against something that's going to fall over, taking them
+with it.For example, if someone says they want to invest in you, there's a
+natural tendency to stop looking for other investors.  That's why
+people proposing deals seem so positive: they want you to
+stop looking.  And you want to stop too, because doing deals is a
+pain.  Raising money, in particular, is a huge time sink.  So you
+have to consciously force yourself to keep looking.Even if you ultimately do the first deal, it will be to your advantage
+to have kept looking, because you'll get better terms.  Deals are
+dynamic; unless you're negotiating with someone unusually honest,
+there's not a single point where you shake hands and the deal's
+done. There are usually a lot of subsidiary questions to be cleared
+up after the handshake, and if the other side senses weakness-- if
+they sense you need this deal-- they will be very tempted to screw
+you in the details.VCs and corp dev guys are professional negotiators.  They're trained
+to take advantage of weakness. 
+[8]
+So while they're often nice
+guys, they just can't help it.  And as pros they do this more than
+you.  So don't even try to bluff them.  The only way a startup can
+have any leverage in a deal is genuinely not to need it.  And if
+you don't believe in a deal, you'll be less likely to depend on it.So I want to plant a hypnotic suggestion in your heads: when you
+hear someone say the words "we want to invest in you" or "we want
+to acquire you," I want the following phrase to appear automatically
+in your head: don't get your hopes up.  Just continue running
+your company as if this deal didn't exist.  Nothing is more likely
+to make it close.The way to succeed in a startup is to focus on the goal of getting
+lots of users, and keep walking swiftly toward it while investors
+and acquirers scurry alongside trying to wave money in your face.
+Speed, not MoneyThe way I've described it, starting a startup sounds pretty stressful.
+It is.  When I talk to the founders of the companies we've funded,
+they all say the same thing: I knew it would be hard, but I didn't
+realize it would be this hard.So why do it?  It would be worth enduring a lot of pain and stress
+to do something grand or heroic, but just to make money?  Is making
+money really that important?No, not really.  It seems ridiculous to me when people take business
+too seriously.  I regard making money as a boring errand to be got
+out of the way as soon as possible.  There is nothing grand or
+heroic about starting a startup per se.So why do I spend so much time thinking about startups?  I'll tell
+you why.  Economically, a startup is best seen not as a way to get
+rich, but as a way to work faster.  You have to make a living, and
+a startup is a way to get that done quickly, instead of letting it
+drag on through your whole life.
+[9]We take it for granted most of the time, but human life is fairly
+miraculous.  It is also palpably short.  You're given this marvellous
+thing, and then poof, it's taken away.  You can see why people
+invent gods to explain it.  But even to people who don't believe
+in gods, life commands respect.  There are times in most of our
+lives when the days go by in a blur, and almost everyone has a
+sense, when this happens, of wasting something precious.  As Ben
+Franklin said, if you love life, don't waste time, because time is
+what life is made of.So no, there's nothing particularly grand about making money.  That's
+not what makes startups worth the trouble.  What's important about
+startups is the speed.  By compressing the dull but necessary task
+of making a living into the smallest possible time, you show respect
+for life, and there is something grand about that.Notes[1]
+Startups can die from releasing something full of bugs, and not
+fixing them fast enough, but I don't know of any that died from
+releasing something stable but minimal very early, then promptly
+improving it.[2]
+I know this is why I haven't released Arc.  The moment I do,
+I'll have people nagging me for features.[3]
+A web site is different from a book or movie or desktop application
+in this respect.  Users judge a site not as a single snapshot, but
+as an animation with multiple frames.  Of the two, I'd say the rate of
+improvement is more important to users than where you currently
+are.[4]
+It should not always tell this to users, however.  For example,
+MySpace is basically a replacement mall for mallrats.  But it was
+wiser for them, initially, to pretend that the site was about bands.[5]
+Similarly, don't make users register to try your site.  Maybe
+what you have is so valuable that visitors should gladly register
+to get at it.  But they've been trained to expect the opposite.
+Most of the things they've tried on the web have sucked-- and
+probably especially those that made them register.[6]
+VCs have rational reasons for behaving this way. They don't
+make their money (if they make money) off their median investments.
+In a typical fund, half the companies fail, most of the rest generate
+mediocre returns, and one or two "make the fund" by succeeding
+spectacularly.  So if they miss just a few of the most promising
+opportunities, it could hose the whole fund.[7]
+The attitude of a running back doesn't translate to soccer.
+Though it looks great when a forward dribbles past multiple defenders,
+a player who persists in trying such things will do worse in the
+long term than one who passes.[8]
+The reason Y Combinator never negotiates valuations
+is that we're not professional negotiators, and don't want to turn
+into them.[9]
+There are two ways to do 
+work you love: (a) to make money, then work
+on what you love, or (b) to get a job where you get paid to work on
+stuff you love.  In practice the first phases of both
+consist mostly of unedifying schleps, and in (b) the second phase is less
+secure.Thanks to Sam Altman, Trevor Blackwell, Beau Hartshorne, Jessica 
+Livingston, and Robert Morris for reading drafts of this.

recipes/experimental/long-context/H2O/data/PaulGrahamEssays/submarine.txt

@@ -0,0 +1,217 @@
+April 2005"Suits make a corporate comeback," says the New
+York Times.  Why does this sound familiar?  Maybe because
+the suit was also back in February,
+
+September
+2004, June
+2004, March
+2004, September
+2003, 
+
+November
+2002, 
+April 2002,
+and February
+2002.
+
+Why do the media keep running stories saying suits are back?  Because
+PR firms tell 
+them to.  One of the most surprising things I discovered
+during my brief business career was the existence of the PR industry,
+lurking like a huge, quiet submarine beneath the news.  Of the
+stories you read in traditional media that aren't about politics,
+crimes, or disasters, more than half probably come from PR firms.I know because I spent years hunting such "press hits."  Our startup spent
+its entire marketing budget on PR: at a time when we were assembling
+our own computers to save money, we were paying a PR firm $16,000
+a month.  And they were worth it.  PR is the news equivalent of
+search engine optimization; instead of buying ads, which readers
+ignore, you get yourself inserted directly into the stories.  [1]Our PR firm
+was one of the best in the business.  In 18 months, they got press
+hits in over 60 different publications.  
+And we weren't the only ones they did great things for.  
+In 1997 I got a call from another
+startup founder considering hiring them to promote his company.  I
+told him they were PR gods, worth every penny of their outrageous   
+fees.  But I remember thinking his company's name was odd.
+Why call an auction site "eBay"?
+SymbiosisPR is not dishonest.  Not quite.  In fact, the reason the best PR
+firms are so effective is precisely that they aren't dishonest.
+They give reporters genuinely valuable information.  A good PR firm
+won't bug reporters just because the client tells them to; they've
+worked hard to build their credibility with reporters, and they
+don't want to destroy it by feeding them mere propaganda.If anyone is dishonest, it's the reporters.  The main reason PR  
+firms exist is that reporters are lazy.  Or, to put it more nicely,
+overworked.  Really they ought to be out there digging up stories
+for themselves.  But it's so tempting to sit in their offices and
+let PR firms bring the stories to them.  After all, they know good
+PR firms won't lie to them.A good flatterer doesn't lie, but tells his victim selective truths
+(what a nice color your eyes are). Good PR firms use the same
+strategy: they give reporters stories that are true, but whose truth
+favors their clients.For example, our PR firm often pitched stories about how the Web  
+let small merchants compete with big ones.  This was perfectly true.
+But the reason reporters ended up writing stories about this
+particular truth, rather than some other one, was that small merchants
+were our target market, and we were paying the piper.Different publications vary greatly in their reliance on PR firms.
+At the bottom of the heap are the trade press, who make most of
+their money from advertising and would give the magazines away for
+free if advertisers would let them.  [2] The average
+trade publication is a  bunch of ads, glued together by just enough
+articles to make it look like a magazine.  They're so desperate for
+"content" that some will print your press releases almost verbatim,
+if you take the trouble to write them to read like articles.At the other extreme are publications like the New York Times
+and the Wall Street Journal.  Their reporters do go out and
+find their own stories, at least some of the time.  They'll listen 
+to PR firms, but briefly and skeptically.  We managed to get press   
+hits in almost every publication we wanted, but we never managed 
+to crack the print edition of the Times.  [3]The weak point of the top reporters is not laziness, but vanity.
+You don't pitch stories to them.  You have to approach them as if
+you were a specimen under their all-seeing microscope, and make it
+seem as if the story you want them to run is something they thought 
+of themselves.Our greatest PR coup was a two-part one.  We estimated, based on
+some fairly informal math, that there were about 5000 stores on the
+Web.  We got one paper to print this number, which seemed neutral   
+enough.  But once this "fact" was out there in print, we could quote
+it to other publications, and claim that with 1000 users we had 20%
+of the online store market.This was roughly true.  We really did have the biggest share of the
+online store market, and 5000 was our best guess at its size.  But
+the way the story appeared in the press sounded a lot more definite.Reporters like definitive statements.  For example, many of the
+stories about Jeremy Jaynes's conviction say that he was one of the
+10 worst spammers.  This "fact" originated in Spamhaus's ROKSO list,
+which I think even Spamhaus would admit is a rough guess at the top
+spammers.  The first stories about Jaynes cited this source, but
+now it's simply repeated as if it were part of the indictment.   
+[4]All you can say with certainty about Jaynes is that he was a fairly
+big spammer.  But reporters don't want to print vague stuff like
+"fairly big."  They want statements with punch, like "top ten." And
+PR firms give them what they want.
+Wearing suits, we're told, will make us 
+3.6
+percent more productive.BuzzWhere the work of PR firms really does get deliberately misleading is in
+the generation of "buzz."  They usually feed the same story to    
+several different publications at once.  And when readers see similar
+stories in multiple places, they think there is some important trend
+afoot.  Which is exactly what they're supposed to think.When Windows 95 was launched, people waited outside stores
+at midnight to buy the first copies.  None of them would have been
+there without PR firms, who generated such a buzz in
+the news media that it became self-reinforcing, like a nuclear chain
+reaction.I doubt PR firms realize it yet, but the Web makes it possible to  
+track them at work.  If you search for the obvious phrases, you
+turn up several efforts over the years to place stories about the  
+return of the suit.  For example, the Reuters article 
+
+that got picked up by USA
+Today in September 2004.  "The suit is back," it begins.Trend articles like this are almost always the work of
+PR firms.  Once you know how to read them, it's straightforward to
+figure out who the client is.  With trend stories, PR firms usually
+line up one or more "experts" to talk about the industry generally. 
+In this case we get three: the NPD Group, the creative director of
+GQ, and a research director at Smith Barney.  [5] When
+you get to the end of the experts, look for the client. And bingo, 
+there it is: The Men's Wearhouse.Not surprising, considering The Men's Wearhouse was at that moment 
+running ads saying "The Suit is Back."  Talk about a successful
+press hit-- a wire service article whose first sentence is your own
+ad copy.The secret to finding other press hits from a given pitch
+is to realize that they all started from the same document back at
+the PR firm.  Search for a few key phrases and the names of the
+clients and the experts, and you'll turn up other variants of this 
+story.Casual
+fridays are out and dress codes are in writes Diane E. Lewis
+in The Boston Globe.  In a remarkable coincidence, Ms. Lewis's
+industry contacts also include the creative director of GQ.Ripped jeans and T-shirts are out, writes Mary Kathleen Flynn in
+US News & World Report.  And she too knows the 
+creative director of GQ.Men's suits
+are back writes Nicole Ford in Sexbuzz.Com ("the ultimate men's
+entertainment magazine").Dressing
+down loses appeal as men suit up at the office writes Tenisha
+Mercer of The Detroit News.
+Now that so many news articles are online, I suspect you could find
+a similar pattern for most trend stories placed by PR firms.  I
+propose we call this new sport "PR diving," and I'm sure there are
+far more striking examples out there than this clump of five stories.OnlineAfter spending years chasing them, it's now second nature
+to me to recognize press hits for what they are.  But before we
+hired a PR firm I had no idea where articles in the mainstream media
+came from.  I could tell a lot of them were crap, but I didn't
+realize why.Remember the exercises in critical reading you did in school, where
+you had to look at a piece of writing and step back and ask whether
+the author was telling the whole truth?  If you really want to be
+a critical reader, it turns out you have to step back one step
+further, and ask not just whether the author is telling the truth,
+but why he's writing about this subject at all.Online, the answer tends to be a lot simpler.  Most people who
+publish online write what they write for the simple reason that
+they want to.  You
+can't see the fingerprints of PR firms all over the articles, as
+you can in so many print publications-- which is one of the reasons,
+though they may not consciously realize it, that readers trust
+bloggers more than Business Week.I was talking recently to a friend who works for a
+big newspaper.  He thought the print media were in serious trouble,
+and that they were still mostly in denial about it.  "They think
+the decline is cyclic," he said.  "Actually it's structural."In other words, the readers are leaving, and they're not coming
+back.
+Why? I think the main reason is that the writing online is more honest.
+Imagine how incongruous the New York Times article about
+suits would sound if you read it in a blog:
+   The urge to look corporate-- sleek, commanding,
+  prudent, yet with just a touch of hubris on your well-cut sleeve--
+  is an unexpected development in a time of business disgrace.
+   
+The problem
+with this article is not just that it originated in a PR firm.
+The whole tone is bogus.  This is the tone of someone writing down
+to their audience.Whatever its flaws, the writing you find online
+is authentic.  It's not mystery meat cooked up
+out of scraps of pitch letters and press releases, and pressed into 
+molds of zippy
+journalese.  It's people writing what they think.I didn't realize, till there was an alternative, just how artificial
+most of the writing in the mainstream media was.  I'm not saying
+I used to believe what I read in Time and Newsweek.  Since high
+school, at least, I've thought of magazines like that more as
+guides to what ordinary people were being
+told to think than as  
+sources of information.  But I didn't realize till the last  
+few years that writing for publication didn't have to mean writing
+that way.  I didn't realize you could write as candidly and
+informally as you would if you were writing to a friend.Readers aren't the only ones who've noticed the
+change.  The PR industry has too.
+A hilarious article
+on the site of the PR Society of America gets to the heart of the   
+matter:
+   Bloggers are sensitive about becoming mouthpieces
+  for other organizations and companies, which is the reason they
+  began blogging in the first place.  
+PR people fear bloggers for the same reason readers
+like them.  And that means there may be a struggle ahead.  As
+this new kind of writing draws readers away from traditional media, we
+should be prepared for whatever PR mutates into to compensate.  
+When I think   
+how hard PR firms work to score press hits in the traditional   
+media, I can't imagine they'll work any less hard to feed stories
+to bloggers, if they can figure out how.
+Notes[1] PR has at least   
+one beneficial feature: it favors small companies.  If PR didn't  
+work, the only alternative would be to advertise, and only big
+companies can afford that.[2] Advertisers pay 
+less for ads in free publications, because they assume readers 
+ignore something they get for free.  This is why so many trade
+publications nominally have a cover price and yet give away free
+subscriptions with such abandon.[3] Different sections
+of the Times vary so much in their standards that they're
+practically different papers.  Whoever fed the style section reporter
+this story about suits coming back would have been sent packing by
+the regular news reporters.[4] The most striking
+example I know of this type is the "fact" that the Internet worm   
+of 1988 infected 6000 computers. I was there when it was cooked up,
+and this was the recipe: someone guessed that there were about
+60,000 computers attached to the Internet, and that the worm might
+have infected ten percent of them.Actually no one knows how many computers the worm infected, because
+the remedy was to reboot them, and this destroyed all traces.  But
+people like numbers.  And so this one is now replicated
+all over the Internet, like a little worm of its own.[5] Not all were
+necessarily supplied by the PR firm. Reporters sometimes call a few
+additional sources on their own, like someone adding a few fresh 
+vegetables to a can of soup.
+Thanks to Ingrid Basset, Trevor Blackwell, Sarah Harlin, Jessica 
+Livingston, Jackie McDonough, Robert Morris, and Aaron Swartz (who
+also found the PRSA article) for reading drafts of this.Correction: Earlier versions used a recent
+Business Week article mentioning del.icio.us as an example
+of a press hit, but Joshua Schachter tells me 
+it was spontaneous.

recipes/experimental/long-context/H2O/data/PaulGrahamEssays/superangels.txt

@@ -0,0 +1,302 @@
+
+
+Want to start a startup?  Get funded by
+Y Combinator.
+
+
+
+
+October 2010After barely changing at all for decades, the startup funding
+business is now in what could, at least by comparison, be called
+turmoil.  At Y Combinator we've seen dramatic changes in the funding
+environment for startups.  Fortunately one of them is much higher
+valuations.The trends we've been seeing are probably not YC-specific.  I wish
+I could say they were, but the main cause is probably just that we
+see trends first—partly because the startups we fund are very
+plugged into the Valley and are quick to take advantage of anything
+new, and partly because we fund so many that we have enough data
+points to see patterns clearly.What we're seeing now, everyone's probably going to be seeing in
+the next couple years.  So I'm going to explain what we're seeing,
+and what that will mean for you if you try to raise money.Super-AngelsLet me start by describing what the world of startup funding used
+to look like.  There used to be two sharply differentiated types
+of investors: angels and venture capitalists.  Angels are individual
+rich people who invest small amounts of their own money, while VCs
+are employees of funds that invest large amounts of other people's.For decades there were just those two types of investors, but now
+a third type has appeared halfway between them: the so-called
+super-angels. 
+[1]
+  And VCs have been provoked by their arrival
+into making a lot of angel-style investments themselves.  So the
+previously sharp line between angels and VCs has become hopelessly
+blurred.There used to be a no man's land between angels and VCs.  Angels
+would invest $20k to $50k apiece, and VCs usually a million or more.
+So an angel round meant a collection of angel investments that
+combined to maybe $200k, and a VC round meant a series A round in
+which a single VC fund (or occasionally two) invested $1-5 million.The no man's land between angels and VCs was a very inconvenient
+one for startups, because it coincided with the amount many wanted
+to raise.  Most startups coming out of Demo Day wanted to raise
+around $400k.  But it was a pain to stitch together that much out
+of angel investments, and most VCs weren't interested in investments
+so small.  That's the fundamental reason the super-angels have
+appeared.  They're responding to the market.The arrival of a new type of investor is big news for startups,
+because there used to be only two and they rarely competed with one
+another.  Super-angels compete with both angels and VCs.  That's
+going to change the rules about how to raise money.  I don't know
+yet what the new rules will be, but it looks like most of the changes
+will be for the better.A super-angel has some of the qualities of an angel, and some of
+the qualities of a VC.  They're usually individuals, like angels.
+In fact many of the current super-angels were initially angels of
+the classic type.  But like VCs, they invest other people's money.
+This allows them to invest larger amounts than angels:  a typical
+super-angel investment is currently about $100k.  They make investment
+decisions quickly, like angels.  And they make a lot more investments
+per partner than VCs—up to 10 times as many.The fact that super-angels invest other people's money makes them
+doubly alarming to VCs. They don't just compete for startups; they
+also compete for investors.  What super-angels really are is a new
+form of fast-moving, lightweight VC fund.   And those of us in the
+technology world know what usually happens when something comes
+along that can be described in terms like that.  Usually it's the
+replacement.Will it be?  As of now, few of the startups that take money from
+super-angels are ruling out taking VC money.  They're just postponing
+it.  But that's still a problem for VCs.  Some of the startups that
+postpone raising VC money may do so well on the angel money they
+raise that they never bother to raise more.  And those who do raise
+VC rounds will be able to get higher valuations when they do.  If
+the best startups get 10x higher valuations when they raise series
+A rounds, that would cut VCs' returns from winners at least tenfold.
+[2]So I think VC funds are seriously threatened by the super-angels.
+But one thing that may save them to some extent is the uneven
+distribution of startup outcomes: practically all the returns are
+concentrated in a few big successes.  The expected value of a startup
+is the percentage chance it's Google.  So to the extent that winning
+is a matter of absolute returns, the super-angels could win practically
+all the battles for individual startups and yet lose the war, if
+they merely failed to get those few big winners.  And there's a
+chance that could happen, because the top VC funds have better
+brands, and can also do more for their portfolio companies.  
+[3]Because super-angels make more investments per partner, they have
+less partner per investment.  They can't pay as much attention to
+you as a VC on your board could.  How much is that extra attention
+worth?  It will vary enormously from one partner to another.  There's
+no consensus yet in the general case.  So for now this is something
+startups are deciding individually.Till now, VCs' claims about how much value they added were sort of
+like the government's.  Maybe they made you feel better, but you
+had no choice in the matter, if you needed money on the scale only
+VCs could supply.  Now that VCs have competitors, that's going to
+put a market price on the help they offer.  The interesting thing
+is, no one knows yet what it will be.Do startups that want to get really big need the sort of advice and
+connections only the top VCs can supply?  Or would super-angel money
+do just as well?  The VCs will say you need them, and the super-angels
+will say you don't.  But the truth is, no one knows yet, not even
+the VCs and super-angels themselves.   All the super-angels know
+is that their new model seems promising enough to be worth trying,
+and all the VCs know is that it seems promising enough to worry
+about.RoundsWhatever the outcome, the conflict between VCs and super-angels is
+good news for founders.  And not just for the obvious reason that
+more competition for deals means better terms.  The whole shape of
+deals is changing.One of the biggest differences between angels and VCs is the amount
+of your company they want.  VCs want a lot.  In a series A round
+they want a third of your company, if they can get it.  They don't
+care much how much they pay for it, but they want a lot because the
+number of series A investments they can do is so small.  In a
+traditional series A investment, at least one partner from the VC
+fund takes a seat on your board.  
+[4]
+ Since board seats last about
+5 years and each partner can't handle more than about 10 at once,
+that means a VC fund can only do about 2 series A deals per partner
+per year. And that means they need to get as much of the company
+as they can in each one.  You'd have to be a very promising startup
+indeed to get a VC to use up one of his 10 board seats for only a
+few percent of you.Since angels generally don't take board seats, they don't have this
+constraint.  They're happy to buy only a few percent of you.  And
+although the super-angels are in most respects mini VC funds, they've
+retained this critical property of angels.  They don't take board
+seats, so they don't need a big percentage of your company.Though that means you'll get correspondingly less attention from
+them, it's good news in other respects.  Founders never really liked
+giving up as much equity as VCs wanted.  It was a lot of the company
+to give up in one shot.  Most founders doing series A deals would
+prefer to take half as much money for half as much stock, and then
+see what valuation they could get for the second half of the stock
+after using the first half of the money to increase its value.  But
+VCs never offered that option.Now startups have another alternative.  Now it's easy to raise angel
+rounds about half the size of series A rounds.  Many of the startups
+we fund are taking this route, and I predict that will be true of
+startups in general.A typical big angel round might be $600k on a convertible note with
+a valuation cap of $4 million premoney.  Meaning that when the note
+converts into stock (in a later round, or upon acquisition), the
+investors in that round will get .6 / 4.6, or 13% of the company.
+That's a lot less than the 30 to 40% of the company you usually
+give up in a series A round if you do it so early.  
+[5]But the advantage of these medium-sized rounds is not just that
+they cause less dilution.  You also lose less control.  After an
+angel round, the founders almost always still have control of the
+company, whereas after a series A round they often don't.  The
+traditional board structure after a series A round is two founders,
+two VCs, and a (supposedly) neutral fifth person.  Plus series A
+terms usually give the investors a veto over various kinds of
+important decisions, including selling the company.  Founders usually
+have a lot of de facto control after a series A, as long as things
+are going well.  But that's not the same as just being able to do
+what you want, like you could before.A third and quite significant advantage of angel rounds is that
+they're less stressful to raise.  Raising a traditional series A
+round has in the past taken weeks, if not months.  When a VC firm
+can only do 2 deals per partner per year, they're careful about
+which they do.  To get a traditional series A round you have to go
+through a series of meetings, culminating in a full partner meeting
+where the firm as a whole says yes or no.  That's the really scary
+part for founders: not just that series A rounds take so long, but
+at the end of this long process the VCs might still say no.  The
+chance of getting rejected after the full partner meeting averages
+about 25%.  At some firms it's over 50%.Fortunately for founders, VCs have been getting a lot faster.
+Nowadays Valley VCs are more likely to take 2 weeks than 2 months.
+But they're still not as fast as angels and super-angels, the most
+decisive of whom sometimes decide in hours.Raising an angel round is not only quicker, but you get feedback
+as it progresses.  An angel round is not an all or nothing thing
+like a series A.  It's composed of multiple investors with varying
+degrees of seriousness, ranging from the upstanding ones who commit
+unequivocally to the jerks who give you lines like "come back to
+me to fill out the round." You usually start collecting money from
+the most committed investors and work your way out toward the
+ambivalent ones, whose interest increases as the round fills up.But at each point you know how you're doing.  If investors turn
+cold you may have to raise less, but when investors in an angel
+round turn cold the process at least degrades gracefully, instead
+of blowing up in your face and leaving you with nothing, as happens
+if you get rejected by a VC fund after a full partner meeting.
+Whereas if investors seem hot, you can not only close the round
+faster, but now that convertible notes are becoming the norm,
+actually raise the price to reflect demand.ValuationHowever, the VCs have a weapon they can use against the super-angels,
+and they have started to use it.   VCs have started making angel-sized
+investments too.  The term "angel round" doesn't mean that all the
+investors in it are angels; it just describes the structure of the
+round.  Increasingly the participants include VCs making investments
+of a hundred thousand or two.  And when VCs invest in angel rounds
+they can do things that super-angels don't like.  VCs are quite
+valuation-insensitive in angel rounds—partly because they are
+in general, and partly because they don't care that much about the
+returns on angel rounds, which they still view mostly as a way to
+recruit startups for series A rounds later.  So VCs who invest in
+angel rounds can blow up the valuations for angels and super-angels
+who invest in them. 
+[6]Some super-angels seem to care about valuations.  Several turned
+down YC-funded startups after Demo Day because their valuations
+were too high.  This was not a problem for the startups; by definition
+a high valuation means enough investors were willing to accept it.
+But it was mysterious to me that the super-angels would quibble
+about valuations.  Did they not understand that the big returns
+come from a few big successes, and that it therefore mattered far
+more which startups you picked than how much you paid for them?After thinking about it for a while and observing certain other
+signs, I have a theory that explains why the super-angels may be
+smarter than they seem.  It would make sense for super-angels to
+want low valuations if they're hoping to invest in startups that
+get bought early.  If you're hoping to hit the next Google, you
+shouldn't care if the valuation is 20 million.  But if you're looking
+for companies that are going to get bought for 30 million, you care.
+If you invest at 20 and the company gets bought for 30, you only
+get 1.5x.  You might as well buy Apple.So if some of the super-angels were looking for companies that could
+get acquired quickly, that would explain why they'd care about
+valuations.  But why would they be looking for those?   Because
+depending on the meaning of "quickly," it could actually be very
+profitable.  A company that gets acquired for 30 million is a failure
+to a VC, but it could be a 10x return for an angel, and moreover,
+a quick 10x return.  Rate of return is what matters in
+investing—not the multiple you get, but the multiple per year.
+If a super-angel gets 10x in one year, that's a higher rate of
+return than a VC could ever hope to get from a company that took 6
+years to go public.  To get the same rate of return, the VC would
+have to get a multiple of 10^6—one million x.  Even Google
+didn't come close to that.So I think at least some super-angels are looking for companies
+that will get bought.  That's the only rational explanation for
+focusing on getting the right valuations, instead of the right
+companies.  And if so they'll be different to deal with than VCs.
+They'll be tougher on valuations, but more accommodating if you want
+to sell early.PrognosisWho will win, the super-angels or the VCs?  I think the answer to
+that is, some of each.  They'll each become more like one another.
+The super-angels will start to invest larger amounts, and the VCs
+will gradually figure out ways to make more, smaller investments
+faster.  A decade from now the players will be hard to tell apart,
+and there will probably be survivors from each group.What does that mean for founders?  One thing it means is that the
+high valuations startups are presently getting may not last forever.
+To the extent that valuations are being driven up by price-insensitive
+VCs, they'll fall again if VCs become more like super-angels and
+start to become more miserly about valuations.  Fortunately if this
+does happen it will take years.The short term forecast is more competition between investors, which
+is good news for you.  The super-angels will try to undermine the
+VCs by acting faster, and the VCs will try to undermine the
+super-angels by driving up valuations.  Which for founders will
+result in the perfect combination: funding rounds that close fast,
+with high valuations.But remember that to get that combination, your startup will have
+to appeal to both super-angels and VCs.  If you don't seem like you
+have the potential to go public, you won't be able to use VCs to
+drive up the valuation of an angel round.There is a danger of having VCs in an angel round: the so-called
+signalling risk.  If VCs are only doing it in the hope of investing
+more later, what happens if they don't?  That's a signal to everyone
+else that they think you're lame.How much should you worry about that?  The seriousness of signalling
+risk depends on how far along you are.  If by the next time you
+need to raise money, you have graphs showing rising revenue or
+traffic month after month, you don't have to worry about any signals
+your existing investors are sending.  Your results will speak for
+themselves.  
+[7]Whereas if the next time you need to raise money you won't yet have
+concrete results, you may need to think more about the message your
+investors might send if they don't invest more.  I'm not sure yet
+how much you have to worry, because this whole phenomenon of VCs
+doing angel investments is so new. But my instincts tell me you
+don't have to worry much.  Signalling risk smells like one of those
+things founders worry about that's not a real problem.  As a rule,
+the only thing that can kill a good startup is the startup itself.
+Startups hurt themselves way more often than competitors hurt them,
+for example.  I suspect signalling risk is in this category too.One thing YC-funded startups have been doing to mitigate the risk
+of taking money from VCs in angel rounds is not to take too much
+from any one VC.  Maybe that will help, if you have the luxury of
+turning down money.Fortunately, more and more startups will.  After decades of competition
+that could best be described as intramural, the startup funding
+business is finally getting some real competition.  That should
+last several years at least, and maybe a lot longer. Unless there's
+some huge market crash, the next couple years are going to be a
+good time for startups to raise money.  And that's exciting because
+it means lots more startups will happen.
+Notes[1]
+I've also heard them called "Mini-VCs" and "Micro-VCs." I
+don't know which name will stick.There were a couple predecessors.  Ron Conway had angel funds
+starting in the 1990s, and in some ways First Round Capital is closer to a
+super-angel than a VC fund.[2]
+It wouldn't cut their overall returns tenfold, because investing
+later would probably (a) cause them to lose less on investments
+that failed, and (b) not allow them to get as large a percentage
+of startups as they do now.  So it's hard to predict precisely what
+would happen to their returns.[3]
+The brand of an investor derives mostly from the success of
+their portfolio companies.  The top VCs thus have a big brand
+advantage over the super-angels.  They could make it self-perpetuating
+if they used it to get all the best new startups.  But I don't think
+they'll be able to.  To get all the best startups, you have to do
+more than make them want you.  You also have to want them; you have
+to recognize them when you see them, and that's much harder.
+Super-angels will snap up stars that VCs miss.  And that will cause
+the brand gap between the top VCs and the super-angels gradually
+to erode.[4]
+Though in a traditional series A round VCs put two partners
+on your board, there are signs now that VCs may begin to conserve
+board seats by switching to what used to be considered an angel-round
+board, consisting of two founders and one VC.  Which is also to the
+founders' advantage if it means they still control the company.[5]
+In a series A round, you usually have to give up more than
+the actual amount of stock the VCs buy, because they insist you
+dilute yourselves to set aside an "option pool" as well.  I predict
+this practice will gradually disappear though.[6]
+The best thing for founders, if they can get it, is a convertible
+note with no valuation cap at all.  In that case the money invested
+in the angel round just converts into stock at the valuation of the
+next round, no matter how large.  Angels and super-angels tend not
+to like uncapped notes. They have no idea how much of the company
+they're buying.  If the company does well and the valuation of the
+next round is high, they may end up with only a sliver of it.  So
+by agreeing to uncapped notes, VCs who don't care about valuations
+in angel rounds can make offers that super-angels hate to match.[7]
+Obviously signalling risk is also not a problem if you'll
+never need to raise more money.  But startups are often mistaken
+about that.Thanks to Sam Altman, John Bautista, Patrick Collison, James
+Lindenbaum, Reid Hoffman, Jessica Livingston and Harj Taggar
+for reading drafts
+of this.

recipes/experimental/long-context/H2O/data/PaulGrahamEssays/todo.txt

@@ -0,0 +1,22 @@
+April 2012A palliative care nurse called Bronnie Ware made a list of the
+biggest regrets
+of the dying.  Her list seems plausible.  I could see
+myself — can see myself — making at least 4 of these
+5 mistakes.If you had to compress them into a single piece of advice, it might
+be: don't be a cog.  The 5 regrets paint a portrait of post-industrial
+man, who shrinks himself into a shape that fits his circumstances,
+then turns dutifully till he stops.The alarming thing is, the mistakes that produce these regrets are
+all errors of omission.  You forget your dreams, ignore your family,
+suppress your feelings, neglect your friends, and forget to be
+happy.  Errors of omission are a particularly dangerous type of
+mistake, because you make them by default.I would like to avoid making these mistakes.  But how do you avoid
+mistakes you make by default?  Ideally you transform your life so
+it has other defaults.  But it may not be possible to do that
+completely. As long as these mistakes happen by default, you probably
+have to be reminded not to make them.  So I inverted the 5 regrets,
+yielding a list of 5 commands
+
+   Don't ignore your dreams; don't work too much; say what you
+   think; cultivate friendships; be happy.
+
+which I then put at the top of the file I use as a todo list.

recipes/experimental/long-context/H2O/data/PaulGrahamEssays/unions.txt

@@ -0,0 +1,42 @@
+May 2007People who worry about the increasing gap between rich and poor
+generally look back on the mid twentieth century as a golden age.
+In those days we had a large number of high-paying union manufacturing
+jobs that boosted the median income.  I wouldn't quite call the
+high-paying union job a myth, but I think people who dwell on it
+are reading too much into it.Oddly enough, it was working with startups that made me realize
+where the high-paying union job came from.  In a rapidly growing
+market, you don't worry too much about efficiency.  It's more
+important to grow fast.  If there's some mundane problem getting
+in your way, and there's a simple solution that's somewhat expensive,
+just take it and get on with more important things.  EBay didn't
+win by paying less for servers than their competitors.Difficult though it may be to imagine now, manufacturing was a
+growth industry in the mid twentieth century.  This was an era when
+small firms making everything from cars to candy were getting
+consolidated into a new kind of corporation with national reach and
+huge economies of scale.  You had to grow fast or die.  Workers
+were for these companies what servers are for an Internet startup.
+A reliable supply was more important than low cost.If you looked in the head of a 1950s auto executive, the attitude
+must have been: sure, give 'em whatever they ask for, so long as
+the new model isn't delayed.In other words, those workers were not paid what their work was
+worth.  Circumstances being what they were, companies would have
+been stupid to insist on paying them so little.If you want a less controversial example of this phenomenon, ask
+anyone who worked as a consultant building web sites during the
+Internet Bubble.  In the late nineties you could get paid huge sums
+of money for building the most trivial things.  And yet does anyone
+who was there have any expectation those days will ever return?  I
+doubt it.  Surely everyone realizes that was just a temporary
+aberration.The era of labor unions seems to have been the same kind of aberration, 
+just spread
+over a longer period, and mixed together with a lot of ideology
+that prevents people from viewing it with as cold an eye as they
+would something like consulting during the Bubble.Basically, unions were just Razorfish.People who think the labor movement was the creation of heroic union
+organizers have a problem to explain: why are unions shrinking now?
+The best they can do is fall back on the default explanation of
+people living in fallen civilizations.  Our ancestors were giants.
+The workers of the early twentieth century must have had a moral
+courage that's lacking today.In fact there's a simpler explanation.  The early twentieth century
+was just a fast-growing startup overpaying for infrastructure.  And
+we in the present are not a fallen people, who have abandoned
+whatever mysterious high-minded principles produced the high-paying
+union job.  We simply live in a time when the fast-growing companies
+overspend on different things.

recipes/experimental/long-context/H2O/data/PaulGrahamEssays/useful.txt

@@ -0,0 +1,225 @@
+February 2020What should an essay be? Many people would say persuasive. That's
+what a lot of us were taught essays should be. But I think we can
+aim for something more ambitious: that an essay should be useful.To start with, that means it should be correct. But it's not enough
+merely to be correct. It's easy to make a statement correct by
+making it vague. That's a common flaw in academic writing, for
+example. If you know nothing at all about an issue, you can't go
+wrong by saying that the issue is a complex one, that there are
+many factors to be considered, that it's a mistake to take too
+simplistic a view of it, and so on.Though no doubt correct, such statements tell the reader nothing.
+Useful writing makes claims that are as strong as they can be made
+without becoming false.For example, it's more useful to say that Pike's Peak is near the
+middle of Colorado than merely somewhere in Colorado. But if I say
+it's in the exact middle of Colorado, I've now gone too far, because
+it's a bit east of the middle.Precision and correctness are like opposing forces. It's easy to
+satisfy one if you ignore the other. The converse of vaporous
+academic writing is the bold, but false, rhetoric of demagogues.
+Useful writing is bold, but true.It's also two other things: it tells people something important,
+and that at least some of them didn't already know.Telling people something they didn't know doesn't always mean
+surprising them. Sometimes it means telling them something they
+knew unconsciously but had never put into words. In fact those may
+be the more valuable insights, because they tend to be more
+fundamental.Let's put them all together. Useful writing tells people something
+true and important that they didn't already know, and tells them
+as unequivocally as possible.Notice these are all a matter of degree. For example, you can't
+expect an idea to be novel to everyone. Any insight that you have
+will probably have already been had by at least one of the world's
+7 billion people. But it's sufficient if an idea is novel to a lot
+of readers.Ditto for correctness, importance, and strength. In effect the four
+components are like numbers you can multiply together to get a score
+for usefulness. Which I realize is almost awkwardly reductive, but
+nonetheless true._____
+How can you ensure that the things you say are true and novel and
+important? Believe it or not, there is a trick for doing this. I
+learned it from my friend Robert Morris, who has a horror of saying
+anything dumb. His trick is not to say anything unless he's sure
+it's worth hearing. This makes it hard to get opinions out of him,
+but when you do, they're usually right.Translated into essay writing, what this means is that if you write
+a bad sentence, you don't publish it. You delete it and try again.
+Often you abandon whole branches of four or five paragraphs. Sometimes
+a whole essay.You can't ensure that every idea you have is good, but you can
+ensure that every one you publish is, by simply not publishing the
+ones that aren't.In the sciences, this is called publication bias, and is considered
+bad. When some hypothesis you're exploring gets inconclusive results,
+you're supposed to tell people about that too. But with essay
+writing, publication bias is the way to go.My strategy is loose, then tight. I write the first draft of an
+essay fast, trying out all kinds of ideas. Then I spend days rewriting
+it very carefully.I've never tried to count how many times I proofread essays, but
+I'm sure there are sentences I've read 100 times before publishing
+them. When I proofread an essay, there are usually passages that
+stick out in an annoying way, sometimes because they're clumsily
+written, and sometimes because I'm not sure they're true. The
+annoyance starts out unconscious, but after the tenth reading or
+so I'm saying "Ugh, that part" each time I hit it. They become like
+briars that catch your sleeve as you walk past. Usually I won't
+publish an essay till they're all gone — till I can read through
+the whole thing without the feeling of anything catching.I'll sometimes let through a sentence that seems clumsy, if I can't
+think of a way to rephrase it, but I will never knowingly let through
+one that doesn't seem correct. You never have to. If a sentence
+doesn't seem right, all you have to do is ask why it doesn't, and
+you've usually got the replacement right there in your head.This is where essayists have an advantage over journalists. You
+don't have a deadline. You can work for as long on an essay as you
+need to get it right. You don't have to publish the essay at all,
+if you can't get it right. Mistakes seem to lose courage in the
+face of an enemy with unlimited resources. Or that's what it feels
+like. What's really going on is that you have different expectations
+for yourself. You're like a parent saying to a child "we can sit
+here all night till you eat your vegetables." Except you're the
+child too.I'm not saying no mistake gets through. For example, I added condition
+(c) in "A Way to Detect Bias" 
+after readers pointed out that I'd
+omitted it. But in practice you can catch nearly all of them.There's a trick for getting importance too. It's like the trick I
+suggest to young founders for getting startup ideas: to make something
+you yourself want. You can use yourself as a proxy for the reader.
+The reader is not completely unlike you, so if you write about
+topics that seem important to you, they'll probably seem important
+to a significant number of readers as well.Importance has two factors. It's the number of people something
+matters to, times how much it matters to them. Which means of course
+that it's not a rectangle, but a sort of ragged comb, like a Riemann
+sum.The way to get novelty is to write about topics you've thought about
+a lot. Then you can use yourself as a proxy for the reader in this
+department too. Anything you notice that surprises you, who've
+thought about the topic a lot, will probably also surprise a
+significant number of readers. And here, as with correctness and
+importance, you can use the Morris technique to ensure that you
+will. If you don't learn anything from writing an essay, don't
+publish it.You need humility to measure novelty, because acknowledging the
+novelty of an idea means acknowledging your previous ignorance of
+it. Confidence and humility are often seen as opposites, but in
+this case, as in many others, confidence helps you to be humble.
+If you know you're an expert on some topic, you can freely admit
+when you learn something you didn't know, because you can be confident
+that most other people wouldn't know it either.The fourth component of useful writing, strength, comes from two
+things: thinking well, and the skillful use of qualification. These
+two counterbalance each other, like the accelerator and clutch in
+a car with a manual transmission. As you try to refine the expression
+of an idea, you adjust the qualification accordingly. Something
+you're sure of, you can state baldly with no qualification at all,
+as I did the four components of useful writing. Whereas points that
+seem dubious have to be held at arm's length with perhapses.As you refine an idea, you're pushing in the direction of less
+qualification. But you can rarely get it down to zero. Sometimes
+you don't even want to, if it's a side point and a fully refined
+version would be too long.Some say that qualifications weaken writing. For example, that you
+should never begin a sentence in an essay with "I think," because
+if you're saying it, then of course you think it. And it's true
+that "I think x" is a weaker statement than simply "x." Which is
+exactly why you need "I think." You need it to express your degree
+of certainty.But qualifications are not scalars. They're not just experimental
+error. There must be 50 things they can express: how broadly something
+applies, how you know it, how happy you are it's so, even how it
+could be falsified. I'm not going to try to explore the structure
+of qualification here. It's probably more complex than the whole
+topic of writing usefully. Instead I'll just give you a practical
+tip: Don't underestimate qualification. It's an important skill in
+its own right, not just a sort of tax you have to pay in order to
+avoid saying things that are false. So learn and use its full range.
+It may not be fully half of having good ideas, but it's part of
+having them.There's one other quality I aim for in essays: to say things as
+simply as possible. But I don't think this is a component of
+usefulness. It's more a matter of consideration for the reader. And
+it's a practical aid in getting things right; a mistake is more
+obvious when expressed in simple language. But I'll admit that the
+main reason I write simply is not for the reader's sake or because
+it helps get things right, but because it bothers me to use more
+or fancier words than I need to. It seems inelegant, like a program
+that's too long.I realize florid writing works for some people. But unless you're
+sure you're one of them, the best advice is to write as simply as
+you can._____
+I believe the formula I've given you, importance + novelty +
+correctness + strength, is the recipe for a good essay. But I should
+warn you that it's also a recipe for making people mad.The root of the problem is novelty. When you tell people something
+they didn't know, they don't always thank you for it. Sometimes the
+reason people don't know something is because they don't want to
+know it. Usually because it contradicts some cherished belief. And
+indeed, if you're looking for novel ideas, popular but mistaken
+beliefs are a good place to find them. Every popular mistaken belief
+creates a dead zone of ideas around 
+it that are relatively unexplored because they contradict it.The strength component just makes things worse. If there's anything
+that annoys people more than having their cherished assumptions
+contradicted, it's having them flatly contradicted.Plus if you've used the Morris technique, your writing will seem
+quite confident. Perhaps offensively confident, to people who
+disagree with you. The reason you'll seem confident is that you are
+confident: you've cheated, by only publishing the things you're
+sure of.  It will seem to people who try to disagree with you that
+you never admit you're wrong. In fact you constantly admit you're
+wrong. You just do it before publishing instead of after.And if your writing is as simple as possible, that just makes things
+worse. Brevity is the diction of command. If you watch someone
+delivering unwelcome news from a position of inferiority, you'll
+notice they tend to use lots of words, to soften the blow. Whereas
+to be short with someone is more or less to be rude to them.It can sometimes work to deliberately phrase statements more weakly
+than you mean. To put "perhaps" in front of something you're actually
+quite sure of. But you'll notice that when writers do this, they
+usually do it with a wink.I don't like to do this too much. It's cheesy to adopt an ironic
+tone for a whole essay. I think we just have to face the fact that
+elegance and curtness are two names for the same thing.You might think that if you work sufficiently hard to ensure that
+an essay is correct, it will be invulnerable to attack. That's sort
+of true. It will be invulnerable to valid attacks. But in practice
+that's little consolation.In fact, the strength component of useful writing will make you
+particularly vulnerable to misrepresentation. If you've stated an
+idea as strongly as you could without making it false, all anyone
+has to do is to exaggerate slightly what you said, and now it is
+false.Much of the time they're not even doing it deliberately. One of the
+most surprising things you'll discover, if you start writing essays,
+is that people who disagree with you rarely disagree with what
+you've actually written. Instead they make up something you said
+and disagree with that.For what it's worth, the countermove is to ask someone who does
+this to quote a specific sentence or passage you wrote that they
+believe is false, and explain why. I say "for what it's worth"
+because they never do. So although it might seem that this could
+get a broken discussion back on track, the truth is that it was
+never on track in the first place.Should you explicitly forestall likely misinterpretations? Yes, if
+they're misinterpretations a reasonably smart and well-intentioned
+person might make. In fact it's sometimes better to say something
+slightly misleading and then add the correction than to try to get
+an idea right in one shot. That can be more efficient, and can also
+model the way such an idea would be discovered.But I don't think you should explicitly forestall intentional
+misinterpretations in the body of an essay. An essay is a place to
+meet honest readers. You don't want to spoil your house by putting
+bars on the windows to protect against dishonest ones. The place
+to protect against intentional misinterpretations is in end-notes.
+But don't think you can predict them all. People are as ingenious
+at misrepresenting you when you say something they don't want to
+hear as they are at coming up with rationalizations for things they
+want to do but know they shouldn't. I suspect it's the same skill._____
+As with most other things, the way to get better at writing essays
+is to practice. But how do you start? Now that we've examined the
+structure of useful writing, we can rephrase that question more
+precisely. Which constraint do you relax initially? The answer is,
+the first component of importance: the number of people who care
+about what you write.If you narrow the topic sufficiently, you can probably find something
+you're an expert on. Write about that to start with. If you only
+have ten readers who care, that's fine. You're helping them, and
+you're writing. Later you can expand the breadth of topics you write
+about.The other constraint you can relax is a little surprising: publication.
+Writing essays doesn't have to mean publishing them. That may seem
+strange now that the trend is to publish every random thought, but
+it worked for me. I wrote what amounted to essays in notebooks for
+about 15 years. I never published any of them and never expected
+to. I wrote them as a way of figuring things out. But when the web
+came along I'd had a lot of practice.Incidentally, 
+Steve 
+Wozniak did the same thing. In high school he
+designed computers on paper for fun. He couldn't build them because
+he couldn't afford the components. But when Intel launched 4K DRAMs
+in 1975, he was ready._____
+How many essays are there left to write though? The answer to that
+question is probably the most exciting thing I've learned about
+essay writing. Nearly all of them are left to write.Although the essay 
+is an old form, it hasn't been assiduously
+cultivated. In the print era, publication was expensive, and there
+wasn't enough demand for essays to publish that many. You could
+publish essays if you were already well known for writing something
+else, like novels. Or you could write book reviews that you took
+over to express your own ideas. But there was not really a direct
+path to becoming an essayist. Which meant few essays got written,
+and those that did tended to be about a narrow range of subjects.Now, thanks to the internet, there's a path. Anyone can publish
+essays online. You start in obscurity, perhaps, but at least you
+can start. You don't need anyone's permission.It sometimes happens that an area of knowledge sits quietly for
+years, till some change makes it explode. Cryptography did this to
+number theory. The internet is doing it to the essay.The exciting thing is not that there's a lot left to write, but
+that there's a lot left to discover. There's a certain kind of idea
+that's best discovered by writing essays. If most essays are still
+unwritten, most such ideas are still undiscovered.Notes[1] Put railings on the balconies, but don't put bars on the windows.[2] Even now I sometimes write essays that are not meant for
+publication. I wrote several to figure out what Y Combinator should
+do, and they were really helpful.Thanks to Trevor Blackwell, Daniel Gackle, Jessica Livingston, and
+Robert Morris for reading drafts of this.

recipes/experimental/long-context/H2O/data/PaulGrahamEssays/vb.txt

@@ -0,0 +1,129 @@
+January 2016Life is short, as everyone knows. When I was a kid I used to wonder
+about this. Is life actually short, or are we really complaining
+about its finiteness?  Would we be just as likely to feel life was
+short if we lived 10 times as long?Since there didn't seem any way to answer this question, I stopped
+wondering about it.  Then I had kids.  That gave me a way to answer
+the question, and the answer is that life actually is short.Having kids showed me how to convert a continuous quantity, time,
+into discrete quantities. You only get 52 weekends with your 2 year
+old.  If Christmas-as-magic lasts from say ages 3 to 10, you only
+get to watch your child experience it 8 times.  And while it's
+impossible to say what is a lot or a little of a continuous quantity
+like time, 8 is not a lot of something.  If you had a handful of 8
+peanuts, or a shelf of 8 books to choose from, the quantity would
+definitely seem limited, no matter what your lifespan was.Ok, so life actually is short.  Does it make any difference to know
+that?It has for me.  It means arguments of the form "Life is too short
+for x" have great force.  It's not just a figure of speech to say
+that life is too short for something.  It's not just a synonym for
+annoying.  If you find yourself thinking that life is too short for
+something, you should try to eliminate it if you can.When I ask myself what I've found life is too short for, the word
+that pops into my head is "bullshit." I realize that answer is
+somewhat tautological.  It's almost the definition of bullshit that
+it's the stuff that life is too short for.  And yet bullshit does
+have a distinctive character.  There's something fake about it.
+It's the junk food of experience.
+[1]If you ask yourself what you spend your time on that's bullshit,
+you probably already know the answer.  Unnecessary meetings, pointless
+disputes, bureaucracy, posturing, dealing with other people's
+mistakes, traffic jams, addictive but unrewarding pastimes.There are two ways this kind of thing gets into your life: it's
+either forced on you, or it tricks you.  To some extent you have to
+put up with the bullshit forced on you by circumstances.  You need
+to make money, and making money consists mostly of errands.  Indeed,
+the law of supply and demand insures that: the more rewarding some
+kind of work is, the cheaper people will do it.  It may be that
+less bullshit is forced on you than you think, though.  There has
+always been a stream of people who opt out of the default grind and
+go live somewhere where opportunities are fewer in the conventional
+sense, but life feels more authentic.  This could become more common.You can do it on a smaller scale without moving.  The amount of
+time you have to spend on bullshit varies between employers.  Most
+large organizations (and many small ones) are steeped in it.  But
+if you consciously prioritize bullshit avoidance over other factors
+like money and prestige, you can probably find employers that will
+waste less of your time.If you're a freelancer or a small company, you can do this at the
+level of individual customers.  If you fire or avoid toxic customers,
+you can decrease the amount of bullshit in your life by more than
+you decrease your income.But while some amount of bullshit is inevitably forced on you, the
+bullshit that sneaks into your life by tricking you is no one's
+fault but your own.  And yet the bullshit you choose may be harder
+to eliminate than the bullshit that's forced on you.  Things that
+lure you into wasting your time have to be really good at
+tricking you.  An example that will be familiar to a lot of people
+is arguing online.  When someone
+contradicts you, they're in a sense attacking you. Sometimes pretty
+overtly.  Your instinct when attacked is to defend yourself.  But
+like a lot of instincts, this one wasn't designed for the world we
+now live in.  Counterintuitive as it feels, it's better most of
+the time not to defend yourself.  Otherwise these people are literally
+taking your life.
+[2]Arguing online is only incidentally addictive. There are more
+dangerous things than that. As I've written before, one byproduct
+of technical progress is that things we like tend to become more
+addictive.  Which means we will increasingly have to make a conscious
+effort to avoid addictions — to stand outside ourselves and ask "is
+this how I want to be spending my time?"As well as avoiding bullshit, one should actively seek out things
+that matter.  But different things matter to different people, and
+most have to learn what matters to them.  A few are lucky and realize
+early on that they love math or taking care of animals or writing,
+and then figure out a way to spend a lot of time doing it.  But
+most people start out with a life that's a mix of things that
+matter and things that don't, and only gradually learn to distinguish
+between them.For the young especially, much of this confusion is induced by the
+artificial situations they find themselves in. In middle school and
+high school, what the other kids think of you seems the most important
+thing in the world.  But when you ask adults what they got wrong
+at that age, nearly all say they cared too much what other kids
+thought of them.One heuristic for distinguishing stuff that matters is to ask
+yourself whether you'll care about it in the future.  Fake stuff
+that matters usually has a sharp peak of seeming to matter.  That's
+how it tricks you.  The area under the curve is small, but its shape
+jabs into your consciousness like a pin.The things that matter aren't necessarily the ones people would
+call "important."  Having coffee with a friend matters.  You won't
+feel later like that was a waste of time.One great thing about having small children is that they make you
+spend time on things that matter: them. They grab your sleeve as
+you're staring at your phone and say "will you play with me?" And
+odds are that is in fact the bullshit-minimizing option.If life is short, we should expect its shortness to take us by
+surprise. And that is just what tends to happen.  You take things
+for granted, and then they're gone.  You think you can always write
+that book, or climb that mountain, or whatever, and then you realize
+the window has closed.  The saddest windows close when other people
+die. Their lives are short too.  After my mother died, I wished I'd
+spent more time with her.  I lived as if she'd always be there.
+And in her typical quiet way she encouraged that illusion.  But an
+illusion it was. I think a lot of people make the same mistake I
+did.The usual way to avoid being taken by surprise by something is to
+be consciously aware of it.  Back when life was more precarious,
+people used to be aware of death to a degree that would now seem a
+bit morbid.  I'm not sure why, but it doesn't seem the right answer
+to be constantly reminding oneself of the grim reaper hovering at
+everyone's shoulder.  Perhaps a better solution is to look at the
+problem from the other end. Cultivate a habit of impatience about
+the things you most want to do. Don't wait before climbing that
+mountain or writing that book or visiting your mother.  You don't
+need to be constantly reminding yourself why you shouldn't wait.
+Just don't wait.I can think of two more things one does when one doesn't have much
+of something: try to get more of it, and savor what one has.  Both
+make sense here.How you live affects how long you live.  Most people could do better.
+Me among them.But you can probably get even more effect by paying closer attention
+to the time you have.  It's easy to let the days rush by.  The
+"flow" that imaginative people love so much has a darker cousin
+that prevents you from pausing to savor life amid the daily slurry
+of errands and alarms.  One of the most striking things I've read
+was not in a book, but the title of one: James Salter's Burning
+the Days.It is possible to slow time somewhat. I've gotten better at it.
+Kids help.  When you have small children, there are a lot of moments
+so perfect that you can't help noticing.It does help too to feel that you've squeezed everything out of
+some experience.  The reason I'm sad about my mother is not just
+that I miss her but that I think of all the things we could have
+done that we didn't.  My oldest son will be 7 soon.  And while I
+miss the 3 year old version of him, I at least don't have any regrets
+over what might have been.  We had the best time a daddy and a 3
+year old ever had.Relentlessly prune bullshit, don't wait to do things that matter,
+and savor the time you have.  That's what you do when life is short.Notes[1]
+At first I didn't like it that the word that came to mind was
+one that had other meanings.  But then I realized the other meanings
+are fairly closely related.  Bullshit in the sense of things you
+waste your time on is a lot like intellectual bullshit.[2]
+I chose this example deliberately as a note to self.  I get
+attacked a lot online.  People tell the craziest lies about me.
+And I have so far done a pretty mediocre job of suppressing the
+natural human inclination to say "Hey, that's not true!"Thanks to Jessica Livingston and Geoff Ralston for reading drafts
+of this.

recipes/experimental/long-context/H2O/data/PaulGrahamEssays/vcsqueeze.txt

@@ -0,0 +1,124 @@
+November 2005In the next few years, venture capital funds will find themselves
+squeezed from four directions.  They're already stuck with a seller's
+market, because of the huge amounts they raised at the end of the
+Bubble and still haven't invested.  This by itself is not the end
+of the world.  In fact, it's just a more extreme version of the
+norm
+in the VC business: too much money chasing too few deals.Unfortunately, those few deals now want less and less money, because
+it's getting so cheap to start a startup.  The four causes: open
+source, which makes software free; Moore's law, which makes hardware
+geometrically closer to free; the Web, which makes promotion free
+if you're good; and better languages, which make development a lot
+cheaper.When we started our startup in 1995, the first three were our biggest
+expenses.  We had to pay $5000 for the Netscape Commerce Server,
+the only software that then supported secure http connections.  We
+paid $3000 for a server with a 90 MHz processor and 32 meg of
+memory.  And we paid a PR firm about $30,000 to promote our launch.Now you could get all three for nothing.  You can get the software
+for free; people throw away computers more powerful than our first
+server; and if you make something good you can generate ten times
+as much traffic by word of mouth online than our first PR firm got
+through the print media.And of course another big change for the average startup is that
+programming languages have improved-- or rather, the median language has.  At most startups ten years
+ago, software development meant ten programmers writing code in
+C++.  Now the same work might be done by one or two using Python
+or Ruby.During the Bubble, a lot of people predicted that startups would
+outsource their development to India.  I think a better model for
+the future is David Heinemeier Hansson, who outsourced his development
+to a more powerful language instead.  A lot of well-known applications
+are now, like BaseCamp, written by just one programmer.  And one
+guy is more than 10x cheaper than ten, because (a) he won't waste
+any time in meetings, and (b) since he's probably a founder, he can
+pay himself nothing.Because starting a startup is so cheap, venture capitalists now
+often want to give startups more money than the startups want to
+take.  VCs like to invest several million at a time.  But as one
+VC told me after a startup he funded would only take about half a
+million, "I don't know what we're going to do.  Maybe we'll just
+have to give some of it back." Meaning give some of the fund back
+to the institutional investors who supplied it, because it wasn't
+going to be possible to invest it all.Into this already bad situation comes the third problem: Sarbanes-Oxley.
+Sarbanes-Oxley is a law, passed after the Bubble, that drastically
+increases the regulatory burden on public companies. And in addition
+to the cost of compliance, which is at least two million dollars a
+year, the law introduces frightening legal exposure for corporate
+officers.  An experienced CFO I know said flatly: "I would not
+want to be CFO of a public company now."You might think that responsible corporate governance is an area
+where you can't go too far.  But you can go too far in any law, and
+this remark convinced me that Sarbanes-Oxley must have.  This CFO
+is both the smartest and the most upstanding money guy I know.  If
+Sarbanes-Oxley deters people like him from being CFOs of public  
+companies, that's proof enough that it's broken.Largely because of Sarbanes-Oxley, few startups go public now.  For
+all practical purposes, succeeding now equals getting bought.  Which
+means VCs are now in the business of finding promising little 2-3
+man startups and pumping them up into companies that cost $100
+million to acquire.   They didn't mean to be in this business; it's
+just what their business has evolved into.Hence the fourth problem: the acquirers have begun to realize they
+can buy wholesale.  Why should they wait for VCs to make the startups
+they want more expensive?  Most of what the VCs add, acquirers don't
+want anyway.  The acquirers already have brand recognition and HR
+departments.  What they really want is the software and the developers,
+and that's what the startup is in the early phase: concentrated
+software and developers.Google, typically, seems to have been the first to figure this out.
+"Bring us your startups early," said Google's speaker at the Startup School.  They're quite
+explicit about it: they like to acquire startups at just the point
+where they would do a Series A round.  (The Series A round is the
+first round of real VC funding; it usually happens in the first
+year.) It is a brilliant strategy, and one that other big technology
+companies will no doubt try to duplicate.  Unless they want to have 
+still more of their lunch eaten by Google.Of course, Google has an advantage in buying startups: a lot of the
+people there are rich, or expect to be when their options vest.
+Ordinary employees find it very hard to recommend an acquisition;
+it's just too annoying to see a bunch of twenty year olds get rich
+when you're still working for salary.  Even if it's the right thing   
+for your company to do.The Solution(s)Bad as things look now, there is a way for VCs to save themselves.
+They need to do two things, one of which won't surprise them, and  
+another that will seem an anathema.Let's start with the obvious one: lobby to get Sarbanes-Oxley  
+loosened.  This law was created to prevent future Enrons, not to
+destroy the IPO market.  Since the IPO market was practically dead
+when it passed, few saw what bad effects it would have.  But now 
+that technology has recovered from the last bust, we can see clearly
+what a bottleneck Sarbanes-Oxley has become.Startups are fragile plants—seedlings, in fact.  These seedlings
+are worth protecting, because they grow into the trees of the
+economy.  Much of the economy's growth is their growth.  I think
+most politicians realize that.  But they don't realize just how   
+fragile startups are, and how easily they can become collateral
+damage of laws meant to fix some other problem.Still more dangerously, when you destroy startups, they make very
+little noise.  If you step on the toes of the coal industry, you'll
+hear about it.  But if you inadvertantly squash the startup industry,
+all that happens is that the founders of the next Google stay in 
+grad school instead of starting a company.My second suggestion will seem shocking to VCs: let founders cash  
+out partially in the Series A round.  At the moment, when VCs invest
+in a startup, all the stock they get is newly issued and all the 
+money goes to the company.  They could buy some stock directly from
+the founders as well.Most VCs have an almost religious rule against doing this.  They
+don't want founders to get a penny till the company is sold or goes
+public.  VCs are obsessed with control, and they worry that they'll
+have less leverage over the founders if the founders have any money.This is a dumb plan.  In fact, letting the founders sell a little stock
+early would generally be better for the company, because it would
+cause the founders' attitudes toward risk to be aligned with the
+VCs'.  As things currently work, their attitudes toward risk tend
+to be diametrically opposed: the founders, who have nothing, would
+prefer a 100% chance of $1 million to a 20% chance of $10 million,
+while the VCs can afford to be "rational" and prefer the latter.Whatever they say, the reason founders are selling their companies
+early instead of doing Series A rounds is that they get paid up
+front.  That first million is just worth so much more than the
+subsequent ones.  If founders could sell a little stock early,
+they'd be happy to take VC money and bet the rest on a bigger
+outcome.So why not let the founders have that first million, or at least
+half million?  The VCs would get same number of shares for the   
+money.  So what if some of the money would go to the  
+founders instead of the company?Some VCs will say this is
+unthinkable—that they want all their money to be put to work
+growing the company.  But the fact is, the huge size of current VC
+investments is dictated by the structure
+of VC funds, not the needs of startups.  Often as not these large  
+investments go to work destroying the company rather than growing
+it.The angel investors who funded our startup let the founders sell
+some stock directly to them, and it was a good deal for everyone. 
+The angels made a huge return on that investment, so they're happy.
+And for us founders it blunted the terrifying all-or-nothingness
+of a startup, which in its raw form is more a distraction than a
+motivator.If VCs are frightened at the idea of letting founders partially
+cash out, let me tell them something still more frightening: you
+are now competing directly with Google.
+Thanks to Trevor Blackwell, Sarah Harlin, Jessica
+Livingston, and Robert Morris for reading drafts of this.

recipes/experimental/long-context/H2O/data/PaulGrahamEssays/vw.txt

@@ -0,0 +1,75 @@
+January 2012A few hours before the Yahoo acquisition was announced in June 1998
+I took a snapshot of Viaweb's
+site.  I thought it might be interesting to look at one day.The first thing one notices is is how tiny the pages are.  Screens
+were a lot smaller in 1998.  If I remember correctly, our frontpage
+used to just fit in the size window people typically used then.Browsers then (IE 6 was still 3 years in the future) had few fonts
+and they weren't antialiased.  If you wanted to make pages that
+looked good, you had to render display text as images.You may notice a certain similarity between the Viaweb and Y Combinator logos.  We did that
+as an inside joke when we started YC.  Considering how basic a red
+circle is, it seemed surprising to me when we started Viaweb how
+few other companies used one as their logo.  A bit later I realized
+why.On the Company
+page you'll notice a mysterious individual called John McArtyem.
+Robert Morris (aka Rtm) was so publicity averse after the 
+Worm that he
+didn't want his name on the site.  I managed to get him to agree
+to a compromise: we could use his bio but not his name.  He has
+since relaxed a bit
+on that point.Trevor graduated at about the same time the acquisition closed, so in the
+course of 4 days he went from impecunious grad student to millionaire
+PhD.  The culmination of my career as a writer of press releases
+was one celebrating
+his graduation, illustrated with a drawing I did of him during
+a meeting.(Trevor also appears as Trevino
+Bagwell in our directory of web designers merchants could hire
+to build stores for them.  We inserted him as a ringer in case some
+competitor tried to spam our web designers.   We assumed his logo
+would deter any actual customers, but it did not.)Back in the 90s, to get users you had to get mentioned in magazines
+and newspapers.  There were not the same ways to get found online
+that there are today.  So we used to pay a PR
+firm $16,000 a month to get us mentioned in the press.  Fortunately
+reporters liked
+us.In our advice about
+getting traffic from search engines (I don't think the term SEO
+had been coined yet), we say there are only 7 that matter: Yahoo,
+AltaVista, Excite, WebCrawler, InfoSeek, Lycos, and HotBot.  Notice
+anything missing?  Google was incorporated that September.We supported online transactions via a company called 
+Cybercash,
+since if we lacked that feature we'd have gotten beaten up in product
+comparisons.  But Cybercash was so bad and most stores' order volumes
+were so low that it was better if merchants processed orders like phone orders.  We had a page in our site trying to talk merchants
+out of doing real time authorizations.The whole site was organized like a funnel, directing people to the
+test drive.
+It was a novel thing to be able to try out software online.  We put
+cgi-bin in our dynamic urls to fool competitors about how our
+software worked.We had some well
+known users.  Needless to say, Frederick's of Hollywood got the
+most traffic.  We charged a flat fee of $300/month for big stores,
+so it was a little alarming to have users who got lots of traffic.
+I once calculated how much Frederick's was costing us in bandwidth,
+and it was about $300/month.Since we hosted all the stores, which together were getting just
+over 10 million page views per month in June 1998, we consumed what
+at the time seemed a lot of bandwidth.  We had 2 T1s (3 Mb/sec)
+coming into our offices.  In those days there was no AWS.  Even
+colocating servers seemed too risky, considering how often things
+went wrong with them.  So we had our servers in our offices.  Or
+more precisely, in Trevor's office.  In return for the unique
+privilege of sharing his office with no other humans, he had to
+share it with 6 shrieking tower servers.  His office was nicknamed
+the Hot Tub on account of the heat they generated.  Most days his
+stack of window air conditioners could keep up.For describing pages, we had a template language called RTML, which
+supposedly stood for something, but which in fact I named after
+Rtm.  RTML was Common Lisp augmented by some macros and libraries,
+and concealed under a structure editor that made it look like it
+had syntax.Since we did continuous releases, our software didn't actually have
+versions.  But in those days the trade press expected versions, so
+we made them up.  If we wanted to get lots of attention, we made
+the version number an
+integer.  That "version 4.0" icon was generated by our own
+button generator, incidentally.  The whole Viaweb site was made
+with our software, even though it wasn't an online store, because
+we wanted to experience what our users did.At the end of 1997, we released a general purpose shopping search
+engine called Shopfind.  It
+was pretty advanced for the time.  It had a programmable crawler
+that could crawl most of the different stores online and pick out
+the products.

recipes/experimental/long-context/H2O/data/PaulGrahamEssays/want.txt

@@ -0,0 +1,43 @@
+November 2022Since I was about 9 I've been puzzled by the apparent contradiction
+between being made of matter that behaves in a predictable way, and
+the feeling that I could choose to do whatever I wanted. At the
+time I had a self-interested motive for exploring the question. At
+that age (like most succeeding ages) I was always in trouble with
+the authorities, and it seemed to me that there might possibly be
+some way to get out of trouble by arguing that I wasn't responsible
+for my actions. I gradually lost hope of that, but the puzzle
+remained: How do you reconcile being a machine made of matter with
+the feeling that you're free to choose what you do?
+[1]The best way to explain the answer may be to start with a slightly
+wrong version, and then fix it. The wrong version is: You can do
+what you want, but you can't want what you want. Yes, you can control
+what you do, but you'll do what you want, and you can't control
+that.The reason this is mistaken is that people do sometimes change what
+they want. People who don't want to want something — drug addicts,
+for example — can sometimes make themselves stop wanting it. And
+people who want to want something — who want to like classical
+music, or broccoli — sometimes succeed.So we modify our initial statement: You can do what you want, but
+you can't want to want what you want.That's still not quite true. It's possible to change what you want
+to want. I can imagine someone saying "I decided to stop wanting
+to like classical music." But we're getting closer to the truth.
+It's rare for people to change what they want to want, and the more
+"want to"s we add, the rarer it gets.We can get arbitrarily close to a true statement by adding more "want
+to"s in much the same way we can get arbitrarily close to 1 by adding
+more 9s to a string of 9s following a decimal point. In practice
+three or four "want to"s must surely be enough. It's hard even to
+envision what it would mean to change what you want to want to want
+to want, let alone actually do it.So one way to express the correct answer is to use a regular
+expression. You can do what you want, but there's some statement
+of the form "you can't (want to)* want what you want" that's true.
+Ultimately you get back to a want that you don't control.
+[2]
+Notes[1]
+I didn't know when I was 9 that matter might behave randomly,
+but I don't think it affects the problem much. Randomness destroys
+the ghost in the machine as effectively as determinism.[2]
+If you don't like using an expression, you can make the same
+point using higher-order desires: There is some n such that you
+don't control your nth-order desires.
+Thanks to Trevor Blackwell,
+Jessica Livingston, Robert Morris, and
+Michael Nielsen for reading drafts of this.

recipes/experimental/long-context/H2O/data/PaulGrahamEssays/web20.txt

@@ -0,0 +1,299 @@
+
+
+Want to start a startup?  Get funded by
+Y Combinator.
+
+
+
+
+November 2005Does "Web 2.0" mean anything?  Till recently I thought it didn't,
+but the truth turns out to be more complicated.  Originally, yes,
+it was meaningless.  Now it seems to have acquired a meaning.  And
+yet those who dislike the term are probably right, because if it
+means what I think it does, we don't need it.I first heard the phrase "Web 2.0" in the name of the Web 2.0
+conference in 2004.  At the time it was supposed to mean using "the
+web as a platform," which I took to refer to web-based applications.
+[1]So I was surprised at a conference this summer when Tim O'Reilly
+led a session intended to figure out a definition of "Web 2.0."
+Didn't it already mean using the web as a platform?  And if it
+didn't already mean something, why did we need the phrase at all?OriginsTim says the phrase "Web 2.0" first
+arose in "a brainstorming session between
+O'Reilly and Medialive International." What is Medialive International?
+"Producers of technology tradeshows and conferences," according to
+their site.  So presumably that's what this brainstorming session
+was about.  O'Reilly wanted to organize a conference about the web,
+and they were wondering what to call it.I don't think there was any deliberate plan to suggest there was a
+new version of the web.  They just wanted to make the point
+that the web mattered again.  It was a kind of semantic deficit
+spending: they knew new things were coming, and the "2.0" referred
+to whatever those might turn out to be.And they were right.  New things were coming.  But the new version
+number led to some awkwardness in the short term.  In the process
+of developing the pitch for the first conference, someone must have
+decided they'd better take a stab at explaining what that "2.0"
+referred to.  Whatever it meant, "the web as a platform" was at
+least not too constricting.The story about "Web 2.0" meaning the web as a platform didn't live
+much past the first conference.  By the second conference, what
+"Web 2.0" seemed to mean was something about democracy.  At least,
+it did when people wrote about it online.  The conference itself
+didn't seem very grassroots.  It cost $2800, so the only people who
+could afford to go were VCs and people from big companies.And yet, oddly enough, Ryan Singel's article
+about the conference in Wired News spoke of "throngs of
+geeks."  When a friend of mine asked Ryan about this, it was news
+to him.  He said he'd originally written something like "throngs
+of VCs and biz dev guys" but had later shortened it just to "throngs,"
+and that this must have in turn been expanded by the editors into
+"throngs of geeks."  After all, a Web 2.0 conference would presumably
+be full of geeks, right?Well, no.  There were about 7.  Even Tim O'Reilly was wearing a   
+suit, a sight so alien I couldn't parse it at first.  I saw
+him walk by and said to one of the O'Reilly people "that guy looks
+just like Tim.""Oh, that's Tim.  He bought a suit."
+I ran after him, and sure enough, it was.  He explained that he'd
+just bought it in Thailand.The 2005 Web 2.0 conference reminded me of Internet trade shows
+during the Bubble, full of prowling VCs looking for the next hot
+startup.  There was that same odd atmosphere created by a large  
+number of people determined not to miss out.  Miss out on what?
+They didn't know.  Whatever was going to happen—whatever Web 2.0
+turned out to be.I wouldn't quite call it "Bubble 2.0" just because VCs are eager
+to invest again.  The Internet is a genuinely big deal.  The bust
+was as much an overreaction as
+the boom.  It's to be expected that once we started to pull out of
+the bust, there would be a lot of growth in this area, just as there
+was in the industries that spiked the sharpest before the Depression.The reason this won't turn into a second Bubble is that the IPO
+market is gone.  Venture investors
+are driven by exit strategies.  The reason they were funding all  
+those laughable startups during the late 90s was that they hoped
+to sell them to gullible retail investors; they hoped to be laughing
+all the way to the bank.  Now that route is closed.  Now the default
+exit strategy is to get bought, and acquirers are less prone to
+irrational exuberance than IPO investors.  The closest you'll get 
+to Bubble valuations is Rupert Murdoch paying $580 million for   
+Myspace.  That's only off by a factor of 10 or so.1. AjaxDoes "Web 2.0" mean anything more than the name of a conference
+yet?  I don't like to admit it, but it's starting to.  When people
+say "Web 2.0" now, I have some idea what they mean.  And the fact
+that I both despise the phrase and understand it is the surest proof
+that it has started to mean something.One ingredient of its meaning is certainly Ajax, which I can still
+only just bear to use without scare quotes.  Basically, what "Ajax"
+means is "Javascript now works."  And that in turn means that
+web-based applications can now be made to work much more like desktop
+ones.As you read this, a whole new generation
+of software is being written to take advantage of Ajax.  There
+hasn't been such a wave of new applications since microcomputers
+first appeared.  Even Microsoft sees it, but it's too late for them
+to do anything more than leak "internal"  
+documents designed to give the impression they're on top of this
+new trend.In fact the new generation of software is being written way too
+fast for Microsoft even to channel it, let alone write their own
+in house.  Their only hope now is to buy all the best Ajax startups
+before Google does.  And even that's going to be hard, because
+Google has as big a head start in buying microstartups as it did
+in search a few years ago.  After all, Google Maps, the canonical
+Ajax application, was the result of a startup they bought.So ironically the original description of the Web 2.0 conference
+turned out to be partially right: web-based applications are a big
+component of Web 2.0.  But I'm convinced they got this right by 
+accident.  The Ajax boom didn't start till early 2005, when Google
+Maps appeared and the term "Ajax" was coined.2. DemocracyThe second big element of Web 2.0 is democracy.  We now have several
+examples to prove that amateurs can   
+surpass professionals, when they have the right kind of system to 
+channel their efforts.  Wikipedia
+may be the most famous.  Experts have given Wikipedia middling
+reviews, but they miss the critical point: it's good enough.  And   
+it's free, which means people actually read it.  On the web, articles
+you have to pay for might as well not exist.  Even if you were    
+willing to pay to read them yourself, you can't link to them.    
+They're not part of the conversation.Another place democracy seems to win is in deciding what counts as
+news.  I never look at any news site now except Reddit.
+[2]
+ I know if something major
+happens, or someone writes a particularly interesting article, it   
+will show up there.  Why bother checking the front page of any
+specific paper or magazine?  Reddit's like an RSS feed for the whole
+web, with a filter for quality.  Similar sites include Digg, a technology news site that's
+rapidly approaching Slashdot in popularity, and del.icio.us, the collaborative
+bookmarking network that set off the "tagging" movement.  And whereas
+Wikipedia's main appeal is that it's good enough and free, these
+sites suggest that voters do a significantly better job than human
+editors.The most dramatic example of Web 2.0 democracy is not in the selection
+of ideas, but their production.  
+I've noticed for a while that the stuff I read on individual people's
+sites is as good as or better than the stuff I read in newspapers
+and magazines.  And now I have independent evidence: the top links
+on Reddit are generally links to individual people's sites rather  
+than to magazine articles or news stories.My experience of writing
+for magazines suggests an explanation.  Editors.  They control the
+topics you can write about, and they can generally rewrite whatever
+you produce.  The result is to damp extremes.  Editing yields 95th
+percentile writing—95% of articles are improved by it, but 5% are
+dragged down.  5% of the time you get "throngs of geeks."On the web, people can publish whatever they want.  Nearly all of
+it falls short of the editor-damped writing in print publications.
+But the pool of writers is very, very large.  If it's large enough,
+the lack of damping means the best writing online should surpass  
+the best in print.
+[3]  
+And now that the web has evolved mechanisms
+for selecting good stuff, the web wins net.  Selection beats damping,
+for the same reason market economies beat centrally planned ones.Even the startups are different this time around.  They are to the  
+startups of the Bubble what bloggers are to the print media.  During
+the Bubble, a startup meant a company headed by an MBA that was   
+blowing through several million dollars of VC money to "get big
+fast" in the most literal sense.  Now it means a smaller, younger, more technical group that just      
+decided to make something great.  They'll decide later if they want  
+to raise VC-scale funding, and if they take it, they'll take it on
+their terms.3. Don't Maltreat UsersI think everyone would agree that democracy and Ajax are elements
+of "Web 2.0."  I also see a third: not to maltreat users.  During
+the Bubble a lot of popular sites were quite high-handed with users.
+And not just in obvious ways, like making them register, or subjecting
+them to annoying ads.  The very design of the average site in the   
+late 90s was an abuse.  Many of the most popular sites were loaded
+with obtrusive branding that made them slow to load and sent the
+user the message: this is our site, not yours.  (There's a physical
+analog in the Intel and Microsoft stickers that come on some
+laptops.)I think the root of the problem was that sites felt they were giving
+something away for free, and till recently a company giving anything
+away for free could be pretty high-handed about it.  Sometimes it
+reached the point of economic sadism: site owners assumed that the
+more pain they caused the user, the more benefit it must be to them.  
+The most dramatic remnant of this model may be at salon.com, where   
+you can read the beginning of a story, but to get the rest you have
+sit through a movie.At Y Combinator we advise all the startups we fund never to lord
+it over users.  Never make users register, unless you need to in
+order to store something for them.  If you do make users register,   
+never make them wait for a confirmation link in an email; in fact,
+don't even ask for their email address unless you need it for some
+reason.  Don't ask them any unnecessary questions.  Never send them
+email unless they explicitly ask for it.  Never frame pages you
+link to, or open them in new windows.  If you have a free version 
+and a pay version, don't make the free version too restricted.  And
+if you find yourself asking "should we allow users to do x?" just 
+answer "yes" whenever you're unsure.  Err on the side of generosity.In How to Start a Startup I advised startups
+never to let anyone fly under them, meaning never to let any other
+company offer a cheaper, easier solution.  Another way to fly low 
+is to give users more power.  Let users do what they want.  If you 
+don't and a competitor does, you're in trouble.iTunes is Web 2.0ish in this sense.  Finally you can buy individual
+songs instead of having to buy whole albums.  The recording industry
+hated the idea and resisted it as long as possible.  But it was
+obvious what users wanted, so Apple flew under the labels.
+[4]
+Though really it might be better to describe iTunes as Web 1.5.     
+Web 2.0 applied to music would probably mean individual bands giving
+away DRMless songs for free.The ultimate way to be nice to users is to give them something for
+free that competitors charge for.  During the 90s a lot of people   
+probably thought we'd have some working system for micropayments     
+by now.  In fact things have gone in the other direction.  The most   
+successful sites are the ones that figure out new ways to give stuff
+away for free.  Craigslist has largely destroyed the classified ad
+sites of the 90s, and OkCupid looks likely to do the same to the
+previous generation of dating sites.Serving web pages is very, very cheap.  If you can make even a   
+fraction of a cent per page view, you can make a profit.  And
+technology for targeting ads continues to improve.  I wouldn't be
+surprised if ten years from now eBay had been supplanted by an      
+ad-supported freeBay (or, more likely, gBay).Odd as it might sound, we tell startups that they should try to
+make as little money as possible.  If you can figure out a way to
+turn a billion dollar industry into a fifty million dollar industry,
+so much the better, if all fifty million go to you.  Though indeed,
+making things cheaper often turns out to generate more money in the
+end, just as automating things often turns out to generate more
+jobs.The ultimate target is Microsoft.  What a bang that balloon is going
+to make when someone pops it by offering a free web-based alternative 
+to MS Office.
+[5]
+Who will?  Google?  They seem to be taking their
+time.  I suspect the pin will be wielded by a couple of 20 year old
+hackers who are too naive to be intimidated by the idea.  (How hard
+can it be?)The Common ThreadAjax, democracy, and not dissing users.  What do they all have in  
+common?  I didn't realize they had anything in common till recently,
+which is one of the reasons I disliked the term "Web 2.0" so much.
+It seemed that it was being used as a label for whatever happened
+to be new—that it didn't predict anything.But there is a common thread.  Web 2.0 means using the web the way
+it's meant to be used.  The "trends" we're seeing now are simply
+the inherent nature of the web emerging from under the broken models
+that got imposed on it during the Bubble.I realized this when I read an  interview with
+Joe Kraus, the co-founder of Excite.
+[6]
+
+  Excite really never got the business model right at all.  We fell 
+  into the classic problem of how when a new medium comes out it
+  adopts the practices, the content, the business models of the old
+  medium—which fails, and then the more appropriate models get
+  figured out.
+
+It may have seemed as if not much was happening during the years
+after the Bubble burst.  But in retrospect, something was happening:
+the web was finding its natural angle of repose.  The democracy 
+component, for example—that's not an innovation, in the sense of
+something someone made happen.  That's what the web naturally tends
+to produce.Ditto for the idea of delivering desktop-like applications over the
+web.  That idea is almost as old as the web.  But the first time    
+around it was co-opted by Sun, and we got Java applets.  Java has
+since been remade into a generic replacement for C++, but in 1996
+the story about Java was that it represented a new model of software.
+Instead of desktop applications, you'd run Java "applets" delivered
+from a server.This plan collapsed under its own weight. Microsoft helped kill it,
+but it would have died anyway.  There was no uptake among hackers.
+When you find PR firms promoting
+something as the next development platform, you can be sure it's
+not.  If it were, you wouldn't need PR firms to tell you, because   
+hackers would already be writing stuff on top of it, the way sites    
+like Busmonster used Google Maps as a
+platform before Google even meant it to be one.The proof that Ajax is the next hot platform is that thousands of  
+hackers have spontaneously started building things on top
+of it.  Mikey likes it.There's another thing all three components of Web 2.0 have in common.
+Here's a clue.  Suppose you approached investors with the following
+idea for a Web 2.0 startup:
+
+  Sites like del.icio.us and flickr allow users to "tag" content
+  with descriptive tokens.  But there is also huge source of
+  implicit tags that they ignore: the text within web links.
+  Moreover, these links represent a social network connecting the   
+  individuals and organizations who created the pages, and by using
+  graph theory we can compute from this network an estimate of the
+  reputation of each member.  We plan to mine the web for these 
+  implicit tags, and use them together with the reputation hierarchy
+  they embody to enhance web searches.
+
+How long do you think it would take them on average to realize that
+it was a description of Google?Google was a pioneer in all three components of Web 2.0: their core
+business sounds crushingly hip when described in Web 2.0 terms, 
+"Don't maltreat users" is a subset of "Don't be evil," and of course
+Google set off the whole Ajax boom with Google Maps.Web 2.0 means using the web as it was meant to be used, and Google
+does.  That's their secret.    They're sailing with the wind, instead of sitting  
+becalmed praying for a business model, like the print media, or   
+trying to tack upwind by suing their customers, like Microsoft and 
+the record labels.
+[7]Google doesn't try to force things to happen their way.  They try   
+to figure out what's going to happen, and arrange to be standing 
+there when it does.  That's the way to approach technology—and 
+as business includes an ever larger technological component, the
+right way to do business.The fact that Google is a "Web 2.0" company shows that, while
+meaningful, the term is also rather bogus.  It's like the word
+"allopathic."  It just means doing things right, and it's a bad   
+sign when you have a special word for that.
+Notes[1]
+From the conference
+site, June 2004: "While the first wave of the Web was closely  
+tied to the browser, the second wave extends applications across    
+the web and enables a new generation of services and business
+opportunities."  To the extent this means anything, it seems to be
+about 
+web-based applications.[2]
+Disclosure: Reddit was funded by 
+Y Combinator.  But although
+I started using it out of loyalty to the home team, I've become a
+genuine addict.  While we're at it, I'm also an investor in
+!MSFT, having sold all my shares earlier this year.[3]
+I'm not against editing. I spend more time editing than
+writing, and I have a group of picky friends who proofread almost
+everything I write.  What I dislike is editing done after the fact  
+by someone else.[4]
+Obvious is an understatement.  Users had been climbing in through  
+the window for years before Apple finally moved the door.[5]
+Hint: the way to create a web-based alternative to Office may
+not be to write every component yourself, but to establish a protocol
+for web-based apps to share a virtual home directory spread across
+multiple servers.  Or it may be to write it all yourself.[6]
+In Jessica Livingston's
+Founders at
+Work.[7]
+Microsoft didn't sue their customers directly, but they seem 
+to have done all they could to help SCO sue them.Thanks to Trevor Blackwell, Sarah Harlin, Jessica Livingston, Peter
+Norvig, Aaron Swartz, and Jeff Weiner for reading drafts of this, and to the
+guys at O'Reilly and Adaptive Path for answering my questions.

recipes/experimental/long-context/H2O/data/PaulGrahamEssays/weird.txt

@@ -0,0 +1,30 @@
+August 2021When people say that in their experience all programming languages
+are basically equivalent, they're making a statement not about
+languages but about the kind of programming they've done.99.5% of programming consists of gluing together calls to library
+functions. All popular languages are equally good at this. So one
+can easily spend one's whole career operating in the intersection
+of popular programming languages.But the other .5% of programming is disproportionately interesting.
+If you want to learn what it consists of, the weirdness of weird
+languages is a good clue to follow.Weird languages aren't weird by accident. Not the good ones, at
+least. The weirdness of the good ones usually implies the existence
+of some form of programming that's not just the usual gluing together
+of library calls.A concrete example: Lisp macros. Lisp macros seem weird even to
+many Lisp programmers. They're not only not in the intersection of
+popular languages, but by their nature would be hard to implement
+properly in a language without turning it into a dialect of
+Lisp. And macros are definitely evidence of techniques that go
+beyond glue programming. For example, solving problems by first
+writing a language for problems of that type, and then writing
+your specific application in it. Nor is this all you can do with
+macros; it's just one region in a space of program-manipulating
+techniques that even now is far from fully explored.So if you want to expand your concept of what programming can be,
+one way to do it is by learning weird languages. Pick a language
+that most programmers consider weird but whose median user is smart,
+and then focus on the differences between this language and the
+intersection of popular languages. What can you say in this language
+that would be impossibly inconvenient to say in others? In the
+process of learning how to say things you couldn't previously say,
+you'll probably be learning how to think things you couldn't
+previously think.
+Thanks to Trevor Blackwell, Patrick Collison, Daniel Gackle, Amjad
+Masad, and Robert Morris for reading drafts of this.

recipes/experimental/long-context/H2O/data/PaulGrahamEssays/wisdom.txt

@@ -0,0 +1,320 @@
+February 2007A few days ago I finally figured out something I've wondered about
+for 25 years: the relationship between wisdom and intelligence.
+Anyone can see they're not the same by the number of people who are
+smart, but not very wise.  And yet intelligence and wisdom do seem
+related.  How?What is wisdom?  I'd say it's knowing what to do in a lot of
+situations.  I'm not trying to make a deep point here about the
+true nature of wisdom, just to figure out how we use the word.  A
+wise person is someone who usually knows the right thing to do.And yet isn't being smart also knowing what to do in certain
+situations?  For example, knowing what to do when the teacher tells
+your elementary school class to add all the numbers from 1 to 100?
+[1]Some say wisdom and intelligence apply to different types of
+problems—wisdom to human problems and intelligence to abstract
+ones.  But that isn't true.  Some wisdom has nothing to do with
+people: for example, the wisdom of the engineer who knows certain
+structures are less prone to failure than others.  And certainly
+smart people can find clever solutions to human problems as well
+as abstract ones. 
+[2]Another popular explanation is that wisdom comes from experience
+while intelligence is innate.  But people are not simply wise in
+proportion to how much experience they have.  Other things must
+contribute to wisdom besides experience, and some may be innate: a
+reflective disposition, for example.Neither of the conventional explanations of the difference between
+wisdom and intelligence stands up to scrutiny.  So what is the
+difference?  If we look at how people use the words "wise" and
+"smart," what they seem to mean is different shapes of performance.Curve"Wise" and "smart" are both ways of saying someone knows what to
+do.  The difference is that "wise" means one has a high average
+outcome across all situations, and "smart" means one does spectacularly
+well in a few.  That is, if you had a graph in which the x axis
+represented situations and the y axis the outcome, the graph of the
+wise person would be high overall, and the graph of the smart person
+would have high peaks.The distinction is similar to the rule that one should judge talent
+at its best and character at its worst.  Except you judge intelligence
+at its best, and wisdom by its average.  That's how the two are
+related: they're the two different senses in which the same curve
+can be high.So a wise person knows what to do in most situations, while a smart
+person knows what to do in situations where few others could.  We
+need to add one more qualification: we should ignore cases where
+someone knows what to do because they have inside information. 
+[3]
+But aside from that, I don't think we can get much more specific
+without starting to be mistaken.Nor do we need to.  Simple as it is, this explanation predicts, or
+at least accords with, both of the conventional stories about the
+distinction between wisdom and intelligence.  Human problems are
+the most common type, so being good at solving those is key in
+achieving a high average outcome.   And it seems natural that a
+high average outcome depends mostly on experience, but that dramatic
+peaks can only be achieved by people with certain rare, innate
+qualities; nearly anyone can learn to be a good swimmer, but to be
+an Olympic swimmer you need a certain body type.This explanation also suggests why wisdom is such an elusive concept:
+there's no such thing.  "Wise" means something—that one is
+on average good at making the right choice.  But giving the name
+"wisdom" to the supposed quality that enables one to do that doesn't
+mean such a thing exists.  To the extent "wisdom" means anything,
+it refers to a grab-bag of qualities as various as self-discipline,
+experience, and empathy.  
+[4]Likewise, though "intelligent" means something, we're asking for
+trouble if we insist on looking for a single thing called "intelligence."
+And whatever its components, they're not all innate.  We use the
+word "intelligent" as an indication of ability: a smart person can
+grasp things few others could.  It does seem likely there's some
+inborn predisposition to intelligence (and wisdom too), but this
+predisposition is not itself intelligence.One reason we tend to think of intelligence as inborn is that people
+trying to measure it have concentrated on the aspects of it that
+are most measurable.  A quality that's inborn will obviously be
+more convenient to work with than one that's influenced by experience,
+and thus might vary in the course of a study.  The problem comes
+when we drag the word "intelligence" over onto what they're measuring.
+If they're measuring something inborn, they can't be measuring
+intelligence.  Three year olds aren't smart.   When we describe one
+as smart, it's shorthand for "smarter than other three year olds."SplitPerhaps it's a technicality to point out that a predisposition to
+intelligence is not the same as intelligence.  But it's an important
+technicality, because it reminds us that we can become smarter,
+just as we can become wiser.The alarming thing is that we may have to choose between the two.If wisdom and intelligence are the average and peaks of the same
+curve, then they converge as the number of points on the curve
+decreases.  If there's just one point, they're identical: the average
+and maximum are the same.  But as the number of points increases,
+wisdom and intelligence diverge.  And historically the number of
+points on the curve seems to have been increasing: our ability is
+tested in an ever wider range of situations.In the time of Confucius and Socrates, people seem to have regarded
+wisdom, learning, and intelligence as more closely related than we
+do.  Distinguishing between "wise" and "smart" is a modern habit.
+[5]
+And the reason we do is that they've been diverging.  As knowledge
+gets more specialized, there are more points on the curve, and the
+distinction between the spikes and the average becomes sharper,
+like a digital image rendered with more pixels.One consequence is that some old recipes may have become obsolete.
+At the very least we have to go back and figure out if they were
+really recipes for wisdom or intelligence.  But the really striking
+change, as intelligence and wisdom drift apart, is that we may have
+to decide which we prefer.  We may not be able to optimize for both
+simultaneously.Society seems to have voted for intelligence.  We no longer admire
+the sage—not the way people did two thousand years ago.  Now
+we admire the genius.  Because in fact the distinction we began
+with has a rather brutal converse: just as you can be smart without
+being very wise, you can be wise without being very smart.  That
+doesn't sound especially admirable.  That gets you James Bond, who
+knows what to do in a lot of situations, but has to rely on Q for
+the ones involving math.Intelligence and wisdom are obviously not mutually exclusive.  In
+fact, a high average may help support high peaks.  But there are
+reasons to believe that at some point you have to choose between
+them.  One is the example of very smart people, who are so often
+unwise that in popular culture this now seems to be regarded as the
+rule rather than the exception.  Perhaps the absent-minded professor
+is wise in his way, or wiser than he seems, but he's not wise in
+the way Confucius or Socrates wanted people to be. 
+[6]NewFor both Confucius and Socrates, wisdom, virtue, and happiness were
+necessarily related.  The wise man was someone who knew what the
+right choice was and always made it; to be the right choice, it had
+to be morally right; he was therefore always happy, knowing he'd
+done the best he could.  I can't think of many ancient philosophers
+who would have disagreed with that, so far as it goes."The superior man is always happy; the small man sad," said Confucius.
+[7]Whereas a few years ago I read an interview with a mathematician
+who said that most nights he went to bed discontented, feeling he
+hadn't made enough progress.  
+[8]
+The Chinese and Greek words we
+translate as "happy" didn't mean exactly what we do by it, but
+there's enough overlap that this remark contradicts them.Is the mathematician a small man because he's discontented?  No;
+he's just doing a kind of work that wasn't very common in Confucius's
+day.Human knowledge seems to grow fractally.  Time after time, something
+that seemed a small and uninteresting area—experimental error,
+even—turns out, when examined up close, to have as much in
+it as all knowledge up to that point.  Several of the fractal buds
+that have exploded since ancient times involve inventing and
+discovering new things.  Math, for example, used to be something a
+handful of people did part-time.  Now it's the career of thousands.
+And in work that involves making new things, some old rules don't
+apply.Recently I've spent some time advising people, and there I find the
+ancient rule still works: try to understand the situation as well
+as you can, give the best advice you can based on your experience,
+and then don't worry about it, knowing you did all you could.  But
+I don't have anything like this serenity when I'm writing an essay.
+Then I'm worried.  What if I run out of ideas?  And when I'm writing,
+four nights out of five I go to bed discontented, feeling I didn't
+get enough done.Advising people and writing are fundamentally different types of
+work.  When people come to you with a problem and you have to figure
+out the right thing to do, you don't (usually) have to invent
+anything.  You just weigh the alternatives and try to judge which
+is the prudent choice.  But prudence can't tell me what sentence
+to write next.  The search space is too big.Someone like a judge or a military officer can in much of his work
+be guided by duty, but duty is no guide in making things.  Makers
+depend on something more precarious: inspiration.  And like most
+people who lead a precarious existence, they tend to be worried,
+not contented.  In that respect they're more like the small man of
+Confucius's day, always one bad harvest (or ruler) away from
+starvation. Except instead of being at the mercy of weather and
+officials, they're at the mercy of their own imagination.LimitsTo me it was a relief just to realize it might be ok to be discontented.
+The idea that a successful person should be happy has thousands of
+years of momentum behind it.  If I was any good, why didn't I have
+the easy confidence winners are supposed to have?  But that, I now
+believe, is like a runner asking "If I'm such a good athlete, why
+do I feel so tired?" Good runners still get tired; they just get
+tired at higher speeds.People whose work is to invent or discover things are in the same
+position as the runner.  There's no way for them to do the best
+they can, because there's no limit to what they could do.  The
+closest you can come is to compare yourself to other people.  But
+the better you do, the less this matters.  An undergrad who gets
+something published feels like a star.  But for someone at the top
+of the field, what's the test of doing well?  Runners can at least
+compare themselves to others doing exactly the same thing; if you
+win an Olympic gold medal, you can be fairly content, even if you
+think you could have run a bit faster.  But what is a novelist to
+do?Whereas if you're doing the kind of work in which problems are
+presented to you and you have to choose between several alternatives,
+there's an upper bound on your performance: choosing the best every
+time.  In ancient societies, nearly all work seems to have been of
+this type.  The peasant had to decide whether a garment was worth
+mending, and the king whether or not to invade his neighbor, but
+neither was expected to invent anything.  In principle they could
+have; the king could have invented firearms, then invaded his
+neighbor.  But in practice innovations were so rare that they weren't
+expected of you, any more than goalkeepers are expected to score
+goals. 
+[9]
+In practice, it seemed as if there was a correct decision
+in every situation, and if you made it you'd done your job perfectly,
+just as a goalkeeper who prevents the other team from scoring is
+considered to have played a perfect game.In this world, wisdom seemed paramount.  
+[10]
+Even now, most people
+do work in which problems are put before them and they have to
+choose the best alternative.  But as knowledge has grown more
+specialized, there are more and more types of work in which people
+have to make up new things, and in which performance is therefore
+unbounded.  Intelligence has become increasingly important relative
+to wisdom because there is more room for spikes.RecipesAnother sign we may have to choose between intelligence and wisdom
+is how different their recipes are.  Wisdom seems to come largely
+from curing childish qualities, and intelligence largely from
+cultivating them.Recipes for wisdom, particularly ancient ones, tend to have a
+remedial character.  To achieve wisdom one must cut away all the
+debris that fills one's head on emergence from childhood, leaving
+only the important stuff.  Both self-control and experience have
+this effect: to eliminate the random biases that come from your own
+nature and from the circumstances of your upbringing respectively.
+That's not all wisdom is, but it's a large part of it.  Much of
+what's in the sage's head is also in the head of every twelve year
+old.  The difference is that in the head of the twelve year old
+it's mixed together with a lot of random junk.The path to intelligence seems to be through working on hard problems.
+You develop intelligence as you might develop muscles, through
+exercise.  But there can't be too much compulsion here.  No amount
+of discipline can replace genuine curiosity.  So cultivating
+intelligence seems to be a matter of identifying some bias in one's
+character—some tendency to be interested in certain types of
+things—and nurturing it.  Instead of obliterating your
+idiosyncrasies in an effort to make yourself a neutral vessel for
+the truth, you select one and try to grow it from a seedling into
+a tree.The wise are all much alike in their wisdom, but very smart people
+tend to be smart in distinctive ways.Most of our educational traditions aim at wisdom. So perhaps one
+reason schools work badly is that they're trying to make intelligence
+using recipes for wisdom.  Most recipes for wisdom have an element
+of subjection.  At the very least, you're supposed to do what the
+teacher says.  The more extreme recipes aim to break down your
+individuality the way basic training does.  But that's not the route
+to intelligence.  Whereas wisdom comes through humility, it may
+actually help, in cultivating intelligence, to have a mistakenly
+high opinion of your abilities, because that encourages you to keep
+working.  Ideally till you realize how mistaken you were.(The reason it's hard to learn new skills late in life is not just
+that one's brain is less malleable.  Another probably even worse
+obstacle is that one has higher standards.)I realize we're on dangerous ground here.  I'm not proposing the
+primary goal of education should be to increase students' "self-esteem."
+That just breeds laziness.  And in any case, it doesn't really fool
+the kids, not the smart ones.  They can tell at a young age that a
+contest where everyone wins is a fraud.A teacher has to walk a narrow path: you want to encourage kids to
+come up with things on their own, but you can't simply applaud
+everything they produce.  You have to be a good audience: appreciative,
+but not too easily impressed.  And that's a lot of work.  You have
+to have a good enough grasp of kids' capacities at different ages
+to know when to be surprised.That's the opposite of traditional recipes for education.  Traditionally
+the student is the audience, not the teacher; the student's job is
+not to invent, but to absorb some prescribed body of material.  (The
+use of the term "recitation" for sections in some colleges is a
+fossil of this.) The problem with these old traditions is that
+they're too much influenced by recipes for wisdom.DifferentI deliberately gave this essay a provocative title; of course it's
+worth being wise.  But I think it's important to understand the
+relationship between intelligence and wisdom, and particularly what
+seems to be the growing gap between them.  That way we can avoid
+applying rules and standards to intelligence that are really meant
+for wisdom.  These two senses of "knowing what to do" are more
+different than most people realize.  The path to wisdom is through
+discipline, and the path to intelligence through carefully selected
+self-indulgence.  Wisdom is universal, and intelligence idiosyncratic.
+And while wisdom yields calmness, intelligence much of the time
+leads to discontentment.That's particularly worth remembering.  A physicist friend recently
+told me half his department was on Prozac.  Perhaps if we acknowledge
+that some amount of frustration is inevitable in certain kinds
+of work, we can mitigate its effects.  Perhaps we can box it up and
+put it away some of the time, instead of letting it flow together
+with everyday sadness to produce what seems an alarmingly large
+pool.  At the very least, we can avoid being discontented about
+being discontented.If you feel exhausted, it's not necessarily because there's something
+wrong with you.  Maybe you're just running fast.Notes[1]
+Gauss was supposedly asked this when he was 10.  Instead of
+laboriously adding together the numbers like the other students,
+he saw that they consisted of 50 pairs that each summed to 101 (100
++ 1, 99 + 2, etc), and that he could just multiply 101 by 50 to get
+the answer, 5050.[2]
+A variant is that intelligence is the ability to solve problems,
+and wisdom the judgement to know how to use those solutions.   But
+while this is certainly an important relationship between wisdom
+and intelligence, it's not the distinction between them.  Wisdom
+is useful in solving problems too, and intelligence can help in
+deciding what to do with the solutions.[3]
+In judging both intelligence and wisdom we have to factor out
+some knowledge. People who know the combination of a safe will be
+better at opening it than people who don't, but no one would say
+that was a test of intelligence or wisdom.But knowledge overlaps with wisdom and probably also intelligence.
+A knowledge of human nature is certainly part of wisdom.  So where
+do we draw the line?Perhaps the solution is to discount knowledge that at some point
+has a sharp drop in utility.  For example, understanding French
+will help you in a large number of situations, but its value drops
+sharply as soon as no one else involved knows French.  Whereas the
+value of understanding vanity would decline more gradually.The knowledge whose utility drops sharply is the kind that has
+little relation to other knowledge.  This includes mere conventions,
+like languages and safe combinations, and also what we'd call
+"random" facts, like movie stars' birthdays, or how to distinguish
+1956 from 1957 Studebakers.[4]
+People seeking some single thing called "wisdom" have been
+fooled by grammar.  Wisdom is just knowing the right thing to do,
+and there are a hundred and one different qualities that help in
+that.  Some, like selflessness, might come from meditating in an
+empty room, and others, like a knowledge of human nature, might
+come from going to drunken parties.Perhaps realizing this will help dispel the cloud of semi-sacred
+mystery that surrounds wisdom in so many people's eyes.  The mystery
+comes mostly from looking for something that doesn't exist.  And
+the reason there have historically been so many different schools
+of thought about how to achieve wisdom is that they've focused on
+different components of it.When I use the word "wisdom" in this essay, I mean no more than
+whatever collection of qualities helps people make the right choice
+in a wide variety of situations.[5]
+Even in English, our sense of the word "intelligence" is
+surprisingly recent.  Predecessors like "understanding" seem to
+have had a broader meaning.[6]
+There is of course some uncertainty about how closely the remarks
+attributed to Confucius and Socrates resemble their actual opinions.
+I'm using these names as we use the name "Homer," to mean the
+hypothetical people who said the things attributed to them.[7]
+Analects VII:36, Fung trans.Some translators use "calm" instead of "happy."  One source of
+difficulty here is that present-day English speakers have a different
+idea of happiness from many older societies.  Every language probably
+has a word meaning "how one feels when things are going well," but
+different cultures react differently when things go well.  We react
+like children, with smiles and laughter.  But in a more reserved
+society, or in one where life was tougher, the reaction might be a
+quiet contentment.[8]
+It may have been Andrew Wiles, but I'm not sure.  If anyone
+remembers such an interview, I'd appreciate hearing from you.[9]
+Confucius claimed proudly that he had never invented
+anything—that he had simply passed on an accurate account of
+ancient traditions.  [Analects VII:1] It's hard for us now to
+appreciate how important a duty it must have been in preliterate
+societies to remember and pass on the group's accumulated knowledge.
+Even in Confucius's time it still seems to have been the first duty
+of the scholar.[10]
+The bias toward wisdom in ancient philosophy may be exaggerated
+by the fact that, in both Greece and China, many of the first
+philosophers (including Confucius and Plato) saw themselves as
+teachers of administrators, and so thought disproportionately about
+such matters.  The few people who did invent things, like storytellers,
+must have seemed an outlying data point that could be ignored.Thanks to Trevor Blackwell, Sarah Harlin, Jessica Livingston,
+and Robert Morris for reading drafts of this.

recipes/experimental/long-context/H2O/run_needle_haystack_test.py

@@ -0,0 +1,107 @@
+import os
+import tqdm
+import glob
+import json
+import copy
+import math
+
+import torch
+import logging
+import argparse
+
+import numpy as np
+
+from transformers import AutoModelForCausalLM, AutoTokenizer, AutoConfig
+from utils.llama import H2OLlamaForCausalLM
+
+def set_seed(args):
+    np.random.seed(args.seed)
+    torch.manual_seed(args.seed)
+    torch.cuda.manual_seed_all(args.seed)
+
+if __name__ == '__main__':
+
+    parser = argparse.ArgumentParser()
+
+    parser.add_argument("--input-path", type=str, default="")
+    parser.add_argument("--output-path", type=str, default="")
+
+    parser.add_argument("--model-provider", type=str, default="Huggingface")
+    parser.add_argument("--model-name", type=str, default="")
+
+    parser.add_argument("--enable_h2o_generation", action='store_true')
+    parser.add_argument("--num_heavy_hitter_tokens", type=int, default=128)
+    parser.add_argument("--num_window_length", type=int, default=256)
+
+    parser.add_argument("--enable_position_rolling", action='store_true')
+
+    parser.add_argument("--max_new_tokens", type=int, default=1024)
+    parser.add_argument("--temperature", type=float, default=0.1)
+    parser.add_argument("--seed", type=int, default=42, help="random seed for initialization")
+
+    args = parser.parse_args()
+
+    set_seed(args)
+
+    model_name = args.model_name
+    input_path = args.input_path
+    output_path = args.output_path
+    model_provider = args.model_provider
+    os.makedirs(os.path.dirname(output_path), exist_ok=True)
+
+    config = AutoConfig.from_pretrained(model_name)
+    tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=True)
+
+    if args.enable_h2o_generation:
+        config.num_heavy_hitter_tokens = args.num_heavy_hitter_tokens
+        config.num_window_length = args.num_window_length
+        config.enable_position_rolling = args.enable_position_rolling
+        model = H2OLlamaForCausalLM.from_pretrained(model_name,
+            torch_dtype=torch.float16,
+            device_map='auto',
+            low_cpu_mem_usage=True,
+            config=config)
+    else:
+        model = AutoModelForCausalLM.from_pretrained(model_name,
+            torch_dtype=torch.float16,
+            device_map='auto',
+            low_cpu_mem_usage=True,)
+
+    # load the testing prompts
+    for filename in tqdm.tqdm(glob.glob(f'{input_path}/{args.model_provider}_*_prompts.json')):
+        with open(filename, 'r') as f:
+            input_data = json.load(f)
+            prompt = input_data[0]['content']+'\n'+input_data[1]['content']
+
+            input = tokenizer(prompt, truncation=False, return_tensors="pt").to(model.device)
+            context_length = input.input_ids.shape[-1]
+            output = model.generate(
+                **input,
+                max_new_tokens=args.max_new_tokens,
+                num_beams=1,
+                temperature=args.temperature,
+                pad_token_id=tokenizer.eos_token_id,
+            )
+            pred = tokenizer.decode(output[0][context_length:], skip_special_tokens=True)
+            pred = pred.strip()
+
+        basename = os.path.basename(filename)
+        newname = basename.replace('.json', '.txt').replace('_prompts', '')
+        with open(f'{output_path}/{newname}', 'w') as f:
+            f.write(pred)
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+

recipes/experimental/long-context/H2O/run_streaming.py

@@ -0,0 +1,91 @@
+import torch
+import argparse
+import json
+import os
+import time
+import re
+import sys
+
+from utils.streaming import load, download_url, load_jsonl, greedy_generate
+
+from transformers import AutoModelForCausalLM, AutoTokenizer, AutoConfig
+from utils.llama import H2OLlamaForCausalLM
+from utils.cache import Cache, HHCache, StaticCache
+
+
+@torch.no_grad()
+def streaming_inference_h2o(model, tokenizer, config, prompts, max_gen_len=1000, enable_h2o_generation=False):
+    past_key_values = None
+    for idx, prompt in enumerate(prompts):
+        prompt = "USER: " + prompt + "\n\nASSISTANT: "
+        print("\n" + prompt, end="")
+        input_ids = tokenizer(prompt, return_tensors="pt").input_ids
+        input_ids = input_ids.to(model.device)
+        seq_len = input_ids.shape[1]
+
+        past_key_values = greedy_generate(
+            model, tokenizer, input_ids, past_key_values, max_gen_len=max_gen_len
+        )
+        if enable_h2o_generation:
+            space_needed = seq_len + max_gen_len
+            past_key_values = HHCache.from_legacy_cache(config.num_window_length, config.num_heavy_hitter_tokens, past_key_values)
+            past_key_values.evict_for_space(space_needed)
+            past_key_values = past_key_values.to_legacy_cache()
+
+
+def main():
+    parser = argparse.ArgumentParser()
+
+    parser.add_argument("--input-path", type=str, default="")
+    parser.add_argument("--model-name", type=str, default="lmsys/vicuna-13b-v1.5")
+
+    parser.add_argument("--enable_h2o_generation", action='store_true')
+    parser.add_argument("--num_heavy_hitter_tokens", type=int, default=128)
+    parser.add_argument("--num_window_length", type=int, default=256)
+
+    parser.add_argument("--enable_position_rolling", action='store_true')
+
+    parser.add_argument("--seed", type=int, default=42, help="random seed for initialization")
+
+    args = parser.parse_args()
+
+    model_name = args.model_name
+    data_root = args.input_path
+
+    config = AutoConfig.from_pretrained(model_name)
+    tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=True)
+
+    if args.enable_h2o_generation:
+        config.num_heavy_hitter_tokens = args.num_heavy_hitter_tokens
+        config.num_window_length = args.num_window_length
+        config.enable_position_rolling = args.enable_position_rolling
+        model = H2OLlamaForCausalLM.from_pretrained(model_name,
+            torch_dtype=torch.float16,
+            device_map='auto',
+            low_cpu_mem_usage=True,
+            config=config)
+    else:
+        model = AutoModelForCausalLM.from_pretrained(model_name,
+            torch_dtype=torch.float16,
+            device_map='auto',
+            low_cpu_mem_usage=True,)
+
+    test_filepath = os.path.join(data_root, "mt_bench.jsonl")
+    print(f"Loading data from {test_filepath} ...")
+
+    if not os.path.exists(test_filepath):
+        download_url(
+            "https://raw.githubusercontent.com/lm-sys/FastChat/main/fastchat/llm_judge/data/mt_bench/question.jsonl",
+            data_root,
+        )
+        os.rename(os.path.join(data_root, "question.jsonl"), test_filepath)
+
+    list_data = load_jsonl(test_filepath)
+    prompts = []
+    for sample in list_data:
+        prompts += sample["turns"]
+
+    streaming_inference_h2o(model, tokenizer, config, prompts, enable_h2o_generation=args.enable_h2o_generation)
+
+if __name__ == "__main__":
+    main()

research/long-context-llama/H2O/run_summarization.py

@@ -20,8 +20,7 @@ from utils.llama import H2OLlamaForCausalLM
 def set_seed(args):
     np.random.seed(args.seed)
     torch.manual_seed(args.seed)
-    if args.n_gpu > 0:
-        torch.cuda.manual_seed_all(args.seed)
+    torch.cuda.manual_seed_all(args.seed)
 
 if __name__ == '__main__':
 
@@ -38,13 +37,11 @@ if __name__ == '__main__':
 
     parser.add_argument("--enable_position_rolling", action='store_true')
 
-    parser.add_argument("--sample_num", type=int, default=1000)
+    parser.add_argument("--sample_num", type=int, default=500)
     parser.add_argument("--seed", type=int, default=42, help="random seed for initialization")
 
     args = parser.parse_args()
 
-    args.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
-    args.n_gpu = torch.cuda.device_count()
     set_seed(args)
 
     model_name = args.model_name
@@ -59,11 +56,16 @@ if __name__ == '__main__':
         config.num_heavy_hitter_tokens = args.num_heavy_hitter_tokens
         config.num_window_length = args.num_window_length
         config.enable_position_rolling = args.enable_position_rolling
-        model = H2OLlamaForCausalLM.from_pretrained(model_name, config=config)
+        model = H2OLlamaForCausalLM.from_pretrained(model_name,
+            torch_dtype=torch.float16,
+            device_map='auto',
+            low_cpu_mem_usage=True,
+            config=config)
     else:
-        model = AutoModelForCausalLM.from_pretrained(model_name)
-
-    model.half().eval().cuda()
+        model = AutoModelForCausalLM.from_pretrained(model_name,
+            torch_dtype=torch.float16,
+            device_map='auto',
+            low_cpu_mem_usage=True,)
 
     # loading inference data
     requests = []
@@ -100,6 +102,7 @@ if __name__ == '__main__':
                 do_sample=True,
                 num_return_sequences=request['n'],
                 return_dict_in_generate=True, output_scores=True,
+                pad_token_id=tokenizer.eos_token_id
             )
 
             tokens = tokenizer.convert_ids_to_tokens(output_sequences['sequences'].squeeze(0))[len(input_ids[0]):]

recipes/experimental/long-context/H2O/src/streaming.sh

@@ -0,0 +1,23 @@
+method=$1
+if [[ ${method} == 'h2o' ]]; then
+    python -u run_streaming.py \
+        --input-path data \
+        --model-name lmsys/vicuna-13b-v1.5 \
+        --enable_h2o_generation \
+        --num_heavy_hitter_tokens 2048 \
+        --num_window_length 4096 \
+        --enable_position_rolling
+elif [[ ${method} == 'full' ]]; then
+    python -u run_streaming.py \
+        --input-path data \
+        --model-name lmsys/vicuna-13b-v1.5
+else
+    echo 'unknown argment for method'
+fi
+
+
+
+
+
+
+

research/long-context-llama/H2O/utils/cache.py

@@ -518,6 +518,36 @@ class HHCache(Cache):
                 cache.update(key_states, value_states, layer_idx, accumulated_attention_scores=accumulated_attention_scores)
         return cache
 
+    def evict_for_space(self, space_needed: int):
+        num_layers = len(self.key_cache)
+
+        # Update score metrics (Accumulated attention scores)
+        if len(self.accumulated_attention_scores) < num_layers:
+            raise ValueError("The accumulated_attention_scores should be updated before evicting the cache.")
+
+        for layer_idx in range(num_layers):
+            # Update KV Cache, Evict for new coming prompts
+            if self.get_seq_length(layer_idx) + space_needed > self.window_length:
+                if self.window_length - self.num_hh_tokens <= space_needed:
+                    raise ValueError("The space_needed should be less than the window_length - num_hh_tokens.")
+
+                seq_scores = self.accumulated_attention_scores[layer_idx][:, :, :-self.window_length + self.num_hh_tokens + space_needed]
+                _, keep_hh_index = torch.topk(seq_scores, self.num_hh_tokens, dim=-1)
+                keep_hh_index = keep_hh_index.sort().values
+
+                keep_local_index = torch.arange(self.get_seq_length(layer_idx) - self.window_length + self.num_hh_tokens + space_needed, self.get_seq_length(layer_idx), device=keep_hh_index.device).repeat(keep_hh_index.shape[0], keep_hh_index.shape[1], 1)
+                keep_index = torch.cat([keep_hh_index, keep_local_index], dim=-1)
+
+                mask = torch.zeros(self.accumulated_attention_scores[layer_idx].shape, dtype=torch.bool).to(keep_hh_index.device)
+                mask = mask.scatter(-1, keep_index, 1)
+
+                bsz, num_heads, _, head_dim = self.key_cache[layer_idx].shape
+                self.key_cache[layer_idx] = self.key_cache[layer_idx][mask].view(bsz, num_heads, -1, head_dim)
+                self.value_cache[layer_idx] = self.value_cache[layer_idx][mask].view(bsz, num_heads, -1, head_dim)
+                self.accumulated_attention_scores[layer_idx] = self.accumulated_attention_scores[layer_idx][mask].view(bsz, num_heads, -1)
+
+
+
 
 class StaticCache(Cache):
     """

research/long-context-llama/H2O/utils/llama.py

@@ -1,5 +1,7 @@
 import math
 from typing import Any, Dict, List, Optional, Tuple, Union
+import warnings
+warnings.filterwarnings("ignore")
 
 import pdb
 import types
@@ -182,16 +184,18 @@ class H2OLlamaAttention(nn.Module):
                 # decoding stage
                 key_position_ids = torch.arange(kv_seq_len, device=hidden_states.device).unsqueeze(0)
                 query_position_ids = key_position_ids[:, -1].unsqueeze(0)
+            elif not kv_seq_len == position_ids.shape[-1]:
+                # prefilling stage with evicting
+                query_position_ids = position_ids
+                key_position_ids = torch.arange(kv_seq_len, device=hidden_states.device).unsqueeze(0)
             else:
+                # prefilling stage
                 query_position_ids = position_ids
                 key_position_ids = position_ids
 
             key_cos, key_sin = self.rotary_emb(value_states, key_position_ids)
             query_cos, query_sin = self.rotary_emb(value_states, query_position_ids)
 
-            if self.layer_idx == 0:
-                print(kv_seq_len, query_position_ids, key_position_ids)
-
             query_states = apply_rotary_pos_emb_single(query_states, query_cos, query_sin)
             key_states = apply_rotary_pos_emb_single(key_states, key_cos, key_sin)
 

recipes/experimental/long-context/H2O/utils/needle_test/config-eval.yaml

@@ -0,0 +1,7 @@
+model:
+  model_provider: "OpenAI"
+  model_name: "gpt-4"
+
+prompt:
+  needle: "\nThe best thing to do in San Francisco is eat a sandwich and sit in Dolores Park on a sunny day.\n"
+  retrieval_question: "What is the best thing to do in San Francisco?"

recipes/experimental/long-context/H2O/utils/needle_test/config-prompt.yaml

@@ -0,0 +1,22 @@
+prompt:
+  needle: "\nThe best thing to do in San Francisco is eat a sandwich and sit in Dolores Park on a sunny day.\n"
+  haystack_dir: "data/PaulGrahamEssays"
+  retrieval_question: "What is the best thing to do in San Francisco? Here is the most relevant sentence in the context:" # We use the Anthropic's retrieval question as the default one
+
+context:
+  min_len: 1000
+  max_len: 8000
+  interval: 10
+  manually_select_list: null  # null or a list of context lengths to manually select
+
+document_depth:
+  min_percent: 10
+  max_percent: 90
+  interval: 10
+  interval_type: "linear"  # "linear", "sigmoid" or null
+  manually_select_list: null  # null or a list of document percents to manually select
+
+tokenizer:
+  tokenizer_type: "Huggingface"
+
+save_dir: 'data/needle_test/Huggingface'

recipes/experimental/long-context/H2O/utils/needle_test/prompt.py

@@ -0,0 +1,334 @@
+'''
+    Generate prompts for the LLM Needle Haystack.
+    Source code from: 
+        https://github.com/gkamradt/LLMTest_NeedleInAHaystack/tree/main
+        https://github.com/THUDM/LongAlign/tree/main/Needle_test
+'''
+from dotenv import load_dotenv
+import os
+import tiktoken
+import glob
+import json
+import yaml
+import argparse
+from anthropic import Anthropic
+import numpy as np
+import asyncio
+from asyncio import Semaphore
+from transformers import AutoTokenizer
+
+load_dotenv()
+
+class Prompter:
+    """
+    This class is used to test the LLM Needle Haystack.
+    """
+    def __init__(self,
+                 needle="\nThe best thing to do in San Francisco is eat a sandwich and sit in Dolores Park on a sunny day.\n",
+                 haystack_dir="PaulGrahamEssays",
+                 retrieval_question="What is the best thing to do in San Francisco?",
+                 context_lengths_min = 1000,
+                 context_lengths_max = 200000,
+                 context_lengths_num_intervals = 35,
+                 context_lengths = None,
+                 document_depth_percent_min = 0,
+                 document_depth_percent_max = 100,
+                 document_depth_percent_intervals = 35,
+                 document_depth_percents = None,
+                 document_depth_percent_interval_type = "linear",
+                 tokenizer_type = "OpenAI",
+                 model_name = "gpt-4-1106-preview",
+                 num_concurrent_requests = 1,
+                 final_context_length_buffer = 200,
+                 save_dir = "prompts",
+                 print_ongoing_status = True):
+        """        
+        :param needle: The needle to be found in the haystack. Default is None.
+        :param haystack_dir: The directory of text files to use as background context (or a haystack) in which the needle is to be found. Default is Paul Graham Essays.
+        :param retrieval_question: The question which with to prompt the model to do the retrieval.
+        :param results_version: In case you would like to try the same combination of model, context length, and depth % multiple times, change the results version other than 1
+        :param num_concurrent_requests: Due to volume, this object is set up to run concurrent requests, default = 1. Be careful of rate limits.
+        :param save_results: Whether or not you would like to save your contexts to file. Warning: These will get long! Default = True
+        :param save_contexts: Whether or not you would like to save your contexts to file. Warning: These will get long! Default is True.
+        :param final_context_length_buffer: The amount of cushion you'd like to leave off the input context to allow for the output context. Default 200 tokens
+        :param context_lengths_min: The minimum length of the context. Default is 1000.
+        :param context_lengths_max: The maximum length of the context. Default is 200000.
+        :param context_lengths_num_intervals: The number of intervals for the context length. Default is 35.
+        :param context_lengths: The lengths of the context. Default is None.
+        :param document_depth_percent_min: The minimum depth percent of the document. Default is 0.
+        :param document_depth_percent_max: The maximum depth percent of the document. Default is 100.
+        :param document_depth_percent_intervals: The number of intervals for the document depth percent. Default is 35.
+        :param document_depth_percents: The depth percentages of the document. Default is None.
+        :param document_depth_percent_interval_type: The type of interval for the document depth percent. Must be either 'linear' or 'sigmoid'. Default is 'linear'.
+        :param model_provider: The provider of the model. Must be either 'OpenAI' or 'Anthropic'. Default is 'OpenAI'.
+        :param openai_api_key: The API key for OpenAI. Default is None.
+        :param anthropic_api_key: The API key for Anthropic. Default is None.
+        :param model_name: The name of the model. Default is 'gpt-4-1106-preview'.
+        :param seconds_to_sleep_between_completions: The number of seconds to sleep between completions. Default is None.
+        :param print_ongoing_status: Whether or not to print the ongoing status. Default is True.
+        """
+        if not needle or not haystack_dir or not retrieval_question:
+            raise ValueError("Needle, haystack, and retrieval_question must be provided.")
+        
+        self.needle = needle
+        self.haystack_dir = haystack_dir
+        self.retrieval_question = retrieval_question
+        self.num_concurrent_requests = num_concurrent_requests
+        self.final_context_length_buffer = final_context_length_buffer
+        self.print_ongoing_status = print_ongoing_status
+        self.tokenizer_type = tokenizer_type
+        self.model_name = model_name
+        self.testing_results = []
+
+        if context_lengths is None:
+            if context_lengths_min is None or context_lengths_max is None or context_lengths_num_intervals is None:
+                raise ValueError("Either context_lengths_min, context_lengths_max, context_lengths_intervals need to be filled out OR the context_lengths_list needs to be supplied.")
+            else:
+                self.context_lengths = np.round(np.linspace(context_lengths_min, context_lengths_max, num=context_lengths_num_intervals, endpoint=True)).astype(int)
+        else:
+            self.context_lengths = context_lengths
+
+        if document_depth_percents is None:
+            if document_depth_percent_min is None or document_depth_percent_max is None or document_depth_percent_intervals is None:
+                raise ValueError("Either document_depth_percent_min, document_depth_percent_max, document_depth_percent_intervals need to be filled out OR the document_depth_percents needs to be supplied.")
+            else:
+                if document_depth_percent_interval_type == 'linear':
+                    self.document_depth_percents = np.round(np.linspace(document_depth_percent_min, document_depth_percent_max, num=document_depth_percent_intervals, endpoint=True)).astype(int)
+                elif document_depth_percent_interval_type == 'sigmoid':
+                    self.document_depth_percents = [self.logistic(x) for x in np.linspace(document_depth_percent_min, document_depth_percent_max, document_depth_percent_intervals)]
+        else:
+            self.document_depth_percents = document_depth_percents
+
+        if document_depth_percent_interval_type not in [None, "linear", "sigmoid"]:
+            raise ValueError("document_depth_percent_interval_type must be either None, 'linear' or 'sigmoid'. If you'd like your own distribution give a list of ints in via document_depth_percent_intervals")
+        
+        if self.tokenizer_type == "OpenAI":
+            assert self.model_name is not None, "If you're using OpenAI, you must provide a model name."
+            self.enc = tiktoken.encoding_for_model(self.model_name)
+        elif self.tokenizer_type == "Anthropic":
+            self.enc = Anthropic().get_tokenizer()
+        elif self.tokenizer_type == "Huggingface":
+            self.enc = AutoTokenizer.from_pretrained(self.model_name, trust_remote_code=True, use_fast=False)
+        else:
+            raise ValueError("tokenizer_type is not supported. Must be either 'tiktoken', 'Anthropic', or 'Huggingface'")
+        
+        self.save_dir = save_dir
+
+    def logistic(self, x, L=100, x0=50, k=.1):
+        if x == 0:
+            return 0
+        if x == 100:
+            return 100
+        return np.round(L / (1 + np.exp(-k * (x - x0))), 3)
+    
+    async def bound_evaluate_and_log(self, sem, *args):
+        async with sem:
+            await self.evaluate_and_log(*args)
+
+    async def run_test(self):
+        sem = Semaphore(self.num_concurrent_requests)
+
+        # Run through each iteration of context_lengths and depths
+        tasks = []
+        for context_length in self.context_lengths:
+            for depth_percent in self.document_depth_percents:
+                task = self.bound_evaluate_and_log(sem, context_length, depth_percent)
+                tasks.append(task)
+
+        # Wait for all tasks to complete
+        await asyncio.gather(*tasks)
+
+    def generate_prompt(self, context):
+        return [
+            {
+                "role": "system",
+                "content": "You are a helpful AI bot that answers questions for a user. Keep your response short and direct"
+            },
+            {
+                "role": "user",
+                "content": context + '\n\n' + f"Read the document and answer: {self.retrieval_question}"
+            },
+        ]
+
+    async def evaluate_and_log(self, context_length, depth_percent):
+        # Checks to see if you've already checked a length/percent/version.
+        # This helps if the program stop running and you want to restart later
+
+        # Go generate the required length context and place your needle statement in
+        context = await self.generate_context(context_length, depth_percent)
+        print('Generate for context length:', context_length, 'depth percent:', depth_percent)
+
+        # Prepare your message to send to the model you're going to evaluate
+        prompt = self.generate_prompt(context)
+
+        context_file_location = f'{self.tokenizer_type}_len_{context_length}_depth_{int(depth_percent*100)}'
+
+        # Save the prompts to file for retesting
+        if not os.path.exists(self.save_dir):
+            os.makedirs(self.save_dir)
+
+        # Save the result to file for retesting
+        with open(f'{self.save_dir}/{context_file_location}_prompts.json', 'w') as f:
+            json.dump(prompt, f)
+
+    async def generate_context(self, context_length, depth_percent):
+        # Load up tiktoken so we navigate tokens more easily
+
+        # Get your Paul Graham files loaded into a string
+        context = self.read_context_files()
+
+        # Truncate the Paul Graham essays to the context length you desire
+        context = self.encode_and_trim(context, context_length)
+
+        # Insert your random statement according to your depth percent
+        context = self.insert_needle(context, depth_percent, context_length)
+
+        return context
+    
+    def encode_text_to_tokens(self, text):
+        if self.tokenizer_type == "OpenAI":
+            return self.enc.encode(text)
+        elif self.tokenizer_type == "Anthropic":
+            # Assuming you have a different encoder for Anthropic
+            return self.enc.encode(text).ids
+        elif self.tokenizer_type == "Huggingface":
+            return self.enc(text, truncation=False, return_tensors="pt", add_special_tokens=False).input_ids.view(-1).tolist()
+        else:
+            raise ValueError("tokenizer_type must be either 'OpenAI', 'Anthropic', or 'Huggingface'")
+    
+    def insert_needle(self, context, depth_percent, context_length):
+        tokens_needle = self.encode_text_to_tokens(self.needle)
+        tokens_context = self.encode_text_to_tokens(context)        
+
+        # Reducing the context length by 150 buffer. This is to account for system message, the user question, and response.
+        context_length -= self.final_context_length_buffer
+
+        # If your context + needle are longer than the context length (which it will be), then reduce tokens from the context by the needle length
+        if len(tokens_context) + len(tokens_needle) > context_length:
+            tokens_context = tokens_context[:context_length - len(tokens_needle)]
+
+        if depth_percent == 100:
+            # If your depth percent is 100 (which means your needle is the last thing in the doc), throw it at the end
+            tokens_new_context = tokens_context + tokens_needle
+        else:
+            # Go get the position (in terms of tokens) to insert your needle
+            insertion_point = int(len(tokens_context) * (depth_percent / 100))
+
+            # tokens_new_context represents the tokens before the needle
+            tokens_new_context = tokens_context[:insertion_point]
+
+            # We want to make sure that we place our needle at a sentence break so we first see what token a '.' is
+            period_tokens = self.encode_text_to_tokens('.')
+            period_tokens = [30930]
+            
+            # Then we iteration backwards until we find the first period
+            while tokens_new_context and tokens_new_context[-1] not in period_tokens:
+                insertion_point -= 1
+                tokens_new_context = tokens_context[:insertion_point]
+
+            # Once we get there, then add in your needle, and stick the rest of your context in on the other end.
+            # Now we have a needle in a haystack
+            tokens_new_context += tokens_needle + tokens_context[insertion_point:]
+
+        # Convert back to a string and return it
+        new_context = self.decode_tokens(tokens_new_context)
+        return new_context
+
+    def get_context_length_in_tokens(self, context):
+        if self.tokenizer_type == "OpenAI":
+            return len(self.enc.encode(context))
+        elif self.tokenizer_type == "Anthropic":
+            # Assuming you have a different encoder for Anthropic
+            return len(self.enc.encode(context).ids)
+        elif self.tokenizer_type == "Huggingface":
+            return self.enc(context, truncation=False, return_tensors="pt").input_ids.shape[-1]
+        else:
+            raise ValueError("tokenizer_type must be either 'OpenAI', 'Anthropic', or 'Huggingface'")
+
+    def read_context_files(self):
+        context = ""
+        max_context_length = max(self.context_lengths)
+        while self.get_context_length_in_tokens(context) < max_context_length:
+            for file in glob.glob(f"{self.haystack_dir}/*.txt"):
+                with open(file, 'r') as f:
+                    context += f.read()
+        return context
+
+    def get_tokens_from_context(self, context):
+        if self.tokenizer_type == "OpenAI":
+            return self.enc.encode(context)
+        elif self.tokenizer_type == "Anthropic":
+            # Assuming you have a different encoder for Anthropic
+            return self.enc.encode(context).ids
+        elif self.tokenizer_type == "Huggingface":
+            return self.enc(context, truncation=False, return_tensors="pt").input_ids.view(-1).tolist()
+        else:
+            raise ValueError("tokenizer_type must be either 'OpenAI', 'Anthropic', or 'Huggingface'")
+        
+    def decode_tokens(self, tokens, context_length=None):
+        if self.tokenizer_type == "OpenAI":
+            return self.enc.decode(tokens[:context_length])
+        elif self.tokenizer_type == "Anthropic":
+            # Assuming you have a different decoder for Anthropic
+            return self.enc.decode(tokens[:context_length])
+        elif self.tokenizer_type == "Huggingface":
+            decoded = self.enc.decode(tokens[:context_length], skip_special_tokens=True)
+            return decoded
+        else:
+            raise ValueError("tokenizer_type must be either 'OpenAI', 'Anthropic', or 'Huggingface'")
+
+    def encode_and_trim(self, context, context_length):
+        tokens = self.get_tokens_from_context(context)
+        if len(tokens) > context_length:
+            context = self.decode_tokens(tokens, context_length)
+        return context
+    
+    def get_results(self):
+        return self.testing_results
+    
+    def print_start_test_summary(self):
+        print ("\n")
+        print ("Starting Prompt Generation ...")
+        print (f"- Tokenizer: {self.tokenizer_type}")
+        print (f"- Context Lengths: {len(self.context_lengths)}, Min: {min(self.context_lengths)}, Max: {max(self.context_lengths)}")
+        print (f"- Document Depths: {len(self.document_depth_percents)}, Min: {min(self.document_depth_percents)}%, Max: {max(self.document_depth_percents)}%")
+        print (f"- Needle: {self.needle.strip()}")
+        print ("\n\n")
+
+    def start_test(self):
+        if self.print_ongoing_status:
+            self.print_start_test_summary()
+        asyncio.run(self.run_test())
+
+
+if __name__ == '__main__':
+    with open('utils/needle_test/config-prompt.yaml', 'r') as file:
+        config = yaml.load(file, Loader=yaml.FullLoader)
+
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--model_name', type=str, default='None')
+    args = parser.parse_args()
+
+    ht = Prompter(
+        needle=config['prompt']['needle'],
+        haystack_dir=config['prompt']['haystack_dir'],
+        retrieval_question=config['prompt']['retrieval_question'],
+
+        context_lengths_min=config['context']['min_len'],
+        context_lengths_max=config['context']['max_len'],
+        context_lengths_num_intervals=config['context']['interval'],
+        context_lengths=config['context']['manually_select_list'],
+
+        document_depth_percent_min=config['document_depth']['min_percent'],
+        document_depth_percent_max=config['document_depth']['max_percent'],
+        document_depth_percent_intervals=config['document_depth']['interval'],
+        document_depth_percents=config['document_depth']['manually_select_list'],
+        document_depth_percent_interval_type=config['document_depth']['interval_type'],
+
+        tokenizer_type=config['tokenizer']['tokenizer_type'],
+        model_name=args.model_name,
+
+        save_dir=config['save_dir'],
+    )
+
+    ht.start_test()

recipes/experimental/long-context/H2O/utils/needle_test/vis.py

@@ -0,0 +1,89 @@
+'''
+    Generate prompts for the LLM Needle Haystack.
+    Source code from: 
+        https://github.com/gkamradt/LLMTest_NeedleInAHaystack/tree/main
+        https://github.com/THUDM/LongAlign/tree/main/Needle_test
+'''
+
+import pandas as pd
+import seaborn as sns
+import matplotlib.pyplot as plt
+from matplotlib.colors import LinearSegmentedColormap
+import json
+import os
+import re
+import glob
+import argparse
+
+if __name__ == '__main__':
+    # Using glob to find all json files in the directory
+
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--input-path", type=str, default="")
+    parser.add_argument("--exp-name", type=str, default="")
+    args = parser.parse_args()
+
+    json_files = glob.glob(f"{args.input_path}/*.json")
+
+    if not os.path.exists('vis'):
+        os.makedirs('vis')
+
+    print(json_files)
+
+    # Iterating through each file and extract the 3 columns we need
+    for file in json_files:
+        print(file)
+        # List to hold the data
+        data = []
+
+        with open(file, 'r') as f:
+            json_data = json.load(f)
+            
+            for k in json_data:
+                pattern = r"_len_(\d+)_"
+                match = re.search(pattern, k)
+                context_length = int(match.group(1)) if match else None
+
+                pattern = r"depth_(\d+)"
+                match = re.search(pattern, k)
+                document_depth = eval(match.group(1))/100 if match else None
+
+                score = json_data[k]['score']
+
+                # Appending to the list
+                data.append({
+                    "Document Depth": document_depth,
+                    "Context Length": context_length,
+                    "Score": score
+                })
+
+        # Creating a DataFrame
+        df = pd.DataFrame(data)
+
+        pivot_table = pd.pivot_table(df, values='Score', index=['Document Depth', 'Context Length'], aggfunc='mean').reset_index() # This will aggregate
+        pivot_table = pivot_table.pivot(index="Document Depth", columns="Context Length", values="Score") # This will turn into a proper pivot
+        
+        # Create a custom colormap. Go to https://coolors.co/ and pick cool colors
+        cmap = LinearSegmentedColormap.from_list("custom_cmap", ["#F0496E", "#EBB839", "#0CD79F"])
+
+        # Create the heatmap with better aesthetics
+        plt.figure(figsize=(17.5, 8))  # Can adjust these dimensions as needed
+        sns.heatmap(
+            pivot_table,
+            # annot=True,
+            fmt="g",
+            cmap=cmap,
+            cbar_kws={'label': 'Score'},
+            vmin=1,
+            vmax=10,
+        )
+
+        # More aesthetics
+        plt.title(f'Pressure Testing\nFact Retrieval Across Context Lengths ("Needle In A HayStack")\n{args.exp_name}')  # Adds a title
+        plt.xlabel('Token Limit')  # X-axis label
+        plt.ylabel('Depth Percent')  # Y-axis label
+        plt.xticks(rotation=45)  # Rotates the x-axis labels to prevent overlap
+        plt.yticks(rotation=0)  # Ensures the y-axis labels are horizontal
+        plt.tight_layout()  # Fits everything neatly into the figure area
+        # Show the plot
+        plt.savefig(f"vis/{file.split('/')[-1].replace('.json', '')}.png")

recipes/experimental/long-context/H2O/utils/streaming.py

@@ -0,0 +1,123 @@
+"""
+    Source Code: https://github.com/mit-han-lab/streaming-llm/blob/main/streaming_llm/utils.py
+"""
+
+import torch
+import argparse
+from transformers import (
+    AutoTokenizer,
+    AutoModelForCausalLM,
+)
+import os.path as osp
+import ssl
+import urllib.request
+import os
+import json
+
+
+def load(model_name_or_path):
+    print(f"Loading model from {model_name_or_path} ...")
+    # however, tensor parallel for running falcon will occur bugs
+    tokenizer = AutoTokenizer.from_pretrained(
+        model_name_or_path,
+        trust_remote_code=True,
+    )
+    model = AutoModelForCausalLM.from_pretrained(
+        model_name_or_path,
+        device_map="auto",
+        torch_dtype=torch.float16,
+        trust_remote_code=True,
+    )
+    if tokenizer.pad_token_id is None:
+        if tokenizer.eos_token_id is not None:
+            tokenizer.pad_token_id = tokenizer.eos_token_id
+        else:
+            tokenizer.pad_token_id = 0
+
+    model.eval()
+
+    return model, tokenizer
+
+
+def download_url(url: str, folder="folder"):
+    """
+    Downloads the content of an url to a folder. Modified from \
+    https://github.com/pyg-team/pytorch_geometric/tree/master/torch_geometric
+
+    Args:
+        url (string): The url of target file.
+        folder (string): The target folder.
+
+    Returns:
+        string: File path of downloaded files.
+    """
+
+    file = url.rpartition("/")[2]
+    file = file if file[0] == "?" else file.split("?")[0]
+    path = osp.join(folder, file)
+    if osp.exists(path):
+        print(f"File {file} exists, use existing file.")
+        return path
+
+    print(f"Downloading {url}")
+    os.makedirs(folder, exist_ok=True)
+    ctx = ssl._create_unverified_context()
+    data = urllib.request.urlopen(url, context=ctx)
+    with open(path, "wb") as f:
+        f.write(data.read())
+
+    return path
+
+
+def load_jsonl(
+    file_path,
+):
+    list_data_dict = []
+    with open(file_path, "r") as f:
+        for line in f:
+            list_data_dict.append(json.loads(line))
+    return list_data_dict
+
+
+
+@torch.no_grad()
+def greedy_generate(model, tokenizer, input_ids, past_key_values, max_gen_len):
+    outputs = model(
+        input_ids=input_ids,
+        past_key_values=past_key_values,
+        use_cache=True,
+    )
+    past_key_values = outputs.past_key_values
+    pred_token_idx = outputs.logits[:, -1, :].argmax(dim=-1).unsqueeze(1)
+    generated_ids = [pred_token_idx.item()]
+    pos = 0
+    for _ in range(max_gen_len - 1):
+        outputs = model(
+            input_ids=pred_token_idx,
+            past_key_values=past_key_values,
+            use_cache=True,
+        )
+        past_key_values = outputs.past_key_values
+        pred_token_idx = outputs.logits[:, -1, :].argmax(dim=-1).unsqueeze(1)
+        generated_ids.append(pred_token_idx.item())
+        generated_text = (
+            tokenizer.decode(
+                generated_ids,
+                skip_special_tokens=True,
+                clean_up_tokenization_spaces=True,
+                spaces_between_special_tokens=False,
+            )
+            .strip()
+            .split(" ")
+        )
+
+        now = len(generated_text) - 1
+        if now > pos:
+            print(" ".join(generated_text[pos:now]), end=" ", flush=True)
+            pos = now
+
+        if pred_token_idx == tokenizer.eos_token_id:
+            break
+    print(" ".join(generated_text[pos:]), flush=True)
+    return past_key_values
+