Interview with Ray Dillinger

[Note: the 10th anniversary of the Bitcoin whitepaper is this month.  Below is a detailed interview with one of the first individuals to have interacted with Satoshi both in public and private: Ray Dillinger.

All of the written responses are directly from Ray with no contributions from others.]

Logo from 2010: Source

Q1: Tell us about yourself, what is your background?

A1: I am originally from Kansas.  At about the same time I entered high school I became interested in computers as a hobbyist, although hobby computers were still mostly useless at that time.  I got involved in early BBS systems when DOS hadn’t been released yet, modems were acoustically coupled and ran at 300 bits per second or slower, and software was stored mostly either on notebook paper or cassette tapes.

The early interest in computers is part of my lifelong tendency to become deeply involved in technology and ideas that are sufficiently interesting. This has led me to develop interests, obsessions, and expertise in a huge variety of things most of which the public does not discover reasons to care about until much later.

I graduated from KU with a degree in Computer Science in December of 1995 after spending far too long alternating between semesters of attending classes and semesters of working to pay for classes.

After graduation I moved to the San Francisco Bay area.  I worked for several AI startups in the next seven years and hold a couple of patents in natural-language applications from that work.  After that, I worked the night shift for FedEx for some years while doing occasional security consulting gigs during daytime hours.  I am currently doing AI algorithm research and implementation (and some cryptographic protocol/document design) at a FinTech startup.  I work on General AI projects on my own time.

I am somewhat pessimistic by nature and tend to assume until given reason to believe otherwise that anyone trying to sell me something or convince me of something is a scammer.  I know that’s irrational, but knowing doesn’t make the belief stop.  I have an abiding hatred of scammers and find them viscerally disgusting.

I consider making noise to be rude, avoid crowds and public appearances, and distrust anyone speaking faster than they can think.  Although I write a great deal, I rarely speak and strongly dislike talking on the phone.

In spite of my peculiar interests and asocial tendencies, I somehow managed to get married to a wonderful woman who tolerates an unbelievable degree of geekdom in an unbelievable variety of subjects, ranging from mild interest to full-on mad scientist levels in scope. I am tremendously thankful to have her in my life, and to whatever degree
I might be considered social, she deserves most of the credit.

I became marginally involved with Bitcoin in its early development because cryptocurrency, and the application of block chains to cryptocurrency in particular, are interesting.  I ceased to be involved in Bitcoin when the next steps would necessarily involve salesmanship, frequent talking, and social interaction, because those things are not interesting.

Q2: Perry Metzger created the now infamous Cryptography mailing list years ago.  When did you join and what made you interested in cryptography?

A2: I joined so many years ago it’s hard to remember.  It was pretty much as soon as I became aware of the list, but I’m sure it was more than fifteen years ago. It may have been late 2001 or early 2002.

I think I may even have been one of the first twenty or thirty posters on that list – it was still very young.

I remember being vaguely annoyed that it hadn’t been available when I was actually still in college and doing a crypto project in a grad-level networking course – I’d been a member of the even-earlier ‘cypherpunks’ list back when I was in school, but its strident political ideologues (including a guy named Hal Finney, whom you’ve probably heard of)
annoyed me, even back then.

‘cypherpunks’ was where I became aware of and started corresponding with Hal.  Although, way back then, I think we were both mostly annoying to each other.  And possibly to others as well.  Hal had been stridently political all the way from those days (and probably before) to the day he died, and in retrospect, I think I really needed some ‘remedial human-being lessons’ and some wider education at the time.  I’ve learned a lot since then – and perspective outside the narrow specialties we studied in school really does matter.

Q3: There were a lot of other non-cryptocurrency related discussions taking place simultaneously in November 2008 and many of the frequent posters didn’t comment on Bitcoin when it was first announced.  What interested you in it?  How involved would you say you were with providing coding suggestions prior to the genesis block that following January?

A3: I was interested in it for several reasons.  First, Bitcoin was a digital cash protocol, and digital cash protocols have some significant challenges to overcome, and I’d been interested in them for a long time already.  I’d even designed a couple by then.  The first I designed was unsound. The second, which is the only one worth talking about, which I’ll talk more about below.

Second, Bitcoin used a central proof chain (which we now call a block chain) as means of securing the history of each note, and I had known for a long time that any successful digital cash protocol had to use proof chains in some form or it couldn’t circulate (couldn’t be spent onward by someone who’d been paid in it).  And I was very, very much interested in proof chains, especially for a digital cash protocol.  I had already used proof chains (very differently) for a digital cash protocol when I extended Chaum’s e-cash protocol in 1995.

(see Digression #1 below to understand the differences between my protocol and Satoshi’s, and their effect on protocol design.)

Third, Satoshi eventually convinced me that he wasn’t a scammer.  I’m sort of a natural pessimist at heart, and digital cash protocols have a long history of scammers, so at first I had assumed the worst.  I think a lot of others also assumed the worst, which would be why few of them responded.  I made my first couple of replies without even having read it yet, to see how he responded before I wasted mental effort on something that would probably turn out to be a scam.

When I finally bothered to actually read the white paper, and spent the mental effort to understand it, I realized that (A) it wasn’t the usual incompetent bullshit we’d seen in far too many earlier digital-cash proposals, and (B) Its structure really and truly contained no Trusted Roles – meaning the opportunity to scam people was NOT built into the structure of it the way it had been with e-gold, e-cash, etc.

Fourth, and absolutely the clincher for me; it was very very INTERESTING!  It was an entirely new paradigm for a digital cash protocol, and had no Trusted Roles!  Nobody had EVER come up with a digital cash protocol having no Trusted Roles before!

Of course it wasn’t a “serious” proposal, I thought. It wouldn’t work for any kind of widespread adoption (I thought at the time) because of course people would conclude that spent hashes which absolutely couldn’t be redeemed for the electricity or computer power that had been used to create them were valueless.  And it would never scale beyond small communities or specialized applications of course because of its completely stupid bandwidth requirements.

But it was INTERESTING!

I could never have come up with Bitcoin because of the tremendous bandwidth.  Without Satoshi’s proposal, the idea of transmitting every transaction to every user would just have bounced off my mind as inconceivable.  Hell, I didn’t even understand it the first couple of times through the white paper because I was looking for ANY WAY AT ALL to parse those sentences and ‘transmitting every transaction to every user wasn’t even a POSSIBLE parse for me until Satoshi explicitly told me yes, that really was what he meant.

When I finally understood, I started doing math to prove to him that it was impossible, tried to relate bandwidth to rate of adoption and got a largest possible answer that’s only about one-eighth of today’s number of nodes.   I was assuming transaction volume proportional to userbase, which would be at least three times the transactions that today’s blocksize-limited block chain handles, and looking at a version of the protocol which doubled it by transmitting every transaction twice.  So,GIGO, I was wrong – but for good reasons and in the correct order of magnitude anyway.

But that was a couple orders of magnitude larger than the highest answer I had expected to get!  And that meant Satoshi’s idea actually seemed…. surprisingly plausible, if people really didn’t care about bandwidth.

The fact that bandwidth seemed to be available enough for the proposal to be technically plausible was sort of mind-boggling.  So was the idea that so many people did not care, at all, about bandwidth costs.

(See digression #2 to understand why it was hard for me to accept that
people now consider bandwidth to be valueless.)

Anyway, problems aside, it was INTERESTING! If the proof-of-concept actually sort-of worked at least on scales like for a campus or community merchandise token or something it would extend our understanding of protocol design!

What I had done back in 1995 had been INTERESTING for a different reason. At that time nobody had ever come up with a digital cash protocol that allowed people who’d been paid digital coins to respend them if they wanted instead of taking them right back to the issuer.  Of course it wouldn’t work for general adoption because of its own problems, but it had extended our understanding of protocol design back then, so back then that had been INTERESTING!

And before that, Chaum had demonstrated a digital cash protocol that worked at all, and at the time that was INTERESTING!

And in between a whole bunch of people had demonstrated ways to cooperate with bankers etc to have different kinds of access to your checking account or whatever.  Some of those had had privacy features v. the other users, which were also INTERESTING!

And so on.  I was very much looking at things that improved our understanding of digital cash protocols, and had no idea that Bitcoin was intended for widespread release.

Anyway, Satoshi and I talked offlist about the problems, and possible solutions, and use of proof chains for digital cash, and my old protocol, and several previous types of digital cash, and finally he sent me the proof chain code for review.

And the proof chain code was solid, but I freaked out when I saw that it used a Floating Point type rather than an Integer type for any kind of accounting. Accounting requirements vs. floating point types have a long and horrible history.

So that prompted some more discussion. He was designing specifically so that it would be possible to implement compatible clients in languages (*cough* Javascript *cough*) in which no other numeric type is available, so he wanted to squish rounding-error bugs in advance to ensure compatibility.  If anybody gets different answers from doing the same calculation the chain forks, so it’s sort of important for everybody to get the same answer.  Because Javascript clients were going to use double float, and he wanted them to get the same answer, he was going to make sure he got correct answers using double floats.

He was trying to avoid rounding errors as a way of future-proofing: making it completely consistent so clients with higher-precision representations wouldn’t reject the blocks of the old chain – but on the ground he wanted to be damn sure that the answers from Javascript clients, which *would* by necessity use double float, could be compatible with checking the block chain.

The worst that could happen from a rounding error, as long as everybody gets the *SAME* rounding error, is that the miner (whose output is unspecified in the block and defined as “the rest of the TxIn values input”) gets a few satoshis more or less than if the rounding error hadn’t happened, and no satoshis would be created or destroyed.

But if people on different clients get *DIFFERENT* rounding errors, because of different representation or differently implemented operations, the chain forks. And That Would Be Bad.

I would have said *screw* Javascript, I want rounding errors to be impossible, and used integers.  If the Javascripters want to write a float client, they’d better accept accurate answers, even if they have to allow for answers different than their code generates.  And if they make transactions containing rounding errors, let everybody in the universe reject them and not allow them into blocks.  But that’s me.

It was when we started talking about floating-point types in accounting code that I learned Hal was involved in the effort. Hal was reviewing the transaction scripting language, and both the code he had, and the code I had, interacted with the accounting code. So Satoshi brought him in for the discussion on floating point, and both of us reviewed the accounting code. Hal had a lot of experience doing exact math in floating point formats – some of his crypto code in PGP even used float types for binary operations. So he wasn’t as freaked-out about long doubles for money as I was. We talked a lot about how much divisibility Bitcoins ought to have; whether to make ‘Satoshis’ an order of magnitude bigger just to have three more bits of cushion against rounding errors, or keep them near the limit of precision at 10e-8 bitcoins in order to assure that rounding errors would always fail. Failing, immediately, detectably, and hard, at the slightest error, is key to writing reliable software.

So I went over the accounting code with a fine-toothed comb looking for possible rounding errors.  And I didn’t find any.

Which is more than a little bit astonishing.  Numeric-methods errors are so ubiquitous nobody even notices them.  Inevitably someone multiplies and divides in the wrong order, or combines floats at different magnitudes causing rounding, or divides by something too small, or makes equality comparisons on real numbers that are only equal 65535 times out of 65536, or does too many operations between sequence points so that they can be optimized differently in different builds, or uses a compiler setting that allows it to do operations in a different sequence, or checks for an overflow/rounding in a way that the compiler ignores because it can prove algebraically that it’s “dead code” because it will never be activated except in case of undefined behavior (like eg, the roundoff or overflow that someone is checking for)!  Or SOMETHING.  I mean, in most environments you absolutely have to FIGHT both your language semantics and your compiler to make code without rounding errors.

Clearly I hadn’t been the first pessimistic screaming hair-triggered paranoid aware of those issues to go over that accounting code; I could not find a single methods error.  The ‘satoshi’ unit which is the smallest unit of accounting, is selected right above the bit precision that can be handled with NO rounding in the double float format, and every last operation as far as I could find was implemented in ways that admit no rounding of any bits that would affect a unit as large as a satoshi.

To cause rounding of satoshis in the Bitcoin code, someone would have to be adding or subtracting more than 21 million Bitcoins (I think it’s actually 26 million, in fact…).  So, the Bitcoin chain is, I believe, rounding-free and will continue to check regardless of whether clients use any higher floating point precision.

For comparison Doge, which has so many coins in circulation that amounts larger than 26 million Doge are actually transacted, has rounding errors recorded in its block chain.  If a new client ever uses a higher-precision float format, their old chain won’t check on that client.  Which would be seen as a bug in the new client, and “corrected” there (by deliberately crippling its accuracy when checking old blocks). In fact it’s a bug in the Doge coin design which will never be fixed because they’ve already committed too much to it.

Integers.  Even with code that is meticulously maintained and tested for consistency, even where methods errors have been boiled out by somebody’s maniacal obsessive dedication, Integers would have been so much cleaner and easier to check.

Digression #1:
Why I was VERY interested in proof chains and digital cash protocols.

When I extended Chaum’s protocol in 1995, I had used proof chains attached to each ‘coin’, which grew longer by one ‘link’ (nowadays we say ‘block’) every time the coin changed hands. That allowed coins to circulate offline because all the information you needed to make another transaction was in the chains attached to the individual coins.  In order to make it possible to catch double spenders, the ‘links’ contained secret splits which, if two or more contradictory links were combined, would reveal the identity of the spender.

So, it could circulate offline and make transfers between users who weren’t even connected to the Internet. It didn’t have the ferocious bandwidth expense and even more ferocious proof-of-work expense of Bitcoin. Double spenders couldn’t be caught until the differently-spent copies of a coin were compared, potentially after going through several more hands which meant you had to have some kind of resolution process. And a resolution process meant you absolutely had to have a Trusted Certificate Authority with a database that could link UserIDs to RealWorld IDs in order to figure out who the RealWorld crook was.

Buyer and seller had to have valid UserIDs issued by the Trusted CA, which were known to each other even if to no one else.  And although not even The Trusted CA could link UserIDs and transactions except in case of a double spend, the parties to each transaction definitely could. Either party could later show and cryptographically prove the details of the transaction including the counterparty’s UserID, so your transactions were “Private”, not “Secret”. Finally, the ‘coins’ were non-divisible meaning you had to have exact change.

It was, at best, clunky compared to Bitcoin, and not being able to identify double spends until unspecified-time-later would probably be a deal-killer for acceptance. But it also had some advantages: It didn’t create a central permanent ledger that everybody can datamine later the way Bitcoin does, so Trusted CA or not it might actually have been better privacy in practice.  It was completely scalable because no transaction needed bandwidth between anybody except buyer and seller. And it had no proof-of-work expense.  But it needed a God-Damned Trusted Certificate Authority built directly into the design, so that CA’s database was open to various kinds of abuse.

Digression #2:

I had no comprehension of modern attitudes toward bandwidth costs.

I mean, I knew it had gotten cheap, but it was still taking me hours, for example, to download a complete Linux distribution. I figured other people noticed big delays like that too, and wide adoption of Bitcoin would mean slowing down EVERYTHING else they (full nodes anyway) did.  I just hadn’t understood that – and still have trouble with – the idea that by 2008, nobody even cared about bandwidth any more.

I got my first computer, because at that time privately owned computers were INTERESTING!  So I had to, even though they were also mostly useless.  (See a pattern here?)

But at that time, computers were not communications devices.  At All. If you hadn’t invested in something called a “LAN”, which anyway could only work inside one building, probably cost more than the building itself, and was useless unless you’d also invested in multiple computers, you moved data back and forth between your machines and your friends’ machines using cassette tapes.  Or, if and your friend were both rich enough to buy drives, or had been lucky enough dumpster diving to get drives you could repair, and had access to the very expensive media through some kind of industrial or business supply place, you might have done it using floppy disks.  Which held eighty kilobytes.

I got my first modem a few years later, and modems at the time were flaky hardware only BARELY supported by single-tasking systems that had never been designed to handle any signal arriving anywhere at a time they did not choose.  If your computer didn’t respond fast enough to interrupts, a modem could crash it.  If you were running anything that didn’t suspend and resume its business correctly (and most things didn’t because they’d never had to before) or anything that was coded to use the same interrupt, the modem would crash it.  If the software on your end ever started taking too long to execute per input character, the modem would fill up the short hardware buffer faster than your software could empty it, and crash it.  If you transmitted characters faster than the software running on the remote system could handle them, you’d crash the remote system.  There were no error correcting protocols because none of us had the compute power to run them fast enough to avoid a crash at the speeds the modems ran.

And that modem couldn’t transmit or receive characters even as fast as I could type. Sometimes you could crash the remote system just by accidentally typing too fast for a minute or two.

Computer security wasn’t a thing. Pretty much anybody you allowed to connect could at least crash your system and probably steal anything on your computer or delete everything on your computer if they really wanted to.   The host programs weren’t *intended* to allow that, but something as simple as transmitting an unexpected EOT signal could often crash them – sometimes crashing the whole machine, sometimes leaving the caller at the all-powerful command-line prompt. Stuff like that happened all the time, just by accident!  So people were understandably reluctant to let strangers connect to their systems.

There was one place in my whole state that I could call with it where I found people who’d leave a modem running on their machine despite the risk of crashes, and would allow a stranger on their system.  That sysop, in an act of sheer grace that he didn’t have to extend and which nobody was paying him for, allowed me to connect to it.  There were no such things as commercial providers; they could not exist until at least some system security actually worked.

There was barely even any commercial software: Every machine came with its own BIOS and Operating System, and the ONLY way to distribute a program that would run on more than a tiny fraction of systems was to distribute it as source code which people could tweak and fix and adapt in order to get it running, and commercial vendors didn’t want to distribute any source code.

So our software was all shared.  It came from fellow hobbyists, and unless we were physically in the same room to exchange media (and had the ability to read and write media compatible with the other’s systems), we could not share it without using bandwidth.

Long distance calls were over a dollar a minute, modems ran at 160 or 300 bits per second, and I could have burned through my entire monthly paper route income in under three hours.

Finally, every second I was connected to that remote system, that phone line was busy and everybody else couldn’t use it. And the other users needed it for reasons FAR more important than I did. They were military veterans, some of them profoundly not okay after Viet Nam, using it as sort of a hobby-mediated support group, and I was a fifteen-year-old kid hobbyist with a paper route.  Hobby in common or not, I had no illusions about the relative value of our access.   So I tried to be a good guest; I took my turns as fast as possible, at times least likely to conflict as possible, using as many pre-recorded scripts (played off a cassette tape deck!) as possible to waste absolutely no time, and got off.  I didn’t want to keep anybody out of something which was that important to them.

That’s the way things were when I started learning about the value of bandwidth.

No matter how much bandwidth I’ve got now, no matter how cheap it becomes, I’m still aware of it and it’s still important to me to not waste it.  I’ve sweated every byte every time I’ve designed a protocol.

And that’s why – to me anyway – universal distribution of a globally writable block chain is still amazing.  Just the fact that it’s now POSSIBLE seems incredible.

Q4: When Satoshi released the white paper, you had many public exchanges with her on that mailing list.  For instance, you asked her about inflation and Satoshi seemed to think that there could be some price stability if the number of people using it increased at the same level as the supply of bitcoins increased.  But, relative to the USD, there has never really been much price stability in its history to date.  Is there a way to re-engineer Bitcoin and/or future cryptocurrencies to do so without having to rely on  external price feeds or trusted third parties?

A4: Whoof…  that’s a hard question.  “Is not Gross Matter Interchangeable with Light?”  was considered impossible until Einstein figured it out. And the people who’d been asking that question didn’t even recognize or care about Einstein’s answer because his answer wasn’t about bodies and souls and the afterlife.  If the answer is ‘yes’ but the re-engineering involved changes the fundamental qualities that make you (or anybody) value cryptocurrencies, then is the answer really yes?

Satoshi tried to do it by anticipating the adoption curve.  We know how that turned out.

I think it’s fundamentally impossible to plot an adoption curve before launch.  I mean, I was the pessimist who assumed that there’d be a small group, formed early, that wasn’t going to be growing at all as these additional millions of coins pumped into that campus or that community economy.  So I figured, some initial value and rapid inflation thereafter.

Satoshi was far less pessimistic in figuring a widespread and fairly gradual adoption, and had picked the logarithmic plot to put coins into the economy at about the rate envisioned for adoption, assuming Bitcoin would follow a logarithmic adoption curve. It wasn’t a bad guess, as it’s a decent approximation to the Bass Diffusion Model, but the
parameters of the curve were completely unknown, and the Bass curve often appears after something’s been around a long time – not just when it’s launched.

Most importantly, nobody anticipated Bitcoin’s primary use as being a vehicle of financial speculation. The Bass Diffusion Model isn’t applicable to speculative commodities, because price changes in speculative commodities are responsive to PREVIOUS price changes in the speculative commodity.  That makes them nonlinear and chaotic.

And that, I think, is what it comes down to.  If people will be using something as a vehicle of speculation, then its price point is chaotic and defies all attempts to stabilize it by predicting and compensating for it.  So I think we need to abandon that notion.

You’ve already ruled out the idea of external price feeds and trusted third parties, because those would change the fundamental qualities that make you value cryptocurrencies.

That leaves internal price feeds:  If a cryptocurrency is used as a medium of exchange in other fungible assets, and those exchanges are recorded in its own block chain, then exchanges of crypto for dollars and exchanges of crypto for, eg, gold bars are visible in the block chain and could at least in theory be used to detect economic conditions and adjust the rate of issue of cryptocoins.

But the fly in that ointment is, again, the fact that the crypto is being used as a speculative asset.  People can read the block chain before the changes are made, anticipate what changes the code is about to make, and will front-run them.  Or, operating as “Sybil and her Sisters”, make a thousand completely bogus transactions in order to fool the software into doing something crazy.  Either way reintroducing positive feedback via market manipulation.

Most schemes aimed at stabilizing the value of a coin via any automatic means assume that the price can be changed by changing the rate of issue.  But the more coins are in circulation, the less possible it becomes for changes in the rate of issue to shift the price, meaning it devolves back to the first case of nonlinear and chaotic feedback.  IOW, the new coins being added represent a much smaller fraction of the available supply, and withholding them will affect almost no one except miners.

Honestly I’m very surprised Tethercoin isn’t dead yet.  What they propose, economically speaking, simply will not work.  They got themselves somehow declared to be the only way to get money OUT of a major wallet, which props up their transaction volume, but if the people haven’t already walked away with most of the money they’re supposedly holding but won’t say where, then I’m very surprised.

Q5: About a year ago you wrote a highly-commented upon, passionate retrospection published on LinkedIn.  You called out a lot of the nonsense going on then, is there anything that has been on your mind since then that you wanted to expand upon?

A5: Um.  Artificial Intelligence, Financial Markets, Human Brains and how they are organized, the nature, origins and mechanisms of consciousness and emotion, a generalization of neuroevolution algorithms intended to scale to recurrent networks of much greater complexity than now possible, scope of political corruption and the politics of divisiveness, gene migration and expression, the way cells control and regulate mutation in different kinds of tissues, directed apoptosis via a multiplicity of P53 genes as a preventive for cancer (happens naturally in elephants; easy to do with CRISPR; engineered humans would probably be radiation-resistant enough for lifetimes in space, or just plain longer-lived, or both), history of the Balkans, history of the Roman Empire, ancient religions, writing a science fiction novel ….

You know, things that are INTERESTING!  I actually _can’t_ turn my brain off.  It’s a problem sometimes.

I have had a few thoughts about cryptocurrencies, however, which is probably what you intended to ask about.

The first:

I have figured out how to redesign the cryptographically secured history database built by cryptocurrencies so that you don’t need any full nodes.  There are other ways to organize the blocks that give the proof property you need; They don’t have to form something that’s only a chain, and you don’t have to have specialized nodes for the purpose of holding them because everybody can hold just the blocks they need to show the validity of their own txOuts.

In order to verify the validity of any txOut, you need three things:  to see the block where it was created, to be sure that block is part of the same database as that proposed for the transaction, and to be sure that no block exists between those two in which that txOut was spent in another transaction.

Call it a “Block Hyperchain”, by reference to the N-dimensional hypercube it’s based on and the block chain it replaces.

I should be clear and say there are things it does and things it doesn’t do.  If your goal is to check all transactions, you’ll download a scattering of blocks for each transaction that soon add up to most of the block database, so someone who wants to check every transaction will rapidly accumulate the whole database.

But most users should be happy with just the few blocks they need to demonstrate the validity of the txOuts they hold, and it’s damn nice to be able to download a client, open it up, and just use it with minimal delay because someone offered to pay you bitcoins one minute ago and you want to be able to make sure the transaction he’s offering is valid RIGHT NOW, instead of waiting to accumulate the whole chain to check anything.

Suppose we pick a base, for convenience, of 10.  This helps make things easy to explain because we work with base-10 numbers, but we could have picked 16 and used hexadecimal for our explanations.

In a base-10 Block Hyperchain, every block that’s published has its own set of transactions, and the hashes of the blocks  10^N blocks ago for every integer value of N from N=0 to N <= log10 of block height.

Every block would record its own transactions, and also one list of destroyed txOuts per integer value N over the same range.

Each destroyed-txOut list would be all txOuts created in blocks whose block numbers match (modulo 10^N) the current block number, that have been destroyed in the last 10^N blocks.

Example:
If someone shows me a transaction seeking to spend a txOut, I want to check and see if it’s valid.  Ie, I want to see the block where it was created, and see evidence that it hasn’t been spent since.

So I can look at that txOut’s ID and know it was created in block 124. If the current block is 7365,  I get block 7365 and 7364 to make sure it hasn’t been spent in those, the same way we can do with a block chain.

Then I have a block whose last digit matches the last digit of the block where the txOut was created.  So I start checking the 10-block txOut-destroyed lists.  I check the list in block 7364 to make sure it wasn’t spent in blocks 7354 to 7363.

Then, jumping back by 10-block increments (relying on the second recorded hash in the header), I can check to make sure it hasn’t been spent in the previous ten blocks to each of blocks 7354, 7344, and 7334.  Then I get block 7324.

Now I’m at a block whose last 2 digits match the block where the txOut was created, so I can start checking the previous hundred blocks using the second txOut-destroyed list, and jumping back by hundred-block increments using the third recorded hash.  So I get blocks 7224 and 7124.

Finally, I’m at a block whose last 3 digits match the block where the txOut was created, so I can start jumping back by thousand-block increments, checking the thousand-block txOut-destroyed lists.  So I get blocks 6124, 5124, 4124, 3124, 2124, 1124, and finally 124.

So finally, I have a txOut created over 7200 blocks previous to the current block, and I have downloaded a total of 15 blocks to make sure that it was created in the same Hyperchain and hasn’t been spent since.

The number of blocks downloaded is proportional to the log base 10 of the number of blocks in the chain.

The blocks I’ve downloaded are larger because of the spent-txOut lists, but the spent-txOut lists have an average length that is the same regardless of the span of blocks they cover.  Lists that report transactions from a set 10x as long, only need to report individual transactions from that set 1/10 as often.

With more efficient access to the history database, it is possible to substantially raise transaction bandwidth.  People who make transactions during the next 7 blocks or so would need to see that block;  Later on, people who accept txOuts created during that block will need to see that block. And there’ll be about 49 blocks worth of txOuts,  scattered through the earlier history, that someone eventually has to traverse this block to verify.

All this means you have drastically smaller bandwidth requirements (remember I obsess on bandwidth costs?) for the same transaction volume but larger data-at-rest requirements (for any weirdo who for whatever reason feels like they need to collect the WHOLE database in one place, and why would anybody do that?) by a factor of seven.

And I keep thinking I’m going to do it, because it’s INTERESTING! And I ought to do it, because it’s VALUABLE!  But then I think about the current state of the cryptocurrency world and the quality of the people it would bring me into contact with and the ways people would try to scam with it and the number of people who’d find reasons to lie to me or about me, and then I get a sour stomach and go on to do something ELSE!

And feel vaguely guilty for not doing it, because it actually would be valuable.

It’s really hard for me to be motivated or enthusiastic about a cryptocurrency project, until the whole field is more full of people I’d be happy to interact and exchange ideas with and less full of ….  um.

The words that come to mind really shouldn’t be printed.  [This is fine meme]  I don’t mind if people know I’m sort of upset with the conditions and business ethics out there, or even that being so upset is literally preventing me from doing something useful.  But I’d rather not have it expressed in terms that are an incitement to violence.

Anyway, moving on;  In order to mine, someone would have to be able to see seven of the previous blocks; a different set of seven every time. But if I thought bandwidth was going to waste, that doesn’t even START to address the costs of hashing!  Deploying something that saves bandwidth without also figuring out a way to save hashing would fail to address a critical point.

So, I’ve had a bunch of thoughts about mining.  Most of which aren’t as interesting or valuable as the thought about how to organize the history database.  In favor of mining, it’s good that someone is able to join the network permissionlessly, help secure it, get paid, and initially get coin into circulation going from “none” to “some”.

My thoughts for securing a chain without proof-of-work are something I suppose I ought to call “Proof-of-Total-Stake.”

Congratulations!  This conversation with you got me to name it!  I had been calling it “proof-of-activity” but I see that name has acquired a much more specific meaning than it had when I started calling this by it, and no longer fits.

I still need to figure out what to call my revised structure for the block history database though.

Proof-of-Total-Stake  means measuring the priority of a fork by the total value of TxOuts that existed BEFORE the fork that have been spent AFTER the fork.  In other words, the total stake: how much of EVERYBODY’s money the blocks formed after the fork represent.  That is a well-founded mechanism for security that doesn’t involve trusted parties nor burning hashes.  It’s the only one I’ve come up with.  In the long run, unless somebody comes up with another fundamentally new idea, or accepts the idea at least of trusted block signers, that’s what I think a proper cryptocurrency would have to wind up with.

But there’s a problem with it.

Proof-of-Total-Stake, by itself, doesn’t provide an obvious way to determine who gets to form the next block – which can be a CRUCIALLY important security concern.

And Proof-of-Stake, including Proof-of-Total-stake, doesn’t handle the initial, permissionless, distribution of coins.  They can’t go from “none” to “some.”  They can only go from “some” to “some more.”

So I think it could only be deployed along with some kind of mining.

Q6: We first started interacting some four years ago when I was doing some research on dead cryptocurrencies, most of which were just direct clones or copies of Bitcoin.  At the time you were doing the heavy lifting categorizing how they died in a BitcoinTalk thread.  Today sites like Deadcoins.com have tried to do something similar.  Even though loud advocates at events like to claim blockchains ” live immutably forever” empirically there are probably just as many dead blockchains than living blockchains.  What do you think the top reason for why so many blockchains lose support to the point of death and do you think those reasons will change much in the future?

A6: By far the vast majority of those people were not doing anything INTERESTING!  A lot of the honest ones discovered that it was a lot of work and had other commitments in life.  A lot of the dishonest ones made their money and walked away leaving the  suckers behind.  A lot of people discovered that maintaining a codebase needed more programming chops than they actually possessed, and quietly withdrew from the field. A fair number ran into scammers and crooks whose utterly disgusting behavior left them convinced they wanted to do something else rather than meeting any more of those guys.

But the most important point? Hardly any of those coins was ever used in any transaction for an actual thing – not even an initial experiment like Laszlo’s Pizza.

Most of them were only ever mined by people who intended absolutely nothing beyond immediately converting them into Bitcoin, and only ever held by people who daily watched their value trying to guess the right time to sell them for Bitcoin.

It’s not so much that most of them *failed* – it’s more the case that the vast majority never even remotely began to *succeed*. There was no economic activity, meaning sales of merchandise or payment for work, that they facilitated.  Put bluntly, they just didn’t do anything beyond providing a temporary and completely discardable medium for speculation and scamming.  And, as surely as atomic decay, they got used, for that purpose only, and discarded.

Q7: Based on the original white paper, the intent of Bitcoin was to be an e-cash  payment system which could be utilized without needing to disclose a real identity to an administrator.  It seems that over time several different tribes have popped up, including those who market Bitcoin as a form of “e-gold.”  What do you think of the visible fracture that has occurred between the various Bitcoin tribes?  Does proof-of-work really act as a type of DRM for coin supply or do all the forks we have seen turn the advertised “digital scarcity” and “digital gold” into an oxymoron?

A7: That endless fight, starting with the block size fight, with everybody yelling and nobody listening, pretty much convinced me that the “community” which had grown around Bitcoin was in deep trouble.

The differences between the various proposed technical changes to the block chain, are far less important to the futures of those forks, than the integrity of the people who support and do business using them.

But the technical merits were never discussed by most. Instead, repetitive sound bites and slogans about them containing absolutely no new information were shouted.  Integrity was seldom displayed either. Instead, the fight was carried forward almost exclusively by partisans who had already decided what was the only possible solution that they would accept, and in many cases using tactics that inspire an absolute refusal to support their interests, or even participate in the communities where they are found.

If someone hires a troll army to attack a community by astroturfing fake support for something, can you respect that person?  If someone drives people who disagree away with personal abuse, is that a reasonable method for coming to an agreement about a protocol?  Is it a valid form of technical reasoning to launch a sabotage against a block chain based consensus mechanism?  What can you say about someone who buys existing accounts of users whom others trust in order to fake trusted support for their agenda? How about when it happens after those users whom others trust have been driven away or left in disgust?  Is it a respectful negotiator interested in the insights of others in solving a problem, whose negotiating skills include locking the damn doors and refusing to let someone leave the room until they get his signature on an “agreement” that they wrote without his knowledge before he even got there?

Is someone who would participate in a fight, on those terms, someone whose agenda or business interests you really want to support?  Hint: You already know that people who fake support for their agenda, or tell lies about other in order to discredit them, or who deliberately deceive others about the merits of their own proposal or others’ proposals, are doing business by means of fraud.  Do you want to carry on until the fraud is financial and the victim is you?

These factions had no interest whatsoever in reaching a consensus.  And nothing prevented each from implementing their idea and launching, with no hard feelings from anybody and no fight.  The only thing they were really fighting over was the name “Bitcoin,”  which was absolutely unrelated to the technical merit of any proposal.  And, to a first approximation, the other merits of having the name is a thing that none of them even mentioned during the fight.

Technically speaking, there is not much wrong with any of these forks. They address certain problems in different ways slightly favoring the interests of different groups, but not seriously to anyone’s disadvantage.  None of them was entirely without technical merit.

On the other hand none of them make more than a tiny amount of difference.  None helped with the bandwidth or transaction volume by anything more than a small constant factor, so the problem they were supposedly about solving was not in fact solved, nor even very much affected.

So while none of the proposed changes were objectionable in themselves, there was really no *very* compelling reason for any of them to be implemented.  Each of those ideas is merely a stopgap that pushes the rock down the road another foot or two without moving it out of the way. If you want to move that rock out of the road, you will need a much more powerful idea.

Q8: You’ve mentioned that limited supplies simply incentivizes hoarding which leads to low economic activity.  You have proposed a type of “proof-of-activity” replacement.  Can you expand more on either of these views?

A8: Suppose you have an economy that’s growing (more value is being created) but has a constant supply of coins.

In that case your coins represent, let’s say, one-millionth or so of the money that’s in circulation.

And, as the economy continues to grow, your coins will continue to represent one-millionth or so of the money that’s in circulation.  But that will be one-millionth or so of a lot more actual wealth.  In fact, your money, just sitting there in your wallet, is GUARANTEED to rise in value by the same fraction that the economy is growing by.  In our terms, this would be exactly the market average, as though you were holding stocks invested in ALL the businesses in your economy in proportion according to their  capitalization.  This is what index funds and IRAs make, mostly, but it’s making it with no risk.

Now, if you offer any investor a risk-free investment that’s guaranteed to make the same return as the market average, that investor would be mad to pass it up.  No investor is confident that she’ll beat the market average in any given year.  That’s why they call it “AVERAGE!”  And volatility – variance in return – is an unqualified bad thing because it will always take an 11% gain to make up for a 10% loss.  That money sitting right there in her wallet is the best investment she could possibly make.  There might be things that would make as much or more money, but all of them involve risk out of proportion to their marginal return.  Let other investors do that; they’re suckers and she’ll make the same money they do.

The problem with that is that the other investors are looking at the same question.  And reaching the same conclusion.  Why invest in companies doing anything productive, and expose yourself to risk, when you can make the same money just by holding your investment in your wallet?

And then who invests in the businesses that, if they were working, would actually create the value these people all intend to have some share in?

… (sound of crickets chirping) …  Suckers.

Suckers who lose more often than they win, because it takes an 11% gain to recover a 10% loss.  And the money the lose? Eventually trickles into the hands of the people who are hoarding it.

With no reason for investors to invest in business, the businesses eventually starve and the economy shrinks.  And all those coins that represent one-millionth of the economy’s wealth start representing one-millionth of less and less actual value.

This is what happened to ancient Rome.  They used metals (gold and silver and bronze) as currency, and their economy collapsed WHILE people had plenty enough money to keep it going!  Everybody stashed all their coins expecting to benefit later from prospering businesses, and the businesses, for want of capital, did not prosper.

Then the death spiral started: everybody stashed their coins waiting for the economy to come back so the coins would be worth their “real” value, and the economy never came back.  The coins were never worth their “real” value, until the people who remembered where the coins were buried had also been buried.

It’s a millennium-and-a-half later and we are STILL finding stashes of Roman coins!  The people who could have gotten their economy moving again, if they had EVER supported a business, instead buried their money in sacks.

The government tried to get it moving again, or pretend for a while that it hadn’t collapsed, making coins with increasingly ridiculous adulterated alloys.  But that didn’t change the underlying dynamic.

The Gold bugs of course have all told each other a different version of this story, where the adulterated coins were the cause of the collapse rather than the increasingly desperate attempt to recover from it.  And it’s pointless to try to convince them otherwise; they believe they already know the only possible truth. But for those actually motivated to investigate, the chronology of the events is reasonably clear.

===============================================================

The next thing is about “Proof-of-Total-Stake”, which I guess is what I’m going to call this idea for securing the chain.

The fundamental idea behind Proof-of-Total-Stake is that the priority of any branch of a fork is the total amount of EVERYBODY’s money which that fork represents.  That means, coins generated in that fork and pre-existing coins brought into the fork by transactions.

Coins generated in a fork are the coinbase transactions; Coins moved into the fork from earlier parts of the chain are TxOuts from earlier in the block chain that have been spent during the fork.

But we have to know which BRANCH of the fork they were spent into. ie, someone trying to create a fork should not be able to stick transactions from the valid branch of the chain into it, or they can match the txOut spending from earlier in the chain.  This is the basic problem with most implementations of proof-of-stake, which some writers have called “nothing at stake.”   Whatever resource you are using to secure the chain is meaningless when it can be used to secure *BOTH* forks of the chain.

In order to prevent the replay attack, each transaction would have to “stake” a recent block, making a commitment to supporting only forks which include that block.  This adds a field to each transaction.

The new field would give the (hash) ID of a block, indicating that this particular transaction is not valid in any branch of the chain which does not include the staked block.

So, let’s say that two transactions “coffee” and “eggs” are made at the same time,  after the chain forks at block 50.  “Coffee” stakes block 48 and “eggs” stakes block 51A.

When “coffee” appears in block 51B, the total stake of fork B is increased by that amount; its weight counts toward that resolution of the fork.

Then “eggs” is added to block 52A, and can’t be placed in chain B because it staked a block doesn’t exist in chain B.  Now “eggs” counts as stake in favor of the A branch and “coffee” counts as stake in favor of the B branch.

But then “coffee” appears in branch 53A, where it is also valid because the same block 48 is behind both branches.  This cancels out its support for branch B, just by being equal – revealing that stake which can be used in favor of both chains counts for nothing.

Security happens because some finite resource (coins created before the branch point and spent in transactions that are staked after the branch point) is committed detectably and irrevocably to the support of one branch (by staking after the branch point), and cannot be used to support any other.

This is exactly what Bitcoin does with hashes:  Hashes per second and number of seconds spent hashing are finite.  Hashes are irrevocably used in support of one branch (because the hash preimage can never be made to match a different block).  And the fact that they are used to support a particular branch is detectable.

Well, strictly speaking there’s only one “detectable” hash in each block. All we know about the others is, on average, how rare that one “detectable” one was and therefore, on average, how many they must have been.

But it’s still the same basic criteria.  Some finite resource, committed detectably and irrevocably to the support of one branch, which cannot be used to support conflicting branches.  And proof-of-total stake says that resource is the amount of EVERYBODY’s coins that branch represents.

With transactions supporting the basic security of the chain, and the idea behind coinbases being that they are payment for providing chain security, we want our “coinbases” to reward the people who make transactions that stake recent blocks.

PoTS is strong in the long run, or when the chain is seeing a high volume of legitimate transactions, but has its own problems.

Transactions in most cryptocurrencies are a very bursty use of something with long latent periods.  Absent heavy transaction volume, you can’t really expect PoTS to definitively reject a branch in such a way that a crook couldn’t resurrect it with a very large spend.  If the crook has more coins than the difference in total-stake between the two forks, the crook could resurrect the “dead” fork.

This is why the “interest” payments (actually per-transaction coinbases of a particular sort) when a transaction staking a recent block are made. To encourage a fairly constant stream of transactions that support one particular version of the chain up to a very recent block.

But the peril with that is that you want to structure it in such a way that you don’t incentivize people to overwhelm your bandwidth by transferring every coin they own from their left pocket to their right every block either.  So the actual design would come down to some compromise between transaction fees, and interest payments on transactions staked in very recent blocks, where the breakevens represent the transaction volume you want.

And there are a couple of final things to address together.  First, PoTS, while it has a workable rule for figuring out which branch of forks is preferred, is pretty silent about who gets to form blocks and how.  Second, Interest on coins spent has the “nothing to something” problem where if you don’t have anything in the system to start with, you won’t have anything ever.  These are both classic problem with PoTS coins.  The final design has to include some additional kind of coin creation that doesn’t depend on previous holdings (even if it gets de-emphasized after a while) and some way to determine who forms the next block.

Q9: ICOs have been around in some form or fashion for about five years now.  What’s your view on these fundraising schemes?

A9: The SEC is bouncing on them pretty hard, and as far as I can see it’s pretty much deserved.  Everybody wants something they can freely trade on secondary markets, and sell on the basis of its future value, but they also want to lie about it by saying it isn’t a security.

It is a security.  If a security is sold by a company to raise money, but does not represent a bond (a promise to buy it back) nor a stock (a share in future earnings) then an investor is getting nothing for her money – except maybe a receipt for having made a donation.

Another investor (a “real” investor who knows and understands a broad market, not a speculator who made a lot of money by a couple of strokes of sheer luck) will not buy it from them, at any price.  Such a thing has only speculative value.

If something’s continued value depends on a company, but the company’s continued existence doesn’t depend on that thing having value, it would be an excellent thing to not buy.

And all of that, we can say without ever touching on ethics and business practices of the people who run them.  But when we do touch on the people who run them, the story gets worse.  Much worse.  Much, much worse.  In this most are following the path trod by Altcoins.  And racking up a very similar ratio of efforts that fail, or which never even start to succeed.

Q10: You have alluded to tokenized securities in the LinkedIn article as well as our correspondence, what is your take on this topic?  What are the advantages versus say, simply doing what Carta (formerly eShares) does?

A10: I would have to answer that admitting to some degree of ignorance about Carta.  As I remember eShares, it was very much a top-down stock and option management tool, in that a private company with (non-traded) shares typically uses it to keep track of who owns what – actually issuing assets or recording changes in their status, making info about them available for the holders but mostly just to view online.

What it does not do, as I understand it, is directly enable the shareholders to trade those shares or options with each other.  Nor does it handle securities involved with or created by more than one company at a time.  It is a management interface, not a market.

I envision a block chain – sigh, now I have to come up with a name again.  Phooey.  I never care about naming anything, and then someone wants me to talk about one of my ideas and I have to come up with a name for it on the spot.  Let’s go for the pun and call it the Stock Trading and Options CryptoAsset Keeper.  I could come up with  something even dumber, but for the sake of exposition, call it STOCK.

The idea is that STOCK would act both as a Transfer Agent (which Carta does) AND a market (which AFAIK Carta does not).   A company could issue securities such as stocks and bonds directly on the STOCK block chain (“cryptoassets”) and the block chain could record trades in those issues against its native cryptocurrency.  The benefit here is the clear record and history to keep track of all trades and the current disposition of all the different cryptoassets – the stocks, the bonds, and the “cash” used to trade in them, would all be on the chain.

As long as no off-chain assets like bushels of wheat or truckloads of sneakers need to be delivered, and dividends/prices/etc accruing to these instruments are paid out (or in) in the cryptocurrency, the block chain could then function directly as market, transfer agent, means of delivery, and payment channel.  The task of converting the cryptocurrency to and from actual fiat, and the heavily regulated business of delivering the fiat currency, could be left to already-established cryptocurrency markets.

Trading in stocks/bonds/etc is highly regulated, and debts (NEGATIVE amounts) can crop up unexpectedly when companies go south or options traders go bust. Stuff gets into the RealWorld quickly when someone has to be found for debt payments, served process, and/or prosecuted for fraud, etc.  So STOCK couldn’t be an  “anonymous/permissionless” chain, at least not for regulated trades.  Each person or entity authorized to actually make securities trades would have to have a vetted, verified ID as specified by KYC laws, and would have to sign each such transaction with a public/private key pair proving Identity.

From the point of view of investors, STOCK would be a very sluggish market – submit your trade, have a completely random execution window averaging ten minutes (or whatever) during which the price might change, then a whole block of transactions all fly past at once and everybody’s waiting for the next completely randomly-timed block.  On the other hand, you don’t need an agent, or a broker, or a company transfer agent, or a registrar, or a clearance period, or ANY of those people who normally collect fees on every trade.  You could actually have a market where the buyer and seller get the exact same price with no ‘float’ whatsoever.  And you don’t have to worry about what time it is.  NASDAQ closes at 5PM new york time, and then a whole bunch of “off-market,” “private,” and “over the counter”  trades that nobody but the insiders can participate in or see happen. But STOCK would go on making blocks twenty-four hours a day seven days a week.  Why should it ever stop?

The SEC would be all over it of course; they’d be sticking a microscope up the butts of everybody involved to make sure that there was absolutely no scamming the investors.  Which is, after all, their job. And they’d require KYC compliance, and a whole lot of other regulatory compliance.  But, y’know, that’s kind of how starting any _legitimate_ business in financial services works.  No need to feel special or particularly victimized about that.

And the regulators would need some privileged keys that could be used to “seize” assets when a court orders them to, as part of a settlement for fraud or theft or something.  And everything else.  There’s a great irony that they’re interested in nobody having the opportunity to scam the investors, but they structurally require, just to be able to do their fundamental mission, builtins to the protocol that if misused would allow somebody to scam the investors.

But once satisfied and functioning within the law, I think they’d welcome STOCK as something that puts down a visible, provable, inalterable, unfakeable history of all trades.

Q11: Is there any cryptocurrency you think could become widely used outside of geeks, cypherpunks, and ideologues?  If not, what would need to change and how?  Has any popular coin ossified to such an extent that it can’t meaningfully evolve?

A11: Homer Husband and Harriet Housewife want convenience and familiarity. Which is mostly about form factor and compatibility.  They do not want to deal with key management in any form.

To do that, you have to make a hardware wallet small enough to fit into a wallet or a purse.  It doesn’t have to be literally credit card sized, but couldn’t be much bigger.  It should be the size of a stack of five credit cards, at most.  Or maybe it gets stuck back-to-back onto their cell phone.  It has to have an end that acts like a chip card, or an edge that acts like a mag stripe, or both, so that it can interact with the grocery stores, auto shops, restaurants, etc that Homer and Harriet already do business with.

That’s very very important, because Homer and Harriet aren’t evangelists.  The mechanic they’ve been going to for fifteen years has never heard of cryptocurrency and is never going to deal with the inconvenience of getting set up to accept it.  He wants people to pay cash or pay with a card, and Homer and Harriet would NEVER consider arguing with him about it, don’t want to go to the effort of explaining it to him, and probably couldn’t explain it very well anyway.  If they have to do any of those things, that’s a deal-breaker.

After that you have to get your cryptocurrency onto the Plus or Cirrus network, using the same interface as a foreign fiat currency.  That would allow Homer and Harriet to automate the sale and exchange to whatever local people think is money, or the purchase and exchange to crypto, when they want to spend or accept stuff from that “card.”  This will mean that they get hit with some extra fees when they use it, but
those fees are both unavoidable if you want to be on those networks, and relatively familiar to them.

Finally, there’s that key management thing.  You could handle most of it by making the wallet do it.  But sooner or later, that hardware wallet is going to fall and bounce of the curb, and go crunch under the tires of a bus.  Or, you know, get dropped into the ocean accidentally, or just get lost.

Homer and Harriet are NOT willing to accept that this is not something they can recover.  The only thing that they accept not being able to recover, when they lose their wallet, is familiar, folding fiat currency.  And that’s why they don’t keep very much of folding fiat actually in their regular wallets.

If you do convince them that losing the wallet makes the funds unrecoverable, they will never want to have more than fifteen dollars on it, which will mean it isn’t useful.  So, your hardware wallet has to interact with SOMETHING that keeps enough information about what’s on it, to enable a new wallet to recover everything that got lost.

Q12: Mining farms, mining pools, and ASICs. Many accounts are that Satoshi did not anticipate the full industrial scale these would reach.  Do you agree with this?  What are your views on mining pools and ASICs as we know them know today (specifically as described by Eric Budish’s paper)?

A12: My first problem with ASICs is that they can be used for exactly two things:  Mining cryptocurrencies, and carrying out attacks on cryptocurrencies.

Every day of every year, people who own those enormous ASIC farms are deciding which is the most profitable use of them, on that day.

And the rewards for mining cryptocurrencies ratchet downward every couple of years.

That seems problematic.  I keep watching to see what emerges each time the reward ratchets down, but I haven’t seen evidence yet that any of the big ASIC farms have turned around on any large scale.

My second problem with ASICs is that they are sucking up ridiculous amounts of energy that can never be recovered or used for anything else. I don’t so much mind this when converting the energy into heat is actually useful – replacing electric heaters in the basement of a building with a bank of Antminers that use the same amount of power is
energy-neutral and helps secure the chain.

But that’s not what happens in huge ASIC farms.  All that heat is just waste. Nobody’s home is made more comfortable, no furnace’s power bill is alleviated, no greenhouses are enabled to grow food in the winter, nobody’s oven gets to bake bread with that heat, and all that energy is just plain gone.

The Bitcoin chain issues the same number of coins per day regardless of how much energy is spent; I’d like to think that spending a whole lot less of it, at least in ways where the heat produced isn’t useful, would be better.

But then we get back to the first problem;  If honest miners start spending a whole lot less on the energy costs of hashing, then there’s a whole lot of ASICs not being used, and the owners of those are going to be looking around making their daily decision about what’s more profitable….

So the logic finally does work out the same. Security requires the vast majority of those ASIC boxes to be in use mining.  It just seems such a colossal expenditure of power, and it might be that a different design could have achieved chain security without that global cost.

My third problem with ASICs is that they have become a way for their owners to steal money from the taxpayers in many nations.  Countries that mean to do a good thing for everybody, create “development zones” with subsidized electricity, paid for by the taxpayers of that country. And then people move in with ASIC farms to suck up that electricity which the public paid for, and convert it into bitcoins in their private possession.  These are business that employ very few people, drive the development of no other resources, and otherwise do pretty much nothing for the development of the local economy.  IOW, the taxpayers who paid for that electricity are definitely not getting their money’s worth in economic development.

My fourth problem with ASICs is that there really is no way to monitor centralization of hashing power.  People keep pretending that they’re tracking whether a 51% attack is underway, but I think most of them probably suspect, as I do, that what they’re really tracking is probably nothing more than whether or not the cabal of ASIC farm owners
remembered to configure that new warehouse full of machinery to use a different identifier.

In all fairness, this last thing results directly from anonymous, permissionless mining, which is something that was a very specific and very much desired part of Satoshi’s vision; he wanted anybody to be able to connect and participate, without any interference of a gatekeeper. But there can never be security from a Sybil attack when you don’t have any way of tracking RealWorld identities, and a “majority” can never be
relied on to be more than the front for some cabal or business interest, as long as a Sybil attack is possible.

And that was what Proof-of-work was supposed to prevent.  In those early days everyone was thinking of hashing power as a side effect of computing infrastructure that was likely to be there, or be useful, for other purposes when it wasn’t hashing.  And EVERYBODY has a use for warehouses full of computers, so it was easy to think that hashing power would remain at least somewhat distributed.  The idea that someone would amass enormous numbers of special-purpose machines which made every other kind of computer in the world utterly useless for mining and which are themselves utterly useless for any other job (except attacking the network), was not, I think, really considered.

Satoshi definitely understood, and planned, that there would probably be server farms devoted to mining and that economies of scale and infrastructure would eventually drive individuals with ordinary desktop machines out of the mining business by being more efficient and making it unprofitable for the less efficient machines.

But I’m pretty sure he didn’t think of miners in places with artificially low subsidized rates for electricity outcompeting all other miners because of that advantage, driving the concentration of the vast majority of hashing power into just one country where it’s subject to the orders and whims of just one government and a few businessmen who
pal around with each other.

So he probably figured, yes, there’d be a few dozen large-ish server farms and a couple hundred small-ish server farms, but I’m pretty sure he envisioned them being scattered around the planet, wherever people find it worthwhile to install server farms for other reasons.

I’m fairly sure Satoshi’s notion of the eventual centralization of hashing power didn’t really encompass todays nearly-complete centralization in a single country, owned by a set of people who are subject to the whims and commands of a single government, who very clearly know each other and work together whenever it’s convenient.

And I find it worrisome.

Those enormous mining farms, and the way economics drove them together, are a structural problem with converting electricity into security.

I am not comfortable with the implication that, for any Proof-of-Work block chain including Bitcoin, economics will eventually devolve to the point where, when Beijing says ‘jump’ the mining and security of that block chain says ‘how high?’

And that is one of the greatest reasons why I look around for a different means of securing block chains.

El Fin

External facing appearances for the final months of the year

The past several months have been pretty productive especially in terms of education.

For instance, my “Eight Things” article had over 100,000 views in its first week alone thanks largely to landing on the front page Hacker News and reshares on social media. I may write-up an article breaking down its reception at a later date.

And interestingly, one of my older articles from 2014 recently ended up on the front of /r/DataIsBeautiful generating 15k+ views over a couple of days.

Below are some of my outward facing appearances.  If you’re interested in chatting about the topics below, feel free to reach me via Post Oak Labs.

Interviews

Cited and acknowledged

Panels and presentations

A panel on smart contracts with industry developers and educators

Earlier today I participated in a virtual panel covering smart contracts called, “Let’s Talk Smart Contracts.”

The panel included: Adam Krellenstein (Counterparty), Oleg Andreev (CoreBitcoin), Pamela Morgan (Empowered Law), Stefan Thomas (Codius, Ripple Labs), Stephan Tual (Ethereum), Tim Swanson (Of Numbers), Yurii Rashkovskii (Trustatom) and it was moderated by Roman Snitko with Straight.

Below are some transcribed notes of my own statements.

Introduction starting at 09:06:

Hey guys, great to be here.  Thanks for the invite, thanks for organizing this.  So I’m here because you guys needed another white guy from Europe or something like that (that’s a joke).  So the definition I have of smart contracts, I have written a couple books in this space, and the definition I use is a smart contract is “a proposed tool to automate human interactions: it is a computer protocol – an algorithm – that can self-execute, self-enforce, self-verify, and self-constrain the performance of a contract.”  I think I got most of that definition from Nick Szabo’s work.  For those of you who are familiar with him, look up some of his past writings.  I think that the primary work he is known for is the paper, “Formalizing and Securing Relationships on Public Networks.”  And he is basically considered the [intellectual] grandfather of this space.  I’m here basically to provide education and maybe some trolling.

From 22:02 -> 24:15

I think I see eye-to-eye with Adam here.  Basically the idea of how we have a system that is open to interpretation, you do have reversibility, you do have nebulousness.   These are things that Nick Szabo actually discussed in an article of his called “Wet code and dry” back in 2008.  If you look back at some of the earlier works of these “cypherpunks” back in the ’90s, they talked about some of these core issues that Oleg talked about in terms of being able to mitigate these trusted parties.  In fact, if you look at the Bitcoin whitepaper alone, the first section has the word “reverse” or “reversibility” around 5 times and the word “trust” or “trusted” appears 11 times in the body of the work.  This was something that whoever created Bitcoin was really interested in trying to mitigate the need for any kind of centralized or third party involved in the process of transactions to reduce the mediation costs and so forth.

But I suppose my biggest criticism in this space, it is not pointed to anyone here in particular, is how we have a lot of “cryptocurrency cosplay.”  Like Mary Sue Bitcoin.  I’m not sure if you guys are familiar with who Mary Sue is: she is this archetype who is this kind of idealized type of super hero in a sense.  So what happens with Bitcoin and smart contracts is that you have this “Golden Age” [of Comics] where you had the limited ideas of what it could do.  Like Superman for example, when he first came out he could only jump over a building and later he was pushed to be able to fly because it looks better in a cartoon.  You have only a limited amount of space [time] and it takes too long to jump across the map.  So that’s kind of what I see with Bitcoin and smart contracts.  We can talk about that a little bit later, just how they have evolved to encompass these attributes that they’re probably not particularly good at.  Not because of lack of trying but just because of the mechanisms of how they work in terms of incentives for running mining equipment and so on.  So, again we can talk about that later but I think Adam and Oleg have already mentioned the things that are pretty important at this point.

40:18 -> 41:43

I’m the token cynic, huh?  So actually before I say anything, I would like to mention to the audience other projects that you might be interested in looking at: BitHalo; NotaryChains is a new project that encompasses some of these ideas of Proof of Existence created by Manuel Araoz, he is the one who did POE.  NotaryChains is a new project I think that sits on top of Mastercoin.  The issue that people should consider is that proof of existence/proof of signature: these are just really hi-tech forms of certification.  Whether or not they’re smart contracts I guess is a matter of debate.

There is another project: Pebble, Hyperledger, Tezos, Tendermint, Nimblecoin.  With Dogethereum their project is called Eris which apparently is the first DAO ever.  A DAO for the audience is a decentralized autonomous organization, it’s a thing apparently. SKUChain is a start-up in Palo Alto, I talk about them in chapter 16.  They have this interesting idea of what they call a PurchaseChain which is a real use-case for kind of updating the process from getting a Letter of Credit to a Bill of Lading and trying to cut out time and mediation costs in that process.  There are a few others in stealth mode.  So I really don’t have a whole lot to add with cynicism at this point, we can go on and come back to me in a little bit.

59:41 -> 1:02:35

The go to deficiency guy, huh?  They’re not really saying anything particularly controversial, these things are fundamentally — at least from an engineering perspective — could be done.  The problem though I think runs into is what Richard Boase discussed in — if listeners are interested — he went to Kenya and he did a podcast a few weeks ago on Let’s Talk Bitcoin #133.  I really recommend people listen to it.  In it he basically talks about all of these real world issues that run into this idealized system that the developers are building.  And as a result, he ended up seeing all of these adoption hurdles, whether it was education or for example tablets: people were taking these tablets with bitcoin, and they could just simply resell it on a market, the tablet itself was worth more than they make in a year basically; significant more money.  He talked about a few issues like P2P giving, lending and charity and how that doesn’t probably work like we think it does.

I guess the biggest issue that is facing this space, if you want issues, is just the cost benefit analysis of running these systems.  There is a cost somewhere to run this stuff on many different servers, there is different ways to come up with consensus for this: for example, Ripple, Stellar, Hyperledger, they’re all using consensus ledgers which require a lot less capital expenditures.  But when you end up building something that requires some kind of mining process itself, that costs money.  So I think fundamentally in the long-run it won’t be so much what it can do but what can it economically do.

So when you hear this mantra of let’s decentralize everything, sure that’s fine and dandy but that’s kind of like Solutionism: a solution looking for a problem.  Let’s decentralize my hair — proof of follicle — there is a certain reductio ad absurdum which you come to with this decentralization.  Do you want to actually make something that people are actually going to use in a way that is cheaper than an existing system or we just going to make it and throw it out there and think they’re going to use it because we designed [wanted] it that way.  So I think education is going to be an issue and there are some people doing that right now: Primavera De Fiillipi, she’s over at Harvard’s Berkman Center — she’s got something called the Common Accord program.  And also Mike Hearn; listeners if you’re interested he’s made about 7 or 8 use-cases using the existing Bitcoin blockchain including assurance contracts — not insurance contracts — assurance contracts.  And he’s got a program called Lighthouse which hopes to build this onto the actual chain itself.  So there are things to keep in mind, I’m sure I’ll get yelled at in a minute here.

1:23:58 -> 1:28:10

Anyone listening to this wanting to get involved with smart contracts: hire a lawyer, that’s my immediate advice.  I will preface by saying I don’t necessarily agree with policies that exist and so on; I don’t personally like the status quo but there is no reason to be a martyr for some crusade led by guys in IRC, in their little caves and stuff like that.  That’s not towards anyone here in this particular chat but you see this a lot with “we’re going to destroy The Fed” or “destroy the state” and the reality is that’s probably not going to happen.  But not because of lack of trying but because that’s not how reality works.

Cases right now are for example: DPR, Shavers with the SEC, Shrem now with the federal government, Karpeles [Mt. Gox] went bankrupt.  What’s ended up happening is in 2009, with Bitcoin for example, you started with a system that obviated the need of having trusted third parties but as users started adopting it you ended up having scams, stolen coins, people losing coins so you ended up having an organic growth of people wanting to have insurance or some way to mediate these transactions or some way to make these things more efficient.  And I think that it will probably happen — since we’re guessing, this is speculative — I think that this will kind of happen with smart contracts too.  That’s not to say smart contracts will fail or anything like that.  I’m just saying that there will probably just be a few niche cases initially especially since we don’t have much today, aside I guess from Bitcoin — if you want to call it a smart contract.

What has ironically happened, is that we have created — in order to get rid of the middlemen it looks like you’ve got to reintroduce middlemen.  I’m not saying it will always be the case.  In empirical counter-factual it looks like that’s where things are heading and again obviously not everyone will agree with me on that and they’ll call me a shill and so on.  But that’s kind of where I see things heading.

I have a whole chapter in a book, chapter 17.  I interviewed 4 or 5 lawyers including Pamela [Morgan] of different reasons why this could take place.  For example, accredited investor — for those who are unfamiliar just look up ‘accredited investor.’  If you’re in the US, in order to buy certain securities that are public, you need to have gone through certain procedure to be considered a ‘sophisticated investor.’  This is one of the reasons why people do crowdsales outside of the US — Ethereum — because you don’t want to have to interact with the current legal system in the US.  The reason I mention that is because you end up opening yourselves to lawsuit because chains — like SWARM — cannot necessarily indemnify users.  That’s legal terminology for being able to protect your users from lawsuits from third parties; they just do not have the money, the revenue to support that kind of legal defense.  Unlicensed practice of law (UPL) is another issue.  If you end up putting up contracts on a network one of the issues could be, at least in the US, are bar associations.  Bar associations want to protect their monopoly so they go after people who practice law without a license.  I’m not saying it will happen but it could happen.

My point with this is, users, anyone listening to this should definitely do your due diligence, do your education.  If you plan to get involved with this space either as an investor or developer or so on, definitely at least talk to a lawyer that has some inkling of of an idea [on this].  The ones I recommend, in addition to Pamela here are: Ryan Straus, he is a Seattle-based attorney with Riddell Williams; Austin Brister and James Duchenne they’re with a program called Satoshi Legal; and then Preston Byrne, who’s out in London and he’s with Norton Rose Fulbright.

1:52:20 -> 1:54:43

Guys look, I understand that sounds cool in theory and it’s great to have everything in the background, but the reason you have to see these “shrink wrapped” EULAs [end user license agreements] and TOSs [terms of service] is because people were hiding stuff inside those agreements.  So if you hide what’s actually taking place in the contract you end up making someone liable for something they might not actually agree to.  So I’m not sure, I think it’s completely debatable at this point.  If we’re trying to be transparent, then you’re going to have to be transparent with the terms of agreement.

I should point out by the way, check out Mintchalk.com, it’s run by guys named James and Aaron in Palo Alto, they’re doing contract building.  ACTUS is a program from the Stevens Institute, they’re trying to come with codified language for contracts.  Mark S. Miller, he’s got a program over at Google, he does something with e-rights.

I mention all of this because, we already have a form of “polycentric law” if you will in terms of internationally with 200 different jurisdictions vying for basically jurisdiction arbitrage.  Ireland and the Netherlands have a tax agreement that Facebook, Google, Pfizer they take advantage of.  It’s this Double Irish With a Dutch Sandwich.  In fact my own corporation is incorporated in Delaware because of the legal arbitrage [opportunities].  Obviously smart contracts might add some sort of new wrinkle to that, but people who are listening to this, don’t expect to be living in some Galt’s Gulch tomorrow or something like that.

For example, when you have something that is stolen, there is something called Coinprism which is a colored coin project.  They can issue dividends on stock.  The cool thing with that is, “hey, you get to decentralize that.”  The double-edged side of that is if that when that get’s stolen: people steal stuff like bitcoins and so forth, what happens to the performance of that dividend?  If the company continues paying that dividend in knowing that the person had been stolen from: if somebody stole from me and I tell the company, “hey, it was stolen” and they continue paying, then I can sue them for continuing to pay a thief.  If they stop paying then it defeats the purpose of decentralization because anonymity is given up, identity has taken place.  Obviously this moves into another area called “nemo dat” it’s another legal term talking about what can be returned to the rightful owner, that’s where the term “bona fide” comes from.  Anyways, I wanted to get that out there.  Be wary of disappearing EULAs, those have a purpose because people were being sued for hiding stuff in there.

2:10:05 -> 2:12:23

So I think everybody and all these projects are well-intentioned and have noble goals but they’re probably over-hyped in the short-run, just like the Segway was.  It eventually leads to some kind of burnout, or over-promise and under-delivering.  I’m not saying this will happen, I’m just saying it could happen.  I actually think the immediate future will be relatively mundane, such as wills and trusts kind of like Pamela was talking about.

One particular program is in Kenya there is something called Wagenitech which is run by Robin Nyaosi and he is wanting to help farmers move, manage and track produce to market to bypass the middleman.  That doesn’t seem like something really “sexy,” that doesn’t seem like the “Singularity” kind of thing that everyone likes to talk about.  But that is needed for maybe that particular area and I think we might see more of that along with PurchaseChain, NotaryChains, some of these things that we already do with a lot of the paperwork.

Again, blockchains and distributed ledgers are pretty good at certain things, but not everything.  It has real limitations that vocal adopters on the subreddit of Bitcoin like to project their own philosophical views onto it and I think that it does it a very big disservice to this technology long-term.  For example, LEGO’s can be used to make a car but you wouldn’t want to go driving around in one.  A laptop could be used as a paper weight but it’s not particularly cost effective to do that.  And so what I think we’ll end up running into a tautology with smart contracts, it’s going to be used by people who need to use them.  Just like bitcoin is.  So what we’re going to have is a divergence between what can happen, this “Superman” version of Bitcoin and smart contracts, versus the actual reality.

So for example, people say it’s [Bitcoin] going to end war.  You had the War of Spanish Succession, there was a Battle of Denain, a quarter million people fought that in 1712 and it was gold-based [financed by specie].  Everyone that says bitcoin is going to destroy fiat, if the state exists as it does today there’s always going to be these institutions and types of aggression.  I do think smart contracts do add collateral and arbitration competition and it does take away the problem of having trust in the system itself, but the edges are the kryptonite.  And always will be.  So we need to focus on education and creating solutions to real actual problems today with the actual technology and not just some hypothetical “Type 2” civilization where we are using [harvesting] the Sun for all of our energy.

On the ground relevant data and knowledge on the mainland

Shanghaiist recently did a must-read interview with Shaun Rein.  I had the fortune of interviewing him for my book (see Ch 4 & 13), but this new interview is even better/more detailed in my opinion.

He’s one of the few laowai that truly understands what is going on here, approaching the business atmosphere with a very balanced mindset.  Check out his book and if you’re looking for a consumer research firm, he happens to manage one (CMR) as well.

An Interview with Kenneth Walker, founder of Bubba’s Texas-Style BBQ

bubbas-sitelogoOn March 25, 2013 I took a taxi out to Hongqiao, a western district in Shanghai, to visit Bubba’s Texas-Style BBQ (now one of three locations).  Having grown up in the Lone Star state it is always interesting to meet fellow Texans in East Asia (there are not many of us… we do not have a reputation for being globe-trotters) and as it so happens the founder of Bubba’s is Kenneth Walker, from South Austin.

Kenneth has spent the last 16 years in Asia, 8 in Hong Kong and 8 now in Shanghai.  Prior to Asia he also had an 8 year stint in New York City, working as a PR specialist for Weber Shandwick, a large global PR firm.  And as he explained it to me, although he enjoyed cooking BBQ as a hobby throughout high school, it was not his intention to create his own barbecue joint in China, rather it was a matter of happenstance.

According to him, “While I was doing PR work in New York and Hong Kong, creating a BBQ restaurant was not something that I had really thought about.  It was not until much later, after I had arrived in Shanghai and sampled the local so-called “BBQ” that I realized there was an unmet market opportunity and I began putting together a business plan.  One of the initial challenges for me then and one of the challenges that all restaurant start-ups today must face in this city is finding a location.  When I finally decided to give this idea a try, I originally tried scouting locations whose tenants were already out of business.  The problem for me was two-fold: the first is that I had actually only been in the metro for less than a year, so I did not have a lot of connections throughout the city.  The second is that in contrast to other parts of the world, after a business closes shop, the ideal time to lease the real estate is not after it is fully closed down.  By then it is too late, as someone else always manages to make arrangements before you do – as I found time and again.  So I changed my strategy to look for establishments that were on their way down – that had seen better days.  The original Bubba’s here, was just that – a woman’s bar called Dragon Bar that was run by a couple of expat women.  A friend noticed the place and informed me about it.  So I stopped by one day and spoke to the lady who was thrilled to hear about the buyout idea.”

Kenneth was able to finally check something off the list: found the location.  And as I mention in Chapter 3, in busy areas like Raffles City (来福士广场), one of the most popular shopping malls in Shanghai, space may cost more than 15 RMB per square meter per day (a rate, which for the typical sit-down restaurant footprint can amount to several million RMB a year).  As a consequence several notable restaurants have closed this past year including Purple Onion, Funky Chicken, Public and The Fat Olive (yes those are real names, visit SmartShanghai for more).

What about supplies and decorations?  For anyone that has lived abroad for any long-term period, it can be difficult to find a number of creature comforts from back home.  And as is the case of football, to the chagrin of North American fans, sport memorabilia can be hard to find.  Yet walking around in Bubba’s, North Americans would feel at home with dozens of sport pennants and authentic jersey’s attached to the wall and even a trophy case with autographed footballs (including one from Tony Dorsett and Mack Brown).

Where did he get these?

“It started with a Michigan State fan a few years ago.  He gave me a Spartan pennant that I hung up on the wall.  And after other patrons saw this, they became riled up.  So I made it an open policy: if you bring it, I’ll hang it.  BYOB: bring your own banner.  So little by little I began to accrue what you see before you today.  Prior to this ambiance, all I had initially brought were a few of things I had with me: a flag of Texas, saddles and a golf club whose Tom Kite autograph was accidentally rubbed off later by a cleaning crew.”

Again, for disclosure purposes I will note that I do not own any equity in his firm.  I even paid for my own baby back ribs.  But as I later told my friends, these were the most authentic tasting Texas ribs I have eaten in this time zone and this is coming from a guy who growing up, regularly ate at Spring Creek and Dickies.  How did he do this?

“As I said, I had sampled the food of other local restaurants and what I found did not meet the standards I was used to back home.  And to make good BBQ you need to actually smoke the meat.  So I began looking for a smoker to import.  I had never done this before and I was unfamiliar with the import laws.  I did find a company in the US called Southern Pride who told me they could sell me one for $15,000 and that they had sold them abroad before.  After hearing they had exported 15 smokers to Asia back in the ‘90s, I then began trying to track down the one that ended up in Shanghai.  Somewhere in this city was a real smoker.  And as luck would have it, I found out that Hard Rock Cafe were the owners of the smoker – and that they closed the restaurant a few years earlier.  I got a hold of their previous GM and he told me that all of the store equipment was now in storage out by the airport and that it was all about to be removed in the next month.  So I drove out to this storage site, climbed around for a couple of hours with a flashlight in my hands and after crawling around old chairs, tables, automobiles and motorcycles, I found the smoker.  While it had been used as a rotisserie oven for chicken, it had never been used as a smoker, so fortunately it did not have any of the old smoker smell stained into it.  I was able to buy it for $2,000 and with the help of about 10 Chinese mover guys we put it onto a flatbed  truck and brought it to the now gutted Dragon Bar.”

While you may not be as fortunate as Kenneth was in acquiring kitchenware for your own restaurant, today would-be entrepreneurs have websites like Craigslist, Shanghai Expat and Delta Bridges to talk with others (both locals and laowai) to find equipment and potential store locations.

So Kenneth has found a location, a smoker and has a team to work with.  How did he turn it into the real deal?

“The next hurdle is finding some type of wood to smoke with.  Any fruit wood would work just fine, and I knew that apple wood was available as some of the Beijing duck restaurants were using it to smoke their ducks.   I found a local duck restaurant that used Applewood to smoke their ducks, and while they gave me a few sample pieces of wood, they would not divulge the supplier name or contact.  Later however, a friend I had placed in charge of logistics tracked down a company in Beijing that sells Applewood by the ton.  I was not very sure what a ton of wood looked like and was told they would deliver it in a 6-ton truck.  Thus I had a dilemma because I did not want to waste the storage capacity either.  At the same time I did not want to scare the neighbors or the authorities, if they saw a new business opening with piles of wood, what would they think?  So while I initially ordered a full 6 tons, at the last moment I cancelled half of the order.  When the truck finally arrived and we unloaded the wood, all 3 tons that were delivered ended up taking up half of the restaurant.  I got lucky because if I had ordered all 6, there would be no room in the restaurant.  Afterwards we manually moved each cord out behind the restaurant wall and this supply lasted for two-and-a-half years.”

Is the rest history?  Not quite, after all, even with everything in place you still need to cook a produce customers are willing to buy.

“Even though I knew how to cook BBQ, the first 3 months were hard.  I was here from 5am to midnight each day.  I eventually taught some local hires how to properly smoke meat and our initial menu included chicken, ribs and potato salad.  Sometimes, for recipes like a breakfast sausage that tastes like Jimmy Dean sausage for instance, I would do a lot of trial and error before finding the right combination of flavors that the customer was used to and wanted.  The restaurant also features my own secret barbecue cause and dry rub.  Later we expanded the menu to include burgers, pizza, seafood and Cajun style fish& chips.  We did have some Mexican food at one point but felt the menu was getting too cluttered so we culled it.  We might expand the menu again though, due to changing customer demand.”

What are some expansion possibilities and opportunities?

“I opened up the first Texas BBQ in all of China.  Because of my professional background I have done a lot of branding for merchandise like t-shirts and now social media.  I also organize an annual chili cookoff and an annual BBQ cookoff with multiple teams made up of local cooks around the metro.  Because the chili cookoff is an officially sanctioned event by CASI, the top 3 finishers earn automatic bids to the world championship held in Terlingua, Texas.  In addition, we have live bands at these events and more than 1,000 people attend, with about an 80% – 20% demographic split (80% foreigners, 20% local), which is about the same demographic customer base at Bubba’s locations.  We even work with Crown Relocations services to help manage the logistics of the chili teams, by picking up the kitchen equipment the day before and setting it up at the event grounds.”

CASI is the Chili Appreciation Society International and all Texans born north of the Guadalupe River are legally required to belong to it.   Crown Relocations services is 45 years old and according to its official FAQ, generates $766 million in annual revenue.

What are some of the challenges that business owners face?

“This can still be a challenging business culture because of the need to maintain guanxi, to maintain relationships with suppliers and various governmental bodies.  But larger cities like Shanghai, Beijing and Guangzhou have become increasingly business friendly.  There is a formal structure with forms and permits that have been streamlined over the years.  Potential business owners should also make sure to set-up a WFOE because it can be very risky creating a JV with a local partner like a friend or wife.  Too many risks associated with that.”

For more about guanxi Chapter 1 and WFOE (Wholly Foreign Owned Enterprise) see Chapter 10.

And other opportunities?

“While it will vary from location to location, but the team that gutted and renovated our first restaurant did so brick by brick.  They used the wood paneling from the bar to construct a ladder and reused the bricks to create a new wall in a different location.  This kind of resourcefulness occurs throughout many other construction sites all throughout the country.  And because of time saving methods like this, we were able to gut and then reopen the restaurant in about 3 months – a process which probably would have taken longer in other cities in the US.  Other opportunities that I see are a “dive” steak restaurant.  It is hard to find a really good steak for a decent price in this city.  When I was working in New York there were a couple of very inelegant restaurants that had nothing but graffiti on the walls, no frills or ambiance whatsoever.  The money you paid for the meat was used to buy the best meat for the money, so you got a great product, a delicious steak.  I think there is room for several of these “dive” restaurants throughout China, especially in bigger cities.  In addition, there are these pop-up trailers that are popular in certain cities of the US.  They are mobile units that cook ethnic food like from Vietnam and I have yet to see them here on the mainland, but could see them being very popular late at night and early in the morning near bars and clubs.”

For a review of these pop-up mobile food trailers in Austin see: Scrumptious Chef

With varnished bar stools hovering around wooden tables, a flat-screen TV with sports on in the background, sport pennants and white walls (as opposed to the pink and black décor of Dragon Bar), Bubba’s illustrates how a simple idea and bit of tenacity and luck can create a successful business in China.

For more info about Kenneth and Bubba’s BBQ see: Shanghai’s First Annual BBQ Cookoff: What’s Bubba got to say about it? from ShanghaiExpat and Austinite who introduced Shanghai to Texas barbecue to compete in Terlingua from Austin360

Chapter 3 – Food and beverage

[Note: below is Chapter 3 from Great Wall of Numbers]

There is a famous proverb in Chinese called min yi shi wei tian (民以食为天) – eating is as important as the sky.1 Despite regional differences in dialects, in regulations and in climate, one universal rule on the mainland is that eating is one of the most important activities of the day.  While this may sound like a fortune cookie truism (which by the way, do not exist in China – I have yet to see one), with recent memories of plagues and famines in their mind, many Chinese residents pay close attention to what their next meal will be.  And how it will be cooked and increasingly, where it was grown.

And while they may have a reputation for spending some of the highest amounts of their annual income on food (28% in China versus 10% in the US, this is called Engel’s Law)2 and are simultaneously highly elastic (e.g., if food prices increase they will switch over to cheaper substitutes), their perpetual gastronomical vigilance is not unwarranted.3

During the summer of 2008, China was faced with a series of nationwide milk powder scandals in which 300,000 babies and infants were poisoned and six died from a chemical called melamine which local producers had added “to save money.”456 As a consequence, there was a subsequent surge in the importation of milk powder from abroad including New Zealand and Australia.7 All told 80% of imported dairy products were from New Zealand in 2012.8 And in Australia, Victorian dairy farms recorded a record $144 million in exports to China in 2011.9 In fact, China is now the largest powder milk importer globally as many families – out of concern of repeat poisoning – currently place higher value on imported brands.1011 As a consequence, Tmall, the largest e-commerce platform on the mainland, has begun importing baby formula from companies such as Nestle and Danone to be sold directly to Chinese customers.12 And milk is not the only dairy product that Chinese consumers are importing.  For example, despite being a new commodity to the mainland $139 million worth of cheese was imported in 2011.13

And to “secure” these supply lines according to Financial Review, China Investment Corporation (CIC), one of the world’s largest sovereign wealth funds, is actively seeking agriculture investment opportunities in Australia and other countries.14 In fact, the Beidahung Group, the biggest Chinese agricultural conglomerate recently purchased and is leasing 100,000 hectares in Western Australia and plans to invest up to $4 billion in Australian agriculture.15

In addition to the melamine fatalities above there have been scandals involving “gutter oil,” whereby cooking oil is collected and dredged from restaurant drains; clenbuterol and other chemicals are added to meat to “enhance” the taste yet is toxic; arsenic in frozen calamari and even tainted steamed buns crop up throughout the year.16 It is not unsurprising then that according to a 2012 survey conducted by Horizon Research Consultancy, a Beijing-based polling company found that 41% of those surveyed said food safety was a major problem, up from 12% in 2008.17 One residual ramification from this stark rise in concern comes from a 2012 report from Ipsos, a market research company, which found that due to food scandals “more than 60 percent of people would choose foreign brands more often.”1819

At the same time, in 2010 China exported $41 billion in agriculture such as garlic and onions, yet even among some of this purportedly screened produce, there have been food safety issues.  For example, this past summer a batch of strawberries originally grown in Shandong ended up poisoning thousands of German schoolchildren.  While the investigation is ongoing, the initial findings were that some of strawberries may have been contaminated with norovirus.20

Not quite soylent green

In a bid to protect consumers, an independent, private consumer watch-dog group called Zhichuchuangwai was started in 2011 to chronicle all of the nationwide stories involving food and beverage containments.21 In addition, the world’s largest food retailers (hypermarkets) including WalMart, Tesco, Carrefour and Metro have spearheaded an independent non-profit quality assurance consortium called Global Food Safety Initiative to produce food safety guidelines, harmonize food safety standards and create a certification framework for suppliers and distributors.

As China develops and the middle class grows, the demand for higher quality goods – safe goods – has led to an increase in opportunities for foreign brands which are perceived as meeting the highest safety standards.  For example, the USDA trade office in Shanghai reported that “the 82 foreign hypermarkets [in Shanghai] accounted for 78.6% of the total hypermarket sales volume in 2008.”22 These same hypermarkets also contain an increasing amount of food products (60% as of 2009).  Collectively the mainland hypermarkets is a $81 billion industry and growing.23 Yet before you decide to jump in and create yet-another-hypermarket, this area is fraught with nebulous legislation.  For example, in an effort to “protect” small and medium suppliers, on December 19, 2011 five ministries and committees issued a joint regulatory plan whereby they arbitrarily removed hypermarkets from being able to charge delivery fees, slotting fees, holiday fees and several other fees.24 This caused a lot of confusion and as a consequence retailers were essentially forced to resign their previous contracts with local suppliers.  In addition, these foreign “big box” companies face zoning restrictions that prevent them from competing with domestic retailers like Suning or Gome.2526

For perspective in this segment, in 2010 the two largest retailers in China were Vanguard and Lianhua, both of whom are state owned enterprises (SOEs), and they both generated more than $10 billion in revenue.  For comparison, privately run RT Mart (owned by Sun Art), Carrefour and Walmart generated 25-35% less, yet with a fraction of stores compared with their domestic competition.  For instance, Carrefour generated almost $7 billion in revenue from 182 stores compared with Vanguard, who had 3,155 stores.  In addition, not all private companies perform the same.  In 2012 Tesco closed several outlets in August and scaled down its expansion plans and Carrefour’s same store sales declined 6.1% in Q3 2012 yet RT Mart has thrived and plans to open 105 hypermarkets in 2013.27

In my own anecdotal observations, these large foreign-owned hypermarkets are continuously filled by Chinese and expats alike, even though some of the products cost significantly more than locally owned supermarkets are charging.  For example, as I mention later in Chapter 6, I had a bad bout with food poisoning while in China during 2011 and as a consequence both my Chinese and American doctors recommended that it is better to be safe than sorry – better to spend more on safer food now than pay for it later as an in-patient.  So several times a week I shop at a nearby CityShop.28 CityShop is an up-and-coming, locally owned and expat managed supermarket with 10 stores in the Shanghai and Beijing areas.  More than 80% of their products are imported from the US, Germany, Australia and other developed countries.29 And like Walmart and Carrefour, CityShop is packed with shoppers – both Chinese and expats – throughout the day.

In March 2013 I spoke with Lawrence D’souza, customer service director for CityShop.  He is originally from Goa, India and has spent his career managing a variety of supermarket chains around the globe.  For the last three years he has worked on the mainland at his current position and in his view there are a couple of challenges that while not unique to China, can be a hurdle for entrepreneurs.

In D’souza’s words, “one of the issues that all grocery stores face when trying to import goods is changes in duties and clearing customs.  So for example, recently it has become difficult to import certain products like organic foods which require the CCC stamp from the government yet may not be approved for a variety of unstated reasons.  On the other hand, items like organic milk from Australia are typically allowed entry.  So supermarket owners need to keep up-to-date otherwise they will have backlogs in their supply chain.  Another reason this is important to a grocery store like ours has to do with location.  Each of our store locations is placed in a different neighborhood with differing demographics.  So for example, one location may cater to a client base that is comprised entirely of mainland Chinese.  Another location may have a mix of businesspeople from Japan, Hong Kong and Taiwan.  And yet other locations may have 50% of their customers that are expats from the West.  As a result, we have to stock each store with different items that the target market wants, thus the consistency and flow of items is different.  Which brings us back again to keeping up-to-date with changes in the import rules so that way we can remain flexible to the demands of our diverse customer base.”  As mentioned in Chapter 1, CCC stands for China Compulsory Certificate and is required in order to import any item.  Furthermore, import taxes such as those on wine or breakfast cereal can vary throughout the year and the list of which is maintained by the General Administration of Customs (海关总署).  Additionally, all organic foods imported or produced locally needs to be approved by the China Organic Food Certification Center (COFCC).30

At the same time D’souza sees several untapped opportunities in this segment.  According to him, “because of the rapid development in the last several decades and scarcity of shelf space, there is a product gap, a lack of certain flavors and tastes as some of the food that supermarkets import on the mainland may not necessarily be of interest to customers from India, Russia or Southeast Asia.  For instance, there are between 40 to 50,000 Indian businesspeople and traders that work in Shanghai throughout the year, yet finding specific food like basmati rice can be difficult to locate in the metroplex.  One example in our own stores is that based on customer feedback we added avocados and guacamole throughout the year due to the popularity of Mexican food.  So other entrepreneurs could likewise satiate demand from ethnic niches and become successful doing so.  On the other extreme is volume and scale: the bigger the store, the more products you can stock.  Entrepreneurs and managers could try to emulate big-box stores such as Costco which may be able to keep costs low and process bulk orders to cost conscious buyers – but this also has its own share of inherent challenges.”

Another challenge that D’souza and others that I interviewed raised that is not unique to China is fixed capital costs.  Or in other words, leasing real-estate.  In busy areas like Raffles City (来福士广场), one of the most popular shopping malls in Shanghai, space may cost 15 RMB per square meter per day.  It is significantly higher in the case of supermarkets, 10,000 to 15,000 RMB per square meter or up to 50% of the total operating costs may come solely from rent.  Entrepreneurs should also be aware of the gestation period in receiving import permits and licenses.  According to several businesspeople that I spoke with, it may take 6 or more months to have all the paperwork processed and approved (photocopied, translated, verified), especially if it is a new type of food or ingredient that has never been sold in China.

However, before you come away thinking that hypermarkets and supermarkets throughout China are filled row-after-row with imported goods, the same USDA report noted that even international retailers “typically carry less than 1% imported SKUs” and that “imports rarely constitute more than 5% of total SKUs” even in high profile stores in large cities like Shanghai and Beijing (an SKU is a stock-keeping unit).  One of the reasons why this is the case is that few of the retailers have been able to build out a distribution network (e.g., cold storage) that they already have in Western countries on the scale that KFC has managed to do (as I note in Chapter 16).  What this means is that by-and-large, these same retailers typically still source their food from local suppliers.  This presents an opportunity to foreign distribution and supply chain management experts – to bridge the wide gulf between an increasingly wealthier consumer that would like to purchase imported products for quality and safety reasons versus the amount of imported goods that retailers are able to continuously stock.

Another opportunity is for cold storage experts as well.  For example, PFS (Preferred Freezer Services) is a joint venture between an American company and a Dalian-based company (Yida Group).  During the next several years they will spend more than 7 billion RMB ($1.1 billion) to build out a cold storage network across China, including a 40,000 ton facility in Shanghai.31 And according to Li Wanqiu, head of Zhongde which is a cold-storage consultancy, “Beijing alone built over 50,000 square meters worth of new cold storage warehouses in 2011.”32 The reason these are needed is that according to Datamonitor, “China’s frozen food market grew 9.9 percent annually between 2004 and 2009.”33  One of the reasons for why this marked increase in consumption has occurred is that as an economic develops and urban residents work longer hours, they have less time to cook.  So the demand for quick, easy-to-make meals such as TV dinners (e.g., Hungry-Man) increases.  Can your firm take advantage of this opportunity?

You don’t need to bet the farm

 In Chapter 16 I discuss the failures and successes of foreign owned and operated restaurants (such as KFC and McDonald’s) but even with their entry there is potential for additional competition.  For example, in October 2012 I interviewed Glenn Wilkinson an Australian who has lived in China for the past 25 years.  He is a Senior Consultant at Beacon Consulting, a Shanghai-based firm that specializes in corporate training and HR staffing.34 In his mind, one of the biggest opportunities for both foreign and local entrepreneurs is in the “food industry,” a very vibrant and dynamic market.

What he means by food for example, are the restaurants I listed above.  One of the reasons this is a vibrant market is that according to the National Bureau of Statistics, the restaurant industry as a whole has been growing 14% annually since 2007 “reaching two trillion Chinese yuan (US $319 billion) in 2011 to account for 11.3% of all revenue in the consumer products category.”35 The food-service industry (e.g., catering) jumped to $99 billion in 2011 also up 14% from a year earlier.36 Furthermore, as noted by Wilkinson and others interviewed, there is a high demand for a quality product due in part to safety concerns and in part because of a growing middle class.  For comparison, retail sales in China have risen an average of 17% for each of the last five years and the luxury goods market (see Chapter 11) is expected to grow 20% a year for the next decade.37

Yet to be even handed, enthusiasm should be tempered due to policy changes from the top.  According to the China Cuisine Association (CCA) that due in large part to Xi Jinping’s (the new President of China) fight to stop waste of public money on extravagant meals, the CCA recently conducted a survey that found, “60 per cent of nearly 100 restaurants saw bookings cancelled recently, with one Beijing-based outlet reporting an 80-per-cent drop in sales.”38 Furthermore, “The survey found that business owners felt pessimistic about the outlook of the industry.  They think it’s necessary to readjust their business models to adapt to the new market conditions.”  Prior to the new national policy, the average monthly failure rate of restaurants was 15% in mid-2012.39 Thus the risks involved in setting up a new food & beverage establishment arguably have changed and that failure rate may increase in the short and medium term.

While each city has different rules regarding local partnerships and minimum registered capital requirements, the restaurant business is relatively open to market participation (e.g., certain districts in Shanghai require a $150,000 minimum in registered capital).40 While you would need to do your own fact-finding exploration to measure the return-on-investment, based on anecdotal evidence, it appears that mainland Chinese are apt to eat foreign food just as voraciously as they eat domestic food.

For example, in 2002, Element Fresh was founded by two foreigners, Scott Minole and Sheldon Habiger, has since opened 13 stores in Shanghai, Beijing and Guangzhou.  While a bit on the expensive side, in my mind the quality of the salads more than make up for the price.   In 1999, Bob Boyce from Montana co-founded Blue Frog with his business partner Kathleen Lau.41 I have visited two of its 9 locations (there are 6 in Shanghai and 3 in Beijing) in part because they have great specials on Monday’s and because the burgers are some of the best in the city.   Also in 1999, John Christensen of Denmark founded Wagas, a delicious sandwich shop that I have personally frequented several dozen times.  There are now 25 and counting Wagas locations in Shanghai alone and one in Beijing.

One of the reasons these Western-style restaurants are finding success is as I note in Chapter 15, you have several million Chinese residents who have lived, studied and worked overseas and some of them now enjoy Western food.  For example, according to one estimate by the government, 186,000 Chinese living abroad moved back to China in 2011.42 According to the China Tourism Agency, 70 million Chinese tourists traveled overseas in 2011 and an estimated 82 million traveled overseas in 2012 (who spent $98 billion).4344 McKinsey & Company estimates this number will climb to 94 million by 2015.45 And by 2020, Boston Consulting Group predicts that “China’s total outbound market [will] likely be three times as big as Japan’s.”46 In addition, nearly one million Chinese students now study overseas, including more than 190,000 in the US alone.47 Thus, the Chinese middle class is increasingly familiar with Western-flavors and styles.  And if my own anecdotal experience is any indication, restaurants like Wagas and Blue Frog, while popular with expats, are also quite popular with locals as well – some nights accounting for 90+% of the customer base.  Perhaps you could create a BBQ or Tex-Mex franchise, both of which there are currently few market participants.

Simultaneously, domestic firms whose management understands these dynamic tastes are not sitting idly by.  For example, the Alibaba Group, the largest ecommerce internet company in China (owner of Taobao, Tmall and Alibaba.com) is developing a new procurement system to bridge consumer demand with foreign, international suppliers.4849 According to its founder, Jack Ma, “Tmall will work with the center to build a database of international suppliers that Chinese consumers are most interested in.  It would then collect orders to make group purchases.”50 The new system is expected to be rolled out within the next two years and part of the domestic plans is to deliver anywhere on the mainland within 24 hours.51

Tastes and flavors from home

In March 2013 I spoke with Charles Zeng, founder of Piro, a restaurant and bar located in Shanghai near Jing’an Temple.  Zeng is originally from New York City and previously worked in the financial industry before moving to Shanghai.  After some cursory research he saw an unmet demand: tasty Western food with a normal price tag.  Thus two years ago he setup shop, teaching himself how to cook, learning as he went.  In his words, “while both of my parents were originally from Shanghai, they moved to the US about 30 years ago.  Yet despite this cultural connection, starting up the restaurants was still difficult for me due to a lack of guanxi and knowledge of the regulatory climate.  The learning curve was steep but based on my research I found that there was not enough American-style food for a price that both expats and locals could afford.  And so despite the hurdles I have turned this project into a profitable business venture and definitely think that there are a wide range of opportunities for more competition in this food and beverage area.”

In his view, obtaining the necessary licenses and permits and meeting the food code regulations are an ongoing challenge that all business owners must face.  Specific opportunities beyond food that he sees are “niches such as micro brews, craft brews – there are currently not many out here despite the enormous consumption of beer and liquor.  More specifically, craft beers that are higher value, top-shelf products.”  In 2011, 50 billion liters of beer were consumed in China compared with 24 billion in the US and 9 billion in Germany.52  With $1 billion in industry profits in 2012, China is the largest beer market by sales and Nomura forecasts that profits will rise to $9 billion in 2021.  Yet according to Accenture, 85% of the domestic beer market is “comprised of low-end domestic beer brands” such as Tsingtao which sells a 330 mL bottle for $.32 (for comparison, a similarly sized Budweiser costs $1).53 Thus, Zeng sees this as an opportunity to serve a niche market that will invariably grow as consumers become more familiar with what the market offers.  Maybe your local microbrew club could find success, like Carlsberg has attempted to do, as it recently bid to takeover Chongqing Brewery Company for $461 million.54

Yet to temper visions of immediate grandeur, consider champagne and chocolate.  In contrast to the large amounts of wine importation (see Chapter 11), the consumption of champagne remains relatively subdued.  Only 1.3 million bottles of champagne were sold in China in 2011 compared with 1.3 billion bottles of red wine during the same year.55 Similarly sales of other red wines to China from areas such as Germany remained muted because of lack of brand awareness (i.e., Chinese consumers are unfamiliar with German brands).56 In other food segments, chocolate consumption also remains low-key on the mainland.  The average Chinese consumes a mere 100g of chocolate a year; in comparison the average Japanese eats 11 times as much, an American eats 44 times as much and a German eats 82 times as much.57

And if you own a farm

Roughly 2% of all American’s work in the most productive agricultural industry in the world; an industry which not only feeds the 3rd largest populace but also grows and exports significant portions of the world’s caloric intake (up to 20%).  In 2011, the US exported a record $137 billion in agricultural products globally and China imported a record $20 billion from the US (surpassing Canada).58 In fact, due to a variety of reasons, in 2011 China became the largest importer of agriculture.59 Among other products, US farms exports soybean, rice, corn, cotton and pork to China.60 And in part because of a variety of domestic policies in China (discussed in Mark DeWeaver’s new book61 ), China is essentially dependent on the US for food security and thus is investing in and buying secure supply channels to improve its livestock.

For example, the New York Times noted this past spring that the US, “exported a record $664 million worth of breeding stock and genetic material like semen in 2011.”62 Who is buying this material?  In 2011, Chinese companies “bought $41 million worth of live breeding animals and genetics.”

Why are they buying this?  The New York Times quoted, Ronald Lemenager, a professor of animal sciences at Purdue University in Indiana, who said, “[w]hen you have a nation’s diet changing as rapidly as China’s, the most efficient way to build up production is to improve your animal genetics. We have the genetics they want.”

Thus, if you own or operate a farm in the US, not only can you export your products to China, but you can probably provide services to improve China’s knowledge base of breeding and husbandry.63

Domestic initiatives in agribusiness 

As I detail later in Chapter 14, NetEase is the 2nd largest tech company in China.  Its founder is William Ding who is investing a significant portion of his personal multi-billion dollar wealth in agriculture.  He along with his company “have set aside $16 million for agricultural investments” such as an organic piggery stocked with 5,000 pigs “raised and sold under conditions that can satisfy health-conscious consumers spooked by China’s many food safety scandals.”64

How large is this organic industry?  According to Du Xiangge, chairman of the China Federation of Organic Agricultural Movements (CFOAM), “Only 1.9 million hectares of land are used for growing organic vegetables in the country, less than 1 percent of all farmable land.  It is possible that the ratio could reach 5 percent over time.”65 Thus there is ripe potential for investors such as SAIF partners a domestic private equity (PE) fund that finances specialty shops like LohaoCity which sells organic health food in Beijing and Tianjin.

And on the other end of the spectrum is Peter Zhang a former-chemical engineer at a large SOE (and an autodidact) who is originally from Heilongjiang in the northeast.  Over the past several years he has retrained and retooled to become an English teacher, yet in an effort to hedge against food safety he has retrained yet again, becoming proficient in agribusiness and has leased several dozen hectares in Southeast Anhui province to grow organic produce, primarily fruits and vegetables, for his friends and family.  Zhang told me in November 2012 that, “I am concerned about food safety issues and have invested my personal savings into growing quality produce for my family and friends.  I belong to the middle class, I should be able to afford the quality of food that previous generations have in the past, yet due to inflation and a number of other macro factors, cannot.  So I have invested our savings into a dozen hectares of crops including carrots, rice, pears, cherries, grapes and even free-range chickens.  And after you get through the inconvenient paperwork, land is relatively cheap to lease outside of urban areas thus making the whole endeavor worthwhile.”  He is leasing roughly 80 acres (around 500 mu or亩) for about 30,000 RMB ($4,800) a year based on a 55-year lease between the government and rural farmers who exchanged the land with him (the typical lease on the mainland is 50-70 years, after which time ownership automatically reverts back to the state).

According to Zhang and others I have spoken to, inflation has pushed the price of chicken past a psychological “100 yuan per pound” line, a price that makes eating high-quality chickens unaffordable to those living, ironically, in larger cities.66 Why have these prices increased?  According to the Ministry of Agriculture, “urban Chinese increased their consumption of chicken 219% per capita from 1983 to 2006.”67 What Zhang is also referring to are consumer price index (CPI) increases which have added transportation and storage costs for farmers that bring poultry and produce from outlying farms due to a nascent supply chain network and cold storage network discussed earlier in the chapter.  For example, while CPI increased at a relatively low 1.7% in October 2012, roughly two years ago in January and February of 2011, CPI in China increased by nearly 5%, led in part by a 10% increase in food costs.6869  Similarly, the CPI index rose 3.2% in February 2013, a 10-month high due to a 6% increase in food costs – more specifically, residents in Beijing pay more per pound than their peers in Boston70 Yet to give you an idea of how this fluctuates and differs according to category, in mid-February the average prices of 21 different vegetables declined 11.2%.71 And since Chinese consumers spend a significantly larger portion of their disposable incomes on food (28% in China versus 10% in the US72 ), even a relatively small increase in price of staple goods is immediately felt.

Zhang also noted that one of the main reasons he and his family have become increasingly vigilante about what they eat is “because our trust and our faith in domestically owned restaurants has been shattered due to milk powder poisoning, gutter oil, fake honey and even moon cake scandals.”  While each region varies, roughly half of all honey currently sold in Shanghai is reportedly fake, comprised of substantially cheaper substitutes made of syrup and gum.73 Moon cakes are a traditional dessert made and given as gifts during Midautumn Festival usually held in September.  Over the past several years, investigations have uncovered several domestic mooncake producers, who in an effort to reduce costs have reused and resold both filling and entire inventories of mooncakes – made from previous years – to customers believing that they were buying newly made desserts.  For example, in 2003 a Shanghai-based company, Guanshengyuan, “was caught making mooncakes with expired and mildewed fillings.”74

He also mentioned that similar scandals have taken place at grocery stores and restaurants that dyed rotten pork to make it look younger and “leaner,” dyed noodles and even sold fake steamed buns.75  He is referring to a scandal in 2011 in which 17 noodle manufacturers in Dongguan added ink and paraffin wax “to give their products the look and texture of more expensive varieties.”76 In addition, while there have been several steam bun scandals, one of the most recent notables cases is the Shanghai Shenglu Food company, which added food coloring to lower costs (e.g., turning corn flour buns into a different color) and repackaged expired buns.77

This is not to say foreign restaurant chains are scandal free.  KFC advertised that its soybean milk was freshly ground, when it was not; and its chicken suppliers in Shandong may have used antibiotics to fatten the chicks faster (same-store sales declined 37% in January 2013 as a result).78 In fact, to alleviate food safety concerns, in February 2013 KFC announced that it was launching a new quality assurance program and cutting out small farmers due to the difficulty in overseeing them.79 Ajisen Ramen (a Japanese noodle restaurant) claimed its soup was made from bone-based broth, which upon further investigation turned out to be highly diluted (e.g., “a concentrate”).80  And a McDonald’s outlet in Shenyang reportedly served laundry detergent instead of a Coca-Cola.81

Yet perhaps by partnering with these entrepreneurs, foreign agriculture companies can establish a foot-hold on the mainland and satiate consumer demand.  And as I discuss later in several other chapters (notably 11 & 12), branding, trust and market perception are distinct advantages that foreign firms typically have when entering the mainland market.  This is in part because of the immense resources invested in quality control programs (e.g. Six Sigma) by foreign brands in order to proactively innovate and prevent any potential quality-based scandals from ruining their company images.  In contrast, this kind of branding issue is not taken as seriously on the mainland as it is elsewhere which itself creates an opportunity for brand marketing consultants.

Changing times

According to China Daily, a substantial portion of businesspeople in Zhejiang have moved away from the low-end, low-margin manufacturing industry to agriculture.82 In fact according to the Zhejiang Provincial Administration for Industry and Commerce, “the average annual amount of money invested in agriculture by Zhejiang businessmen has exceeded 10 billion yuan over the past five years.  The total amount reached 20 billion yuan last year.”83 Could your ag firm work with these businessmen in modernizing their farms?

Or maybe foreign companies that build or design automated farming equipment (e.g., robotic fruit pickers) can find demand for products in an industry that is still largely based on manual labor.  For example, according to a 2010 statement from China’s Ministry of Agriculture, “in corn production, the mechanization rate for sowing has reached 72.5%, but the mechanization rate for harvesting is only 16.9% and has become the bottleneck for corn production.”84 In 2012 this figure was updated and the new estimates for the overall mechanization rate for corn harvesting is now 38%.85 In contrast, in the US both planting and harvesting of corn are fully mechanized.  In fact, through the use of mechanization and genetically modified crops, an acre of US farmland “yields twice as much corn as in China or Eastern Europe and four times as much as in India.”86 Yet mechanization, as shown in the statistics above, is increasing rapidly on the mainland and according to a recent Reuters report, “[m]ore than half of China’s ploughing, planting and harvesting is carried out by machines, compared with a third a decade ago.”87 In fact, the 2011 harvest yields in Heilongjiang province broke nation-wide records, rising 11% over the previous year due to “bigger and better machinery for threshing and plowing.”88

And according to Der Spiegel, one of the reasons German companies purchased the imported strawberries from Shandong in the first place (see the strawberry story at the beginning of the chapter) was because strawberry picking robots capable of washing and cutting are an unknown variable – hence the relatively cheaper labor costs in China provided a cost advantage that neighboring countries did not have (at the time).89  In fact, because of relatively high labor costs in California, farm companies have begun looking for robotic alternatives such as prototypes from Vision Robotics that while still on the drawing board, have the potential to assist and replace manual human labor.90 Similarly, German and Californian agribusinesses may even be interested in a project unveiled two years ago: Japanese researchers demonstrated a robotic system that can identify the ripeness of a strawberry which enables the machines to “cut harvesting time from 500 hours to 300 hours.”91 Thus if you and your company build the agribusiness machines or software that powers the machines, you may be able to find new revenue sources in China.

Yet there are two sides to every coin.  As one Chinese source recently told me, “the potential for opportunities for foreign firms remains high in the agriculture industry because it is still largely underdeveloped.  Compounding the issue is that much arable land has been seized from farmers for real estate development during the urbanization process, and major labor forces have migrated from the farmlands into the cities, which leads to worries that the current food production capacity may not meet the growing food demand for the populace.”9293 For example, between 1996 and 2008, arable land decreased from 130 million acres to 121 million acres.  Another estimate put the loss at 123 million mu (one mu is about 1/6 of an acre).9495 Thus China must either import food or modernize its agricultural industry to increase production to make up for the food shortage.96 Simultaneously there are regulatory hurdles that sometimes require technology transfers from foreign agriculture firms to Chinese companies.  For example, seed companies like Monsanto must team up with local partners in order to gain market access.  Yet, these provisions have not prevented Monsanto from increasing both earnings and market share – and it plans to further boost investment on the mainland.97

Takeaway: as China develops, its middle class will have more funds and resources to allocate towards food and beverages.  US businesses and entrepreneurs are already providing both products and services in the form of agriculture, knowledge and physical storefronts.  Yet because of the continued growth, there are still opportunities to start new restaurants or even restructure and train a largely non-mechanized agriculture workforce to the industrial-scale agribusiness that is the envy of emerging markets.  Chapter 12 discusses how you can establish a brand in China through its diverse domestic social media networks.


Endnotes:

  1. A Chinese friend suggested that I provide an explanation to this phrase.  The complete phrase is wang zhe yi min wei tian, min yi shi wei tian.  While wang zhe means emperor, according to him the key to this is the last word “tian.”  Tian on its own is literally ‘heaven’ or the sky over our heads.  But this phrase should be better appreciated in the context of its originator, a Minister Guan Zhong of the powerful Qi Kingdom of the Spring Autumn (1st half Eastern Zhou) period, who meant to use tian here as in “tian ming” – the mandate from heaven (to rule over the people).  Properly understood, the five character phrase ending should correctly translate to something like “the government’s mandate (king or prince) to rule is founded upon its ability to feed the people.”  Or, in the more sophisticated form of political advice: “hunger breeds discontentment.” []
  2. See Agriculture Commissioner Todd Staples says that Americans spend less of their disposable income on food than individuals in Mexico, China and Russia from PolitiFact and Meet the 2020 Chinese Consumer from McKinsey & Co. []
  3. Seniors aged 55 to 65 in China’s largest cities spend half their expenditures on food and only 7% on apparel, according to Ogilvy data, while those a decade younger allocate 38% of their spending to food and 13% to apparel.  See Targeting Grandpa: China’s Seniors Hunger for Ads from The Wall Street Journal []
  4. See Chinese figures show fivefold rise in babies sick from contaminated milk from The Guardian, Two get death in tainted milk case from China Daily, Timeline: China milk scandal from BBC, and 毒牛奶事件 from Yunnan News []
  5. China dairy industry whistle-blower dies after assault from South China Morning Post []
  6. Another reason milk powder is in high demand is that milk powder companies have successfully convinced families that powder makes babies more “chubby” and therefore healthier than being breast-fed.  See Breastfeeding faces challenges in China from Xinhua  and Breastfeeding flashmobs: Chinese mothers are abandoning formula from The Telegraph []
  7. See Baby food sails out with Chinese crews from The New Zealand Herald and Dollar falls as tariff raised from The New Zealand Herald []
  8. This trend may not last as Chinese consumers and government officials have began investigating a claim regarding dicyandiamide, or DCD that has purportedly contaminated some milk powder from New Zealand.  See Ministry acts on dairy safety from China Daily []
  9. Farmers to milk China’s taste for cheese from The Australian []
  10. Hopes wane for China whole milk imports from agrimoney []
  11. 美赞臣等洋奶粉仍热销 国产奶粉再沦陷消费者失去信心 from Qbaobei []
  12. Tmall announces cooperation with foreign baby formula companies from Xinhua []
  13. Foreign cheese firms eye big slice of China’s market from China Daily []
  14. China targets dairy industry from Financial Review []
  15. Chinese buy farms for food from The Western Australia []
  16. See The Shandong Oilman from Caixin and Maggots in the Pasta: Europe Screens Tainted Chinese Food from The New York Times []
  17. See Survey: Half of Chinese like US ideas on democracy from Associated Press and McDonald’s says food giveaway not tied to China’s TV show on Corporate Shame from South China Morning Post []
  18. The consuming challenge of food safety from China Daily []
  19. For an illustration of why and which foreign items are purchased in China see this excellent infographic: Why Do Chinese Consumers Pay So Much for Foreign Brands? from East-West Connect []
  20. See The Hidden Price of Food from China from Der Spiegel and No contamination found on China-exported strawberries: watchdog from Xinhua []
  21. The consumer report site is Zhichuchuangwai []
  22. China Retail Report from the US Department of Agriculture Trade Office in Shanghai []
  23. Sea Bass With Barbie Dolls Challenge Wal-Mart in China from Bloomberg []
  24. China Retail Report from the US Department of Agriculture Trade Office in Shanghai []
  25. Chinese retailers give global giants run for money from The Hindu []
  26. Walmart, Tesco, Carrefour finding it tough to do business in China from The Economic Times []
  27. Sea Bass With Barbie Dolls Challenge Wal-Mart in China from Bloomberg []
  28. This is not an endorsement of their services as there are other chains that provide high-quality imported food as well, such as Ole’ – which is owned by CR Vanguard (華潤萬家), the largest grocery company on the mainland.  Another competitor is Metro (麦德龙), a German-owned chain. []
  29. CityShop []
  30. China Organic Food Certification Center (中绿华夏) []
  31. Cold storage industry sees a hot market in mainland from China Daily []
  32. In Beijing, cool profits from sub-zero storage from smartplanet []
  33. Ibid []
  34. Beacon Consultancy []
  35. China’s restaurant industry shows strong growth potential from Want China Times []
  36. This double digit increase is expected to decline due to a maturing market and a crackdown on government banquets in 2013.  See New Bureaucratic Diet Takes Bite Out of Restaurants, Hotels from The Wall Street Journal []
  37. Chocolate-makers seek to whet China’s appetite from Asia One []
  38. Mainland restaurant takings plummet as party order cadres to tighten belts from South China Morning Post []
  39. Ibid []
  40. For a step-by-step guide on forming an WFOE see China’s Approval Process for Inbound Foreign Direct Investment from the US Chamber of Commerce.  See also Forming A China WFOE. How Long Will That Be Going On? from China Law Blog and Selling In And Into China. Four Good Tips And Mine. from China Law Blog []
  41. Interview: Bob Boyce, owner of Blue Frog and KABB from Shanghaiist []
  42. Reverse brain drain: China engineers incentives for “brain gain” from Christian Science Monitor []
  43. See Demystifying the Chinese traveler from CNN and Chinese rush overseas for holiday from China Daily []
  44. While Chinese consumers typically trust foreign brands (as shown in Chapter 3) they are increasingly vigilant against scams and cons as well, especially while traveling abroad.  See Tourist: We were conned from The New Zealand Herald []
  45. Chinese Choosing Prada Over Louvre Boost Luxury Shares from Bloomberg []
  46. Chinese check-ins from The Economist []
  47. In 2011, the US embassy in China issued more than 160,000 student visas for Chinese students to study at American schools.  Yet a November 2012 report from Open Doors notes that the actual number is even higher, 194,029.  See Ten Years of Rapid Development of China-US Relations from Xinhua and Students from China add $5b to US economy from China Daily []
  48. The goal Sales at Tmall and Taobao combined to reach $157 billion in December 2012, a new record.  See RMB 1 TRILLION: Alibaba Shopping Sites Hit a Sales Milestone from Alizila []
  49. According to a recent Morgan Stanley research note, Alibaba is worth between $66-128 billion and Alibaba continues to diversify into other areas of e-commerce including notably a new search engine through its Aliyun brand.  See Morgan Stanley’s Latest Alibaba Estimates Suggest It’s Worth $66 – 128 Billion from Forbes and China’s E-Commerce Giant Now Has a Search Engine to Take on Baidu and Google from Tech In Asia []
  50. Tmall Plans to Link China’s Consumers with Foreign Goods from Caixin []
  51. The plans are being rolled out together however the domestic 24-hour delivery service is expected to be completed within 1 year.  See 马云1000亿建电商物流 目标全国任何地区24小时内送达from ifeng. []
  52. China beer consumption hits the 50 billion litre mark for first time in 2011 from Mintel []
  53. In the battle for China’s beer drinkers, the $0.32 brew is still king from Quartz []
  54. Carlsberg launches take-over offer for Chinese brewer from Reuters []
  55. Putting some fizz into the wine market from China Daily []
  56. German winemakers seek to win over the Chinese from Deutsche Welle []
  57. Chocolate-makers seek to whet China’s appetite from AsiaOne []
  58. See US agricultural exports to China become costly in times of drought from Global Post and U.S. chief agricultural negotiator sees bright future for exports from the Agricultural Communication Services []
  59. China Overtakes U.S. as Largest Crop Importer, WTO Data Show from Bloomberg []
  60. Chinese farmers produced 50 million tons of pork in 2012, more than half of the world’s total.  See How China’s love affair with pork is creating a pollution problem from The Guardian []
  61. Animals Spirits with Chinese Characteristics by Mark DeWeaver []
  62. From the U.S., a Future Supply of Livestock for China from The New York Times []
  63. According to one estimate at the US Department of Agriculture that I spoke with, building a new corn or soybean farm in the US may be profitable with current prices, especially since both of these crops are in high demand from China. []
  64. Game Boy: Billionaire William Ding Lei Has A Few Fantasies Of His Own from Forbes []
  65. Backed by Profit-hungry Investors, New Approach to Farming Takes Root from Caixin []
  66. This is not to say all produce has increased in costs.  For example, there has been a cabbage glut that has put many farmers in northern China in a bind.  The cost of cabbage harvest and transportation costs more than what could be made selling the produce at the market.  Thus some farms have allowed the public to gather cabbages for free and in some cases even let the cabbage rot in the fields.  See Cabbage price drop hurts growers, wholesalers, grocers from China Daily []
  67. Cheap food may be a thing of the past in U.S. from Los Angeles Times []
  68. See China’s food inflation leaving a bad taste from Globe & Mail and China’s inflation eases to 1.7 percent in October, giving room for more stimulus from Washington Post []
  69. China CPI in December 2012 rose to 2.5% and has spurred interest in looking for ways to reduce farm distribution costs.  See China moves to cut farm produce distribution costs from Xinhua []
  70. Meat prices add to China’s inflation, policy risks from Reuters []
  71. China’s farm produce prices down from China Daily []
  72. See Agriculture Commissioner Todd Staples says that Americans spend less of their disposable income on food than individuals in Mexico, China and Russia from PolitiFact and Meet the 2020 Chinese Consumer from McKinsey & Co. []
  73. Fake honey sales called rampant, hard to detect from Shanghai Daily []
  74. Bad moon rising: China’s mooncakes won’t keep from Want China Times []
  75. From Milk to Peas, a Chinese Food-Safety Mess from International Health Tribune []
  76. See China wrestles with food safety problems from Los Angeles Times and Noodle makers in hot water from China Daily []
  77. 3 arrested over Shanghai steamed bun scandal from China Daily []
  78. See Yum stumbles badly in China, warns on profit from Reuters, ‘Kentucky Fried China’ no more? from Reuters, Yum’s Yuck Factor in China from The Wall Street Journal, Yum’s chicken in China contained excessive levels of drugs from Reuters, CCTV report says KFC chickens are being fattened with illegal drugs from South China Morning Post and China Chicken Probe Hurts Profit as KFC Ends Deal: Liuhe from Bloomberg []
  79. Yum Makes Cuts to Supply Chain in China from The Wall Street Journal []
  80. Some foreign fast food is harder to swallow from China Daily []
  81. Shenyang McDonald’s apologizes after woman served detergent from Want China Times []
  82. Manufacturing, mining and construction represent approximately 45-48% of China’s GDP; in contrast, services accounted for 44.6% of China’s GDP in 2012.  While there are and will continue to be opportunities for manufacturing, the service industry continues to grow at a fast clip.  Yet as a number of the people I interviewed noted, services are intangible, physical goods are much more tangible so it can be a lot of hard work educating consumers about the value proposition in paying for something less concrete than they are used to.  How long will it take to educate them to appreciate this quality?  How to differentiate your company from local competitors like Newegg tried to do?  See Chinese Graduates Say No Thanks to Factory Jobs from The New York Times and Served in China from The Economist []
  83. Entrepreneurs turn to the land for profit from China Daily []
  84. Another innovation that may assist in the merging, acquisition and development of agricultural land is satellite imagery which is now being used as part of a land reform project being piloted in Anhui.  See China’s big step in rural reform; mapping tiny plots of farm land from Reuters, 加快推进玉米收获机械化 力争2015年玉米主产区机收水平超过50% from the Ministry of Agriculture and All About Corn by Cathy Gao Jing []
  85. 农业部:全国“三秋”农业机械化生产迅速展开 from The Central People’s Government of the People’s Republic of China []
  86. Monsanto Needs to Put Doubts to Rest from The Wall Street Journal []
  87. Analysis: China turns to machines as farmers seek fresh fields from Reuters []
  88. Ibid []
  89. The Hidden Price of Food from China from Der Spiegel []
  90. Farms Fund Robots to Replace Migrant Fruit Pickers from Wired []
  91. Strawberry-picking robot knows when they’re ripe from c|net []
  92. This is not to say similar “eminent domain” cases do not take place in other countries.  See China’s cabinet warns of rural land expropriation from Xinhua, China’s giant, deserted malls wait for reluctant consumers from The Globe & Mail and Henan city refuses to stop clearance of graves to make farmland from South China Morning Post []
  93. This is not suggesting that China is facing a looming plague or food shortage as a whole.  Yet despite a record grain output of 589 million tons in 2012, according to Xinhua, China throws away enough food to feed 200 million people each year.  Another estimate from China Agricultural University states that 50 million tons of food is wasted annually (10% of China’s annual grain output) due to poor storage techniques and rot during transportation.  These statistics have become talking points recently during a government crackdown on waste and corruption targeted at government banquets where food is often left uneaten.  For comparison, according to the Food and Agriculture Organization (FAO), one third of all food grown globally is lost or wasted each year (amounting to approximately $1 trillion).  Another recent report from the Institution of Mechanical Engineers states that as much as half of all food produced in the world ends up wasted each year.  See Curbing food waste from Xinhua, China’s anti-waste campaign revives frugal spirit from Xinhua, Grain supplies still not secure from China Daily, Assuring Chinese Finish Their Live Lobster Sashimi from Bloomberg and Almost half of the world’s food thrown away, report finds from The Guardian []
  94. Shortage of farms and water threatens grain output targets from China Daily []
  95. Shrinking arable land threatens grain security from China Daily []
  96. This is called a “grain security” issue that Chinese policy makers and analysts have been increasingly discussing over the past several years.  Some have noted this is equivalent to the “Western dependence” on Middle East petroleum.  See Keep a red line for arable land from China Daily and Grain supplies still not secure from China Daily []
  97. See Monsanto Sees Greater China Investment on Par With Brazil from Bloomberg and Monsanto Needs to Put Doubts to Rest from The Wall Street Journal []