A brief history of R3 – the Distributed Ledger Group

What’s in a name?

I was at an event last week and someone pulled me aside asked: why do you guys at R3 typically stress the phrase “distributed ledger” instead of “blockchain”?

The short answer is that they are not the same thing.

In simplest terms: a blockchain involves stringing together a chain of containers called blocks, which bundle transactions together like batch processing, whereas a distributed ledger, like Corda, does not and instead validates each transaction (or agreement) individually.1

The longer answer involves telling the backstory of what the R3 consortium is in order to highlight the emphasis behind the term “distributed ledger.”

Inspired by IMF report, page 8


R3 (formerly R3 CEV) started out as a family office in 2014.2 The “3” stood for the number of co-founders: David Rutter (CEO), Todd McDonald (COO), and Jesse Edwards (CFO). The “R” is the first initial of the CEO’s last name.  Very creative!

During the first year of its existence, R3 primarily looked at early stage startups in the fintech space.  The “CEV” was an acronym: “crypto” and “consulting,” “exchanges,” and “ventures.”

Throughout 2014, the family office kept hearing about how cryptocurrency companies were going to obliterate financial institutions and enterprises.  So to better understand the ecosystem and drill into the enthusiasm around cryptocurrencies, R3 organized and held a series of round tables.

The first was held on September 23, 2014 in NYC and included talks from representatives of: DRW, Align Commerce, Perkins Coie, Boost VC, and Fintech Collective.  Also in attendance were representatives from eight different banks.

The second round table was held on December 11, 2014 in Palo Alto and included talks from representatives of: Stanford, Andreessen Horowitz, Xapo, BitGo, Chain, Ripple, Mirror, and myself.  Also in attendance were representatives from 11 different banks.

By the close of 2014, several people (including myself) had joined R3 as advisors and the family office had invested in several fintech startups including Align Commerce.

During the first quarter of 2015, David and his co-founders launched two new initiatives.  The first was LiquidityEdge, a broker-dealer based in NYC that built a new electronic trading platform for US Treasurys.3  It is doing well and is wholly unrelated to R3’s current DLT efforts.

The second initiative was the incorporation of the Distributed Ledger Group (DLG) in Delaware in February 2015.  By February, the family office had also stopped actively investing in companies in order to focus on both LiquidityEdge and DLG.

In April 2015 I published Consensus-as-a-Service (CaaS) which, at the time, was the first paper articulating the differences between what became known as “permissioned” and “permissionless” blockchains and distributed ledgers.  This paper was then circulated to various banks that the small R3 team regularly interacted with.

The following month, on May 13, 2015, a third and final round table was held in NYC and included talks from representatives of Hyperledger (the company), Blockstack, Align Commerce and the Bank of England.  Also in attendance were representatives from 15 banks as well as a market infrastructure operator and a fintech VC firm.  In addition to the CaaS paper, the specific use-case that was discussed involved FX settlement.4

The transition from a working group to a commercial entity was formalized in August and the Distributed Ledger Group officially launched on September 1, 2015 although the first press release was not until September 15.  In fact, you can still find announcements in which the DLG name was used in place of R3.

By the end of November, phase one of the DLG consortium – now known as the R3 consortium – had come to a conclusion with the admission of 42 members.  Because of how the organization was originally structured, no further admissions were made until the following spring (SBI was the first new member in Phase 2).

So what does this all have to do with “distributed ledgers” versus “blockchains”?

Well, for starters, we could have easily (re)named or (re)branded ourselves the “Blockchain Group” or “Blockchain Banking Group” as there are any number of ways to plug that seemingly undefinable noun into articles of incorporation.  In fact, DistributedLedgerGroup.com still exists and points to R3members.com.5 So why was R3 chosen?  Because it is a bit of a mouthful to say DistributedLedgerGroup!

Corda’s genesis

Upon launch, the architecture workstream lead by our team in London (which by headcount is now our largest office), formally recognized that the current hype that was trending around “blockchains” had distinct limitations.  Blockchains as a whole were designed around a specific use-case – originally enabling censorship-resistant cryptocurrencies. This particular use-case is not something that regulated financial institutions, such as our members, had a need for.

While I could spend pages retracing all of the thought processes and discussions surrounding the genesis of what became Corda, Richard Brown’s view (as early as September 2015) was that there were certain elements of blockchains that could be repurposed in other environments, and that simply forking or cloning an existing blockchain – designed around the needs of cryptocurrencies – was a non-starter.  At the end of that same month, I briefly wrote about this view in a post laying out the Global Fabric for Finance (G3F), an acronym that unfortunately never took off. In the post I specifically stated that, “[i]t also bears mentioning that the root layer may or may not even be a chain of hashed blocks.”

In October 2015, both James Carlyle and Mike Hearn formally joined the development team as Chief Engineer and lead platform engineer respectively.  During the fall and winter, in collaboration with our members, the architecture team was consumed in the arduous process of funneling and filtering the functional and non-functional requirements that regulated financial institutions had in relation to back office, post-trade processes.

By the end of Q1 2016, the architecture team gestated a brand new system called Corda.  On April 5, 2016, Richard published the first public explanation of what Corda was, what the design goals were and specifically pointed out that Corda was not a blockchain or a cryptocurrency.  Instead, Corda was a distributed ledger.

Prior to that date, I had personally spent dozens of hours clarifying what the difference between a blockchain and a distributed ledger was to reporters and at events, though that is a different story.  Unfortunately even after all these explanations, and even after Richard’s post, the Corda platform was still inappropriately lumped into the “blockchain” universe.

Following the open sourcing of Corda in November 2016, we formally cut the “CEV” initials entirely from the company name and are now known simply as R3.  Next year we plan to make things even shorter by removing either the R or 3, so watch out domain squatters!


As of February 2017, the R3 consortium is formally split into two groups that share knowledge and resources: one group is focused on building out the Corda platform and the other, the Lab and Research Center, is focused on providing a suite of services to our consortium members.  I work on the services side, and as described in a previous post, my small team spends part of its time filtering vendors and projects for the Lab team which manages several dozen projects at any given time for our consortium members.

The Lab team has completed more than 20 projects in addition to 40 or so ongoing projects.  Altogether these involved (and in some cases still involve) working with a diverse set of platforms including Ethereum, Ripple, Fabric, Axoni, Symbiont and several others including Corda.  Since we are member driven and our members are interested in working and collaborating on a variety of different use-cases, it is likely that the services side will continue to experiment with a range of different technologies in the future.

Thus, while it is accurate to call R3 a technology company focused on building a distributed ledger platform and collaborating with enterprises to solve problems with technology, it is not accurate to pigeonhole it as a “blockchain company.”  Though that probably won’t stop the conflation from continuing to take place.

If you are interested in understanding the nuances between what a blockchain, a database, and a distributed ledger are, I highly recommend reading the multitude of posts penned by my colleagues Antony Lewis and Richard Brown.

  1. Blockchains inspired by cryptocurrencies such as Bitcoin used blocks because Satoshi wanted identity-free consensus (e.g., pseudonymity).  That implies miners can come and go at will, without any kind of registration, which eliminated the choice of using any existing consensus algorithm.

    As a result, Satoshi’s solution was proof-of-work (PoW).  However, PoW is susceptible to collisions (e.g., orphan blocks).  When a collision occurs you have to wait longer to obtain the same level of work done on a transaction. Thus you want to minimize them, which resulted in finding a PoW on average every ten minutes.  This means that in a network with one minute propagation delays, not unlikely in a very large network (BGP sees such propagation times) then you waste ~10% of total work done, which was considered an acceptable loss rate in 2008 when Satoshi was designing and tweaking the parameters of the system.

    Distributed ledgers such as Corda, use a different design because it is an identified network, where members cannot just come and go at will, and do have to register. With Corda, the team also assumes relatively low propagation times between members of a notary cluster.  One of the key differences between mere PoW (i.e. hashcash) and a blockchain is that in the latter, each block references the prior – thus PoWs aggregate.  It can be tough to do that unless all transactions are visible to everyone and there is a single agreed upon blockchain but if you do not, you will not get enough PoW to yield any meaningful security. []

  2. The R3CEV.com domain was created on August 13, 2014. []
  3. It may look like an odd spelling, but Treasurys is the correct spelling. []
  4. At the time, I was an advisor to Hyperledger which was acquired by Digital Asset the following month. []
  5. The DistributedLedgerGroup.com domain was created on December 23, 2014 and R3members.com was created on March 15, 2016. []
Send to Kindle

Layer 2 and settlement

Nary a week goes by without having to hear a startup claim their service will have the ability to “settle” a cryptocurrency or virtual asset or something “smart,” on to Layer 2.  In this instance, Layer 2 refers to a separate network that plugs into a cryptocurrency via off-chain channels.1

This often comes up in conjunction with conversations surrounding the Bitcoin block size debate: specifically around (hypothetically) scaling to enable Visa-like transaction throughput vis-a-vis projects like the Thunder and Lightning network proposals which are often characterized as Layer 2 solutions.2

As Wolfgang Pauli might say, this is not even wrong.

Why?  For starters, the comparisons are not the same.

Visa is a credit clearing and authentication network, not a settlement network; in contrast no cryptocurrency has credit lines baked-in.  In addition – as I penned a year ago – in practice “settlement” is a legal concept and typically requires ties into the existing legal infrastructure such as courts and legally approved custodians. 3

Two simplified examples: (1) If Bob wanted to settle cash electronically and he lived in just about any country on the globe, the only venue that this electronic cash ultimately settles in right now is a central bank usually via its real-time gross settlement (RTGS) network.  And (2), if Bob owned the title to a (dematerialized) security and he is trying to transfer ownership of it to someone else, the security ultimately settles in a central securities depository (CSD) such as the DTCC or Euroclear.

What does this have to do with the world of blockchains and DLT?

As of this writing, no central bank-backed digital currency (CBDC) exists.4 As a consequence, there is no real digital cash settlement taking place on any ledger outside of a banks’ own ledger (yet).

One of the key goals for DLT platforms is to eventually get “cash on-ledger” issued by one or more central bank.  For instance, at R3 we are currently working on a couple of CBDC-related projects including with the Bank of Canada and Monetary Authority of Singapore.  And other organizations are engaged in similar efforts.


In short, one of the potential advantages of using a CBDC issued onto a distributed ledger is the enabling of network participants (such as financial institutions) to settle dematerialized (digitized) asset transfers without relying on outside reconciliation processes. Delivery versus Payment (DvP), the simultaneous exchange of an asset and its payment, could actually take place on-chain.5

However, today if participants on a distributed ledger wanted to settle a trade in cash on a distributed ledger, they could not. They would still need to settle via external processes and mechanisms, which according to an estimate from Autonomous research, collectively costs the industry $54 billion a year.  As a result, the industry as a whole is attempting to reduce and – if possible – remove frictions such as these post-trade processes.6  And according to a recent paper from the Bank of England as well as a new paper from the Federal Reserve, CDBCs are one invention that potentially could reduce some of these associated frictions and processes.

So how does that tie back in to a hypothetical Layer 2 or 3, 4, 5, connected to a cryptocurrency network?

Assuming one or more of the Lightning implementations is built, deployed, and goes “into production,” the only object that is being tracked and confirmed is a cryptocurrency.7

Cryptocurrencies, as I have written before, are anarchic: purposefully divorced from legal infrastructure and regulatory compliance.

As a result, it cannot be said that “Layer 2” will act as a settlement layer to anything beyond the cryptocurrency itself, especially since the network it attaches to can at most by design only guarantee probabilistic finality.  In fact, the most accurate description of these add-on networks is that each Lightning implementation requires building completely separate networks run and secured by different third parties: pseudonymous node operators acting as payment processors.  What are the service-level agreements applied to these operators?  What happens if it is no longer profitable or sustainable to operate these nodes?  Who are you going to call when something – like routing – doesn’t work as it is supposed to?

And like most cryptocurrencies, Lightning (the generic Lightning) is developed as a public good, which – as a recent paper explored – may have hurdles from a fiduciary, governance, and accountability perspective.

Assuming the dev teams working on the various implementations solve for decentralized routing and other challenges, at most Lightning will be a clearing network for a cryptocurrency, not electronic cash or securities.  Therefore proponents of existing Layer 2 network proposals might want to drop the “settlement” marketing language because settlement probably isn’t actually occurring.  Trade confirmations are.

But what about colored coins?  Can’t central banks just use the Bitcoin network itself and “peg” bitcoins directly to cash or set-up a Bitcoin-like system that is backed by the central bank itself?

These are tangential to “Layer 2” discussion but sure, they could in theory.  In fact, the latter is an idea explored by JP Koning in a soon-to-be published paper on “Fedcoin.”  In practice this is probably not ideal for a variety of reasons including: privacy, confidentiality, recourse, security, scalability, public goods problems, and the fact that pseudonymous miners operating outside the purview of national regulatory bodies would be in charge of monetary policy (among many other regulatory compliance issues).

Why not just use an existing database to handle these regulated financial instruments then?  This is a topic that has and will fill academic journals in the years to come (e.g., RSCoin).  But for starters I recommend looking at a previous post from Richard Brown and two newer posts from Antony Lewis.


There are real, non-aesthetic reasons why aviation designers and manufacturers stopped building planes with more than two or three wings, namely aerodynamics.  Creative ideas like Lightning may ultimately be built and deployed by cryptocurrency-related companies and organizations, but it is unclear how or why any regulated enterprise would use the existing proposals since these networks are not being architected around requirements surrounding settlement processes.

Perhaps that will change in time, but laws covering custody, settlement, and payment processing will continue to exist and won’t disappear because of anarchic “Layer 2” proposals.  Maybe it is possible to borrow and clone some of the concepts, reusing them for alternative environments, just like some of the “blockchain”-inspired platforms have reused some of the ideas underlying cryptocurrencies to design new financial market infrastructure.  Either way, both worlds will continue to co-exist and potentially learn from one another.

  1. From a word choice, it is arguably a misnomer to call Lightning a “layer” at all because relatively little is being built on top of Bitcoin itself.  These new networks are not powered by mining validators whereas colored coin schemes are. []
  2. While he doesn’t delve too much into any of these specific projects, Vitalik Buterin’s new paper on interoperability does briefly mention a couple of them.  Also note that the Teechan proposal is different than Lightning in that the former scales via trusted hardware, specifically Intel’s SGX tech, and sidesteps some of the hurdles facing current Lightning proposals. []
  3. This topic is a ripe area for legal research as words need to be precisely defined and used.  For instance, if bitcoins do not currently “settle” (in the sense that miners and users do not tie on-chain identities into court recognized identity, contract, and ledger systems thereby enabling traditional ownership transfer), does this impact government auctions of seized cryptocurrencies?  What was the specific settlement process involved in the auction process and are encumbrances also transferred?  It appears in practice, that in these auctions bitcoins do transfer in the sense that new entities take control of the private key(s), is this settlement? []
  4. An argument can be made that there are at least 3 publicly known exceptions to this, though it depends on the definition of an in-production CBDC.  This includes vendors working with: Senegal, Tunisia, and Barbados. []
  5. In blockchain parlance this is called an “atomic transfer.” []
  6. It is not just reconciliation processes, it is the actual DvP itself (plus the subsequent “did you get it yet” reconciliation processes). []
  7. As an aside, what are the requirements for “being in production?”  In the enterprise world, there is a difference between being in a sandbox and being in production.  Which blockchain(s) have been vetted for and secured against real production level situations and fulfilled functional requirements such as scaling and preserving confidentiality? []
Send to Kindle

Non-technical Corda whitepaper released

Earlier today our architecture team released its first public whitepaper on Corda.

The WSJ covered it here and here.

Consequently I am somewhat puzzled by news stories that still refer to a “blockchain” as “Bitcoin technology.”  After all, we don’t refer to combustion engines in cars as “horse-powered technology” or an airplane turbine engine as “bird-powered technology.”

A more accurate phrase would be to say something like, “a blockchain is a type of data structure popularized by cryptocurrencies such as Bitcoin and Ethereum.”  After all, chronologically someone prior to Satoshi could have assembled the pieces of a blockchain into a blockchain and used it for different purposes than censorship-resistant e-cash.  In fact, both Guardtime and Z/Yen Group claim to have done so pre-2008, and neither involves ‘proof-of-work.’

Fun fact: Corda is not a blockchain, but is instead a distributed ledger.

Send to Kindle

Code is not law

This past Sunday I gave a new presentation at the Palo Alto Ethereum meetup — it was largely based on my previous two blog posts.

Note: all of the references and citations can be found within the notes section of the slides.  Also, I first used the term “anarchic chain” back in April 2015 based on a series of conversations with Robert Sams.  See p. 27.

Special thanks to Ian Grigg for his constructive feedback.



Send to Kindle

Archy and Anarchic Chains

[Note: the views expressed below are solely my own and do not necessarily represent the views of my employer or any organization I advise.]

Yesterday, at block height 1920000, many elements of the Ethereum community coordinated a purposeful hardfork.

After several weeks of debate and just over a couple weeks of preparation, key stakeholders in the community — namely miners and exchanges — attempted to create a smooth transition from Ethereum Prime (sometimes referred to as Ethereum Classic) into Ethereum Core (Ethereum One).1

Users of exchange services such as Kraken were notified of the fork and are now being allowed to withdraw ETH to Ethereum Core, which many miners and exchanges now claim as “mainnet.”

Was the hardfork a success?  To answer that question depends on which parallel universe (or chain) you resided on.  And it also depends on the list of criteria for what “failure” or “success” are measured by.

For instance, if you ended up with ETH on the “unsupported” fork (Classic), who was financially responsible for this and who could attempt to file a lawsuit to rectify any loses?

Maybe no one.  Why?  Because public blockchains intentionally lack terms of service, EULA, and service level agreements, therefore it is difficult to say who is legally liable for mistakes or loses.

For instance, if financial instruments from a bank were sent to miners during the transition phase and are no longer accessible because the instruments were sent to the “unsupported” chain, who is to blame and bears responsibility?  Which party is supposed to provide compensation and restitution?

De facto versus de jure

This whole hardfork exercise visualizes a number of issues that this blog has articulated in the past.

Perhaps the most controversial is that simply: there is no such thing as a de jure mainnet whilst using a public blockchain.  The best a cryptocurrency community could inherently achieve is a de facto mainnet.2

What does that mean?

Public blockchains such as Bitcoin and Ethereum, intentionally lack any ties into the traditional legal infrastructure.  The original designers made it a point to try and make public blockchains extraterritorial and sovereign to the physical world in which we live in.  In other words, public blockchains are anarchic.

As a consequence, lacking ties into legal infrastructure, there is no recognized external authority that can legitimately claim which fork of Bitcoin or Ethereum is the ‘One True Chain.’  Rather it is through the proof-of-work process (or perhaps proof-of-stake in the future) that attempts to attest to which chain is supposed to be the de facto chain.3

However, even in this world there is a debate as to whether or not it is the longest chain or the chain with the most work done, that is determines which chain is the legitimate chain and which are the apostates.4 5

And this is where, fundamentally, it becomes difficult for regulated institutions to use a public blockchain for transferring regulated data and regulated financial instruments.

For instance, in March 2013 an accidental, unintended fork occurred on what many participants claimed as the Bitcoin mainnet.

To rectify this situation, over roughly four hours, operators of large mining pools, developers, and several exchanges met on IRC to coordinate and choose which chain they would support and which would be discarded.  This was effectively, at the time, the largest fork-by-social-consensus attempted (e.g., proof-of-nym-on-IRC).

There were winners and losers.  The losers included: OKPay, a payment processor, lost several thousand dollars and BTC Guild, a large mining pool who had expended real capital, mined some of the now discarded blocks.

In the Bitcoin world, this type of coordination event is slowly happening again with the never ending block size debate.

One team, Bitcoin Classic, is a small group of developers that supports a hardfork to relatively, quickly increase the block size from 1 MB to 2 MB and higher.  Another group, dubbed Bitcoin Core, prefers a slower role out of code over a period of years that includes changes that would eventually increase the block size (e.g., segwit). 6

Yet as it lacks a formal governance structure, neither side has de jure legitimacy but instead relies on the court of public opinion to make their case.  This is typically done by lobbying well-known figureheads on social media as well as mining pools directly.  Thus, it is a bit ironic that a system purposefully designed for pseudonymous interactions in which participants were assumed to be Byzantine and unknown, instead now relies on known, gated, and trusted individuals and companies to operate.

Note: if the developers and miners did have de jure legitimacy, it could open up a new can of worms around FinCEN administrative requirements. 7  Furthermore, the miners are always the most important stakeholders in a proof-of-work system, if they were not, no one would host events just for them.

arthur twitter pow

Source: Twitter


With this backstory it is increasingly clear that, in the legal sense, public blockchains are not actual distributed ledgers.  Distributed, yes; ledgers, no.

As Robert Sams articulates:8

I think the confusion comes from thinking of cryptocurrency chains as ledgers at all. A cryptocurrency blockchain is (an attempt at) a decentralised solution to the double spending problem for a digital, extra-legal bearer asset. That’s not a ledger, that’s a log.

That was the point I was trying to make all along when I introduced the permissioned/permissionless terminology!9 Notice, I never used the phrase “permissionless ledger” — Permissionless’ness is a property of the consensus mechanism.

With a bearer asset, possession of some instrument (a private key in the cryptocurrency world) means ownership of the asset. With a registered asset, ownership is determined by valid entry in a registry mapping an off-chain identity to the asset. The bitcoin blockchain is a public log of proofs of instrument possession by anonymous parties. Calling this a ledger is the same as calling it “bearer asset ledger”, which is an oxymoron, like calling someone a “married bachelor”, because bearer assets by definition do not record their owners in a registry!

This taxonomy that includes the cryptocurrency stuff in our space (“a public blockchain is a permissionless distributed ledger of cryptocurrency”) causes so much pointless discussion.

I should also mention that the DLT space should really should be using the phrase “registry” instead of “ledger”. The latter is about accounts, and it is one ambition too far at the moment to speak of unifying everyone’s accounts on a distributed ledger.

As I have discussed previously, public blockchains intentionally lack hooks into off-chain legal identification systems.

Why?  Because as Sams noted above: a KYC’ed public blockchain is effectively an oxymoron.  Arguably it is self-defeating to link and tie all of the participants of the validation (mining) process and asset transfer process (users) to legal identities and gate them from using (or not using) the network services.  All you have created is a massively expensive permissioned-on-permissionless platform.

But that irony probably won’t stop projects and organizations from creating a Kimberely Process for cryptocurrencies.

I cannot speak on behalf of the plethora of “private chain” or “private ledger” projects (most of which are just ill-conceived forks of cryptocurrencies), but we know from public comments that some regulators and market structures might only recognize blockchains and distributed ledgers that comply with laws (such as domestic KYC / AML regulations) by tying into the traditional legal infrastructure.10 This means tying together off-chain legal identities with on-chain addresses and activity.


There are multiple reasons, but partly due to the need to reduce settlement risks: to create definitive legal settlement finality and identifying the participants involved in that process.11


As illustrated with the purposeful Ethereum One hardfork and the accidental Bitcoin fork in 2013, public blockchains by design, can only provide probabilistic settlement finality.

Sure, the data inside the blocks itself is immutable, but the ordering and who does the ordering of the blocks is not.

What does this mean?  Recall that for both Ethereum and Bitcoin, information (usually just private keys) are hashed multiple times by a SHA algorithm making the information effectively immutable.12 It is unlikely given the length of time our star is expected to live, that this hash function can be reversed by a non-quantum computer.

However, blocks can and will be reorganized, they are not immutable.  Public blockchains are secured by social and economic consensus, not by math.

As a consequence, there are some fundamental problems with any fork on public blockchains: they may actually increase risks to the traditional settlement process.  And coupled with the lack of hooks for off-chain identity means that public blockchains — anarchic blockchains — are not well-suited or fit-for-purpose for regulated financial institutions.

After all, who is financially, contractually, and legally responsible for the consequences of a softfork or hardfork on a public blockchain?

  • If it is no one, then it might not be used by regulated organizations because they need to work with participants who can be held legally accountable for actions (or inactions).
  • If it is someone specifically (e.g., a doxxed individual) then you have removed the means of pseudonymous consensus to create censorship resistance.

In other words, public blockchains, contrary to the claims of social media, are not “law” because they do not actually tie into the legal infrastructure which they were purposefully designed to skirt.  By attempting to integrate the two worlds — by creating a KYC’ed public blockchain — you end up creating a strange hydra that lacks the utility of pseudonymity (and censorship resistance) yet maintains the expensive and redundant proof-of-work process.

These types of forks also open up the door for future forks: what is the criteria for forking or not in the future?  Who is allowed and responsible to make those decisions?  If another instance like the successful attack and counter-attack on The DAO takes place, will the community decide to fork again?  If 2 MB blocks are seen as inadequate, who bears the legal and financial responsibility of a new fork that supports larger (or smaller) blocks?  If any regulated institution lose assets or funds in this forking process, who bears responsibility?  Members of IRC rooms?

If the answers are caveat emptor, then that level of risk may not be desirable to many market participants.


Who are you going to sue when something doesn’t go according to plan?  In the case of The DAO, the attacker allegedly threatened to sue participants acting against his interests because he claimed: code is law.  Does he have legal standing?  At this time it is unclear what court would have accepted his lawsuit.

But irrespective of courts, it is unclear how smart contract code, built and executed on an anarchic platform, can be considered “legal.”  It appears to be a self-contradiction.

As a consequence, the fundamental need to tie contract code with legal prose is one of the key motivations behind how Richard Brown’s team in London approached Corda’s design.  If you cannot tie your code, chain, or ledger into the legal system, then it might be an unauthoritative ledger from the perspective of courts.13

And regulated institutions can’t simply just ignore regulations as they face real quantifiable consequences for doing so.  To paraphrase George Fogg, that’s akin to putting your head in the sand.

We continue to learn from the public blockchain world, such as the consequences of forks, and the industry as a whole should try to incorporate these lessons into their systems — especially if they want anyone of weight to use them.  Anarchic blockchains will continue to co-exist with their distributed ledger cousins but this dovetails into a conversation about “regtech,” which is a topic of another post.


  1. Rejecting Today’s Hard Fork, the Ethereum Classic Project Continues on the Original Chain: Here’s Why from Bitcoin Magazine []
  2. This doesn’t mean that regulators and/or financial institutions won’t use public blockchains for various activities; perhaps some of them will be comfortable after quantifying the potential risks associated with them. []
  3. Ethereum developers plan to transition Ethereum from proof-of-work to proof-of-stake within the next year. []
  4. See Arthur Breitman’s interview on Epicenter Bitcoin and Mike Hearn’s interview on Money & Tech []
  5. Philosophically when Bob connects to “The Bitcoin Network” — how does Bob know he is actually connected to the “real” Bitcoin network?  One method is to look at the block header: it should take a specific amount of time to recreate the hash with that proof-of-work. This proves which network has the most work done.  However, in the meantime, Bob might connect to other ‘pretenders’ claiming to be “The Bitcoin Network.”  At this time, there does not appear to be any legal recognition of a specific anarchic chain. []
  6. The Bitcoin Core fork, which is euphemistically called a softfork, is basically a hardfork spread over a long period of time. []
  7. See Section 3.4 []
  8. Personal correspondence: March 9, 2016 []
  9. See Blockchain Finance by Robert Sams []
  10. This is not to say that regulators, governments, and various market participants will not use public blockchains for other activity. []
  11. See Section 3.1 []
  12. For proof-of-work mining, Ethereum uses ethash instead of SHA256.  For hashing itself, Ethereum uses SHA-3 which is part of the Keccak family (some people use the terms interchangeably but that isn’t technically correct). []
  13. See Section 9 []
Send to Kindle

What’s the deal with DAOs?

[Disclaimer: I do not own any cryptocurrencies nor have I participated in any DAO crowdfunding.]

This post will look at the difference between a decentralized autonomous organization (DAO) and a project called The DAO.

Brief explanation

The wikipedia entry on DAOs is not very helpful.  However, Chapters 2 through 5 may be of some use (although it is dated information).

In terms of the uber hyped blockchain world, at its most basic kernel, a DAO is a bit of code — sometimes called a “smart contract” (a wretched name) — that enables a multitude of parties including other DAOs to send cryptographically verifiable instructions (such as a digitally signed vote) in order to execute the terms and conditions of the cloud-based code in a manner that is difficult to censor.

One way to think of a simple DAO: it is an automated escrow agent that lives on a decentralized cloud where it can only distribute funds (e.g., issue a dividend, disperse payroll) upon on receiving or even not receiving a digital signal that a task has been completed or is incomplete.

For instance, let us assume that a small non-profit aid organization whose staff primarily work in economically and politically unstable regions with strict capital controls, set up a DAO — an escrow agent — on a decentralized cloud to distribute payroll each month.

This cloud-based escrow agent was coded such that it would only distribute the funds once a threshold of digital signatures had signed an on-chain contract — not just by staff members — but also from independent on-the-ground individuals who observed that the staff members were indeed doing their job.  Some might call these independent observers as oracles, but that is a topic for a different post.1

Once enough signatures had been used to sign an on-chain contract, the escrow agent would automatically release the funds to the appropriate individuals (or rather, to a public address that an individual controls via private key).  The terms in which the agent operated could also be amended with a predetermined number of votes, just like corporate board’s and shareholder’s vote to change charters and contracts today.

The purported utility that decentralization brings to this situation is that it makes censoring transactions by third parties more difficult than if the funds flowed through a centralized rail.  There are trade-offs to these logistics but that is beyond the scope of this post.

The reason the DAO acronym includes the “organization” part is that the end-goal by its promoters is for it to provide services beyond these simple escrow characteristics such as handling most if not all administrative tasks such as hiring and firing.

Watch out Zenefits, the cryptocurrency world is going to eat your lunch!  Oh wait.

A short history

It is really easy to get caught up in the euphoria of a shiny new toy.  And the original goal of a DAO sounds like something out of science fiction —  but these undertones probably do it a disservice.

Prior to 2014 there had been several small discussions around the topic of autonomous “agents” as it related to Bitcoin.

For instance, in August 2013, Mike Hearn gave a presentation at Turing Festival (see above), describing what was effectively a series of decentralized agents that operated logistical companies such as an autonomous car service.

Several months later, Vitalik Buterin published the Ethereum white paper which dove into the details of how to build a network — in this case a public blockchain — which natively supported code that could perform complex on-chain tasks: or what he dubbed as a decentralized autonomous organization.


The impetus and timing for this post is based on an ongoing crowdsale / crowdfunding activity for the confusingly named “The DAO” that has drawn a lot of media attention.

Over the past year, a group of developers, some of whom are affiliated with the Ethereum Foundation and others affiliated with a company called Slock.it have created what is marketed as the first living and breathing DAO on the Ethereum network.

The organizers kicked off a month long token sale and at the time of this writing just over 10 million ether (the native currency of the Ethereum blockchain) — or approximately 13% of all mined ether — has been sent to The DAO.  This is roughly equivalent to over $100 million based on the current market price of ether (ETH).

In return for sending ether to The DAO, users receive an asset called a DAO Token which can be used in the future to vote on projects that The DAO wants to fund.2 It is a process that Swarm failed at doing.

An investment fund or a Kickstarter project?

I would argue that, while from a technical standpoint it is possible to successfully set up a DAO in the manner that The DAO team did, that there really isn’t much utility to do so in an environment in which censorship or the theft of funds by third parties will probably not occur.

That is to say, just as I have argued before that permissioned-on-permissionless is a shortsighted idea, The DAO as it is currently set up, is probably a solution to a problem that no one really has.3

Or in short, if you “invested” in The DAO crowdsale thinking you’re going to make money back from the projects via dividends, you might be better off investing in Disney dollars.


Putting aside securities regulations and regulators such as the SEC for a moment, most of the crowdsale “investors” probably don’t realize that:

  1. crowdfunding in general has a checkered track record of return-on-investment4
  2. crowdfunding in the cryptocurrency world almost always relies on the future appreciation of token prices in order to break-even and not through the actual creation of new features or tools (e.g., see Mastercoin/Omni which effectively flopped)
  3. that the funds, when dispersed to Slock.it and other “products,” could take years, if ever to return a dividend

Why would this pool of capital provide any better expected return-on-investment than others?

Or as Nick Zeeb explained to me:

My sense about The DAO is that it’s a fascinating experiment that I do not want to be part of. I also do not think that a committee of over 1,000 strangers will make wise investment decisions. Most good investment decisions are taken by courageous individuals in my opinion. Anything that can get past a big committee will probably not be the next Google. Imagine this pitch: “Hi I’m Larry and this is Sergey and we want to build the world’s 35th search engine.”

While it probably wasn’t the 35th search engine, tor those unfamiliar with the history of Google, Larry Page and Sergey Brin are the co-founders who created a search engine in what was then though a very crowded market.

So why the excitement?

I think part of it is quite simply: if you own a bunch of ether, there really isn’t much you can do with it right now.  This is a problem that plagues the entire cryptocurrency ecosystem.

Despite all the back-patting at conferences, the market is already filled with lots of different tokens. There is a glut of tokens which do not currently provide many useful things that you couldn’t already do with existing cash systems.5

Part of it also is that most probably think they will some become rich quick through dividends, but that probably won’t happen anytime soon, if at all.

With The DAO, only the development teams of projects that are voted and approved by The DAO (e.g., the thousands of users with DAO Tokens), will see any short term gains through a steady paycheck.  And it is only after they build, ship and sell a product that the original investors may begin seeing some kind of return.

Or in other words: over the past several weeks, the pooling of capital has taken place for The DAO.  In the future there will be various votes as to where that capital goes.  Shortly thereafter, some capital is deployed and later KPI’s will be assessed in order to determine whether or not funding should continue.  All the while some type of profit is sought and dividend returned.

Why, I asked another friend, would this pool of capital offer any better risk adjusted return-on-investment than other asset classes?

In his view:

The return might be high but so is the risk. Always adjust for risk. I think The DAO is better compared to a distributed venture capital firm. Whether that’s better or worse I don’t know — I mean you have the crowd deciding on investments. Or more realistically: nerds who know how to obtain ether (ETH) get to decide on investments.

Does that make them better VCs? Probably not. However, The DAO can decide to hire people with actual credentials to manage and select the investments, admitting its own weakness which would then turn into a strength. I think this can go either way but given the regulator is not prepared for any of this it will probably not work out in the short term.

Does the ‘design-by-giant-nerd-committee’ process work?

Over the past year we have already seen the thousands, probably tens-of-thousands of man-hours dropped into the gravity well that is known as the “block size debate.”  In which hundreds of passionate developers have seemingly argued non-stop on Slack, Twitter, reddit, IRC, conferences and so forth without really coming to an amicable decision any one group really likes.

So if block size-design-by-committee hasn’t worked out terribly well, will the thousands of investors in The DAO take to social media to influence and lobby one another in the future?  And if so, how productive is that versus alternative investment vehicles?

Redistributing the monetary base

Assuming Ethereum has an economy (which it probably doesn’t by most conventional measures), will The DAO create a deflationary effect on the Ethereum economy?

For instance, at its current rate, The DAO could absorb about 20% of the ether (ETH) monetary base.

Does that mean it permanently removes some of the monetary base?  Probably not.

For example, we know that there will be some disbursements to projects such as Slock.it, so there will be some liquidity from this on-chain entity.  And that future DAOs will spend their ether on expenses and development like a normal organization.

But we also know that there is a disconnect between what The DAO is, an investment fund, with what many people see it as: a large vault filled with gold laying in Challenger Deep that will somehow appreciate in value and they will be able to somehow extract that value.

Sure, we will all be able to observe that the funds exist at the bottom of the trench, but someone somewhere has to actually create value with the DAO Tokens and/or ether.

For the same reason that most incubators, accelerators and VC funds fail, that entrepreneur-reliant math doesn’t change for The DAO.  Not only does The DAO need to have a large volume of deal flow, but The DAO needs to attract legitimate projects that — as my friend point out above — have a better risk adjusted return-on-investment than other asset classes.

Will the return-on-investment of the DAO as an asset class be positive in the “early days”?  What happens when the operators and recipients of DAO funds eventually confront the problem of securities regulation?

So far, most of the proposals that appear to be geared up for funding are reminiscent to hype cycles we have all seen over the past couple of years.

Let’s build a product…

  • 2014: But with Bitcoin
  • 2015: But with Blockchain
  • 2016: But with DAO

Maybe the funds will not all be vaporized, but if a non-trivial amount of ETH ends up being held in this DAO or others, it could be the case that with sluggish deal flow, a large portion of the funds could remain inert.  And since this ether would not touching any financial flows; it would be equivalent to storing a large fraction of M0 in your basement safe, siloed off from liquid capital markets.

Ten observations

  1. Since the crowdsale / crowdfund began on April 30, the market price of ETH has increased ~30%; is that a coincidence or is there new demand being generated due to The DAO crowdsale?
  2. A small bug has been discovered in terms of the ETH to DAO Token conversion time table
  3. The DAO surpassed the Ethereum Foundation to become the largest single holder of ether (note: the linked article is already outdated)
  4. In terms of concentration of wealth: according to Etherscan, the top 50 DAO Token holders collectively “own” 38.49% of The DAO
  5. The top 500 DAO Token holders collectively “own” 71.39% of The DAO
  6. As of this writing there are over 15,000 entities (not necessarily individuals) that “own” some amount of a DAO Token
  7. Why is “own” in quotation marks? Because it is still unclear if controlling access to these private keys is the same thing as owning them.  See also: Watermarked Tokens as well as The Law of Bitcoin
  8. Gatecoin, which facilitated the crowdsale of both The DAO and DigixDAO was recently hacked and an estimated $2 million in bitcoins and ether were stolen
  9. Yesterday Gavin Wood, a co-founder of Ethereum, announced that he is stepping down as a “curator” for The DAO.  Curators, according to him, are effectively just individuals who identify whether someone is who they say they are — and have no other duties, responsibilities or authority.
  10. Three days ago, the Slock.it dev team — some of whom also worked on creating The DAO — did a live Q/A session that was videotaped and attempted to answer some difficult questions, like how many DAO Tokens they individually own.


About 17 months ago I put together a list of token crowdsales.  It would be interesting to revisit these at some point later this year to see what the return has been for those holders and how many failed.

For instance, there hasn’t really been any qualitative analysis of crowdsales or ICOs in beyond looking at price appreciation.6 What other utility was ultimately created with the issuance of say, factoids (Factom tokens) or REP (Augur tokens)?

Similarly, no one has really probed Bitcoin mining (and all POW mining) through the lens of a crowdsale on network security. Is every 10 minutes an ICO? After all, the scratch-off contest ties up capital seeking rents on seigniorage and in the long run, assuming a competitive market, that seigniorage is bid away to what Robert Sams has pointed out to where the marginal cost equals the marginal value of a token. So you end up with this relatively large capital base — divorced from the real world — that actually doesn’t produce goods or services beyond the need to be circularly protected via capital-intensive infrastructure.

Other questions to explore in the future include:

  • what are the benefits, if any, of using a centralized autonomous organization (CAO) versus decentralized autonomous organization (DAO) for regulated institutions?
  • how can a party or parties sue a decentralized autonomous organization? 7
  • what are the legal implications of conducting a 51% attack on a network with legally recognized DAOs residing on a public blockchain?8
  • will the continued concentration of ether and/or DAO Tokens create a 51% voting problem identified in the “Curator” section?

Still don’t fully understand what The DAO is?  Earlier this week CoinDesk published a pretty good overview of it.

[Special thanks to Raffael Danielli, Robert Sams and Nick Zeeb for their thoughts]


  1. Note: for the purposes of The DAO, “curators” are effectively identity oracles. []
  2. It appears that currently, once a quorum is achieved, a relatively small proportion of token holders can vote “yes” to a proposal to trigger a large payout. []
  3. The current line-up of goods and services are not based around solving for problems in which censorship is a threat, such as those facing an aid worker in a politically unstable region. []
  4. That is not to say that they all fail. In fact according to one statistic from Kickstarter, there was a 9% failure rate on its platform. Thus, it depends on the platform and what the reward is. []
  5. CoinGecko is tracking several hundred tokens. []
  6. ICO stands for “initial coin offering” — it is slight twist to the term IPO as it relates to securities. []
  7. An added wrinkle to identifying liable parties is: what happens when systems like Zcash launch? []
  8. This presupposes that a DAO will gain legal recognition and/or a public blockchain gains legal standing as an actual legal record. []
Send to Kindle

What is the difference between Hyperledger and Hyperledger?

hyperledgerI am frequently asked this question because there is some confusion related to the legacy name and the current branding of certain technology. The two are distinct. And how we got there involves a little history.

Hyper, the parent company of Hyperledger, was founded by Dan O’Prey and Daniel Feichtinger in the spring of 2014. Fun fact: one of the alternative names they considered using was “Mintette.com” — after the term coined by Ben Laurie in his 2011 paper.

The simplest way to describe Hyperledger, the technology platform from Hyper, during its formative year in 2014 was: Ripple without the XRP. Consensus was achieved via PBFT.1 There were no blocks, transactions were individually validated one by one.

Hyperledger, the technology platform from Hyper, was one of the first platforms that was pitched as, what is now termed a permissioned distributed ledger: validators could be white listed and black listed. It was designed to be first and foremost a scalable ledger and looked to integrate projects like Codius, as a means of enabling contract execution.

Most importantly, Hyperledger in 2014 was not based off of the Bitcoin codebase.

Note: in the fall of 2014 Richard Brown and I both became the first two advisors to Hyper, the parent company of Hyperledger.  Our formal relationship ended with its acquisition by DAH.2

In June 2015, DAH acquired Hyper (the parent company of Hyperledger) which included the kit and caboodle: the name brand, IP and team (the two Dans).  During the same news release, it was announced that DAH had acquired Bits of Proof, a Hungary-based Bitcoin startup that had designed a Java-based reimplementation of Bitcoin (which previously had been acquired by CoinTerra).3

It was proposed at that time that Hyperledger, the Hyper product, would become the permissioned ledger project from DAH.  It’s product landing page (courtesy of the Internet Archive) uses roughly the same terminology as the team had previously pitched it (see also the October homepage older homepage for DAH as well).

digital asset homepage october 2015

Source: Digital Asset / Internet Archive

On November 9, 2015, on a public blog post DAH announced that it was “Retiring Hyperledger Beta, Re-Open Sourcing Soon, and Other Changes.”

The two most notable changes were:

(1) development would change from the languages of Erlang and Elixir to Java and Scala;

(2) switch to the UTXO transaction model

The team noted on its blog in the same post:

We are also switching from our simplistic notion of accounts and balances to adopt to de facto standard of the Bitcoin UTXO model, lightly modified. While Hyperledger does not use Bitcoin in any way, the Bitcoin system is still extremely large and innovative, with hundreds of millions of dollars invested. By adopting the Bitcoin transaction model as standard, users of Hyperledger will benefit from innovation in Bitcoin and vice versa, as well as making Hyperledger more interoperable.

During this same time frame, IBM was working on a project called OpenChain, which for trademark reasons was later renamed (now internally referred to as OpenBlockchain).4

IBM’s first public foray into distributed ledgers involved Ethereum vis-a-vis the ADEPT project with Samsung (first announced in January 2015). Over the subsequent months, IBM continued designing its own blockchain (see its current white paper here).

In December 2015, the Linux Foundation publicly announced it was creating a new forum for discussion and development of blockchain technology.  Multiple names were proposed for the project including Open Ledger (which was the name originally used in the first press release). However, in the end, the name “Hyperledger” was used.

How did that occur?

DAH, one of the founding members of the project, donated two things to the Linux Foundation: (1) the brand name “Hyperledger” and (2) the codebase from Bits of Proof.

Recall that Bits of Proof was the name of a Bitcoin startup that was acquired by DAH in the fall of 2014 (the Chief Ledger Architect at DAH was the co-founder of Bits of Proof). 5 Architecturally, Bits of Proof is a Java-implementation of Bitcoin. 6

In other words: today the term “Hyperledger” represents an entirely different architectural design and codebase than the original Hyperledger built by Hyper.7

The major architectural switch occurred in November 2015, which as noted above involved adopting the UTXO transaction set and Java language that Bits of Proof was built with.  Therefore, Hyperledger circa 2016 is not the same thing as Hyperledger circa 2014.

Over the past two months there have been multiple different codebases donated to the Linux Foundation all of which is collectively called “Hyperledger” including the IBM codebase (partly inspired by Ethereum) as well as the DAH and Blockstream codebase (one is a clone of Bitcoin and the other is a set of extensions to Bitcoin). The technical discussions surrounding this can be found on both the public Linux Foundation mailing list and its Slack channel.

How do different, incompatible codebases work as one?

This technical question is being discussed in the Linux Foundation. It bears mentioning that as of now, the codebases are incompatible largely due to the fact that Bitcoin uses the UTXO transaction set and OpenBlockchain uses an “accounts” based method for handling balances.  There are other reasons for incompatibility as well, including that they are written in completely different languages: Java/Scala versus Go versus C++ (Blockstream).

How extensive is the reuse of the Bits of Proof Bitcoin codebase donated to the Linux Foundation from the DAH team?  According to a quick scan of their GitHub repo:

So when someone asks “what is Hyperledger technology?” the short answer is: it is currently the name of a collective set of different codebases managed by the Linux Foundation and is not related to the original distributed ledger product called Hyperledger created by Hyper. The only tenuous connection is the name.

Timeline in brief: Hyperledger was originally created in Spring 2014 by Hyper; Hyper was acquired in June 2015 by DAH; the original Hyperledger architecture was entirely replaced with Bits of Proof in November 2015; the Hyperledger brand name and Bits of Proof code was donated to the Linux Foundation in December 2015.

  1. Interestingly enough, the current OpenBlockchain project from IBM also uses PBFT for its consensus mechanism and uses an “accounts” based method; two characteristics that the original Hyperledger platform from Hyper had too. []
  2. For more info on the original Hyperledger, see the Innotribe pitch; the description in Consensus-as-a-service from April 2015 and the Epicenter Bitcoin interview. []
  3. Following the bankruptcy of CoinTerra, the Bits of Proof team became independent once again. []
  4. CoinPrism launched a project called OpenChain, before IBM did. []
  5. Sometimes there is a confusion between Bits of Proof and Bits of Gold.  Bits of Proof was the independent Java-implementation of Bitcoin (which is not the same thing as bitcoinj).  Bits of Gold is an Israeli-based Bitcoin exchange.  A co-founder of Bits of Gold also works at DAH and is their current CTO. []
  6. In the future it may contain some modifications including Elements from Blockstream. []
  7. What was once the original Hyperledger GitHub repo has been handed over to the Linux Foundation but some of the original code base and documentation from the 2014 project can still be viewed elsewhere. []
Send to Kindle

Watermarked tokens and pseudonymity on public blockchains

As mentioned a couple weeks ago I have published a new research paper entitled: “Watermarked tokens and pseudonymity on public blockchains

In a nutshell: despite recent efforts to modify public blockchains such as Bitcoin to secure off-chain registered assets via colored coins and metacoins, due how they are designed, public blockchains are unable to provide secure legal settlement finality of off-chain assets for regulated institutions trading in global financial markets.

The initial idea behind this topic started about 18 months ago with conversations from Robert Sams, Jonathan Levin and several others that culminated into an article.

The issue surrounding top-heaviness (as described in the original article) is of particular importance today as watermarked token platforms — if widely adopted — may create new systemic risks due to a distortion of block reorg / double-spending incentives.  And because of how increasingly popular watermarked projects have recently become it seemed useful to revisit the topic in depth.

What is the takeaway for organizations looking to use watermarked tokens?

The security specifications and transaction validation process on networks such as the Bitcoin blockchain, via proof-of-work, were devised to protect unknown and untrusted participants that trade and interact in a specific environment.

Banks and other institutions trading financial products do so with known and trusted entities and operate within the existing settlement framework of global financial markets, with highly complex and rigorous regulations and obligations.  This environment has different security assumptions, goals and tradeoffs that are in some cases opposite to the designs assumptions of public blockchains.

Due to their probabilistic nature, platforms built on top of public blockchains cannot provide definitive settlement finality of off-chain assets. By design they are not able to control products other than the endogenous cryptocurrencies they were designed to support.  There may be other types of solutions, such as newer shared ledger technology that could provide legal settlement finality, but that is a topic for another paper.

This is a very important issue that has been seemingly glossed over despite millions of VC funding into companies attempting to (re)leverage public blockchains.  Hopefully this paper will help spur additional research into the security of watermarking-related initiatives.

I would like to thank Christian Decker, at ETH Zurich, for providing helpful feedback — I believe he is the only academic to actually mention that there may be challenges related to colored coins in a peer-reviewed paper.  I would like to thank Ernie Teo, at SKBI, for creating the game theory model related to the hold-up problem.  I would like to thank Arthur Breitman and his wife Kathleen for providing clarity to this topic.  Many thanks to Ayoub Naciri, Antony Lewis, Vitalik Buterin, Mike Hearn, Ian Grigg and Dave Hudson for also taking the time to discuss some of the top-heavy challenges that watermarking creates.  Thanks to the attorneys that looked over portions of the paper including (but not limited to) Jacob Farber, Ryan Straus, Amor Sexton and Peter Jensen-Haxel; as well as additional legal advice from Juan Llanos and Jared Marx.  Lastly, many thanks for the team at R3 including Jo Lang, Todd McDonald, Raja Ramachandran and Richard Brown for providing constructive feedback.

Watermarked Tokens and Pseudonymity on Public Blockchains

Send to Kindle

What are a few direct and indirect costs of the “block size debate”?

About six weeks ago I mentioned a dollar figure during a panel at the Consensus event in NYC: $6 million. Six million USD is a loose estimate — for illustrative purposes — of the amount of engineering time representing thousands of man hours over the past 7-9 months that has gone into a productivity black hole surrounding the Bitcoin block size debate.

A little recent history

While there had been some low intensity discussions surrounding block size(s) over the past several years, most of that simmered in the background until the beginning of 2015.

On January 20th Gavin Andresen posted a 20 MB proposal which was followed over the subsequent weeks by a number of one-and-done counterpoints by various developers.

About four months later, beginning on May 4, Gavin posted a series of blog articles that kicked things up a notch and spurred enormous amounts of activity on social media, IRC, web forums, listservs, podcasts and conferences.

The crescendo of public opinion built up over the summer and reached a new peak on August 15th with a post from Mike Hearn, that Bitcoin would fork into two by the beginning of next year.

The passionate enthusiasts on all sides of the spectrum took to social media once again to voice their concerns.  During the final two weeks of August, the debate became particularly boisterous as several moderators on reddit began to ban discussions surrounding Bitcoin XT (among other forks and proposals).  There was even an academic paper published that looked at the sock puppets involved in this period: Author Attribution in the Bitcoin Blocksize Debate on Reddit by Andre Haynes.

Ignoring the future evolution of block size(s), with respect to the opportunity costs of the debate itself: investors and consumers have unintentionally funded what has turned out to be a battle between at least two special interest groups. 1

So where does the $6 million figure come from?

Of the roughly $900 million of VC funding related to Bitcoin itself that has been announced over the past 3 years, about half has been fully spent and went towards legal fees, domain names, office rent, conference sponsorship’s, buying cryptocurrencies for internal inventory and about a dozen other areas.2

At the current burn rate, Bitcoin companies collectively spend about $8-$10 million a month, perhaps more.  And since the debate is not isolated to development teams, because upper management at these companies are involved in letter writing campaigns (and likely part of the sock puppet campaigns), then it could be the case that 5-10% of on-the-clock time at certain companies was spent on this issue.

Consequently, this translates into about $400,000 to $1 million each month which has been redirected and spent funding tweets, reddit posts, blog posts, conferences, research papers and industry conferences.3

What about specific numbers?

For instance, with around 150-200 attendees the Montreal scalability conference likely absorbed $250,000 from everyone involved (via travel, lodging, food, etc.).  Similarly, one independent estimate that Greg Maxwell mentioned at the same Consensus event was his back-of-the-envelope projection of the opportunity costs: a few hundred thousand USD in the first couple weeks of May alone as engineers were distracted with block sizes instead of shipping code.

While a more precise number (+/-) could probably be arrived at if someone were to link individual developer activity on the dev mailing list/reddit/twitter with their estimated salaries on Glassdoor — since this past spring roughly $6 million or so has probably gone towards what has amounted to basically two diametrically opposed political campaigns.

And the issue is still far from resolved as there are more planned scalability conferences, including one in Hong Kong in early December.

Why is it a black hole though?  Surely there is utility from the papers and projects like Lightning, right?

It’s a money pit because it doesn’t and cannot resolve the coordination problem that decentralized governance creates.  I have an upcoming paper that briefly touches on this issue (in Appendix A): the key point is that any time decision making is decentralized then specific trade-offs occur.

In this case, due to an intentional power vacuum in which there is no “leader,” special interest groups lobby one another for the de facto right to make decisions.  Some decisions, like raising the minimum transaction relay fees involve less tweets and downvotes and are for various reasons considered less important as others.  Yet ultimately, de jure decision making remains out of reach.

Not the first time to a rodeo

Because decentralized governance (and external social consensus) was/is a key feature for many cryptocurrencies, this type of political activity could happen again with say, increasing the money supply from 21 million or if KYC becomes mandatory for all on-chain interactions.

Again, this was bound to happen because of the tragedy of the commons: because the Bitcoin network is a public good that lacks an explicit governance structure.  Anytime you have a lack of formal governance you often end up with an informal power structure that makes it difficult to filter marketing fluff from sock puppets like Cypherdoc (aka Marc Lowe) from actual fact-filled research.

And this subsequently impacts any project that relies on the Bitcoin network as its security mechanism.  Why?  According to anecdotes, projects from new organizations and enterprises have reconsidered using public blockchains due to the aforementioned inherent governance hurdles alone.

After all, who do they call when the next Mexican standoff, block reorg or mutually assured destruction situation arises?  There is no TOS, EULA or service-level agreement and as a result they look at other options and platforms.4

  1. It is probably too simplistic to say that, with $6 million in funding, these same developers could have simply created a new system, like Ethereum, from scratch that factors in scalability challenges from day one.  It is unlikely that these same developers would have come to agreement on what to spend those funds on as well. []
  2. See What impact have various investment pools had on Bitcoinland? and Flow of investments funds in Bitcoinland []
  3. The academic term for this is single-issue politics. []
  4. For instance, Tezos was designed specifically with a self-amending chain in mind due to this issue. []
Send to Kindle

A few results from the first intentional stress test on a communal blockchain

I have covered the issue of increasing the Bitcoin block size a few times in the past:

Three days ago several individuals within the development community (and on reddit) — in order to test to see how the network would handle (and is impacted by) a large increase in transactions — went ahead and repeatedly sent transactions (via scrypts) onto the network.

Below are multiple graphs illustrating what this traffic looked like relative to “normal” days:

blockrio graphs

Source: blockr.io (over the past 30 days)

Above are two charts from Blockr.io illustrating the block sizes over time and average block fee over the past 30 days.

transaction fees in USD

Source: Blockchain.info (fees denominated in USD)

transactions per day

Source: Blockchain.info (number of transactions per day including popular addresses)

excluding chains

Source: Blockchain.info (excluding chains longer than 10)

Above are three charts from Blockchain.info covering the past year (365 days) activity related to: fees to miners, transactions to all addresses (including popular), transactions excluding chains longer than 10 (see Slicing data for an explanation).

statoshi clearing

Data Source: Statoshi.info / Image source (reddit thread)

Above is a screengrab from Statoshi.info (run by @lopp).  It illustrates the roughly 20 hour time period in which this stress test took place.


There were multiple reddit threads that attempted to break down the findings, below are some of their comments with slight amendments

  • A peak of approximately 24,000 unconfirmed Bitcoin transactions occurred
  • Nearly 133,000 transactions were included in blocks during one day, a new all time high
  • Blocks became full starting at block 358596 at 23:38 UTC
  • And remained consistently full until block 358609 at 03:21 UTC
  • The majority of mining pools cap block size at 0.75 MB instead of 1 MB
  • Some transactions were “mysteriously” not broadcast until 2 hours post their actual broadcast time (Broadcast between 23- 24:00 UTC, shows 02:54 UTC)
  • The majority of low fee/minimum fee transactions required 3-4 hours for the first confirmation

Brute force fan fiction

While not necessarily a surprise, for approximately $3,000 an individual can effectively spam the network, filling up blocks and annoying users for several hours.  Because it became increasingly expensive for transactions to be included within blocks, the “attack” probably is not the most effective way to cause many transactions to be permanently slowed down.

Yet it does show that the Maginot Line narrative — that the only way to “attack” the network is to acquire hundreds of millions of dollars in hashing power to brute force the network — is just fan fiction.  A well-organized and minimally financed group of savvy internet users — not even professional hackers — can create headaches for settlement systems, payment processors or anyone else running time sensitive applications reliant on a public blockchain.

Thus, as Robert Sams pointed out a couple weeks ago: it would probably be financially irresponsible for a large organization like NASDAQ to use a communal blockchain — whose pseudonymous validators are not held contractually liable or accountable for transaction processing (or attacks thereof) — to clear and settle off-chain assets (Ryan Selkis briefly touched on a similar point last week as well).  Whether this kind of test convinces NASDAQ and others to rethink their pilot programs on a public blockchain is an open question.

Governance issues with “the commons”

Over the past 4-5 weeks there are probably well over a hundred reddit threads, blog posts and Bitcoin Talk forum posts related to increasing the block size.

Instead of rehashing all of the arguments here, the decision to increase block sizes seems to boil down to two things:

  1. Conflicts in governance (e.g., politics and special interest groups)
  2. Subjectivity in how many nodes represent “decentralization”

The first issue is much harder, perhaps impossible to solve because no one owns the network — it is a communal, public good.  Chronically lacking a clear and effective governance model, decisions are typically made based on: how many retweets someone gets, how many upvotes a poster receives, or increasingly, Six Degrees of Satoshi: how often Satoshi directly responded to your comments in the past.

We see this quite frequently with the same clique of developers using a type of argument from authority.  Perhaps they are correct and one person was left “in charge” by fiat — by Satoshi one spring morning in 2011.  Yet it was not Satoshi’s network to “give” in the first place — he was not the bonafide owner.  No one is, which presents a problem for any kind of de jure governance.1

gavin mike hearn

Source: reddit

The second issue, in terms of how many validating nodes are needed for decentralization, this is an issue that Vitalik Buterin, Jae Kwon and several others have been talking about for over six months, if not longer.

In short, as block sizes increase in size, fewer validating nodes will operate on the network due to a number of factors but largely related to the economic costs of running them (bandwidth is typically cited as the biggest consideration).  We see this empirically occur over the past 18 months on the Bitcoin blockchain (with validators dropping from over 13,000 in March 2014 to just under 6,000 today).

Appealing to amorphous social contracts

Social contracts historically fall apart due to their nebulous mandate and they also — non-governmental versions specifically — typically lack explicit enforcement mechanisms.

Bitcoin suffers from both.  There is no terms of service or explicit service agreement to the end user.  Nor is there a way to enforce an “ethos” onto a physically decentralized userbase.

Yet ironically several key developers are now appealing to a social contract to make decisions for how block sizes should and should not evolve.

Irrespective of what is decided on social media, there will ultimately be a solution that arises in the coming months, but not everyone will be happy.

How to solve this in the future?  What are other projects doing?

Tezos, if we come to believe that it is valuable or safe (because others are using it, or is scientifically verified), has a self-amending model which bakes in governance into the code itself.

Ethereum is also trying to create specific, technical ways for “explicit governance” to direct its evolution as it achieves certain milestones.  For instance, its developers plan to eventually transition the proof-of-work process into a proof-of-stake network (via a poorly marketed “bomb“).

Whether either of these projects is successful is another topic, but at least the developers recognize the governance issue as paramount to the ultimate “success” of the project.

Other projects in the distributed ledger arena, such as the “permissioned” ledgers I did a report (pdf) on earlier last month, also do not have this type of governance problem due to the fact that they each have a private sponsor (sometimes in the form of an NGO, others in the form of a company) where the buck finally, explicitly stops.

There may be non-technical ways to govern (via organizational structure), but Bitcoin’s model is both ad hoc and largely devolves into unproductive shouting matches.  Is this really how a financial system and series of products is best developed?  Probably not.

But this is a topic for political archaeologists to pour through in the coming years.

Other experts weigh in

Chun Wang, who is a member of the F2Pool operating team (F2Pool, also known as Discus Fish, is one of the largest mining pools), made the following comment two days ago on the Bitcoin development mailing list:

Hello. I am from F2Pool. We are currently mining the biggest blocks on
the network. So far top 100 biggest bitcoin blocks are all from us. We
do support bigger blocks and sooner rather than later. But we cannot
handle 20 MB blocks right now. I know most blocks would not be 20 MB
over night. But only if a small fraction of blocks more than 10 MB, it
could dramatically increase of our orphan rate, result of higher fee
to miners. Bad miners could attack us and the network with artificial
big blocks. As yhou know, other Chinese pools, AntPool, BW, they
produces ASIC chips and mining mostly with their own machines. They do
not care about a few percent of orphan increase as much as we do. They
would continue their zero fee policy. We would be the biggest loser.
As the exchanges had taught us, zero fee is not health to the network.
Also we have to redevelop our block broadcast logic. Server bandwidth
is a lot more expensive in China. And the Internet is slow. Currently
China has more than 50% of mining power, if block size increases, I
bet European and American pools could suffer more than us. We think
the max block size should be increased, but must be increased
smoothly, 2 MB first, and then after one or two years 4 MB, then 8 MB,
and so on. Thanks.

I reached out to Andrew Geyl (Organ of Corti) to see what was on his mind.  He independently concurred with LaruentMT, who suggested re-running the tests a few more times for more data:

The transaction “stress test” was well overdue. It’s impossible to understand exactly how increasing block sizes (or even reducing time between blocks) will affect transaction confirmations if we’re only using the network to capacity, and Testnet won’t be much use.

By ensuring that there were more transactions than could be confirmed, we understand a little more about the limits of the network’s transaction transmission capacity. As soon as I get access to relevant data I’ll be trying to determine what factors limited the rate of transactions per block per second.

I think this “stress test” should be run again at some point on a Sunday (when it will have least impact on network users) and – to account for variance in block making – for longer than just 8 hours. Maybe 24 hours? If we are are warned ahead of time, this might be more palatable to the bitcoin users. Think of it as preventative maintenance.

I also reached out to Dave Hudson, proprietor of HashingIt.com.  He has run a number of models over the past year; two notable posts still stick out: 7 Transactions Per Second? Really? and The Myth Of The Megabyte Bitcoin Block.  Below are his new comments:

I’d really like to have time to think about the stress test some more and to look at the numbers, but it demonstrates something that I’m pretty sure a number of people have considered before: 51% attacks are not the biggest cause for concern with Bitcoin; there are dramatically easier ways to attack the system than to build 350 PH/s of hardware.

The delays resulting from large numbers of TX’s sent to the network were entirely predictable (I did the sims months ago).

I doubt this is the only problem area. Consider (and this has been raised a lot in discussions over block size increases) that a lot of miners use the relay network. Attacking that, or shutting it down via some means would certainly set things backwards, especially if we do see larger block sizes.

Other attacks would be massive-scale Sybil attacks. I know there’s the whole argument that it can’t be done, but of course it can. It would be trivial to set up malware that turned 100s of thousands of compromised systems into Bitcoin nodes (even better if this could be done against something embedded where users don’t run malware detection).

It seems to me that the fact this hasn’t happened before is because those people interested in Bitcoin at the moment are more interested in seeing it useful than in bringing it down. When cybercriminals are extorting money in Bitcoin then they want to see it succeed too, but my guess is that if they could find some other equally anonymous way to get paid then we’d have seen some large-scale assaults, not just a few thousand extra TXs done as a thought experiment.

The problem here is that most software designers can build really good working systems. They can follow secure coding rules to ensure that their software doesn’t have resource leaks and network security vulnerabilities, but then they don’t consider any part of the system that might not be under their direct control. It’s the assumed-correct behaviour of the rest of the world that tends to be where major risks come in. Constructing a Maginot Line is a waste of time and money when the attacker bypasses it instead. In fact the perceived strengths of a defence usually lead to complacence. The stress test was a great example of this; huge amounts of time have been spent analyzing 51% attacks when this was probably the least likely attack even years ago. It’s essentially back to the crypto geek cartoon where the super-strong password is not cracked technologically, but instead by threatening its owner.

Despite what some entrepreneurs and venture capitalists have proclaimed — that there is a “scalability roadmap” — this is probably not the last time we look at this.

There are certainly proposed roadmaps that scale, to a point, but there are many trade offs. And it appears that some of the hosted wallet and payment processors that have publicly stated they are in favor of Gavin Andresen’s proposal are unaware of the impact that this type of block size increase has.  How it likely accelerates the reduction of nodes and how that likely creates a more centralized network (yet with the costs of decentralization).  Or maybe they are and simply do not think it is a real issue.  Perhaps they are correct.

One final comment — and this is tangential to the conversation above — is that by looking at the long chain exclusion chart we observe that the additional “stress test transactions” appear as normal unchained transactions.

This is interesting because it illustrates how easy it is to inflate the transaction volume metric making it less useful in measuring the health or adoption of the network.  Thus it is unlikely that some (all?) Bitprophets actually know what comprises transactions when they claim the Bitcoin network has reached “an all time high.”  Did they do forensics and slice the data?

See also: Creating a decentralised payment network: A study of Bitcoin by Jonathan Levin and Eclipse Attacks on Bitcoin’s Peer-to-Peer Network by Heilman et al.

  1. See Bitcoin faces a crossroads, needs an effective decision-making process by Arvind Narayanan []
Send to Kindle

Consensus-as-a-service: a brief report on the emergence of permissioned, distributed ledger systems

I have spent the past month compiling research that took place between August and the present day.  This was much more of a collaborative process than my previous publications as I had to talk with not just 8 geographically dispersed teams to find out what their approach was in this nascent field but also find out who is working on ideas that are closely related to these projects (as seen in Appendix A).

The culmination of this process can be found in this report: Permissioned distributed ledgers

Fortunately I had the help of not just astute practitioners in the industry who did the intellectual heavy lifting, but the resources and experience of the R3 CEV team where I am an advisor.

I think the three strongest areas are:

  • Richard Brown’s and Jo Lang’s description and visualization of smart contracts.  I loathe the term smart contracts (I prefer “banana” and Preston Byrne prefers “marmot”) and fortunately they distilled it to a level where many professionals can probably begin to understand it
  • Meher Roy’s excellent OSI-model for an “internet of money”
  • Robert Sams mental model of the core attributes of a permissioned distributed ledger

I think the weakest part is in the beginning of Section 8 regarding TCP/IP.  That is reflective of the fact that there is no perfect analogy because Bitcoin was designed to do many things that no other system does right now so there probably is no single apple’s to apple’s comparison.

While you do not need special internetcoins or fun buxx to use the internet (as it were), there is still a cost to someone to connect to the net.  So perhaps, the frictional differences between obtaining and securing an internet connection versus obtaining and securing a bitcoin at this time is probably something that should be highlighted more if the report is updated.

Wither Bitcoin?

For cryptocurrencies such as Bitcoin to do what it does best on its own terms, its competitive advantage lays with the native token and not representing real-world assets: its community needs to come to terms about what it is and is not good for.  Because of its inability to control off-chain assets its developers should stop promising that bitcoins — or metacoins and watermarked-coins that use Bitcoin as a transportation layer — as a panacea for managing off-chain assets, assets the network cannot control.  At most Bitcoin’s code base and node network operates as its own legal system for non-watermarked bitcoins.

Consequently, the advantage a cryptocurrency system has is endogenous enforcement of contractual terms — or as Taulant Ramabaja calls it: “fully blockchain endogenous state transition without any external dependencies.”  Or on-chain, dry code to dry code.

I wonder if someone in the future will call themselves a full “dry code” stack developer?

Consensus-as-a-service: a brief report on the emergence of permissioned, distributed ledger systems

Send to Kindle

Panel from Blockchain University Demo Day

A couple weeks ago I moderated a panel at Blockchain University, wrapping up the inaugural cohort.

Panelists included Atif Nazir (co-founder of Block.io), Matthieu Riou (co-founder of BlockCypher) and Greg Slepak (co-founder of okTurtles Foundation).  All three were instructors for the course this past winter.

Send to Kindle

A panel on smart contracts with industry developers and educators

Earlier today I participated in a virtual panel covering smart contracts called, “Let’s Talk Smart Contracts.”

The panel included: Adam Krellenstein (Counterparty), Oleg Andreev (CoreBitcoin), Pamela Morgan (Empowered Law), Stefan Thomas (Codius, Ripple Labs), Stephan Tual (Ethereum), Tim Swanson (Of Numbers), Yurii Rashkovskii (Trustatom) and it was moderated by Roman Snitko with Straight.

Below are some transcribed notes of my own statements.

Introduction starting at 09:06:

Hey guys, great to be here.  Thanks for the invite, thanks for organizing this.  So I’m here because you guys needed another white guy from Europe or something like that (that’s a joke).  So the definition I have of smart contracts, I have written a couple books in this space, and the definition I use is a smart contract is “a proposed tool to automate human interactions: it is a computer protocol – an algorithm – that can self-execute, self-enforce, self-verify, and self-constrain the performance of a contract.”  I think I got most of that definition from Nick Szabo’s work.  For those of you who are familiar with him, look up some of his past writings.  I think that the primary work he is known for is the paper, “Formalizing and Securing Relationships on Public Networks.”  And he is basically considered the [intellectual] grandfather of this space.  I’m here basically to provide education and maybe some trolling.

From 22:02 -> 24:15

I think I see eye-to-eye with Adam here.  Basically the idea of how we have a system that is open to interpretation, you do have reversibility, you do have nebulousness.   These are things that Nick Szabo actually discussed in an article of his called “Wet code and dry” back in 2008.  If you look back at some of the earlier works of these “cypherpunks” back in the ’90s, they talked about some of these core issues that Oleg talked about in terms of being able to mitigate these trusted parties.  In fact, if you look at the Bitcoin whitepaper alone, the first section has the word “reverse” or “reversibility” around 5 times and the word “trust” or “trusted” appears 11 times in the body of the work.  This was something that whoever created Bitcoin was really interested in trying to mitigate the need for any kind of centralized or third party involved in the process of transactions to reduce the mediation costs and so forth.

But I suppose my biggest criticism in this space, it is not pointed to anyone here in particular, is how we have a lot of “cryptocurrency cosplay.”  Like Mary Sue Bitcoin.  I’m not sure if you guys are familiar with who Mary Sue is: she is this archetype who is this kind of idealized type of super hero in a sense.  So what happens with Bitcoin and smart contracts is that you have this “Golden Age” [of Comics] where you had the limited ideas of what it could do.  Like Superman for example, when he first came out he could only jump over a building and later he was pushed to be able to fly because it looks better in a cartoon.  You have only a limited amount of space [time] and it takes too long to jump across the map.  So that’s kind of what I see with Bitcoin and smart contracts.  We can talk about that a little bit later, just how they have evolved to encompass these attributes that they’re probably not particularly good at.  Not because of lack of trying but just because of the mechanisms of how they work in terms of incentives for running mining equipment and so on.  So, again we can talk about that later but I think Adam and Oleg have already mentioned the things that are pretty important at this point.

40:18 -> 41:43

I’m the token cynic, huh?  So actually before I say anything, I would like to mention to the audience other projects that you might be interested in looking at: BitHalo; NotaryChains is a new project that encompasses some of these ideas of Proof of Existence created by Manuel Araoz, he is the one who did POE.  NotaryChains is a new project I think that sits on top of Mastercoin.  The issue that people should consider is that proof of existence/proof of signature: these are just really hi-tech forms of certification.  Whether or not they’re smart contracts I guess is a matter of debate.

There is another project: Pebble, Hyperledger, Tezos, Tendermint, Nimblecoin.  With Dogethereum their project is called Eris which apparently is the first DAO ever.  A DAO for the audience is a decentralized autonomous organization, it’s a thing apparently. SKUChain is a start-up in Palo Alto, I talk about them in chapter 16.  They have this interesting idea of what they call a PurchaseChain which is a real use-case for kind of updating the process from getting a Letter of Credit to a Bill of Lading and trying to cut out time and mediation costs in that process.  There are a few others in stealth mode.  So I really don’t have a whole lot to add with cynicism at this point, we can go on and come back to me in a little bit.

59:41 -> 1:02:35

The go to deficiency guy, huh?  They’re not really saying anything particularly controversial, these things are fundamentally — at least from an engineering perspective — could be done.  The problem though I think runs into is what Richard Boase discussed in — if listeners are interested — he went to Kenya and he did a podcast a few weeks ago on Let’s Talk Bitcoin #133.  I really recommend people listen to it.  In it he basically talks about all of these real world issues that run into this idealized system that the developers are building.  And as a result, he ended up seeing all of these adoption hurdles, whether it was education or for example tablets: people were taking these tablets with bitcoin, and they could just simply resell it on a market, the tablet itself was worth more than they make in a year basically; significant more money.  He talked about a few issues like P2P giving, lending and charity and how that doesn’t probably work like we think it does.

I guess the biggest issue that is facing this space, if you want issues, is just the cost benefit analysis of running these systems.  There is a cost somewhere to run this stuff on many different servers, there is different ways to come up with consensus for this: for example, Ripple, Stellar, Hyperledger, they’re all using consensus ledgers which require a lot less capital expenditures.  But when you end up building something that requires some kind of mining process itself, that costs money.  So I think fundamentally in the long-run it won’t be so much what it can do but what can it economically do.

So when you hear this mantra of let’s decentralize everything, sure that’s fine and dandy but that’s kind of like Solutionism: a solution looking for a problem.  Let’s decentralize my hair — proof of follicle — there is a certain reductio ad absurdum which you come to with this decentralization.  Do you want to actually make something that people are actually going to use in a way that is cheaper than an existing system or we just going to make it and throw it out there and think they’re going to use it because we designed [wanted] it that way.  So I think education is going to be an issue and there are some people doing that right now: Primavera De Fiillipi, she’s over at Harvard’s Berkman Center — she’s got something called the Common Accord program.  And also Mike Hearn; listeners if you’re interested he’s made about 7 or 8 use-cases using the existing Bitcoin blockchain including assurance contracts — not insurance contracts — assurance contracts.  And he’s got a program called Lighthouse which hopes to build this onto the actual chain itself.  So there are things to keep in mind, I’m sure I’ll get yelled at in a minute here.

1:23:58 -> 1:28:10

Anyone listening to this wanting to get involved with smart contracts: hire a lawyer, that’s my immediate advice.  I will preface by saying I don’t necessarily agree with policies that exist and so on; I don’t personally like the status quo but there is no reason to be a martyr for some crusade led by guys in IRC, in their little caves and stuff like that.  That’s not towards anyone here in this particular chat but you see this a lot with “we’re going to destroy The Fed” or “destroy the state” and the reality is that’s probably not going to happen.  But not because of lack of trying but because that’s not how reality works.

Cases right now are for example: DPR, Shavers with the SEC, Shrem now with the federal government, Karpeles [Mt. Gox] went bankrupt.  What’s ended up happening is in 2009, with Bitcoin for example, you started with a system that obviated the need of having trusted third parties but as users started adopting it you ended up having scams, stolen coins, people losing coins so you ended up having an organic growth of people wanting to have insurance or some way to mediate these transactions or some way to make these things more efficient.  And I think that it will probably happen — since we’re guessing, this is speculative — I think that this will kind of happen with smart contracts too.  That’s not to say smart contracts will fail or anything like that.  I’m just saying that there will probably just be a few niche cases initially especially since we don’t have much today, aside I guess from Bitcoin — if you want to call it a smart contract.

What has ironically happened, is that we have created — in order to get rid of the middlemen it looks like you’ve got to reintroduce middlemen.  I’m not saying it will always be the case.  In empirical counter-factual it looks like that’s where things are heading and again obviously not everyone will agree with me on that and they’ll call me a shill and so on.  But that’s kind of where I see things heading.

I have a whole chapter in a book, chapter 17.  I interviewed 4 or 5 lawyers including Pamela [Morgan] of different reasons why this could take place.  For example, accredited investor — for those who are unfamiliar just look up ‘accredited investor.’  If you’re in the US, in order to buy certain securities that are public, you need to have gone through certain procedure to be considered a ‘sophisticated investor.’  This is one of the reasons why people do crowdsales outside of the US — Ethereum — because you don’t want to have to interact with the current legal system in the US.  The reason I mention that is because you end up opening yourselves to lawsuit because chains — like SWARM — cannot necessarily indemnify users.  That’s legal terminology for being able to protect your users from lawsuits from third parties; they just do not have the money, the revenue to support that kind of legal defense.  Unlicensed practice of law (UPL) is another issue.  If you end up putting up contracts on a network one of the issues could be, at least in the US, are bar associations.  Bar associations want to protect their monopoly so they go after people who practice law without a license.  I’m not saying it will happen but it could happen.

My point with this is, users, anyone listening to this should definitely do your due diligence, do your education.  If you plan to get involved with this space either as an investor or developer or so on, definitely at least talk to a lawyer that has some inkling of of an idea [on this].  The ones I recommend, in addition to Pamela here are: Ryan Straus, he is a Seattle-based attorney with Riddell Williams; Austin Brister and James Duchenne they’re with a program called Satoshi Legal; and then Preston Byrne, who’s out in London and he’s with Norton Rose Fulbright.

1:52:20 -> 1:54:43

Guys look, I understand that sounds cool in theory and it’s great to have everything in the background, but the reason you have to see these “shrink wrapped” EULAs [end user license agreements] and TOSs [terms of service] is because people were hiding stuff inside those agreements.  So if you hide what’s actually taking place in the contract you end up making someone liable for something they might not actually agree to.  So I’m not sure, I think it’s completely debatable at this point.  If we’re trying to be transparent, then you’re going to have to be transparent with the terms of agreement.

I should point out by the way, check out Mintchalk.com, it’s run by guys named James and Aaron in Palo Alto, they’re doing contract building.  ACTUS is a program from the Stevens Institute, they’re trying to come with codified language for contracts.  Mark S. Miller, he’s got a program over at Google, he does something with e-rights.

I mention all of this because, we already have a form of “polycentric law” if you will in terms of internationally with 200 different jurisdictions vying for basically jurisdiction arbitrage.  Ireland and the Netherlands have a tax agreement that Facebook, Google, Pfizer they take advantage of.  It’s this Double Irish With a Dutch Sandwich.  In fact my own corporation is incorporated in Delaware because of the legal arbitrage [opportunities].  Obviously smart contracts might add some sort of new wrinkle to that, but people who are listening to this, don’t expect to be living in some Galt’s Gulch tomorrow or something like that.

For example, when you have something that is stolen, there is something called Coinprism which is a colored coin project.  They can issue dividends on stock.  The cool thing with that is, “hey, you get to decentralize that.”  The double-edged side of that is if that when that get’s stolen: people steal stuff like bitcoins and so forth, what happens to the performance of that dividend?  If the company continues paying that dividend in knowing that the person had been stolen from: if somebody stole from me and I tell the company, “hey, it was stolen” and they continue paying, then I can sue them for continuing to pay a thief.  If they stop paying then it defeats the purpose of decentralization because anonymity is given up, identity has taken place.  Obviously this moves into another area called “nemo dat” it’s another legal term talking about what can be returned to the rightful owner, that’s where the term “bona fide” comes from.  Anyways, I wanted to get that out there.  Be wary of disappearing EULAs, those have a purpose because people were being sued for hiding stuff in there.

2:10:05 -> 2:12:23

So I think everybody and all these projects are well-intentioned and have noble goals but they’re probably over-hyped in the short-run, just like the Segway was.  It eventually leads to some kind of burnout, or over-promise and under-delivering.  I’m not saying this will happen, I’m just saying it could happen.  I actually think the immediate future will be relatively mundane, such as wills and trusts kind of like Pamela was talking about.

One particular program is in Kenya there is something called Wagenitech which is run by Robin Nyaosi and he is wanting to help farmers move, manage and track produce to market to bypass the middleman.  That doesn’t seem like something really “sexy,” that doesn’t seem like the “Singularity” kind of thing that everyone likes to talk about.  But that is needed for maybe that particular area and I think we might see more of that along with PurchaseChain, NotaryChains, some of these things that we already do with a lot of the paperwork.

Again, blockchains and distributed ledgers are pretty good at certain things, but not everything.  It has real limitations that vocal adopters on the subreddit of Bitcoin like to project their own philosophical views onto it and I think that it does it a very big disservice to this technology long-term.  For example, LEGO’s can be used to make a car but you wouldn’t want to go driving around in one.  A laptop could be used as a paper weight but it’s not particularly cost effective to do that.  And so what I think we’ll end up running into a tautology with smart contracts, it’s going to be used by people who need to use them.  Just like bitcoin is.  So what we’re going to have is a divergence between what can happen, this “Superman” version of Bitcoin and smart contracts, versus the actual reality.

So for example, people say it’s [Bitcoin] going to end war.  You had the War of Spanish Succession, there was a Battle of Denain, a quarter million people fought that in 1712 and it was gold-based [financed by specie].  Everyone that says bitcoin is going to destroy fiat, if the state exists as it does today there’s always going to be these institutions and types of aggression.  I do think smart contracts do add collateral and arbitration competition and it does take away the problem of having trust in the system itself, but the edges are the kryptonite.  And always will be.  So we need to focus on education and creating solutions to real actual problems today with the actual technology and not just some hypothetical “Type 2” civilization where we are using [harvesting] the Sun for all of our energy.

Send to Kindle

The advantages and challenges of mining bitcoins in China

I received some feedback from a veteran of the mining subindustry in China regarding my previous research on this space.

According to him there are a number of other moving pieces at play that are fluid will not necessarily last.

For instance, providers such as HashRatio have succeeded, not by designing their own chip but by figuring out the best combination of system and power configurations.  Going from chip to working system is non-trivial.   The end result are systems which are not necessarily pretty to look at, but they work.

One of the issues this new source had with my report was that because of guanxi is relatively hard to quantify, knowing whether or not you have the best price of a particular resource (like energy) is always a lingering question.  That is to say, even if Alice knows the boss of a coal mine, another competitor, Bob, may know his bosses boss which gives Bob even cheaper rates than what you thought you were receiving.  Improving guanxi is a millennia old Herculean task.

Some other highlights according to the source:

  •        If Alice’s metric is purely dollars per ghash, the analysis was correct. This is because there are two important figures: Alice’s new ASIC kWh/hash multiplied by her electricity cost / kWh.
  •        While Moses Lake is quoted in many news reports at being 1.7 cents per kWh, there are many other parts of the state which are very low, some averaging 2.3 cents per kWh.  And Washington has a much better infrastructure (both for electricity and internet) than China which makes it a very competitive geographic region.
  •        Similarly, Russia is 1 to 1.2 cents per kWh, though, you would be in Russia.
  •        China is cheap relative to a lot of countries, but relative to Washington and Russia the community capacity is still limited by State Grid, a large state owned enterprise (SOE) with a flat rate of 0.3 RMB kWh buying in any power station linked to it.  Miners will likely be unable to go under that.
  •        While Alice can do some meter fiddling or go off grid power, those options are hard to find and probably will not last long.
  •        State Grid has likely heard of bitcoin mining, but the wattage usage is not big enough to pique their interest or oversight.
  •        Inner Mongolia, as part of China, has overinvested in wind farms.  Yet there are large areas that are not linked to the grid yet.  And due to the unstable nature of wind, as well as poor internet infrastructure, none of the mining pools has gone there yet.  And it is sparsely populated which leads to potential difficulties in sourcing human capital and talent to run a pool.
  •        Mongolia, the country, imports roughly 10-20% of its electricity from Russia, so Bob might as well go to Russia if he is willing to set up a facility in Mongolia.
Send to Kindle

Chapter 15 – Human resource and infrastructure challenges

[Note: below is Chapter 15 from Great Wall of Numbers]

Consulting firms in China are abundant and usually just a stone’s throw away.  The primary reason has to do with China’s developmental status: China currently lacks expertise and experience in several fields.  As a consequence many domestic companies are willing and increasingly have the funds to hire foreign experts to guide, manage and even direct operations at companies.  To paint a clearer picture of the situation, according to Chen Yuyu, associate professor at Peking University, “[h]igh-end jobs that should have been produced by industrialization, including research, marketing and accounting etc., have been left in the West.”1 As a consequence, because they are faced with dilemma of working in low-skilled, low-waged professions, a recent survey found that “among people in their early 20s, those with a college degree were four times as likely to be unemployed as those with only an elementary school education.”2

At the same time, it is advised that rosy enthusiasm – get-rich-quick in China – be tempered with a dose of reality.  For example, the Wall Street journal ran a piece in March 2012 which details the gradual shift away from recruiting expats at all corporate levels.3 This is due in part to increasingly expensive compensation packages needed to lure experienced expats and because of a growing talent pool of educated Chinese returning from overseas dubbed “sea-turtles” (hǎiguī). This changing outlook is best summed up by hedge fund manager Mark DeWeaver who recently told me in an interview,

I don’t think immigrating to China would be a logical choice for most foreigners.  There just wouldn’t be that many job opportunities for them, particularly if they don’t speak the language.  They would also be competing with the many Chinese graduates of US colleges that return home after graduating.4

Between 2000-2009, more than 630,000 Chinese-born immigrants received US green cards.5 Over the past 30 years more than 1.2 million Chinese studied abroad, approximately 20% of who matriculated to US schools and institutions.678 During the 2012-2013 school year, more than 190,000 Chinese students studied at US schools (up from 160,000 the year before) – they also comprise a quarter of all international students in the US.910 In fact, 37% of all international graduate students in the US now are Chinese nationals.11  While there is some overlap between the two groups and some manage to stay and attain green cards, some of the remaining – well-trained and educated – return home to join the Chinese workforce.12

According to the Ministry of Education, due in part to the incentives mentioned above in Chapter 9 (“1,000 Talents”) approximately 186,000 overseas Chinese returned to China in 2011, an increase of nearly 40%.1314 While some do move back to the West again, others stay.  For example, Kevin Woo is a Shanghai native who received an LL.M. from the University of Wisconsin yet works as an auditor for a large Chinese real-estate firm.  He returned to Shanghai in part due to the soft labor market in the US.  Anthony Wang received his bachelor’s and master’s from the University of Waikato and now works at his family-owned factory in Anhui.  Tony Wu received his bachelor’s from the University of Stirling and now works for AMER International Group, a large Chinese resource company in Shanghai.

So before packing your bags and flying out to China to open an office, you and your company need to answer the following questions: are you really a foreign expert?  Make a list of things you can do comparatively better than your Chinese counterpart.  What is your marginal productive value and what is the typical salary an expat with your skill set makes in China?  What are the advantages and disadvantages of opening an office overseas headed by a foreigner?

If you hesitated to answer at least one of these questions, remember that you and your company can always hire Western educated local Chinese who understand not only the complex culture of China but also can usually communicate effectively in English and understand many aspects of the West as well.

With that said, there are still a large number of multinationals that have moved in and set up shop on the mainland, recruiting both locals and expats alike.  Some notable examples in Shanghai are Indianapolis-based Eli Lilly which manages about 2,000 in the Pudong and Xintiandi districts; Sunnyvalle-based Intel which operates a 2,000 person division in Minhang, Shanghai, and another smaller office in Beijing of less than 1,000 workers (less than 5% are foreigners) both divisions focus on software development of chipset drivers; English First (EF), a Lucerne-based Swedish company which is the world’s biggest EFL training company and employs more than 2,100 full-time employees in the Shanghai metro alone, approximately 15% of which are expats.  In contrast, BP’s Pudong office has 200 employees, 10 of whom are foreigners; AIA’s Shanghai office only has about 150 non-sales employees; and Geneva-based Mercuria – a $75 billion resource multinational company (MNC) – operates a small corporate office of about 25 people also in Pudong.15

Some other auxiliary issues to consider before opening an office in China: according to the 2012 Expat Explorer survey, half of the expats recruited expect not only to earn more money upon relocation to China but also perks.16 For perspective consider that according to one October 2012 estimate that the per capita income of Tier 1 cities such as Guangzhou ($9200), Beijing ($8980) and Shanghai ($8325) are significantly higher than the average urban annual salary ($3,430).1718 For comparison according to the Social Security Administration the national average wage in the US in 2011 was $42,979.19 Yet Mercer’s 2012 ranking report on the most expensive cities notes that the cost of living for expats in China is disproportionally higher in these same cities: (being closer to 1st means more expensive) Shanghai is 16th, Beijing is 17th and  Guangzhou is 31st.2021 Similarly, an ECA International cost-of-living survey published in December 2012 found that Beijing is the 22nd and Shanghai is the 26th most expensive cities globally for expats.22 Why?  Because according to Lee Quane of ECA, “[e]ssentially what’s happening in China is that prices are rising at a faster rate than they are in the West, and that’s caused Beijing to leapfrog all those other locations in the rankings.”23 Or in other words, make sure to get firm budgetary numbers for the costs of: expat compensation packages (transportation costs, hardship perks, recruiting bonus) and rental property expenditures.

How large are these mainland cities?  Shanghai is the largest, with 23 million permanent residents, Beijing is slightly smaller with 20 million residents and Guangzhou has 16 million.24 Furthermore in terms of internet penetration rates across the country, Shenzhen has the highest (76.8%) followed by Guangzhou (72.9%), Beijing (70.3%) and Shanghai (66.2%).25 In contrast, Hong Kong is 68.7% and Singapore is 77.2%.  In terms of foreigners, despite the fact that more than 57 million inbound tourists visited the mainland in 2011 (see Chapter 4) there are only 600,000 foreigners who are permanent residents and 220,000 foreigners legally working on the mainland.26 Shanghai itself is home to the most foreign residents (200,000), roughly a third of all foreign residents on the mainland (in contrast Hong Kong has about 400,000 foreign residents).27

What city should you set up your first office and hire local labor from?  In addition to doing your due diligence regarding business licenses, you and your company should perform a cost-benefit analysis of mainland cities.  While labor costs are significantly cheaper in Tier 2 & 3 cities, salaries in Tier 1 cities also varied.  For example, the average monthly salary for an internet censor in Beijing is $653 whereas a similar censor in Tianjin is paid $480 a month.28 Similarly while land rental rates may be cheaper inland, larger metros like Shanghai, Guangzhou and Beijing typically have modern infrastructure (e.g., subways, well-maintained highways) which in turn attracts multinational corporations (MNCs).  Shanghai, which was according to a recent Forbes report is the best city for business on the mainland, itself has roughly 60 MNCs – more than any other city on the mainland.2930 This is due in part to subsidies and duty-free policies.  For example, a MNC can now receive an 8 million RMB ($1.3 million) subsidy for 3 years plus duty-free imports at facilities by setting up an office in Shanghai.31

Attracting, retaining and discovering talent and connections

Another seemingly mundane recruitment issue facing foreign and domestic companies alike is the labor hiring cycle.  Simply put, some recruiting months are not the same as others.  While most firms in large cities use the Gregorian calendar year for GAAP accounting, nearly every domestic firm celebrates holidays based on the traditional lunar calendar.  The biggest holiday of the year is Spring Festival or Chinese New Year, typically at the end of January to beginning of February.  Like their Japanese counterparts, it is customary for domestic companies to award significant bonuses – 20-50% of a month’s salary and even higher – to each employee just before Spring Festival.32 As a consequence, it becomes increasingly difficult to hire qualified workers after Mid-Autumn festival (also called Moon festival which is usually held in September) because employees not only would lose their potential bonus at the first, current company but would only be eligible to receive a reduced bonus at the new company.  This is just one more cultural issue US firms should be aware of before starting up a domestic office.

How hard is it to hire expats?  I posed this question to nearly every person I interviewed and the answer was unsurprisingly the same as it would be in other countries: compensation packages are usually the top priority.  And specifically, full medical insurance with coverage and reimbursements to private hospitals (see Chapter 19).  One of the reasons why this was important is that in the eyes of these managers, directors and CEOs, expats typically feel more comfortable in a foreign country if they knew they could have access to doctors and medical providers that spoke their native language.  As a consequence, firms looking to attract overseas talent may need to factor in the costs of medical reimbursements which can run up to 20,000 RMB ($3200) a night at some of the foreign owned and operated medical facilities.

Natalia Shuman, the new COO of Kelly Services’ in China mentioned in a recent interview that the top challenge in China for 2013 is,

I think retention and hiring talent are still going to be challenging.  More multinational companies are expanding their Chinese operations.  And the war for talent continues.  From the staffing and recruitment industry prospective, the operating environment here in China is tough: the competition is strong, limited collaboration between players, not enough regulations from the staffing associations, quality issues, cost pressures and price wars.33

In terms of retaining employees, in November 2012 I spoke with one foreign executive at a technology company in Shanghai who has employed a unique strategy in an attempt to kill two birds with one stone: retaining skilled employees and maintaining information integrity.  After losing several key staff to competitors, instead liquidating his assets he decided to go a different path, a “hollowed castle” route based on a strategy from Zhuge Liang in Romance of the Three Kingdoms (三国演义).

In a nutshell, there was a volatile period two thousand years ago when what we now know as China was divided into three warring states (三国时代).  One of the states (Shu) had its capital in Chengdu, in the contemporary western province of Sichuan.  This kingdom was ruled by a calculating leader named Liu Bei who had under his command an able minister and war general, Zhuge Liang.  At one point in this time period Liang had ordered all of his troops to leave the city and engage the enemy (Wei) capital of Chang’an (now Xi’an) to the north.  Yet his enemy took a different route avoiding a clash with the Shu, moving rapidly towards Chengdu whereupon they began preparations to lay siege to the Shu castle.  Liang, with little recourse attempted an unusual tactic: he opened the castle doors, disguised the remaining soldiers as civilians and played music from the top gates.  The leader of the opposing forces, Sima Yi, knew that Liang was a shrewd and calculating opponent and thus came to the conclusion that this ploy must be a trap.  So Yi withdrew his forces.  This type of reverse psychology is termed the Empty Fort Strategy (空城计).  Similarly this executive has since brought his subsequent teams into the fold, explicitly imparting the knowledge that they alone hold the key to their own long-term success – and that they could walk away at any time.  His staff turnover was subsequently lower largely due to what he considers from this frankness towards future revenue generation and employee trust.

In terms of specific retention examples in the service industry, when I spoke with both Scott Freeman and Richard Qi (see Chapter 13) they both noted that based on their experiences in the domestic IT industry there is a usually a dividing line of 1985.  That is to say, that the turnover rate is substantially higher for those born post-1985 (50-60%) than those born before it (20%).  Or in other words, the younger the employee, the riskier they may be – yet simultaneously, the younger the employee the more familiar they may be with new ways of thinking differently.  And it is a conundrum that is not endemic to China.

Another way to utilize and attract talent is a method used at Motion Global (MG).  Nira Binderer is an HR manager at MG in Shanghai and noted in our May 2012 interview that MG unequivocally sees China as the long-term home for its future base of operations.34 While it is no secret that from a wage and salary perspective it may make financial sense to hire local talent (e.g.,Chinese graduates from lesser known schools earn less than $350 a month at their first job35 ), Binderer said that MG had a unique external hiring strategy: hire expats as interns.  According to her, MG will give each intern a small stipend each month, but requires the interns to pay their own way (flights, accommodation, food) to show just how dedicated and genuine they are regarding SEO (search-engine optimization) and internet marketing.  After a two-month probation, their salaries increase proportionally to the success of their SEO campaigns (judged by analytic tools measuring click through rates, bounce rates, etc).  When I visited their Shanghai headquarters in May, more than two dozen foreigners (typically recent college graduates) were working side by side with local Chinese.  I spoke with one former intern, Miles Vaughn – now in Florida, who noted that he learned more in the months he was at MG than in any classroom.  In his words, “I hit the ground running and had a chance to not only learn as I went but each week we had a chance to talk with SEO teams from other companies including Google.”

While it would be difficult to convince the average expat to pay his or her own way just to be an intern at your new China office, hiring interns in general could help tide your firm over during cyclical periods such as the post-Mid-Autumn holiday (when fewer workers are willing to leave their employers due to bonus incentives).  Interns can also be viewed as an ongoing-asset.  After all your firm has invested both time and money in them, perhaps they can eventually be promoted to a permanent position in the future.

As mentioned several times previously as well as in Chapter 13, one of the problems that Larry Chang specifically faced when setting up shop in Shanghai five years ago was a lack of local contacts.  He did not know any businessmen or government employees and the locals were unfamiliar with him because he did not go to school with any of their colleagues, teachers or family members.  Thus building his guanxi (social, business, personal connections) was a challenge that required significant attention – one that he still focuses on.  In Chang’s words, “SMEs cannot make their own guanxi over night.”  One way he has successfully gotten his foot into the door is by meeting with consultants who act as his ‘air force’ – while he trains a figurative army of software designers, he relies on consultant connections to help put him into contact with suppliers, vendors and other contacts.  After years of meetings, this has enabled his firm to grow 30% annually.  Thus entrepreneurs should be cognizant of this all-encompassing cultural trait that Matt Garner described in Chapter 1 as “relationship focused” – in contrast to the “results focused” in the West.

Telecom infrastructure

ClarkMorgan runs a very tight ship in Shanghai’s Changning district, next to Jing’an Temple.36 Founded more than a decade ago by Australian Andy Clark and Briton Morry Morgan, it is primarily known as a firm specializing in corporate training, yet I would argue it publishes one of the top quarterly magazines on HR-related issues in China.  When I visited their office in May 2012, I had a chance to see firsthand the typical workday in which expat and local employees worked side by side, sometimes even sharing the same scarce tables.  The Shanghai office is staffed by approximately 20 full time employees (half expats) and as Gary Isse explained to me in an interview, “one of the challenges we continually face is maintaining a reliable network connections both internally and externally.”

This is one of the struggles that all potential firms wanting to move to China will face: how to deal with a relatively static telecom industry within each city.  While China is home to state-of-the-art telecom gear manufacturers such as Huawei and ZTE who produce modern equipment, its domestic broadband build-out is lagging neighboring peers such as Japan and Korea in part because its internal telecom infrastructure is organized into two disparate tiers.

In October 2012 I spoke with Scott Freeman, CEO of ITBN, a private internet service provider (ISP) that provides broadband connectivity solutions in Beijing and Shanghai.37 ITBN was founded in 2000 and offers a range of connections from dedicated ISDN lines to full fiber connections.  While ITBN charges a premium for their services, they also provide something that these SOEs cannot: reliability and bi-lingual telephone support and thus have captured a significant percent of the urban market share.

While much speculation exists about the telecom infrastructure in China, Freeman described the seemingly complicated national network thusly, “there are hundreds of licensed and probably thousands of unlicensed ISPs in China. Some have national licenses (like we do); others have only provincial licenses. Many more operate without licenses. The official differentiation between ISPs like us and the big state-run ones is that we are called “second-tier” ISPs, whereas they are called “first-tier” ones.  Theoretically the first-tier telcos are supposed to control all of the physical connections in and out of the country.  Other than that it’s not so clear what else differentiates them on the ground, other than the fact that the big state-run telcos have a lot more money and the extra job of content monitoring.”

As Freeman noted, aside from a few licensed private firms such as ITBN, there are essentially only three tier 1 ISPs in the whole country (China Telecom, Unicom and Mobile) all of whom are state-owned enterprises.38 In fact, on a user basis China Telecom and China Unicom (both SOEs) are the largest ISPs in the world.39 And while there may be developmental reasons for relatively slower bandwidth speeds (compared with their neighbors), in terms of throughput, according to their Q3 2012 speed survey, ChinaCache noted that while the overall speeds are a little slower than previous speed rankings, Shanghai currently leads the country in average speeds at roughly 3.44 Mb/s and Beijing is 10th at around 2.5 Mb/s.40

While the quality of wireless telephony signals between the US and China is debatable, the Ministry of Industry and Information Technology announced in September 2012 that it plans to being issuing 4G licenses within the following year.41 Thus while Western countries are finishing rolling out 4G networks, aside from a pilot roll-out in a dozen cities such as Chengdu, Hangzhou and Wenzhou (from China Mobile), the majority of Chinese users unfortunately have another couple of years before 4G becomes an installed reality.42 And in the case of ClarkMorgan, there just are not many broadband packages that fit their needs at the prices expat managers are accustomed to (e.g., choice between a relatively inexpensive 5 mb/s DSL versus an expensive dedicated T1).

Takeaway: before opening up a Chinese branch for your company be sure to research the costs of living, property rental prices and telecom infrastructure availability in the area.  In addition, paying attention to hiring cycles, offering internships and recruiting hǎiguī may also give your company a significant advantage over your competition that fails to do so.


  1. See China’s Graduates Face Glut from The Wall Street Journal and University Graduates Have Hard Time Finding Job, Initial Survey Finds from Caixin []
  2. In other words, many college graduates are typically uninterested in low-wage, low-skilled factory work.  See Chinese Graduates Say No Thanks to Factory Jobs from The New York Times []
  3. Asia’s Endangered Species: The Expat from The Wall Street Journal []
  4. Animal Spirits with Chinese Characteristics: An Interview with Mark DeWeaver from The Libertarian Standard []
  5. Legal and Unauthorized Chinese Immigrant Population from Migration Policy Institute []
  6. According to a 2008 report from Reuters, “Of the 1.2 million Chinese people who have gone abroad to study in the past 30 years, only one fourth of them have returned, according to the Chinese government.”  In addition to Chapter 19, see China’s Brain Drain at the High End by Cong Cao, China’s Brain Drain Dilemma: Elite Emigration from The Jamestown Foundation and China fears brain drain as its overseas students stay put from The Guardian and China goes on the road to lure “sea turtles” home from Reuters []
  7. Unrealistic U.S. Immigration Policies Push Away China’s Best And Brightest by Forbes []
  8. An Export of Students: Where Are China’s Ultra-Rich Sending Their Children to Study? from Good Infographics []
  9. In 2011, the US embassy in China issued more than 160,000 student visas for Chinese students to study at American schools.  Yet a November 2012 report from Open Doors notes that the actual number is even higher, 194,029.  See Ten Years of Rapid Development of China-US Relations from Xinhua and Students from China add $5b to US economy from China Daily []
  10. Spreading their wings early from China Daily []
  11. U.S. a Hot Spot for Chinese Grad Students from The Wall Street Journal []
  12. Tough US job market sends Chinese students home from China Daily []
  13. Reverse brain drain: China engineers incentives for “brain gain” from The Christian Science Monitor []
  14. Is overseas returnee working as driver a waste of talents? from People’s Daily []
  15. It should also be noted that due in part to an economic slow-down on the mainland and because of political tensions, numerous Japanese firms are purportedly considering relocating elsewhere.  According to Reuters a “quarter of Japanese manufacturers are rethinking their investment plans in China and some may shift future production elsewhere.”  For perspective, since 1990, Japanese firms have invested almost $1 trillion the mainland.  And despite these tensions, in 2012, “Chinese consumers bought nearly 3 million Japanese cars and trucks.”  See As China tensions simmer, Japan pulls back from “world’s factory” from Reuters and Five Predictions for China’s Auto Industry in the Year of the Snake from The Wall Street Journal []
  16. Expat preference for the growing Chinese economy is apparent from HSBC []
  17. Other estimates such as the National Bureau of Statistics put the per capita averages higher:  (~$13,000/capita), Beijing (~$12,500) and Guangzhou (~$13,000).  The reason for the disparity involves not just sample size but also what geographic districts are included or excluded (e.g., in the NBS case they divided total GDP by population in the region).  See Guangzhou has highest average salaries for cities in mainland China from South China Morning Post []
  18. Charting China’s Family Value from The Wall Street Journal []
  19. National Average Wage Index from Social Security Administration []
  20. Modern China: A tale of luxury villas and displaced villagers from McClatchy []
  21. Worldwide Cost of Living Survey 2012 from Mercer []
  22. Beijing, Shanghai Cost-of-Living Leaps from The Wall Street Journal []
  23. Ibid []
  24. See The Current Demographic Profiles of Shanghai (2011) from Shanghai Municipal Population and Family Planning Commission, Beijing’s temporary population fell in 2011 from China Daily and Guangzhou seeks opinions on population draft from China Daily []
  25. Shenzhen Has the Highest Weibo Penetration Rate in China from China Internet Watch []
  26. See 593,832 foreigners live on Chinese mainland: census data from Xinhua and Plan to reduce minimum stay for foreign workers from Shanghai Daily []
  27. Shanghai’s foreign population above 200,000 from Want China Times []
  28. Wages have also decreased for certain professions over time.  For example, in 2000 a computer science graduate could earn $725 a month in Shenzhen, a wage that has decreased to $550 a month due to more competition from graduates.  See China’s ‘Manhattan’ becomes censorship capital from Financial Times and Chinese Graduates Say No Thanks to Factory Jobs from The New York Times []
  29. See Shanghai tops China’s “best city for business” from Sina and Top 10 best cities for business in China 2012 from China.org.cn []
  30. Another estimate puts the number of Asia-Pacific headquarters in Shanghai at 393.  See Almost 400 MNCs have their Asia HQs in Shanghai from IANS and Shanghai tops China in attracting multinational headquarters from Xinhua []
  31. Shanghai policies woo multinational headquarters from Xinhua []
  32. This is different than shūshin koyō (employment for life).  For a dated yet clear explanation of the Japanese bonus system see Bonuses and Employment in Japan from Journal of the Japanese and International Economies, 1987 []
  33. Developing a Competitive Edge from Insight []
  34. Motion Global []
  35. China’s Graduates Face Glut from The Wall Street Journal []
  36. Clark Morgan []
  37. ITBN and ITR []
  38. While there are a few large, private, independent ISPs on the mainland such as 263.net (网络通信) and Great Wall Broadband (recently acquired in Q4 2012 by Chengdu-based Dr. Peng Telecom, 成都鹏博士电信传媒集团股份有限公司) nearly all traffic is still routed through the three SOE tier 1 backbone monopolies.  China Tietong Telecommunications (中国铁通集团有限公司) which used to be China Railcom, merged with China Mobile in May 2008.  China Netcom (CNC) merged with China Unicom in October 2008.  See Users angry at slow Internet speeds from Global Times []
  39. Just two Chinese ISPs serve 20% of the world broadband users from ArsTechnica []
  40. ChinaCache Releases Third Quarter 2012 China Internet Connection Speed Rankings from China Web Report []
  41. China 4G licenses to be issues in 2013 from ZDNet []
  42. See China Mobile Network Costs Mean First Net Drop Since ’99 from Bloomberg, China Mobile Builds First 4G Base Station In Chengdu from China Tech News and China Unicom Books 50% Net Profit Growth In 2012 from China Tech News []
Send to Kindle

Chapter 20 – VPN and infrastructure services

[Note: below is Chapter 20 from Great Wall of Numbers]

Any company wanting to conduct international business on the mainland, irrespective of whether it is local or foreign, must invariably factor in the additional costs of communicating electronically beyond the mainland.  Beginning in 1996 and launching in 1999, the Ministry of Public Security and other governmental organizations began implementing and enforcing a series of regulations involving data filtering that ultimately matured into what is commonly referred to as the “Great Firewall” (GFW).12 Collectively, the types of filtering techniques employed by the Ministry through Tier 1 ISPs (all SOEs as noted in Chapter 15) include: IP blocking, DNS filtering, URL filtering, packet filtering and connection resets.  Simultaneously, the Ministry maintains a continuously updated “black list” of websites that mainland users are unable to access through this dynamic filtering and blocking mechanism.

This presents an opportunity to virtual private network (VPN) providers overseas.  A VPN is a type of technology that effectively creates a secure tunnel from one computer to another, isolating its data stream from the surrounding traffic.  This can be done by means of encryption and as a consequence many banks, financial institutions and national security centers – irrespective of the region or hemisphere – typically use some form of VPN to securely communicate with outside parties (e.g., for wiring money, discussing trade secrets, or diplomacy).3

There is no shortage of VPN providers in Western countries and there are in fact, Chinese-based VPN providers as well – the efficacy and reliability of which is debatable.4 In my own anecdotal experience, even with a paid service based in a foreign country, data can still be throttled and your connection reset.5 One reason is that the GFW is not a passive system – it is continually tweaked and changed.  In an interview in 2011, Fang Binxing, the Father of the Great Firewall explained that he himself has “six VPNs on my home computer.”6 He uses them to “test which side wins: the GFW or the VPN.”  And in his opinion, “[s]o far, the GFW is lagging behind and still needs improvement.”7

One world, two internets

As I mentioned in Chapter 12, while there are any number of domestically made and managed counterparts and clones of foreign social media services (e.g., Sina Weibo is the equivalent to Twitter), there is still niche demand for foreign-based web services.  For example, as I mentioned in Chapter 9, there are now about a million Chinese students studying overseas each year; more than 190,000 Chinese students studied in the US this past year alone.8 In addition, 1.36 million Chinese tourists visited the US in 2011.9 What this means is that as I mentioned in Chapter 3, Chinese consumers are increasingly exposed to Western and in particular, American tastes and services.

Yet to temper the optimism that a VPN provider could immediately sell several million service packages to individual mainlanders, consider this rough facsimile: while we may never know the real number, Facebook’s own indirect estimate of mainland usage of its social network is close to 600,000.10 While there are a number of other niche services in demand, especially from financial service firms, this 600,000 number can be used as a proxy to estimate the general demand for VPNs.

It should also be noted that not all foreigners use Facebook in China nor do all foreigners want to pay for a VPN to access it.  Furthermore, based on my own anecdotal experience at various institutions, the average Chinese user does not currently have access to a VPN or other fan qiang (“Wall Climbing”) software such as UltraSurf or Hotspot.  In their mind, why should they have to pay to access foreign services when there is a similar Chinese version available for free?  This is not to say that they could not gain access to the services if they were motivated and inclined to do so.  In my own anecdotal experience virtually none of the several thousand students I have had at various cities on the mainland over the past four years have had active Facebook accounts.  A few however have had VPN accounts so they could play online games like World of Warcraft on servers outside the mainland (e.g., “gold farmers,” see Chapter 14).11

Assuming the number is around 600,000, how much can a foreign-based service provider expect to generate?  Currently, the average monthly rates at PandaPow, Astril and most others are roughly $10 a month.12 And because a large portion of a VPN package is based on software that is open-source and free, the initial setup costs are minimal.13 Yet bandwidth charges, hardware purchases, customer support labor and utilities charges all vary and will depend on how large you plan to scale your company to.  Thus before investing in this segment, do your due diligence.

Entrepreneurs should also consider this: Bill Bishop has cogently noted numerous times over the past several years that while mainland users are effectively prevented from using some foreign web services, the mainland equivalents are not only more easily accessible and relatively comparable (e.g. same features), but the interconnectivity issues (e.g., latency, bandwidth) with them are relatively muted.14  In other words, assuming you have access to a VPN, it is still more convenient for mainland based users to stream videos on Youku than it would be to stream from Youtube because of the increased bandwidth throughput and reduced lag due to closer proximities to the content delivery networks (CDNs) for Youku-like providers such as ChinaCache.15 David Wolf, a partner at Allison+Partners (a consulting firm) echoed similar reasoning recently in an interview with The Wall Street Journal, noting that “What they [national government] prefer is that Chinese users decide it is just too much trouble and by default use onshore sites, or sites that are mirrored onshore.”16 As a consequence, because of the sheer size of the Chinese-based internet (see Chapter 12 and Chapter 13), there is now “one world, two internets.”17

Climbing the wall

I spoke with security expert David Veksler (see also Chapter 13), CEO of CryptAByte, who has given security workshops and seminars about these issues.18 In his view, “the GFW presents a fundamental problem for domestic researchers.  Because significantly large portions of foreign-based information are blocked and denied, only researchers with VPNs are able to keep up-to-date with their foreign counterparts.  Those without VPNs are left trying to use Google which is frequently blocked and misdirected or Baidu, which outputs few useful or useable results.  Thus they become discouraged, often times quitting and are ultimately unable to do the necessary research – idea investigations – for innovation.”

How does this create opportunities?  According to Veksler, this ties into another Catch-22 that domestic firms find themselves in, this endless cycle of benchmarking and cloning.  That irrespective as to whether or not they want to innovate, they are in a prisoner’s dilemma, “every competitor on the mainland expects to have the lowest costs.  Yet if they increase their research and development – creating higher quality products – consumers do not believe them, because consumers also expect that domestic companies are cutting corners, so why pay extra for a product that is probably just the same as the rest?”

He also likens this dilemma to a game theory scenario: the first domestic company to make that leap into quality is punished because consumers simply do not trust the product quality due to a history of scandals.  Thus any firm that does it is unable to recoup the capital costs of the research and development.  In contrast, foreign companies have spent decades building up their brands and reputations based on quality control programs (e.g., Six Sigma) and as a consequence are readily more trusted on the mainland.  Yet he remains optimistic, “the first domestic company to make a concerted, long-term leap into quality will not only be monetarily successful, but will help end this never ending cycle of benchmarking and cloning.”

Thus Veksler thinks that foreign brand managers, experts like Matt Garner, will be able to find opportunities to work within the entire spectrum of industries as their participants build national and internationalization expansion plans.

Chicken and egg problem

It is hard to measure the impact that an apparatus like the GFW has on productivity and creativity which business start-ups should take into consideration.  Consider Silicon Valley and Moore’s Law.  Much like other projects and mian zi gong cheng, there have been several public initiatives to replicate Silicon Valley in China, such as Zhongguancun in Beijing.  And yet for every successful start-up like iQiyi or venture capitalist like Kai-Fu Lee (see Chapter 12), large quantities of resources have been misallocated towards supercomputers that when installed – while capturing headlines for theoretical peak performances – are unable to be fully utilized because there are not enough trained software engineers to develop the sophisticated machines.19 Similarly despite 2 billion RMB ($320 million) in investment since 2010, Jike, a new search engine developed by People’s Daily (an SOE) has managed to capture a mere 0.0001% marketshare forcing the organization to lay off 20% of its staff.20

Empirically speaking, if central planners were to be the creators of Silicon Valley, they would have created Silicon Valley.  If central planners were to be the creators of Moore’s Law, they would have created Moore’s Law.2122 For example, the Soviets spent decades and relatively large budgets to overtake the West in computing innovations, yet failed at every turn.  In fact, it was not just one or two half-hearted attempts, it was a concerted effort directed from the top.  Mikhail Gorbachev himself made advancements in microprocessor technology a cornerstone part of Perestroikain 1985 (encompassing the 14th Five Year Plan).

Just how much effort was put into their centrally planned machine industry? Consider what the USSR tech industry was like circa 1988:

Machine building is the sector of industry on which Gorbachev is relying to ensure the success of his [Perestroika] strategy.  The hub of Soviet [computing] industry, this complex employs over 16 million workers at more than 9,000 research institutes, design bureaus, and production and enterprises, and is responsible for designing, developing, and producing over one-fourth of the country’s industrial output.  Of the 17 industrial ministries that make up the machine-building complex (detailed in foldout at back of paper), nine — collectively referred to as the defense industry — specialize in military hardware. The other eight produce primarily consumer goods and equipment for investment in the civil sector.23

Gorbachev recognized that “a high-investment, high-growth strategy must, at a minimum, continue through at least the first few years of the period to renew the sector’s capital stock.”  Yet ultimately, the Soviets tried, consumed their capital base, and failed.2425 Instead, hundreds of private companies, entrepreneurs, venture capitalists, designers, and one relatively free market created a semiconductor industry that accounts for the number one export of the United States.26 Furthermore, this is not to say that technological activity will not take place in China, nor that Chinese institutions and researchers will not produce usable technology.  The question is rather, can it be cutting edge and innovative?  And if your firm hopes to tap into the innovation potential of the mainland, how does this impact your firms’ investment?

Many of these artificial technology and science research parks conflate cause and effect.  For example, during World War II, the Allies used Pacific islands as forward operating bases to protect their overseas supply routes.  On many of the islands the Allied forces built airstrips, including one on Vanuatu.  Following post-war demilitarization, most of these islands were vacated as the warring militaries returned home.  On Vanuatu, many of the islanders wanted the supply ships to return and provide modern goods to their pre-industrial society.  As a consequence, the islanders staged “drills” and “marches” with mock soldiers while others attempted to man the airstrips – all under the belief that it is these superficial motions and actions that originally brought the Western supplies.  Richard Feynman dubbed this “cargo cultism” (e.g., a cult that dreamt of Western cargo).27

In November 2012 I spoke with Mark Thornton, an economist at the Ludwig von Mises Institute and an expert in the boom-bust investment cycle.28  According to him, “Research parks are all about inventing technology for commercial and other purposes. Generally we are speaking of higher order goods, the types of goods associated with the boom phase of the business cycle. Therefore we would expect that research park projects tend to be established during booms when profits are high, the cost of capital is low, and where retained earnings are more than sufficient to support additional projects. If research parks are established at or near the peak in the business cycle then it would be wise to avoid contracting with research parks that have few tenants.  Traditionally one of the main benefits of research parks is synergy.  If your research park has no tenants then you do not have the type of synergies that successful research parks generate.  New companies, new technologies and products, as well as successful research parks (e.g. Stanford Research Park and Research Triangle Park) tend to get their starts during bad economic times.  During recessions land, labor, capital are cheaper and budding entrepreneurs are more abundant.”  In economic terms, higher order goods are goods used to produce consumer goods (e.g., those which require a long-term investment such as building a factory which in turn creates consumer goods).29

Similarly, many of these research parks and endeavors – not just in China – arguably exhibit patterns of modern-day cargo cultism.  Thornton noted that, “The next Silicon Valley will not look like Silicon Valley.  It will have some new features and not have all the same features as Silicon Valley.  You cannot just build “it” and expect them to come.  Silicon Valley is more than just Stanford Research Park and Stanford University. There are tangible and intangible factors that matter. They include things like the weather, demographics, culture, and relatively limited regulatory impact from the government. Even some factors we just do not know. Government can subsidize research parks but it takes a free market and entrepreneurs to actually weave the fibers of something extremely complex like Silicon Valley.”

In fact, in the US, nearly every state has erected several tech parks with the hopes of “creating” another Silicon Valley; there are dozens of research and technology centers across the country.  This raises the question: if you build it, will they (the creative classes) come?

In February 2013 I spoke with Becky Wu a native of Jiangsu province and a project manager at Xi-Tong Scientific & Technology Industrial Park located in Nantong, Jiangsu province.30 The primary task of her job is attracting and relocating foreign firms so that they will build and setup operations in the industrial park.  According to her, “we provide incentives and subsidies to attract firms from abroad.  For example, if land prices were with 230,000 RMB per mu, depending on how promising the project is and what industry your firm is in we can lower the price to 200,000 RMB or even 150,000 per mu.  This helps attract firms, enticing them to construct their new offices in the park.  We will also provide free temporary offices for new companies for up to 6 months while their new office is being built.  The utilities are also free of charge as well.”  As noted earlier in Chapter 3, a mu is 1/6th of an acre.

Wu also explained that there are other rebates and training subsidies that firms can receive.  She noted that, “we also offer new companies subsidies for research and to train personnel that can be allocated and spent without strings attached.  For example, we can provide up to $1,000 a year per person, up to 10 people to help offset training and research costs.  In terms of income taxes, we provide rebates to specific workers, typically managers and high-level executives for 3-5 years.  The way this works is that if you have to pay 100 RMB in taxes, 60% goes to the central government, 8% goes to the provincial government, the remaining portion goes to Nantong, thus we at the park can reimburse the remaining 32% back to you.”  Clients such as Caterpillar, BIC, Accuma and Kopron have taken advantage of these incentives over the past several years.

Does the return-on-investment pay for the capital expenditures which were originally expended?  While it is impossible to say yes or no for all the cases, what can be said is that the GFW itself probably does not create innovation, foster creativity or act as an incentive to attracting outside talent.  If it did, the Chinese computing industry would not be reliant on Western semiconductors, Western software and foreign know-how.31 And as a consequence, mainlanders conducting research are left using a virtual straw in order to access, view and communicate with the outside world.

How is this relevant and how does this affect your company?  Without virtual openness to new ideas, the domestic, indigenous engineering industries – while not autarchic – will probably always be laggards due to what Veksler noted above (e.g., getting frustrated and quitting).  To this point, last year the American Chamber of Commerce in Beijing conducted a survey of its members, “nearly three-quarters of about 300 businesses it surveyed said unstable Internet access impedes their efficiency. About 40% said China’s censorship efforts have a negative business impact.”32 Similarly, economist Arthur Kroeber, founder of Dragonomics research noted in March 2013 that one obstacle to growth is the GFW.  In his view, innovation in the modern world today comes from “the sharing of knowledge and information across a variety of fields.  Innovation comes when you take knowledge in one area and it migrates over to another area and someone comes up with a new way of using it.  China seems to have a political system that mentally at its core is opposed to those networks ever becoming viable.”33 Thus, in addition to the issues raised in Chapter 15, this obstacle is another consideration that all firms looking to recruit talent must take account for.34

While there are occasional opportunities and projects like “1,000 talents” (mentioned in Chapter 9 and Chapter 15) that provide monetary and other perks and incentives to relocate, these well-intentioned plans may be unable to offset the hurdles created by the GFW and as a consequence there has been a “brain drain” that all firms and HR departments should be aware of.35

Yet to be even handed, Larry Chang mentions that he works within this system on purpose because it is “an untapped opportunity.”  He only hires fresh mainland graduates with the sole purpose of building an indigenous software industry.  And in his opinion, with more than 6 million college students graduating each year, there are bound to be creative, outside-the-box thinkers.  Similarly, at the 2013 Unleashing Innovation conference recently held in Singapore, Ya-Qin Zhang, chairman of Microsoft’s Asia Pacific research and development group, noted that “Chinese engineers are well equipped to produce the kind of innovative work that their more illustrious American rivals are renowned for” and continued with, “[t]he scale of innovators and the scale of the market will converge and eventually make China a key [innovation] center in the region.”36 Thus it may just be a matter of time before the right combination of inputs brings about the transition up the value chain as described in Chapter 7.

Opportunities in the rough

Again, even with these seemingly insurmountable challenges there are also opportunities.  For example, as I noted in Chapter 17, foreign architects are in high demand to help build and design buildings, bridges and even office parks.  Perhaps your firm can find new revenue streams by helping to build out domestic content delivery networks (CDNs) and cloud computing initiatives that are part of these technology parks.  As I mentioned in Chapter 13, according to IDC, $286 million was spent on cloud-computing infrastructure in China in 2011 and this is expected to increase to $1 billion by 2016.37 And this segment is quickly professionalizing, for example, ChinaCache is the largest CDN on the mainland with 53% of the marketshare.38 It was initially funded by the likes of Intel and is now listed on NASDAQ.

Another opportunity is with corporate VPNs.  While the individual market may seem like a logical way to establish a steady revenue stream, according to David Veksler, corporate enterprises – both domestic and foreign – will eventually want and need to have VPNs to secure their communication with clients, vendors and essentially anyone.  Irrespective of the GFW, Veksler’s own estimate is that there is an unlimited amount of potential growth for VPNs because very few domestic firms currently recognize the need to protect their assets.  But Veskler suggests, “this attitude will probably change, due to the increasing security vulnerabilities publicly acknowledged by even the largest of enterprises.”

But there is also a challenge regarding foreign owned and run VPNs on the mainland, as the Global Times recently quoted Fang Binxing (father of the GFW as noted above) that, “[u]nregistered VPN service providers are not protected by Chinese laws, and any company running a VPN business should realize they have a responsibility to register.”39 More directly, an employee in the Ministry of Industry and Information Technology pointed out in the same report that, “only Chinese companies and Sino-foreign joint ventures can apply to establish a VPN business.”  This is not to say that is illegal to connect to a VPN outside of the mainland.  Currently there are no laws which prohibit users in China from connecting to an overseas VPN.40

In December 2012 I spoke with an American executive at a large IT company that provides dedicated internet connections to enterprises and institutions primarily in Tier 1 cities.  According to him, “no foreign IT company and few domestic companies advertise their VPN services yet many of them will bundle it as part of a package to corporate clients.  Furthermore, Chinese regulators typically permit VPNs so as long as it is privately – not publicly – accessible as well as the stipulation that consumers use leased-lines.  A typical dedicated leased-line will cost over 3,000 RMB a month for 1 mb/s, this scales linearly (e.g., if you need 4 mb/s you are charged around 12,000 RMB), thus this option is typically out of reach by most consumers outside of the corporate and foreign communities.  In addition, you can find a number of local firms that will provide point-to-point VPN services within the mainland.  So if you are an expat that works for a foreign company that operates a VPN network elsewhere, then you will be able to securely connect from your local VPN to their secure environment overseas.”

Similarly, as an entrepreneur you can utilize these tech parks in China since they are not going to disappear overnight, if ever.  For example, Larry Chang merged all company divisions under one roof in a research park located on a campus of a local college in Changning, Shanghai.  His firm was provided incentives such as reduced rental rates for doing so.  Similarly, Richard Qi mentioned that a new area in Shanghai called Cloud City – a tech park – provides perks and benefits to foreign software, engineering and IT firms.  For example, Cloud City provides discounted office property, assistance in communicating with governmental organizations, stipends form the government and as the name-sake suggests, access to cloud services.  Prior to relocating to this tech park, Qi mentioned that it was often difficult as a foreign service provider to issue invoices because of unclear laws (e.g., Shanghai and other municipalities are currently transitioning from a business tax to a VAT) and it was hard to find the government contacts needed to settle these transactions.  In addition, perhaps your software or semiconductor firm can also take advantage of these inducements created by the 2011 policy which provides a tax holiday for several years, reduces the subsequent tax rates and provides exemptions on profits.41

Takeaway:  Due to a variety of regulations and policies on the mainland, certain telecommunication restrictions have germinated into a formidable barrier called the GFW.  And with several million technologically-inclined consumers familiar with Western tastes and styles, there exists a potentially new customer base for VPN service providers.  Yet just because there is potential for growth does not necessarily mean that the potential customers will purchase your goods and services (e.g., “if you build it, will they come?”).


  1. According to Fang Binxing, the ‘Father of the Great Firewall,’ it was “reportedly launched in 1998 [and] came online about 2003.”  See Great Firewall father speaks out from Global Times []
  2. Splinternet Behind the Great Firewall of China from Association for Computing Machinery []
  3. To bypass copyright restrictions, VPN uptake has increased over the past several years in several Western countries, as consumers move to alternative methods for downloading copyrighted content.  According to a study from Lund University in Sweden, there has “been a 40% rise in the number of 15 to 25-year-olds using such [VPN] services since 2009.”  See File-sharers look to VPNs to overcome Pirate Bay ban from BBC []
  4. Even with encryption algorithms like AES, third parties which have direct access to even one end of a data stream can conduct packet sniffing and other “side channel” attacks. []
  5. See Five Myths about the Chinese Internet from Foreign Policy and Florida pet spa mystery link to China’s great firewall from New Scientist []
  6. Great Firewall father speaks out from Global Times []
  7. In January 2013, Han Weili, a software instructor at Fudan University in Shanghai publicly solicited applications for employment to improve the GFW.  In his view there are two problems with the GFW technology, “The first is a lack of transparency in strategy, the second is that Great Firewall strategy execution has a false-report rate that is too high.”  See Great Firewall Engineer Han Weili Calls for Job Applications from Fei Chang Dao []
  8. In 2011, the US embassy in China issued more than 160,000 student visas for Chinese students to study at American schools.  Yet a November 2012 report from Open Doors notes that the actual number is even higher, 194,029.  See Ten Years of Rapid Development of China-US Relations from Xinhua and Students from China add $5b to US economy from China Daily []
  9. Chinese tourists spend more in US in 2011 from China Daily []
  10. No, Facebook does not have 63.5 million active users in China from The Next Web []
  11. Approximately half of World of Warcraft’s 10-12 million userbase is estimated to be from mainland China.  See “Gold Farming”: Real-World Production in Developing Countries for the Virtual Economies of Online Games by Richard Heeks and Converting the Virtual Economy into Development Potential: Knowledge Map of the Virtual Economy from the World Bank []
  12. Disclosure: I do not currently have any stakes in these products, services or companies.  See Testing five VPNs that’ll get you back on YouTube, Facebook in China from c|net []
  13. A user can remotely set up their own VPN practically anywhere using software such as OpenVPN.  The primary key issue is locating a computer outside of the mainland where it can be installed on and reliably connected to. []
  14. Sinocism []
  15. To better understand the importance of CDNs see, Google and Netflix Make Land Grab On Edge Of Internet from Wired []
  16. China’s ‘Wall’ Hits Business from The Wall Street Journal []
  17. See One World, Two Internets by Bill Bishop and Iran’s network in a bottle from The Boston Globe []
  18. CryptAByte []
  19. According to one estimate regarding software application investment for supercomputers in China, “Less than 10% of supercomputing funding goes to developing such applications, said Chinese researchers who complain that political leaders press them to build headline-grabbing new machines rather than focus on whether they are used to their full capabilities.”  See China’s Not-So-Super Computers from The Wall Street Journal []
  20. See People’s Search Engine Denies Layoff Rumors; Says More Jobs Open from Caijing, Jike’s attempt to censor news about its 0.0001% market share has backfired from Shanghaiist and You’ve been Jiked! from China Media Project []
  21. Debt as Tall as Dubai, or How the Singularity Is Not a Guaranteed Phenomenon by Tim Swanson []
  22. They cannot a priori due to the economic calculation problem.  See Economic Calculation In The Socialist Commonwealth by Ludwig von Mises []
  23. The Soviet Machine-Building Complex: Perestroyka’s Sputtering Engine from the Office of Soviet Analysis published by the Directorate of Intelligence []
  24. Throughout its existence the Soviet Union tried to incorporate technology in its Pyatiletka — Five Year Plans.  They even tried to recreate Silicon Valley through the construction of numerous science and research parks called Naukograd.  Numerous other countries have also tried to emulate the success of the Bay Area with little measurable return-on-investment; this includes Silicon Taiga in Novosibirsk.  The Soviet Union was unable to incubate something akin to Moore’s Law for the same reason the Soviet Union ultimately failed: without prices, you cannot make efficient allocation decisions.  Prices only arise from market interactions, through profit and loss — which signal to entrepreneurs when to buy, sell, trade, and invest capital.  Without this organic knowledge Soviet planners were left using arbitrary coefficients to plug into their various economic models with the net result: planned chaos.  See Planned Chaos by Ludwig von Mises. []
  25. One frequently cited myth regarding Japan is that it was successful in its attempts to centrally plan scientific innovation.  This is untrue.  See The Fifth Generation Fallacy by J. Marshall Unger.  See also Chapter 9 in Animal Spirits with Chinese Characteristics by Mark DeWeaver []
  26. According to the Semiconductor Industry Association, “three quarters” of all semiconductor design and manufacturing takes place in the United States and that 82% of semiconductor sales are outside the United States.  See America’s #1 Export Industry Applauds Passage of Free Trade Agreements from the Semiconductor Industry Association []
  27. See Cargo Cult Science by Richard Feynman and In John They Trust from Smithsonian []
  28. Skyscrapers and Business Cycles by Mark Thornton []
  29. See Chapter 1 in Principles of Economics by Carl Menger and Chapter 16 in Human Action by Ludwig von Mises []
  30. Xi-Tong Scientific & Technology Industrial Park []
  31. According to recent reports, Chinese policy makers are attempting to build a 100-petaflop supercomputer which would be five times faster than the current record holder (Titan).  As part of this plan, Zhang Yunquan, a professor at the Institute of Software Chinese Academy of Sciences, noted that domestically designed chips may be used.  These domestic chips, called Loongson are based on MIPS, a chip design developed by a Sunnyvalle-based technology firm (MIPS Technologies).  Similarly, Chinese policy makers are frustrated by the fact that Android (which is managed by Google) has the lion’s share of marketshare and would prefer to have a domestic, homegrown OS used by smartphone makers instead.  See China is building a 100-petaflop supercomputer from IT World, China’s godson gamble from IEEE SpectrumWhy China Can’t Make Its Own Mobile OS from Tech In Asia and Google controls too much of China’s smartphone sector: ministry from Reuters []
  32. China’s ‘Wall’ Hits Business from The Wall Street Journal []
  33. Economist: China Plenty Creative, Just Not in Right Ways from The Wall Street Journal []
  34. China’s self-defeating war with information by Andy Yee []
  35. See Rich Chinese want to buy happiness — by emigrating from Los Angeles Times and Wary of Future, Professionals Leave China in Record Numbers from The New York Times []
  36. Microsoft’s Zhang Sees China as Asia’s Innovation Center from The Wall Street Journal []
  37. Cloud computing investment ‘to hit $1b’ from China Daily []
  38. ChinaCache investor relations []
  39. Foreign-run VPNs illegal in China: govt from Global Times []
  40. Adding Some Key Facts In WSJ.com’s China’s Internet ‘Wall’ Hits Business Article from VPN Instructions []
  41. China offers new incentives to further boost software and semiconductor industries by Peng Tao []
Send to Kindle