Six bedtime stories from 2017

[Note: I neither own nor have any trading position on any cryptocurrency.  I was not compensated by any party to write this.  The views expressed below are solely my own and do not necessarily represent the views of my employer or any organization I advise.  See Post Oak Labs for more information.]

2017 taught us many things, including the fact that no one reads (or writes) or pays for long-form content any more.  Even with lovable memes and animated gifs, keeping an audience’s attention is hard.

Already too distracted to read further?  How about a quick video from JP Sears on how to appropriately Bitcoin Shame your friends and family:

The other takeaway for 2017 is that, if in doubt, open up hundreds of social media accounts and shill your way to riches.  The worst thing that could happen is no one buys your coin.  The best thing that happens is that someone buys your coin and you can then convert the coin into real money, retire, and act like you are super-wise thought leader with oodles of entrepreneurial and investing experience.

Some other stories with revisiting from the past year:

(1) “Legitimization”

If we were being intellectually honest we would say that the only goal post anyone cared about this year was that the price of cryptocurrencies, as measured in real money, and how high they soared.1 And that the main reason this occurred is because Bob knew Alice and Carol were both going to buy a lot of say, bitcoin, thereby pushing up the price, so he did too.  The Economist called it “the greater fool theory.”  But The Economist are great fools for not buying in at $1, so let’s ignore them.

Basically none of the feel-good goals about lowering remittance fees or increasing financial inclusion promoted in previous years by enthusiasts have really materialized.  In fact, at-risk users and buyers in developing economies probably got screwed on the ICO bandwagon as insiders and sophisticated investors who were given privileged early access to pre-sales, dumped the coins on secondary markets and hoi polloi ended up holding the bag on dozens of quarter-baked ICOs.2

Oh, but transaction fees for Bitcoin are at all-time highs, that’s a real milestone right?

There are many reasons for this, including the fact that Bitcoin Core’s scaling roadmap has thus far failed to achieve its advertised deadlines (see section 5 below).3 Maybe that will change at some point.

Shouldn’t higher fees be a cause for celebration with “champaign” (sic)? 4

Some Bitcoin Core representatives and surrogates have created an ever expanding bingo card of scapegoats and bogeymen for why fees have gone up, ranging from:

  • blaming Roger Ver and Jihan Wu as demonic-fueled enemies of Bitcoin
  • to labeling large chunks of transactions as ‘spam attacks’ from nefarious Lizard-led governments5
  • to flat out bitcoinsplaining: higher fees is what to expect when mass adoption takes place!

I’m sure you’ll be on their bingo card at some point too.

Just like Visa and other widely used payment network operators charge higher and higher rates as more and more users join on… oh they don’t.6 But that’s because they censor your freedom loving transactions!  Right?

So what’s the interim solution during this era of higher fees?  Need to send a bitcoin payment to someone?

You know how supermarkets used to hold items on layaway?  They still do, but it’s not as common to use, hence why you googled the term.  Well, in light of high fees, some Bitcoin Core developers are publicly advising people to open up a “tab” with the merchant.  You know, just like you do with your favorite local bartender.

Fun fact: the original title of the Satoshi whitepaper was, Bitcoin: a peer-to-peer electronic layaway system.

This faux comparison didn’t age well.  In 2014 this was supposed to be a parody. (Source)

For example, the ad above was promoted far and wide by Bitcoin enthusiasts, including Andreas Antonopoulos who still tries to throw sand in Western Union’s eye.  Seriously, watch the linked video in which Antonopoulos claims that Bitcoin will somehow help the poor masses save money such that they can now invest in and acquire clean water.  It’s cringe worthy.  Did Bitcoin, or Bitcoin-related businesses, actually do any of the things he predicted?  Beyond a few one-time efforts, not really.7 Never mind tangible outcomes, full steam ahead on the “save the world” narrative!

Many enthusiasts fail to incorporate in their cartoonish models: that the remittance and cross border payment markets have a set of inflexible costs that have led the price structure to look the way it does today, and a portion of those costs, like compliance, have nothing to do with the costs of transacting.8  There may be a way of reducing those costs, but it is disingenuous (and arguably unethical) to pull on the heart strings of those living on subsistence in order to promote your wares.9

Rather than repeat myself, check out the break down I provided on the same Western Union example back in 2014.  Or better yet, look at the frequently updated post from Save on Send, who has the best analysis bar none on the topic.

Back to loathing about ‘adoption’ numbers: few people were interested in actual usage beyond arbitrage opportunities and we know this because no one writes or publishes usage numbers anymore.10 I’ll likely have a new post on this topic next quarter but for a quick teaser: BitPay, like usual, still puts out headline numbers of “328% growth” but doesn’t say what the original 2016 baseline volume was in order to get the new number today.

I don’t strive to pick on BitPay (to be fair they’re like the only guys to actually publish something) but unfortunately for them, the market still has not moved their way: Steam recently dropped support for Bitcoin payments and a Morgan Stanley research note (below) showed that acceptance from top 500 eCommerce merchants dropped from 5 in 2016 to 3 in 2017.11

“This is possibly the saddest bitcoin chart ever” – BI. Source: Morgan Stanley

Due to a lack of relevant animated gifs, a full break down on the topic wouldn’t fit in this article.  But just a quick note, there were a number of startups that moved decisively away from their original stated business case of remittances and instead in to B2B plays (BitPesa, Bitspark) or to wallets (Abra). 12  These would be worth revisiting in a future article.13

So what does this all have to do with “legitimization”?

If you haven’t seen the Godfather trilogy, it’s worth doing so during or after the holiday break.14

This year we have collectively witnessed the techbro re-enactment of Godfather: Part 3 with the seeming legitimization of online bucket shops and dodgy casinos, aka cryptocurrency intermediaries, you wouldn’t talk about in polite company.

All of the worst elements of society, like darknet market operators, hate groups, and malware developers, effectively got eff you money and a cleansing mainstream “exit” courtesy of financial institutions coming in and regulators overwhelmed by all of the noise.15  Just like in No Country for Old Men, the bad guy(s) sometimes win.  This isn’t the end of that story but the takeaway for entrepreneurs and retail investors: don’t work or build anything. Just shill for coins on social media morning, noon, and night.

(2) Red Scares

I am old enough to remember back in 2013 when Bitcoin “thought leaders” welcomed Chinese Bitcoin users.  In late 2013, during the second bull run of that year, there were frequent reddit threads about how mainland Chinese could use Bitcoin to route around censorship and all the other common civil libertarian tropes.

Guess what happened?  On December 5th, 2013, the People’s Bank of China and four other ministries issued guidance which restricted activities that domestic banks could do with cryptocurrencies, thereby putting spot exchanges in a bit of a bind, causing panic and subsequently a market crash.  Within days there were multiple “blame China” threads and memes that still persist to this day.  Case in point: this thread titled, “Dear China” which had Mr. Bean flipping off people in cars, was voted to the top of /r/bitcoin within a couple months of the government guidance.  Classy.

As I detailed in a previous post, earlier in the autumn, several state organs in China finally closed down the spot exchanges, which in retrospect, was probably a good decision because of the enormous amounts of scams and deception going on while no one in the community was policing itself.16 In fact, some of the culprits that led Chinese exchanges into the dishonesty abyss are still around, only now they’re working for other high-profile Bitcoin companies. 17  Big surprise!

For example, Reuters did an investigation into some of the mainland exchanges this past September, prior to the closure of the spot exchanges.  They singled out BTCC (formerly BTC China) as having a checkered past:

Internal customer records reviewed by Reuters from the BTCChina exchange, which has an office in Shanghai but is stopping trading at the end of this month, show that in the fall of 2015, 63 customers said they were from Iran and another nine said they were from North Korea – countries under U.S. sanctions.

It’s unclear how much volume BTCC processed on behalf of North Koreans, one former employee says the volumes were definitely not zero.18 These were primarily North Koreans working in China, some in Dandong (right across the border).

For perspective: North Korea has been accused of masterminding the WannaCry ransomware attack and also attacking several South Korea exchanges to the tune of around $7 million this year.  Sanctions are serious business, check out the US Department of Treasury resource center to learn more.19

Isn’t China the root of all problems in Bitcoinland?

Source: Twitter

The sensationalism (above) is factually untrue yet look how many people retweeted and liked the quickly debunked conspiracy theory.  It’s almost as if, in the current mania, no one cares about facts.

As Hitchens might say: that which can be asserted without evidence, can be dismissed without evidence.  So to are the conspiracies around Bitcoin in China:

  • Is the Chinese government nationalizing Bitcoin?  No.
  • Is the Chinese government responsible for Bitcoin Cash.  No.
  • Is the Chinese government behind the rise in CryptoKitties. No.

In this bull market it is unclear why Paul has to resort to PR stunts, like making fearmongering tweets or opening a strike/call option at LedgerX with the bet that bitcoin will be worth $50,000 next year.20 There are many other ways to better utilize this capital: rethink investing in funds run by managers who are not only factually wrong but who spread fake rumors around serious issues like nationalization.

For instance, I don’t normally publicly write about who I meet, but this past July, while visiting Beijing I sat down with about a dozen members of their ‘Digital Money‘ team (part of the People’s Bank of China group involved in exploring and researching blockchain-related topics). 21 They had already spoken with my then-current employer as well as many other teams and companies (apparently the Zcash team saw them the very next day). While I don’t want to be perceived as endorsing their views, based on my in-depth discussion that day, this Digital Money team had clearly done their homework and heard from all corners of the entire blockchain ecosystem, both cryptocurrency advocates and enterprise vendors. They were interested in the underlying tech: how could the big umbrella of blockchain-related technology improve their financial market infrastructure?

Look at it another way: the Chinese government (or any government for that matter) has no need to nationalize Bitcoin, what value would it bring to them?  It would just be a cost center for them as miners don’t run for free.22  In contrast, their e-RMB team, based out of Shenzhen, has been experimenting with forks/clones of Ethereum.  This is public information.

But what about Jihan and Bitmain?  Aren’t they out to kill Bitcoin?

I can’t speak on his intentions but consider this: as a miner who manufacturers and sells SHA256 hardware that can be used by both Bitcoin and Bitcoin Cash (as well as any SHA256 proof-of-work coin), Bitmain benefits from repeat business and satisfied customers.  It is now clear that the earlier Antbleed campaign effort to demonize Bitmain was a massive PR effort to create a loss of confidence in Bitmain as it was promoted by several well known Bitcoin Core supporters and surrogates to punish Bitmain for its support for an alternative Bitcoin scaling roadmap and client.  In fact, as of this day, no one has brought forth actual evidence beyond hearsay, that covert ASICBoost is/was taking place.  Maybe they did, but you’d need to prove this with evidence.

Speaking of PR campaigns and mining…

(3a) Energy usage / mining

Over the past two months there have probably been more than a dozen articles whitewashing proof-of-work mining energy consumption numbers.  Coin Center, a lobbying group straight out of Thank You for Smoking, has its meme team out on continuous social media patrols trying to conduct damage control: no one must learn that Bitcoin mining isn’t free or that it actually consumes resources!

Source: Twitter

The title of the article above is complete clickbait BS.  Empirically proof-of-work mining is driving miners to find regions of the world that have a good combination of factors including: low taxes, low wages, low energy costs, quick time-to-market access (e.g., being able to buy and install new hashing equipment), reliable energy, reliable internet access, and low political turmoil (aka stability).23  Environmental impact and “clean energy” are talking points that Van Valkenburgh allege, but don’t really prove beyond one token “we moved to renewables!” story.  The next time Coin Center pushes this agenda item, be sure to just ask for evidence from miners directly.24.

Another example is in a recent Bloomberg View column from Elaine Ou (note: the previous company that she co-founded was shut down by the SEC).  She wrote:

Digital currency is wasteful by design. Bitcoin “miners,” who process transactions in return for new currency, must race to solve extremely difficult cryptographic puzzles. This computational burden helps keep the transaction record secure — by raising the bar for anyone who would want to tamper with it –- but also requires miners to build giant farms of servers that consume vast amounts of energy. The more valuable bitcoin becomes, the more miners are willing to spend on equipment and electricity.

Mining a proof-of-work coin (such as Bitcoin) can only be as ‘cheap‘ or ‘efficient’ as the block reward is worth. As the market price of a coin increases so too does the capital expended by miners chasing seigniorage.  This, we both agree on.

In the long run, proof-of-work miners will invest and consume capital up to the threshold in which the marginal costs of mining (e.g., land, labor, electricity, taxes, etc.) roughly equals the marginal revenue they receive from converting the bitcoins into foreign currency (aka real money) to pay those same costs.  This, we also both agree on.

What Ou makes a mistake on is in her first sentence: digital currencies are not all wasteful, only the proof-of-work variety are.  Digital currency != cryptocurrency.25

I know, I know, all other digital currencies that are not proof-of-work are crap coins and those who make them are pearl-clutching morons.  Contra Ou and Coin Center, it is possible for central banks, and even commercial banks, to issue their own digital currency — and they could do so without using resource intensive proof-of-work.26  The Bank of International Settlements recently published a good paper on the various CBDC models out there, well worth a read.  And good news: no mountains of coal are probably used in the CBDC issuance and redemption process.27

Back to proof-of-work coins: a hypothetically stable $1 million bitcoin will result in a world in which miners as a whole expend up to $1 million in capital to mine.  If the network ever became cheaper to operate it would also mean it is cheaper to permanently fork the network.  You can’t have both a relatively high value proof-of-work coin and a simultaneously non-resource intensive network.

While it is debatable as to whether or not Bitcoin mining is wasteful or not, it empirically does consume real resources beyond the costs of energy and the externalization of pollution onto the environment.  The unseen costs of hash generation for a $20,000 bitcoin is at least $13 billion in capital over a year that miners will eventually consume in their rent-seeking race albeit from a combination of resources.

Data source: BitInfoCharts

I quickly made the chart (above) to illustrate this revenue (or costs depending on the point of view).28 These are the eight largest proof-of-work-based cryptocurrencies as measured by real money market prices.

There are a few caveats: (1) some of the block rewards adjust more frequently than others (like XMR); (2) some of the coins have relatively low transaction fees which equates to negligible revenue so they were not included; (3) the month of December has seen some very high transaction fees that may or may not continue into 2018; (4) because block generation for some of these is based on an inhomogeneous Poisson process, blocks may come quicker than what was supposed to be “average.”

How to interpret the table?

The all-time high price for Bitcoin was nearly $20,000 per coin this year.  If in the future, that price held stable and persisted over an entire year, miners would receive about $13 billion in block rewards alone (not including transaction fees).  Empirically we know that miners will deploy and consume capital up to the point where the marginal costs equals the marginal value of the coin.29  So while there are miners with large operating margins right now, those margins will be eaten up such that about $13 billion will eventually be deployed to chase and capture those rewards.  Consequently, if all 8 of these proof-of-work coins saw their ATH extended through 2018, ceteris paribus, miners would collectively earn about $32.6 billion in revenue (including some fees).

There are a variety of sites that attempt to gauge what the energy consumption is to support the network hashrate.  Perhaps the most frequently cited is Digiconomist.  But Bitcoin maximalists don’t like that site, so let’s put together an estimate they cannot deny (yes, there are climate change denialists in the cryptocurrency world).

For the month of December, the network hashrate for Bitcoin hovered around 13.5 exahash/second or 13.5 million terahash/second (TH/s).

To get a lowerbound on how many hash-generating machines are being used, let’s look at a product called the S9 from Bitmain.  It is considered to be the most “efficient” off-the-shelf product that public consumers can order in volume.30 This mining unit generates around 13.5 TH/s.

So, if we were to magically wave our hands and replace all of the current crop of Bitcoin mining machines into the most efficient off-the-shelf product, we’d need about 1 million of these to be manufactured, shipped, installed, and maintained in order to generate the equivalent hashrate that the Bitcoin network has today.  Multiply 1 million S9’s times the amount of energy individually used by a S9 and you’d get a realistic lowerbound energy usage for the network today.31

Note: this doesn’t factor in land prices, energy costs, wages for employees, building the electrical infrastructure (e.g., installing transformers), and many other line items that are unseen in the chart above.  It also doesn’t include the most important factor: as more mining hashrate is added and the difficulty rating adjust upward, it dilutes the existing labor force (e.g., your mining unit does not improve or become more productive over time).

(3b) Energy usage upperbound

So what are the upperbound costs?

Source: Twitter

The tweet above is not a rare occurrence.  If you are reading this, you probably know someone who tried to mine a cryptocurrency from an office computer or maybe their computer was the victim of ransomware.

You may not think of much of the externalization and socialization of equipment degradation that is taking place, but because mining is a resource intensive process, the machines used for that purpose depreciate far faster than those with normal office usage.32  To date, no one has done a thorough analysis of just how many work-related computers have been on the receiving end of the mining process but we know that employees sometimes get caught, like the computer systems manager for the New York City Department of Education or the two IT staffers in Crimea.33

Even if miners eventually fully utilize renewable energy resources, most hash-generating machines currently deployed do not and will not next year.  These figures also do not factor in the fully validating nodes that each network has that run out of charity (people run them without any compensation) yet consume resources.  According to Bitnodes, Bitcoin has around 11,745 nodes online. According to EtherNodes, Ethereum has around 26,429 nodes online.

So is there an actual upperbound number?

There is, by dividing hashpower by cost and comparing to costs of various known processor types.  For instance, see this footnote for the math on how two trillion low-end laptop CPUs could be used.3435

Just looking at the hash-generating machines, according to Chen Min (a chip designer at Avalon Mining), as of early November, 5% of all transistors in the entire semiconductor industry is now used for cryptocurrency mining and that Ethereum mining alone is driving up DRAM prices.

This is not to say you should march in the streets demanding that miners should forgo the use of coal power plants and only use solar panels (which of course, require consumption of resources including semiconductors), there are after all, many other activities that are relatively wasteful.

But some Bitcoin and cryptocurrency enthusiasts are actively whitewashing the environmental impact of their anarchic systems and cannot empirically claim that their proof-of-work-based networks are any less wasteful or resource intensive than the traditional foreign capital markets they loathe.

In point of fact, while the traditional financial markets will continue to exist and grow without having to rely on cryptocurrencies for rationally pricing domestic economic activity, in 2018, as in years prior, Bitcoinland is still fully dependent on the stability of foreign economies providing liquidity and pricing data to the endogenous labor force of Bitcoin.  Specifically, I argue in a new article, that miners cannot calculate without using a foreign unit of account; that economic calculations on whether or not to deploy and consume capital for expanding mining operations can only be done with stable foreign currency.36

Keep in mind that cryptocurrencies such as Bitcoin only clear (not settle) just one coin (or token) whereas traditional financial markets manage, transact, clear and settle hundreds of different financial instruments each day. 37  For comparison, the Federal Reserve estimates that on any given day about 600 million payment, clearing, and settlement transactions take place in the US representing over $11 trillion in value.38  But this brings up a topic that is beyond the scope of this article.  Next section please.

(4) MIT’s Digital Currency Initiative

On the face of it, MIT’s DCI effort makes a lot of sense: one of the world’s most recognized institutions collaborating with cryptocurrency developers and projects worldwide.

But beneath the slick facade is a potential conflict of interest that has not been looked at by any media outlet.  Specifically, around its formal foray into building tools for central bank digital currency (CBDC).  Rob Ali, a well-respected lawyer turned research scientist (formerly with the Bank of England), was hired earlier this year by DCI to build and lead a team at MIT for the purpose of continuing the research he had started at the BoE.  This is no secret.

Less known is how this research has now morphed into a two-fold business:

  1. DCI charges central banks about $1 million a year to be a partner.39  What this allows the central bank to do is send staff to MIT and tap into its research capabilities.  This includes MIT representatives co-authoring a couple of papers each year focused on topics that the central bank is keen to explore.  Multiple central banks have written checks and are working together with DCI at this time.
  2. Building and licensing tools and modules to central banks and commercial banks.  DCI has hired several Bitcoin developers whom in turn have cloned/forked Bitcoin Core and Lightning.  Using this code as a foundation, DCI is building IP it aims to license to central banks who want to build and issue central bank digital currency.

Where is the conflict of interest?

DCI is housed within MIT’s Media Lab, whose current director is Joi Ito.  Ito is also the co-founder and director of Digital Garage.  Digital Garage is an investor in Blockstream and vocal advocate of Lightning; coincidentally Blockstream is building its own Lightning implementation. Having made several public comments in favor of Bitcoin Core’s hegemony, Ito also appears to be a critic of alternative blockchain implementations.

In looking at his publicly recorded events on this topic Ito does not appear to disclose that the organizations he co-runs and invests in, directly benefit from the marketing efforts that Bitcoin Core and Lightning receive.  Perhaps this is just miscommunication.

I’m all for competition in the platform and infrastructure space and think central bank digital currencies are legit (again check out this BIS paper) but this specific DCI for-profit business should probably be spun off into an independent company.  Why?  Because it would help reduce the perception that Ito – and others developers involved in it – benefits from these overlapping relationships.  After all, Bitcoin Core arguably has a disproportional political clout that his investment (Blockstream) potentially benefits from if/when Lightning goes into production.40 And again, this is not to say there shouldn’t be any private-public partnerships or corporate sponsorships of academic research or that researchers should be prohibited in investing in companies, rather just a recommendation for disclosure and clarity.

(5) Lightning Network

If you haven’t seen The Money Pit (with Tom Hanks), it is well worth it for one specific reason: the contractors and their staff who are renovating Hanks’ home keep telling Hanks that it will be ready in two weeks.

And after those two weeks are over, Hanks is informed yet again that it will be ready in another two weeks.

The Lightning Network, as a concept, was first announced via a draft paper in February 2015. Its authors, Tadge Dryja and Joseph Poon, had initially sketched out some of the original ideas at their previous employer Vaurum (now called Mirror).

Lightning, as it is typically called, is commonly used in the same breath as “the scaling solution,” a silver bullet answer to the current transactional limitations on the Bitcoin network.41 Nearly three years later, after enormous hype and some progress, a decentralized routing version still has not gone into production.  Maybe it will eventually but not one of its multiple implementations is quite ready today unless you want to use a centralized hub.42  Strangely, some of the terminology that its advocates frequently use, “Layer 2 for settlement,” is borderline hokum and probably has not been actually vetted to see if it fulfills the requirements for real “settlement finality.”43

And like multiple other fintech infrastructure projects, some of its advocates repeatedly said it would be ready in less than 6 months, several times.  For instance:

  • On October 7, 2015, Pete Rizzo interviewed multiple developers including Tadge Dryja and Joseph Poon regarding Lightning.  Rizzo wrote that: “In interview, Dryja and Poon suggested that, despite assertions project development could take years, Lightning could take as little as six months to be ready for launch.”
  • On April 5, 2016, Kyle Torpey interviewed Joseph Poon regarding expected time lines, stating that: “Lightning Network co-creator Joseph Poon recently supplied some comments to CoinJournal in regards to the current status of the project and when it will be available for general use. Poon claimed a functional version of the Lightning Network should be ready this summer.”
  • A month later, on May 5, 2016, Kyle Torpey interviewed Adam Back regarding his roadmap.  Torpey wrote that: “While all of these improvements are being implemented on Bitcoin’s base layer, various layer-2 solutions, such as the Lightning Network, can also happen in parallel. The Lightning Network only needs CHECKSEQUENCYVERIFY (along with two other related BIPs) and Segregated Witness to be accepted by the network before it can become a reality on top of the main Bitcoin blockchain.”
  • On November 12, 2016, Alyssa Hertig interviewed several developers including Pierre-Marie Padiou, CEO of ACINQ, one of the startups trying to building a Ligthning implementation.  According to Padiou: “The only blocker for a live Lightning implementation is SegWit. It’s not sure how or when it will activate, but if SegWit does activate, there is no technical thing that would prevent Lightning from working.”

Segregated Witness (SegWit) was activated on August 24, 2017.  More than four months later, Lightning is still not in production without the use of hubs.

Source: Twitter

Not to belabor the point, just this past week, one of the executives at Lightning Labs (which is building one of the implementations) was interviewed on Bloomberg but wasn’t asked about their prior rosy predictions for release dates.  To be fair, there is only so much they could cover in a six minutes allocation.

“Building rock solid infrastructure is hard,” is a common retort.

Who could have guessed it would take longer than 6 months?  Yes, for regular readers of my blog, I have routinely pointed out for several years that architecting and deploying financial market infrastructure (FMI) is a time consuming, laborious undertaking which has now washed out more than a handful of startups attempting to build “enterprise” blockchains.

For example, Lightning as a concept predates nearly every single enterprise-focused DLT vendor’s existence.  While not an equal comparison (they are trying to achieve different goals), there are probably ~5 enterprise-focused, ‘permissioned’ platforms that are now being used in mature pilots with real institutional customers and a couple could flip the “production” button on in the next quarter or so.4445

For what it is worth, enterprise DLT vendors as a whole did a very poor job managing expectations the past couple of years (which I mentioned in a recent interview).  And they certainly had their own PR campaigns during the past couple of years too, there is no denying that.  Someone should measure and quantify the amount of mentions on social media and news stories covering enterprise vendors and proposals like Lightning.46

Better late than never, right?  So what about missed time frames?

In a recent (unscientific) poll I did via Twitter (the most scientific voting platform ever!) found that of the more than 1,600 voters, 81% of respondents thought that relatively inexpensive anonymous Lightning usage won’t really be good to go for at least 6+ months.

Just as Adam Back proposed a moratorium on nebulous “contention” for six months (beginning in August), I propose a moratorium on using the term “Lightning” as a trump card until it is actually live and works without relying on hubs.  But don’t expect to see the crescendo of noise (and some signal) to die down in the meantime, especially once exchanges and wallets begin to demonstrate centralized, MSB-licensed implementations.47

With that suggestion, I can see it now: all of the Lightning supporters flaming me in unison on Twitter for not being a vocal advocate.  Sure beats shipping code!  To be even handed, Lightning’s collective PR effort was just one of many others (hello sofachains!) that could be scrutinized.  A future post could look at all funded infrastructure-related efforts to improve cryptocurrency networks.  Which ones, if any, showed much progress in 2017. 48

Interested in reading more contrarian views on the Lightning Network?  See Gerard and Stolfi (and Stolfi2x) (and Stolfi3x).  Let’s revisit in 6 months to see what has been launched and is in production.

(6) Objective reporting and analysis

Without sugar coating it: with the exception of a few stories, coin media not only dropped the ball on critically, objectively covering ICO mania this past year, but was largely complicit in its mostly corrupt rise.  This includes The Information, which is usually stellar, but seems to have fallen in the tank with the ICO pumpers.  That is, unless you’re a fake advisor and then they’ve got your number.

It took some time, but eventually mainstream and a few not-so-mainstream coverage has brought a much needed spotlight on some of the shady actions that took place this year. There were also a number of good papers from lawyers and academics published throughout 2017.

Your holiday reading list in no particular order:

One of my favorite articles this year should be yours too:

Just a few short months after Stephen Palley published the article above, a lawsuit occurred in which, surprise surprise, the plaintiffs highlighted specific claims in the white paper:

Source: Twitter

Note: that the SEC’s order against the Munchee ICO also relied on highlighting specific claims in the white paper.

Concluding remarks

Unfortunately 2017 will probably go down as the year in which several generations of nerds turned into day-trading schmucks, with colorful technical charts and all.50 This included even adopting religious slogans like:  Buy the dip!  Weakhands!  HODL!  We are the new 1%!  The dollar is crashing!  It’s not a bubble, it’s an adoption curve!

A few parting bits of advice: unfollow anyone that says this time things are different or the laws of economics have changed or calls themselves a “cryptolawyer” or who previously got shutdown by the SEC or who doesn’t have a LinkedIn page.  Rethink donating or investing funds to anyone who makes up rumors about mining nationalization or who was fired for gambling problems or has a communications team solely dedicated to designing memes for Twitter.51

Cryptocurrencies aren’t inherently bad and ideas like ERC721 are even cool.52 But as neat as some of the tech ideas may be, magic internet coins sure as heck continue to attract a lot of Scumbag Steves who are enabled by participants that have turned a blind eye.  It’s all good though, because everyone will somehow get a Moonlambo after the final boss is beaten, right?

Coda

I will have a separate post discussing predictions for 2018 but since we are reflecting on 2017, below are a few other areas worth looking into now that you’re a paper zillionare:

  • We have real empirical observation of hyperdeflation occurring: in which it is more rational to hoard the coin instead of spend it.  As a result, Bitcoin-focused companies that have accumulated bitcoin are still raising capital from external financial markets denominated in foreign currency instead of deploying (consuming) their own bitcoin. And these same startups are receiving valuations measured, not in terms of bitcoin, but in terms of a foreign unit of account.  What would change this trend?
  • Bitcoinland, with its heavy concentration of wealth, looks a lot like a feudal agrarian economy completely dependent on other countries and external financial markets in order to rationally deploy capital and do any economic calculation. Is there a way to build a dynamically adjustable cryptocurrency that does not rely on foreign capital or foreign reference rates?
  • How much proof-of-work related pollution has been externalized and socialized on the public at large due to subsidies in various regions like Venezuela?  What are the effects, if any, on global energy markets?
  • As traditional financial markets add products and solutions with direct ties to cryptocurrencies (futures, options, payments, custody), by the end of 2018 how much of the transactional activity on Bitcoin’s edges will be based on non-traditional financial markets (e.g., LocalBitcoins)?
  • There were a lot of publicity stunts this year.  Working backwards chronologically, the Andreas Antonopoulos donation could have been a publicity stunt, it also could be real.  The argument goes: how is someone with a best selling book, who charges $20,000+ for speaking engagements, and who has been receiving bitcoins for years (here is the public address), still in debt.  Maybe he is, maybe his family fell on hard times.  But few asked any questions when an anonymous person sent what amounted to $1 million in bitcoin enabling him to reset his tax basis.  (Hate me for writing this?  As an experiment, earlier this month I put up a Bitcoin and Ethereum address on the sidebar of the home page, feel free to shower me with your magic coins and prove me wrong.  I promise to convert it all into dirty filthy statist bucks.)  A few months prior to that, Jamie Dimon was accused of everything but eating babies after he said “Bitcoin is a fraud.”  Dozens of “Dear Jamie” letters were written begging him to see Bitcoin with their pure rose-tinted eyes.  At what point will Bitcoin enthusiasts grow some thick skin and ignore the critics they claim don’t matter?  And while we can continue to add PR stunts forever, the “fundraiser” for Luke-Jr’s home after Hurricane Irma had zero proof that it was his house, just a picture that Luke-Jr. says it was and the rest of the Bitcoin Core fan club promoting it.  Trust but verify?

[Note: if you found this research note helpful, be sure to visit Post Oak Labs for more in the future.]

Acknowledgements

Many thanks to the following for their constructive feedback: VB, YK, RD, CM, WG, MW, PN, JH

End notes

  1. Bitcoin fans basically walked onto the field before the football game, toppled the goal posts, and carried it outside the stadium declaring themselves victorious without having actually played the match. []
  2. How many of these unsophisticated buyers have subsequently lost the corresponding private keys?  See “Nearly 4 Million Bitcoins Lost Forever, New Study Says” from Fortune []
  3. I am sure I will be accused of being a “Bitcoin Cash shill” (which obviously I must be, there is no other explanation!) for pointing this out, but last week, one vocal Bitcoin Core supporter even proposed a commit to change the wording on Bitcoin.org surrounding low fees: “These descriptions of transaction features are somewhat open to interpretation; it would probably be best not to oversell Bitcoin given the current state of the network.” []
  4. As an actor on a classic Saturday Night Live sketch said: “You may ask how we at the Change Bank, make money? It’s simple, volume.” []
  5. I take issue with anyone claiming to be able to label transactions specifically as spam without doing an actual graph analysis.  See Slicing Data for more. Proof-of-lizard is not to be conflated with lizardcoin. []
  6. Note: this is not an endorsement of Visa, I do not have any equity or financial stake in Visa. []
  7. One reviewer commented: “One problem that affects all cryptocurrencies whether proof of work or of stake: What reason do most people have for using them that won’t run afoul of social policy objectives? As long as people need to convert them to regular fiat currencies, they have a distinct disadvantage. The only exception would be in failed economies where stable fiat currencies are restricted, until those governments see a cryptocurrency as a potential substitute and ban it. It is not even clear why a government would need to issue a cryptocurrency (not a CBDC). If it wants to serve unbanked people it could open or subsidize a bank for them which is what is being attempted in a few developing countries.” []
  8. One reviewer commented: “Fully peer-to-peer without banks ultimately leads to creating a new currency. A new currency means that for international payments you have the additional costs of converting into the currency and converting out of the currency. A currency not linked to a real world economy is always going to have a more volatile price (assuming it has any price at all). Volatility in FX always, always leads to higher transaction costs for exchange because the bid offer spread has to be wider. This is before you even get into the mining proof or work model and all its inherent flaws, which again ultimately result from trying to build a financial system without banks.” []
  9. One reviewer noted that: “Transferwise, Currency Fair, Revolut, Mondo and other startups are already doing it. And they’re doing it without having to break the rules and laws banks and Western Union have to play by. They’re building actual real, potentially sustainable businesses that are useful to society. They’re just not grabbing the headlines like the greater fool / Nakamoto Scheme is. When you build a real business, your scope for false promise making behind incoherent computer science jargon is pretty small.” []
  10. I even stopped aggregating numbers 18 months ago because fewer companies were making usage numbers public: it’s hard to write about specific trends when that info disappears.  Note: if you think you have some interesting info, feel free to send it my way. []
  11. BitPay has diversified its portfolio of services now, expanding far beyond the original merchant acceptance and recently closed a $30 million funding round.  However, the problem with their growth claims is they are typically measured in $USD volume. So, as the value of bitcoin has grown 10-20x (as measured in USD) in the past year, it is unclear how much BitPay has really grown in terms of new customers and additional transactions.  Note: the same can be said for most Bitcoin-specific companies making big growth-related claims, BitPay is just one example. []
  12. Movements occurred in other areas too, on the enterprise side, Chain was perhaps the most well known company to pivot away from that vertical. []
  13. One reviewer commented: “2017 was a good year for B2B players with some prominent funding rounds (e.g., Bitspark, Veem, BitPesa) and some claimed growth on blockchain “rails” (but also on non-blockchain) namely Veem and BitPesa. A big surprise of 2017 was a much broader awareness of cryptocurrencies, i.e., free massive PR. The Coinbase app became more popular than Venmo (and far ahead of any bank). As a result, one of the most intriguing questions right now for 2018 is if/how Coinbase could capitalize on this opportunity to become a full-fledged bank leveraging the best of banking-like services from players like Xapo, Uphold, and Luno?” []
  14. I suppose it is safe to assume that if you’re reading this, you are coin millionaire so you don’t worry about fiat-mandated holiday breaks like the rest of us. []
  15. Not all medium-to-large coin holders are the adopters you now see wearing suits on television talk shows.  Most coin holders, including the abusive trolls and misogynists on social media, have seen a large pay raise, enabling the worst elements to continue their bullying attacks and illicit activities.  See Alt-right utilizes bitcoin after crackdown on hate speech from The Hill []
  16. Worth pointing out that Ryan Selkis is attempting to push forward with a the self-regulatory effort called Messari.  See also: The Brooklyn Project. []
  17. Earlier this year, right after the law enforcement raids in China, one of the senior executives left BTCC but still remains on the board of the parent company that operates BTCC.  He quickly found a new senior role at another high-profile Bitcoin-focused company and uses his social media accounts to vigorously promote Bitcoin Core and maximalism. []
  18. As explored in a previous post, fake volumes among the Chinese exchanges was not uncommon and several of the large exchanges attempted to gain funding from venture capitalists while simultaneously faking the usage numbers. As one former employee put it: “That was an extraordinary attempt at fraud — faking the numbers through wash trading and simply printing trades, while using that data to attract investment and establish their valuation.” []
  19. Coinbase got into some problems in early 2015 when one of its investor decks highlighted the fact that cryptocurrencies, such as Bitcoin, could be used to bypass sanctions. []
  20. Ari Paul runs a small “crypto” hedge fund called BlockTower Capital (estimated to have between around $50-$80 million AUM) that like many companies in this space, faces an ongoing lawsuit.  Unclear why LPs didn’t just buy and hold cryptocurrencies themselves and cut out the hysteria and management fees. []
  21. Yea, I know, “money” is already digital… I didn’t give them that name, they did. []
  22. One reviewer noted: “The fact remains that if you replace the mining process with a a centralized system for validation of transactions and up-to-date of balances you could run the whole thing on an ordinary sized server for a few thousand dollars per year. Centralisation and a more logical data model are vastly better technically speaking. And it would be far easier to add in compliance and links to banks for more robust and honest methods for exchanging between a centralized bitcoin and fiat. What would the Chinese government gain from mining?” []
  23. One of the often overlooked benefits of setting up a mining farm in China is that many of the parts and components of mining equipment are either manufactured in China and/or final assembly takes place in China.  So logistically it is much quicker to transport and install the hardware on-site within China versus transport and use overseas. []
  24. I know a bunch and could maybe introduce them though some of them make public appearances at conferences so they can usually be approached or emailed. []
  25. In fact, many regulators, such as the ECB, categorize cryptocurrency as a type of “virtual currency,” separate from a “digital currency.” []
  26. There is often confusion conflating “transaction processing” and “hash generation,” the two are independent activities.  Today mining pools handle the transaction processing and have sole discretion to select any transactions from the memory pool to process (historically there have been thousands of ’empty’ blocks) — yet mining pools are still paid the full block reward irrespective of how many transactions they do or not process.  Hash generation via mining farms has been a discrete service for more than 5 years — think of mining pools as the block makers who outsource or subcontract the hash generation out to a separate labor force (mining farms) and then a mining pool packages the transactions into a block once they receive the correct proof-of-work.  Note: “fees” to miners is a slightly different but related topic. []
  27. CBDCs have their own issues, like the risk of crowding out ordinary banks in market for deposits in a low interest rate environment but they have little in common with anarchic crytocurrencies. []
  28. Many thanks to Vitalik Buterin for his feedback and suggestions here. []
  29. See also: Some Crypto Quibbles with Threadneedle Street from Robert Sams []
  30. There are other mining manufacturers, including some who only build for themselves, such as Bitfury. []
  31. Interestingly enough, the market price for one of these machines is around $2,000.  And if you do the math, you’ll see exactly what all professional miners do: it’d only cost $2 billion to buy enough machines to generate 100% of the network hashrate and claim all the $13 billion in rewards to yourself!  In other words, the seigniorage is big, fat, and juicy… and will attract other miners to come and bid up the price of mining to the equilibrium point. []
  32. There are many walk-throughs of bitcoin mining facilities, including this video from Quartz. []
  33. In the process of writing this article, a new story explained how more than 105,000 users of a Chrome extension were unknowingly mining Monero.  Heroic theft of CPU cycles, right? []
  34. In theory, and practice, the upperbound is not infinite.  We know from the hashrate being generated that there are a finite amount of cycles being spent repeatedly multiplying SHA256 over and over.  Perhaps a possible, but improbable way to gauge the upperbound is to take the processing speed of a low-end laptop CPU (which is not as efficient at hashing as its ASIC cousins are).  At 6 MH/s, how many seventh generation i3 chips would it take to generate the equivalent of 13.5 million TH/s?  On paper, over 2 trillion CPUs.  Note: 1 terahash is 1 million megahashes.  So 1 million laptop CPUs each generating 6 MH/s on paper, would collectively generate around 6 TH/s.  The current network hashrate is 13.5 exahash/s.  So you’d need to flip on north of 2 trillion laptop CPUs to reach the current hashrate.  In reality, you’d probably need more because to replace malfunctioning machines: a low-end laptop isn’t usually designed to vent heat from its CPU throttled to the max all day long. []
  35. One China-based miner reviewed this scenario and mentioned another method to arrive at an upperbound: “Look at the previous generation of ASICs which run at 2-3x watt per hash higher.  The previous generation machines normally get priced out within 18 months.  But with differing electricity costs and a high enough price, these machines get turned on.  Or they go to cheap non-petrodollar countries like Russia or Venezuela. So your base load of 1 million machines will have an upperbound of 2x to 3x depending on prevailing circumstances.” []
  36. It may be also worth pointing out that the “evil Chinese miners blocking virtuous Core” narrative is hard to justify because Bitcoin’s current relatively high fees are a direct result of congestion and has consequently increased miner revenue by 33% (based on December’s fees).  So in theory, it’s actually in the miners interest to now promote the small block position.  Instead, in reality, most miners were and are the ones advocating for bigger block sizes, and certain Bitcoin Core representatives were blocking those proposals as described elsewhere but we’re not going down that rabbit hole today. []
  37. One reviewer commented: “Financial instruments that either directly perform a service to our economy and even indirectly via speculation, enable price discovery for things that are important to people’s lives. Who’s lives is Bitcoin really important to right now? To this day the only markets it can claim to have any significant market share in, let alone be leader in, is illicit trade and ransomware. The rest appears to be just people looking to pump and shill.” []
  38. It’s also probably not worth trying to start a discussion about what the benefits, if any, there is for society regarding cryptocurrency mining relative to the resources it collectively consumes, as the comments below or on social media would simply result in a continuous flame war.  Note: colored coins and metacoins create distortions in the security assumptions (and rewards) for the underlying networks.  Watermarked tokens are neither secure nor proper for financial market infrastructure. []
  39. It is not $1 million straight, there are multiple levels and tiers. []
  40. There is an ongoing controversy around key decision makers within Bitcoin Core (specifically those who approve of BIPs) and their affiliation with Blockstream.  One of Blockstream’s largest investors, Reid Hoffman, said Blockstream would “function similarly to the Mozilla Corporation” (the Mozilla Corporation is owned by a nonprofit entity, the Mozilla Foundation). He likened this investment into “Bitcoin Core” (a term he used six times) as a way of “prioritiz[ing] public good over returns to investors.” []
  41. Because it is its own separate network, it actually has cross-platform capabilities.  However, historically it has been promoted and funded for initial uses on the Bitcoin network moreso than others. []
  42. Yes, I am aware of the demo from Alex Bosworth, it is a big step forward that deserves a pat on the back.  Now to decentralize routing and provide anonymity to users and improve the UI/UX for normal users. []
  43. To start with, see the Principles for Financial Market Infrastructures. []
  44. This is not an endorsement of a specific platform or vendor or level of readiness, but examples would include: Fabric, Quorum, Corda, Axcore, Cuneiform, and Ripple Connect/RCL. []
  45. While Lightning implementations should not be seen as a rival to enterprise chains (it is an apples to oranges comparison), the requirements gathering and technical hurdles needed to be overcome, are arguably equally burdensome and maybe moreso for enterprise-focused companies.  Why?  Because enterprise-focused vendors each need approval from multiple different stakeholders and committees first before they deploy anything in production especially if it touches a legacy system; most Lightning implementations haven’t actually formally defined who their end-customer is yet, let alone their needs and requirements, so in theory they should be able to “launch” it faster without the check-off. []
  46. For instance, CoinDesk currently has 229 entries for “lightning,” 279 entries for “DLT,” and 257 entries for “permissioned.” []
  47. It bears mentioning that Teechain, can achieve similar KPIs that Lightning can, via the use of hardware, and does so today.  BitGo’s “Instant” and payment channels from Yours also attempt to achieve one similar outcome: securely transmitting value quickly between participants (albeit in different ways). []
  48. We’d need to separate that from the enterprise DLT world because again, enterprise vendors are trying to solve for different use cases and have different customers altogether.  Speaking of which, on the corporate side, there is a growing impatience with “pilots” and some large corporates and institutions are even pulling back.  By and large, “blockchain stuff” (people don’t even agree on a definition still or if it is an uncountable noun) remains a multi-year play and aside from the DA / ASX deal, there were not many 2017 events that signaled a shorter term horizon. []
  49. Note: both the Fedcoin and CAD-coin papers were actually completed and sent to consortium members in November 2016 then three months later, published online. []
  50. One reviewer commented: “There seems to be a whole new wave of both suckers and crooks to exploit the geeks. I have read some the Chartist analysis on forums for more traditional forms of day-trading such as FX day-trading and it is exactly the same rubbish of trying to inject the appearance of intelligence and analysis into markets that the day-traders (and those encouraging them) simply do not understand.” []
  51. A former Coinbase employee, now running a “crypto” hedge fund, was allegedly fired for gambling issues.  Maybe he wasn’t but there are a lot of addicts of many strains actively involved in trading and promoting cryptocurrencies; remember what one of the lessons of Scarface was? []
  52. ERC20 and ERC721 tokens may end up causing a top-heavy problem for Ethereum. See Watermarked tokens and also Integrating, Mining and Attacking: Analyzing the Colored Coin “Game” []

Bitcoin Is Now Just A Ticker Symbol and Stopped Being Permissionless Years Ago

Financial market infrastructure in just one country (Source)

What is FMI?  More on that later.  But first, let’s talk about Bitcoin.

If you aren’t familiar with the Bitcoin block size war and its endless online shouting matches which have evolved into legal and even death threats, then you have probably been a very productive human being and should sell hugs and not wander into a non-stop social media dance off.

Why?  Because tens of thousands of man (and woman) hours have collectively been obliterated over a struggle that has illuminated that Bitcoin’s development process is anything but permissionless.

It also illuminates the poor fiduciary care that some VCs have towards their LPs.  In this case, more than a handful of VCs do not seem to really care about what a few of their funded companies actually produce, unless of course the quarterly KPIs include “have your new Bitcoin meme retweeted 1,000 times once a week.”

In some documented cases, several dozen executives from VC-backed Bitcoin companies have spent thousands of hours debating this size attribute instead of building and shipping commercializable products.  But hey, at least they sell cool hats and built up very large Twitter followings, right?

Fact #1: Satoshi Nakomoto did not ask anyone’s permission to launch, change, or modify the codebase she unilaterally released in 2009.

Fact #2: In 2009, when Satoshi Nakomoto issued and minted a new currency (or commodity or whatever these MLIC are) she did so without asking anyone else’s approval or for their “ack.”

In the approximately seven years since she stopped posting under her pseudonym, influential elements of Bitcoin’s anarchic community have intentionally created a permissioned developer system commonly referred to as the Bitcoin Improvement Proposal (BIP) process.  “Bitcoin Core” is the name for the group that self-selected itself to vet BIPs; involvement is empirically permissioned because you can get kicked off the island.1 There are a small handful of decision makers that control access to the code repository.

For example, if you’re a developer that wants to create and launch a new implementation of Bitcoin that includes different block sizes… and you didn’t get it approved through this BIP process, guess what?  You are doing permissionlessness wrong because you didn’t get permission from the BIP approval committee to do so.

Oh, but you realize that and still want to launch this new Bitcoin implementation with the help of other elements of the community, such as some miners and exchanges?

According to some vocal members of the current BIP approval committee (Bitcoin Core) and its surrogates, this is an attack on Bitcoin.  Obviously this is absurd because there is no de jure or legally defined process for changing or forking Bitcoin, either the chain itself or the code.

There is no terms of service or contract which explicitly states what Bitcoin is and who controls its development process.  Or more historically: if Satoshi didn’t need permission from a (non-existent) BIP approval committee to launch a cryptocurrency, then no other Bitcoin developer needs to either.

Tickers

Fast forward to this current moment in time: if the Bitcoin Cash or Segwit2X forks are an attack on network because either fork did not get ack’ed (approved) by the right people on the BIP approval committee or retweeted by the right “thought leaders” on social media, then transitively every 10 minutes (when a block is generated by a miner) arguably could be an attack on Bitcoin.

Why?  At any time a block maker (miner) could use a different software implementation with different consensus rules.  They, like Satoshi before them, do not need permission to modify the code.

Oh, but other miners may not build on top of that block and some exchanges may not recognize those blocks as “legitimate” Bitcoin blocks?

That is certainly a risk.  In fact, several exchanges are now effectively white listing and black listing — permissioning — Bitcoin-related blocks.

For instance, Bittrex, a large crypto-to-crypto exchange, has said:

The “BTC” ticker will remain the Bitcoin Core chain before the hard fork block. Bittrex will observe the Bitcoin network for a period of 24 to 48 hours to determine if a chain split has occurred and the outcome.

In the event of a chain split, “BTC” will remain the existing Bitcoin chain with 1 MB blocks until the industry and ecosystem demonstrates a clear chain preference for Bitcoin.

Bitfinex, the largest (and most nebulous) cryptocurrency exchange in the world, took this even further by stating:

The incumbent implementation (based on the existing Bitcoin consensus protocol) will continue to trade as BTC even if the B2X chain has more hashing power.

After heavy public (and private) lobbying by members and surrogates of Bitcoin Core, other exchanges have instituted similar policies favoring the incumbent.2  So what can alternative implementations to do?  Bend the knee?

Daenerys Targaryen, Breaker of Chains

Historically miners have built on the chain that is both the longest and also has the most accumulated difficulty… and one that has enough profitability to pay for the electricity bills.  It just happens that this collective block building activity is never called an “attack” because in general, most participants have been happy enough with the status quo.

Visions of what Bitcoin is and how it should be defined have clearly, empirically shifted over time.  But since this network was purposefully designed to be self-sovereign and anarchic — lacking contracts and hooks into any legal system — no one group can claim legitimacy over its evolution or its forks.

As a result, recent war cry’s that Segwit2X is a “51% attack” on Bitcoin are a red herring too because there is no consensus on the definition of what Bitcoin is or why the previous block – in which approximately 51% of the hashrate created a block – is not an attack on Bitcoin. 3

This has now morphed into what the “BTC” ticker on exchanges represents.  Is it the longest chain?  The chain with the most accumulated difficulty?  The chain maintained by Bitcoin Core or now defunct NYA developers?  If a group of block makers can build blocks and exchanges are willing to list these coins as “BTC” then that specific chain has just as much legitimacy as any other fork other miners build on top of and exchanges may list.

Furthermore, if the BIP approval committee gets to say what software miners or exchanges should or should not use (e.g., such as increasing or decreasing the block size), that could mean that existing network is a managed and even administered.  And this could have legal implications.  Recall that in the past, because block making and development were originally separate, FinCEN and other regulators issued guidance stating that decentralized cryptocurrencies were exempt from money transmission laws.

Despite what the trade associations and Bitcoin lobbying groups would like the narrative to be, I recently published an article that went into this very topic in depth and have publicly asked several prominent “crypto lawyers” to provide evidence to the contrary (they have yet to do so).  An argument could be made that these dev groups are not just a loose collective of volunteers.

Financial market infrastructure

I’m not defending S2X or XT or Bitcoin Unlimited.  In fact, I have no coins of any sort at this time.

But even if you don’t own any bitcoins or cryptocurrencies at all, the block size debate could impact you if you have invested in the formal financial marketplace.

For example, if and when the CME (and similar exchanges) get CFTC approval to list cryptocurrency-related futures products and/or the NYSE (and similar exchanges) get SEC approval to list cryptocurrency-related ETFs, these products will likely result in a flood of institutional money.

Once institutions, regulators, and sophisticated investors enter the picture, they will want to hold people accountable for actions.  This could include nebulous “general partnerships” that control GitHub repositories.  Recall, in its dressing down of The DAO, the SEC defined the loose collective building and maintaining The DAO as a ‘general partnership.’  Is Bitcoin Core or other identifiable development teams a “general partnership”?

Maybe.  In fact, the common refrain Bitcoin Core and its surrogates continually use amounts to arguments in favor of a purported natural monopoly.

For instance, Joi Ito, Director of MIT’s MediaLab, recently stated that:

“We haven’t won the battle yet. [But] I think the thing that is interesting is that Bitcoin Core has substantially more brain fire power than any of the other networks.”

This is problematic for a couple reasons.

First, Joi Ito is not a disinterested party in this debate.  Through Digital Garage (which he co-founded) it has invested in Blockstream, a company that employs several influential Bitcoin Core devs.4  Ignoring the potential conflict of interest, Ito’s remarks echo a similar sentiment he also made last year, that Core is basically “The Right Stuff” for NASA: they are the only team capable of sending humans into space.

But this is an empirically poor analogy because it ignores technology transfer and aerospace education… and the fact that multiple countries have independently, safely sent humans, animals, and satellites into space.

It also ignores how competitive verticals typically have more than just one dominant enterprise: aerospace, automobiles, semiconductor manufacturers, consumer electronic manufacturers (smart phones), etc.  Each of these has more than one company providing goods and services and even usually more than just one product development team developing those.  Intel, for example, has dozens of design teams working on many new chips at any given time of the year.  And they are just one of the major semiconductor companies.

Even in the highly regulated markets like financial services there is more than one bank.  In fact, most people are unaware of this but banks themselves utilize what is called “Core Banking Software” and there are more than a dozen vendors that build these (see image below).

It is a bit ironic that Bitcoin Core seeks to have a monopoly on the BIP process yet even banks have more than one vendor to choose from for mission critical software securely managing and processing trillions of dollars in assets each day.5

On the enterprise (non-anarchic) blockchain side of the ecosystem, there are well over a dozen funded teams shipping code, some of which is being used in pilots by regulated institutions that are liable if a system breaks.  Note: this is something I discussed in my keynote speech (slides) at the Korea Financial Telecommunications and Clearings Institute last year.

But as one vocal Core supporter in a WeChat room recently said, Bitcoin Core is equivalent to Fedwire or Swift, there is only one of each; so too does it make sense for only one Bitcoin dev team to exist.

Firstly, this conflates at least four different things: a specific codebase, with permissioned dev roles, with acceptance processes, with a formal organization.

It is also not a good analogy because there are many regulatory reasons why these two systems (Swift and Fedwire) exist the way they do, and part of it is because they were either setup by regulators and/or regulated organizations.  In effect, they have a bit of a legally ring-fenced marketplace to solve specific industry problems (though this is somewhat debatable because there are some alternatives now).6

If this supporter is equating Core, the codebase, with real financial market infrastructure (FMI), then they should be prepared to be potentially regulated.  Bitcoin Core and many other centralized development teams are comprised of self-appointed, vocal developers that are easy to identify (they have setup verified Twitter accounts and attend many public events), so subpoenas and RFI’s can be sent their way.

As I mentioned in my previous article: with great power comes great accountability.  Depending on the jurisdiction, Core and other teams could end up with regulatory oversight since they insist on having a monopoly on the main (only) implementation and process by which the implementation is managed.7

Remember that Venn diagram at the very top?  The companies and organizations that manage FMI today for central banks (RTGSs), central securities depositories (CSDs), and other intermediaries such as custodians and CCPs, have specific legal and contractual obligations and liabilities.

Following the most recent financial crisis, the G-20 and other counties and organizations established the Financial Stability Board (FSB) to better coordinate and get a handle on systemic risks (among other issues).  And while the genesis of the principles for financial market infrastructures (PFMI) had existed prior to the creation of the FSB, how many of the international PFMI standards and principles does Bitcoin Core comply with?

Spoiler alert: essentially none, because Satoshi intentionally wasn’t trying to solve problems for banks.  So it is unsurprising that Bitcoin isn’t up to snuff when it comes to meeting the functional and non-functional requirements of a global payments platform for regulated institutions.  Fact-check me by reading through the PFMI 101 guide.

When presented with these strong legal accountability and international standards that are part and parcel with running a payment system, there is lots of hand waving excuses and justifications from Core supporters (and surrogates) as to why they are exempt but if Core wants to enforce its monopoly it can’t have it both ways.  Depending on the jurisdiction they may or may not be scrutinized as FMI.

But in contrast, in looking at the evolution and development of the enterprise chain ecosystem – as I described in multiple previous articles – there are valuable lessons that can be learned from these vendors as to how they plan to operate a compliant network.  I recall one conversation with several managing directors at a large US investment bank over a year ago: maybe the enterprise side should just have CLS run a blockchain system since they have all the right business connections and fulfill the legal and regulatory check boxes.

Note: CLS is a very important FMI operator.  Maybe existing FMI operators will do just that.  Speaking of which, will Bitcoin Core (or other dev teams) apply to participate with organizations like the FSB that monitor systemically important financial institutions and infrastructure?

Angela Walch has argued (slides) that some coders, especially of anarchic chains, are a type of fiduciary.8  Even if this were not true, many countries have anti-monopoly and anti-trust laws, with some exceptions for specific market segments and verticals.  There are also laws against organized efforts involved in racketeering; in the US these are found within the RICO Act.

Watch the Godfather trilogy

I haven’t seen a formal argument as to why Core or other development teams could meet the litmus test for being prosecuted under RICO laws (though the networks they build and administer are frequently used for money laundering and other illicit activity).  But trying to use the “decentralization” trump card when in fact development is centralized and decisions are made by a few key individuals, might not work.

Look no further than the string-pulling Mafia which tried to decentralize its operations only for the top decision makers to ultimately be held liable for the activities of their minions.9  And using sock puppets and pseudonyms might not be full proof once forensic specialists are brought in during the discovery phase.10

Concluding remarks

Based on observations from how Bitcoin Core evolved and consolidated its power over time (e.g. removing participants who have proposed alternative scaling solutions), the focus on what Bitcoin is called and defined has landed in the hands of exchanges and really just highlights the distance that Bitcoin has walked away from a “peer-to-peer electronic cash” that initially pitched removing intermediaries.  To even care about what ticker symbol ‘Bitcoin’ is on an exchange is to acknowledge the need for a centralized entity that establishes what the “price” is and by doing so takes away the bitcoin holder’s “self-sovereignty.”11

While the power struggles between various factions within the Bitcoin development community will likely rage on for years, by permissioning off the development process, Bitcoin Core (and any other identifiable development groups), have likely only begun to face the potential regulatory mine field they have foisted on themselves.12

Historically blockchain-based systems have and still are highly dependent on the input and decision-making by people: somebody has to be in charge or nothing gets done and upgrades are a mess.  And the goal of appointing or choosing specific teams on anarchic chains seems to be based around resolving political divisions without disruptive network splits.13

The big questions now are: once these teams are in charge, what will governments expectations be?  What legal responsibilities and regulatory oversight will the developers have?  Can they be sued for anti-trust and/or RICO violations?  With billions of dollars on the line, will they need to submit upgrade and road map proposals for approval?

Endnotes

  1. Examples of developers who were removed: Alex Waters, Jeff Garzik, Gavin Andresen []
  2. Thanks to Ciaran Murray for identifying these exchanges. []
  3. Bitcoin mining is in fact based on an inhomogeneous Poisson process; a participant could theoretically find a block with relatively little hash rate.  Although due to the probabilities involved, most miners pool their resources together to reduce the variance in payouts. []
  4. According to one alleged leak, Digital Garage is testing Confidential Assets, a product of Blockstream. []
  5. According to a paper from the Federal Reserve: payment, clearing, and settlement systems in the United States “process approximately 600 million transactions per day, valued at over $12.6 trillion.” []
  6. On AngelList, there are about 3,400 companies categorized as “payments” — most of these live on top of existing FMI, only a handful are trying to build new independent infrastructure. []
  7. A key difference between Bitcoin and say Ethereum is that with Ethereum there are multiple different usable implementations managed by independent teams and organizations; not so with how Bitcoin has evolved with just one (Bitcoin Core) used by miners.  In addition, the Ethereum community early on formally laid out a reference specification of the EVM in its yellow paper; Bitcoin lacks a formal reference specification beyond the Core codebase itself. []
  8. See also The Bitcoin Blockchain as Financial Market Infrastructure: A Consideration of Operational Risk from Angela Walch []
  9. Thanks to Stephen Palley for providing this observation. []
  10. It is unclear why the current Bitcoin Core team is put onto a pedestal.  There are many other teams around the world building and shipping blockchain-related system code used by companies and organizations (it is not like there is only just one dev team that can build all databases or operating systems).  At the time of this writing Core has not publish any papers in peer-reviewed journals and many of them do not have public resumes or LinkedIn profiles because they have burned business and professional relationships in the past.  Irrespective of what their bonafides may or may not be, it is arguably a non sequitur that ‘permissionless’  coordination in open-source code development has to lead to a monopoly on said development. []
  11. Thanks to Colin Platt for this “appeal to authority” observation. []
  12. Bitcoin stopped being permissionless when developers, miners, and exchanges needed to obtain permission to make and use different code.  And likely there are and will be more other cryptocurrency development teams that follow that same path. []
  13. For an informed contrarian view on governance and distributed ledger technology, see The blockchain paradox: Why distributed ledger technologies may do little to transform the economy by Vili Lehdonvirta []

Intranets and the Internet

It is early into 2017 and at fintech events we can still hear a variety of analogies used to describe what blockchains and distributed ledger technology (DLT) are and are not.

One of the more helpful ones is from Peter Shiau (formerly of Blockstack.io) who used an automobile analogy involving the Model T to describe magic internet chains:1

The Ford Motor Company is well known for its production engineering innovation that gave us the Model T. To this day, the Ford Model T is one of the best selling automobiles of all-time thanks to the sheer number produced and affordability for American middle class families.  And while it was remarkable that Ford was able to sell so many cars, it is well understood Ford’s true innovation was not the Model T but in fact the modern assembly line.

It was this breakthrough that enabled Ford to build a new car every 93 minutes, far more quickly than any of its competitors. Not unlike the Model T, cryptocurrencies like Bitcaoin, are every bit the product of a similar innovative process breakthrough that today we call a “blockchain.”

Carrying the analogy a little further, what is even more powerful about this modern equivalent of the assembly line is that it is not just useful for building cars but also vans and trucks and boats and planes. In just the same way, a blockchain is not just useful for creating a cryptocurrency, but can be applied to a many different processes that multiple parties might rely on to reach agreement on the truth about something.

Less helpful, but all the same plentiful, are the many red herrings and false equivalences that conferences attendees are subjected to.

Arguably, the least accurate analogy is that public blockchains can be understood as being “like the internet” while private blockchains “are like intranets”.

Why is this one so wrong and worthy of comment?

Because it is exactly backwards.

For example, if you want to use a cryptocurrency like Bitcoin, you have to use bitcoin; and if you want to use Ethereum, you have to use ether.  They are not interoperable.  You have to use their proprietary token in order play in their walled garden.

As described in detail below, the internet is actually a bunch of private networks of internet service providers (ISPs) that have legal agreements with the end users, cooperate through “peering” agreements with other ISPs, and communicate via a common, standardized routing protocols such as BGP which publishes autonomous system numbers (ASNs).

In this respect, what is commonly called “the Internet” is closer to interoperable private, distributed ledger networks sharing a common or interoperable communication technology than anarchic, public cryptocurrency blockchain networks, which behave more like independent isolated networks.

Or in short: by design, cryptocurrencies are intranet islands whereas permissioned distributed ledgers — with interoperability hooks (“peering” agreements) — are more like the internet.2

Sidebar

Let’s do a short hands-on activity to see why the original analogy used at fintech conferences is a false equivalence with implications for how we need to frame the conversation and manage expectations in order to integrate DLT in to our reference and business architecture.

If you are using a Windows-based PC, open up a Command window.  If you’re using a Mac or Android device, go to a store and buy a Windows-based PC.

Once you have your Command window open, type in a very simple command:

tracert: www.google.com

Wait a few seconds and count the hops as your signal traces the route through various network switches and servers until you finally land on your destination.  From my abode in the SF area, it took 10 hops to land at Google and 7 hops to land at Microsoft.

If you did this exercise in most developed countries, then the switches and servers your signal zigged and zagged through were largely comprised of privately owned and operated networks called ISPs.  That is to say, what is generally described as “the internet” is just a bunch of privately run networks connected to one another via several types of agreements such as: transit agreements, peering agreements, and interconnect agreements.

By far the most widely used agreement is still done via the proverbial “handshake.”  In fact, according to a 2012 OECD report, 99.5% of internet traffic agreements are done via handshakes.  There is also depeering, but more on that later.

What do all these agreements look like in practice?

According to the 2016 Survey of Internet Carrier Interconnection Agreements (pdf):

The Internet, or network of networks, consists of 7,557 Internet Service Provider (ISP) or carrier networks, which are interconnected in a sparse mesh. Each of the interconnecting links takes one of two forms: transit or peering. Transit agreements are commercial contracts in which, typically, a customer pays a service provider for access to the Internet; these agreements are most prevalent at the edges of the Internet, where the topology consists primarily of singly connected “leaf” networks that are principally concerned with the delivery of their own traffic. Transit agreements have been widely studied and are not the subject of this report. Peering agreements – the value-creation engine of the Internet – are the carrier interconnection agreements that allow carriers to exchange traffic bound for one another’s customers; they are most common in the core of the Internet, where the topology consists of densely interconnected networks that are principally concerned with the carriage of traffic on behalf of the networks which are their customers.

Colloquially it is a lot easier to say “I want to use the Internet” instead of saying “I want to connect with 7,557 ISPs interconnected in a sparse mesh.”

Back to topology, each ISP is able to pass along traffic that originated from other networks, even if these external networks and the traffic therein originate from foreign countries, because the physical systems can speak to one another via standardized transport protocols like TCP and UDP and route via BGP.3 4

Thus there is no such thing as a physical “internet rail,” only an amalgam of privately and publicly owned networks stitched together.

And each year there is inevitably tension between one more ISP and consequently depeering takes place.  A research paper published in 2014 identified 26 such depeering examples and noted that while depeering exists:

Agreements are very quite affair and are not documented for, they are mostly handshake agreements where parties mutually agree  without  any  on  record  documentation.  This  argument is supported by the fact that 141,512 Internet Interconnection Agreements out of 142,210 Internet Agreements examined till March 2011 were Handshake Agreements.

This is the main reason you do not hear of disputes and disagreements between ISPs, this also dovetails into the “net neutrality” topic which is beyond the scope of this post.

Intranets

Just as the internet is an imperfect analogy for blockchains and DLT in general, so is its offspring the “intranet” is a poor analogy for a permissioned blockchains.  As noted above, the internet is a cluster of several thousand ISPs that typically build business models off of a variety of service plans in both the consumer and corporate environments.

Some of these server plans target corporate environments and also includes building and maintaining “private” intranets.

What is an intranet?

An intranet is a private network accessible only to an organization’s staff. Generally a wide range of information and services from the organization’s internal IT systems are available that would not be available to the public from the Internet. (Source)

And while more and more companies migrate some portion of their operations and work flows onto public and private “clouds,” intranets are expected to be maintained given their continued utility.  From an infrastructure standpoint, notwithstanding that an intranet could be maintained one or more more servers through Software Defined Networks (SDNs), it is still a subset of a mash up of ISPs and mesh networks.

What does this have to do with magic internet chains?

A private blockchain or private distributed ledger, is a nebulous term which typically means that the validation process for transactions is maintained by known, identified participants, not pseudonymous participants.  Depending on the architecture, it can also achieve the level of privacy that is associated with an intranet while staying clear of the hazards associated with preserving true pseudonymity.

Why is the “intranet” analogy so misleading and harmful?

For multiple reasons.

For starters, it is not really valid to make a sweeping generalization of all identity-based blockchains and distributed ledgers, as each is architected around specific use-cases and requirements.  For instance, some vendors insist on installing on-premise nodes behind the firewall of an enterprise.  Some vendors setup and run a centralized blockchain, from one or two nodes, for an enterprise. Some others tap into existing operational practices such as utilizing VPN connections.  And others spin up nodes on public clouds in data centers which are then operated by the enterprise.

There are likely more configurations, but as noted above: from a topological perspective in some cases these private blockchains and distributed ledgers operate within an intranet, or on an ISP, or even as an extranet.

Fundamentally the biggest difference between using an ISP (“the internet”) and using an intranet is about accessibility, who has access rights.  And this is where identity comes into play: most ISPs require the account holder to provide identification materials for what is effectively KYC compliance.

Thus while you may be visit a coffee shop like Starbucks who provides “free” access, Starbucks itself is an identified account holder with an ISP and the ISP could remove Starbucks access for violating its terms of service.  Similarly, most coffee shops, airports, schools, etc. require users to accept a terms of service acknowledging that their access can be revoked for violating it.

Source: FireFox 51.0.1

In short, both the internet and intranet are in effect part of identity and permission-based networks.  There is no such thing as an identity-less internet, only tools to mask the users identity (e.g., Tor, Peerblock, Whisper).  In the same way that, “private” intranets are a fallacy.

Anarchic chains, which were designed to operate cryptocurrencies like Bitcoin, attempt to create an identity-less network on top of an identifiable network, hence the reason people involved in illicit activities can sometimes be caught.

Identity

Interestingly, where the internet analogy does hold up is in how public, anarchic blockchains are no less challenged by the effort and complexity of truly masking identity. I mentioned this in a footnote in the previous post, but it deserves being highlighted once more. Anarchic blockchains inspired by cryptocurrencies such as Bitcoin, used blocks because Satoshi wanted identity-free consensus (e.g., pseudonymity).  That implies miners can come and go at will, without any kind of registration, which eliminated the choice of using any existing consensus algorithm.

As a result, Satoshi’s solution was proof-of-work (PoW).  However, PoW is susceptible to collisions (e.g., orphan blocks).  When a collision occurs you have to wait longer to obtain the same level of work done on a transaction. Thus you want to minimize them, which resulted in finding a PoW on average every ten minutes.  This means that in a network with one minute propagation delays, not unlikely in a very large network (BGP sees such propagation times) then you waste ~10% of total work done, which was considered an acceptable loss rate in 2008 when Satoshi was designing and tweaking the parameters of the system.

Distributed ledgers such as Corda, use a different design and exist precisely as an identified network, where members cannot just come and go at will, and do have to register. With Corda, the team also assumes relatively low propagation times between members of a notary cluster.  One of the key differences between mere PoW (i.e. hashcash) and a blockchain is that in the latter, each block references the prior – thus PoWs aggregate.  It can be tough to do that unless all transactions are visible to everyone and there is a single agreed upon blockchain but if you do not, you will not get enough PoW to yield any meaningful security

When fintech panels talk about the notion of “open” or “closed” networks, this is really a red herring because what is being ignored is how identity and permission work and are maintained on different types of networks.

From the standpoint of miner validation, in practice cryptocurrencies like Bitcoin are effectively permission-based: the only entity that validates a transaction is effectively 1 in 20 semi-static pools each day.  And the miners/hashers within those pools almost never individually generate the appropriate/winning hash towards finding a block.  Each miner generates trillions of invalid hashes each week and are rewarded with shares of a reward as the reward comes in.

And if you want to change something or possibly insert a transaction, you need hashrate to do so.  Not just anyone running a validating node can effect change.

More to the point, nearly all of these pools and many of the largest miners have self-doxxed themselves.  They have linked their real world identities to a pseudonymous network whose goals were to mask identities via a purposefully expensive PoW process.  As a result, their energy and telecommunication access can be revoked by ISPs, energy companies, and governments.  Therefore calling anarchic or public blockchains “open” is more of a marketing gimmick than anything else at this stage.

Clarity

AOL and CompuServe were early, successful ISPs; not intranets.5  Conflating these terms makes it confusing for users to understand the core technology and identify the best fit use-cases. 6

Alongside the evolution of both the “cloud” and ISP markets, it will be very interesting to watch the evolution of “sovereign” networks and how they seek to address the issue of identity.

Why?

Because of national and supranational laws like General Data Protection Regulation (GDPR) that impacts all network users irrespective of origin.

For instance, Marley Gray (Principal Program Manager Blockchain at Microsoft) recently explained in an interview (above) how in order to comply with various data regulations (data custody and sovereignty), Microsoft acquired fiber links that do not interact with the “public” internet.  That is to say, by moving data through physically segregated “dark” networks, Microsoft can comply with requirements of its regulated customers.

And that is what is missing from most fintech panels on this topic: at the end of the day who is the customer and end-user.

If it is cypherpunks and anarchists, then anarchic chains are built around their need for pseudonymous interactions.  If it is regulated enterprises, then identity-based systems are built around the need for SLAs and so forth.  The two worlds will continue to co-exist, but each network has different utility and comparative advantage.

Acknowledgements: I would like to thank Mike Hearn, Stephen Lane-Smith, Antony Lewis, Marcus Lim, Grant McDaniel, Emily Rutland, Kevin Rutter, and Peter Shiau for their constructive feedback. This was originally sent to R3 members on March 31, 2017.

Endnotes

  1. His analogy is reused with permission. []
  2. From a network perspective, some of the integration and interop challenges facing DLT platforms could be similar to the harried IPv4 vs IPv6 coexistence over the past decade.  Who runs the validating nodes, the bridges — the links between the chains and ledgers — still has to be sorted out.  One reviewer noted that: If you equate IPv4 (TCP/UDP/ICMP) to DLTv4 where BGPv4 enables IPv4 networks to interact, we need an equivalent for BPGv4, say DLTGPv4 (DLT Gateway Protocol) for DLTv4 fabrics (ISPv4s) to interact and the same thing for IPv6 and DLTv6 where DLTv6 is a different DLT technology than DLTv4.  So the basic challenge here is solving integration of like DLT networks. []
  3. Venture capitalists such as Marc Andreessen and Fred Wilson have stated at times that they would have supported or invested in something akin to TCPIPcoins or BGPcoins.  That is to say, in retrospect the missing element from the “internet stack” is a cryptocurrency.  This is arguably flawed on many levels and if attempted, would likely have stagnated the growth and adoption of the internet, see page 18-19. []
  4. One reviewer noted that: Because of the IPv4 address restrictions (address space has been allocated – relying on auctions etc for organizations to acquire IPv4 addresses), some sites now only have an IPv6 address.  Most devices today are dual stack (support IPv4 and IPv6), but many ISPs and older devices still only support IPv4 creating issues for individuals to access IPv6 resulting in the development of various approaches for IPv4 to IPv6 (e.g. GW46 – my generic label).  I think, the question with DLTGW46 is whether to go dual stack or facilitate transformation between v4 and v6. []
  5. A reviewer who previously worked at AOL in the mid ’90s noted that: “In its early days, AOL was effectively a walled garden.  For example, it had its own proprietary markup language called RAINMAN for displaying content. And access to the internet was carefully managed at first because AOL wanted its members to stay inside where content was curated and cultural norms relatively safer — and also desirable for obvious business reasons.” []
  6. One reviewer commented: “In my opinion, the “internet” cannot be created by a single party. It is an emergent entity that is the product of multiple ISPs that agree to peer – thus the World Wide Web. DLT-based and blockchain-based services first need to develop into their own robust ecosystems to serve their own members. Eventually, these ecosystems will want to connect because the value of assets and processes in multiple ecosystems will increase when combined.” []

DLT as FMI in Korea

Yesterday I gave a keynote talk at “The Future of Financial Payment Services Driven by Technology Innovation” organized and hosted by the Korea Finance Telecommunications & Clearings Institute (KFTC).

It was their 30th Anniversary Seminar and was held in Seoul, South Korea.

Below are the slides I presented on “Distributed Ledger Technology as Financial Market Infrastructure”:

What is the difference between Hyperledger and Hyperledger?

hyperledgerI am frequently asked this question because there is some confusion related to the legacy name and the current branding of certain technology. The two are distinct. And how we got there involves a little history.

Hyper, the parent company of Hyperledger, was founded by Dan O’Prey and Daniel Feichtinger in the spring of 2014. Fun fact: one of the alternative names they considered using was “Mintette.com” — after the term coined by Ben Laurie in his 2011 paper.

The simplest way to describe Hyperledger, the technology platform from Hyper, during its formative year in 2014 was: Ripple without the XRP. Consensus was achieved via PBFT.1 There were no blocks, transactions were individually validated one by one.

Hyperledger, the technology platform from Hyper, was one of the first platforms that was pitched as, what is now termed a permissioned distributed ledger: validators could be white listed and black listed. It was designed to be first and foremost a scalable ledger and looked to integrate projects like Codius, as a means of enabling contract execution.

Most importantly, Hyperledger in 2014 was not based off of the Bitcoin codebase.

Note: in the fall of 2014 Richard Brown and I both became the first two advisors to Hyper, the parent company of Hyperledger.  Our formal relationship ended with its acquisition by DAH.2

In June 2015, DAH acquired Hyper (the parent company of Hyperledger) which included the kit and caboodle: the name brand, IP and team (the two Dans).  During the same news release, it was announced that DAH had acquired Bits of Proof, a Hungary-based Bitcoin startup that had designed a Java-based reimplementation of Bitcoin (which previously had been acquired by CoinTerra).3

It was proposed at that time that Hyperledger, the Hyper product, would become the permissioned ledger project from DAH.  It’s product landing page (courtesy of the Internet Archive) uses roughly the same terminology as the team had previously pitched it (see also the October homepage older homepage for DAH as well).

digital asset homepage october 2015

Source: Digital Asset / Internet Archive

On November 9, 2015, on a public blog post DAH announced that it was “Retiring Hyperledger Beta, Re-Open Sourcing Soon, and Other Changes.”

The two most notable changes were:

(1) development would change from the languages of Erlang and Elixir to Java and Scala;

(2) switch to the UTXO transaction model

The team noted on its blog in the same post:

We are also switching from our simplistic notion of accounts and balances to adopt to de facto standard of the Bitcoin UTXO model, lightly modified. While Hyperledger does not use Bitcoin in any way, the Bitcoin system is still extremely large and innovative, with hundreds of millions of dollars invested. By adopting the Bitcoin transaction model as standard, users of Hyperledger will benefit from innovation in Bitcoin and vice versa, as well as making Hyperledger more interoperable.

During this same time frame, IBM was working on a project called OpenChain, which for trademark reasons was later renamed (now internally referred to as OpenBlockchain).4

IBM’s first public foray into distributed ledgers involved Ethereum vis-a-vis the ADEPT project with Samsung (first announced in January 2015). Over the subsequent months, IBM continued designing its own blockchain (see its current white paper here).

In December 2015, the Linux Foundation publicly announced it was creating a new forum for discussion and development of blockchain technology.  Multiple names were proposed for the project including Open Ledger (which was the name originally used in the first press release). However, in the end, the name “Hyperledger” was used.

How did that occur?

DAH, one of the founding members of the project, donated two things to the Linux Foundation: (1) the brand name “Hyperledger” and (2) the codebase from Bits of Proof.

Recall that Bits of Proof was the name of a Bitcoin startup that was acquired by DAH in the fall of 2014 (the Chief Ledger Architect at DAH was the co-founder of Bits of Proof). 5 Architecturally, Bits of Proof is a Java-implementation of Bitcoin. 6

In other words: today the term “Hyperledger” represents an entirely different architectural design and codebase than the original Hyperledger built by Hyper.7

The major architectural switch occurred in November 2015, which as noted above involved adopting the UTXO transaction set and Java language that Bits of Proof was built with.  Therefore, Hyperledger circa 2016 is not the same thing as Hyperledger circa 2014.

Over the past two months there have been multiple different codebases donated to the Linux Foundation all of which is collectively called “Hyperledger” including the IBM codebase (partly inspired by Ethereum) as well as the DAH and Blockstream codebase (one is a clone of Bitcoin and the other is a set of extensions to Bitcoin). The technical discussions surrounding this can be found on both the public Linux Foundation mailing list and its Slack channel.

How do different, incompatible codebases work as one?

This technical question is being discussed in the Linux Foundation. It bears mentioning that as of now, the codebases are incompatible largely due to the fact that Bitcoin uses the UTXO transaction set and OpenBlockchain uses an “accounts” based method for handling balances.  There are other reasons for incompatibility as well, including that they are written in completely different languages: Java/Scala versus Go versus C++ (Blockstream).

How extensive is the reuse of the Bits of Proof Bitcoin codebase donated to the Linux Foundation from the DAH team?  According to a quick scan of their GitHub repo:

So when someone asks “what is Hyperledger technology?” the short answer is: it is currently the name of a collective set of different codebases managed by the Linux Foundation and is not related to the original distributed ledger product called Hyperledger created by Hyper. The only tenuous connection is the name.

Timeline in brief: Hyperledger was originally created in Spring 2014 by Hyper; Hyper was acquired in June 2015 by DAH; the original Hyperledger architecture was entirely replaced with Bits of Proof in November 2015; the Hyperledger brand name and Bits of Proof code was donated to the Linux Foundation in December 2015.

  1. Interestingly enough, the current OpenBlockchain project from IBM also uses PBFT for its consensus mechanism and uses an “accounts” based method; two characteristics that the original Hyperledger platform from Hyper had too. []
  2. For more info on the original Hyperledger, see the Innotribe pitch; the description in Consensus-as-a-service from April 2015 and the Epicenter Bitcoin interview. []
  3. Following the bankruptcy of CoinTerra, the Bits of Proof team became independent once again. []
  4. CoinPrism launched a project called OpenChain, before IBM did. []
  5. Sometimes there is a confusion between Bits of Proof and Bits of Gold.  Bits of Proof was the independent Java-implementation of Bitcoin (which is not the same thing as bitcoinj).  Bits of Gold is an Israeli-based Bitcoin exchange.  A co-founder of Bits of Gold also works at DAH and is their current CTO. []
  6. In the future it may contain some modifications including Elements from Blockstream. []
  7. What was once the original Hyperledger GitHub repo has been handed over to the Linux Foundation but some of the original code base and documentation from the 2014 project can still be viewed elsewhere. []

Cryptocurrency KYSF: Know Your Source of Funds part 2

ecommerceA few days ago I was asked a number of questions from a reporter at CoinDesk regarding on-chain trade volume; this was a follow-up from some questions back in early May.

A few of my responses were published in a new article today: Dark Web Markets ‘Processed more Bitcoin than BitPay in 2014’

Below are my unabbreviated comments:

Q: How have the recent posts from Coinbase and BitPay impacted the diagram you outlined in that previous post? Has it had any impact at all?

A: The most striking data point from the Coinbase and BitPay posts was what was missing: actual real user numbers.  Neither one of them is willing to publicly say how many monthly active users (MAU) they have which stands in contrast to other fintech companies, financial institutions and “social media” startups they like to compare themselves to.

For instance, even though Coinbase claims to have 2.4 million users/3.1 million wallets, what does that mean?  Are these all fully KYC’ed accounts?  What percent have logged on in the past month?  What percent have actually used Coinbase’s services?  How many simply create an account, deposit $10 and never log on again?

Similarly, BitPay numbers are actually pretty sobering.  We know demographically from both the CoinDesk report and the leaked Coinbase pitch deck that the over 80% of all bitcoin holders/owners are males between the ages of 18-45.  And that the majority of the overall users reside in North America.  Yet according to the BitPay charts, North American volume has been relatively flat the last 6 quarters.

So if the largest group of bitcoin owners are not using their holdings despite a marked increase in available merchants, that is probably not an indication that they are interested in spending their funds and probably see bitcoins as an investable asset than actual money.  BitPay also does not disclose aggregate USD or euro volume.  Startups like to make noise when they are doing good or can show growth; if the value of their volume was actually growing, they probably would say.

And while transaction count in Europe and Latin America appear to be growing, perhaps the collective value has stayed the same (the Latin America numbers are also a bit misleading; it’s easy to show large growth percentages when you start from 0).

Another point about BitPay’s post is that they don’t really say what “IT services” is.  Notably absent from this post, compared with their post in April, is what “mining” related activity is.  Recall that some miners, such as KnC and now defunct BFL were (are) using BitPay as their payment processor.  In fact, in BitPay’s post earlier this year, “Bitcoin Mining” — by volume — represented the largest share of volume processed.  Does “IT services” now include this previously large segment?

Lastly, one number they do not include is the total aggregate transactions by each quarter.  Eye-balling it, it appears for Q2 2015 they processed about 180,000 transactions.  Divided by 60,000 merchants comes to around 3 transactions per quarter or 1 transaction per month per merchant.

In all likelihood usage follows a power law or a 80-20 rule, that 20% of the merchants account for the majority of transaction volume.  My understanding is that Gyft uses (or used BitPay) as their payment processor and since 9% of all bitcoin-related transactions last quarter were related to gift cards, it is likely that the lionshare of this “gift card” activity in the power law distribution is represented by just one or two companies (e.g., FoldApp and Purse.io are a couple potential ones to look at as well).

Startups like Blockseer, Sabr, Coinalytics and Chainalysis have APIs and address labeling that may be able to tell us more about specific merchant/payment processor activity,

Q: Also, are clearnet tx outweighed by darknet tx with bitcoin? Silk Road and other marketplaces were the first use case for bitcoin, but are they still the biggest?

A: According to a new paper (Soska and Christin 2015), if you look at Figure 5 and the discussion involved, prior to Operation Olympus, six large dark net marketplaces collectively accounted for more than $600,000 in sales per day.  It is unclear how much of that activity was expressly illegal, although the paper does attempt to break down the amount of illicit drugs being sold on the same sites.

dark net market volume

Source: Soska and Christin

During the same time frame (most of 2014), volume at payment processors such as BitPay and Coinbase were relatively flat with a few outliers during days with speculative and media frenzies as well as ‘Bitcoin Black Friday.’

As of today it is unclear what activity is the “biggest” — we would need to aggregate all of the dark net marketplaces and compare that with the reused addresses BitPay uses plus the self-disclosed numbers from Coinbase.

In the chart above, illustrating off-chain activity between August 14, 2014 – August 13, 2015, it is also unclear from Coinbase’s number what a “off-chain” transaction is.  Is it only related to merchant activity?  Does it also include movement between users or with cold storage as well?

Therefore based on past historical trends (above) I do not think that “clearnet” or on-chain “licit” activity outweighs illicit transactions.  One darknet market alone — Evolution — processed roughly the same amount of bitcoins last year as BitPay did.

Q: Do you think consumer volumes will change significantly in the next year – what would it take for this to happen?

A: It depends on what we mean by “consumer volume.”  If this includes both illicit and licit activity, sure, maybe.  If it also includes “off-chain” transactions, then yes, probably as well.  But it is important to note you are not using Bitcoin (or bitcoin) when you go off-chain.  The transparency and auditability trail disappears and a user is now reliant on a trusted third party — many of whom in the “Bitcoin space” have a checkered past on financial controls — to protect and secure your privkeys.

I think we have already largely witnessed what the “killer apps” that incentivize increased usage of on-chain bitcoin activity are: censorship-resistant activities.

If the goal of Bitcoin was to provide a censorship-resistant payment processing platform (the word “payment” appears 12 times in the white paper) then it is safe to say that: dark net markets, casino sites, ransomware and other activities that require censorship-resistance and cannot be globally accessed on permissioned networks will continue to attract users towards it.1

It is my view that the following two laws explain the on-chain phenomenon we observe on a regular basis.  Folk law: “Anything that needs censorship-resistance will gravitate towards censorship-resistant systems.”  In contrast is Sams’ law: “Anything that doesn’t need censorship-resistance will gravitate towards non censorship-resistant systems.”

As far as other “apps” such as sites like Zapchain, while boasting growth numbers, appears to recreate a trusted third party system (e.g., facilitate deposit-taking and MSB activities like other hosted wallets) all while simultaneously scraping content from other sites.2

So Buzzfeed, but with bitcoins.

Does it have legs?  Porter Bibb would probably say no.

In closing, one last comment related to real on-chain trade (as opposed to spam-like “long-chain transactions“) is the recent announcement / non-announcement from TigerDirect.  Jorge Stolfi, a computer science professor in Brazil, probably best summarized the nebulous responses from the electronic retailer:

  • How much have you been making in bitcoin payments? “While Expedia has seen a decrease in bitcoin payments, TigerDirect shared a different story.”
  • How many customers are paying with bitcoin? “46 percent of customers purchasing with bitcoin are new users”
  • Sorry, how much did you say you made with bitcoin payments? “the average order placed with bitcoin is 30 percent larger than the average order.”
  • Yes, but, how much are you selling with bitcoin? “TigerDirect sees the highest volume of bitcoin orders during periods of volatility for bitcoin price.”
  • We would really like to know how much, roughly, you are getting from bitcoin payments. “TigerDirect has still seen consistent bitcoin transaction volume.”
  1. According to Kotov and Rajpal, bitcoins are now the most common method of payment for ransomware.  See Understanding Crypto-Ransomware. []
  2. Zapchain uses Coinbase as a wallet provider for deposits — the tipping of transactions is done via via BlockCypher. []

Buckets of Permissioned, Permissionless, and Permissioned Permissionlessness Ledgers

A few hours ago I gave the following presentation to Infosys / Finacle in Mysore, India with the Blockchain University team.  All views and opinions are my own and do not represent those of either organization.

The Distributed Ledger Landscape: Who is developing shared, replicated ledgers and why

Earlier today I gave a presentation for Blockchain University hosted at PricewaterhouseCoopers in San Francisco.  It covers the different startups developing permissioned ledgers, the use-cases they are looking at and the reasons for why permissionless systems are currently inadequate to fulfill similar business requirements.

A gift card economy: breaking down BitPay’s numbers

Two days ago BitPay, the largest payment processor in the cryptocurrency space, published a new infographic filled with a number of new stats.

BitPay claims that in 2014:

  • $158,800,000 total value processed (an increase from $107 million in 2013)
  • 563,568 total number of transactions (an increase from 209,420 in 2013)
  • $281 average order value (a decline from $513 in 2013)

They also state that there is a reason for the decline in average order value:

This number is dropping as adoption increases and Bitcoin moves from an investment commodity to a payment method.

At best that is just a guess.  While it is neat that BitPay is one of a very few companies in this space willing to publicly release some numbers, we cannot determine what the actual cause for this trend with the available information.  Correlation (drop in prices or average order value) does not mean the real cause is payment adoption.

correlation

Source: XKCD

According to Jonathan Levin, head of business development at Chainalysis:

The fall in the average order value seems likely to be attributed to the increase in difficulty and the fall in the number of home miners.

Unless they publish weekly or monthly bar charts (which they used to), or what merchants are their largest by volume each week, it is unclear what could be skewing that number (e.g., large block sales from miners in 2013 and 2014?).

For instance, in December 2013, the chart below was published on the official BitPay blog (it has since been removed):

bitpay 2013The spike in transactions during November 2013 is probably related to two things:

  1. the Bitcoin Black Friday marketing event
  2. simultaneous run-up in prices during the contemporary bubble that early adopters / miners were likely able to capitalize off of by exiting positions

Are there any other numbers?

bitpay 2014Above is the last known public chart of BitPay transaction volume.  The dates on the chart corresponds with April 2013 – March 2014 and the image comes from the Cryptolina conference held in August 2014.

Although the quality is a little fuzzy, transaction volume appears to have reached around 70,000 in March 2014.  Token prices during March ranged from approximately $450 – $650 which they likely weighted and multiplied by the total amount of bitcoins received each day to come up with a figure of $1 million processed each day (note: at the end of May 2014, BitPay announced it was processing $1 million in bitcoins a day).

Yet as we shall see, in terms of fiat transaction equivalent, there is less than half as much today as there was last year.

bitpay chartThe chart above is part of the original BitPay infographic released on Wednesday.

In terms of transaction volume, bitcoin mining alone accounts for the next 4 largest segments combined.  For those who believe this will change in the future, recall that if mining somehow becomes cheaper then it is also cheaper to attack the network.  So as long as there are rents to be extracted, miners will continue to fight for and bid up the slivers of seigniorage up to where the marginal cost eventually reaches the marginal value of the token; and that translates into continuous streams of mining revenue (not necessarily economic profit) that are converted into fiat to pay for land, labor, taxes and electricity.

Furthemore, because bitcoin mining is not on the top 5 list of in terms of number of transactions this likely means that the miners that do use BitPay likely sell large blocks and are therefore large manufacturers or farms or both (and of those miners, most probably come from large entities such as BFL and KnC paying their utility bills).

The second chart to the right states that gift cards as a class represent the lion share for number of transactions processed.  This is actually kind of humorous and unhumorous.  What this means is that the majority of BitPay users (and probably bitcoin users in general) are not doing economic calculation in BTC (the unit of account) but instead some kind of fiat.  And to do so, they are going through a Rube Goldberg-like process to convert bitcoins into fiat-based utility.

This is mostly borne out through a roundabout process such as bitcoins sent to Gyft -> Gamestop -> ShellCard (the gas company).  Or Gyft->Amazon->Purse.io.

What are other motivations?  Some users, based on social media posts, claim to do this in order to reduce identification (KYC) paper trails so taxes will not have to be declared and sometimes to take part in illicit trade (e.g., sell these gift cards at a discount for actual cash for illicit wares).

Based on their chart, roughly $345,000 of merchant activity is processed on a daily basis.  Of that, $277,000 comes from precious metals and bitcoin mining.  The remaining  $68,000 is for unidentified e-commerce, IT services and travel.  Or in other words, nearly 80% of bitcoins processed by BitPay in 2014 went to paying for security (mining) and buying (or selling) gold and silver.

As I have written about previously, that for roughly every $1 spent on security (via mining), there was roughly $1 spent on actual retail commerce which translates into a quantitatively (not qualitatively) oversecured network.1 But based on this new data: more capital is probably being spent securing the network than retail commerce by a factor of at least 2x.2

Recall that bitcoin mining represents just under half of all transaction volume processed by BitPay, and BitPay itself has about 1/3 to 1/2 of the global market share for payment processing, so it is probably a good sample size of world wide non-darkmarket “activity.”

What about others?

The second largest payment processor is Coinbase.  And based on their self-reported transaction volume (below), the “off-chain” trend over the past year is similar to what BitPay processed:

coinbase chartAs described in Wallet Growth, approximately six months ago, in October 2014, Brian Armstrong and Fred Ehrsam, co-founders of Coinbase, did a reddit AMA.  At the 31:56 minute mark (video), Ehrsam discussed merchant flows:

One other thing I’ve had some people ask me IRL and I’ve seen on reddit occasionally too, is this concept of more merchants coming on board in bitcoin and that causing selling pressure, or the price to go down. [Coinbase is] one of the largest merchant processors, I really don’t think that is true.  Well one, the volumes that merchants are processing aren’t negligible but they’re not super high especially when compared to people who are kind of buying and selling bitcoin.  Like the trend is going in the right direction there but in absolute terms that’s still true.  So I think that is largely a myth.

Perhaps those volumes will change, but according to the chart above, that does not appear to be the case.

And as discussed in Slicing Data, the noticeable pattern of higher activity on weekdays versus the weekend is apparent irrespective of holidays with Coinbase too. Consequently, on most days these self-reported numbers comprise between 3-5% of the total transactions on the Bitcoin blockchain.  However, as Jonathan Levin, has pointed out, it is not clear from these numbers alone are or what they refer to: Coinbase user to user, user to merchant, and possible user wallet to user vault?

What does this mean for BitPay?

BitPay has three tiers of customer pricing.  The first plan is free, the second charges $300 for the first month and the third is for enterprise clients.  They claim that there are no transaction fees at all.

While they probably do sign up customers on their 2nd and 3rd tier, it is unclear how much.  Speculatively it may not be very much due to the low transaction volumes overall (e.g., why would Microsoft pay more in customer service than they generate in actual revenue?).  Thus their margins may be razor thin at ~1% which translates to roughly $1.5 million in annual revenue (it has to be below 2-3% otherwise merchants would not perceive an advantage for using their service).  BitPay also charges (collects) a spread through a process called the BitPay Best Bid (BBB) rate.

Based on the current head count of between 70-100 people (9 were probably laid off after the “Bitbowl“), it may be the case that the revenue generated annually covers the labor costs for just one or two months.  Perhaps this will change if prices rebound and/or if volume increases (recall that payment processors sometimes have to put coins on their books if they cannot find a counterparty to sell to in the time frame so in the likely event that BitPay holds coins on their books, they can gain or lose through forex movements).

bitpay twitterOn this point, four months ago I was involved in a mini-twitter debate with Jeff Garzik (a developer with BitPay) and Antonis Polemitis (an investor with Ledra Capital).  It partially centered around some of the findings that Jorge Stolfi (a computer science professor in Brazil) posted the previous month regarding BitPay’s transaction volume.

As discussed on Twitter, their burn rate on labor — as in almost all startups — is most certainly higher than the revenue they generate.  This should not be seen as “picking on BitPay” (because virtually every US-based VC-backed Bitcoin-related startup is in the same boat, see Buttercoin and probably ChangeTip) but they probably are not generating much additional revenue from “monthly SaaS subscriptions and payroll API customers.”

How do we know this?  Again, why would Demandware pay more for a SaaS subscription than they generate via revenue?  Altruism?  Perhaps a few do (like NewEgg or TigerDirect) but even if 1,000 customers paid $300 a month, that is still just $300,000 a month far less than the $1 million (speculatively) needed to cover labor alone.

Clustering

I contacted Fabio Federici, co-founder of Coinalytics which specializes in building data intelligence tools to analyze activities on the blockchain.  Using data from WalletExplorer.com (which identifies reused addresses of payment processors, pools, gambling services and such), his team was able to create visual aides covering BitPay.

It bears mentioning that there is a ~10% discrepancy between the WalletExplorer numbers and BitPay and this is likely a result of the clustering heuristic (by WalletExplorer) which will not give 100% coverage and is not dishonesty from BitPay (e.g., WalletExplorer data set identifies just over 600,000 transactions last year whereas BitPay cites roughly 650,000 transactions).

bitpay daily number of transactionsThe time frame for the chart above takes place between July 2, 2011 and April 13, 2015.  The chart visualizes the Daily Number of Transactions.  The green line is the important line as it represents the incoming transaction amount that BitPay receives each day.  It shows that aside from a brief outlier in the winter of 2014, volume has remained relatively flat at around 1,200 – 1,500 transactions per day for the past 15 months.Daily Volume Btc (2013-2015) [Log] xThe time frame for the log chart above is slightly shorter, between January 1, 2013 and February 28, 2015 (there is a strange drop starting in March that is likely a problem with the clustering heuristic, so it was removed).  The chart visualizes the Daily Volume of bitcoin that BitPay receives.  The green line is the important line as it represents the aggregate of how many bitcoins BitPay received each day.  While there are some days where the total reaches to 8,000 or even 9,000 bitcoins, these are outliers.  Conversely some slower days reach around 500 bitcoins per day.  On average, between January 1, 2013 and February 28, 2015, the daily amount is 1,138 bitcoins.

Other specific ranges:

  • Average February 2013 – February 2015 = 1,209 bitcoins daily
  • Average February 2014 – February 2015 = 850 bitcoins daily

One explanation for the discrepancy is that there is a large incoming transaction of 28,790 bitcoins on March 25, 2013 which skews the average in the first date range.  It the same day that the Cyprus international bailout was announced.  While this coincides with the ‘bull run’ in the spring of 2013, it is unclear from public data what this one sale may have been.  Looking at some other charts, at around that date roughly 52,694,515 bitcoin days were destroyed (BDD) and total output volume (TOV) was around 4 million (which is about 4x higher than today).  During this time frame fees to miners were also about 3x-4x higher than they are today.  And on this specific day, 159 bitcoins in fees were sent to miners, the fifth highest total ever.  While speculative it could have been an “early adopter” or even a company overseas cashing out (market price was around $73.60 per bitcoin on March 25, 2013).

Daily Number of Transactions (2013-2015) [Log] xThe log chart above visualizes the daily number of transactions for BitPay between January 1, 2013 and February 28, 2015.  The interesting phenomenon is the flip that occurred in the fall of 2014.  Whereas previously the number of outgoing transactions exceeded the internally held coins, in late September this appears to have changed.  It is unclear what the reason(s) for this is.  Perhaps more merchants decided to keep coins instead of exchanging for fiat.  Or perhaps due to the continued price decline, BitPay had to hold more coins on their balance sheet due to the inability to liquidate merchant requests fast enough (e.g., between August 1 – November 1, market prices declined from around $558 to $336 per bitcoin).

Other noticeable phenomenon on the green line above include a rapid run-up during the collapse of Mt. Gox in February 2014 and then later Bitcoin Black Friday followed by Cyber Monday in November 2014.

Why are there recognizable patterns for the green line in all of the charts?  Again, since the bulk of payments are related to mining, it is likely that miners sell blocks on a regular basis.  Denominated in USD, when paired up with bitcoin volume between February 2013 and February 2015, the plot would likely look like a left-modal bell curve.

Perspectives and conclusions

On average BitPay processed 1,544 transactions worth $435,068 per day in 2014.  Once mining and precious metals are removed, the BitPay “economy” involves $57.5 million per year.  Even if the full amount, $158 million, were classified as actual economic activity, it is less money than what Harvard Business School generates from selling case studies each year (~$200 million) or roughly the same amount that the University of Texas athletic department generates each year.

If Coinbase and the rest of the bitcoin-to-fiat merchant economy sees similar patterns of activity, that would mean that above-board economic “activity” may currently hover around $350 million a year.  This is just slightly more than venture capital was invested in the Bitcoin space last year (~$315 million) and roughly equivalent to the fund that Lux Capital raised last month for funding science-related startups.  For comparison, Guatemalan’s working abroad remitted more than $500 million back to their families in one month alone last year.

In terms of payments the competitive landscape for Bitcoinland is not just other cryptocurrencies but also incumbent payment providers and tech companies such as Google, Apple, Facebook and Microsoft (the latter has been collecting money transmitter licenses), each of which has launched or is planning to launch an integrated payments system.  Startups such as Venmo and Square, both of which were launched the same year as Bitcoin, have seen some actual traction.  For instance, in the forth quarter of 2014 Venmo payment volume came in just over $900 million, up from $700 million processed in the third quarter (Square Cash claims to have an annualized volume run rate of $1 billion).

And although it is not a completely fair comparison, Second Life from Linden Lab is still around “with 900,000 active users a month, who get payouts of $60 million in real-world money every year” (note: there is some debate over specific user numbers).

When mining payments are removed, Bitcoin, as an above-board economy, appears to generate less in return than the venture capital funds have gone into it (so far).  Perhaps this will change as more of the capital is deployed but it may be the case that Bitcoinland cannot securely grow exponentially (as the bullish narrative envisions) while maintaining a fixed amount of outputs.

In his recent conversation with International Business Times, Wouter Vonk, BitPay’s European marketing manager, described the trends from the infographic, stating:

As bitcoin becomes a more established technology, we expect to see more consumers using it. The investors are usually the first ones to hop on new technology, but as bitcoin circulates more, and as the amount of transactions increases, we should see bitcoin being used by more and more average consumers. We see bitcoin being used in emerging markets as a supplement to the current banking and monetary systems.  Bitcoin breaks down the barriers to financial tools that many people in emerging countries are facing.

Empirically, regarding “more consumer using it,” this does not seem to be true.  Nor is there evidence that bitcoin is circulating “more” — in fact, based on age of last use, more than 70% of coins have not moved in more than 6 months (slightly older figure).  And while cryptocurrencies may play a role in developing countries, so far there is little evidence this is actually occurring beyond talk at conferences.  Again, perhaps this will change as new data could reinforce Vonk’s narrative, but so far that is not the case.

For perspective I contacted Dave Hudson, proprietor of HashingIt, a leading network analysis site.  According to him:

One thing that I did notice is that their earlier “incoming” graphs all look highly correlated to the transaction volume in the Bitcoin network after long chains are removed.  This gets back to the usual Bitcoin transaction volume question of what’s really in a transaction and what’s change?  It seems their transaction volumes have really only crept up in the last 12 months, much slower than the rate of growth in transactions (or non-long-chain transactions) on the main network (increased competition?).

What does this look like?  The chart below measures Number of Transactions Excluding Chains Longer Than 10 between April 2013 – April 2015.

blockchain long chainsWhat are long chains again?  Rather than rehashing the entire paper, recall that in Slicing Data, it was observed that a significant fraction of total transaction volume on any given day was likely inflated through a variety of sources such as faucets, coin mixing and gambling.

As we can see above, while there is indeed an upward trend line over the past two years, it is clearly not growing exponentially but rather linearly, and particularly in spurts around “macro” events (e.g., bubble in late 2013 and collapse of Mt. Gox).

Based on the public data from address clustering, consumer adoption is empirically not growing near the same level as merchant adoption.  In fact, consumer adoption in terms of actual non-mining, retail-usage, has basically plateaued over the past year.  We know this is the case since merchants accepting bitcoin for payments has roughly quintupled over the same time frame (20,000 to 100,000) and includes several large marquis (such as Microsoft) yet without any surge in usage by bitcoin owners in aggregate.

Other companies that have actively promoted bitcoin for payments have likely also been impacted by sluggish sales.

For instance, in February 2015, Overstock.com (which has been using Coinbase as a payment processor for over a year) tried to obfuscate weak traction by using a strange method: measuring orders per 1 million residents.

overstock bitcoin

The top 3 were:

  • New Hampshire has a population of 1,326,813 and according to the chart above Overstock received 131 bitcoin orders per million residents.  This comes out to roughly 175 orders in 2014.
  • Utah has a population of 2,949,902 and according to the chart, Overstock received 89 bitcoin orders per million residents.  This comes out to roughly 270 orders in 2014.
  • Washington D.C. has a population of 658,893 and according tot he chart above Overstock received 85 bitcoin orders per million residents.  This comes out to roughly 56 orders in 2014 (although if the greater D.C. metro population was used, the order number would be about 9x larger).
  • Fighting for last place: Puerto Rico trounced Mississippi, which came in dead last.  Puerto Rico has a population of 3,667,084 and according to the report, Overstock received 12 bitcoin orders per million residents.  This comes out to about 44 orders in 2014.   In comparison, Mississippi, with a population of 2,994,079 had 8 order per million residents.  This comes to about 24 orders.

According to Overstock, in 2014 approximately 11,100 customers paid with bitcoin at both its US and international websites.  Altogether this represented roughly $3 million in sales which when coupled with low margin products (based on the top 10 list of things sold on Overstock) is an initiative that Stone Street Advisors labeled “distracting” (see slides 21, 32, 33, 37, 58).

In addition, since gift cards represent about 16% of all transactions processed by BitPay, they can be added to the list of non-negligible reasons for fluctuation in blockchain transaction volume.  That is to say, on any given day there are roughly 242 gift card related transactions through BitPay which should appears on the blockchain.  This is about the same amount of Counterparty transactions that may take place on a slow day.

Thus, as discussed in Slicing Data, the daily components of blockchain transactions are likely: faucet outputs (which may be “long chains”), mining rewards, some retail activity, coin mixing, gambling, watermarked assets (e.g., Counterparty, Mastercoin), P2SH, movement to ‘change’ addresses, wallet shuffling and now gift cards.

While their new infographic does not come to any direct conclusions as to macro growth of Bitcoinland it is likely that there are still only a few profitable businesses and projects in the ecosystem and most are unrelated to Bitcoin itself:

  • Fabrication plants such as TSMC and designers like Alchip
  • Utility companies (hydroelectric dams in Washington, coal power plants in Inner Mongolia)
  • Large mining farms with access to the newest ASIC batches reducing overall operating costs relative to marginal players (Bitfury in the Republic of Georgia)
  • Some mining pools (Organ sometimes has a break down of block makers)
  • Law firms (such as Perkins Coie)
  • Conference organizers such as Inside Bitcoins (but not The Bitcoin Foundation)
  • A handful of bitcoin-to-fiat exchanges (BTC-e, Bitfinex and a few others)
  • Scams (Moolah from Alex Green/Ryan Kennedy, GAW/PayCoin from Josh Garza, BFL, MyCoin and at least 42 others, more likely hundreds)
  • Botnet operators (botnet mining still exists, externalizing operating costs with “other people’s electricity”)
  • Ransomeware (CryptoLocker, KEYHolder, CryptoWall and a few dozen others)
  • Darknet Markets (Evolution “exit,” Sheep Marketplace “hack“; some low-hanging fruit exists for academics studying operators and providers that transitioned from Liberty Reserve to other DNMs, after it was shut down 2 years ago)

Perhaps all of this will change and this snapshot is “too early” as the bullish narrative claims.  Trends may change, no one has a crystal ball.

[Special thanks to CukeKing, Fabio Federici, Dave Hudson, Jonathan Levin and Pete Rizzo for their feedback and info]

  1. See Are there changes in the volume of retail transactions through Bitpay this past year?, Will colored coin extensibility throw a wrench into the automated information security costs of Bitcoin? and A brief history of Bitcoin “wallet” growth []
  2. In Chapter 14 in The Anatomy: “If the labor force of bitcoin is spending $10 million on protecting the network yet real commerce is only $30 million, this would be equivalent to a mall issuing 1 out of 3 customers a personal security detail to go shopping.  Or in other words it is, arguably, quantitatively oversecure (it is not qualitatively trustless as shown by the trifecta of DeepBit, BTC Guild and GHash.io).” []

Consensus-as-a-service: a brief report on the emergence of permissioned, distributed ledger systems

I have spent the past month compiling research that took place between August and the present day.  This was much more of a collaborative process than my previous publications as I had to talk with not just 8 geographically dispersed teams to find out what their approach was in this nascent field but also find out who is working on ideas that are closely related to these projects (as seen in Appendix A).

The culmination of this process can be found in this report: Permissioned distributed ledgers

Fortunately I had the help of not just astute practitioners in the industry who did the intellectual heavy lifting, but the resources and experience of the R3 CEV team where I am an advisor.

I think the three strongest areas are:

  • Richard Brown’s and Jo Lang’s description and visualization of smart contracts.  I loathe the term smart contracts (I prefer “banana” and Preston Byrne prefers “marmot”) and fortunately they distilled it to a level where many professionals can probably begin to understand it
  • Meher Roy’s excellent OSI-model for an “internet of money”
  • Robert Sams mental model of the core attributes of a permissioned distributed ledger

I think the weakest part is in the beginning of Section 8 regarding TCP/IP.  That is reflective of the fact that there is no perfect analogy because Bitcoin was designed to do many things that no other system does right now so there probably is no single apple’s to apple’s comparison.

While you do not need special internetcoins or fun buxx to use the internet (as it were), there is still a cost to someone to connect to the net.  So perhaps, the frictional differences between obtaining and securing an internet connection versus obtaining and securing a bitcoin at this time is probably something that should be highlighted more if the report is updated.

Wither Bitcoin?

For cryptocurrencies such as Bitcoin to do what it does best on its own terms, its competitive advantage lays with the native token and not representing real-world assets: its community needs to come to terms about what it is and is not good for.  Because of its inability to control off-chain assets its developers should stop promising that bitcoins — or metacoins and watermarked-coins that use Bitcoin as a transportation layer — as a panacea for managing off-chain assets, assets the network cannot control.  At most Bitcoin’s code base and node network operates as its own legal system for non-watermarked bitcoins.

Consequently, the advantage a cryptocurrency system has is endogenous enforcement of contractual terms — or as Taulant Ramabaja calls it: “fully blockchain endogenous state transition without any external dependencies.”  Or on-chain, dry code to dry code.

I wonder if someone in the future will call themselves a full “dry code” stack developer?

Consensus-as-a-service: a brief report on the emergence of permissioned, distributed ledger systems

Interview with core developer, Peter Todd

A new interview is up with IamSatoshi Network and Bitcoin core developer, Peter Todd.  While the first part explains the politics of getting code into (or out of the protocol) — which many enthusiasts gloss over — I especially found Peter’s discussions in Part 2 of germane interest due to the sky-is-falling on reddit surrounding Ghash.io the last few days.  Here is one such comment thread.  And I think this comment sums it up the best:

If a 51% can occur, all trust in bitcoin should be lost forever. An investment that relies on people begging random strangers on reddit not to ruin it every couple of weeks is not really something that seems like a great thing to pour actual money into, tbqh.

Tbqh means “to be quite honest” (that same user made other good points about this issue too).  And here is also another interesting subthread (mulligan for that decentralization + neutrality notion…).

Ghash.io is the largest mining pool on the Bitcoin network (it actually supports merged mining for Devcoin, Namecoin and Ixcoin as well) and it hit 48% of the network hashrate this past weekend (it is now 43%, see this chart).  Its parent operating company is Cex.io and the system is run in a cloudhashing manner — customers rent hashrate by purchasing contracts with bitcoin.  Interestingly enough, the cost of these contracts is now more than what you receive as a reward for hashing, leading to the joke that Ghash (and other such services) are pay-for faucets.  That is to say, faucets are way to distribute tokens, for free (usually by filling out some Captcha once a day).  Yet in this case, because of bitcoin volatility the past 5 months, users are actually paying to receive a minute amount of bitcoins — they might as well terminate their contracts and buy bitcoins on the open market.

One common refrain that some Bitcoin advocates say about mitigating 51% attacks is that hashers in mining pools can simply move and/or point their hashing equipment at another pool.  This may be possible in the “early” days of today, yet there are two problems as time goes on:

1) As we approach the top of the S-curve in ASIC tech improvements, mining farms (and pools) will gravitate to locations with the best and cheapest network and energy infrastructure.  This itself creates centralization risks that I and many others have written about.  If you are renting out equipment (hashing systems) from a cloud provider at one of these locations, you can no longer physically move the equipment to another farm and perhaps in some cases, you may not be able to direct your miner to other farms (there is one proposal by Greg Maxwell that hasn’t been reported on involving tamper resistant private keys physically built into the gear, but that’s a story for another post).

2) Block size increases.  In order to make the Bitcoin network more competitive as a payments and transportation network, there have been many proposals to increase the hard cap of 1 MB block sizes by several orders of magnitude.  To date however, the average block size is around 350 KB, with an average of 0.7 transactions per second — thus the need to increase it is low (primarily because few people actually use the chain for much activity such as commerce).  If block sizes are increased, without the use of something like tree chains, then centralization will occur because miners (and fully validating nodes) will need to pay for larger bandwidth options, larger hard drives, etc. which squeezes out marginal players.  This is a known issue but Peter Todd highlights this as a hurdle for hashers wanting to move to another pool for the same reasons mentioned in point #1.

Outside funding of cryptocurrency and Bitcoin startups

According to CB Insights, VCs spent $74 million across 40 BTC-related deals in 2013,  the two largest rounds were Coinbase ($25m) and Circle ($9m).

Despite the increased media attention, even if these numbers are repeated again this year this may not help boost the poor performance for VC funds as a whole.1 Even with the optimistic outlook many of the VC firms apparently now have, their actual results at ~6% per annum over the past decade have underperformed the Russel 2000.2

Why?  Some VCs not as nimble at feeling out business models with actual revenue generating capabilities as many angel investors are.

Changes over four decades

Consistent with secular theme of ubiquitous adoption of open source software as well as cloud computing that has lowered the cost of developing software and more importantly the costs associated with launching new companies, so too has this trend lowered the threshold for tech investments.  Where previously the funding of start-ups was limited to deep-pocketed professional investors, namely VCs, the deflationary landscape has increasingly enabled greater numbers of individual investors, angels to compete in funding environment.

The new class of angel investors is more astute than the passive and non-tech-savvy high net worth investor of yesteryear.  Increasingly, angel investors today have deep domain experience.  Many have worked in the sector that they are funding, are entrepreneurs and experienced operators themselves and visionary at feeling out new business and innovative trends.  The historical barrier to entry for angel investing is one of risk given the magnitude of investment commitment.  With lower costs of starting businesses, this hurdle is largely gone.  Smart angels with deep operational domain expertise is disruptive to the traditional VC universe.  They may be better attuned and friendlier with terms that are less predatory than the historical VC norm.

This is not to say that VCs will not flourish once again, however as it stands most angels began as entrepreneurs and learned how to generate sales and revenue first hand.  Furthermore, as noted above, over the past decade technological costs that have driven down expenses.  For example, relatively cheap cloud services like github and Compute Engine provide services (CaaS, SaaS and IaaS) that allow many tech start-ups to be leaner than before in terms of what funding they require to cover operating costs.  On top of this are better organized angels who now have an entire ecosystem of choices to fund through such as AngelList, 500 Startups and Y Combinator.  In fact, over the past six months, BitAngels.co have invested $7 million in 12 crypto projects globally.

Another way that cryptocurrency-related startups are being funded through are crowdfunded IPOs.  This includes Mastercoin, which raised $5 million in part by 4,700 bitcoins from “investors.”3  NextCoin (Nxt) and the upcoming Ethereum IPO have also included raising funds through bitcoin transfers.  While I am not necessarily endorsing any of these particular fundraising models, this illustrates how small (and perhaps large) development teams can financially cover costs without seed funding by VCs.

See also: MoneyTree Report from PricewaterhouseCoopers and the every-growing list of funded Bitcoin companies listed on CrunchBase

[Special thanks to DA for his comments and feedback.]

  1. Kauffman Foundation Bashes VCs For Poor Performance, Urges LPs To Take Charge from The Wall Street Journal and Most venture capital funds lose money from CNN|Fortune []
  2. Venture capital kingpin Kleiner Perkins acknowledges weak results from Reuters []
  3. Backed by $5 Million in Funding (4,700 BTC), Mastercoin Is Building a Flexible, New Layer of Money on Bitcoin from MarketWired []

Evolution of the cloud

Spent several days earlier this week with some brilliant software engineers who not only were domain experts but were very articulate about topics beyond the sci-tech world.  I posted a couple of tweets (here and here).  I’d like to thank Matthew Wilson for arranging the brainstorming sessions as well as Patrick Michaud, Larry Wall, Jonathan Worthington and Ingy for their hard work and creative collaboration.

Some of the topics and projects we discussed:

  • Firebase
  • Hadoop ecosystem
  • CaaS/SaaS/PaaS/IaaS (OpenStack, Docker, CloudFoundry, Stackato)
  • Intentional Software
  • Semantic Web, Programmable Web
  • Git
  • Domain-driven paradigm (Eclipse Xtext/DSLT, OMeta, Colm)
  • Joyent Manta
  • Rackspace ZeroVM
  • Meteor
  • Reactive paradigm
  • Cloud Haskell, Persistent Haskell

For those interested, if you really want to know about the hottest trends and innovations in software, be sure to look at the upcoming FOSDEM conference schedule.

Google Reader: An end of an era?

Midway in pursuing my grad studies years ago, a friend of mine, Michael Ewens, convinced me to switch from Bloglines to Google Reader.

What are those?  They are news aggregators that use syndication feeds based on a couple of popular formats: RSS and Atom.  Back in 2003 I wrote a lengthy series of posts regarding the various strains of RSS for a website that no longer exists (and its url is currently being squatted upon by a Eastern European malware owner so I won’t link to it right now).  While Netscape created the first version of RSS, it was further enhanced by Dave Winer over at Radio UserLand and then yet another fork was created in part by the late Aaron Swartz called Atom.  All are based on XML and each has the potential to tap into the ontological web.

While reading social media feeds from Twitter, Facebook, LinkedIn and their Chinese equivalents is very popular for both power users and the average Joe (or Zhou) alike, RSS/Atom is still a widely used syndication/aggregation method for millions of readers, including myself.  In fact, I prefer not to scan through the thousands of Weibo posts or Facebook smack downs to find links of information.  Opinions, sure, but actual data and original content that is longer than 140 character sound bites — traditional websites is where that information is still at, not behind subscription-only or friend-only silos.  While I personally am a proponent of the Open Access/Open content (hence the reason all of my writings are CC licensed), in practice it appears that the trend away from information silos that began in the ’90s with the original hobbyist intertubes has done a U-turn back into a new form of walled gardens (social media sites).  And while some disdain this trend, it would be fallacious to say whether this phenomenon is either good or bad because it is based solely on user subjective preferences (if you do not like AOL in 1994 for its “walled garden” in terms of accessing sites outside of the AOL ecosystem, no one is forcing you to subscribe to their service just like no one is forcing you to use FB today).

With that said, July 1, 2013 marks the end of a great service that Google provided in the form of Google Reader.  While its users were all freeloaders (there was no monetization or monthly subscription costs to it), when Google announced it was ending the service several months back, among the weeping and gnashing of teeth, one of the claims that I saw posted several times on social media sites is: RSS is dead.  Why was RSS dead?  Because it purportedly has no roadmap or development.

While there are many reasons to end the Google Reader service (such as the capital costs of maintaining it, for free to end users and how it is apparently hard to integrate it into Google+ due to licensing/copyright issues), this particular argument put forth above seems like a non sequitur.  RSS/Atom are not programming languages, they are not operating systems, they are not SDKs or APIs.

Among others, one objective of RSS/Atom was to help make it easier for machine-based solutions to grab the content from your site and allow other machine-based technologies (aggregators) to translate the code into something readable and organized to humans (and eventually AI itself).  It does that and it does that efficiently.  Whether or not it is effective is debatable as the duplication issues are related to an aggregation itself, not the XML code defining parameters in RSS/Atom.  This type of service can and will still be done so as long as sites still create and support the feeds, which I suspect will continue for many more moons — unless it is replaced by something technically superior.  Like what?  Perhaps information providers such as Reuters or Bloomberg (which most associate with news broadcasting but have huge budgets and teams working towards information processing) may develop a syndication method that satiates and unmet need.  Or maybe RSS and Atom are good enough for content producers and consumers for decades to come.

What solutions are there for news junkies to continue their habit?  Bloglines is still around, but slow (at least for me).  Digg released theirs, but it is inaccessible here in China without a VPN (it times out over and over).  Feedly doesn’t automatically insert the url of the articles when you email them.  The Old Reader has similar issues.  And AOL surprisingly has a reader now, one that I’m now using, that looks and feels snappy — but when you want to email the story to someone it opens up Outlook by default (I put in a request to have that changed to other email addresses and received a response from their dev team that a future feature is in the works to change this).

So basically, nothing matches the current form of Google’s own solution.  It is their service, so of course they have every right to close it down.  However, it will probably not push the millions of users towards Google+ which was managements original (desired) intention.  Until social media sites allow for integration of RSS/Atom, then power users will continue to find solutions to their information needs.

As an aside, to give readers an idea of how often I used Google Reader, below is a snapshot from the statistics page today.  On average, about 225-250 stories are aggregated through all of the feeds each weekday (weekends oddly enough have relatively little published), perhaps 15% of the stories are duplicates (especially the science/tech sites).  I dislike posting stories on FB or Twitter unless they are very important (but obviously I’m in a small minority) and consequently enjoy emailing them to friends, family and colleagues (hence the 300+ emails this past month).  Note: “clicked” means a user clicked the url in the headline of the article, usually that specific url is a Feed Burner link (called “feedproxy”).  Unfortunately, here in China, those url’s are blocked by the GFW and clicking it kills the link (one last tangent, it is because of Google Reader that many blocked stories are able to get past the GFW here sans a VPN).  Fortunately most sites like io9 or Slashdot have a “Read more here” link which is what I click (I am unaware of statistics that say which specific link is more prevalent to be clicked).

Long live, RSS and long live Google Reader!

Update: be sure to read Lockdown for more details and analysis

google reader stats

Are MOOCs a solution for the skillset mismatch?

While few people have a desire to be unemployed (or unemployable) the current higher education system in China has a number of issues in that many of the programs students have enrolled and graduate from do not prepare many of them for the labor market.  For example, roughly 6.99 million students will graduate from a Chinese college this summer (exams are typically held at the end of June, early July) yet in large cities like Guangdong, Beijing or Shanghai, only ~30% or so of new graduates have signed contracts for employment.

This is not a new issue or discovery, in fact, I wrote about it last year (as did the WSJ).  In Chapter 9 I discuss several of the opportunities that comes from this skillset mismatch, namely the need for retraining — some of which may take place online.  Massive open online courses (MOOCs) could be one solution.

Below is a very interesting, very concise write-up of the current problem as shown at a recently held Shanghai job fair, where neither candidate nor employer is incentivized by the other.  From MarketPlace education:

Hundreds of HR managers carefully eye prospective employees who, resumes in hand, crowd the floor at a Shanghai job fair.

Here’s the problem: neither group is interested in each other.

Nicole Li is looking to hire college graduates for her property management company. “We need technicians to fix software problems, but college grads don’t have these skills,” says Li, frowning. “We need people for exhibitions who can do presentations in English, but they can’t do that, either.”

Li needs to hire people for 60 high-skilled jobs. She says among the thousands of candidates here today, she’ll be lucky if she finds one.

Tong Huiqin comes to this job fair every Friday. He graduated from the Shanghai Finance University six years ago. Since then, he’s jumped from one job to the next. “It isn’t hard to find a job,” says Tong.  “It’s hard to find the right job.”

He’s worked as a supervisor for a bunch of companies, but hasn’t found the right fit. “You could have five hundred graduates and five hundred job openings here, and none of them would match up,” he says.

Tong blames Chinese universities. He says they need to do a better job at preparing people for the country’s rapidly changing labor market. Xiong Bingqi is the deputy dean of the 21st Century Education Research Institute, a nonprofit think tank in Beijing. “The scale of China’s higher education system has developed so fast that we’re failing to produce college graduates with the right skills for the jobs that are out there,” says Xiong.

For those with means, that’s meant sending your college-age children instead to universities in the U.S., Australia, or Europe. But most young Chinese can’t afford that, so they’re stuck in a Chinese university. And after they graduate — according to a recent state survey — their unemployment rate is four times higher than for those who didn’t get past elementary school.

Inside the job fair, young graduates linger in front of a booth for Bao Steel, China’s largest steel manufacturer. A big sign says that people from parts of Sichuan, Henan, Anhui, and Hunan are not allowed to apply. A guy applying for a job says people from those provinces can’t be trusted. It’s sort of like a booth at a New York job fair banning applicants from, say, Minnesota, Wisconsin and North Dakota. But this is typical in China, where even state-owned enterprises don’t bother to hide their discrimination.

At a neighboring booth, Jason Zhang is hiring people to work at a chain of nightclubs. He doesn’t care where his job candidates are from. He’s more concerned whether they’re willing to work. “I think today’s graduates are less appealing than people who were born in the ’70s and ’80s,” says Zhang. “They tend to be overly confident and they don’t want to work very hard.”

I turn around and ask 22-year-old Wang Qianmin, who’s about to graduate from Shanghai Normal University with a teaching degree, what she’s looking for at the job fair. “I don’t know,” she says with a pout. “Most of the jobs here aren’t really interesting. I’m looking for a company that’ll give me a high salary, money for meals and that’ll pay my rent — a place where the working hours aren’t too long.”

Wang says she wants to be a teacher. Or maybe a wedding planner.

She can’t decide.

Jason Zhang, the recruiter who has years of experience hiring people, rolls his eyes at this type of candidate. “Chinese college graduates these days think they’re really special,” he says with a smile. “The problem is — they’re the only ones who think that.”

Zhang says Wang and many others in China’s class of 2013 will go all summer thinking they’ve got lots of options, and will probably end up unemployed.

Stat of the day: internet speeds in China

As mentioned in Chapter 9, according to their Q3 2012 speed survey, ChinaCache, the largest domestic content delivery network (CDN), notes that while the overall speeds are a little slower than previous speed rankings, Shanghai currently leads the country in average speeds at roughly 3.44 Mb/s and Beijing is 10th at around 2.5 Mb/s.12 Akamai Technologies (a global  content delivery network provider) ranked China’s average internet-connection speed at 94th globally, at 1.6 Mb/s.3

Here is a new estimate from Qihoo 360 (in mb/s):

china-internet-speed-province-2013

Via Tech In Asia

  1. ChinaCache Releases Third Quarter 2012 China Internet Connection Speed Rankings from China Web Report []
  2. For comparison, the average download bandwidth in the US is 11.6 Mb/s.  See International Broadband Data Report (Third) from the Federal Communications Commission []
  3. China’s ‘Wall’ Hits Business from The Wall Street JournalI []

The art of cui bono and frowning upon conclusions

MG, an acquaintance, writes:

Tim, the reality is that China remains a society held captive to the Chinese Communist Party, who is committed to stealing intellectual property from American entrepreneurs and companies. Doing business with the Chinese is extremely costly and a strategic mistake for most US companies.  My comments are directed to the broader issues of doing business in China and the Chinese government’s posture on its domestic economic policies, its foreign trade practices, and its unprecedented role in global intellectual property theft. These issues have been the subject of a great deal of analysis here in Washington DC (some of which we’ve engaged in).

Ignoring the agitprop boilerplate about the Party (it exists, it will continue to exist), I mention some of those IP issues in a couple different places and explain to the reader that they should talk with an experienced attorney and IT security specialist before setting up shop on the mainland: see Chapter 10 (legal services) and Chapter 13 (internet security).  In fact, there are several cases discussed in detail in Chapter 13 but rather than rehashing those statements, MG raises a problematic issue here: financial disclosure.

For instance, this past week Bloomberg published a story entitled, “Cybersecurity Lobby Surges as Congress Considers New Laws.”  While cybersecurity is a real, present danger for every firm in any country there is a Latin phrase that summarizes the conflict of interest in MG’s position: cui bono (who benefits).  As Bloomberg notes:

There were 513 filings by consultants and companies to press Congress on cybersecurity by the end of 2012, up 85 percent from 2011 and almost three times as many as in 2010, according to U.S. Senate filings. Twelve firms have submitted new registrations this year on behalf of companies including Google Inc. (GOOG)’s Motorola Mobility unit, Symantec Corp. (SYMC), United Parcel Service Inc. (UPS) and Ericsson Inc., the U.S. subsidiary of Stockholm-based Telefonaktiebolaget LM Ericsson.

“Cybersecurity is a lobbyist’s dream,” Rogan Kersh, provost at Wake Forest University in Winston-Salem, North Carolina, who researches political influence, said in an interview.

How do we know there is a conflict of interest in this segment?  CISPA, or Cyber Intelligence Sharing and Protection Act, is a proposed law which would enable government agencies to monitor and share private electronic communication (similar to SOPA).  This past week, Representative Mike Rogers, Chairman of the House Intelligence Committee accidentally tweeted and then deleted that the “House Intelligence Committee received 15 times more from pro-CISPA groups than anti-CISPA orgs.”  This is another example of cui bono as the same organizations lobbying for the bill are the same ones that will financially gain if it is passed.  Similarly, MG works at a firm that gains financially due to the hype and theater surrounding this issue.

This is not to say that hacking and cybersecurity are not real problems that firms and entrepreneurs should ignore.  Rather consider what General Electric Vice Chairman, John Rice recently said, “Despite hacking and other issues in China, foreign companies need to be there, due to the country’s potential as the world’s biggest marketplace.  The greater risk lies in staying away.”

Thus it comes down to who you want to trust: MG, a contractor who financially gains from hyping a purported threat or John Rice, who is willing to weigh risks and potentially capitalize off knowledge and technology arbitrage (e.g., bringing specific technological know-how to .the mainland).

Experienced expat response

Last night I spoke with David Veksler, CEO of CryptAByte and a cybersecurity professional in China who I interviewed for Chapter 13.  He told me in an email exchange that:

“Doing business in China is no doubt risky.  But betting your company’s future on a stagnating domestic markets is risky too.  Every business must balance the risk of IP theft and broken contracts against opportunities from the world’s biggest consumer market and low cost suppliers.

While Chinese companies are well known for intellectual property theft, we must keep in mind that there is no monolithic entity, even within the Party.  There are many competing interests, and each case is different.  By doing their due diligence, it is quite possible to protect one’s interests and secrets while being successful in China.  To claim otherwise is to say that information security is futile.

In the long run, industrial secrets may be impossible to keep, but in any dynamic industry competitive advantage and profits are made in the near future.  Businesses that substitute innovation with copying the competition will not succeed in a competitive market.”

Unconventional analysis

I also reached out and spoke with a patent attorney about the issues MG raises and spoke with Stephan Kinsella (the same attorney in Chapter 7).  Here is what Kinsella wrote in an email exchange yesterday:

“There may be a grain of truth in the complaint about Chinese companies not respecting Americans’ IP, insofar as some Chinese companies seem less willing or able to abide by contractual restrictions designed to keep certain information proprietary. But this danger exists for businesses in all societies, even in the US, especially as employee mobility increases and employees move back and forth between employers, taking ideas with them.

But the bulk of the complaint seems to be focused on American-style IP law, namely patent and copyright, and to assume that patent and copyright are legitimate types of property rights. This is what permits the author to refer to competition by Chinese companies as “stealing”: he has accepted the IP mentality. But copying and emulating others in the process of competing with them is part of the free market. Patent and copyright are anti-property, anti-market systems designed to protect companies from competition.

Thus, the author here is siding with protectionism and against the market, and Chinese companies who compete with American companies because they have relatively fewer IP laws to shackle them, more more capitalistic. It is true that some American companies whose business model depends on the protection from competition afforded to them by IP law would prefer that other countries, like China, also offer them protections from competition. But this does not mean that such laws make sense.”

Secret central plans

Tangentially related to cybersecurity and hacking is the myth of the secret plan hidden somewhere.  Or rather, the myth that hackers in other countries have of the US — that there is a super secret master plan that directs all activities of the federal government.  This was humorously brushed aside last month by Ezra Klein at the Washington Post, who explained that:

I almost feel bad for the Chinese hackers. Imagine the junior analysts tasked with picking through the terabytes of e-mails from every low-rent think tank in Washington, trying to figure out what matters and what doesn’t, trying to make everything fit a pattern. Imagine all the spurious connections they’re drawing, all the fundraising bluster they’re taking as fact, all the black humor they’re reading as straight description, all the mundane organizational chatter they’re reading.

This weekend Reuters published a story about a cybersecurity program at Jiao Tong university with PLA connections (the same PLA unit in the Mandiat report that made headlines earlier this year).  While formal ties with this college may exist, to be even handed, we should keep in mind that Stuxnet and Flame were designed by the NSA and Israel, to take out Siemens-designed software systems located in Iran.1 However, this raises a number of questions (e.g., when is state-sponsored cyber espionage justified) that detract from the immediate issue at hand.

Some cybersecurity threats are real, others imagined.  Before investing in any domicile be sure to do your due diligence for security threats (even the old-fashioned variety) and speak to a lawyer or risk assessment expert to qualify potential threats.  For more on this hype and cybersecurity, be sure to follow Techdirt and Bruce Schneier.

  1. See Flame and Stuxnet Cousin Targets Lebanese Bank Customers, Carries Mysterious Payload from Wired and Resource 207: Kaspersky Lab Research Proves that Stuxnet and Flame Developers are Connected from Kaspersky Lab []

The market for massive open online courses in China

A quick update and illustration regarding online education in Chapter 9.  South China Morning Post published an article this morning about massive open online courses (MOOCs) which are increasingly popular in China.

If you are from North America you have probably heard about a couple MOOCs including edX and Udacity.  The SCMP piece noted that Chinese University of Hong Kong is “committing” (does that mean developing?) some classes to the Coursera platform, which was founded by two Stanford professors last year.  For reference, Khan Academy is probably the best known as it has served more than 200 million classes and Wendy Bao cited it specifically in her interview as a future model for education in Chapter 9.

While millions of students are enrolled in these classes globally, it is difficult to track and measure the Chinese matriculation in terms of marketshare because, unfortunately the GFW (Great Firewall) blocks most of these sites.

What are the opportunities then?  What can be done about this?  Are there any domestic startups? From the SCMP piece:

NetEase, a leading China-based internet technology company, said that the number of open course subjects had doubled from last year to the present 12,000. It reported 4.6 million subscribers via PCs and smart phones and about one million student visits a day.

However, John Zhang, co-founder of mainland-based online career and recruitment consultancy Guolairen.com said open courses delivered by internet portals simply served as a “media platform”.

In contrast, he said Mooc platforms provided students with a classroom-setting learning experience and opportunities to receive credit and recognition for their work. Already five subjects provided via Coursera had been recommended by the American Council on Education.

“This is a moment of historic significance in higher education because Mooc is not just a complement to existing higher learning, it opens up a new era of online teaching,” Zhang said.

Guolairen.com which unveiled its own Mooc platform in a low-key launch in October, the first on the mainland, has recorded 35,000 enrolments for 200 courses from 100,000 sign-ups.

Zhang said the company would invest US$30 million in its Mooc platform over the next three years, in co-operation with leading international universities such as Harvard, Columbia University and MIT, to provide a specialised platform for career-minded Chinese youth.

Potentially large numbers

Upon looking at the Guolairen ( 过来人) site right now, it is not very obvious that MOOC is a central part of this portal.  In fact, it is currently geared towards job-seekers, primarily for new graduates (including info about cover letters and interviews).  Thus, if 过来人 is the go-to model, it may be too early to declare any domestic market leader yet.

Again, as I remind readers repeatedly (for good reason), it is incredibly easy to get caught up when big numbers are thrown around or when it comes to potential customers in the middle kingdom.

What are some actual estimates for clientele in this segment?  At the tail end of Chapter 9 I note the following data:

Yet for those willing to face these technical challenges, the financial rewards could be lucrative.  According to one recent estimate, up to 380 million people in China will “need high-quality education and training resources across the country” from 2012 to 2017.1 And a large percentage (~30%) of these people are expected to utilize online services and tools, creating a potential market worth an estimated $11 billion in revenue.  However, to temper any get-rich-quick enthusiasm, the amount of investment into Chinese education companies fell to $46 million in 2012, less than a quarter of the previous year.2 Why?  David Chen of AngleVest – a venture capital group focusing on angel rounds – noted that “the timeframe for growing an education business can be drawn-out, and a challenge for fund managers who have to achieve returns by a specific date.”3 Thus once again, while there is potential revenue there is also required patience for returns on investment.

In addition to infrastructure issues another challenge that MOOCs may face are cultural stigmas attached to learning from non-traditional, non-accredited sources (also discussed in Chapter 9).  This may change though, as the country develops and the middle class begins to seek ways of self-help and autodidactism (e.g., their upcoming Wikipedia generation).  For what it is worth, my current employment is involved on the periphery of this industry, though not at the scale of an MOOC though.  If you are looking to work in this area I would keep your eyes open and try to attend events like 500 Startups or Barcamp Shanghai to find young techy entrepreneurs wanting to create a domestic MOOC.

Odds and ends

I mentioned it several times in Chapter 9, but there are roughly 300 million English learners in China primarily because of institutional inertia at this point (e.g., taught from primary school on up).4  And while there has always been some amount of resistance to teach this subject due to factors like nationalism, over the past few weeks I have noticed a number of op-eds published in a few national newspapers which for me, is a first.

For instance, about 10 days ago, Zhang Shuhua, a CPPCC deputy and head of the Intelligence Research Committee said that Chinese education was facing an unprecedented “destructive” crisis because of the English language requirements.  Another more recent one is from today’s China Daily which discusses reform and removing English requirements from core curriculum and examinations for college majors that do not use it (such as Chinese literature researchers).  While this probably will not be changed over night, it is something to also keep you eye on.  Or maybe you can capitalize off this nascent sentiment and create training centers for those niches.

Via Sinocism

  1. Tencent Eyes Growing Online Education Market in China from Caijing []
  2. China Investors: We Don’t Need No Edukation from The Wall Street Journal []
  3. Ibid []
  4. Chinese Learn English the Disney Way from The Wall Street Journal []

Chapter 13 – IT and software services

[Note: below is Chapter 13 from Great Wall of Numbers]

At the various schools, colleges and organizations I have worked at on the mainland, each facility was staffed by employees with a diverse range of technical abilities.  In addition, the equipment ranged from slightly dated to cutting edge.  While I have had the chance to work on a SugarCRM and Drupal wire frame development project domestically, I think some general statistics will give you a better idea of the size, scope and marketshare of the software and IT service industries in China.

According to their 2012 annual report, the Ministry of Industry and Information Technology estimated that China’s software and information services in 2011 had an output of $60 billion, “up nearly 40 percent year-on-year.”1 IBISWorld estimates that the entire software and IT industry in China “generated revenue of $284.02 billion in 2011, up 35.1 percent from 2010.”2

For perspective, India’s business process outsourcing and IT industries generated $100 billion in revenue in 2011.3

In terms of BPO growth – which is commonly called offshoring in the West – NASSCOM estimates that Indian firms generated $11 billion in BPO revenue in 2008 and $32 billion in 2012.4 In comparison, by one estimate the Chinese BPO sector “generated revenues of US$3.52 billion in 2009.”5 Another estimate, by XMG Global, shows that Chinese outsourcing firms generated $43.1 billion in revenue in 2012 (compared with $63.2 billion in India).6

Since its humble beginnings as an importer of DEC computers in the late 1970s (e.g. the PDP-7 minicomputer) China’s software development and IT services industry have grown dramatically and by one optimistic estimate, could generate $635 billion by 2015.7

Yet for perspective, the US software industry generated $261 billion in 2007 and the ten largest US software companies alone generated over $235 billion in 2010.8 Furthermore 63 of the world’s largest software companies are headquartered in the US compared with 2 in China.9

Big numbers, big opportunities

What this means is that for US-based firms, there are numerous opportunities to provide both software and related-services to the Chinese market.  And while market access and intellectual property (IP) infringement issues continue to dominate bilateral forums, there is still potential for foreign firms – especially those that focus on services – to gain substantial market share.

For example, in November 2012 I spoke with Larry Chang, the CEO of Pro-Lambda Solutions which specializes in Computer Aided Engineering (CAE) solutions and provides CAE software packages.10 Chang is originally from Taipei and had spent 25 years working in the CAE industry including in the US.  After conducting due diligence, he created a startup in Shanghai five years ago based on some surprising market research: there is no domestic CAE software company that actually develops and sells its products abroad (yet).  Or in Chang’s words, “zero engineering software products that are made in China are sold outside of China.  As a consequence everything is by-and-large still imported from other countries.  Obviously, something is missing here; if and when we can provide this missing part to the society, the economic payback will follow.  That is the opportunity we see and value.”

This is not to say that Chinese individuals and software companies do not make innovative or exportable software.  For example, Kingsoft (金山软件) is a Chinese developer that develops antivirus software and a office productivity suite called WPS.  It has 50 million monthly active users globally.11 Internet giant Baidu recently invested in the firm as well.12 Similarly, local software engineers like Ni Chao, a developer in Beijing, can and do create innovative solutions to large-scale problems such as purchasing train tickets during peak hours.13 Innovation takes place outside of the computer world as Reuters recently aired a story about various inventions used by migrant workers on their long journey home during Spring Festival, such as a local designed “seat sleeper” that enables passengers without beds to sleep on a mobile tray that can be leaned on.14 And in another fulfillment of Plato’s dictum “necessity is the mother of invention,” The Telegraph discovered a Chinese man of modest means who hand-built a working dialysis machine that has kept him alive for the past 13 years.15

Yet Arthur Kroeber, founder of the research firm Dragonomics sees scalability issues even with this promising amount of creativity.  In March 2013 he told a literary panel in Beijing that, “What’s sad is the amount of creativity you see in China is phenomenal but it’s not always directed in ways that are ultimately productive.”16 He likened it to figuring out how to create homemade solutions to a car whose parts are no longer on the market, yet running into problems trying to create “innovative solutions which are scalable throughout the entire world.”

As a consequence, Chang’s long-term vision is to become the first mover, to build and design engineering software in China which is then exported abroad.  There is a small twist to his strategy.  One of the problems he (and others like David Veksler cited later) have noted is that if you build and try to sell a product in China, most Chinese consumers will consider the quality is of lesser value.  That a product is perceived to be “better” if it originated from a foreign country is a stigma that Chang is hoping to reverse.  Thus in August 2012, Pro Lambda began selling its software solutions to the international market with the intention of giving his team experience, credibility and real-world feedback, before they attempt to sell directly on the mainland.

While traditional software solutions may be a risky business, services also have its share of challenges.  According to Chang, “one of the problems with the service industry as a whole and the software industry in particular is that this value added service is relatively unknown – and quite a suspicious concept to most Chinese consumers and businesses.  For example, upon buying your software they often think ‘why do we have to pay for your services since we just bought your software?  You owe me, not the reverse.’  Thus, this is a long-term challenge but I think enterprises and developers have begun making inroads as a younger generation of consumers has begun to understand the importance and value of this business model.”

There are also a few reasons why this lack of engineering software exports exists.  Yet according to Chang, this absence presents an opportunity for those willing to do the training needed.  For example, he notes that “software architects continue to live and work outside of China as do nearly all software product managers and development facilitating teams.  As a consequence, what has moved to China in the past decade is the ‘digital assembly line’ – coders and programmers are pretty much all that currently exists.  These coders and programmers are overseen by a project manager who coordinates with the foreign-based research and development office.  Yet, there is no facilitating team and no product team for engineering software on the mainland.”  Chang’s comment about a dearth of software architects was recently echoed by Ji Yongqing.  Ji is a technology author on the mainland who noted that while there are many programmers in China, relatively small amounts of resources are put into long-term projects to generate high-end skills, ideas and fundamental software research.  In his words, “Even now in the internet industry, everyone talks about product managers and no one talks about software architects, but in truth the two are equally important.”17

Furthermore, there are at least two systemic issues for this phenomenon as David Veksler (see below) and Chang both note: the first is that most Chinese students typically did not participate in team-based activities throughout school.  Thus when they are required to work as a team on larger scale projects, they often have difficulties adjusting to cooperation-based tasks – because they have been culturally raised to always compete and silo off information that can be traded and exchanged like currency.  Or in other words, whereas many Western education systems encourage teamwork and cooperation, older generations in China were taught a different style which relies more on trust networks (e.g., only share information with those you know, with whom you have guanxi) instead of “being a team player.”

Another key issue which is being addressed and discussed at every level and corner of Chinese society is fostering innovative thinking and creativity – taking the initiative to “think different” (see Chapter 20 too).  Yet there is a Chinese phrase that describes and explains why this same phenomenon is being repressed (and one that many Westerners are familiar with): 树大招风、枪打出头鸟or in English, “the stake that sticks up gets hammered down.”18 There are numerous requirements to build a “creative class” – yet there are also numerous cultural and institutional hammers that prevent this from germinating and blossoming on the mainland.  And while rote memorization and a lack of institutionalized ‘free thinking’ (e.g., ‘free expression’) are typically cited as the two main reasons, there are a number of additional factors that explain the constraints on domestic creativity, those would fill volumes if fully discussed.

Yet to be even handed, this is not to say that Chinese people are not creative or innovative.  For example, there is an entire industry of shanzhai (山寨) products such as customized smartphones which are cobbled together in a MacGyver-like fashion (though some segments are being shut down).1920 Similarly, web services such as Sina Weibo actually made it very easy to find and maintain trackbacks which illustrates indigenous ingenuity.  On that point, Gary Wang, founder of Tudou (a video streaming site that merged with Youku last year) recently told The Wall Street Journal that Chinese incubators, app-makers and innovators actually have cutting-edge, top-quality ideas comparable to those in Silicon Valley.21 However in his view they fall short due to a lack of experience and skills because of “the educational system and shorter start-up culture.”  Thus there is long-term potential as Larry Chang noted, for utilizing and training local talent for research and development.

Proprietary leakage

Later on in this chapter I discuss trade secrets and IT security issues, but one real-world case study that entrepreneurs should be aware of is what Chang himself faced several years ago.  His sales team abruptly left and took corporate proprietary information with them and as a consequence his sales bottom line was “burned.”

Instead of offering higher pay and enforcing stricter rules, he simply showed the predicament of the start-up company to his employees.  What he does is explain to each employee that while they could become temporarily richer by leaving and selling proprietary information, if they stayed and continued to build the company the results and rewards would be substantially larger in the long-run.22 Thus he considers his employees as partners, not employees – continuously trusting them with vital information while painting a picture of the future in which they are compensated significantly more than they might have otherwise in the immediate short-run.  As a consequence, Chang figuratively keeps the door open for all staff and is certain that any proprietary information that does leave would find little market value due to his focus on branding (i.e., why buy a pirated copy of software for the same price as the legitimate software?).

And while it remains a challenging market, as he also noted that “while a younger generation of engineers are willing to buy some types of software and government institutions are required by law to stymie digital piracy, many of the top enterprises, institutions and organizations on the mainland still typically use pirated copies and do not feel bad about it.  This presents an opportunity though and I do not begrudge them,” Chang said, “for example, in order to export a product domestic firms will have to eventually benchmark it with a legitimate copy of the software in order for foreign customers to trust its quality.  As it stands now, piracy is a form of free marketing and advertising.  As subsequent generations of users adopt and use the software they will begin to trust the product and eventually buy both the product and support services.  Take Hollywood films for example.  If copyright enforcement and penalties had been very strict, it is highly likely that no one would have watched the films to begin with.”  This last point is germane to the rapid growth of video stream sites like Youku, who arguably would not have gained preeminence if they had not stored and streamed copies of Hollywood films (Youku has now signed agreements with every Hollywood studio, see Chapter 14 for more).23

As a consequence, after hiring his first software architect five years ago, Chang’s firm now has about 30 employees, with growth rate targets of 30% annually, the profit of which is recycled and reinvested back into the company.

Services

In December 2012 I spoke with Richard Qi, the director of SR Force Consultants, a Brisbane-based software consulting firm that focuses on providing SugarCRM solutions to the Chinese marketplace – specifically to joint-ventures and foreign-owned firms.24 CRM stands for customer relationship management; it is a type of organizational and productivity software that creates a streamlined method for tracking, converting and managing leads and is used at nearly every large enterprise in Western countries.  Qi is originally from Dongbei (中国东北) and worked in Australia for 10 years before returning to the mainland two years ago.  According to him, “while there is a lot of growth potential, one of the challenges to providing technical services and solutions is that many local firms simply have not done the necessary due diligence to implement and fully utilize a lot of the software and services they purchase.  For example, SAP implementations have a roughly 70% failure rate on the mainland (e.g., initial production goals were unmet) because local customers and decision makers typically do not know what to do after the software is installed and integrated.”

Thus one of the reasons why Qi caters to joint-ventures and foreign firms is that, “they usually have detailed operational meetings and specific milestones providing both their internal IT team and external contractor with the necessary requirements gathering to build and use the functionality of the system.  They know what they are getting into.  In contrast most domestic customers are not fully cognizant of the limitations and features of their IT department let alone something more complex like a CRM system.  They may know how to run and synch a Windows server with Outlook yet they typically do not have the necessary enterprise management skill base to utilize some of the more complex packages and projects that are initially funded and installed.”

Thus in his mind, one opportunity that service firms such as his provides is “filling in the blanks with locally sourced expertise.  We hire all of our consultants locally based on both bilingual abilities and technical proficiency.  Yet services such as ours do not have to be strictly focused on CRM; business consultancy in general is about delivering value to customers and not necessarily every functionality imaginable.”  Yet one of the challenges is that “many local businesses are family managed so they typically do not have the training necessary to make long-term strategic growth plans – they are focused on immediate short-term profits that result in millions of different business paths that are often counterproductive.  In the past when we have provided solutions to these local firms, the initial service requests typically involve functionality issues (“do you have a PDF convertor”?) rather than strategic long-term issues (“how to distinguish a lead from a contact?”).  As a consequence, a challenge that other service firms will face is that if they focus solely on domestic companies, your firm may become part of an endless feature-focused loop that prevents your firm from growing and keeping pace with your international peers.”

Another issue that Qi explained and is not necessarily endemic to China is budgeting constraints.  Often time because enterprise software implementation is new to most domestic firms, the allocated budget is usually not adequate.  For example, in projects like implementing a CRM typically for every $1 spent on software, $2 needs to be spent on services just in case new modules need to be added or modified or technical support issues crop up.  Yet due to aggressive timelines, many firms face budget overruns that can prevent the systems from working efficiently or providing value to the end-user.

Cloud services

Another challenge for software makers in general is that, irrespective of trade secret issues, a large portion of traditionally developed software (e.g., shrink wrapped packages) has already been emulated, copied and installed at Chinese enterprises.  For example, one estimate of the bootleg rate in China is 77% (down from 92% in 2003).252627

So where does that leave your firm?

Perhaps your company can build out cloud computing on the mainland.  For instance, according to IDC, $286 million was spent on cloud-computing specific infrastructure in China in 2011 and this is expected to increase to $1 billion by 2016.28 There are currently 430,000 data centers and more than 5 million servers on the mainland.29 Furthermore according to IDC, over the next five years the cloud computing data center market as a whole on the mainland “is valued at 2 trillion RMB ($320 billion).”30

In terms of specific build outs, Jingdong Century, owners of 360buy.com (a leading e-commerce site on the mainland), recently invested 4 billion RMB ($750 million) building two new datacenters and in January 2013 opened a new cloud R&D facility in Beijing.3132 In September 2012 Baidu announced that it is investing $1.6 billion in building a cloud computing center.33 In March 2013 EMC, an information management company, said that it expects to land 1,000 projects over the next five years by focusing on niche segments like healthcare and education in over 300 cities on the mainland.34 Also in March, the Weather Company International, producers of the Weather Channel, announced that it would further expand its cloud and data services on the mainland where it already has more than 35 clients.35 And in addition to the Kyocera’s newly launched cloud-based network security services other firms like the Alibaba Group (Taobao, Tmall, Alibaba) are already among the leading local cloud service providers as its sites host tens of thousands of storefronts for SMEs.3637

During my interview with Eric Azumi, vice president of information services at EF (see Chapter 9) he noted cloud computing as one area on the mainland ripe for opportunities primarily because local players are still largely fragmented, inexperienced and unfamiliar with international ‘best practices.’  For example, due to various legal issues (see below) it is difficult for foreign companies to set up and directly own a data center on the mainland.  Thus Salesforce.com built a new center in Japan and Europe because according to Azumi, “there is no big money for the cloud China for the largest international participants at this time but there probably will be in the future.”  Yet concurrently he sees abundant openings for experienced foreign firms to still come in and train and provide other ancillary services to this segment.

One word of caution however, “[f]oreign companies that wish to operate cloud service in China must have governmental license.”38 As a consequence, Microsoft actually leases room in a China Telecom’s data center and outsources data management to a local firm, 21Vianet.  And Amazon recently suspended their cloud rollout due to these regulatory requirements.  Thus foreign firms specializing in cloud services should investigate the necessary legal requirements before entering this segment as well.

While moving to the cloud is increasingly popular, another area where US expertise and experience still thrives and cannot be easily copied is support services.  For example, Gartner forecasts software-as-a-service (SaaS) reached $14.5 billion globally in 2012, with US-firms taking the lions share at $9.1 billion.39 And Parks Associates estimates that the US tech support industry will “grow from $9.6 billion in 2011 to more than $20 billion by year-end 2015.”40 Can you or your company provide such services?

There is an app for that

Another potential area for US and foreign software companies is modifying their iOS and Android apps for the Chinese market.  As I mentioned in Chapter 6, China is now the world’s largest smartphone market, overtaking the US this past summer.  In addition, there are certain demographic groups, such as the elderly (aged 55+) that have been thus far overlooked for targeted apps, specifically games.41

What is the breakdown for app ecosystems?

While iOS remains relatively popular within China at more than 17% market share as of Q2 2012, more than 80% of all smartphones sold within China were Android-based.42 And in Q3 2012 Android marketshare on the mainland reached 90.1%.43 This mirrors global adoption rates, as of November 2012 Android-based devices now account for 72.4% of the global market (iOS is 13.9%).44 Unsurprisingly this has brought the total Android ecosystem to more than 50% total market share in China.45 This has also led Eric Schmidt, chairman of Google, to actively woo Chinese developers to the Android ecosystem.46 Yet despite this huge potential market, nearly all of these Android phones have been stripped of Google ad-supported services as well as Google Play – replaced by custom 3rd party applications and app stores.4748 In fact, 80% of Android phones in China use a preinstalled version of the Baidu-powered search tool instead.49 Or in other words, modern smartphones with Chinese characteristics.

What this means is that for US app developers, there are some opportunities to port and translate their apps and games to the Chinese market.  For example, as I also mentioned in Chapter 6, in terms of smartphones and tablets, less than 10% of the Chinese user base are older adults (55+).  This same demographic group comprises 7.1% of gaming and entertainment app users compared with substantially larger percentages in the US.50

How much larger in the US?  For instance, while a Pew Internet study found that only 13% of those ages 65+ in the US had a smartphone, Nielsen reported in May 2012 that in the US, “more than 50% of those who play FreeCell, Solitaire, and Hearts are over the age of 55.”5152 And a June 2012 study from Forrester research found that 44% of US seniors play solo games online.5354

In contrast, according to their 2010 report from IDC, only 7.1% of those aged 50+ in China played games.  More specifically, in terms of online chess gamers and mobile gamers, those older than 50 comprised 5.7% and 2.4% of all players respectively.55 Or in short, your grandparents and their peers frequently play computer games yet few software firms design games specifically for them, let alone for their Chinese counterparts.

While there may be cultural reasons for such a dramatic difference (7.1% in China versus 50% in the US), in my own anecdotal experience of walking through the parks and streets throughout the cities I have lived in, elderly Chinese seem just as apt to play memory games, dominoes (mahjong) and poker-style games as their Western counterparts.  And according to China Daily, “the turnover of China’s mobile gaming market is soon going to hit 5.2 billion yuan ($835 million) as the number of players reach 270 million.”5657 Thus in the long run even if the adoption and penetration rate remains relatively low for the elderly demographic group, 7.1% of 202 million (the number of elderly currently in China, see Chapter 18) is a potential niche market for future growth.

And as I mention in Chapter 6, in general, developers looking to port their apps and games over to Chinese markets should consider modifying the games to include Chinese traditions, symbols and cultural tie-ins – or in other words ‘Western video games with Chinese characteristics.’  For example: the color red, number 8, and the Chinese knot (Zhōngguó jié) are all considered lucky.  Perhaps creatively integrating these symbols into your game would prove popular, just as Kung Fu Panda was (see Chapter 14).  And since Macau now generates more than six times as much as gambling revenue as Las Vegas (Macau overtook it in 2007) maybe there is a legal way to capture this market.58 Or rather, because gambling is popular across all demographic groups perhaps designing a social gambling game or non-monetary betting app would find success across the mainland.59

Based on the wide variety of demographic groups playing games on the subway in Shanghai and Guangzhou and standing in line at restaurants, casual games such as those from PopCap (e.g., Peggle, Bejeweled, Plants vs Zombies), Imangi Studios (Temple Run), ZeptoLab (Cut the Rope), Halfbrick Studios (Fruit Ninja) and Rovio (the Angry Birds series) are also popular.  In fact, “Cut the Rope” has more daily users in China than any other country and according to the Financial Times, “around a quarter of all Angry Birds downloads are conducted in China.”60 It is so popular in fact that Rovio recently turned Shanghai’s skyscrapers green to market their new product and simultaneously launch a native version for the Chinese market.61

Another advantage US-firms currently have in porting their apps to the Chinese marketplace: English is the 2nd largest language in the Chinese iOS app store.62 And this presents an opportunity for Western developers: in their September 2012 report, Distimo found that after introducing a native language app, their “download volumes on the iPhone [increased] by more than 128 percent during the next week that followed.”  And sales revenue increased by 26 percent in the same week.  Either way you look at it, even if your company does not create a Chinese-version of its apps, the potential competitive marketshare even in English remains in reach of your company.

Understanding the market

You might be asking yourself, how does the app store function in China?  Are they run by Apple and Google and are they censored?

Apple opened its first official app store in China on October 27, 2010.63 By June 2011, China became the second largest source of app downloads for Apple.6465 And China sales for Apple products and services now accounts for 15% of Apple’s total revenue, $23.8 billion in fiscal 2012.66 In fact, Apple is actively courting Chinese developers by translating their tools and guides into Chinese.67 In addition to the large Android userbase, there are more than 70 Android app stores in China, which is estimated to eventually consolidate down to 10 within the coming years.6869

In terms of censorship, as reported by the New York Times, Apple has been selectively censoring applications in its app store based on requests by the government.70 And because of Google’s on-again-off-again legal fights with Chinese regulators, it is oftentimes unclear of what is being censored in the Android marketplace.  For instance, in the fall of 2011 there was a week-long period in which both the Android marketplace and Gmail application worked intermittently.71 This occurred once again in the early parts of the summer and fall of 2012 yet service was restored in both cases.72

This also raises another visceral point.  Despite its off-and-on wrestling with Chinese regulatory authorities, with a mere 4.72% search marketshare, Google’s revenue in “China’s mobile-app ad market will probably more than double to about 1.8 billion yuan ($283 million) this year [2012], exceeding the 1.2 billion yuan from mobile-search queries.”7374 In fact, despite these ongoing disputes with Chinese regulators, Google is “still the 3rd largest advertising revenue generator in that country doing $640 million a year (annualized).”75  And despite being hard to access at times Google has roughly 15% of the search engine market on the mainland.76 If they can achieve this in the face of never ending challenges, then your firm has potential as well.77

An app that helps find customers

Over the past 18-months Windisch-based coresystems has been working on a cloud-based digital assistant called Mila (an app) that was a finalist in the GMIC G-Startup competition held in October 2012.78) Mila allows entrepreneurs and SMEs to create an online assistant and unified online store front which is hosted on the cloud for free.  The assistant (Mila) can then search social media sites like Twitter to look for potential customers based on what your company provides as services.  And once a match is found, it then guides you through a streamlined sales process including invoicing using a smartphone.

In October 2012 I spoke with Andrea Chang, the marketing manager for Mila’s branch in China.  According to Chang, in their effort to localize the brand on the mainland, Mila has partnered with China Unicom (the second largest telecom company in China).79 Together they have modified Mila to integrate with Sina Weibo (which I noted in Chapter 12 is the world’s 2nd largest microblog site) and Alipay (the largest online payment provider on the mainland).  According to Chang, “the process of opening an online shop is one of the easiest and cheapest ways to generate leads and do business in China.  Using an integrated chat feature that allows customers and business to speak directly to one another, Mila not only communicates directly with your customer but also conduct all transactions, including invoicing.”

Chang also noted that because of the wide proliferation of smartphones and social media in China that one of the advantages of using Mila is that its cloud based transaction model substantially lowers the sales cycle costs (e.g., locating potential customers) while simultaneously providing customer service (e.g., by storing customer contacts).  This in turn allows entrepreneurs and SMEs to compete more on service instead of spending resources on search-engine optimization (SEO) or virtual store fronts.

So how does this help foreign companies wanting to do business in China?

Again, as mentioned in Chapter 12, before your company even establishes a physical presence on the mainland, you can use Mila and other services like Wildfire to search and discover the potential customer base for your company’s products and services.  And as I mentioned in Chapter 12 as well, because Facebook and Twitter are currently blocked on the mainland, you will need a way to localize your customer search to Chinese web services.  Solutions like Mila and Wildfire makes the process easier for your team, even if you are unfamiliar with Chinese customs and culture.

Securing your network

Cybersecurity is a sub industry that is often overlooked and dismissed by many businesses in China.  It has not helped that some media outlets resort to hyperbole to describe the real – and sometimes imagined – dangers for all firms with insecure IT networks.  For example, in July 2012, General Keith Alexander director of the NSA announced that up to $1 trillion in cybercrime damage was done globally each year.  This figure was later debunked.80 Yet determined hackers – both domestic and foreign – can and will compromise trade secrets and other proprietary assets typically without being caught.  Because a lot of theft and digital espionage goes left unnoticed it is very difficult to guess how much damage cybercrimes create.81 However in September 2012, Symantec released arguably one of the most extensive studies related to cybercrime and estimated the damage to be $110 billion a year globally.8283

How does cybercrime affect China, Chinese business and foreigners doing business in China?

In March 2012, Businessweek published a widely circulated report about corporate espionage of a US wind turbine supplier (AMSC) conducted by its Chinese client, Sinovel.84 In short, while AMSC attempted to isolate its trade secrets and proprietary software code outside of China (using an ‘air gapped’ facility), Sinovel still managed to use social engineering (e.g., bribery) to lure one of AMSC’s key Austrian-based programmers to China.  An ‘air gapped’ facility in their case meant the proprietary code – “secret sauce” – was only accessible at a workstation that was not connected to the internet.85 Using the ‘defense in depth’ IT security strategy (e.g., multiple firewalls and secure zones nested within one another) AMSC purposefully built this facility with the sole intention of building a physically isolated silo that could not be easily compromised.  While the case is still being fought in court, this is not an isolated instance.86 According to Akamai, a leading content-delivery network provider, in Q3 2012 one third of all cyberattacks originated from China (the US was second with 13%).87 All told, since 2007 the FBI and the Justice Department have opened more than two dozen cases involving trade secret, economic espionage and embargo circumvention restrictions involving Chinese contractors and individuals.88

One solution – a drastic solution – was detailed by the Washington Post in 2011.89 They interviewed several American executives who frequently traveled between the US and China each year for a variety of meetings.  A few of the executives had a straight forward security solution: buy a new iPad before flying to China, download all of the needed information from the cloud and then never use it again (e.g., throw it away).  Another simple low-tech, yet increasingly popular solution is to simply no longer provide external media outlets like a USB in a terminal with access to sensitive information.  In fact, in some IT security circles, one nickname for the USB is now “Ubiquitous Security Backdoor” due to this chronic problem – the ease in which sensitive information can be removed with a flash drive or in which malware can be conveniently installed, such as Stuxnet and Flame.909192

But what if the hackers simply move and setup shop overseas in your hometown?  In May 2010, NetworkWorld ran a story about an ongoing espionage attempt by unknown Chinese operators and a large US firm in the Midwest.93 Similarly, according to a recent Bloomberg story, right before its attempted $2.4 billion acquisition of Huiyuan Juice Group fell through, Coca-Cola was hacked in 2009 by a Chinese hacker group dubbed Comment Crew.949596 While it is unclear whether either of the espionage activities was successful, the threat of domestic and foreign hacking should motivate your company into proactive risk assessment – even if it does not plan to operate overseas.

Yet it is not just US firms that are on the losing end of cybercrime.  According to the same McAfee study above, malware and phishing attacks cost Chinese consumers $46 billion in 2011, twice as much as the US.97 The Ministry of Information Technology and Industry published a report that said “in 2012 alone that foreign hackers used viruses and other malicious software to seize control of 1,400 computers in China and 38,000 websites.”98 In fact, according to the Anti-Phishing Alliance of China (APAC) between January and November 2012 there were 24,535 phishing websites and scams targeted specifically at China’s online populace.99 In addition, in just a matter of weeks into 2013, a new virus called “Bill Shocker” has already impacted 620,000 users in China targeting the popular QQ messenger (see Chapter 12).100 In another instance, there was a 47% month-to-month phishing surge during Single’s Day (11-11) in November 2012.  This is the biggest online shopping day of the year as mentioned in the previous chapter.  Furthermore, Rising Information Technology, a web security company located in Beijing, estimated in a January 2013 report that nearly 200,000 Chinese websites were hacked in 2011 and at least 60% “of the attacks targeting China’s large companies, government, and scientific research institutions come from overseas.”101 According to Rising’s report, because Internet security typically is overlooked “[a] growing number of Chinese companies are turning to overseas Web security companies for protection, a move which still leaves them vulnerable to attacks.”

However with these challenges come opportunities for foreign security experts such as David Veksler, CEO of CryptAByte based in Shanghai.102 In October 2012 I had a chance to talk with him regarding some of the key opportunities in China’s nascent security industry.  He noted that “Chinese companies and foreign firms doing business on the mainland are equally in need of information protection.  Since retooling and retraining in business is increasingly based on software, losing proprietary information and trade secrets to any competitor, irrespective of physical location, can lead to losing your competitive advantage in innovation.”  Later in Chapter 20 he explains several other challenges and opportunities, but according to him, there are numerous possibilities for security experts since SMEs on the mainland are typically unaware of IT vulnerabilities such as zero-day exploits.  Zero-day exploits (or day zero) are threats and attacks that take place on the first days of a discovered vulnerability, before a developer patches the hole(s).  Thus according to Veksler, security consultants can help train mainland-based IT departments on ‘best practices’ and preventive measures that Western firms have learned the hard way with.

How does this work in practice?  For example, the world economy is shifting from capital intensive retooling which typically involved heavy machinery, to rapid prototypers and 3D printers (see Chapter 7).  This means that capital tools are now software.  Thus if you want to steal a new factory in the 21st century, all you really need to do is pilfer software.  As a consequence, the theft of entire industries could conceivably take place, allowing perpetrators to simply take the data to the cheapest country (e.g., based on land and labor costs) and eat into the marketshare of the original innovator.

This cloak-and-dagger industrial espionage is in Veskler’s words, “actually becoming a prime motivator for innovation.  While competitors could learn trade secrets through hiring former employees or reverse engineering, because you are never quite sure if someone has hacked into your systems or used social engineering – like Kevin Mitnick did – to gain access to proprietary information, every incumbent must now continually innovate.  Otherwise their competition could use a stealth startup and out-maneuver you with your own confidential information.”  In economic theory, when a firm is successful it sends profit signals out to the marketplace (e.g., by satisfying consumer demand you become profitable and other participants take notice).  As a consequence, because the firm realizes it will eventually draw competition with these “signals of success” it has to always keep striving to improve and innovate.

Kevin Mitnick was a hacker in the 1980s who used social engineering (e.g., manipulating secretaries to give him secure access) to compromise corporate networks such as DEC and Motorola.103 Samuel Slater, known as the father of the Industrial Revolution in the US, was born in the UK.  He was an originally an apprentice at a cotton mill based on Richard Arkwright’s design near Cromford Mill in England.  When he immigrated to the US, he later used a design similar to Arkwright’s to kick-start the American Industrial Revolution.  This a common risk noted Kent Kedl of the consultancy Control Risks, who recently told The Economist that, “The easiest way to get intellectual property from a firm is by buying or renting an employee inside it.”104 Thus, a stealth startup today could conceivably appropriate proprietary information (e.g., CAD models, engineering designs) via social engineering, hiring or hacking, build a warehouse in a developing country where resources costs are relatively low, and fill the warehouse with 3D printers.  Then in turn, export the products to world markets.  Some of the practical issues involving VPNs for corporate environments, such as preventing industrial espionage, are discussed later in Chapter 20 as well.105

During my February 2013 interview with Shaun Rein, founder of China Market Research, he noted that “for any company in the world, internet security is an increasingly important issue.  And especially in China I think a lot of MNCs are continuously worried about protecting IP.  As a matter of fact, our firm recently received an RFP [Request for Proposal] from a very large internet company building a marketing expansion strategy on the mainland.  As part of the proposal we are supposed to disclose our firm’s security issues to make sure we are a reliable partner to work with.  In other words, to prevent any proprietary information from being leaked by a vendor they are modifying their risk management to hedge against the possibility of being hacked.  The flip side of this is that there are currently no large barriers to entry for doing internet security consulting because the government is very supportive of intellectual property transfers at this level.  At the same time, it may be more difficult selling antivirus software directly because then you would be competing with domestic forces and local firms like Kingsoft.  But services such as IT security are quite open.”

In January 2013, internet giant Baidu announced that it was investing in Kingsoft, makers of antivirus software (and an office productivity suite).106107 At the beginning of this year, several media outlets such as Businessweek have released additional reports covering Comment Crew (see above), also known as ATP1 (which may be the same as PLA Unit 61398) which has purportedly hacked into nearly 150 companies and organizations in more than a dozen countries over a period of 7 years bringing this IT security issue to the attention of more stakeholders such as MNCs.108

And with all of these local and international security issues laid bare, for another perspective one should also consider the comments from General Electric Vice Chairman, John Rice who recently explained that, “Despite hacking and other issues in China, foreign companies need to be there, due to the country’s potential as the world’s biggest marketplace.  The greater risk lies in staying away.”109 Without going into details, GE is purportedly “improving how it handles threats to its information.”  Thus eternal electronic vigilance may be the new normal but it is something that your competitors (both domestic and foreign) will probably have to overcome as well.

Takeaway: The software development, IT support and security services industry is both alive and growing at a fast pace in China.  US firms relying on traditional revenue models such as selling shrink wrap packaging will need to modify their business model for entry into China.  This may come in the form of cloud computing and software-as-a-service.  Yet either way their expertise and quality management – even at higher costs – are still marketable within China.  In addition, US firms specializing in developing apps have yet another revenue stream they can tap into if they are able to modify and translate their applications for Chinese consumption – the world’s 2nd largest app market.  Furthermore, IT security firms also have potential opportunities to secure and optimize the networks of Chinese enterprises and SMEs whom suffer billions in economic losses each year.


Endnotes:

  1. Software outsourcing on upward curve from China Daily []
  2. Chinese Software Industry to Grow 25% Through 2016: Report from eWeek []
  3. Indian IT-BPO Industry from NASSCOM []
  4. Ibid []
  5. 5 Reasons Why China Will Dominate Business Process Outsourcing from Right Site []
  6. China, not PH, eroding India’s BPO leadership, says consulting firm from InterAksyon []
  7. See The Emerging Market of China’s Computer Industry by Jeff Zhang and Yan Wang and Chinese software, IT revenue to touch $635bn from Times of India []
  8. Software Industry Facts and Figures from Business Software Alliance []
  9. Global Software Top 100 Edition 2011: The Highlights from Software Top 100 []
  10. See Pro-Emfatic and Pro-Lambda Software []
  11. Kingsoft Boasts Over 50 Million Monthly Active WPS Users from China Tech News []
  12. Baidu Invests in Kingsoft, Moves Into Web Security, Qihoo’s CEO Calls it a “Big Joke” from Tech in Asia []
  13. See Train ticketing software highlights China’s innovation paradox from Xinhua and China Train Ticket Site Cost Nearly $100 Million, Seems to Be Harassing Programmers, And Might Be Broken Again from Tech in Asia []
  14. Rubber chickens, ostrich heads ease China’s rough ride home from Reuters []
  15. Chinese man kept alive by self-built dialysis machine from The Telegraph []
  16. Economist: China Plenty Creative, Just Not in Right Ways from The Wall Street Journal []
  17. Why China Can’t Make Its Own Mobile OS from Tech In Asia []
  18. It is a cultural characteristic of many regions in East Asia.  For example, the Japanese equivalent is 出る杭は打たれる. []
  19. See also hackerspaces in Chapter 7.  See Bandit phone king has the last laugh from Financial Times, Imitation Is the Sincerest Form of Rebellion in China from The Wall Street Journal and In China, Knockoff Cellphones Are a Hit from The New York Times []
  20. Number’s up for fake cell phones from Shanghai Daily []
  21. Chinese Companies Getting Good at Attracting Talent from The Wall Street Journal []
  22. The economic term for short versus long-term time horizons is “time preference.” See Chapter 18 in Human Action by Ludwig von Mises. []
  23. Similarly, Hearst president David Carey recently noted that Apple and Steve Jobs “taught people how to buy digital content.”  See Hearst president David Carey: Apple taught people ‘how to buy digital content’ from Engadget []
  24. SRForce []
  25. Microsoft’s newest weapon in China piracy fight from Reuters and Report: China’s software piracy rate falls to new low — of 77% from ZDNet []
  26. To combat piracy of Windows 8 in China, Microsoft will not sell a shrink wrapped package – users can only get it pre-installed by OEMs or by downloading it.  With the release of Office 2013 on the mainland, consumers can still purchase traditional packages via Microsoft’s online store.  See Microsoft Cancels Packaged Windows 8 For Chinese Market from China Tech News and Microsoft Commences Office 2013 Software Sales In China from China Tech News []
  27. One other partnership area could be to pursue a joint-venture such as the kind that Microsoft and Suning (a large mainland retailer) have recently announced.  See Suning, Microsoft Ink Multipart Retail Deal For China from China Tech News []
  28. Cloud computing investment ‘to hit $1b’ from China Daily []
  29. Ministry to set up cloud computing data centers from China Daily []
  30. Ibid []
  31. Tech Bytes: 4 Billion Yuan from China Daily []
  32. China’s 360buy.com Launches Cloud Computing R&D Center In Beijing from China Tech News []
  33. Baidu Shares Plunge on Worries over Mobile Monetization from Caijing []
  34. EMC China’s Growth Focuses On Big Data, Cloud Computing from China Tech News []
  35. Cloudy Days Ahead As Big Data Comes To Chinese Meteorological Administration from China Tech News []
  36. Cloud-based Network Security Suite Launched By Kyocera In China from China Tech News []
  37. Alibaba’s Cloud Computing Platform Combines Storage Services from China Tech News []
  38. Amazon’s cloud service aborted in China, launch of Kindle delayed from Morning Whistle []
  39. Gartner Says Worldwide Software-as-a-Service Revenue to Reach $14.5 Billion in 2012 from Gartner []
  40. Tech Support Industry Webcast Will Examine Opportunities in $9 Billion Market from Parks Associates []
  41. It is highly recommended that game developers and digital entrepreneurs read Digital Game Design for Elderly Users from Association for Computing Machinery.  The study noted a similar finding, including one that I also point out: “the growing 65+ demographic is currently not well served by the majority of commercial games on the market, creating a significant potential niche market for game developers.” []
  42. China’s smartphone market grows 164%, Apple’s iOS takes 17.3% from Apple Insider []
  43. Report: Android Rises to 90% of Smartphone Market in China from Tech In Asia []
  44. Gartner has published two others estimates which put Android marketshare globally at 68.4% in 2012 compared with 19.4% for iOS and later with Android at 69.7% and iOS at 20.9%.  See Strategy Analytics: Android claimed 70 percent of world smartphone share in Q4 2012 from Engadget, Gartner Says Worldwide Sales of Mobile Phones Declined 3 Percent in Third Quarter of 2012; Smartphone Sales Increased 47 Percent from Gartner and Gartner Says Worldwide Mobile Phone Sales Declined 1.7 Percent in 2012 from Gartner []
  45. See Android is winning – if you’re writing apps for China. Elsewhere, though… from The Guardian and Alternative app stores and platform branches: Is Android too open? from Android Authority []
  46. After North Korea trip, Google’s Eric Schmidt swings by China to woo Android developers from The Next Web []
  47. Google is beginning to try and take action to purportedly prevent further fractures and forking of the Android ecosystem.  See The Acer/Google/Alibaba tussle: It’s not about open Android from ZDNet and Acer Apparently Reconciles With Google from Forbes []
  48. While there has been a lot of discussion over the past year over whether or not it is profitable for developers to make Android apps for the Chinese marketplace, there is at least one success story that could be used as a case study: CocoaChina which makes a popular game called Fishing Joy.  See How CocoaChina proved it’s possible to make money on Android in China (to the tune of $2m a month) from The Next Web []
  49. See 80% of Android phones in China will have its default search set to Baidu from The Next Web and Android Takes Off in China, But Google Has Little to Show for It from Forbes []
  50. See Table 3, p. 9 China Gaming Market End-User Survey, 2010 from IDC []
  51. Nearly half of American adults are smartphone owners from PewInternet []
  52. Vintage PC Video Games Still Thrive in World of App from Nielsen []
  53. The Data Digest: Digital Seniors from Forrester []
  54. While unrelated to gaming see also, For the first time, half of adults ages 65 and older are online from PewInternet []
  55. Gamers in a sample size of 29,392.  Online chess gamers in a sample size of 3,050.  Mobile gamers in a sample size of 1,519.  See China Gaming Market End-User Survey, 2010 from IDC []
  56. Internet gaming: ‘A winning gamble’ from China Daily []
  57. Mobile payments are also expected to rise markedly over the next 3-5 years, hitting $112 billion by 2015.  According to Alipay (the largest domestic online payment service), in 2012 the number of people who used mobile payment increased by 223% and “over 4.3 million people spent more money via mobile phones than PCs.”  See China’s Mobile Payments Will Reach Over CNY700 Billion By 2015 from China Tech News and China’s Alipay Reported 546% Wireless Payment Growth In 2012; Tibetan City Tops Ranking from China Tech News []
  58. Revenue hit $38 billion in 2012 and is expected to reach $44 billion in 2013.  See Rolexes Pawned in Macau Signal Further Gains for Casinos from Bloomberg, Macau gaming revenues hit $33.5 billion in 2011, no slowing seen from Las Vegas Review-Journal, Broken Tooth and New Macau from Foreign Policy and Door is about to slam shut on high-rolling holidays to Macau from The Times []
  59. Despite initial reports that suggested a new pilot program was starting at a casino in Sanya, Hainan province (called Jesters), gambling on the mainland is currently banned.  Macau is the only nearby domicile where this is allowed.  Mainland residents must still apply for an entry visa in order to travel to Macau and are typically only allowed to visit it a few times a year.  There are exceptions, for example, if you live nearby in certain cities of Guangdong or if you have relatives living in the SAR.  See Sanya Says It Never Licensed Any Form of Gambling Activities from Caijing, Chinese authorities close cashless casino bar in island resort from Reuters, Macau Casinos Decline on Visa, Credit Limit Concerns from Bloomberg, Macau’s Casino Revenue Reaches Record After Holiday Week from Bloomberg and China Tightens Reins on Macau from Bloomberg []
  60. See For App Makers, China Is Untapped and Untamed from The Wall Street Journal and China: lots of three kingdoms, not enough Angry Birds from Financial Times []
  61. Rovio announces Angry Birds book app: Live from Frankfurt Book Fair from paidContent and Bad Piggies and Angry Birds Hit the Road in China, Turn Shanghai Skyline Green from Tech In Asia []
  62. According to Distimo, “Applications with Chinese as a language in the top 200 were responsible for the largest share of the free downloads in China at 73 percent. English was responsible for only 69 percent of the free downloads among the top 200 in China.” See App Distribution Becomes A Global Game: The Shift Of Power & Impact For Developers from Distimo []
  63. Apple Opens Chinese App Store from The Wall Street Journal []
  64. China Now Apple App Store’s Second Biggest Market from PcMag []
  65. Apple’s App Store made big gains in China in 2011 from GigaOm []
  66. In a January 2013 interview, Apple CEO Tim Cook predicted that China will become the biggest market overall for Apple.  See iPhone 5 launch results in new weekend record for Chinese market from ArsTechinca, Tim Cook: China Accounts for 15% of Apple’s Sales, Will Get iPhone 5 in December from Tech In Asia and iPhone 5 hits China as Apple market share slips from Reuters, Interview: Apple CEO expects China to become biggest market from Xinhua and Apple’s China dilemma: market share or cachet? from Reuters []
  67. Apple is also opening up an R&D center in Shanghai.  See Apple courting Chinese developers to strengthen iOS in China from ArsTechnica and Apple Shanghai R&D center confirmed for summer 2013 from Apple Insider []
  68. See China Has 70 Android App Stores, But That Could Soon Whittle Down To 10 from paidContent and For App Makers, China Is Untapped and Untamed from The Wall Street Journal []
  69. One problem with this fragmentation is that applying security patches is a much longer process and sometimes never occurs, leaving consumers open to fraud schemes such as ‘smishing’ (sending phony text messages).  See ‘Fragmentation’ leaves Android phones vulnerable to hackers, scammers from The Washington Post []
  70. Far-Ranging Support for Google’s China Move from The New York Times []
  71. China Cripples Android With Fitful Blocks of Gmail, Market Apps from paidContent []
  72. Similar blockages have occurred in November during the leadership transition.  Readers may be interested in the developments with GitHub as well.  See What is going on with GMail in China, and how to get around from GreatFire and China, GitHub and the man-in-the-middle from GreatFire []
  73. Google Finally Leads in China – in App Ad Sales from Bloomberg []
  74. Google decline in China continues as its search share falls to 4th place, maps to 6th from The Next Web []
  75. Google Still Does $640 Million In Annual Revenue In China from Forbes []
  76. China Search Engine Market Share in 2012 from China Internet Watch []
  77. According to one recent report, sometime at the beginning of December 2012 Google acquiesced and removed “a feature which had previously informed users from China of censored keywords” and “at the same time, they deleted the help article which explained how to use the feature.”  Yet according to another source “the opportunity to capitulate was lost forever when Google gave the middle finger and left.”  See Google Bows Down To Chinese Government On Censorship from GreatFire.org and Mr Kim, tear down that wall; Mr Xi, carry on from The Economist []
  78. Mila from coresystems can be downloaded from Google Play and Apple’s App Store (WoStore is China Unicom’s equivalent []
  79. China Unicom has its own marketing channel which Mila uses. []
  80. Does Cybercrime Really Cost $1 Trillion? from ProPublica []
  81. Pentagon Warns: ‘Pervasive’ Industrial Spying Targets U.S. Space Tech from Wired []
  82. In February 2013 Microsoft researchers published a report discussing reasons and variables for why certain geographic regions and areas are more or less prone to cybersecurity holes and abuse.  Unsurprisingly economic stages of development played a big role (e.g., wealthy countries have lower rates of malware infection compared with developing countries).  See Wealthy Countries Better At Protecting Citizens…From Malware from The Security Ledger []
  83. 2012 Norton Study: Consumer Cybercrime Estimated at $110 Billion Annually from Symantec []
  84. China Corporate Espionage Boom Knocks Wind Out of U.S. Companies from BusinessWeek []
  85. FAA: Boeing’s New 787 May Be Vulnerable to Hacker Attack from Wired []
  86. China Court to Weigh Corporate-Spy Case from The Wall Street Journal []
  87. China Source of Most CyberAttacks, Says Akamai from PC Magazine []
  88. Summary of Major U.S. Export Enforcement, Economic Espionage, Trade Secret and Embargo-Related Criminal Cases from Department of Justice []
  89. In China, business travelers take extreme precautions to avoid cyber-espionage from Washington Post []
  90. Ubiquitous Security Backdoor from SANS Institute []
  91. This security issue is not endemic to China.  For example, over the past two years, a school in Virginia and a hospital in Oregon accidentally lost USB drives which contained sensitive information.  See OHSU says stolen USB drive contained some patient data from KATU and Students’ personal data exposed after USB drive stolen from SOPHOS []
  92. See Flame and Stuxnet Cousin Targets Lebanese Bank Customers, Carries Mysterious Payload from Wired and Resource 207: Kaspersky Lab Research Proves that Stuxnet and Flame Developers are Connected from Kaspersky Lab []
  93. See Black duck eggs and other secrets of Chinese hackers from NetworkWorld and Michigan Couple Stole GM Secrets for Chinese, U.S. Says from Bloomberg []
  94. Coke Gets Hacked And Doesn’t Tell Anyone from Bloomberg []
  95. Comment Crew (also known as APT1) is also suspected of hacking into other firms (both foreign and domestic) including a high-profile case involving Solid Oak Software, a California-based firm that specializing in developing internet filtering software.  Two other large hacking organizations are the collective known as ‘Beijing Group’ and the PLA’s Unit 61398 whom are suspected of conducting economic espionage (APT1 and 61398 may be one in the same).  See China Mafia-Style Hack Attack Drives California Firm to Brink from Bloomberg, A Chinese Hacker’s Identity Unmasked from Businessweek, Mandiant, the Go-To Security Firm for Cyber-Espionage Attacks from Businessweek and Mandiat APT1 Report from Mandiat []
  96. The New York Times has repeatedly been hacked since October 2012 as have the servers of The Wall Street Journal and The Washinton Post as well.  The perpetrators of all three are purportedly located in China.  See Hackers in China Attacked The Times for Last 4 Months from The New York Times, Chinese Hackers Targeted Wall Street Journal Computers from The Wall Street Journal, The People’s Republic of Hacking from Foreign Policy, A Chinese Hacker’s Identity Unmasked from Businessweek and Chinese hackers suspected in attack on The Post’s computers from The Washington Post []
  97. Chinese lost US$46 billion to cybercrime last year from Shanghai Daily []
  98. U.S. Ready to Strike Back Against China Cyberattacks from Associated Press []
  99. Phishing scams target China’s growing online population from Xinhua []
  100. Malware controls 620,000 phones, sends costly messages from Help Net Security []
  101. Nation under increasing threat from hackers from China Daily []
  102. CryptAByte []
  103. See The Art of Deception: Controlling the Human Element of Security by Kevin Mitnick and Takedown: The Pursuit and Capture of Kevin Mitnick by Tsutomu Shimomura []
  104. Who needs cyber-spying? from The Economist []
  105. Domestic Chinese firms are also investing into this segment.  See Baidu Invests in Kingsoft, Moves Into Web Security, Qihoo’s CEO Calls it a “Big Joke” from Tech in Asia []
  106. Baidu Invests in Kingsoft, Moves Into Web Security, Qihoo’s CEO Calls it a “Big Joke” from Tech In Asia []
  107. In 2002, foreign firms such as Symantec, Trend Micro and Network Associates were required to give code samples (e.g., viruses, rogue wiretaps) to the security ministry in order to receive approval for access to the mainland consumer market.  In addition to Kingsoft, domestic firms now include Qihoo 360 and Rising.  As of Q3 2012, Qihoo 360 had 442 million monthly active users and the enterprise version reached 420,000 users (representing millions of computers).  See China Is Asking Software Firms To Provide Samples of Viruses from The Wall Street Journal and Qihoo 360 Acquires Chinese Web Log Analysis Platform from China Tech News []
  108. See A Chinese Hacker’s Identity Unmasked from Businessweek, Mandiant, the Go-To Security Firm for Cyber-Espionage Attacks from Businessweek and Mandiat APT1 Report from Mandiat []
  109. Being in China Is Less Risky Than Not Being There from The Wall Street Journal []

Chapter 20 – VPN and infrastructure services

[Note: below is Chapter 20 from Great Wall of Numbers]

Any company wanting to conduct international business on the mainland, irrespective of whether it is local or foreign, must invariably factor in the additional costs of communicating electronically beyond the mainland.  Beginning in 1996 and launching in 1999, the Ministry of Public Security and other governmental organizations began implementing and enforcing a series of regulations involving data filtering that ultimately matured into what is commonly referred to as the “Great Firewall” (GFW).12 Collectively, the types of filtering techniques employed by the Ministry through Tier 1 ISPs (all SOEs as noted in Chapter 15) include: IP blocking, DNS filtering, URL filtering, packet filtering and connection resets.  Simultaneously, the Ministry maintains a continuously updated “black list” of websites that mainland users are unable to access through this dynamic filtering and blocking mechanism.

This presents an opportunity to virtual private network (VPN) providers overseas.  A VPN is a type of technology that effectively creates a secure tunnel from one computer to another, isolating its data stream from the surrounding traffic.  This can be done by means of encryption and as a consequence many banks, financial institutions and national security centers – irrespective of the region or hemisphere – typically use some form of VPN to securely communicate with outside parties (e.g., for wiring money, discussing trade secrets, or diplomacy).3

There is no shortage of VPN providers in Western countries and there are in fact, Chinese-based VPN providers as well – the efficacy and reliability of which is debatable.4 In my own anecdotal experience, even with a paid service based in a foreign country, data can still be throttled and your connection reset.5 One reason is that the GFW is not a passive system – it is continually tweaked and changed.  In an interview in 2011, Fang Binxing, the Father of the Great Firewall explained that he himself has “six VPNs on my home computer.”6 He uses them to “test which side wins: the GFW or the VPN.”  And in his opinion, “[s]o far, the GFW is lagging behind and still needs improvement.”7

One world, two internets

As I mentioned in Chapter 12, while there are any number of domestically made and managed counterparts and clones of foreign social media services (e.g., Sina Weibo is the equivalent to Twitter), there is still niche demand for foreign-based web services.  For example, as I mentioned in Chapter 9, there are now about a million Chinese students studying overseas each year; more than 190,000 Chinese students studied in the US this past year alone.8 In addition, 1.36 million Chinese tourists visited the US in 2011.9 What this means is that as I mentioned in Chapter 3, Chinese consumers are increasingly exposed to Western and in particular, American tastes and services.

Yet to temper the optimism that a VPN provider could immediately sell several million service packages to individual mainlanders, consider this rough facsimile: while we may never know the real number, Facebook’s own indirect estimate of mainland usage of its social network is close to 600,000.10 While there are a number of other niche services in demand, especially from financial service firms, this 600,000 number can be used as a proxy to estimate the general demand for VPNs.

It should also be noted that not all foreigners use Facebook in China nor do all foreigners want to pay for a VPN to access it.  Furthermore, based on my own anecdotal experience at various institutions, the average Chinese user does not currently have access to a VPN or other fan qiang (“Wall Climbing”) software such as UltraSurf or Hotspot.  In their mind, why should they have to pay to access foreign services when there is a similar Chinese version available for free?  This is not to say that they could not gain access to the services if they were motivated and inclined to do so.  In my own anecdotal experience virtually none of the several thousand students I have had at various cities on the mainland over the past four years have had active Facebook accounts.  A few however have had VPN accounts so they could play online games like World of Warcraft on servers outside the mainland (e.g., “gold farmers,” see Chapter 14).11

Assuming the number is around 600,000, how much can a foreign-based service provider expect to generate?  Currently, the average monthly rates at PandaPow, Astril and most others are roughly $10 a month.12 And because a large portion of a VPN package is based on software that is open-source and free, the initial setup costs are minimal.13 Yet bandwidth charges, hardware purchases, customer support labor and utilities charges all vary and will depend on how large you plan to scale your company to.  Thus before investing in this segment, do your due diligence.

Entrepreneurs should also consider this: Bill Bishop has cogently noted numerous times over the past several years that while mainland users are effectively prevented from using some foreign web services, the mainland equivalents are not only more easily accessible and relatively comparable (e.g. same features), but the interconnectivity issues (e.g., latency, bandwidth) with them are relatively muted.14  In other words, assuming you have access to a VPN, it is still more convenient for mainland based users to stream videos on Youku than it would be to stream from Youtube because of the increased bandwidth throughput and reduced lag due to closer proximities to the content delivery networks (CDNs) for Youku-like providers such as ChinaCache.15 David Wolf, a partner at Allison+Partners (a consulting firm) echoed similar reasoning recently in an interview with The Wall Street Journal, noting that “What they [national government] prefer is that Chinese users decide it is just too much trouble and by default use onshore sites, or sites that are mirrored onshore.”16 As a consequence, because of the sheer size of the Chinese-based internet (see Chapter 12 and Chapter 13), there is now “one world, two internets.”17

Climbing the wall

I spoke with security expert David Veksler (see also Chapter 13), CEO of CryptAByte, who has given security workshops and seminars about these issues.18 In his view, “the GFW presents a fundamental problem for domestic researchers.  Because significantly large portions of foreign-based information are blocked and denied, only researchers with VPNs are able to keep up-to-date with their foreign counterparts.  Those without VPNs are left trying to use Google which is frequently blocked and misdirected or Baidu, which outputs few useful or useable results.  Thus they become discouraged, often times quitting and are ultimately unable to do the necessary research – idea investigations – for innovation.”

How does this create opportunities?  According to Veksler, this ties into another Catch-22 that domestic firms find themselves in, this endless cycle of benchmarking and cloning.  That irrespective as to whether or not they want to innovate, they are in a prisoner’s dilemma, “every competitor on the mainland expects to have the lowest costs.  Yet if they increase their research and development – creating higher quality products – consumers do not believe them, because consumers also expect that domestic companies are cutting corners, so why pay extra for a product that is probably just the same as the rest?”

He also likens this dilemma to a game theory scenario: the first domestic company to make that leap into quality is punished because consumers simply do not trust the product quality due to a history of scandals.  Thus any firm that does it is unable to recoup the capital costs of the research and development.  In contrast, foreign companies have spent decades building up their brands and reputations based on quality control programs (e.g., Six Sigma) and as a consequence are readily more trusted on the mainland.  Yet he remains optimistic, “the first domestic company to make a concerted, long-term leap into quality will not only be monetarily successful, but will help end this never ending cycle of benchmarking and cloning.”

Thus Veksler thinks that foreign brand managers, experts like Matt Garner, will be able to find opportunities to work within the entire spectrum of industries as their participants build national and internationalization expansion plans.

Chicken and egg problem

It is hard to measure the impact that an apparatus like the GFW has on productivity and creativity which business start-ups should take into consideration.  Consider Silicon Valley and Moore’s Law.  Much like other projects and mian zi gong cheng, there have been several public initiatives to replicate Silicon Valley in China, such as Zhongguancun in Beijing.  And yet for every successful start-up like iQiyi or venture capitalist like Kai-Fu Lee (see Chapter 12), large quantities of resources have been misallocated towards supercomputers that when installed – while capturing headlines for theoretical peak performances – are unable to be fully utilized because there are not enough trained software engineers to develop the sophisticated machines.19 Similarly despite 2 billion RMB ($320 million) in investment since 2010, Jike, a new search engine developed by People’s Daily (an SOE) has managed to capture a mere 0.0001% marketshare forcing the organization to lay off 20% of its staff.20

Empirically speaking, if central planners were to be the creators of Silicon Valley, they would have created Silicon Valley.  If central planners were to be the creators of Moore’s Law, they would have created Moore’s Law.2122 For example, the Soviets spent decades and relatively large budgets to overtake the West in computing innovations, yet failed at every turn.  In fact, it was not just one or two half-hearted attempts, it was a concerted effort directed from the top.  Mikhail Gorbachev himself made advancements in microprocessor technology a cornerstone part of Perestroikain 1985 (encompassing the 14th Five Year Plan).

Just how much effort was put into their centrally planned machine industry? Consider what the USSR tech industry was like circa 1988:

Machine building is the sector of industry on which Gorbachev is relying to ensure the success of his [Perestroika] strategy.  The hub of Soviet [computing] industry, this complex employs over 16 million workers at more than 9,000 research institutes, design bureaus, and production and enterprises, and is responsible for designing, developing, and producing over one-fourth of the country’s industrial output.  Of the 17 industrial ministries that make up the machine-building complex (detailed in foldout at back of paper), nine — collectively referred to as the defense industry — specialize in military hardware. The other eight produce primarily consumer goods and equipment for investment in the civil sector.23

Gorbachev recognized that “a high-investment, high-growth strategy must, at a minimum, continue through at least the first few years of the period to renew the sector’s capital stock.”  Yet ultimately, the Soviets tried, consumed their capital base, and failed.2425 Instead, hundreds of private companies, entrepreneurs, venture capitalists, designers, and one relatively free market created a semiconductor industry that accounts for the number one export of the United States.26 Furthermore, this is not to say that technological activity will not take place in China, nor that Chinese institutions and researchers will not produce usable technology.  The question is rather, can it be cutting edge and innovative?  And if your firm hopes to tap into the innovation potential of the mainland, how does this impact your firms’ investment?

Many of these artificial technology and science research parks conflate cause and effect.  For example, during World War II, the Allies used Pacific islands as forward operating bases to protect their overseas supply routes.  On many of the islands the Allied forces built airstrips, including one on Vanuatu.  Following post-war demilitarization, most of these islands were vacated as the warring militaries returned home.  On Vanuatu, many of the islanders wanted the supply ships to return and provide modern goods to their pre-industrial society.  As a consequence, the islanders staged “drills” and “marches” with mock soldiers while others attempted to man the airstrips – all under the belief that it is these superficial motions and actions that originally brought the Western supplies.  Richard Feynman dubbed this “cargo cultism” (e.g., a cult that dreamt of Western cargo).27

In November 2012 I spoke with Mark Thornton, an economist at the Ludwig von Mises Institute and an expert in the boom-bust investment cycle.28  According to him, “Research parks are all about inventing technology for commercial and other purposes. Generally we are speaking of higher order goods, the types of goods associated with the boom phase of the business cycle. Therefore we would expect that research park projects tend to be established during booms when profits are high, the cost of capital is low, and where retained earnings are more than sufficient to support additional projects. If research parks are established at or near the peak in the business cycle then it would be wise to avoid contracting with research parks that have few tenants.  Traditionally one of the main benefits of research parks is synergy.  If your research park has no tenants then you do not have the type of synergies that successful research parks generate.  New companies, new technologies and products, as well as successful research parks (e.g. Stanford Research Park and Research Triangle Park) tend to get their starts during bad economic times.  During recessions land, labor, capital are cheaper and budding entrepreneurs are more abundant.”  In economic terms, higher order goods are goods used to produce consumer goods (e.g., those which require a long-term investment such as building a factory which in turn creates consumer goods).29

Similarly, many of these research parks and endeavors – not just in China – arguably exhibit patterns of modern-day cargo cultism.  Thornton noted that, “The next Silicon Valley will not look like Silicon Valley.  It will have some new features and not have all the same features as Silicon Valley.  You cannot just build “it” and expect them to come.  Silicon Valley is more than just Stanford Research Park and Stanford University. There are tangible and intangible factors that matter. They include things like the weather, demographics, culture, and relatively limited regulatory impact from the government. Even some factors we just do not know. Government can subsidize research parks but it takes a free market and entrepreneurs to actually weave the fibers of something extremely complex like Silicon Valley.”

In fact, in the US, nearly every state has erected several tech parks with the hopes of “creating” another Silicon Valley; there are dozens of research and technology centers across the country.  This raises the question: if you build it, will they (the creative classes) come?

In February 2013 I spoke with Becky Wu a native of Jiangsu province and a project manager at Xi-Tong Scientific & Technology Industrial Park located in Nantong, Jiangsu province.30 The primary task of her job is attracting and relocating foreign firms so that they will build and setup operations in the industrial park.  According to her, “we provide incentives and subsidies to attract firms from abroad.  For example, if land prices were with 230,000 RMB per mu, depending on how promising the project is and what industry your firm is in we can lower the price to 200,000 RMB or even 150,000 per mu.  This helps attract firms, enticing them to construct their new offices in the park.  We will also provide free temporary offices for new companies for up to 6 months while their new office is being built.  The utilities are also free of charge as well.”  As noted earlier in Chapter 3, a mu is 1/6th of an acre.

Wu also explained that there are other rebates and training subsidies that firms can receive.  She noted that, “we also offer new companies subsidies for research and to train personnel that can be allocated and spent without strings attached.  For example, we can provide up to $1,000 a year per person, up to 10 people to help offset training and research costs.  In terms of income taxes, we provide rebates to specific workers, typically managers and high-level executives for 3-5 years.  The way this works is that if you have to pay 100 RMB in taxes, 60% goes to the central government, 8% goes to the provincial government, the remaining portion goes to Nantong, thus we at the park can reimburse the remaining 32% back to you.”  Clients such as Caterpillar, BIC, Accuma and Kopron have taken advantage of these incentives over the past several years.

Does the return-on-investment pay for the capital expenditures which were originally expended?  While it is impossible to say yes or no for all the cases, what can be said is that the GFW itself probably does not create innovation, foster creativity or act as an incentive to attracting outside talent.  If it did, the Chinese computing industry would not be reliant on Western semiconductors, Western software and foreign know-how.31 And as a consequence, mainlanders conducting research are left using a virtual straw in order to access, view and communicate with the outside world.

How is this relevant and how does this affect your company?  Without virtual openness to new ideas, the domestic, indigenous engineering industries – while not autarchic – will probably always be laggards due to what Veksler noted above (e.g., getting frustrated and quitting).  To this point, last year the American Chamber of Commerce in Beijing conducted a survey of its members, “nearly three-quarters of about 300 businesses it surveyed said unstable Internet access impedes their efficiency. About 40% said China’s censorship efforts have a negative business impact.”32 Similarly, economist Arthur Kroeber, founder of Dragonomics research noted in March 2013 that one obstacle to growth is the GFW.  In his view, innovation in the modern world today comes from “the sharing of knowledge and information across a variety of fields.  Innovation comes when you take knowledge in one area and it migrates over to another area and someone comes up with a new way of using it.  China seems to have a political system that mentally at its core is opposed to those networks ever becoming viable.”33 Thus, in addition to the issues raised in Chapter 15, this obstacle is another consideration that all firms looking to recruit talent must take account for.34

While there are occasional opportunities and projects like “1,000 talents” (mentioned in Chapter 9 and Chapter 15) that provide monetary and other perks and incentives to relocate, these well-intentioned plans may be unable to offset the hurdles created by the GFW and as a consequence there has been a “brain drain” that all firms and HR departments should be aware of.35

Yet to be even handed, Larry Chang mentions that he works within this system on purpose because it is “an untapped opportunity.”  He only hires fresh mainland graduates with the sole purpose of building an indigenous software industry.  And in his opinion, with more than 6 million college students graduating each year, there are bound to be creative, outside-the-box thinkers.  Similarly, at the 2013 Unleashing Innovation conference recently held in Singapore, Ya-Qin Zhang, chairman of Microsoft’s Asia Pacific research and development group, noted that “Chinese engineers are well equipped to produce the kind of innovative work that their more illustrious American rivals are renowned for” and continued with, “[t]he scale of innovators and the scale of the market will converge and eventually make China a key [innovation] center in the region.”36 Thus it may just be a matter of time before the right combination of inputs brings about the transition up the value chain as described in Chapter 7.

Opportunities in the rough

Again, even with these seemingly insurmountable challenges there are also opportunities.  For example, as I noted in Chapter 17, foreign architects are in high demand to help build and design buildings, bridges and even office parks.  Perhaps your firm can find new revenue streams by helping to build out domestic content delivery networks (CDNs) and cloud computing initiatives that are part of these technology parks.  As I mentioned in Chapter 13, according to IDC, $286 million was spent on cloud-computing infrastructure in China in 2011 and this is expected to increase to $1 billion by 2016.37 And this segment is quickly professionalizing, for example, ChinaCache is the largest CDN on the mainland with 53% of the marketshare.38 It was initially funded by the likes of Intel and is now listed on NASDAQ.

Another opportunity is with corporate VPNs.  While the individual market may seem like a logical way to establish a steady revenue stream, according to David Veksler, corporate enterprises – both domestic and foreign – will eventually want and need to have VPNs to secure their communication with clients, vendors and essentially anyone.  Irrespective of the GFW, Veksler’s own estimate is that there is an unlimited amount of potential growth for VPNs because very few domestic firms currently recognize the need to protect their assets.  But Veskler suggests, “this attitude will probably change, due to the increasing security vulnerabilities publicly acknowledged by even the largest of enterprises.”

But there is also a challenge regarding foreign owned and run VPNs on the mainland, as the Global Times recently quoted Fang Binxing (father of the GFW as noted above) that, “[u]nregistered VPN service providers are not protected by Chinese laws, and any company running a VPN business should realize they have a responsibility to register.”39 More directly, an employee in the Ministry of Industry and Information Technology pointed out in the same report that, “only Chinese companies and Sino-foreign joint ventures can apply to establish a VPN business.”  This is not to say that is illegal to connect to a VPN outside of the mainland.  Currently there are no laws which prohibit users in China from connecting to an overseas VPN.40

In December 2012 I spoke with an American executive at a large IT company that provides dedicated internet connections to enterprises and institutions primarily in Tier 1 cities.  According to him, “no foreign IT company and few domestic companies advertise their VPN services yet many of them will bundle it as part of a package to corporate clients.  Furthermore, Chinese regulators typically permit VPNs so as long as it is privately – not publicly – accessible as well as the stipulation that consumers use leased-lines.  A typical dedicated leased-line will cost over 3,000 RMB a month for 1 mb/s, this scales linearly (e.g., if you need 4 mb/s you are charged around 12,000 RMB), thus this option is typically out of reach by most consumers outside of the corporate and foreign communities.  In addition, you can find a number of local firms that will provide point-to-point VPN services within the mainland.  So if you are an expat that works for a foreign company that operates a VPN network elsewhere, then you will be able to securely connect from your local VPN to their secure environment overseas.”

Similarly, as an entrepreneur you can utilize these tech parks in China since they are not going to disappear overnight, if ever.  For example, Larry Chang merged all company divisions under one roof in a research park located on a campus of a local college in Changning, Shanghai.  His firm was provided incentives such as reduced rental rates for doing so.  Similarly, Richard Qi mentioned that a new area in Shanghai called Cloud City – a tech park – provides perks and benefits to foreign software, engineering and IT firms.  For example, Cloud City provides discounted office property, assistance in communicating with governmental organizations, stipends form the government and as the name-sake suggests, access to cloud services.  Prior to relocating to this tech park, Qi mentioned that it was often difficult as a foreign service provider to issue invoices because of unclear laws (e.g., Shanghai and other municipalities are currently transitioning from a business tax to a VAT) and it was hard to find the government contacts needed to settle these transactions.  In addition, perhaps your software or semiconductor firm can also take advantage of these inducements created by the 2011 policy which provides a tax holiday for several years, reduces the subsequent tax rates and provides exemptions on profits.41

Takeaway:  Due to a variety of regulations and policies on the mainland, certain telecommunication restrictions have germinated into a formidable barrier called the GFW.  And with several million technologically-inclined consumers familiar with Western tastes and styles, there exists a potentially new customer base for VPN service providers.  Yet just because there is potential for growth does not necessarily mean that the potential customers will purchase your goods and services (e.g., “if you build it, will they come?”).


Endnotes:

  1. According to Fang Binxing, the ‘Father of the Great Firewall,’ it was “reportedly launched in 1998 [and] came online about 2003.”  See Great Firewall father speaks out from Global Times []
  2. Splinternet Behind the Great Firewall of China from Association for Computing Machinery []
  3. To bypass copyright restrictions, VPN uptake has increased over the past several years in several Western countries, as consumers move to alternative methods for downloading copyrighted content.  According to a study from Lund University in Sweden, there has “been a 40% rise in the number of 15 to 25-year-olds using such [VPN] services since 2009.”  See File-sharers look to VPNs to overcome Pirate Bay ban from BBC []
  4. Even with encryption algorithms like AES, third parties which have direct access to even one end of a data stream can conduct packet sniffing and other “side channel” attacks. []
  5. See Five Myths about the Chinese Internet from Foreign Policy and Florida pet spa mystery link to China’s great firewall from New Scientist []
  6. Great Firewall father speaks out from Global Times []
  7. In January 2013, Han Weili, a software instructor at Fudan University in Shanghai publicly solicited applications for employment to improve the GFW.  In his view there are two problems with the GFW technology, “The first is a lack of transparency in strategy, the second is that Great Firewall strategy execution has a false-report rate that is too high.”  See Great Firewall Engineer Han Weili Calls for Job Applications from Fei Chang Dao []
  8. In 2011, the US embassy in China issued more than 160,000 student visas for Chinese students to study at American schools.  Yet a November 2012 report from Open Doors notes that the actual number is even higher, 194,029.  See Ten Years of Rapid Development of China-US Relations from Xinhua and Students from China add $5b to US economy from China Daily []
  9. Chinese tourists spend more in US in 2011 from China Daily []
  10. No, Facebook does not have 63.5 million active users in China from The Next Web []
  11. Approximately half of World of Warcraft’s 10-12 million userbase is estimated to be from mainland China.  See “Gold Farming”: Real-World Production in Developing Countries for the Virtual Economies of Online Games by Richard Heeks and Converting the Virtual Economy into Development Potential: Knowledge Map of the Virtual Economy from the World Bank []
  12. Disclosure: I do not currently have any stakes in these products, services or companies.  See Testing five VPNs that’ll get you back on YouTube, Facebook in China from c|net []
  13. A user can remotely set up their own VPN practically anywhere using software such as OpenVPN.  The primary key issue is locating a computer outside of the mainland where it can be installed on and reliably connected to. []
  14. Sinocism []
  15. To better understand the importance of CDNs see, Google and Netflix Make Land Grab On Edge Of Internet from Wired []
  16. China’s ‘Wall’ Hits Business from The Wall Street Journal []
  17. See One World, Two Internets by Bill Bishop and Iran’s network in a bottle from The Boston Globe []
  18. CryptAByte []
  19. According to one estimate regarding software application investment for supercomputers in China, “Less than 10% of supercomputing funding goes to developing such applications, said Chinese researchers who complain that political leaders press them to build headline-grabbing new machines rather than focus on whether they are used to their full capabilities.”  See China’s Not-So-Super Computers from The Wall Street Journal []
  20. See People’s Search Engine Denies Layoff Rumors; Says More Jobs Open from Caijing, Jike’s attempt to censor news about its 0.0001% market share has backfired from Shanghaiist and You’ve been Jiked! from China Media Project []
  21. Debt as Tall as Dubai, or How the Singularity Is Not a Guaranteed Phenomenon by Tim Swanson []
  22. They cannot a priori due to the economic calculation problem.  See Economic Calculation In The Socialist Commonwealth by Ludwig von Mises []
  23. The Soviet Machine-Building Complex: Perestroyka’s Sputtering Engine from the Office of Soviet Analysis published by the Directorate of Intelligence []
  24. Throughout its existence the Soviet Union tried to incorporate technology in its Pyatiletka — Five Year Plans.  They even tried to recreate Silicon Valley through the construction of numerous science and research parks called Naukograd.  Numerous other countries have also tried to emulate the success of the Bay Area with little measurable return-on-investment; this includes Silicon Taiga in Novosibirsk.  The Soviet Union was unable to incubate something akin to Moore’s Law for the same reason the Soviet Union ultimately failed: without prices, you cannot make efficient allocation decisions.  Prices only arise from market interactions, through profit and loss — which signal to entrepreneurs when to buy, sell, trade, and invest capital.  Without this organic knowledge Soviet planners were left using arbitrary coefficients to plug into their various economic models with the net result: planned chaos.  See Planned Chaos by Ludwig von Mises. []
  25. One frequently cited myth regarding Japan is that it was successful in its attempts to centrally plan scientific innovation.  This is untrue.  See The Fifth Generation Fallacy by J. Marshall Unger.  See also Chapter 9 in Animal Spirits with Chinese Characteristics by Mark DeWeaver []
  26. According to the Semiconductor Industry Association, “three quarters” of all semiconductor design and manufacturing takes place in the United States and that 82% of semiconductor sales are outside the United States.  See America’s #1 Export Industry Applauds Passage of Free Trade Agreements from the Semiconductor Industry Association []
  27. See Cargo Cult Science by Richard Feynman and In John They Trust from Smithsonian []
  28. Skyscrapers and Business Cycles by Mark Thornton []
  29. See Chapter 1 in Principles of Economics by Carl Menger and Chapter 16 in Human Action by Ludwig von Mises []
  30. Xi-Tong Scientific & Technology Industrial Park []
  31. According to recent reports, Chinese policy makers are attempting to build a 100-petaflop supercomputer which would be five times faster than the current record holder (Titan).  As part of this plan, Zhang Yunquan, a professor at the Institute of Software Chinese Academy of Sciences, noted that domestically designed chips may be used.  These domestic chips, called Loongson are based on MIPS, a chip design developed by a Sunnyvalle-based technology firm (MIPS Technologies).  Similarly, Chinese policy makers are frustrated by the fact that Android (which is managed by Google) has the lion’s share of marketshare and would prefer to have a domestic, homegrown OS used by smartphone makers instead.  See China is building a 100-petaflop supercomputer from IT World, China’s godson gamble from IEEE SpectrumWhy China Can’t Make Its Own Mobile OS from Tech In Asia and Google controls too much of China’s smartphone sector: ministry from Reuters []
  32. China’s ‘Wall’ Hits Business from The Wall Street Journal []
  33. Economist: China Plenty Creative, Just Not in Right Ways from The Wall Street Journal []
  34. China’s self-defeating war with information by Andy Yee []
  35. See Rich Chinese want to buy happiness — by emigrating from Los Angeles Times and Wary of Future, Professionals Leave China in Record Numbers from The New York Times []
  36. Microsoft’s Zhang Sees China as Asia’s Innovation Center from The Wall Street Journal []
  37. Cloud computing investment ‘to hit $1b’ from China Daily []
  38. ChinaCache investor relations []
  39. Foreign-run VPNs illegal in China: govt from Global Times []
  40. Adding Some Key Facts In WSJ.com’s China’s Internet ‘Wall’ Hits Business Article from VPN Instructions []
  41. China offers new incentives to further boost software and semiconductor industries by Peng Tao []