Massively scalable ‘blockchainified’ databases (BigchainDB) … next internet phase … only 100 public databases to organise ‘Big Data’ and every ‘thing’ globally

Massively scalable ‘blockchainified’ databases (BigchainDB) … next internet phase … only 100 public databases to organise ‘Big Data’ and every ‘thing’ globally

Massively scalable 'blockchainified' databases are the next phase of the internet. In its current form, Blockchain is limited in terms of size, scalability and latency and is not a world changing technology. The next incarnation of Blockchain will not be Blockchain, but Blockchainified databases which overcome these limitations and are a profound technology that can scale and change the world. As few as 100 blockchainified databases (by sector or country) could organise (not necessarily store) every 'thing' around the world. This is a Zero Marginal Cost technology which means they could be deployed at practically no cost and could become a free service globally.

This article introduces the applications for Blockchainified databases and then provides a comprehensive overview of the technology. Wisdom Networks bring Wisdom to 'things'. Blockchainified databases store and transact 'things'. The Appendix below contains screenshots and a comprehensive overview of the 'things' in multiple sectors across society.

Massively scalable ‘blockchainified’ databases (BigchainDB) … next internet phase … only 100 public databases to organise every ‘thing’ globally

Blockchain is a distributed database that has gained notoriety in one if its application as a cryptocurrency or Bitcoin. The early incarnation of the Blockchain is, however, severely limited (size, latency and scalability) that renders it unusable in its current form. The next generation of Blockchain is not blockchain at all. The next generation begins with Massively Scalable Databases that are used as the foundation of the world’s largest web sites and adds Blockchain functionality. The outcome is a Blockchain enabled massively scalable database. This database will operate just like the internet. As computers are added to the collective internet, the networks scales and support every increasing users. As database servers (or computers) are added to the collective Blockchainified database, the collective capacity and reach of the database scales. This means that one database could store all the data for an entire country or global sector. 50 wisdom networks could organise the world. Similarly, 100 Blockchain-enabled massively scalable databases could organise society’s ‘things’ around the world in every country and sector.

Blockchainified databases can be configured in a variety of ways. A community model (similar to the internet) would mean that (1) no one node or organisation controls owns the data or the network of databases, and (2) secure exchange of information between 2 parties independent of central or facilitating nodes. Other models could involve private ownership of community ‘things’. It is possible that (like the internet), the storage and organisation of things becomes a global and collective endeavour.

Blockchainified databases reduce society complexity exponentially by consolidating every 'thing' in one distributed network and simultaneously increase transparency, accountability and engagement with all citizens exponentially. Public blockchainified databases would be even more profound and could provide a single network of society's 'things' at Zero Marginal Cost. Data previously lost and isolated in vertical silos become available to all in a public, distributed horizontal network accessible to all.

The community is crowd creating evolution from Industrial Economy to Network Society using concurrent processes. Community crowd created internet (collective connectivity). Community crowd created world wide web (collective information). Community crowd created social networks (collective community). Everyone knows everyone else. Community crowd creates open data (collective things) for every one. Industrial Economy fragments and centralises things in silos. Blockchainified databases overcomes these limitations and stores and transacts society's things in a distributed database that provides collective accessibility, transparency and governance across society. Everyone has access to every thing. Community can crowd creates wisdom (collective knowledge and prioritised action). The Industrial Economy is inefficient and ineffective due to its physical technologies that can't scale and the underlying tools of centralisation and fragmentation. Wisdom Networks overcome these limitations and, in conjunction with other Zero Marginal Cost technologies, can organise collective collaboration, collective knowledge and collective wisdom for every 'thing' in sectors, countries and the Global Village.

Zero Marginal Cost technologies enable the Network Society and Global Village. Zero Marginal Cost techologies include internet (collective connectivity), wisdom networks (collective collaboration, knowledge and wisdom), mobile devices (collective universal access), blockchain-enabled massively scalable databases (collective 'things'). Wisdom Networks bring Wisdom to 'things'. Blockchainified databases store and transact 'things'.

BigchainDB: A Scalable Blockchain Database

For your convenience, we will extract the key parapagraphs from a profound whitepaperb by BigchainDB on blockchain enabled massively scalable databases.

Extract: BigchainDB: A Scalable Blockchain Database (DRAFT), February 10 2016


This paper describes BigchainDB. BigchainDB fills a gap in the decentralization ecosystem: a decentralized database, at scale. It points to performance of 1 million writes per second throughput, storing petabytes of data,and sub-second latency. The BigchainDB design starts with a distributed database (DB), and through a set of innovations adds blockchain characteristics: decentralized control, immutability, and creation & movement of digital assets. BigchainDB inherits characteristics of modern distributed databases: linear scaling in throughput and capacity with the number of nodes, a full-featured NoSQL query language, efficient querying, and permissioning. Being built on an existing distributed DB, it also inherits enterprise-hardened code for most of its codebase. Scalable capacity means that legally binding contracts and certificates may be stored directly on the blockchainified database. The permissioning system enables configurations ranging from private enterprise blockchainified databases to open, public blockchainified databases.

We introduce a concept called blockchain pipelining, which is key to scalability when adding blockchainlike characteristics to the distributed DB. We present a thorough description of BigchainDB, an analysis of latency, and preliminary experimental results. The paper concludes with a description of use cases. (Page 1)

A Public BigchainDB

Decentralization technology has potential to enable a new phase for the Internet that is open and democratic but also easy to use and trust. It is intrinsically democratic, or at the very least disintermediating. It is also trustworthy: cryptography enables the conduct of secure and reliable transactions with strangers without needing to trust them, and without needing a brand as proxy. The discourse is around benefits in both the public and private sector. In the public sector, the most obvious benefit is in the future shape of the Internet and especially the World Wide Web. These technologies have fundamentally reshaped society over the past two decades. The 90s Web started out open, free-spirited, and democratic. In the past 15 years, power has consolidated across social media platforms and the cloud. People around the world have come to trust and rely on these services, which offer a reliability and ease of use that did not exist in the early Internet. However, these services are massively centralized, resulting in both strict control by the central bodies and vulnerability to hacking by criminals and nation-states. Decentralization promises a large positive impact on society. An antidote to these centralized services and concentration of power is to re-imagine and re-create our Internet via decentralized networks, with the goal of giving people control over their data and assets and redistributing power across the network.(Page 45)

Toward a decentralised application stack

The introduction of Bitcoin [1] has triggered a new wave of decentralization in computing. Bitcoin illustrated a novel set of benefits: decentralized control, where 'no one' owns or controls the network; immutability, where written data is tamper-resistant ('forever'); and the ability to create & transfer assets on the network, without reliance on a central entity.

The initial excitement surrounding Bitcoin stemmed from its use as a token of value, for example as an alternative to government-issued currencies. As people learned more about the underlying blockchain technology, they extended the scope of the technology itself (e.g. smart contracts), as well as applications (e.g. intellectual property). With this increase in scope, single monolithic 'blockchain' technologies are being re-framed and refactored into building blocks at four levels of the stack: 1. Applications; 2. Decentralized computing platforms ('blockchain platforms'); 3. Decentralized processing ('smart contracts') and decentralized storage (file systems, databases), and decentralized communication 4. Cryptographic primitives, consensus protocols, and other algorithms. (Page 2)

Centralisation can provide technology efficiency and economies of scale and agility. However these come at the expense of fragility. Even the distributed, synchronised cloud platforms like AWS, Azure, Digital Ocean, come with risks of single-operator systems.

Natural communities are resilient to extinction because they can accommodate the loss of small numbers without losing the functioning of the community as a whole. Communities function as a 'mesh' of interweaving relationships and capabilities with a common vision or purpose. Wisdom Networks are working on the implementation of mesh networks where multiple paths, for process and data, are available and survive the loss of some nodes. The internet works in much the same way but is significantly more fragile - the whole of the current (2016) internet is based around 9 main networks. Wisdom Network participants will create dynamic mesh networks, sometimes bypassing infrastructure providers entirely, to create a robust, always on, network of the future.

The distributed network is made useful with the distributed stack that puts the network to work doing useful things for communities.

Blockchains and Databases

We can frame a traditional blockchain as a database (DB), in the sense that it provides a storage mechanism. If we measure the Bitcoin blockchain by traditional DB criteria, it’s terrible: throughput is just a few transactions per second (tps), latency before a single confirmed write is 10 minutes, and capacity is a few dozen GB. Furthermore, adding nodes causes more problems: with a doubling of nodes, network traffic quadruples with no improvement in throughput, latency, or capacity. It also has essentially no querying abilities: a NoQL1 database. In contrast, a modern distributed DB can have throughput exceeding 1 million tps, capacity of Petabytes and beyond, latency of a fraction of a second, and throughput and capacity that increases as nodes get added. Modern DBs also have rich abilities for insertion, queries, and access control in SQL or NoSQL flavors; in fact SQL is an international ANSI and ISO standard. (Page 2-3)

The need for scale

Decentralized technologies hold great promise to rewire modern financial systems, supply chains, creative industries, and even the Internet itself. But these ambitious goals need scale: the storage technology needs throughput of up to millions of transactions per second (or higher), sub-second latency2, and capacity of petabytes or more. These needs exceed the performance of the Bitcoin blockchain by many orders of magnitude. (Page 2-3)

Wisdom Networks are designed from the ground up to be massively scalable to meet the needs of large population groups and globally distributed communities. Such networks will have millions of people, and each person could be participating in multiple networks. The WN database, of whatever form, will process billions of transactions per minute- if Twitter is a 'flood' then Wisdom Network is a data Tsunami ….. 24x7.

BigchainDB: Blockchain meets Big Data

This paper introduces BigchainDB, which is for database-style decentralized storage: a blockchainified database. BigchainDB combines the key benefits of distributed DBs and traditional blockchains, with an emphasis on scale, as Table 1 summarizes.

We built BigchainDB on top of an enterprise-grade distributed DB, from which BigchainDB inherits high throughput, high capacity, low latency, a full-featured efficient NoSQLquery language, and permissioning. Nodes can be added to increase throughput and capacity.

BigchainDB has the traditional blockchain benefits of decentralized control, immutability, and creation & transfer of assets. The decentralized control is via a federation of nodes with voting permissions, that is, a super-peer P2P network [2]. The voting operates at a layer above the DB’s built in consensus. Immutability / tamper-resistance is via an ordered sequence of blocks where each block holds an ordered sequence of transactions; and a block’s hash is over its transactions and related data, and the previous block’s hash; that is, a block chain. Any entity with asset-issuance permissions can issue an asset; any entity with asset-transfer permissions and the asset’s private key may transfer the asset. This means hackers or compromised system admins cannot arbitrarily change data, and there is no single-point-of-failure risk. Scalable capacity means that legally binding contracts and certificates may be stored directly on the blockchain DB. The permissioning system enables configurations ranging from private enterprise blockchain DBs to open, public blockchain DBs.
(Page 3-4)

Layering BlockChain principles on an 'eventually consistent' database or data layer means that the type of datastore used for bigdata can be something attached to a WN node. The data flowing through the network can be forked out to a massive scale repository for BigData tools to work on the whole pool of data. The analysis and insights can then be fed back into the Wisdom Network as another form 'collective wisdom' that further enhances the behaviour of the underlying WN itself as well as influencing the behaviour of the people and organisations participating in the network.

Forms of AI could be applied to the network both on the streams of ideas, decisions, data as well as using BigData analysis. In addition human moderators could monitor their own WN sectors and provide guidance in ways that machines and AI could not.

BigchainDB in the decentralization ecosystem

Figure 1 illustrates how BigchainDB can be used in a fully decentralized setting, or as a mild extension from a traditional centralized computing context. BigchainDB is complementary to decentralized processing / smart contracts (e.g. Ethereum VM [3][4] or Enigma [5][6]), decentralized file systems (e.g. IPFS [7]), and communication building blocks (e.g. email). It can be included in higher-level decentralized computing platforms (e.g. Eris/Tendermint [8][9]). It can be used side-by-side with identity protocols, financial asset protocols (e.g. Bitcoin [1]), intellectual property asset protocols (e.g. SPOOL [10]), and glue protocols (e.g. pegged sidechains [11], Interledger [12]). Scalability improvements to smart contracts blockchains will help fully decentralized applications to better exploit the scalability properties of BigchainDB. BigchainDB works in compliance with more centralized computing blocks too. One use case is where decentralizing just storage brings the majority of benefit. Another use case is where scalability needs are greater than the capabilities of existing decentralized processing technologies; in this case BigchainDB provides a bridge to an eventual fullydecentralized system.
(Page 1)

Bitcoin Scalability Issues

Bitcoin has scalability issues in terms of throughput, latency, capacity, and network bandwidth.

Throughput. The Bitcoin network processes just 1 transaction per second (tps) on average, with a theoretical maximum of 7 tps. It could handle higher throughput if each block was bigger, though right now making blocks bigger would lead to size issues (see Capacity and network bandwidth, below). This throughput is unacceptably low when compared to the number of transactions processed by Visa (2,000 tps typical, 10,000 tps peak), Twitter (5,000 tps typical, 15,000 tps peak), advertising networks (500,000 tps typical), trading networks, or email networks (global email volume is 183 billion emails/day or 2,100,000 tps). An ideal global blockchain, or set of blockchains, would support all of these multiple high-throughput uses.

Latency. Each block on the Bitcoin blockchain takes 10 minutes to process. For sufficient security, it is better to wait for about an hour, giving more nodes time to confirm the transaction. By comparison, a transaction on the Visa network is approved in seconds at most. Many financial applications need latency of 30 to 100 ms.

Capacity and network bandwidth. The Bitcoin blockchain is about 50 GB; it grew by 24 GB in 2015. It already takes nearly a day to download the entire blockchain. If throughput increased by 2,000x to Visa levels, the additional transactions would result in database growth of 3.9 GB/day or 1.42 PB/year. At 150, 000 tps, the blockchain would grow by 214 PB/year (yes, petabytes). If throughput were 1M tps, it would completely overwhelm the bandwidth of any node’s connection, which is counterproductive to the democratic goals of Bitcoin. (Page 6)

BigchainDB Principles

Rather than trying to scale up blockchain technology, BigchainDB starts with a 'big data' distributed database, and adds blockchain characteristics. It avoids the technology choices that plague Bitcoin, such as full replication and broadcast communication. We built BigchainDB on top of an enterprise-grade distributed DB, from which BigchainDB inherits high throughput, high capacity, a full-featured NoSQL query language, efficient querying, and permissioning. Nodes can be added to increase throughput and capacity.(Page 12)

BigChainDB - High level description

We focused on the adding the following blockchain features to the DB: 1. Decentralized control, where 'no one' owns or controls a network; 2. Immutability, where written data is tamper-resistant ('forever'); and 3. The ability to create & transfer assets on the network, without reliance on a central entity.

Decentralized control is achieved via a DNS-like federation of nodes with voting permissions. Other nodes can connect to read, and propose transactions; this makes it a super-peer P2P network. The voting operates at a layer above the DB’s built in consensus. Quorum is a majority of votes. (Page 13)

Choice of distributed database

The BigchainDB design is flexible enough to have been built on top of a wide variety of existing distributed DBs. Of course, we had to choose a first one to build on. To select which, we first did benchmarking, then added additional criteria. There are > 100 DBs to choose from, listed for example at [50] and [51]. This was our shortlist: Cassandra, HBase, RediS, Riak, MongoDB, RethinkDB, and ElasticSearch.

In the future, we envision a variety of distributed databases being 'blockchainified' according to the approach of this paper. Every relational database, document store and graph store on the planet might someday have a blockchain version.(Page 31-32)

The key to usability will be in having mechanisms that allow for enablement of user initiated events in the WN that are not then tied up waiting for data consistency in the entire network. The principle required is a form of 'fire and forget' where as long as the nearest node gets the transaction ok and validates it, then the WN user can continue to work.

BigChainDB Capacity

Each t2.medium provides 48 TB of storage, so the total capacity of N nodes is N times 48 TB; with 32 nodes, the total capacity is 1536 TB, i.e. more than a Petabyte. For quick reference, Figure 10 shows how total capacity depends on the number of nodes.

(Page 33)

Public vs. Private Bitcoin

The way that BigchainDB is designed, permissioning sits at a layer above the core of the design. However, we have already seen many questions about "private vs. public" versions of BigchainDB, privacy, and authentication. In our view, a rich permissioning framework is the technology foundation. This section explores permissions, roles, private BigchainDBs, and privacy. It then has an extended section on a public BigchainDB, which we believe is tremendously important. It finally discusses authentication and the role of certificate-granting authorities.(page 42)

Private BigchainDB

A private BigchainDB could be set up amongst a group of interested parties to facilitate or verify transactions between them in a wide variety of contexts, such as exchanging of securities, improving supply chain transparency, or managing the disbursement of royalties. (Page 43)

Public BigchainDB

A BigchainDB can be configured to be more public, with permissioning such that anyone can issue assets, trade assets, read assets, and authenticate. (Page 45)

A Public BigchainDB - Example permissioning

Table 5 describes permissioning of a public BigchainDB instance. Here, BigchainDB is configured such that each User (column 6) can do anything, except for sensitive roles such as voting, administration, and authentication. Critically, Users can issue any asset (column 6, row 5) and read all assets (column 6, row 14); this is one of the defining features of an open blockchain.
At the core of a public BigchainDB are the 'Caretakers': organizations with an identity that has a 'Voting Node' role. An identity with that role can vote to approve or reject transactions, and can vote whether to assign the 'Voting Node' role to another identity. (Note: an organization can have more than one identity, so a Caretaker could have two or more identities with a 'Voting Node' role.) To start, the public BigchainDB will have five identities with the Voting Node role: three held by ascribe and two held by other organizations chosen by ascribe. That is, the public BigchainDB will start with three Caretakers: ascribe and two others. From there, additional Caretakers will be selected and added to the federation by existing Caretakers. Caretakers will have divergent interests to avoid collusion, but must have one thing in common: they must have the interests of the Internet at heart. In choosing Caretakers, there will be a preference to organizations that are non-profit or building foundational technology for a decentralized Internet; and for diversity in terms of region, language, and specific mandate.

The right organizational structure will be critical to the success of a public BigchainDB. Governance issues have plagued the Bitcoin blockchain. We can take these as lessons in the design of a public BigchainDB. We are consulting with lawyers, developers, academics, activists, and potential Caretakers to develop a strong, stable system that is transparent enough to be relied on and flexible enough to meet the needs of the network. Ultimately, the public BigchainDB will operate entirely independently under its own legal entity. It will choose its own Caretakers and set its own rules—but it will always work toward the long-term goal of a free, open, and decentralized Internet.(Page 46-47)


This paper has introduced BigchainDB. BigchainDB fills a gap in the decentralization ecosystem: a decentralized database, at scale. BigchainDB performance points to 1 million writes per second, sub-second latency, and petabyte capacity. It has easy-to-use and efficient querying. It features a rich permissioning system that supports public and private blockchains. It is complementary to decentralized processing technologies (smart contracts) and decentralized file systems, and can be a building block within blockchain platforms.(Page 53)

horizontal break


Blockchain and Wisdom Networks are different, but both organise 'things'

Blockchain and Wisdom Networks are very different, but do the same thing. They manage things in society. The blockchain is a means to manage distributed, non-repudiable transactions and crowd creates the resulting register of ‘things’. Blockchain manages flows and stocks. The wisdom network focuses organises ‘things’ (data, information, community, collaboration, knowledge, wisdom) to reveal wisdom and focus the wisdom of crowds on outcomes.

Block Chain and Wisdom Networks are complementary. Wisdom Networks add value to things in blockchain by organising wisdom around every 'thing'. The Blockchain provides a source of ‘things’ to link to and organise.

Zero Marginal Cost Networks (internet, blockchain, wisdom networks etc) displace millions of ‘Industrial Economy’ fragmented technologies and hierarchies that manage ‘things’ by organisation, sector or country across the world at zero cost and with global reach. For example, one 'Equity Market' block chain could crowd create transactions and registers for securities transactions worldwide. One Equity Market Wisdom network could manage the global equity market. One Blockchain could manage land titles globally. Blockchain is another type of global, horizontal Zero Marginal Cost network that manages ‘things’ globally.

Each sector may consist of 50 things. Each country may consist of a few hundred things. Some of these things will be managed by Blockchain technology at either a global country level. The use of Blockchain depends on each ‘thing’. Land and securities need Blockchain. Holes in the road ‘things’ that need to be fixed do not.

Blockchain and Wisdom Networks are part of the inevitable evolutionary ‘More with less’ evolution that means less adminstrative and people are required within the services sector. Like previous Industrial Revolutions, the Third Industrial Revolution will result in a jump in productivity and significant decline in employment to achieve the same outcomes.

More: Blockchain and Wisdom Networks both have global reach and manage society 'things' at Zero Marginal Cost ... 20+ examples of Blockchain applications in society

What does wisdom actually look like? Screenshots!

Wisdom Networks organise society's 'things' within networks in the Internet of Everything (IOE) via people and their mobile devices. Please find some screenshot examples below of 'things' and 'books' in sectors. Please note the tabs at the top of the screen allow a person to switch between various sectors. Each sectors contains different 'things', but the underlying method of linking 'things' into 'books' and sharing them remains the same. One method of organising past, present and future means a dramatic reduction in complexity across society and a scalable solution to organise society and the global village at zero marginal cost.

'things' and 'books'

'things' and 'books'

'things' and 'books'

'things' and 'books'

Management & Organisation
'things' and 'books'

Equity Market
'things' and 'books'

Sector pages and presentations contain further screenshots and workflow diagrams for each sector.

1 consolidated method to exchange 'wisdom'

Each sector is a subset of the Network Society. Within a Network Society, sectors and countries use a common method ( things = books = sectors = countries = global village) to exchange wisdom (data, information, community, collaboration, knowledge wisdom). There are exponential benefits from using a common method of exchanging wisdom across individuals, families, sectors, countries, regions and the global village, rather than fragmenting wisdom and sectors using an Industrial Economy model. Network Society development model enables citizens to actively participate in sector governance and service delivery within the community, rather than citizen non-participation because sector governance and service delivery is isolated in hierarchies outside the community.

NOTE: Society already uses the 'things' into 'books' method to organise their social life. Billions of people have been trained in this method. It just needs to be enabled in other sectors.

One method across sectors delivers exponential benefits

People link 'things' into 'books'
and share them across society

The method of combining 'things' into books and sharing them across every sector is universally applicable to organise wisdom in the Internet of Everything. Some examples:
  • Health Wisdom doctors, nurses and patient crowd creating health (via HealthBook)
  • Education Wisdom: teachers, parents and students crowd creating education ( via EducationBook)
  • Equity Market Wisdom: companies, advisers and investors crowd creating IPO's, ventures, trade sales and innovation
  • Democracy Wisdom: politicians and citizens crowd creating real-time citizen participation and democracy (via PoliticiansBook and Network Democracy
  • Government Wisdom: politicians, bureacrats, citizens and industries crowd creating governance and services outcomes
  • Governance Wisdom: governance is about directing effort to prioritised, merit based outcomes. Wisdom Networks reveal perfect knowledge and optimise action action.
  • Management Wisdom: directors (direct), managers (manage and review) and executives (execute) crowd create management and leadership within an organisation with a 3 role flat management structure
  • Regional integration: people across regions can exchange wisdom about any "thing" in any sector
  • Global: any person anywhere around the world can participate, contribute or seek humanitarian assistance from any sector in any Network Society

Books capture past (audit trail), present (knowledge) and future (plan) about every "thing"

Evolution has been crowd
creating wisdom books
over centuries

Hardware complexity to Software simplicity

The complexity of physics can be simplified to e=mc2. The complexity of society can be simplified to Things = Books = Sectors = Countries = Global Village. This formula represents the design of software that organises Network Society. An Industrial Economy divides people using hierarchies within countries and globally using physical (ie hardware) tools to organise society. Network Society unites people with few networks in sectors, countries and Network Democracy around the world using virtual (ie software) tools to organise society. The primary means for organising society becomes networks of people organised by software at zero marginal cost with immediate outcomes, rather than isolated, physical hierarchies using expensive infrastructure with uncertain outcomes over long timeframes. The 'Global Village' becomes a mesh of countries and sectors. Each country is unified by software as '1 Country'. All countries are unified by software as '1 Global Village'. Network Society simply comes 'Over the Top' (OTT) of society and virtually organises every 'thing' in the cloud.

A switch from physical (hardware) to virtual (software) to organise society

People in networks become primary means to organise society

A shift from local and vertical ... to horizontal and global

Network Society adds 4 new features to Industrial Economy