State management
State in the EVM is quite significantly restricted. It consists of a key-value where each key and value is fixed size at 256 bits. This restriction can make it difficult to manage the different state namespaces as a human developer. As a result, Solidity handles this by automatically handling the location in the 256 bit key space where each storage variable is placed. This is generated based on the hash of the variable name and/or the location in the source file (depending on exact solidity version etc, this can change). The actual way that this is stored in the database by nodes/wallets is using the “state trie”, a Patricia tree. This covers it in more detail than I probably could. The TL;DR; though is that a verifiable cryptographic tree is used so that nodes can always prove, given the “root hash”, that a piece of state exists and is of the expected value. This can’t be done in a standard merkle tree because the entire state of all contracts in the are stored and recalculating a naive merkle tree with hundreds of thousands of different state elements would be too computationally heavy, even if verification would be faster than producing the root hash.
In Qtum-x86, one of the biggest goals was to remove these restrictions of state in the EVM. So, in Qtum-x86 it is possible to have dynamic length keys and values. This required a completely new design. Qtum has always been -like in many of it’s designs, due to being a fork of . In Qtum-x86 we are introducing the DeltaDB design. It is a new database which operates on differences, rather than needing to encode the entire state of the into a single accessible data structure. There is no direct restriction on the size of keys or values of state, though overly large sizes can cause updates to be overly expensive in gas costs, so some state separation is still beneficial to separate large seldom updated data from smaller frequently updated data. All state management in Qtum-x86 is manual for the moment. There is nothing similar to Solidity’s automatic state management for key names. There may be at some later point, but only for higher level languages than C. Regardless, the key space is trivial to manage using typical methods from traditional key-value databases. For instance, if a key-value map named “balances” that is indexed by addresses and stores a simple 64bit integer, then it could be stored like so:
"balances_QddCMpVUf4gKTLseP5XFuVco6xy1YajbK7" -> 1000Of course, “balances_” might be an integer prefix and the address would be stored as raw bytes (or better yet, the UniversalAddress bytecode format) to save on space. Namespaces and keys are thus much easier to manage manually than an equivalent situation in the EVM with 256 bit keys and values. The actual way this state is stored in the node/ databases is very straight forward. Something like so:
"state_XZkyE7XAweUtrizgtry1RoMSv1p4zk9Rw8_balances_QddCMpVUf4gKTLseP5XFuVco6xy1YajbK7" -> 1000This is where “XZkyE7XAweUtrizgtry1RoMSv1p4zk9Rw8” is the contract address. Also, of course this is all stored as encoded bytes and not wasteful string, this is just for illustration. The straight forward manner in which this type of data can be stored makes modern database caches and prediction significantly more effective than the equivalent situation in the EVM. A notorious issue of the EVM is requiring full nodes to use an SSD because of the unpredictable nature of reading data indexed by “random” hashes. There is some historical data duplication in Qtum-x86 nodes for maintaining consensus, but this is also subject to pruning and thus it is safe to delete most historical data after 500 blocks if the node operator is only interested in the current state, rather than historical states. Historical states in particular can be useful for auditing and some specific light applications, but is not needed for most users.
The actual method this state data is proven on the is also more simple than the methods used by (and Qtum’s current version of the EVM). In ’s traditional light “SPV” technology, a merkle tree root hash is stored in the block and this can be used to prove that a transaction occurred in a particular block. However, this concept is too limited for smart contract state as we need to prove more than simply that a contract transaction occurred. We need to prove what state is present, along with transaction receipts and “events” (’s “log” concept).
To do this, Qtum-x86 uses a “staggered” merkle tree with nested per-account merkle trees. Most merkle trees encode a list of some singular type of data. With the staggered approach, the merkle tree instead encodes pairs of data, with the first data being a contract address, and the second data being the account “delta tree”. The staggered approach to the root merkle tree allows for simpler location of accounts of interest. ie, if a light node is interested in all times a contract state changes in some way, it is easy and lightweight to encode a proof that the account was modified in some way. Accounts which are not modified in the current block will not be in the root tree. More importantly, this allows for a form of censorship resistance where the concept can be inverted to construct proofs that an account was not modified. This would work by getting a list of all the accounts modified and their respective delta trees. It’s obvious to detect that not all data was received (as the merkle tree hash would not match the block data) or that some of it was modified.
The delta tree itself encodes a list of “deltas”. A delta indicates a change to the contract account. This can include things like a simple (unpure) contract execution, events being emitted by the contract, state changes, or balance changes. The “transaction receipt” concept of is also encoded into a delta, so that it is easy to prove a contract execution took place and if the contract encountered an error in execution.
The DeltaDB concept is technically more simple than ’s state trie concept, yet gives many benefits:
- Simpler to implement and audit (as Qtum)
- Allows for dynamic length keys and values
- Extremely extensible to allow for additional data to be encoded in the consensus-critical tree in the future without breaking client and programmatic supports (though would still require a hardfork)
- Simpler to scale and handle disk load due to non-random indexed keys
- Allows for censorship detection by SPV nodes of any account
- Allows for (slow, but capable) retrieval of all historical contract state for any and all contracts without replaying transactions, and with included proofs that data is missing.
Of course, it’s not without it’s disadvantages as well:
- Unproven and new design
- Getting all state of a contract of interest as an SPV node could require more data transfer with full nodes, since the untrusted setup can not exclude all historical data generally. Depending on the exact contract setup though, it may be possible to safely exclude some historical data. The amount of round-trip requests also is significantly higher when using censorship-resistance, and thus is not expected to be used as the “normal” mode of communication
- Is expected to increase disk usage on nodes compared to an equivalent setup on Ethereum, due to needed location, intermediate, and temporary (500 block) historical data that would otherwise be expensive to compute more than one time.
Contract Upgrades
Contract upgrades have long been a point in the smart contract ecosystem; the EVM does this no favors. The EVM does not directly allow code changes to take place in an established contract account. The workaround used by the community is to use the concept of . The summary of this is that a special “proxy” contract is used that can respond to certain requests (such as those to conduct upgrades), but for all other functionality delegates to the actual backing contract, which can be pointed to a new contract through upgrade functions. The proxy contract also holds all relevant state for the backing smart contract. This is done so that the code for the contract can be upgraded without needing to redeploy or modify the existing data used by the contract.
EVM smart contracts are of course typically written in Solidity. This further complicates the upgrade process. As discussed in the state database comparison above, Solidity auto-magically handles the location in which all state variables and arrays are placed in the single 256-bit namespace of EVM based storage. In older versions of Solidity, this caused simple refactoring such as placing a new state variable above old state variables would break this, causing Solidity to try to read/write a different location for the old state variables, of course leading to disastrous results. There are many workarounds, some more complex than others. This post covers a good number of them and the advantages and drawbacks. In general though, the trade-off triangle is Difficulty of Refactoring/Upgrades vs Additional Gas Costs vs Difficulty of Initial Implementation. The old adage applies, “pick two that you want to be optimal”.
Qtum-x86 intends to remove this triangle for the most part, and does so in four core ways:
- Simple manual and explicit state management prevents dropping to assembly or other burdens placed on the smart contract developer.
- Direct upgrades on contract code is possible, eliminating the need for proxy contracts.
- New mechanisms for a coalition of (potentially untrusted) smart contracts is introduced to allow access to a central “registry” of state without undue burden in gas costs.
- External code delegation (ie, DELEGATECALL) can be done safely with untrusted or semi-trusted contracts by introducing a permission system to establish what a delegate called contract is allowed to do on the calling contract’s behalf.
This is a lot, but first the explicit state management. We touched on that some in the above section about state databases, but the big takeaway is that the developer is in control of where state is placed. Although 256 bits is plenty of space to arrange any human number of distinct state maps/arrays, it is needlessly complicated, especially since it would require hashing of the keys used for indexing in maps. To do this in Solidity explicitly also requires dropping to the assembly level and typically writing a number of wrapper functions. These wrapper functions used in leu of normal state variables are hard to optimize by the solidity compiler, and the developer is completely on their own to handle uniqueness of locations, bounds checking, field packing, field splitting, etc. In Qtum-x86 state is handled as simply as a typical key-value database. The only real difference between access is that there is no (planned) method of allowing a contract to do a “wildcard” search for keys, such as “give me every key in my database that starts with X”. This kind of query would be quite expensive and would be difficult to guard against exploits which cause an excessive amounts of gas to be consumed. So, although the concept is possible in Qtum-x86’s database, there is no plans to expose that functionality at the moment. Regardless, simple key namespaces are made using prefixes, keys can be stored without requiring hashing (although after some length, Qtum-x86 internally will hash it), and there is no need to break up structures or manually pack bytes to fit into some constant size.
The next big thing is allowing upgrades of a contract’s code. This will be a feature that some contracts may wish to opt-out of for provability reasons, but currently there is no good reason, security-wise, that even a like does not allow it. This allows for the elimination of the need for proxy contracts, and allows simpler implementation of upgrade mechanisms.
In Qtum-x86 it will be possible to read state directly from external contracts. There is no need for state accessor functions like getTokenBalance() to be implemented in an ERC20-like contract. Instead, something like this can be used: externalState(erc20Address, "balance", addressOfInterest, &balance). This allows for any external contract to read state from a common space. Writing state is more complicated as permissions are involved, but basically the state contract would control who has access to writing to that state, either by using the Trusted Library permission features, or by using explicit modifier functions, potentially with validation of the format etc as well as that the party attempting the modification is authorized. The Trusted Library system will allow for modular contract design, using a proxy-like system. However, instead of delegating all code to a single “code” contract, it will be possible to delegate specific functions to specific contracts. In addition the permission system introduced will ensure that even if an exploit were discovered in one of these delegated function contract, the impact would be limited. This means that if an ERC20-like contract delegated out a simple function like “getBalance”, it would not be allowed to modify state or send Qtum out of the contract.
Published at Sun, 19 May 2019 07:14:27 +0000
