IN3-Protocol¶
This document describes the communication between a Incubed client and a Incubed node. This communication is based on requests that use extended JSON-RPC-Format. Especially for ethereum-based requests, this means each node also accepts all standard requests as defined at Ethereum JSON-RPC, which also includes handling Bulk-requests.
Each request may add an optional in3
property defining the verification behavior for Incubed.
Incubed Requests¶
Requests without an in3
property will also get a response without in3
. This allows any Incubed node to also act as a raw ethereum JSON-RPC endpoint. The in3
property in the request is defined as the following:
- chainId
string<hex>
- The requested chainId. This property is optional, but should always be specified in case a node may support multiple chains. In this case, the default of the node would be used, which may end up in an undefined behavior since the client cannot know the default. - includeCode
boolean
- Applies only foreth_call
-requests. If true, the request should include the codes of all accounts. Otherwise only the the codeHash is returned. In this case, the client may ask by calling eth_getCode() afterwards. - verifiedHashes
string<bytes32>[]
- If the client sends an array of blockhashes, the server will not deliver any signatures or blockheaders for these blocks, but only return a string with a number. This allows the client to skip requiring signed blockhashes for blocks already verified. - latestBlock
integer
- If specified, the blocknumberlatest
will be replaced by a blockNumber-specified value. This allows the Incubed client to define finality for PoW-Chains, which is important, since thelatest
-block cannot be considered final and therefore it would be unlikely to find nodes willing to sign a blockhash for such a block. - useRef
boolean
- If true, binary-data (starting with a 0x) will be referred if occurring again. This decreases the payload especially for recurring data such as merkle proofs. If supported, the server (and client) will keep track of each binary value storing them in a temporary array. If the previously used value is used again, the server replaces it with:<index>
. The client then resolves such refs by lookups in the temporary array. - useBinary
boolean
- If true, binary-data will be used. This format is optimzed for embedded devices and reduces the payload to about 30%. For details see the Binary-spec. - useFullProof
boolean
- If true, all data in the response will be proven, which leads to a higher payload. The result depends on the method called and will be specified there. - finality
number
- For PoA-Chains, it will deliver additional proof to reach finality. If given, the server will deliver the blockheaders of the following blocks until at least the number in percent of the validators is reached. - verification
string
- Defines the kind of proof the client is asking for. Must be one of the these values:'never
’ : No proof will be delivered (default). Also noin3
-property will be added to the response, but only the raw JSON-RPC response will be returned.'proof
’ : The proof will be created including a blockheader, but without any signed blockhashes.'proofWithSignature
’ : The returned proof will also include signed blockhashes as required insignatures
.
- signatures
string<address>[]
- A list of addresses (as 20bytes in hex) requested to sign the blockhash.
A example of an Incubed request may look like this:
{
"jsonrpc": "2.0",
"id": 2,
"method": "eth_getTransactionByHash",
"params": ["0xf84cfb78971ebd940d7e4375b077244e93db2c3f88443bb93c561812cfed055c"],
"in3": {
"chainId": "0x1",
"verification": "proofWithSignature",
"signatures":["0x784bfa9eb182C3a02DbeB5285e3dBa92d717E07a"]
}
}
Incubed Responses¶
Each Incubed node response is based on JSON-RPC, but also adds the in3
property. If the request does not contain a in3
property or does not require proof, the response must also omit the in3
property.
If the proof is requested, the in3
property is defined with the following properties:
- proof Proof - The Proof-data, which depends on the requested method. For more details, see the Proofs section.
- lastNodeList
number
- The blocknumber for the last block updating the nodelist. This blocknumber should be used to indicate changes in the nodelist. If the client has a smaller blocknumber, it should update the nodeList. - lastValidatorChange
number
- The blocknumber of the last change of the validatorList (only for PoA-chains). If the client has a smaller number, it needs to update the validatorlist first. For details, see PoA Validations - currentBlock
number
- The current blocknumber. This number may be stored in the client in order to run sanity checks forlatest
blocks oreth_blockNumber
, since they cannot be verified directly.
An example of such a response would look like this:
{
"jsonrpc": "2.0",
"result": {
"blockHash": "0x2dbbac3abe47a1d0a7843d378fe3b8701ca7892f530fd1d2b13a46b202af4297",
"blockNumber": "0x79fab6",
"chainId": "0x1",
"condition": null,
"creates": null,
"from": "0x2c5811cb45ba9387f2e7c227193ad10014960bfc",
"gas": "0x186a0",
"gasPrice": "0x4a817c800",
"hash": "0xf84cfb78971ebd940d7e4375b077244e93db2c3f88443bb93c561812cfed055c",
"input": "0xa9059cbb000000000000000000000000290648fc6f2cb27a2a81dc35a429090872991b92000000000000000000000000000000000000000000000015af1d78b58c400000",
"nonce": "0xa8",
"publicKey": "0x6b30c392dda89d58866bf2c1bedf8229d12c6ae3589d82d0f52ae588838a475aacda64775b7a1b376935d732bb8022630a01c4926e71171eeda938b644d83365",
"r": "0x4666976b528fc7802edd9330b935c7d48fce0144ce97ade8236da29878c1aa96",
"raw": "0xf8ab81a88504a817c800830186a094d3ebdaea9aeac98de723f640bce4aa07e2e4419280b844a9059cbb000000000000000000000000290648fc6f2cb27a2a81dc35a429090872991b92000000000000000000000000000000000000000000000015af1d78b58c40000025a04666976b528fc7802edd9330b935c7d48fce0144ce97ade8236da29878c1aa96a05089dca7ecf7b061bec3cca7726aab1fcb4c8beb51517886f91c9b0ca710b09d",
"s": "0x5089dca7ecf7b061bec3cca7726aab1fcb4c8beb51517886f91c9b0ca710b09d",
"standardV": "0x0",
"to": "0xd3ebdaea9aeac98de723f640bce4aa07e2e44192",
"transactionIndex": "0x3e",
"v": "0x25",
"value": "0x0"
},
"id": 2,
"in3": {
"proof": {
"type": "transactionProof",
"block": "0xf90219a03d050deecd980b16cad9752133333ccdface463cc69e784f32dd981e2e751e34a01dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d4934794829bd824b016326a401d083b33d092293333a830a012892951590f62f4b2802f88e8fddc09c951ad2cac23803e07c4f11e01991907a018a21c8413fc7fc29f09d12f75515993ab38858bfa9e5632670cbba3358f0cfaa02fc4436c96ae4d100921c20b5cb601252de68ddde159bc89f3353555eff0ccccb901009034d281f0400b0920d21f7795b09d8c2b9cd48a939ce476aa84f486c68855684c0804a304a444a17c0ca4420e32a3b29a8218802d9fab5112a82b8d60e12203400084c2a236149a4a44905e120540a1478261a55a399229fe046595236900025de213ea6a000612901d6008080a6f773755182105c9100048a40eb458808a0334a2c5927a9308f300962916898c861a888d8d780508061c2bc54c866078216042497a0cd05dfa65948b8dc4144ca64144883c2422a5280848021328d8a8e41602890d122b0110c27bc014193502a7690d40e00f03a879080b0073f1ae4ab0232b93630c068ecb7b4b923de0012566855524a000502c87906480151e81d2b032870709c2784add128379fab6837a3f58837a12f8845d0b4673987070796520e4b883e5bda9e7a59ee4bb99e9b1bc9329ad43a0e21b342112a946b58fa2f50739166c20aed4647d3ad8e37210d451fb8b243870888f95c17c0647e1f9",
"merkleProof": [
"0xf90131a00150ff50e29f3df34b89870f183c85a82a73f21722d7e6c787e663159f165010a0b8c56f207a223067c7ae5df7420221327c32f89f36cef8a14c33e5a4e67be9cfa0112091138bbf6bde2e20c88b08d10f8ea08ec298f2daac34d76fc8e248379dc5a0c737a71d34faa7c864930707ac7870b2c7cc28e7d489d21330acfa8deb72d805a075811c4bdef2cc74095e57cacce23debab8ea8e6d8937932678d2fd444367ea9a0e79e4e445e517b7b31ad626acabec77a6e0c846207b91f01ac33e804af096325a07065708e1a9e9b865dbd5e19e521224ae554a5d3064257e5401d7cad900f555aa01a71ef57896ce378fd51bf44a1d0b538d3587d9aecdbf3c6c7f6794bbb0f0fa8a0d720eecae23cd40af5c534b90b00f33b7ec0638b11cc7809058110bf984a02d48080808080808080",
"0xf90211a0f4a5e4a1197190f910e4a026f50bd6a169716b52be42c99ddb043ad9b4da6117a09ad1def70dd1d991331d013719cca31d35111cf75d3046dffdc9d1897ecfce29a01ada8fa2d6a7f9b44394a0d7fafe8a59810e48596e1258adb57ca51a6a014024a0eeb2d6482d696d623ae7f868aa3463790041c4863f1d47f84d6629f2d5ee88c5a0f1c04c4bc88aa5f24c7e5ac401c5246cf17834e7e68d4b2c9b656a37f510aff1a040446d66c0039c4806ee13da02ebe408abab366332ec2355367ca0dec5aab273a0775b1f53ad22fdcb6fef814d34b910be6a2e6463febb174d4f2064626baf639fa0bb1668055775f8ba59bf071465ffe68db4f916a7eb0ea07126b71d3e30a8fd70a08ad25a05847cdeec5261154c5ae89f03f2a8a813e8804983c677dc0d39e26bfca0a0c6f9e3e55cabbe3a9c0c6713aeb4e70135b9abe21b50bb6e04e6f4a09888d5a011d5422e577e357d26390492378b9328518b263310574b1e0d9e322031485a22a0c2f4f15a1ba6585a87a0dcca7b45dc0bbcd72830df61888d7abf16fef6a4df72a02bf0d1675ebf1c1f2af6793edf748e3184c2ac5522a6640a1b04d3b7bad7e23ca0c80cf2596da4c35f6c5e5348791c64c10d80ccd7668d6ef73a2454f0f11a0f59a03e54112466dbd3791d6e1e281d25470b884c96406e39bd83e8a806cfc8e60219a00e2cc674fa10aefb4dea53ac114e28c6353d30b315d4ba280ab4741920a60ce280",
"0xf8b020b8adf8ab81a88504a817c800830186a094d3ebdaea9aeac98de723f640bce4aa07e2e4419280b844a9059cbb000000000000000000000000290648fc6f2cb27a2a81dc35a429090872991b92000000000000000000000000000000000000000000000015af1d78b58c40000025a04666976b528fc7802edd9330b935c7d48fce0144ce97ade8236da29878c1aa96a05089dca7ecf7b061bec3cca7726aab1fcb4c8beb51517886f91c9b0ca710b09d"
],
"txIndex": 62,
"signatures": [
{
"blockHash": "0x2dbbac3abe47a1d0a7843d378fe3b8701ca7892f530fd1d2b13a46b202af4297",
"block": 7994038,
"r": "0xef73a527ae8d38b595437e6436bd4fa037d50550bf3840ad0cd3c6ca641a951e",
"s": "0x6a5815db16c12b890347d42c014d19b60e1605d2e8e64b729f89e662f9ce706b",
"v": 27,
"msgHash": "0xa8fc6e2564e496efc5fd7db8e70f03fd50af53e092f47c98329c84c96026fdff"
}
]
},
"currentBlock": 7994124,
"lastValidatorChange": 0,
"lastNodeList": 6619795
}
}
ChainId¶
Incubed supports multiple chains and a client may even run requests to different chains in parallel. While, in most cases, a chain refers to a specific running blockchain, chainIds may also refer to abstract networks such as ipfs. So, the definition of a chain in the context of Incubed is simply a distributed data domain offering verifiable api-functions implemented in an in3-node.
Each chain is identified by a uint64
identifier written as hex-value (without leading zeros). Since incubed started with ethereum, the chainIds for public ethereum-chains are based on the intrinsic chainId of the ethereum-chain. See https://chainid.network.
For each chain, Incubed manages a list of nodes as stored in the server registry and a chainspec describing the verification. These chainspecs are held in the client, as they specify the rules about how responses may be validated.
Registry¶
As Incubed aims for fully decentralized access to the blockchain, the registry is implemented as an ethereum smart contract.
This contract serves different purposes. Primarily, it manages all the Incubed nodes, both the onboarding and also unregistering process. In order to do so, it must also manage the deposits: reverting when the amount of provided ether is smaller than the current minimum deposit; but also locking and/or sending back deposits after a server leaves the in3-network.
In addition, the contract is also used to secure the in3-network by providing functions to “convict” servers that provided a wrongly signed block, and also having a function to vote out inactive servers.
Node structure¶
Each Incubed node must be registered in the ServerRegistry in order to be known to the network. A node or server is defined as:
url
string
- The public url of the node, which must accept JSON-RPC requests.owner
address
- The owner of the node with the permission to edit or remove the node.signer
address
- The address used when signing blockhashes. This address must be unique within the nodeList.timeout
uint64
- Timeout after which the owner is allowed to receive its stored deposit. This information is also important for the client, since an invalid blockhash-signature can only “convict” as long as the server is registered. A long timeout may provide higher security since the node can not lie and unregister right away.deposit
uint256
- The deposit stored for the node, which the node will lose if it signs a wrong blockhash.props
uint64
- A bitmask defining the capabilities of the node:0x01
: proof : The node is able to deliver proof. If not set, it may only serve pure ethereum JSON/RPC. Thus, simple remote nodes may also be registered as Incubed nodes.0x02
: multichain : The same RPC endpoint may also accept requests for different chains.0x04
: archive : If set, the node is able to support archive requests returning older states. If not, only a pruned node is running.0x08
: http : If set, the node will also serve requests on standard http even if the url specifies https. This is relevant for small embedded devices trying to save resources by not having to run the TLS.0x10
: binary : If set, the node accepts request withbinary:true
. This reduces the payload to about 30% for embedded devices.
More properties will be added in future versions.
unregisterTime
uint64
- The earliest timestamp when the node can unregister itself by callingconfirmUnregisteringServer
. This will only be set after the node requests an unregister. The client nodes with anunregisterTime
set have less trust, since they will not be able to convict after this timestamp.registerTime
uint64
- The timestamp, when the server was registered.weight
uint64
- The number of parallel requests this node may accept. A higher number indicates a stronger node, which will be used within the incentivization layer to calculate the score.
The following functions are offered within the registry:
NodeRegistry functions¶
constructor¶
constructor
Development notice: cannot be deployed in a genesis block
Parameters:
- _blockRegistry
BlockhashRegistry
: address of a BlockhashRegistry-contract
convict¶
must be called before revealConvict commits a blocknumber and a hash
Development notice: The v,r,s paramaters are from the signature of the wrong blockhash that the node provided
Parameters:
- _blockNumber
uint
: the blocknumber of the wrong blockhash - _hash
bytes32
: keccak256(wrong blockhash, msg.sender, v, r, s); used to prevent frontrunning.
registerNode¶
register a new node with the sender as owner
Development notice: will call the registerNodeInteral function
Parameters:
- _url
string
: the url of the node, has to be unique - _props
uint64
: properties of the node - _timeout
uint64
: timespan of how long the node of a deposit will be locked. Will be at least for 1h - _weight
uint64
: how many requests per second the node is able to handle
registerNodeFor¶
register a new node as a owner using a different signer address
Development notice: will revert when a wrong signature has been provided
which is calculated by the hash of the url, properties, timeout, weight and the owner
in order to prove that the owner has control over the signer-address he has to sign a message
will call the registerNodeInteral function
Parameters:
- _url
string
: the url of the node, has to be unique - _props
uint64
: properties of the node - _timeout
uint64
: timespan of how long the node of a deposit will be locked. Will be at least for 1h - _signer
address
: the signer of the in3-node - _weight
uint64
: how many requests per second the node is able to handle - _v
uint8
: v of the signed message - _r
bytes32
: r of the signed message - _s
bytes32
: s of the signed message
removeNodeFromRegistry¶
removes an in3-server from the registry
Development notice: only callable in the 1st year after deployment
only callable by the unregisterKey-account
Parameters:
- _signer
address
: the signer-address of the in3-node
returnDeposit¶
only callable after the timeout of the deposit is over returns the deposit after a node has been removed
Development notice: reverts if the deposit is still locked
reverts when there is nothing to transfer
reverts when not the owner of the former in3-node
Parameters:
- _signer
address
: the signer-address of a former in3-node
revealConvict¶
reveals the wrongly provided blockhash, so that the node-owner will lose its deposit
Development notice: reverts when the wrong convict hash (see convict-function) is used
reverts when the _signer did not sign the block
reverts when trying to reveal immediately after calling convict
reverts when trying to convict someone with a correct blockhash
reverts if a block with that number cannot be found in either the latest 256 blocks or the blockhash registry
Parameters:
- _signer
address
: the address that signed the wrong blockhash - _blockhash
bytes32
: the wrongly provided blockhash - _blockNumber
uint
: number of the wrongly provided blockhash - _v
uint8
: v of the signature - _r
bytes32
: r of the signature - _s
bytes32
: s of the signature
transferOwnership¶
changes the ownership of an in3-node
Development notice:
reverts when the sender is not the current owner
reverts when trying to pass ownership to 0x0
reverts when trying to change ownership of an inactive node
Parameters:
- _signer
address
: the signer-address of the in3-node, used as an identifier - _newOwner
address
: the new owner
unregisteringNode¶
doing so will also lock his deposit for the timeout of the node a node owner can unregister a node, removing it from the nodeList
Development notice: reverts when not called by the owner of the node
reverts when the provided address is not an in3-signer
Parameters:
- _signer
address
: the signer of the in3-node
updateNode¶
updates a node by adding the msg.value to the deposit and setting the props or timeout
Development notice: reverts when trying to change the url to an already existing one
reverts when trying to increase the timeout above 10 years
reverts when the signer does not own a node
reverts when the sender is not the owner of the node
Parameters:
- _signer
address
: the signer-address of the in3-node, used as an identifier - _url
string
: the url, will be changed if different from the current one - _props
uint64
: the new properties, will be changed if different from the current onec - _timeout
uint64
: the new timeout of the node, cannot be decreased. Has to be at least 1h - _weight
uint64
: the amount of requests per second the node is able to handle
calcProofHash¶
calculates the sha3 hash of the most important properties in order to make the proof faster
Parameters:
- _node
In3Node
: the in3 node to calculate the hash from
Return Parameters:
bytes32
the hash of the properties to prove with in3
checkNodeProperties¶
function to check whether the allowed amount of ether as deposit per server has been reached
Development notice: will fail when the provided timeout is greater then 1 year
will fail when the deposit is greater then 50 ether in the 1st year
Parameters:
- _deposit
uint256
: the new amount of deposit a server has - _timeout
uint64
: the timeout until a server can receive his deposit after unregister
registerNodeInternal¶
registers a node
Development notice: reverts when either the owner or the url is already in use
reverts when trying to register a node with more then 50 ether in the 1st year after deployment
reverts when provided not enough deposit
reverts when time timeout exceed the MAXDEPOSITTIMEOUT
Parameters:
- _url
string
: the url of a node - _props
uint64
: properties of a node - _timeout
uint64
: the time before the owner can access the deposit after unregister a node - _signer
address
: the address that signs the answers of the node - _owner
address
: the owner address of the node - _deposit
uint
: the deposit of a node - _weight
uint64
: the amount of requests per second a node is able to handle
unregisterNodeInternal¶
handles the setting of the unregister values for a node internally
Parameters:
- _si
SignerInformation
: information of the signer - _n
In3Node
: information of the in3-node
removeNode¶
removes a node from the node-array
BlockHashRegistry functions¶
constructor¶
constructor
searchForAvailableBlock¶
searches for an already existing snapshot
Parameters:
- _startNumber
uint
: the blocknumber to start searching - _numBlocks
uint
: the number of blocks to search for
Return Parameters:
uint
returns a blocknumber when a snapshot had been found. It will return 0 if no blocknumber was found.
recreateBlockheaders¶
if successfull the last blockhash of the header will be added to the smart contract it will be checked whether the provided chain is correct by using the reCalculateBlockheaders function only usable when the given blocknumber is already in the smart contract starts with a given blocknumber and its header and tries to recreate a (reverse) chain of blocks
Development notice: function is public due to the usage of a dynamic bytes array (not yet supported for external functions)
reverts when the chain of headers is incorrect
reverts when there is not parent block already stored in the contract
Parameters:
- _blockNumber
uint
: the block number to start recreation from - _blockheaders
bytes[]
: array with serialized blockheaders in reverse order (youngest -> oldest) => (e.g. 100, 99, 98)
saveBlockNumber¶
stores a certain blockhash to the state
Development notice: reverts if the block can’t be found inside the evm
Parameters:
- _blockNumber
uint
: the blocknumber to be stored
snapshot¶
stores the currentBlock-1 in the smart contract
getParentAndBlockhash¶
returns the blockhash and the parent blockhash from the provided blockheader
Parameters:
- _blockheader
bytes
: a serialized (rlp-encoded) blockheader
Return Parameters:
- parentHash
bytes32
- bhash
bytes32
reCalculateBlockheaders¶
the array of the blockheaders have to be in reverse order (e.g. [100,99,98,97]) starts with a given blockhash and its header and tries to recreate a (reverse) chain of blocks
Parameters:
- _blockheaders
bytes[]
: array with serialized blockheaders in reverse order, i.e. from youngest to oldest - _bHash
bytes32
: blockhash of the 1st element of the _blockheaders-array
Binary Format¶
Since Incubed is optimized for embedded devices, a server can not only support JSON, but a special binary-format. You may wonder why we don’t want to use any existing binary serialization for JSON like CBOR or others. The reason is simply: because we do not need to support all the features JSON offers. The following features are not supported:
- no escape sequences (this allows use of the string without copying it)
- no float support (at least for now)
- no string literals starting with
0x
since this is always considered as hexcoded bytes - no propertyNames within the same object with the same key hash
Since we are able to accept these restrictions, we can keep the JSON-parser simple. This binary-format is highly optimized for small devices and will reduce the payload to about 30%. This is achieved with the following optimizations:
All strings starting with
0x
are interpreted as binary data and stored as such, which reduces the size of the data to 50%.Recurring byte-values will use references to previous data, which reduces the payload, especially for merkle proofs.
All propertyNames of JSON-objects are hashed to a 16bit-value, reducing the size of the data to a signifivant amount (depending on the propertyName).
The hash is calculated very easily like this:
static d_key_t key(const char* c) { uint16_t val = 0, l = strlen(c); for (; l; l--, c++) val ^= *c | val << 7; return val; }
Note
A very important limitation is the fact that property names are stored as 16bit hashes, which decreases the payload, but does not allow for the restoration of the full json without knowing all property names!
The binary format is based on JSON-structure, but uses a RLP-encoding approach. Each node or value is represented by these four values:
- key
uint16_t
- The key hash of the property. This value will only pass before the property node if the structure is a property of a JSON-object. - type
d_type_t
- 3 bit : defining the type of the element. - len
uint32_t
- 5 bit : the length of the data (for bytes/string/array/object). For (boolean or integer) the length will specify the value. - data
bytes_t
- The bytes or value of the node (only for strings or bytes).
The serialization depends on the type, which is defined in the first 3 bits of the first byte of the element:
d_type_t type = *val >> 5; // first 3 bits define the type
uint8_t len = *val & 0x1F; // the other 5 bits (0-31) the length
The len
depends on the size of the data. So, the last 5 bit of the first bytes are interpreted as follows:
0x00
-0x1c
: The length is taken as is from the 5 bits.0x1d
-0x1f
: The length is taken by reading the big-endian value of the nextlen - 0x1c
bytes (len ext).
After the type-byte and optional length bytes, the 2 bytes representing the property hash is added, but only if the element is a property of a JSON-object.
Depending on these types, the length will be used to read the next bytes:
0x0
: binary data - This would be a value or property with binary data. Thelen
will be used to read the number of bytes as binary data.0x1
: string data - This would be a value or property with string data. Thelen
will be used to read the number of bytes (+1) as string. The string will always be null-terminated, since it will allow small devices to use the data directly instead of copying memory in RAM.0x2
: array - Represents an array node, where thelen
represents the number of elements in the array. The array elements will be added right after the array-node.0x3
: object - A JSON-object withlen
properties coming next. In this case the properties following this element will have a leadingkey
specified.0x4
: boolean - Boolean value wherelen
must be either0x1
=true
or0x0
=false
. Iflen > 1
this element is a copy of a previous node and may reference the same data. The index of the source node will then belen-2
.0x5
: integer - An integer-value with max 29 bit (since the 3 bits are used for the type). If the value is higher than0x20000000
, it will be stored as binary data.0x6
: null - Represents a null-value. If this value has alen
> 0 it will indicate the beginning of data, wherelen
will be used to specify the number of elements to follow. This is optional, but helps small devices to allocate the right amount of memory.
Communication¶
Incubed requests follow a simple request/response schema allowing even devices with a small bandwith to retrieve all the required data with one request. But there are exceptions when additional data need to be fetched.
These are:
Changes in the NodeRegistry
Changes in the NodeRegistry are based on one of the following events:
LogNodeRegistered
LogNodeRemoved
LogNodeChanged
The server needs to watch for events from the
NodeRegistry
contract, and update the nodeList when needed.Changes are detected by the client by comparing the blocknumber of the latest change with the last known blocknumber. Since each response will include the
lastNodeList
, a client may detect this change after receiving the data. The client is then expected to callin3_nodeList
to update its nodeList before sending out the next request. In the event that the node is not able to proof the new nodeList, the client may blacklist such a node.
Changes in the ValidatorList
This only applies to PoA-chains where the client needs a defined and verified validatorList. Depending on the consensus, changes in the validatorList must be detected by the node and indicated with the
lastValidatorChange
on each response. ThislastValidatorChange
holds the last blocknumber of a change in the validatorList.Changes are detected by the client by comparing the blocknumber of the latest change with the last known blocknumber. Since each response will include the
lastValidatorChange
a client may detect this change after receiving the data or in case of an unverifiable response. The client is then expected to callin3_validatorList
to update its list before sending out the next request. In the event that the node is not able to proof the new nodeList, the client may blacklist such a node.Failover
It is also good to have a second request in the event that a valid response is not delivered. This could happen if a node does not respond at all or the response cannot be validated. In both cases, the client may blacklist the node for a while and send the same request to another node.
Proofs¶
Proofs are a crucial part of the security concept for Incubed. Whenever a request is made for a response with verification
: proof
, the node must provide the proof needed to validate the response result. The proof itself depends on the chain.
Ethereum¶
For ethereum, all proofs are based on the correct block hash. That’s why verification differentiates between Verifying the blockhash (which depends on the user consensus) the actual result data.
There is another reason why the BlockHash is so important. This is the only value you are able to access from within a SmartContract, because the evm supports a OpCode (BLOCKHASH
), which allows you to read the last 256 blockhashes, which gives us the chance to verify even the blockhash onchain.
Depending on the method, different proofs are needed, which are described in this document.
- Block Proof - Verifies the content of the BlockHeader.
- Transaction Proof - Verifies the input data of a transaction.
- Receipt Proof - Verifies the outcome of a transaction.
- Log Proof - Verifies the response of
eth_getPastLogs
. - Account Proof - Verifies the state of an account.
- Call Proof - Verifies the result of an
eth_call
-response.
Each in3
-section of the response containing proofs has a property with a proof-object with the following properties:
- type
string
(required) - The type of the proof.Must be one of the these values :'transactionProof
’,'receiptProof
’,'blockProof
’,'accountProof
’,'callProof
’,'logProof
’ - block
string
- The serialized blockheader as hex, required in most proofs. - finalityBlocks
array
- The serialized following blockheaders as hex, required in case of finality asked (only relevant for PoA-chains). The server must deliver enough blockheaders to cover more then 50% of the validators. In order to verify them, they must be linkable (with the parentHash). - transactions
array
- The list of raw transactions of the block if needed to create a merkle trie for the transactions. - uncles
array
- The list of uncle-headers of the block. This will only be set if full verification is required in order to create a merkle tree for the uncles and so prove the uncle_hash. - merkleProof
string[]
- The serialized merkle-nodes beginning with the root-node (depending on the content to prove). - merkleProofPrev
string[]
- The serialized merkle-nodes beginning with the root-node of the previous entry (only for full proof of receipts). - txProof
string[]
- The serialized merkle-nodes beginning with the root-node in order to proof the transactionIndex (only needed for transaction receipts). - logProof LogProof - The Log Proof in case of a
eth_getLogs
-request. - accounts
object
- A map of addresses and their AccountProof. - txIndex
integer
- The transactionIndex within the block (for transaactions and receipts). - signatures
Signature[]
- Requested signatures.
BlockProof¶
BlockProofs are used whenever you want to read data of a block and verify them. This would be:
- eth_getBlockTransactionCountByHash
- eth_getBlockTransactionCountByNumber
- eth_getBlockByHash
- eth_getBlockByNumber
The eth_getBlockBy...
methods return the Block-Data. In this case, all we need is somebody verifying the blockhash, which is done by requiring somebody who stored a deposit and would otherwise lose it, to sign this blockhash.
The verification is then done by simply creating the blockhash and comparing this to the signed one.
The blockhash is calculated by serializing the blockdata with rlp and hashing it:
blockHeader = rlp.encode([
bytes32( parentHash ),
bytes32( sha3Uncles ),
address( miner || coinbase ),
bytes32( stateRoot ),
bytes32( transactionsRoot ),
bytes32( receiptsRoot || receiptRoot ),
bytes256( logsBloom ),
uint( difficulty ),
uint( number ),
uint( gasLimit ),
uint( gasUsed ),
uint( timestamp ),
bytes( extraData ),
... sealFields
? sealFields.map( rlp.decode )
: [
bytes32( b.mixHash ),
bytes8( b.nonce )
]
])
For POA-chains, the blockheader will use the sealFields
(instead of mixHash and nonce) which are already RLP-encoded and should be added as raw data when using rlp.encode.
if (keccak256(blockHeader) !== singedBlockHash)
throw new Error('Invalid Block')
In case of the eth_getBlockTransactionCountBy...
, the proof contains the full blockHeader already serilalized plus all transactionHashes. This is needed in order to verify them in a merkle tree and compare them with the transactionRoot
.
Transaction Proof¶
TransactionProofs are used for the following transaction-methods:
- eth_getTransactionByHash
- eth_getTransactionByBlockHashAndIndex
- eth_getTransactionByBlockNumberAndIndex
In order to prove the transaction data, each transaction of the containing block must be serialized
transaction = rlp.encode([
uint( tx.nonce ),
uint( tx.gasPrice ),
uint( tx.gas || tx.gasLimit ),
address( tx.to ),
uint( tx.value ),
bytes( tx.input || tx.data ),
uint( tx.v ),
uint( tx.r ),
uint( tx.s )
])
and stored in a merkle tree with rlp.encode(transactionIndex)
as key or path, since the blockheader only contains the transactionRoot
, which is the root-hash of the resulting merkle tree. A merkle-proof with the transactionIndex of the target transaction will then be created from this tree.
The proof-data will look like these:
{
"jsonrpc": "2.0",
"id": 6,
"result": {
"blockHash": "0xf1a2fd6a36f27950c78ce559b1dc4e991d46590683cb8cb84804fa672bca395b",
"blockNumber": "0xca",
"from": "0x7e5f4552091a69125d5dfcb7b8c2659029395bdf",
"gas": "0x55f0",
"gasPrice": "0x0",
"hash": "0xe9c15c3b26342e3287bb069e433de48ac3fa4ddd32a31b48e426d19d761d7e9b",
"input": "0x00",
"value": "0x3e8"
...
},
"in3": {
"proof": {
"type": "transactionProof",
"block": "0xf901e6a040997a53895b48...", // serialized blockheader
"merkleProof": [ /* serialized nodes starting with the root-node */
"0xf868822080b863f86136808255f0942b5ad5c4795c026514f8317c7a215e218dc..."
"0xcd6cf8203e8001ca0dc967310342af5042bb64c34d3b92799345401b26713b43f..."
],
"txIndex": 0,
"signatures": [...]
}
}
}
Receipt Proof¶
Proofs for the transactionReceipt are used for the following method:
The proof works similiar to the transaction proof.
In order to create the proof we need to serialize all transaction receipts
transactionReceipt = rlp.encode([
uint( r.status || r.root ),
uint( r.cumulativeGasUsed ),
bytes256( r.logsBloom ),
r.logs.map(l => [
address( l.address ),
l.topics.map( bytes32 ),
bytes( l.data )
])
].slice(r.status === null && r.root === null ? 1 : 0))
and store them in a merkle tree with elp.encode(transactionIndex)
as key or path, since the blockheader only contains the receiptRoot
, which is the root-hash of the resulting merkle tree. A merkle proof with the transactionIndex of the target transaction receipt will then be created from this tree.
Since the merkle proof is only proving the value for the given transactionIndex, we also need to prove that the transactionIndex matches the transactionHash requested. This is done by adding another MerkleProof for the transaction itself as described in the Transaction Proof.
Log Proof¶
Proofs for logs are only for the one RPC-method:
Since logs or events are based on the TransactionReceipts, the only way to prove them is by proving the TransactionReceipt each event belongs to.
That’s why this proof needs to provide:
- all blockheaders where these events occured
- all TransactionReceipts plus their MerkleProof of the logs
- all MerkleProofs for the transactions in order to prove the transactionIndex
The proof data structure will look like this:
Proof {
type: 'logProof',
logProof: {
[blockNr: string]: { // the blockNumber in hex as key
block : string // serialized blockheader
receipts: {
[txHash: string]: { // the transactionHash as key
txIndex: number // transactionIndex within the block
txProof: string[] // the merkle Proof-Array for the transaction
proof: string[] // the merkle Proof-Array for the receipts
}
}
}
}
}
In order to create the proof, we group all events into their blocks and transactions, so we only need to provide the blockheader once per block. The merkle-proofs for receipts are created as described in the Receipt Proof.
Account Proof¶
Proofing an account-value applies to these functions:
Each of these values are stored in the account-object:
account = rlp.encode([
uint( nonce),
uint( balance),
bytes32( storageHash || ethUtil.KECCAK256_RLP),
bytes32( codeHash || ethUtil.KECCAK256_NULL)
])
The proof of an account is created by taking the state merkle tree and creating a MerkleProof. Since all of the above RPC-methods only provide a single value, the proof must contain all four values in order to encode them and verify the value of the MerkleProof.
For verification, the stateRoot
of the blockHeader is used and keccak(accountProof.address)
as the path or key within the merkle tree.
verifyMerkleProof(
block.stateRoot, // expected merkle root
keccak(accountProof.address), // path, which is the hashed address
accountProof.accountProof), // array of Buffer with the merkle-proof-data
isNotExistend(accountProof) ? null : serializeAccount(accountProof), // the expected serialized account
)
In case the account does not exist yet (which is the case if none
== startNonce
and codeHash
== '0xc5d2460186f7233c927e7db2dcc703c0e500b653ca82273b7bfad8045d85a470'
), the proof may end with one of these nodes:
- The last node is a branch, where the child of the next step does not exist.
- The last node is a leaf with a different relative key.
Both would prove that this key does not exist.
For eth_getStorageAt
, an additional storage proof is required. This is created by using the storageHash
of the account and creating a MerkleProof using the hash of the storage key (keccak(key)
) as path.
verifyMerkleProof(
bytes32( accountProof.storageHash ), // the storageRoot of the account
keccak(bytes32(s.key)), // the path, which is the hash of the key
s.proof.map(bytes), // array of Buffer with the merkle-proof-data
s.value === '0x0' ? null : util.rlp.encode(s.value) // the expected value or none to proof non-existence
))
Call Proof¶
Call proofs are used whenever you are calling a read-only function of a smart contract:
Verifying the result of an eth_call
is a little bit more complex because the response is a result of executing opcodes in the vm. The only way to do so is to reproduce it and execute the same code. That’s why a call proof needs to provide all data used within the call. This means:
- All referred accounts including the code (if it is a contract), storageHash, nonce and balance.
- All storage keys that are used (this can be found by tracing the transaction and collecting data based on the
SLOAD
-opcode). - All blockdata, which are referred at (besides the current one, also the
BLOCKHASH
-opcodes are referring to former blocks).
For verifying, you need to follow these steps:
- Serialize all used blockheaders and compare the blockhash with the signed hashes. (See BlockProof)
- Verify all used accounts and their storage as showed in Account Proof.
- Create a new VM with a MerkleTree as state and fill in all used value in the state:
// create new state for a vm
const state = new Trie()
const vm = new VM({ state })
// fill in values
for (const adr of Object.keys(accounts)) {
const ac = accounts[adr]
// create an account-object
const account = new Account([ac.nonce, ac.balance, ac.stateRoot, ac.codeHash])
// if we have a code, we will set the code
if (ac.code) account.setCode( state, bytes( ac.code ))
// set all storage-values
for (const s of ac.storageProof)
account.setStorage( state, bytes32( s.key ), rlp.encode( bytes32( s.value )))
// set the account data
state.put( address( adr ), account.serialize())
}
// add listener on each step to make sure it uses only values found in the proof
vm.on('step', ev => {
if (ev.opcode.name === 'SLOAD') {
const contract = toHex( ev.address ) // address of the current code
const storageKey = bytes32( ev.stack[ev.stack.length - 1] ) // last element on the stack is the key
if (!getStorageValue(contract, storageKey))
throw new Error(`incomplete data: missing key ${storageKey}`)
}
/// ... check other opcodes as well
})
// create a transaction
const tx = new Transaction(txData)
// run it
const result = await vm.runTx({ tx, block: new Block([block, [], []]) })
// use the return value
return result.vm.return
In the future, we will be using the same approach to verify calls with ewasm.
RPC-Methods Ethereum¶
This section describes the behavior for each standard-RPC-method.
web3_clientVersion¶
Returns the underlying client version.
See web3_clientversion for spec. No proof or verification possible.
web3_sha3¶
Returns Keccak-256 (not the standardized SHA3-256) of the given data.
See web3_sha3 for spec. No proof returned, but the client must verify the result by hashing the request data itself.
net_version¶
Returns the current network ID.
See net_version for spec. No proof returned, but the client must verify the result by comparing it to the used chainId.
eth_blockNumber¶
Returns the number of the most recent block.
See eth_blockNumber for spec.
No proof returned, since there is none, but the client should verify the result by comparing it to the current blocks returned from others. With the blockTime
from the chainspec, including a tolerance, the current blocknumber may be checked if in the proposed range.
eth_getBalance¶
Returns the balance of the account of a given address.
See eth_getBalance for spec.
An AccountProof, since there is none, but the client should verify the result by comparing it to the current blocks returned from others. With the blockTime
from the chainspec, including a tolerance, the current blocknumber may be checked if in the proposed range.