Nxt Blockchain Tutorial
Introduction
Objective
Use a shell script to scan the Nxt blockchain, with an emphasis on understanding blockchain technology rather than speed. Based on the scan() method found in the Nxt software version 1.3.5 source file nxt/src/java/nxt/BlockchainProcessorImpl.Java
Assumptions
Nxt Java software running on your system, and familiarity with bash shell scripts running in a console window on a linux system. You have the jshon, bc and sha256sum utilities installed.
H2 Database Interfaces
The Nxt blockchain is stored in a Java H2 database (DB), located in your Nxt software installation directory (assumed to be nxt) on the path nxt/nxt_db. There are three tools available for interacting with the DB, two of which (DB Shell and H2 Console) use a browser interface and one of which (DB Shell) can operate concurrently with the Nxt Software without changing the nxt/conf/nxt.properties file.
Shell
To interact with the DB from a linux command line, use the Java Shell tool available in nxt/lib. For example
$ cd nxt; java -cp lib/h2*.jar org.h2.tools.Shell \ -url 'jdbc:h2:./nxt_db/nxt;DB_CLOSE_ON_EXIT=FALSE;CACHE_SIZE=439808;AUTO_SERVER=TRUE' \ -driver 'org.h2.Driver' -user sa -password sa \ -sql 'select base_target from block where height=0;'
returns
BASE_TARGET 153722867 (1 row, 19 ms)
The Shell tool will only connect with the DB if the Nxt software is not running, because it locks the DB. To avoid this problem add
nxt.dbUrl=jdbc:h2:nxt_db/nxt;DB_CLOSE_ON_EXIT=FALSE;AUTO_SERVER=TRUE
to the nxt/conf/nxt.properties file and restart the Nxt software. The AUTO_SERVER clause enables DB access from multiple processes simultaneously.
The Shell tool has an interactive mode accessible by invoking it without the -sql option. The command line mode is slow because a DB connection is established every time.
DB Shell
Enter http://localhost:7876/dbshell into a browser. A page loads with a prompt for an SQL statement. dbshell is built into the Nxt software and so must be used while the latter is running. The SQL query
SELECT * FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME='BLOCK';
gives a list of the columns (fields) of the block table, along with information such as data type. This tool is adequate, but the H2 Console is preferred because of its graphical interface.
H2 Console
In your Nxt software installation directory there is a library subdirectory (lib) with a jar file for the H2 DB. If launched as follows
$ cd nxt; java -cp "lib/h2*:lib/*" org.h2.tools.Console
a page opens in your browser with a DB connection dialog. Specify:
Saved Settings: Generic H2 (Embedded) Setting Name: Generic H2 (Embedded) Driver Class: org.h2.Driver JDBC URL: jdbc:h2:nxt_db/nxt;DB_CLOSE_ON_EXIT=FALSE;AUTO_SERVER=TRUE;CACHE_SIZE=439808 User Name: sa Password: sa
Clicking on Connect will connect with the DB but only if the Nxt software is not running, because it locks the DB. To avoid this problem add
nxt.dbUrl=jdbc:h2:nxt_db/nxt;DB_CLOSE_ON_EXIT=FALSE;AUTO_SERVER=TRUE
to the nxt/conf/nxt.properties file and restart the Nxt software. The AUTO_SERVER clause enables DB access from multiple processes simultaneously.
Use ^C in the shell console to terminate the DB engine, or click on the tool tray icon, then exit.
Database Fields
The BLOCK Table
In the left hand column of the H2 Console all tables in the DB are listed. Expanding the BLOCK table by clicking on "+" shows the block table columns (fields). Clicking the "+" next to each column name shows the column data type:
| Field Name | Data Type | NOT NULL |
|---|---|---|
| DB_ID | BIGINT(19) | yes |
| ID | BIGINT(19) | yes |
| VERSION | INTEGER(10) | yes |
| TIMESTAMP | INTEGER(10) | yes |
| PREVIOUS_BLOCK_ID | BIGINT(19) | no |
| TOTAL_AMOUNT | BIGINT(19) | yes |
| TOTAL_FEE | BIGINT(19) | yes |
| PAYLOAD_LENGTH | INTEGER(10) | yes |
| GENERATOR_PUBLIC_KEY | VARBINARY(32) | yes |
| PREVIOUS_BLOCK_HASH | VARBINARY(32) | no |
| CUMULATIVE_DIFFICULTY | VARBINARY(2147483647) | yes |
| BASE_TARGET | BIGINT(19) | yes |
| NEXT_BLOCK_ID | BIGINT(19) | no |
| HEIGHT | INTEGER(10) | yes |
| GENERATION_SIGNATURE | VARBINARY(64) | yes |
| BLOCK_SIGNATURE | VARBINARY(64) | yes |
| PAYLOAD_HASH | VARBINARY(32) | yes |
| GENERATOR_ID | BIGINT(19) | yes |
These 18 fields define a block on the blockchain in the current version of the Nxt software. Note that the block table has evolved since the genesis block. A complete history of the DB structure can be found in the Nxt software source file nxt/src/java/nxt/NxtDbVersion.java
Note that most fields cannot be NULL. The exceptions are the PREVIOUS_... and NEXT_... fields which link the blocks into a chain both forward and backward. The genesis block has a NULL PREVIOUS_BLOCK_ID and the last (current) block has a NULL NEXT_BLOCK_ID.
Below the column list is a list of indexes. The indexes are all used for sorting various columns for fast retrieval, but the following columns are also restricted to have unique values: DB_ID, HEIGHT, ID, TIMESTAMP. They are all used to uniquely identify blocks. DB_ID is the autoincrement field of the table. It normally increases by one with each new block but gaps can occur in the sequence due to occasional deleted blocks. HEIGHT is zero for the genesis block and increases by one with each block in the blockchain. There are no gaps in this sequence. ID is a unique block ID derived from the hash of some of the block fields. TIMESTAMP is the block creation time measured in number of seconds elapsed since the genesis block.
Note that blocks stored in the BLOCK table are associated with transactions stored in the TRANSACTION table through the fields PAYLOAD_LENGTH and PAYLOAD_HASH, and TOTAL_AMOUNT and TOTAL_FEE. PAYLOAD_LENGTH is the total number of bytes of certain fields of all transactions associated with the block and PAYLOAD_HASH is the hash of all those fields. TOTAL_AMOUNT and TOTAL_FEE are the total amounts and fees of all transactions associated with the block. All four of these block fields are zero when there are no transactions associated with the block.
The TRANSACTION Table
In the left hand column of the H2 Console all tables in the DB are listed. Expanding the TRANSACTION table by clicking on "+" shows the transaction table columns (fields). Clicking the "+" next to each column name shows the column data type:
| Field Name | Data Type | NOT NULL |
|---|---|---|
| DB_ID | BIGINT(19) | yes |
| ID | BIGINT(19) | yes |
| DEADLINE | SMALLINT(5) | yes |
| SENDER_PUBLIC_KEY | VARBINARY(32) | yes |
| RECIPIENT_ID | BIGINT(19) | no |
| AMOUNT | BIGINT(19) | yes |
| FEE | BIGINT(19) | yes |
| HEIGHT | INTEGER(10) | yes |
| BLOCK_ID | BIGINT(19) | yes |
| SIGNATURE | VARBINARY(64) | yes |
| TIMESTAMP | INTEGER(10) | yes |
| TYPE | TINYINT(3) | yes |
| SUBTYPE | TINYINT(3) | yes |
| SENDER_ID | BIGINT(19) | yes |
| BLOCK_TIMESTAMP | INTEGER(10) | yes |
| FULL_HASH | VARBINARY(32) | yes |
| REFERENCED_TRANSACTION_FULL_HASH | VARBINARY(32) | no |
| ATTACHMENT_BYTES | VARBINARY(2147483647) | no |
| VERSION | TINYINT(3) | yes |
| HAS_MESSAGE | BOOLEAN(1) | yes |
| HAS_ENCRYPTED_MESSAGE | BOOLEAN(1) | yes |
| HAS_PUBLIC_KEY_ANNOUNCEMENT | BOOLEAN(1) | yes |
| EC_BLOCK_HEIGHT | INTEGER(10) | no |
| EC_BLOCK_ID | BIGINT(19) | no |
| HAS_ENCRYPTTOSELF_MESSAGE | BOOLEAN(1) | yes |
These 25 fields define a transaction the current version of the Nxt software. Note that the transaction table has evolved since the genesis block. A complete history of the DB structure can be found in the Nxt software source file nxt/src/java/nxt/NxtDbVersion.java
Note that most fields cannot be NULL. The exceptions are RECIPIENT_ID, REFERENCED_TRANSACTION_FULL_HASH, ATTACHMENT_BYTES and the EC_BLOCK... fields. A transaction is valid without any of these fields specified.
Below the column list is a list of indexes. The indexes are all used for sorting various columns for fast retrieval, but the following columns are also restricted to have unique values: DB_ID, ID, FULL_HASH.
Note that transactions stored in the TRANSACTION table are associated with blocks stored in the BLOCK table through the fields HEIGHT, BLOCK_ID and BLOCK_TIMESTAMP.
The Nxt API
We could interact directly with the DB as shown above, but for the purpose of this tutorial, our script will obtain block and transaction data using Nxt software API calls.
The getBlock Call
The getBlock API call can be made from the shell using curl while the Nxt software is running:
$ h=300000; curl -s "http://localhost:7876/nxt?requestType=getBlock&height=$h"
This returns a JSON object for block height 300000. There are no extra spaces or newlines in the object, and the object does not include fields with null values. For clarity the object can be formatted as:
{
"previousBlockHash": "f5006dc3ae97f52aa27dc7c51888db5c6b0553aeeb51ad4bc74350edbef6e84e",
"payloadLength": 370,
"totalAmountNQT": "0",
"generationSignature": "5d08857979a8a900da73d37dc7b8c9d15acbf2644755523dcc3dccae722185ae",
"generator": "15893842156456397710",
"generatorPublicKey": "85d34459911c4c5eff7da30966334388d900fefe1082fe7b4d2cbe18706df422",
"baseTarget": "769068704",
"payloadHash": "0fcff1f8c4b384ee8e8940328827dd99d5d2aa88f401328195e55a5d5afba226",
"generatorRS": "NXT-ELWG-9FPV-KTP6-F7L6T",
"nextBlock": "3635605354783461399",
"requestProcessingTime": 1,
"numberOfTransactions": 2,
"blockSignature": "11a916927177d737ce8869ad892403b61c7b8008d63690ce9ff92250c982630a98d8d9f7b52717af585a97b7475d26d14cd108e6f0704cc5a6cbaef52d21b129",
"transactions": [
"5119943785891977024",
"5340614876563792980"
],
"version": 3,
"totalFeeNQT": "200000000",
"previousBlock": "3095547095745888501",
"cumulativeDifficulty": "11255618847538250",
"block": "16287921892019728450",
"height": 300000,
"timestamp": 32363409
}
The getTransaction Call
The getTransaction API call can be made from the shell using curl while the Nxt software is running:
$ tx=5119943785891977024; curl -s "http://localhost:7876/nxt?requestType=getTransaction&transaction=$tx"
This returns a JSON object for one of the two transactions at block height 300000. There are no extra spaces or newlines in the object, and the object does not include fields with null values. For clarity the object can be formatted as:
{
"senderPublicKey": "2fc5b938a98829d3cbefa60472eac11fd2cc09e6173b370ad6cc0ede666c6475",
"signature": "ae92f26b1ded2fd0f154e20cedf9b949ea0b1a1d637bf8e4ebaa52f6dcce7008c6373d2c85b84ea2f919fdd66a6d495005fa3e6b52c89330f7064fabf91ea870",
"feeNQT": "100000000",
"requestProcessingTime": 0,
"type": 2,
"confirmations": 8217,
"fullHash": "40570f24bfb10d47692bdfa25f3e175768b713b3ccf4bc4cb9cd7cf2ca094bca",
"version": 1,
"ecBlockId": "9591128410195604705",
"signatureHash": "4c9203be1378e5f8ca940794b180e343ec19ac9d38b7fe2f3c2ad9c792e6c305",
"attachment": {
"version.AskOrderCancellation": 1,
"order": "2574190861734682725"
},
"senderRS": "NXT-ADH7-TCCK-R46P-AHZSR",
"subtype": 4,
"amountNQT": "0",
"sender": "10080164044348861925",
"ecBlockHeight": 299990,
"block": "16287921892019728450",
"blockTimestamp": 32363409,
"deadline": 1440,
"transaction": "5119943785891977024",
"timestamp": 32363262,
"height": 300000
}
The getTransactionBytes Call
The getTransactionBytes API call can be made from the shell using curl while the Nxt software is running:
$ tx=5119943785891977024; curl -s "http://localhost:7876/nxt?requestType=getTransactionBytes&transaction=$tx"
This returns the signed and unsigned byte arrays of the transaction fields above packed in a certain order according to a complex algorithm that has changed since the genesis block:
{
"unsignedTransactionBytes": "0214fed2ed01a0052fc5b938a98829d3cbefa60472eac11fd2cc09e6173b...",
"requestProcessingTime": 0,
"confirmations": 12154,
"transactionBytes": "0214fed2ed01a0052fc5b938a98829d3cbefa60472eac11fd2cc09e6173b370ad6cc..."
}
Comparison of Database and API fields
Block
Note that there is not an exact correspondence between the 18 BLOCK table fields and the 21 getBlock fields:
| DB fields | API fields |
|---|---|
| DB_ID | |
| ID | block |
| VERSION | version |
| TIMESTAMP | timestamp |
| PREVIOUS_BLOCK_ID | previousBlock |
| TOTAL_AMOUNT | totalAmountNQT |
| TOTAL_FEE | totalFeeNQT |
| PAYLOAD_LENGTH | payloadLength |
| GENERATOR_PUBLIC_KEY | generatorPublicKey |
| PREVIOUS_BLOCK_HASH | previousBlockHash |
| CUMULATIVE_DIFFICULTY | cumulativeDifficulty |
| BASE_TARGET | baseTarget |
| NEXT_BLOCK_ID | nextBlock |
| HEIGHT | height |
| GENERATION_SIGNATURE | generationSignature |
| BLOCK_SIGNATURE | blockSignature |
| PAYLOAD_HASH | payloadHash |
| GENERATOR_ID | generator |
| generatorRS | |
| numberOfTransactions | |
| transactions | |
| requestProcessingTime |
The missing DB field is DB_ID which is not needed to uniquely identify a block. The extra API fields are the Reed-Solomon version of the generator ID, from which it can be computed, and transaction number and IDs associated with the block, and finally the API request processing time.
Transaction
Note that there is not an exact correspondence between the 25 TRANSACTION table fields and the 25 getBlock fields:
| DB fields | API fields |
|---|---|
| DB_ID | |
| ID | transaction |
| DEADLINE | deadline |
| SENDER_PUBLIC_KEY | senderPublicKey |
| RECIPIENT_ID | recipient |
| recipientRS | |
| AMOUNT | amountNQT |
| FEE | feeNQT |
| HEIGHT | height |
| BLOCK_ID | block |
| SIGNATURE | signature |
| signatureHash | |
| TIMESTAMP | timestamp |
| TYPE | type |
| SUBTYPE | subtype |
| SENDER_ID | sender |
| senderRS | |
| BLOCK_TIMESTAMP | blockTimestamp |
| FULL_HASH | fullHash |
| REFERENCED_TRANSACTION_FULL_HASH | referencedTransactionFullHash |
| ATTACHMENT_BYTES | attachment object |
| VERSION | version |
| HAS_MESSAGE | |
| HAS_ENCRYPTED_MESSAGE | |
| HAS_PUBLIC_KEY_ANNOUNCEMENT | |
| EC_BLOCK_HEIGHT | ecBlockHeight |
| EC_BLOCK_ID | ecBlockId |
| HAS_ENCRYPTTOSELF_MESSAGE | |
| confirmations | |
| requestProcessingTime |
The missing DB fields are DB_ID, which is not needed to uniquely identify a transaction, and the HAS_... booleans which all refer to messages which might be contained in the attachment and which can be determined by examining the attachment. The API has the extra fields signatureHash, which can be obtained from signature, ...RS; the Reed-Solomon versions of IDs, which can be obtained from the IDs; confirmations, which can be computed by comparing height with the last block height on the blockchain; and requestProcessingTime, which is the time to process the API call.
A Shell Script to Scan the Blockchain
Basic Script
Save the following script to a file named block.sh:
#!/bin/bash
function getFields {
local fields=$(echo $json | jshon -k)
local field; local type; local name; local names
for field in $fields; do
name="$1_$field"; names="$names $name"
type=$(echo $json | jshon -e $field -t)
if [[ $type == "string" || $type == "number" ]]; then
declare -g $name="$(echo $json | jshon -e $field -u)"
else
declare -g $name="$(echo $json | jshon -e $field)"
fi
#echo $name=${!name} # dump fields
done
declare -g "$1_fields"="$names"
}
err=; synched=0
if ((${#1})); then h=$1; else h=0; fi
while ((!synched)) || ((${#b})); do
json=$(curl -s "http://localhost:7876/nxt?requestType=getBlock&height=$h")
if ((${#json} == 0)); then err="server not responding"; break; fi
getFields 'bk'
if ((${#bk_errorDescription})); then err="$bk_errorDescription"; break; fi
echo "height=$bk_height, timestamp=$bk_timestamp"
# do checks here that only refer to the current block
if ((h != bk_height)); then err="blocks not in sequential height order"; break; fi
if ((synched)); then # do checks here that refer to the previous block
if ((b != bk_block)); then err="block ID mismatch"; break; fi
if ((pB != bk_previousBlock)); then err="previousBlock ID mismatch"; break; fi
fi
synched=1; ((++h))
pB=$bk_block; b=$bk_nextBlock
for var in $bk_fields; do unset $var; done # clear all block fields
done
if ((${#err})); then echo "Error: $err"; fi
Make the file executable:
$ chmod +x block.sh
Run it starting from the genesis block:
$ ./block.sh
The height and timestamp of each block beginning with the genesis block is printed on the console:
height=0, timestamp=0 height=1, timestamp=793 height=2, timestamp=1064 height=3, timestamp=1148 ...
This simple version of block.sh will scan the entire blockchain in about two hours.
If the "dump fields" line is uncommented
echo $var=${!var} # dump fields
the field variable names and values will be printed for each block.
Now, some explanatory comments about the script.
Call the API
The linux utility curl is used to get a block from the blockchain using the API call getBlock. It returns a JSON-formatted string.
json=$(curl -s "http://localhost:7876/nxt?requestType=getBlock&height=$h")
where $h is the block height and $json is the JSON-formatted block object retrieved.
Loop Control and Command Line Parameter
The variable $h is used to specify block height. It is initialized to the first command line argument $1 (zero default) and incremented near the end of the main block loop, to be checked against the DB field $bk_height in the next loop iteration. To scan the blockchain beginning at height 300000, use:
$ ./block.sh 300000
The block loop terminates when the end of the blockchain is reached. The last block in the chain will have nextBlock null and therefore nextBlock will be missing from the $json string. The value of $bk_nextBlock is stored in $b near the end of the block loop so that when its length is tested at the beginning of the next loop, it will have zero length ${#b} and terminate the loop, since all field variables are cleared at the end of the loop in case one becomes null in the next iteration.
Parsing the JSON Block Object
The getFields function extracts block and transaction fields from the $json string into variables such as $bk_nextBlock and $tx_version. It uses the linux utility program jshon. Lists of variable names $bk_fields and $tx_fields is kept so that the variables can be unset at the end of each loop, in case there are null DB fields omitted from $json in the next iteration.
Handling Errors
In the event of an improper API call, the $json string will contain an errorDescription field instead of block fields. If this is detected, the block loop breaks and an error report is printed.
All errors are handled the same way, by storing an error message in $err, exiting the loop, and finally printing the message.
Blockchain Integrity Checks
If valid block fields are retrieved, integrity checks can be performed in the main body of the block loop. There are two sections for checks. The first is for checks that only refer to the current block, while the second is for checks that refer to the previous block. The synched variable is used to ensure that the second section is only executed after one block has been processed and first-block variables have been stored near the end of the loop for use in the next block.
For example, $bk_block (the block ID of the current block) is stored as pB=$bk_block near the end of the loop, to be compared with $bk_previousBlock (the block ID of the previous block) in the next loop iteration. This is a simple check of blockchain integrity. The field $bk_previousBlock links the current block to the previous block. Likewise, the field $bk_nextBlock (the block ID of the next block) is stored as the variable $b to be compared with $bk_block (the block ID of the current block) in the next loop iteration.
Additional Blockchain Integrity Checks
The basic shell script can be enhanced to perform additional integrity checks on the blockchain. Each new check slows the program down, sometimes significantly, especially when an external, computationally intensive utility is used. For each check below, add it to the appropriate section. If it belongs in the synched section, store the relevant variables near the end of the block loop before the DB fields are cleared for the next iteration.
Transaction Checks
Transaction Number Check
txNum=$(echo $bk_transactions | jshon -l) if ((txNum != bk_numberOfTransactions)); then err="numberOfTransactions mismatch"; break; fi
Transaction Loop
txs=$(echo $bk_transactions | jshon -a -u) # get list of transaction IDs
for tx in $txs; do
json=$(curl -s "http://localhost:7876/nxt?requestType=getTransaction&transaction=$tx")
if ((${#json} == 0)); then err="server not responding"; break; fi
getFields 'tx'
if ((${#tx_errorDescription})); then err="$tx_errorDescription"; break; fi
json=$(curl -s "http://localhost:7876/nxt?requestType=getTransactionBytes&transaction=$tx")
if ((${#json} == 0)); then err="server not responding"; break; fi
getFields 'txb'
if ((${#txb_errorDescription})); then err="$txb_errorDescription"; break; fi
# place transaction checks here
for var in $tx_fields; do unset $var; done # clear all transaction fields
for var in $txb_fields; do unset $var; done # clear all transaction bytes fields
done
if ((${#err})); then break; fi
The getTransaction API call is used to get the transaction fields given the transaction ID. The getTransactionBytes API call is used to get the signed and unsigned transaction byte array (in hex form). The alternative to the latter is to pack together the transaction fields as done below with block fields, except that the packing algorithm is more complex.
The following transaction checks should be placed within the transaction loop as indicated except for designated pre-loop initializations and post-loop checks.
Amount and Fee Checks
Initialize before the transaction loop:
amt=0; fee=0
Accumulate within the loop:
((amt += tx_amountNQT)); ((fee += tx_feeNQT))
Check after the loop:
if ((amt != bk_totalAmountNQT)); then err="tx amountNQT mismatch"; break; fi if ((fee != bk_totalFeeNQT)); then err="tx feeNQT mismatch"; break; fi
Payload Length and Hash Checks
Initialize before the transaction loop:
payLen=0; payHex=
Accumulate within the loop:
((payLen += ${#txb_transactionBytes})); payHex=$payHex$txb_transactionBytes
Check after the loop:
if ((payLen / 2 != bk_payloadLength)); then err="payload length mismatch"; break; fi if [[ $(hexToHash $payHex) != $bk_payloadHash ]]; then err="payload hash mismatch"; break; fi
Check Other Transaction/Block Linkages
if [[ $tx_block != $bk_block ]]; then err="tx/bk block ID mismatch"; break; fi if ((tx_blockTimestamp != bk_timestamp)); then err="tx/bk timestamp mismatch"; break; fi if ((tx_height != bk_height)); then err="tx/bk height mismatch"; break; fi
Transaction Version Check
if ((tx_height <= 213000)); then v=0; else v=1; fi if ((v != tx_version)); then err="tx version mismatch"; break; fi
Transaction Hash and ID Checks
sH=$(hexToHash $tx_signature) if [[ $sH != $tx_signatureHash ]]; then err="tx signature hash mismatch"; break; fi if ((bk_version < 3)); then fH=$(hexToHash $txb_transactionBytes) else fH=$(hexToHash $txb_unsignedTransactionBytes$tx_signatureHash) fi txID=$(hashToID $fH) if [[ $fH != $tx_fullHash ]]; then err="tx full hash mismatch"; break; fi if [[ $txID != $tx_transaction ]]; then err="tx ID mismatch"; break; fi
Transaction Signature Check
if ((bk_height > 0)); then
v=${tx_signature:0:64}; h1=${tx_signature:64}
Y=$(java Curve25519 $v $h1 $tx_senderPublicKey $bk_version)
if (($?)); then err="tx signature invalid"; break; fi
m=$(hexToHash $txb_unsignedTransactionBytes); h2=$(hexToHash $m$Y)
if [[ $h1 != $h2 ]]; then err="tx signature not verified"; break; fi
fi
This check makes use of the Curve25519 Utility to verify the signature. The genesis block is skipped to avoid five bad signatures.
Block Version Check
if ((bk_height == 0)); then v=-1 elif ((bk_height <= 30000)); then v=1 elif ((bk_height <= 132000)); then v=2 else v=3; fi if ((v != bk_version)); then err="incorrect version"; break; fi
previousBlock Hash / ID Check
if ((bk_version > 1)) && (($(hashToID $bk_previousBlockHash) != bk_previousBlock)); then err="previousBlockHash doesn't give previousBlock ID"; break fi
This check makes use of the function
function hashToID { # convert 32-byte hex hash to decimal id
echo $(hexToDec $(hexToLE8 $1))
}
which in turn makes use of the functions
function hexToLE8 { # convert hex string to 8-byte (maximum) little endian
echo $1 | sed 's/../&\n/g' | head -8 | tac | tr -d '\n'
}
function hexToDec { # convert hex string (any length) to decimal
echo "ibase=16; ${1^^}" | bc
}
These functions can be included near the top of the script. They are written to echo a result which can be captured with $(...). $1 is the first function argument, $2 the second.
hexToLE8 converts a hex string to an 8-byte (16 hex digit) maximum little endian (LE) string, meaning that the byte order is reversed. All ID's in Nxt are 8-byte maximum LE unsigned integers, although internally in the Nxt software they are stored as signed integers since that is the only option in Java. Thus only the eight most significant bytes of the 32-byte previousBlockHash are used to compute the previousBlock ID.
hexToDec converts arbitrary size hex strings to decimal numbers. The utility bc (basic calculator) requries the hex sting to be capitalized (accomplished by ${1^^}) but can handle large strings, in contrast to the built-in printf %d 0x$1 which can only handle up to 63 bits.
Block ID Check
blockHex=$(decToLE8 4 $bk_version)$(decToLE8 4 $bk_timestamp) blockHex=$blockHex$(decToLE8 8 $bk_previousBlock)$(decToLE8 4 $bk_numberOfTransactions) tA=$bk_totalAmountNQT; tF=$bk_totalFeeNQT; w=8 if ((bk_version < 3)); then ((tA /= 100000000)); ((tF /= 100000000)); w=4; fi blockHex=$blockHex$(decToLE8 $w $tA)$(decToLE8 $w $tF) blockHex=$blockHex$(decToLE8 4 $bk_payloadLength)$bk_payloadHash$bk_generatorPublicKey$bk_generationSignature if ((bk_version > 1)); then blockHex=$blockHex$bk_previousBlockHash; fi blockHexNoSignature=$blockHex blockHex=$blockHex$bk_blockSignature blockHash=$(hexToHash $blockHex) blockID=$(hashToID $blockHash) if ((blockID != bk_block)); then err="computed block ID mismatch"; break; fi
$blockID is computed from the hash of a byte string composed of certain block fields packed in a certain order. Decimal number fields must first be converted to hex strings with the function decToLE8, which in turn uses the function hexToLE8 previously introduced:
function decToLE8 { # convert $1-byte (8-byte maximum) decimal $2 to little-endian hex
local hex=$(printf %0$(($1 * 2))x $2)
local len=$(($1 * 2 - ${#hex})); if ((len < 0)); then hex=${hex:0-len}; fi # handles version = -1
echo $(hexToLE8 $hex)
}
Once the hex string blockHex is ready, it is hashed using the function hexToHash, which relies on the utility sha256sum:
function hexToHash { # convert hex string to 32-byte hex hash
local hash=($(echo -ne $(echo $1 | sed 's/../\\x&/g') | sha256sum))
echo ${hash[0]}
}
The hex string is prepared by converting it to a byte string as expected by sha256sum; echo must not append a newline nor treat backslashes as escapes, hence echo -ne.
Finally, the block ID is computed from the hash using the previously mentioned hashToID function. It is then checked against the corresponding block field.
Note that the $blockHex string definition depends on the version of the blockchain. Version 1 introduced the previousBlock field, and version 3 replaced NXT with NQT=10-8 NXT in the amount and fee fields.
Block Signature Check
v=${bk_blockSignature:0:64}; h1=${bk_blockSignature:64}
Y=$(java Curve25519 $v $h1 $bk_generatorPublicKey $bk_version)
if (($?)); then err="block signature invalid"; break; fi
m=$(hexToHash $blockHexNoSignature); h2=$(hexToHash $m$Y)
if [[ $h1 != $h2 ]]; then err="block signature not verified"; break; fi
There is no linux utility available for verifying Curve25519 signatures, so the same Curve25519 Java class used by the Nxt software is used. The $bk_blockSignature field is split in half and each half is passed along with $bk_generatorPublicKey to the customized Curve25519 utility, which returns $Y. $Y is hashed with the hash of $blockHexNoSignature, which is just $blockHex without the trailing $bk_blockSignature field.
Curve25519 Utility - Java Version
Curve25519.java can be acquired from the Nxt software directory nxt/src/java/nxt/crypto/Curve25519.java. A main method can be added to the Curve25519 class as follows, after removing the first line (package nxt.crypto;):
final class Curve25519 {
// Usage: java Curve25519 v h publicKey version
// signature is (v 32 bytes + h 32 bytes), public key is 32 bytes, version is integer
public static void main (String[] args) {
if (args.length != 4) System.exit(1);
byte [] v = hexStringToByteArray(args[0]);
byte [] h = hexStringToByteArray(args[1]);
byte [] publicKey = hexStringToByteArray(args[2]);
int version = Integer.parseInt(args[3]);
if (version >= 3) {
byte[] signature = new byte[v.length + h.length];
System.arraycopy(v, 0, signature, 0, v.length);
System.arraycopy(h, 0, signature, v.length, h.length);
if(!Curve25519.isCanonicalSignature(signature)) System.exit(2);
if(!Curve25519.isCanonicalPublicKey(publicKey)) System.exit(3);
}
byte[] Y = new byte[32];
Curve25519.verify(Y, v, h, publicKey);
System.out.println(bytesToHex(Y));
System.exit(0);
}
public static byte[] hexStringToByteArray(String s) {
int len = s.length();
byte[] data = new byte[len / 2];
for (int i = 0; i < len; i += 2) {
data[i / 2] = (byte) ((Character.digit(s.charAt(i), 16) << 4) + Character.digit(s.charAt(i+1), 16));
}
return data;
}
final protected static char[] hexArray = "0123456789ABCDEF".toCharArray();
public static String bytesToHex(byte[] bytes) {
char[] hexChars = new char[bytes.length * 2];
for ( int j = 0; j < bytes.length; j++ ) {
int v = bytes[j] & 0xFF;
hexChars[j * 2] = hexArray[v >>> 4];
hexChars[j * 2 + 1] = hexArray[v & 0x0F];
}
return new String(hexChars);
}
...
}
This method provides a command line interface that accepts and returns hex strings. It also checks the input fields for canonical form when version >= 3.
Compile the modified class with:
$ javac Curve25519.java
The resulting Curve25519.class and Curve25519$long10.class files are now ready for use with the java command
$ java Curve25519 v h publicKey version
curve25519 Utility - C Version
Curve25519.java was ported to C and is available here. Acquire curve25519.c and compile it with:
$ gcc curve25519.c -o curve25519
This version includes all of the cryptographic methods from the original Java source code, plus a main function that provides a convenient interface allowing key pair generation, signing, verifying and key agreement. Here we only need verification. Everywhere where the Java utility is invoked:
Y=$(java Curve25519 $v $h1 ...PublicKey $bk_version)
the C utility can be invoked instead:
Y=$(./curve25519 -v $v $h1 ...PublicKey)
where the executable is assumed to be in the same directory as block.sh. Note that the C version always checks for canonical form, regardless of version; hence the final argument is omitted.
previousBlockHash Check (Synched)
if ((bk_version > 1)) && [[ $pBH != $bk_previousBlockHash ]]; then err="previousBlockHash mismatch"; break; fi
Insert the following near the end of the main block loop:
pBH=$blockHash
generationSignature Check (Synched)
if ((bk_version == 1)); then
v=${bk_generationSignature:0:64}; h1=${bk_generationSignature:64}
Y=$(java Curve25519 $v $h1 $bk_generatorPublicKey $bk_version)
if (($? > 1)); then err="canonical failure"; break; fi
m=$(hexToHash $pGS); h2=$(hexToHash $m$Y)
if [[ $h1 != $h2 ]]; then err="version == 1 generation signature not verified"; break; fi
else
generationSignatureHash=$(hexToHash $pGS$bk_generatorPublicKey)
if [[ $generationSignatureHash != $bk_generationSignature ]]; then
err="version > 1 generation signature not verified"; break
fi
fi
Insert the following near the end of the main block loop:
pGS=$bk_generationSignature
When version = 1, $bk_generationSignature is checked exactly like the $bk_blockSignature. With version 2, a simple hash algorithm not involving a signature is used. The change was made because there was a flaw in the algorithm which used generationSignature in forging calculations due to randomness in Curve22519 signature generation.
baseTarget Check (Synched)
bT=$(((pBT * (bk_timestamp - pT)) / 60)); if ((bT > MAX_BASE_TARGET)); then bT=$MAX_BASE_TARGET;fi if ((bT < pBT / 2)); then ((bT = pBT / 2)); fi if ((bT == 0)); then bT=1; fi pBTx2=$((pBT * 2)); if ((pBTx2 > MAX_BASE_TARGET)); then pBTx2=$MAX_BASE_TARGET; fi if ((bT > pBTx2)); then bT=$pBTx2; fi if ((bT != bk_baseTarget)); then err="baseTarget mismatch"; break; fi
Insert the following near the end of the main block loop:
pBT=$bk_baseTarget; pT=$bk_timestamp
Insert the following constant initializations before the start of the main block loop:
TWO64=$(echo "ibase=16; 10000000000000000" | bc) # 2^64 in decimal MAX_BALANCE_NXT=1000000000 GENESIS_BASE_TARGET=$(echo "$TWO64 / (2 * 60 * $MAX_BALANCE_NXT)" | bc) # 153722867 MAX_BASE_TARGET=$((MAX_BALANCE_NXT * GENESIS_BASE_TARGET))
$MAX_BALANCE_NXT is the total amount of NXT in existence, 1 billion. $GENESIS_BASE_TARGET is the initial $bk_baseTarget at height zero. It is used to determine which Nxt node will forge a new block. A hit value is computed for each node based on an effectively random 8-byte (64-bit) number computed from $bk_generationSignature. All nodes with a hit value less than $bk_baseTarget * $tx_effectiveBalance (computed from transaction data) * (time since last block, in seconds) are permitted to submit new blocks to the Nxt network for inclusion in the blockchain. The candidate blockchain with the greatest $bk_cumulativeDifficulty (inversely related to $bk_baseTarget) is accepted by the network. Eventually some node will get a hit because it becomes easier with each passing second.
If all $MAX_BALANCE_NXT NXT in existence were concentrated in one node, $bk_baseTarget * $tx_effectiveBalance would be TWO64 / (2 * 60) which that node's hit value would have a 50% chance of exceeding in 60 attempts. Theoretically then, a hit occurs on average once per minute throughout the network if all NXT tokens are available for forging. In practice the average elapsed time between blocks is somewhat longer.
$bk_baseTarget is dynamically adjusted to maintain a steady blockchain growth rate. If blocks are being generated too slowly, $bk_baseTarget is increased proportional to the time elapsed since the previous block, but is not allowed to more than double nor less than halve with each new block.
cumulativeDifficulty Check (Synched)
cD=$(echo "$pCD + $TWO64 / $bk_baseTarget" | bc) if [[ $cD != $bk_cumulativeDifficulty ]]; then err="cumulativeDifficulty mismatch"; break; fi
Insert the following near the end of the main block loop:
pCD=$bk_cumulativeDifficulty
The utility bc is necessary, since $TWO64 is out of the normal long integer range and $bk_cumulativeDifficulty grows without limit; it will eventually go out of range.
Complete Script
#!/bin/bash
function hexToLE8 { # convert hex string to 8-byte (maximum) little endian
echo $1 | sed 's/../&\n/g' | head -8 | tac | tr -d '\n'
}
function hexToDec { # convert hex string (any length) to decimal
echo "ibase=16; ${1^^}" | bc
}
function hashToID { # convert 32-byte hex hash to decimal id
echo $(hexToDec $(hexToLE8 $1))
}
function decToLE8 { # convert $1-byte (8-byte maximum) decimal $2 to little-endian hex
local hex=$(printf %0$(($1 * 2))x $2)
local len=$(($1 * 2 - ${#hex})); if ((len < 0)); then hex=${hex:0-len}; fi # handles version = -1
echo $(hexToLE8 $hex)
}
function hexToHash { # convert hex string to 32-byte hex hash
local hash=($(echo -ne $(echo $1 | sed 's/../\\x&/g') | sha256sum))
echo ${hash[0]}
}
TWO64=$(echo "ibase=16; 10000000000000000" | bc) # 2^64 in decimal
MAX_BALANCE_NXT=1000000000
GENESIS_BASE_TARGET=$(echo "$TWO64 / (2 * 60 * $MAX_BALANCE_NXT)" | bc) # 153722867
MAX_BASE_TARGET=$((MAX_BALANCE_NXT * GENESIS_BASE_TARGET))
function getFields {
local fields=$(echo $json | jshon -k)
local field; local type; local name; local names
for field in $fields; do
name="$1_$field"; names="$names $name"
type=$(echo $json | jshon -e $field -t)
if [[ $type == "string" || $type == "number" ]]; then
declare -g $name="$(echo $json | jshon -e $field -u)"
else
declare -g $name="$(echo $json | jshon -e $field)"
fi
#echo $name=${!name} # dump fields
done
declare -g "$1_fields"="$names"
}
err=; synched=0
if ((${#1})); then h=$1; else h=0; fi
while ((!synched)) || ((${#b})); do
json=$(curl -s "http://localhost:7876/nxt?requestType=getBlock&height=$h")
if ((${#json} == 0)); then err="server not responding"; break; fi
getFields 'bk'
if ((${#bk_errorDescription})); then err="$bk_errorDescription"; break; fi
echo "height=$bk_height, timestamp=$bk_timestamp"
# do checks here that only refer to the current block
txNum=$(echo $bk_transactions | jshon -l)
if ((txNum != bk_numberOfTransactions)); then err="numberOfTransactions mismatch"; break; fi
amt=0; fee=0
payLen=0; payHex=
txs=$(echo $bk_transactions | jshon -a -u) # get list of transaction IDs
for tx in $txs; do
json=$(curl -s "http://localhost:7876/nxt?requestType=getTransaction&transaction=$tx")
if ((${#json} == 0)); then err="server not responding"; break; fi
getFields 'tx'
if ((${#tx_errorDescription})); then err="$tx_errorDescription"; break; fi
json=$(curl -s "http://localhost:7876/nxt?requestType=getTransactionBytes&transaction=$tx")
if ((${#json} == 0)); then err="server not responding"; break; fi
getFields 'txb'
if ((${#txb_errorDescription})); then err="$txb_errorDescription"; break; fi
# place transaction checks here
((amt += tx_amountNQT)); ((fee += tx_feeNQT))
((payLen += ${#txb_transactionBytes})); payHex=$payHex$txb_transactionBytes
if [[ $tx_block != $bk_block ]]; then err="tx/bk block ID mismatch"; break; fi
if ((tx_blockTimestamp != bk_timestamp)); then err="tx/bk timestamp mismatch"; break; fi
if ((tx_height != bk_height)); then err="tx/bk height mismatch"; break; fi
if ((tx_height <= 213000)); then v=0; else v=1; fi
if ((v != tx_version)); then err="tx version mismatch"; break; fi
sH=$(hexToHash $tx_signature)
if [[ $sH != $tx_signatureHash ]]; then err="tx signature hash mismatch"; break; fi
if ((bk_version < 3)); then
fH=$(hexToHash $txb_transactionBytes)
else
fH=$(hexToHash $txb_unsignedTransactionBytes$tx_signatureHash)
fi
txID=$(hashToID $fH)
if [[ $fH != $tx_fullHash ]]; then err="tx full hash mismatch"; break; fi
if [[ $txID != $tx_transaction ]]; then err="tx ID mismatch"; break; fi
if ((bk_height > 0)); then
v=${tx_signature:0:64}; h1=${tx_signature:64}
Y=$(java Curve25519 $v $h1 $tx_senderPublicKey $bk_version)
if (($?)); then err="tx signature invalid"; break; fi
m=$(hexToHash $txb_unsignedTransactionBytes); h2=$(hexToHash $m$Y)
if [[ $h1 != $h2 ]]; then err="tx signature not verified"; break; fi
fi
for var in $tx_fields; do unset $var; done # clear all transaction fields
for var in $txb_fields; do unset $var; done # clear all transaction bytes fields
done
if ((${#err})); then break; fi
if ((amt != bk_totalAmountNQT)); then err="tx amountNQT mismatch"; break; fi
if ((fee != bk_totalFeeNQT)); then err="tx feeNQT mismatch"; break; fi
if ((payLen / 2 != bk_payloadLength)); then err="payload length mismatch"; break; fi
if [[ $(hexToHash $payHex) != $bk_payloadHash ]]; then err="payload hash mismatch"; break; fi
if ((h != bk_height)); then err="blocks not in sequential height order"; break; fi
if ((bk_height == 0)); then v=-1
elif ((bk_height <= 30000)); then v=1
elif ((bk_height <= 132000)); then v=2
else v=3; fi
if ((v != bk_version)); then err="incorrect version"; break; fi
if ((bk_version > 1)) && (($(hashToID $bk_previousBlockHash) != bk_previousBlock)); then
err="previousBlockHash doesn't give previousBlock ID"; break
fi
blockHex=$(decToLE8 4 $bk_version)$(decToLE8 4 $bk_timestamp)
blockHex=$blockHex$(decToLE8 8 $bk_previousBlock)$(decToLE8 4 $bk_numberOfTransactions)
tA=$bk_totalAmountNQT; tF=$bk_totalFeeNQT; w=8
if ((bk_version < 3)); then ((tA /= 100000000)); ((tF /= 100000000)); w=4; fi
blockHex=$blockHex$(decToLE8 $w $tA)$(decToLE8 $w $tF)
blockHex=$blockHex$(decToLE8 4 $bk_payloadLength)$bk_payloadHash$bk_generatorPublicKey$bk_generationSignature
if ((bk_version > 1)); then blockHex=$blockHex$bk_previousBlockHash; fi
blockHexNoSignature=$blockHex
blockHex=$blockHex$bk_blockSignature
blockHash=$(hexToHash $blockHex)
blockID=$(hashToID $blockHash)
if ((blockID != bk_block)); then err="computed block ID mismatch"; break; fi
v=${bk_blockSignature:0:64}; h1=${bk_blockSignature:64}
Y=$(java Curve25519 $v $h1 $bk_generatorPublicKey $bk_version)
if (($?)); then err="block signature invalid"; break; fi
m=$(hexToHash $blockHexNoSignature); h2=$(hexToHash $m$Y)
if [[ $h1 != $h2 ]]; then err="block signature not verified"; break; fi
if ((synched)); then # do checks here that refer to the previous block
if ((b != bk_block)); then err="block ID mismatch"; break; fi
if ((pB != bk_previousBlock)); then err="previousBlock ID mismatch"; break; fi
if ((bk_version > 1)) && [[ $pBH != $bk_previousBlockHash ]]; then err="previousBlockHash mismatch"; break; fi
if ((bk_version == 1)); then
v=${bk_generationSignature:0:64}; h1=${bk_generationSignature:64}
Y=$(java Curve25519 $v $h1 $bk_generatorPublicKey $bk_version)
if (($? > 1)); then err="canonical failure"; break; fi
m=$(hexToHash $pGS); h2=$(hexToHash $m$Y)
if [[ $h1 != $h2 ]]; then err="version == 1 generation signature not verified"; break; fi
else
generationSignatureHash=$(hexToHash $pGS$bk_generatorPublicKey)
if [[ $generationSignatureHash != $bk_generationSignature ]]; then
err="version > 1 generation signature not verified"; break
fi
fi
bT=$(((pBT * (bk_timestamp - pT)) / 60)); if ((bT > MAX_BASE_TARGET)); then bT=$MAX_BASE_TARGET;fi
if ((bT < pBT / 2)); then ((bT = pBT / 2)); fi
if ((bT == 0)); then bT=1; fi
pBTx2=$((pBT * 2)); if ((pBTx2 > MAX_BASE_TARGET)); then pBTx2=$MAX_BASE_TARGET; fi
if ((bT > pBTx2)); then bT=$pBTx2; fi
if ((bT != bk_baseTarget)); then err="baseTarget mismatch"; break; fi
cD=$(echo "$pCD + $TWO64 / $bk_baseTarget" | bc)
if [[ $cD != $bk_cumulativeDifficulty ]]; then err="cumulativeDifficulty mismatch"; break; fi
fi
synched=1; ((++h))
pB=$bk_block; b=$bk_nextBlock; pBH=$blockHash; pGS=$bk_generationSignature
pBT=$bk_baseTarget; pT=$bk_timestamp; pCD=$bk_cumulativeDifficulty
for var in $bk_fields; do unset $var; done # clear all block fields
done
if ((${#err})); then echo "Error: $err"; fi
