Bitcoin Q&A: Honest nodes and consensus

“What is the expected pattern that a hash is
reviewed against to determine a valid block?” “Are the attempts to find the nonce
manually or automatically implemented?” Let’s start with the second part. The attempts to find
[the nonce] are absolutely automatically implemented. These involve billions of hashes attempted per second,
billions of different nonces attempted per second. It is usually done entirely in
hardware, not even in software. It is done on application-specific integrated circuits
(ASICs), which is a specialised silicon-based chip. You put a block header into the device,
it plugs nonces and hashes as fast as possible, billions or trillions of times per second. Some of the newest equipment, I believe, is measured
in tera-hashes, which means trillions per second. “What is the expected pattern that the hash
is reviewed against to determine a valid block?” The pattern itself is the difficulty target,
which is a number that the [hash] must be ‘less than.’ Rather than thinking of it as a
pattern, think of it as a number. If you take [one of these] 256-bit hashes, it will look
like a pattern with a lot of zeros in the beginning. If you have a lot of zeroes in the beginning,
that just means it is smaller than another number. The smallest possible number you could [have]
in a hash, is all zeroes followed by one bit. That would be the greatest difficulty. Obviously, if all of the bits are ‘1,’ that is the easiest one
to find because everything is smaller than that. The difficulty is somewhere in that range, a number with
all zeroes in the beginning and [at least] one bit of ‘1.’ Your hash has to be smaller than that number. When the difficulty target gets lower,
finding a smaller number is more difficult. When the difficulty target gets higher,
finding a smaller number is easier. Think of [the target] like a limbo bar. As the limbo bar
gets lower, it becomes harder to squeeze underneath it. That is the general idea here. “What do you mean by [transaction validation]?
How does that process work?” Again, [transaction validation] is not
something that is done manually. All of these [processes] are done
automatically by software [and hardware]. When I run a copy of the Bitcoin Core [client] on my
laptop, it receives transactions several times a second… from other nodes that it is connected to; every time
it receives a transaction, my node will validate it. It will go through twenty or twenty-five
[consensus and validation] rules to make sure… the transaction passes all of these checks. What are some of these checks? Let’s think of some. One of the [rules] we’ve been discussing today: make sure the coins being spent, the inputs of
the transaction, have not been spent before. In order to [check] that, my node
has a copy of the blockchain so far… [to know] which coins have been spent
and which coins have been unspent. It keeps a database of all the unspent coins
called the unspent transaction output (UTXO) set. By keeping that database, nodes can immediately
check if [a transaction contains] unspent coins. If they are, proceed to the next validation. Every transaction has an amount
[of bitcoin] that it sends [as] outputs. What do we know about that amount?
It can’t be less than zero, obviously. We also know it can’t be more than twenty-one
million bitcoin, or twenty-one quadrillion satoshis. [Those rules] are kind of obvious, right?
The transaction validation [process will] make sure… all amounts, in all of the transactions, are
between zero and twenty-one quadrillion satoshis. That is the maximum amount [of bitcoin] that can exist. Next, it [will] check that the amounts add up. The total amount in the inputs must be greater than
or equal to the total amount in the outputs, [with any difference being paid to miners as a fee]. Then it will check that the inputs being
referenced exist and are properly validated. It will check that the scripts in those inputs have,
for example, a signature that is correctly applied. It will check that the scripts in the outputs are valid. It will check the data structure of the transaction, which
includes [parameters] like nLockTime and nSequence. Those [must] be correct, the header [must] be correct.
Once all 20-25 checks are done, it is a valid transaction. Similarly, every time a block is received, nodes
go through a process of validating that block… by checking against a long list of rules. These are the consensus rules. This is how consensus
happens. Consensus is an agreement on what “valid” is. [Consensus] is all of the software rules that
every transaction and block is checked against. To be ‘in consensus’ simply means ‘to be using the
same rules to check a transaction and / or block.’ “What is the difference between miners and nodes?” A node is a computer that is participating
in the peer-to-peer network and protocol. It may store a full copy of the blockchain or not. Usually, most nodes validate all transactions and blocks,
checking everything against the consensus rules. Miners run special-purpose nodes connected to mining
equipment, not only validating transactions and blocks [but] also aggregating transactions from
the mempool in order to produce the next block. So they are managing the queue of
upcoming, [unconfirmed] transactions. I run a node. My node also has a mempool and knows
what transactions have not been confirmed yet. The difference is, every ten minutes a miner’s node will
take a whole load of those [unconfirmed] transactions… and stuff them in a candidate block. It prepares a candidate block by [stuffing in] as many
transactions as it can, maximising the fees it collects. Then it will try to mine that block by finding a
special nonce, that will produce a special pattern… to match the difficulty requirements of the network. The difference between mining nodes and [non-mining]
nodes, is that the mining node is doing more work. It is also mining for new blocks. [But] almost
all nodes validate all transactions and blocks. “What are the upload and download requirements for
an internet connection to be okay with Bitcoin nodes?” Each block, at the moment, is about 1.5 megabytes
[or less], including the Segregated Witness information. In 1.5 megabytes, you will receive every transaction,
a copy of the block, and [then you must] validate them. You will [re-broadcast] every transaction at least once,
more likely two [or more] other adjacent nodes. You will be downloading about 1.5 to 3 megabytes
and uploading 3 to 10 megabytes every ten minutes. Add that up over a day, add that up over a month. That is approximately what it will cost,
in terms of data capacity to run a node. You can actually put restrictions on your node to
limit some of that data flow, but that is up to you. Fabiano asks, “Does the block need to be finished
for the validation of all its transactions?” Not quite sure what this question [means],
but I think I understand what [you are asking]. No, it doesn’t. When a miner receives a block from
someone else, they do not immediately validate it. They start building a new candidate block, with
no transactions in it, to start mining immediately. They validate it while their mining
equipment is already working. The chances that [a transaction was] propagated to
them through the network without being valid, is slim. Every node in the path [to the miner]
had to validate that [transaction or] block. They will validate the basics of proof-of-work
difficulty in the header, but not every transaction… in that block, unless they already
validated it in the mempool. As a shortcut, this is called light mining or SPV mining. It is mining without validation, [which can have]
some downsides, but miners do it anyway… certainly for the five or six seconds it takes to
fully validate the block and all of its transactions. [In that time], they start mining immediately
without validating on the previous block. Fabiana also asks, “Why should I run a full node
if I am not a miner?” The answer is really simple. You should run a full node if you want
to answer the question: ‘Did I get paid?’ There are two ways to answer that question. One way,
the naive way, is [to ask a third-party block explorer]. ‘Does think I got paid?’
‘Does somebody else’s node think I got paid?’ That is not the Bitcoin way. You are trusting somebody
else to [run] the decentralized protocol for you. [Another way], the correct way, of finding the answer to
‘Did I get paid?’ or ‘Have I received payments for this?’ is to have your own node evaluate the consensus rules. I run a node because I want to be able
to [independently] answer that question. For example, somebody just said they sent me
bitcoin, to pay me for [speaking at] a conference. How do I know if I have been paid?
How do I know if it received six confirmations? [How do I know] if it is a valid transaction? Do I ask and trust somebody else?
No, I run my own node, which does its own calculations. It answers that question for me, authoritatively. Authoritative and independent validation of
transactions, is why you run your own node. Perhaps, as an individual, that is not
very important to you. It is for me. I believe in the principle of it very much,
and I think that’s why it is important to me. Certainly if you’re running a merchant service,
an exchange, or anything else [important], you [shouldn’t] trust somebody else’s infrastructure to
tell you if you have been paid, [if you are dealing with…] perhaps hundreds, thousands, or
millions of dollars worth of bitcoin. You [must] make decisions in your software as to
whether you will allow a withdrawal, a trade, and to ship merchandise. These decisions will cost money if you are wrong. Perhaps the person you trust is the one
that wants to steal money from you. They [could] pretend that you have been paid,
when you in fact you haven’t been paid. That’s exactly why you would run your own node. Very good, here is one question from the forums: “On what basis does my node connect to other nodes?
Can I be cheated and connect to dishonest nodes?” That is a great question. Your node connects to other
nodes by a process that starts with discovering… some of the more well-known nodes, which are called
seeds, and then branching out to find other nodes. It does so by making its own decisions about
who it connects to, and then [trusting] these nodes… only as long as it can validate the transactions
and blocks it receives from them. Every node has to constantly prove it [is trustworthy]
by only sending good transactions and blocks. The moment a node we’re connected to
stops sending valid transactions and blocks, and starts sending incorrect transactions and blocks, our node will disconnect from them
and [even] ban them as being dishonest. Once they have discovered a few nodes that they
can trust, if you shut down or restart your node, it will connect to those nodes again, try
them out to see if they are still online. When it connects them, it will also ask,
“Do you know any honest nodes?” [Those nodes will] reply with a
list of other nodes they know. This is the addr (address) message on
the Bitcoin peer-to-peer network protocol. A node will ask [another node] via addr, “Do you have
any addresses for me?” The other node will respond, “I know these nodes that I have been talking to
for a while and have proven to be honest.” Your node will learn, from the nodes
it knows, about more honest nodes. It will connect to them randomly, test them,
and then ask them for more honest nodes. Each node does this [testing] to constantly
maintain a healthy network of honest nodes. Nodes that are dishonest very quickly
get disconnected from the network. “Could someone create enough virtual lieutenants
to bring down the General and replace him with… a new one of their choice — conduct
a blockchain coup d’état, as it were? This is a really critical question, which refers to
something we call a Sybil attack in security systems. A Sybil attack is where you create a lot
of fake nodes that pretend to be real. For example, let’s say you want to persuade people
that your opinion is very popular, when it actually isn’t. You create a lot of accounts on Facebook, all controlled
by you, and they will pretend to have the same opinion. People [may look] at that and think there
are a lot of people who hold this opinion, when in fact all of these accounts are
connected and controlled by a single individual. That is what is called a Sybil attack. These types of
[social] attacks are called sock-puppets sometimes. The key [feature] there is that you can have
one person or actor creating the fake accounts. Bitcoin directly solves the Sybil attack problem. It is the only system that really solves this
as part of the Byzantine Generals’ Problem. It solves the Sybil attack problem by requiring each
node, whether it is controlled by one entity or not, to produce proof-of-work. You can create a lot of fake [nodes], but if
you only have a certain amount of mining power, all you’re doing is splitting it among many systems
and that doesn’t do anything for you. It just [dilutes] the mining power
that each system can bring to bear. Identities don’t matter. The only thing that matters in the Bitcoin network
is the cumulative amount of hash power you bring. The number of “votes” you get, does not
increase when you have more nodes running. At least, they only increase in terms of
the [hash power in] the mining system. When you have more hashing power, whether you
[use it] through one or several nodes doesn’t matter. “Can you create enough virtual lieutenants?” Only if you buy enough hardware and spend enough
electricity, in order to power those lieutenants… in bringing down the General, [if the General
has significantly less hashing power]. That is the power of the Bitcoin consensus algorithm. “How does the network verify that the
winning miner solution was correct?” “How many times is the solution verified,
and by how many [network participants]?” The network verifies the winning
miner’s solution at every point. Let’s say I’m a miner and I have just found
a solution that I believe to be valid. I [will] announce it to all of nodes directly connected
to [me], in order to propagate it across the network. When I tell those nodes I have a new block,
they validate it before accepting the block. They don’t tell any other nodes until they validate it;
nodes do not propagate [transactions or blocks]… across the network until they [and]
every intermediate node validates. They don’t get to reach everyone unless
everyone validates them [independently]. The entire network verifies the solution, as soon as [each
node] sees it and before telling anyone else about it, to ensure that they’re not propagating invalid blocks. If a node propagates an invalid block to its neighbors,
it will get banned. The neighbors will disconnect. [Nodes disconnect] from other nodes that they consider
to be lying, to be propagating invalid information. They will isolate that node by cutting off connections.
You stop talking to people who lie on this network. So, everybody validates everything. Without validation,
[a transaction or block] doesn’t propagate. Only nodes that propagate [valid transactions and
blocks] are valid nodes, so only valid nodes propagate. Miners [usually] only see blocks, if every [other
node who propagated it] thought they are valid. “On my response about cutting connections with a node
that is lying, do all nodes find it at the same time?” “Can there be a mixed understanding of what is
an invalid node? How is consensus [found]?” It is entirely a matter of perspective. [You should
assume that] I only care about my perspective. If I am running a node, the perspective
of my node is the only one that matters. If I find a node that is lying to me,
I will disconnect myself from that node. I don’t really care what anybody else does. As far as [my node is] concerned, that node
isn’t following the consensus rules that I chose. From my perspective, [what they propagate
is] invalid. But is it invalid to everyone? No. Maybe I am the one running consensus
rules that are invalid to everybody else. Maybe I’m the one being cut off; in which
case, I might find myself in the minority, outside of the broader consensus. Consensus is [made through] individual perspectives.
I don’t tell anybody else who [or what] to believe. I don’t believe anybody else. My node will decide,
on its own, which nodes it will talk to, which answers it will believe,
and which rules it will validate by. My goal as a node operator, as someone who
chooses a wallet, or signs up with an exchange, is to pick the one that follows the consensus rules
that I believe are the right for this network.

39 thoughts on “Bitcoin Q&A: Honest nodes and consensus”

  1. Cool insights..
    I also embrace you plain background with out any propaganda, or anything to influence your followers to copy into their mind/life.. unique andreas 👍

  2. Again and again and tirelessly without emotive propaganda you deliver pure truth, thank you for your time Andreas.

  3. 3:12 – "Running a copy of Bitcoin Core" – say it aint so Andreas! Do you really say "Bitcoin Core" when referring to Bitcoin now, or am I hearing that wrong?

  4. Fantastic! Andreas, you make my day and make me feel good for following this amazing knowledge rabbit hole that you generously keep sharing with those who want to improve the world through this beautiful invention. Thank you Andreas…:-)

  5. What would happen if your node was in the process of validating a transaction and you reset your machine/server? Would the transaction be held in a paused status until the node was brought back online?

  6. Better to trust a few others than to rely on your own node. Because the information from your node may be compromised but it is less likely several others are all compromised at the same time

  7. Whats the difference between running a full node on public ip and non public? As far as u understand this, all those 100k passive nodes are behind ISP ip and are not visible for others, so 100k nodes are just downloading all info from that 15k visible nodes?

    How can we check the trx status in full node wallet? I dont see there any option for that..

    Thanks for all that years of educating the world about this great invention in the most friendly and understandable way Andreas!

  8. How do other nodes verify that the hash of a new block actually came from solving an algorithm rather than someone typing a bunch of characters that collectively constitute a number below the target difficulty?

  9. U have a genius brain and u look really tired. Think u need a rest and recharge. Plug in and charge me alien robot

  10. Hi Andreas I just want to give a big shoutout to you from Indonesia! I really hope that you are always well, and you are the voice of the people who does not have a voice. A lot of people are impacted positively in ways no one can imagine indirectly by your work without them ever knowing it! I hope that you will pay a visit to Indonesia, I'd be glad to be your host and bring you around in Jakarta

  11. Andreas: @ 12:13 you said "if it has 6 confirmations" –> why 6? (by the way, great great explanation on nodes and consensus algorithm! 😉 )

  12. Hi, I have some questions
    do the miners validate transactions ?
    should the transaction spread across the network until it reaches all the nodes ?
    do the nodes conserve the same mempool ?
    do miners keep a copy of the mempool?
    which inciste nodes to validate transactions ?
    finally I want to know what is the difference between a wallet and the Bitcoin client ?

Leave a Reply

Your email address will not be published. Required fields are marked *