Participating in the ASMap deployment process and building your own ASMap

[fjahr]

Description

Disclaimer: the process described in the links below are still up for discussion, so details may change.

What is this workshop about? Provide as many details as possible.

We will dive deep into the nuances of internet routing and how this leads us to the insight that using an ASMap file is a good practice for anyone running a node. Then we will discuss how an ASMap file is constructed under the hood, what data sources can be used and what attack vectors exist. We will then practice creating our own ASMap file for use in our personal nodes and review each others ASMap files using publicly available tooling.

What would an attendee learn from this workshop?

• How the usage of ASMap files protects your node and strengthens the network
• A better understanding of how internet routing on clearnet works, including who the participants are, how BGP works, what the issues are, and what other resources are available to improve security
• How the ASMap file to be used in Bitcoin Core is constructed step-by-step on a technical level, leveraging the learnings from the point above and discussing the trade-offs that need to be made
• How they can build their own ASMap file, check its validity and use it in their own Bitcoin Core node
• How the Bitcoin Core deployment process works, how the ASMap file is a part of it and how they can help in that process

Is there anything attendees should read up on before they attend this talk?

They should be familiar with the basic idea of the ASMap feature: https://blog.bitmex.com/call-to-action-testing-and-improving-asmap/ and the more they read the relevant links below, the better, but it's not a prerequisite for successful participation.

If my related talk is accepted it would also be sufficient to just listen to that.

Is there anything attendees should set up before the workshop?

• They should have the bitcoin core repo cloned and build a recent version of master, the latest version of the bitcoin release binary is also fine though it may not work for every single exercise
• I would also help if they had already downloaded the tools we will be using. It's not clear yet which ones it will be but it will most likely be simple python tools, such as https://github.com/fjahr/kartograf
• They should have internet access but maybe I could manage without if I know in advance

Relevant Links

• Longer write-up on the current proposal
• Condensed step-by-step description of the release process
• kartograf

About the Speaker

I have been contributing to Bitcoin Core since 2019. Among other things I have developed the CoinstatsIndex. In 2022, I have spent some time building a better understanding of the larger internet routing infrastructure. This was necessary to understand and evaluate the potential data sources for ASMap files and create a proposal for ASMap in the Bitcoin Core release process. This effort had been pending since 2020 when the feature was merged as an option. I am also an organizer of the Bitdevs Socratic Seminar in Berlin .

Social Links

Github: https://github.com/fjahr
Twitter: https://twitter.com/fjahr
Website: https://fjahr.com/

Workshop Details

Length of workshop

~2 hours

Preferred Day/Time Slot

No preference

[JoseMoranUrena523]

I'll definitely read this.

[iglesiasbrandon]

Hey, @fjahr, thank you for submitting an issue for TABConf 2023! As we get closer to the conference, we will review all the issues, and the chosen ones will be added to the TABConf 2023 schedule: https://github.com/orgs/TABConf/projects/1.

I don't expect us to add any issues to the schedule until a month or two before the conference but keep an eye on this in case someone asks questions. Also, please feel free to update the issue description if you'd like to add anything else as we get closer to the conf.

[iglesiasbrandon]

Hey @fjahr, congratulations, this issue has been accepted for TABConf 2023!

It's been added to the official GitHub Project schedule.
The workshops will be happening on September 6th and 7th; once we accept some more, we will assign the day, time, and location.

[iglesiasbrandon]

Hey @fjahr , We have added a day, time slot, and length to this issue.
You can find it on the schedule in this view: https://github.com/orgs/TABConf/projects/1/views/4

Please acknowledge by commenting 'ack'.

This will confirm your workshop for TABConf 2023!
If we need to make any changes, you will be notified on this issue.

[fjahr]

Hey @fjahr , We have added a day, time slot, and length to this issue. You can find it on the schedule in this view: https://github.com/orgs/TABConf/projects/1/views/4

Please acknowledge by commenting 'ack'.

This will confirm your workshop for TABConf 2023! If we need to make any changes, you will be notified on this issue.

ACK

[miketwenty1]

@fjahr

We are about a week away from the start of the conference. I'm sending this to all accepted upcoming workshop. Please review and ensure you have a clear, up to date, high level, agenda on this GitHub issue. Please buffer in time to help with unexpected problems and questions.

TABConf workshops should aim to involve high levels participation. If you're planning to conduct a live demo, please consider that attendees will require clear instructions on how to prepare for following along, (all steps for setup should be double checked) and any presentation should be structured accordingly.
Participants should leave the workshop either with something built, (be it physical or digital), or with specific knowledge gained from the hands on/interactive experience.

[fjahr]

Here is the latest version of the agenda and notes for preparation. While we do have several hands-on elements in the workshop, it is also fine if you join without having gone through the preparation steps or even without a laptop. If you have used your laptop for development before, we should able to go through the setup steps in a few minutes together (though we are at the mercy of the conference WIFI :p ). We will also discuss each of the steps as a group, so you should still get a lot out of the participation without a machine to follow along.

I am looking forward to seeing you in ATL next week!

Agenda

1. Why ASMap?

• How clearnet works under the hood, especially the "global" routing table, roles of involved parties etc.
• What the issues with BGP are and what the beginnings of the Internet have to do with it
• Discussion of pros and cons of data sources for creating a copy of the global routing table including IRR, RPKI, and others

2. Hands-on: Validating and using an existing ASMap file

• Going through the process currently favored for validating a candidate ASMap file for Bitcoin Core releases
• Loading an ASMap file into Bitcoin Core incl. Health Check Demo in Bitcoin Core

3. Code walkthrough: Creating your own ASMap file

• See how previously discussed issues are solved in Kartograf
• Discuss potential API improvements

4. Open Q&A

• Diving deeper into topics interesting to the group
• Feedback on the release process steps and participation options

Preparation

Please note that the tools have not been tested on Windows machines and I am unsure if rpki-client would work. If in doubt, I would recommend Windows users work within a Linux VM for the workshop.

Mandatory

1. Get Kartograf

$ git clone https://github.com/fjahr/kartograf.git

Install python dependencies

$ cd kartograf
$ pip3 install -r requirements.txt

2. Install RPKI client

This tool is used heavily by Kartograf. Be aware that it may not be available for older OS versions.

Linux/BSD

$ {pkg,dnf,yum,apt} install rpki-client

macOS

$ brew install rpki-client

3. Get asmap-tool

For the compression of the asmap file we need the asmap tool from sipa. To get the latest version we also need to switch the branch nextgen.

$ git clone https://github.com/sipa/asmap.git
$ cd asmap
$ git fetch origin
$ git checkout nextgen

4. Get demo files

This is the raw input data that we will validate as an example. To save us some time, the data is truncated, so don't use the resulting ASMap file for any production nodes!

$ git clone https://github.com/fjahr/asmap-data.git
$ cd asmap-data
$ git fetch origin
$ git checkout tabconf

Now the large source file is stored using Git LFS. Unless you have it set up already you will also need to go through the following additional steps while being in the repo to download the actual file content.

# For Ubuntu/Debian
$ apt-get install git-lfs
# For macOS
$ brew install git-lfs

$ git lfs install
$ git lfs pull

As a final step, unzip the contents of the included ZIP file.

$ unzip 20230901_fjahr_tabconf/reproduction_inputs.zip

Bonuses

If you have more time available and want to be even better prepared, feel free to dive into any of the following suggestions.

1. Refresh your ASMap memory

This write-up by Gleb in the Bitmex blog is partly outdated but still gives a really nice high-level summary of why we want to use ASMap. If you currently have no idea what ASMap even is or why it's interesting, it would be great if you give this a read (5 min).

https://blog.bitmex.com/call-to-action-testing-and-improving-asmap/

2. Check out ASMap Health Check branch of Bitcoin Core

This is still an unmerged PR. Assuming you have already checked out Bitcoin Core, installed dependencies etc. If that is not the case please follow the INSTALL readme first.

$ git fetch upstream pull/27581/head:asmap_health_check
$ git checkout asmap_health_check

Ideally, you can build this branch too, to have it ready. To test that it works, run bitcoind with -debug=net and watch the logs.

3. More reading materials

• ASMap in Bitcoin Core releases/ASMap data sources
• Theoretical ASMap deployment for v25.0 release (be aware that this process I suggested has changed a bit based on feedback)
• How can a malicious ASMap file be created and what mechanisms can detect this?

4. Build your own ASMap file with Katograf

Follow the instructions in the README on how to build an actual ASMap file based on real data. Beware that this is a multistep process that takes many hours and there is currently no convenient way to pause the process, so I would suggest letting this run over night if you want to try.

Questions/Feedback

Let me know if you have questions or feedback on this document. I.e. if the setup didn't work for you or if you would like to discuss specific topics that are currently not on the agenda, comment on this issue or ping me on Twitter X or IRC, I am fjahr pretty much everywhere.

[iglesiasbrandon]

Hey @fjahr , I am closing this issue since you have completed the workshop. Thank you for all of the effort you put into it!