Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RPKI Prefix Origin Validation Support #33

Merged
merged 3 commits into from
Aug 7, 2018
Merged

Conversation

salsh
Copy link
Contributor

@salsh salsh commented May 22, 2018

  • The configure script checks whether the shared library is available or not and supports the option —without-rpki

  • The BGPReader accepts multiple new parameters which allows setting all RPKI options:

    • --rpki enables the historical RPKI validation with default collector (standalone)

    • --rpki-live enables the live RPKI validation with a default RTR-Server (standlone)

    • --rpki-collectors specifies the collectors used for the RPKI validation (standalone),
      Format: "((*|project):(*|(collector(,collectors)))(;)?)",
      e.g. "PJ_1:CC_1,CC_11;PJ_2:CC_2" or "PJ_1:*;PJ_2:CC_2" or "*:*"

    • --rpki-unified enables an unified validation where the ROA dumps of the different collectors are combined

    • --rpki-ssh enables SSH encryption to the RTR Server, format: "username, hostkey_path, privatekey_path"

  • On basis of the parameters the BGPReader creates the ROAFetchlib configuration which is needed for the validation process and stores it in every BGP elem instance

  • During the output process every BGP elem will be validated and the validation result is added to the regular BGPReader output

- The configure script checks whether the shared library is available or not and supports the option ```—without-rpki```

- The BGPReader accepts multiple new parameters which allows setting all RPKI options:
  * ```--rpki``` enables the historical RPKI validation with default collector (standalone)

  * ```--rpki-live``` enables the live RPKI validation with a default RTR-Server (standlone)

  * ```--rpki-collectors``` specifies the collectors used for the RPKI validation (standalone), <br /> Format: "((\*|project):(\*|(collector(,collectors)*))(;)?)*", <br /> e.g. "PJ_1:CC_1,CC_11;PJ_2:CC_2" or "PJ_1:*;PJ_2:CC_2" or "\*:\*"

  * ```--rpki-unified``` enables an unified validation where the ROA dumps of the different collectors are combined

  * ```--rpki-ssh``` enables SSH encryption to the RTR Server, format: "username, hostkey_path, privatekey_path"

- On basis of the parameters the BGPReader creates the ROAFetchlib configuration which is needed for the validation process and stores it in every BGP elem instance

- During the output process every BGP elem will be validated and the validation result is added to the regular BGPReader output
@salsh
Copy link
Contributor Author

salsh commented May 28, 2018

@alistairking @digizeph ping

1 similar comment
@salsh
Copy link
Contributor Author

salsh commented Jun 6, 2018

@alistairking @digizeph ping

@waehlisch
Copy link

Folks, @alistairking @digizeph, can we get some ETA? Would be really nice to know when and how to proceed. Thanks!

configure.ac Outdated
[AS_HELP_STRING([--without-rpki],
[do not compile with rpki support])],
[],
[with_rpki=yes])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

consider default no for rpki support. @alistairking what do you think?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Definitely.
I think we have two options:

  • if roafetchlib is installed, enable rpki by default
  • disable rpki unless explicitly requested

I'm ok with the first option iff the rpki code stays dormant unless explicitly requested at run time, which, i'm assuming is how things work at the moment.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay. Now RPKI support is disabled unless explicitly requested with with-rpki. I chose it because it's a bit more in line with the other options. I don't think the two approaches differ much in practice, because the ROAfetchlib is not included anyway.

@@ -107,6 +107,19 @@ typedef enum {

} bgpstream_elem_type_t;

typedef struct struct_bgpstream_annotations_t {

/** RPKI active */
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor issue: this annotation struct seems to be only about rpki annotation. if we want to keep it generic, we can keep the struct name, and enclose rpki-related fields in #ifdef block. @alistairking

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We already had the same problem the other way around in the last pull request.

99e682f - "2. we always include all the possible fields in the public structures. this is much better since the API will be stable no matter what compile time options are used."

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hah yeah, you're right. thanks for being more awake than me. this should stay as-is.

* Annotations from other libraries
*/
bgpstream_annotations_t annotations;

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

enclose the above with #ifdef ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no, let's go with your other suggestion above.

@@ -211,8 +225,10 @@ static void usage()
fprintf(stderr, " -%c, --%-23s%-15s %s\n", OPTIONS[k].option.val,
OPTIONS[k].option.name, OPTIONS[k].usage, expl_buf);
} else {
#ifdef WITH_RPKI
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@salsh why add this #ifdef?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, i'm confused by this too. and what are the 50x numbers in the argument config structs?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The long args should be visible in usage (help section) only if RPKI support is active. Assumption: all long args belong to RPKI.

if(isalpha(OPTIONS[k].option.val)) {
snprintf(short_options + size, sizeof(short_options) - size,
OPTIONS[k].option.has_arg ? "%c:": "%c", OPTIONS[k].option.val);
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@salsh missing a else here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The "else" part is empty because the non-alpha options (long args) are included in long_options in every iteration anyway (short args also have long args but the RPKI related long args do not have short args).

case 504:
bgpstream_rpki_parse_default(rpki_input);
break;
#endif
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@salsh is the naming of these arguments normal? -50x is hard to understand without actually checking the source code.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh, i see. i guess you (@salsh) wanted long args without short args, so the 50x are a way to ensure no normal argument collides with them.
i think this is fine, but let's create an enum at the top of bgpreader that gives these values a more friendly name.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh, i see. i guess you (@salsh) wanted long args without short args, so the 50x are a way to ensure no normal argument collides with them.

Exactly.

i think this is fine, but let's create an enum at the top of bgpreader that gives these values a more friendly name.

Alright, done!

@alistairking
Copy link
Member

@salsh i think we're pretty close to merging this. see above for various minor comments

@salsh
Copy link
Contributor Author

salsh commented Aug 6, 2018

@alistairking Sounds good.

I know it's a bit of an effort, but I would appreciate it if any of you could really execute it. Just to make sure everything is acceptable to you guys (perhaps some details that are not necessarily visible through a code review).

@digizeph
Copy link
Contributor

digizeph commented Aug 6, 2018

@salsh Thanks for the replies. I will start executing it later today. I'll get back to you real soon.

@digizeph
Copy link
Contributor

digizeph commented Aug 6, 2018

hey @salsh , i got the following error when run with --rpki option.

bgpreader -w 1533558000,1533558001 -c route-views.sfmix -t updates --rpki||less
Error: Could not read ROA file from broker

it then print out regular bgpreader outputs without rpki information.

i configured bgpstream with ./configure --without-transport-kafka --without-kafka --without-betabmp --with-rpki after installing librtr and ROAFetchlib.

@salsh
Copy link
Contributor Author

salsh commented Aug 6, 2018

@digizeph oh sorry, my mistake - I didn't think you would execute it so quickly. Currently only timestamps before 2018-07 are available because we are improving the broker infrastructure at the moment.
Archive: http://roa-broker.realmv6.org/FU-Berlin/
All collectors: http://roa-broker.realmv6.org/info?collector=FU-Berlin:*

@digizeph
Copy link
Contributor

digizeph commented Aug 6, 2018

Thanks @salsh ! I was able to get it working with an earlier timestamp.

bgpreader -w 1527811200,1527811201 -c route-views.sfmix -t updates --rpki||less
U|A|1527811200.000000|routeviews|route-views.sfmix|||32354|206.197.187.5|93.175.149.0/24|206.197.187.5|32354 3257 12779 12654|12654|||FU-Berlin,CC01,valid,12654,93.175.149.0/24-24;
U|A|1527811200.000000|routeviews|route-views.sfmix|||14061|2001:504:30::ba01:4061:1|2001:7fb:fe0a::/48|2001:504:30::ba01:4061:1|14061 3356 3257 12637 12654|12654|3257:3257 3356:2 3356:22 3356:86 3356:502 3356:601 3356:666 3356:2090 14061:2000 14061:2004 14061:4000 14061:4004||FU-Berlin,CC01,valid,12654,2001:7fb:fe0a::/48-48;
U|A|1527811200.000000|routeviews|route-views.sfmix|||14061|2001:504:30::ba01:4061:1|2001:7fb:fe07::/48|2001:504:30::ba01:4061:1|14061 1299 6453 9002 12654|12654|1299:4000 1299:25000 1299:25220 6453:2000 6453:2300 6453:2302 14061:2000 14061:2004 14061:4000 14061:4003||FU-Berlin,CC01,valid,12654,2001:7fb:fe07::/48-48;
U|A|1527811200.000000|routeviews|route-views.sfmix|||14061|2001:504:30::ba01:4061:1|2001:7fb:fe0a::/48|2001:504:30::ba01:4061:1|14061 1299 3257 12637 12654|12654|1299:4000 1299:25000 1299:25220 3257:3257 14061:2000 14061:2004 14061:4000 14061:4003||FU-Berlin,CC01,valid,12654,2001:7fb:fe0a::/48-48;
U|A|1527811200.000000|routeviews|route-views.sfmix|||14061|206.197.187.10|93.175.149.0/24|206.197.187.10|14061 3356 3257 12779 12654|12654|3257:3257 3356:2 3356:22 3356:86 3356:502 3356:666 3356:2090 14061:2000 14061:2004 14061:4000 14061:4004||FU-Berlin,CC01,valid,12654,93.175.149.0/24-24;
U|A|1527811200.000000|routeviews|route-views.sfmix|||14061|206.197.187.10|93.175.149.0/24|206.197.187.10|14061 3356 8784 34019 12654|12654|3356:2 3356:22 3356:100 3356:123 3356:502 3356:2066 8784:502 8784:1002 14061:2000 14061:2004 14061:4000 14061:4004||FU-Berlin,CC01,valid,12654,93.175.149.0/24-24;

The output from bgpstream built without rpki support looks like below:

bgpreader -w 1527811200,1527811201 -c route-views.sfmix -t updates
U|A|1527811200|routeviews|route-views.sfmix|32354|206.197.187.5|93.175.149.0/24|206.197.187.5|32354 3257 12779 12654|12654|||
U|A|1527811200|routeviews|route-views.sfmix|14061|2001:504:30::ba01:4061:1|2001:7fb:fe0a::/48|2001:504:30::ba01:4061:1|14061 3356 3257 12637 12654|12654|3257:3257 3356:2 3356:22 3356:86 3356:502 3356:601 3356:666 3356:2090 14061:2000 14061:2004 14061:4000 14061:4004||
U|A|1527811200|routeviews|route-views.sfmix|14061|2001:504:30::ba01:4061:1|2001:7fb:fe07::/48|2001:504:30::ba01:4061:1|14061 1299 6453 9002 12654|12654|1299:4000 1299:25000 1299:25220 6453:2000 6453:2300 6453:2302 14061:2000 14061:2004 14061:4000 14061:4003||
U|A|1527811200|routeviews|route-views.sfmix|14061|2001:504:30::ba01:4061:1|2001:7fb:fe0a::/48|2001:504:30::ba01:4061:1|14061 1299 3257 12637 12654|12654|1299:4000 1299:25000 1299:25220 3257:3257 14061:2000 14061:2004 14061:4000 14061:4003||
U|A|1527811200|routeviews|route-views.sfmix|14061|206.197.187.10|93.175.149.0/24|206.197.187.10|14061 3356 3257 12779 12654|12654|3257:3257 3356:2 3356:22 3356:86 3356:502 3356:666 3356:2090 14061:2000 14061:2004 14061:4000 14061:4004||
U|A|1527811200|routeviews|route-views.sfmix|14061|206.197.187.10|93.175.149.0/24|206.197.187.10|14061 3356 8784 34019 12654|12654|3356:2 3356:22 3356:100 3356:123 3356:502 3356:2066 8784:502 8784:1002 14061:2000 14061:2004 14061:4000 14061:4004||

Overall, the output looks good.

@digizeph
Copy link
Contributor

digizeph commented Aug 6, 2018

Running with --rpki-live option gives out not-found results, which looks likely to be caused by not being to retrieve "current" rpki records. Is this the correct behavior, @salsh ? I am not too concerned since it only affects the value of the last output element.

bgpreader -w 1527811200,1527811201 -c route-views.sfmix -t updates --rpki-live
U|A|1527811200.000000|routeviews|route-views.sfmix|||32354|206.197.187.5|93.175.149.0/24|206.197.187.5|32354 3257 12779 12654|12654|||FU-Berlin,CC06(RTR),notfound;
U|A|1527811200.000000|routeviews|route-views.sfmix|||14061|2001:504:30::ba01:4061:1|2001:7fb:fe0a::/48|2001:504:30::ba01:4061:1|14061 3356 3257 12637 12654|12654|3257:3257 3356:2 3356:22 3356:86 3356:502 3356:601 3356:666 3356:2090 14061:2000 14061:2004 14061:4000 14061:4004||FU-Berlin,CC06(RTR),notfound;
U|A|1527811200.000000|routeviews|route-views.sfmix|||14061|2001:504:30::ba01:4061:1|2001:7fb:fe07::/48|2001:504:30::ba01:4061:1|14061 1299 6453 9002 12654|12654|1299:4000 1299:25000 1299:25220 6453:2000 6453:2300 6453:2302 14061:2000 14061:2004 14061:4000 14061:4003||FU-Berlin,CC06(RTR),notfound;
U|A|1527811200.000000|routeviews|route-views.sfmix|||14061|2001:504:30::ba01:4061:1|2001:7fb:fe0a::/48|2001:504:30::ba01:4061:1|14061 1299 3257 12637 12654|12654|1299:4000 1299:25000 1299:25220 3257:3257 14061:2000 14061:2004 14061:4000 14061:4003||FU-Berlin,CC06(RTR),notfound;
U|A|1527811200.000000|routeviews|route-views.sfmix|||14061|206.197.187.10|93.175.149.0/24|206.197.187.10|14061 3356 3257 12779 12654|12654|3257:3257 3356:2 3356:22 3356:86 3356:502 3356:666 3356:2090 14061:2000 14061:2004 14061:4000 14061:4004||FU-Berlin,CC06(RTR),notfound;
U|A|1527811200.000000|routeviews|route-views.sfmix|||14061|206.197.187.10|93.175.149.0/24|206.197.187.10|14061 3356 8784 34019 12654|12654|3356:2 3356:22 3356:100 3356:123 3356:502 3356:2066 8784:502 8784:1002 14061:2000 14061:2004 14061:4000 14061:4004||FU-Berlin,CC06(RTR),notfound;

@digizeph
Copy link
Contributor

digizeph commented Aug 6, 2018

@salsh --rpki-collectors seems working just fine, but have problem together with --rpki-live:

bgpreader -w 1527811200,1527811201 -c route-views.sfmix -t updates --rpki --rpki-collectors FU-Berlin:CC01 --rpki-live
Error: Collector not allowed: FU-Berlin:CC01 (only RTR-Server)

I assume this can be handled with better documentation.

@digizeph
Copy link
Contributor

digizeph commented Aug 6, 2018

@salsh --rpki-unified produces a slightly different output than without the option. Specifically, it replaces the first , in rpki section with \ in my test case. I am not sure if that's the desired output.

With --rpki-unified:

U|A|1527811200.000000|routeviews|route-views.sfmix|||32354|206.197.187.5|93.175.149.0/24|206.197.187.5|32354 3257 12779 12654|12654|||FU-Berlin\CC01,valid,12654,93.175.149.0/24-24;

Without --rpki-unified:

U|A|1527811200.000000|routeviews|route-views.sfmix|||32354|206.197.187.5|93.175.149.0/24|206.197.187.5|32354 3257 12779 12654|12654|||FU-Berlin,CC01,valid,12654,93.175.149.0/24-24;

@salsh
Copy link
Contributor Author

salsh commented Aug 7, 2018

@digizeph Thanks for executing.

Running with --rpki-live option gives out not-found results, which looks likely to be caused by not being to retrieve "current" rpki records. Is this the correct behavior, @salsh ? I am not too concerned since it only affects the value of the last output element.

--rpki and --rpki-live are both "standalone" default options for historical and live validation. The "notfound" results were caused because the prefixes of the BGP Announcements were not found in the ROA dump. The validation status is "notfound". It is the correct behavior.

All validation related outputs only affect the last element. The bgpstream output always remains unaffected.

@salsh
Copy link
Contributor Author

salsh commented Aug 7, 2018

@salsh --rpki-collectors seems working just fine, but have problem together with --rpki-live. I assume this can be handled with better documentation.

As pointed out in the last comment --rpki-live is a standalone option (a shortcut for --rpki-collectors FU-Berlin:CC01(RTR)). Maybe the help section should clarify it a bit more. Thanks for the hint. Possible input:

  1. --rpki for default historical validation.
  2. --rpki-live for default live validation.
  3. --rpki-collectors and --rpki-ssh if the connection to the RTR-Server should be encrypted.
  4. --rpki-collectors and --rpki-unified if multiple collectors should be validated unified.

Actually the error message should be clear: For live validation only RTR-Servers (all collectors with "RTR") are allowed e.g. bgpreader -w 1527811200,1527811201 -c route-views.sfmix -t updates --rpki --rpki-collectors FU-Berlin:CC01(RTR)

@salsh
Copy link
Contributor Author

salsh commented Aug 7, 2018

@salsh --rpki-unified produces a slightly different output than without the option. Specifically, it replaces the first , in rpki section with \ in my test case. I am not sure if that's the desired output.

Yes the actual difference is only visible if multiple collectors were defined e.g.:
bgpreader -w 1527811200,1527811201 -c route-views.sfmix -t updates --rpki-collectors "FU-Berlin:*" --rpki-unified

U|A|1527811200.000000|routeviews|route-views.sfmix|||14061|2001:504:30::ba01:4061:1|2001:7fb:fe0a::/48|2001:504:30::ba01:4061:1|14061 3356 3257 12637 12654|12654|3257:3257 3356:2 3356:22 3356:86 3356:502 3356:601 3356:666 3356:2090 14061:2000 14061:2004 14061:4000 14061:4004||FU-Berlin\CC01 FU-Berlin\CC01(RTR) FU-Berlin\CC02 FU-Berlin\CC02(RTR) FU-Berlin\CC03 FU-Berlin\CC03(RTR) FU-Berlin\CC04 FU-Berlin\CC04(RTR) FU-Berlin\CC05 FU-Berlin\CC06 FU-Berlin\CC06(RTR),valid,12654,2001:7fb:fe0a::/48-48;

The different delimiters are for easier parsing and to avoid redundancy.

@salsh
Copy link
Contributor Author

salsh commented Aug 7, 2018

     --rpki                                    validate the BGP records with historical RPKI dumps (standalone)
     --rpki-live                               validate the BGP records with current RPKI dump (standalone)
     --rpki-collectors        <((*|project):(*|(collector(,collectors)*))(;)?)*>  
                                               specify collectors used for historical or live RPKI validation 
     --rpki-unified                            whether the RPKI validation for different collectors is unified
     --rpki-ssh               <user,hostkey,private key>  
                                               enable SSH encryption for the live connection to the RTR server

Is the help section more understandable now?

@digizeph
Copy link
Contributor

digizeph commented Aug 7, 2018

@salsh thanks for the responses. They all make sense to me. I think the current usage text can stay.

@digizeph digizeph merged commit 2018e30 into CAIDA:master Aug 7, 2018
@digizeph
Copy link
Contributor

digizeph commented Aug 7, 2018

@salsh We have merged your PR into master. Thank you for your contribution, Samir!

@salsh
Copy link
Contributor Author

salsh commented Aug 7, 2018

Glad to hear it.

@alistairking @digizeph - Thank you very much for merging the PR, your time and effort. I appreciate it very much.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants