BGPd crashed on 2 routers while running Topotest "test_bgp_l3vpn_to_bgp_vrf.py"

[mdash-vmware]

BGPd crashed on 2 routers while running Topotest "test_bgp_l3vpn_to_bgp_vrf.py"

I just upgraded my Ubuntu kernel from 4.4 (4.4.0-87-generic) to 4.9 (4.9.0-040900-generic) and when I ran "test_bgp_l3vpn_to_bgp_vrf.py" from Topotest, I can see BGPd crashed and core generated on 2 of routers while cleaning up the test.

FRR Version :

Hello, this is FRRouting (version 5.0.1).
Copyright 1996-2005 Kunihiro Ishiguro, et al.

here are the Back Trace.

r4: bgpd crashed. Core file found - Backtrace follows:
[New LWP 29595]
[New LWP 29599]
[New LWP 29600]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/lib/frr/bgpd'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007fc04182c428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
[Current thread is 1 (Thread 0x7fc04291e880 (LWP 29595))]
#0  0x00007fc04182c428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
#1  0x00007fc04182e02a in __GI_abort () at abort.c:89
#2  0x00007fc04254bc8a in core_handler (signo=11, siginfo=0x7fff6763ff70, context=0x7fff6763fe40) at lib/sigevent.c:249
#3  <signal handler called>
#4  0x00005646811585d6 in sendmsg_zebra_rnh (bnc=0x5646853e5840, command=24) at bgp_nht.c:570
#5  0x00005646811587a1 in unregister_zebra_rnh (is_bgp_import_route=<optimized out>, bnc=0x5646853e5840) at bgp_nht.c:641
#6  bgp_unlink_nexthop_check (bnc=0x5646853e5840) at bgp_nht.c:97
#7  0x00005646811588d0 in bgp_unlink_nexthop_check (bnc=<optimized out>) at bgp_nht.c:116
#8  bgp_unlink_nexthop (path=path@entry=0x5646853e6d20) at bgp_nht.c:115
#9  0x0000564681109f00 in bgp_info_free (binfo=0x5646853e6d20) at bgp_route.c:211
#10 bgp_info_unlock (binfo=0x5646853e6d20) at bgp_route.c:236
#11 0x000056468110a009 in bgp_info_reap (rn=rn@entry=0x5646853e6c50, ri=0x5646853e6d20) at bgp_route.c:280
#12 0x000056468110a0e6 in bgp_clear_route_table (peer=peer@entry=0x564682c8bd50, afi=afi@entry=AFI_IP, safi=safi@entry=SAFI_UNICAST, table=<optimized out>, table@entry=0x0) at bgp_route.c:3879
#13 0x000056468110f2fb in bgp_clear_route (peer=peer@entry=0x564682c8bd50, afi=afi@entry=AFI_IP, safi=safi@entry=SAFI_UNICAST) at bgp_route.c:3927
#14 0x000056468110f4a0 in bgp_clear_route_all (peer=peer@entry=0x564682c8bd50) at bgp_route.c:3945
#15 0x00005646810fc888 in bgp_fsm_change_status (peer=peer@entry=0x564682c8bd50, status=status@entry=8) at bgp_fsm.c:924
#16 0x00005646810ee1bd in peer_delete (peer=0x564682c8bd50) at bgpd.c:2224
#17 0x00005646810f2f49 in bgp_delete (bgp=0x564682c8aad0) at bgpd.c:3314
#18 0x00005646810e6708 in bgp_exit (status=0) at bgp_main.c:185
#19 sigint () at bgp_main.c:151
#20 0x00007fc04254bd28 in quagga_sigevent_process () at lib/sigevent.c:103
#21 0x00007fc04255822d in thread_fetch (m=m@entry=0x56468254f630, fetch=fetch@entry=0x7fff676407f0) at lib/thread.c:1359
#22 0x00007fc0425340f3 in frr_run (master=0x56468254f630) at lib/libfrr.c:878
#23 0x00005646810e44be in main (argc=1, argv=0x7fff67640a48) at bgp_main.c:425

r3: bgpd crashed. Core file found - Backtrace follows:
[New LWP 29785]
[New LWP 29795]
[New LWP 29796]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/lib/frr/bgpd'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007f0d1502f428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
[Current thread is 1 (Thread 0x7f0d16121880 (LWP 29785))]
#0  0x00007f0d1502f428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
#1  0x00007f0d1503102a in __GI_abort () at abort.c:89
#2  0x00007f0d15d4ec8a in core_handler (signo=11, siginfo=0x7ffc44bf3a70, context=0x7ffc44bf3940) at lib/sigevent.c:249
#3  <signal handler called>
#4  0x000055d49df715d6 in sendmsg_zebra_rnh (bnc=0x55d4a2aac420, command=24) at bgp_nht.c:570
#5  0x000055d49df717a1 in unregister_zebra_rnh (is_bgp_import_route=<optimized out>, bnc=0x55d4a2aac420) at bgp_nht.c:641
#6  bgp_unlink_nexthop_check (bnc=0x55d4a2aac420) at bgp_nht.c:97
#7  0x000055d49df718d0 in bgp_unlink_nexthop_check (bnc=<optimized out>) at bgp_nht.c:116
#8  bgp_unlink_nexthop (path=path@entry=0x55d4a2ac1550) at bgp_nht.c:115
#9  0x000055d49df22f00 in bgp_info_free (binfo=0x55d4a2ac1550) at bgp_route.c:211
#10 bgp_info_unlock (binfo=0x55d4a2ac1550) at bgp_route.c:236
#11 0x000055d49df23009 in bgp_info_reap (rn=rn@entry=0x55d4a2ac1480, ri=0x55d4a2ac1550) at bgp_route.c:280
#12 0x000055d49df230e6 in bgp_clear_route_table (peer=peer@entry=0x55d4a0350670, afi=afi@entry=AFI_IP, safi=safi@entry=SAFI_UNICAST, table=<optimized out>, table@entry=0x0) at bgp_route.c:3879
#13 0x000055d49df282fb in bgp_clear_route (peer=peer@entry=0x55d4a0350670, afi=afi@entry=AFI_IP, safi=safi@entry=SAFI_UNICAST) at bgp_route.c:3927
#14 0x000055d49df284a0 in bgp_clear_route_all (peer=peer@entry=0x55d4a0350670) at bgp_route.c:3945
#15 0x000055d49df15888 in bgp_fsm_change_status (peer=peer@entry=0x55d4a0350670, status=status@entry=8) at bgp_fsm.c:924
#16 0x000055d49df071bd in peer_delete (peer=0x55d4a0350670) at bgpd.c:2224
#17 0x000055d49df0bf49 in bgp_delete (bgp=0x55d4a034f3f0) at bgpd.c:3314
#18 0x000055d49deff708 in bgp_exit (status=0) at bgp_main.c:185
#19 sigint () at bgp_main.c:151
#20 0x00007f0d15d4ed28 in quagga_sigevent_process () at lib/sigevent.c:103
#21 0x00007f0d15d5b22d in thread_fetch (m=m@entry=0x55d49fc6d630, fetch=fetch@entry=0x7ffc44bf42e0) at lib/thread.c:1359
#22 0x00007f0d15d370f3 in frr_run (master=0x55d49fc6d630) at lib/libfrr.c:878
#23 0x000055d49defd4be in main (argc=1, argv=0x7ffc44bf4538) at bgp_main.c:425

[mdash-vmware]

Attaching the full Topotest Run log.

BGPd_Crash_Topotest_bgp_l3vpn.txt

[qlyoung]

@mdash-vmware can you please apply the patch in #3195 and check if it fixes the issue? if so then we need to backport it to 5.X

[qlyoung]

Closing due to lack of activity