gnrc: nodes crashing with too small packet buffer #5748

OlegHahm · 2016-08-15T10:46:51Z

I just ran into a problem, when I manually configured the packet buffer too small in the gnrc_networking example. In fact, I copied the size from the gnrc_minimal example (512 bytes). Testing this on the iotlab testbed with ~350 nodes (Neighbor cache size set to 100, FIB size to 350) I ran into several crashes. Nodes were dying with hard faults after some time.

(On the pro side: with the default packet buffer size, I was able to transmit a UDP datagram from the one end of the testbed to the other without problems. :))

The text was updated successfully, but these errors were encountered:

miri64 · 2016-08-15T15:03:11Z

I think I have the same problem at the moment. Will look into that.

miri64 · 2016-08-15T15:06:39Z

@OlegHahm are you on native or other platform? (native resets GNRC_PKTBUF_SIZE to 2048)

OlegHahm · 2016-08-15T15:11:35Z

I tested only on iotlab.

miri64 · 2016-08-15T15:38:42Z

My problem was unrelated (see #5749), but nevertheless: will look into this ASAP.

OlegHahm · 2016-08-16T22:47:37Z

I figured that you will probably get more debug information if you revert #5665.

miri64 · 2016-08-30T11:39:28Z

@OlegHahm its really just a hunch, but can you check if #5694 fixes the problem?

OlegHahm · 2016-10-30T22:01:14Z

Just tested on master with same settings and 220 nodes on Lille site: nodes are still crashing.

kYc0o · 2016-11-09T00:18:13Z

Maybe this would be solved by ~~#6083~~ #6086 [edit: @miri64] ?

kYc0o · 2016-11-09T00:19:13Z

Ups sorry, didn't refresh the page and this was already referenced.

RIOT-2016.10 - Release Notes ============================ RIOT is a real-time multi-threading operating system that supports a range of devices that are typically found in the Internet of Things: 8-bit microcontrollers, 16-bit microcontrollers and light-weight 32-bit processors. RIOT is based on the following design principles: energy-efficiency, real-time capabilities, small memory footprint, modularity, and uniform API access, independent of the underlying hardware (this API offers partial POSIX compliance). RIOT is developed by an international open source community which is independent of specific vendors (e.g. similarly to the Linux community) and is licensed with a non-viral copyleft license (LGPLv2.1), which allows indirect business models around the free open-source software platform provided by RIOT. About this release: =================== This release provides a lot of new features as well as it fixes several major bugs. Among these new features are the new simplified network socket API called sock, the GNRC specific CoAP implementation gcoap and several new packages: TinyDTLS, the Aversive++ microcontroller library for robotics, the u8g2 graphic library, and nanocoap. Using the new sock API an implementation of the Simple Time Network Protocol (SNTP) was also introduced, allowing for time synchronization between nodes. New platforms include the Arduino Uno, the Arduino Duemilanove, the Arduino Zero, SODAQ Autonomo, and the Zolertia remote (rev. B). The most significant bug fix was done in native which led to a significantly more robust handling of ISRs and now allows for at least 1,000 native instances running stably on one machine. About 263 pull requests with about 398 commits have been merged since the last release and about 42 issues have been solved. 37 people contributed with code in 100 days. 1006 files have been touched with 166500 insertions and 26926 deletions. Notations used below: ===================== + means new feature/item * means modified feature/item - means removed feature/item New features and changes ======================== General ------- * Verbose behavior for assert() macro Core ---- + MPU support for Cortex-M API changes ----------- + Socket-like sock API (replacing conn) * netdev2: Add Testmodes and CCA modes * IEEE 802.15.4: clean-up Intra-PAN behavior * IEEE 802.15.4: centralize default values * gnrc_pktbuf: allow for 0-sized snips + gnrc_netapi: mbox and arbitrary callback support System libraries ---------------- No new features or changes Networking ---------- + Provide sock-port for GNRC + gcoap: a GNRC-based CoAP implementation + Simple Network Time Protocol (RFC 5905, section 14) + Priority Queue for packet snips + IPv4 header definitions Packages -------- + nanocoap: CoAP header parser/builder + TinyDTLS: DTLS library + tiny-asn1: asn.1/der decoder + Aversive++ microcontroller programming library + u8g2 graphic library Platforms --------- + Support for stm32f2xx MCU family + Low power modes for samd21 CPUs + More Arduino-based platforms: + Arduino Uno + Arduino Duemilanove + Arduino Zero + More boards of ST's Nucleo platforms: + ST Nucleo F030 board support + ST Nucleo F070 board support + ST Nucleo F446 board support + SODAQ Automono + Zolertia remote rev. B Drivers ------- + W5100 Ethernet device + Atmel IO1 Xplained extension + LPD8808 LED strips * at86rf2xx: provide capability to access the RND_VALUE random value register Build System ------------ + static-tests build target for easy local execution of CI's static tests Other ----- + Provide Arduino API to Nucleo boards + Packer configuration file to build vagrant boxes + CC2650STK Debugger Support + ethos: add Ethos over TCP support Fixed Issues from the last release ================================== #534: native debugging on osx fails #2071: native: *long* overdue fixes #3341: netdev2_tap crashes when hammered #5007: gnrc icmpv6: Ping reply goes out the wrong interface #5432: native: valgrind fails Known Issues ============ Networking related issues ------------------------- #3075: nhdp: unnecessary microsecond precision: NHDP works with timer values of microsecond precision which is not required. Changing to lower precision would save some memory. #4048: potential racey memory leak: According to the packet buffer stats, flood-pinging a multicast destination may lead to a memory leak due to a race condition. However, it seems to be a rare case and a completely filled up packet buffer was not observed. #4388: POSIX sockets: open socket is bound to a specific thread: This was an inherit problem of the conn API under GNRC. Since the POSIX sockets are still based on conn for this release, this issue persists #4527: gnrc_ipv6: Multicast is not forwarded if routing node listens to the address (might still be fixable for release, see #5729, #5230: gnrc ipv6: multicast packets are not dispatched to the upper layers) #5016: gnrc_rpl: Rejoining RPL instance as root after reboot messes up routing #5055: cpuid: multiple radios will get same EUI-64 Nodes with multiple interfaces might get the same EUI-64 for them since they are generated from the same CPU ID. #5656: Possible Weakness with locking in the GNRC network stack: For some operations mutexes to the network interfaces need to get unlocked in the current implementation to not get deadlocked. Recursive mutexes as provided in #5731 might help to solve this problem. #5748: gnrc: nodes crashing with too small packet buffer: A packet buffer of size ~512 B might lead to crashes. The issue describes this for several hundret nodes, but agressive flooding with just two nodes was also shown to lead to this problem. #5858: gnrc: 6lo: potential problem with reassembly of fragments: If one frame gets lost the reassembly state machine might get out of sync ### NDP is not working properly #4499: handle of l2src_len in gnrc_ndp_rtr_sol_handle: Reception of a router solicitation might lead to invalid zero-length link-layer addresses in neighbor cache. #5005: ndp: router advertisement sent with global address: Under some circumstances a router might send RAs with GUAs. While they are ignored on receive (as RFC 4861 specifies), RAs should have link-local addresses and not even be send out this way. #5122: NDP: global unicast address on non-6LBR nodes disappears after a while: Several issues (also see #5760) lead to a global unicast address effectively being banned from the network (disappears from neighbor cache, is not added again) #5467: ipv6 address vanishes when ARO (wrongly) indicates DUP caused by outdated ncache at router #5539: Border Router: packet not forwarded from ethos to interface 6 #5790: ND: Lost of Global IPV6 on node after sending lot of UDP frame from BR Timer related issues -------------------- #4841: xtimer: timer already in the list: Under some conditions an xtimer can end up twice in the internal list of the xtimer module #4902: xtimer: xtimer_set: xtimer_set does not handle integer overflows well #5338: xtimer: xtimer_now() not ISR safe for non-32-bit platforms. #5928: xtimer: usage in board_init() crashes: some boards use the xtimer in there board_init() function. The xtimer is however first initialized in the auto_init module which is executed after board_init() #6052: tests: xtimer_drift gets stuck: xtimer_drift application freezes after ~30-200 seconds native related issues --------------------- #495: native not float safe: When the FPU is used when an asynchronous context switch occurs, either the stack gets corrupted or a floating point exception occurs. #2175: ubjson: valgind registers "Invalid write of size 4" in unittests #4590: pkg: building relic with clang fails. #5796: native: tlsf: early malloc will lead to a crash: TLSF needs pools to be initialized (which is currently expected to be done in an application). If a malloc is needed before an application's main started (e.g. driver initialization) the node can crash, since no pool is allocated yet. other platform related issues ----------------------------- #1891: newlib-nano: Printf formatting does not work properly for some numberic types: PRI[uxdi]64, PRI[uxdi]8 and float are not parsed in newlib-nano #2006: cpu/nrf51822: timer callback may be fired too early #2143: unittests: tests-core doesn't compile for all platforms: GCC build-ins were used in the unittests which are not available with msp430-gcc #2300: qemu unittest fails because of a page fault #4512: pkg: tests: RELIC unittests fail on iotlab-m3 #4522: avsextrem: linker sometimes doesn't find `bl_init_clks()` #4560: make: clang is more pedantic than gcc oonf_api is not building with clang. (Partly solved by #4593) #4694: drivers/lm75a: does not build #4737: cortex-m: Hard fault after a thread exits (under some circumstances) #4822: kw2xrf: packet loss when packets get fragmented #4876: at86rf2xx: Simultaneous use of different transceiver types is not supported #4954: chronos: compiling with -O0 breaks #4866: not all GPIO driver implementations are thread safe: Due to non-atomic operations in the drivers some pin configurations might get lost. #5009: RIOT is saw-toothing in energy consumption (even when idling) #5103: xtimer: weird behavior of tests/xtimer_drift: xtimer_drift randomly jumps a few seconds on nrf52 #5361: cpu/cc26x0: timer broken #5405: Eratic timings on iotlab-m3 with compression context activated #5460: cpu/samd21: i2c timing with compiler optimization #5486: at86rf2xx: lost interrupts #5489: cpu/lpc11u34: ADC broken #5603: atmega boards second UART issue #5678: at86rf2xx: failed assertion in _isr #5719: cc2538: rf driver doesn't handle large packets #5799: kw2x: 15.4 duplicate transmits #5944: msp430: ipv6_hdr unittests fail #5848: arduino: Race condition in sys/arduino/Makefile.include #5954: nRF52 uart_write get stuck #6018: nRF52 gnrc 6lowpan ble memory leak other issues ------------ #1263: TLSF implementation contains (a) read-before-write error(s). #3256: make: Setting constants on compile time doesn't really set them everywhere #3366: periph/i2c: handle NACK #4488: Making the newlib thread-safe: When calling puts/printf after thread_create(), the CPU hangs for DMA enabled uart drivers. #4866: periph: GPIO drivers are not thread safe #5128: make: buildtest breaks when exporting FEATURES_PROVIDED var #5207: make: buildest fails with board dependent application Makefiles #5390: pkg: OpenWSN does not compile: This package still uses deprecated modules and was not tested for a long time. #5520: tests/periph_uart not working #5561: C++11 extensions in header files #5776: make: Predefining CFLAGS are parsed weirdly #5863: OSX + SAMR21-xpro: shell cannot handle command inputs larger than 64 chars #5962: Makefile: UNDEF variable is not working as documented #6022: pkg: build order issue Special Thanks ============== We like to give our special thanks to all the companies that provided us with their hardware for porting and testing, namely the people from (in alphabeticalorder): Atmel, Freescale, Imagination Technologies, Limifrog, Nordic, OpenMote, Phytec, SiLabs, UDOO,and Zolertia; and also companies that directly sponsored development time: Cisco Systems, Eistec, Ell-i, Enigeering Spirit, Nordic, FreshTemp LLC, OTAkeys and Phytec. More information ================ http://www.riot-os.org Mailing lists ------------- * RIOT OS kernel developers list devel@riot-os.org (http://lists.riot-os.org/mailman/listinfo/devel) * RIOT OS users list users@riot-os.org (http://lists.riot-os.org/mailman/listinfo/users) * RIOT commits commits@riot-os.org (http://lists.riot-os.org/mailman/listinfo/commits) * Github notifications notifications@riot-os.org (http://lists.riot-os.org/mailman/listinfo/notifications) IRC --- * Join the RIOT IRC channel at: irc.freenode.net, #riot-os License ======= * Most of the code developed by the RIOT community is licensed under the GNU Lesser General Public License (LGPL) version 2.1 as published by the Free Software Foundation. * Some external sources are published under a separate, LGPL compatible license (e.g. some files developed by SICS). All code files contain licensing information.

One hug more; one more hug. One. Hug. More!!!1! https://www.youtube.com/watch?v=IddP8AAIGTQ

gnrc_pktbuf_static: fix #5748

miri64 · 2016-12-16T06:22:07Z

#6086 sadly did not fix this

kYc0o · 2017-04-26T10:13:59Z

Does anyone know if this is still an issue?

kYc0o · 2018-07-20T16:17:40Z

So far, I only found issues with the kw2x driver. With at86rf2xx everything works like a charm. Thus, I conclude there's no problem with gnrc.

Ok to close?

miri64 · 2018-07-20T19:20:13Z

Might be that #9434 fixed this issue as well.

miri64 · 2018-07-20T19:26:12Z

With at86rf2xx everything works like a charm. Thus, I conclude there's no problem with gnrc.

Ok to close?

With how many nodes did you test? How big did you configure the packet buffer to be? How long did your tests run. Though it is not described in the original issue, I remember that this only showed up after a while.

kYc0o · 2018-08-14T17:26:23Z

I'll remake the tests to configure the cases you mention and give a more complete feedback.

miri64 · 2019-01-29T20:20:57Z

@cgundogan can you maybe retest this for the release?

obgm · 2019-03-19T20:24:13Z

Not sure if this is related: Testing via tap on native, after some successful UDP transmissions, _gnrc_netif_thread always gets stuck. DEBUG says:

gnrc_netif: waiting for incoming messages
gnrc_netif: GNRC_NETDEV_MSG_TYPE_EVENT received
gnrc_netif: event triggered -> 2
gnrc_netif_ethernet: cannot allocate pktsnip.
gnrc_netif: waiting for incoming messages
gnrc_netif: GNRC_NETDEV_MSG_TYPE_EVENT received
gnrc_netif: event triggered -> 2
gnrc_netif_ethernet: cannot allocate pktsnip.

Increasing GNRC_PKTBUF_SIZE increases the number of successful transmissions but sooner or later, the same error will occur.

miri64 · 2019-05-23T12:39:35Z

Not sure if this is related: Testing via tap on native, after some successful UDP transmissions, _gnrc_netif_thread always gets stuck. DEBUG says:
gnrc_netif: waiting for incoming messages
gnrc_netif: GNRC_NETDEV_MSG_TYPE_EVENT received
gnrc_netif: event triggered -> 2
gnrc_netif_ethernet: cannot allocate pktsnip.
gnrc_netif: waiting for incoming messages
gnrc_netif: GNRC_NETDEV_MSG_TYPE_EVENT received
gnrc_netif: event triggered -> 2
gnrc_netif_ethernet: cannot allocate pktsnip.
Increasing GNRC_PKTBUF_SIZE increases the number of successful transmissions but sooner or later, the same error will occur.

This hints at a leak in the packet buffer. So it might be interesting which modules you are including and what your scenario it is so we can reproduce it. I don't see the relation to a smaller packet buffer however.

miri64 · 2019-09-24T14:08:06Z

Ping @obgm?

obgm · 2019-09-24T14:27:08Z

Sorry for the delay: I wanted to re-test with my code and up-to-date fork. I will hopefully be able to provide recent data within the next days.

miri64 · 2019-09-24T14:31:27Z

I'm swamped in work anyways, so I won't be able to fix this immediately, but it's good to keep the issue rolling ;-).

miri64 · 2020-07-08T09:13:15Z

I was trying to reproduce this issue but I was not able to. I'm not even getting leaks in the packet buffer. Here is an detailed account of what I did:

https://gist.github.com/miri64/4b88032f857a444bb39bdf0054f6e059

OlegHahm added Type: bug The issue reports a bug / The PR fixes a bug (including spelling errors) Area: network Area: Networking GNRC labels Aug 15, 2016

OlegHahm added this to the Release 2016.10 milestone Aug 15, 2016

OlegHahm assigned miri64 Aug 15, 2016

This was referenced Nov 8, 2016

tests: resize packet buffer for gnrc_sock_udp test #6084

Merged

02-tests/task 02 Release 2016.10 - RC3 RIOT-OS/Release-Specs#26

Closed

miri64 added a commit to miri64/RIOT that referenced this issue Nov 8, 2016

gnrc_pktbuf_static: fix RIOT-OS#5748

6d820d1

miri64 mentioned this issue Nov 8, 2016

gnrc_pktbuf_static: fix #5748 #6086

Merged

miri64 modified the milestones: Release 2016.10, Release 2017.01 Nov 11, 2016

miri64 mentioned this issue Nov 15, 2016

gnrc: crash with (excessive) traffic in native #6123

Closed

miri64 added a commit to miri64/RIOT that referenced this issue Dec 14, 2016

fixup! gnrc_pktbuf_static: fix RIOT-OS#5748

d46c603

One hug more; one more hug. One. Hug. More!!!1! https://www.youtube.com/watch?v=IddP8AAIGTQ

miri64 closed this as completed in 5e236cd Dec 16, 2016

miri64 added a commit that referenced this issue Dec 16, 2016

Merge pull request #6086 from miri64/gnrc_pktbuf_static/fix/i5748

490b907

gnrc_pktbuf_static: fix #5748

PeterKietzmann modified the milestones: Release 2017.01, Release 2017.04 Jan 26, 2017

aabadie modified the milestones: Release 2017.04, Release 2017.07 Jun 21, 2017

aabadie modified the milestones: Release 2017.07, Release 2017.10 Jul 14, 2017

miri64 removed the GNRC label Oct 5, 2018

miri64 added this to the Release 2020.07 milestone Jun 30, 2020

miri64 closed this as completed Jul 8, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gnrc: nodes crashing with too small packet buffer #5748

gnrc: nodes crashing with too small packet buffer #5748

OlegHahm commented Aug 15, 2016

miri64 commented Aug 15, 2016

miri64 commented Aug 15, 2016

OlegHahm commented Aug 15, 2016

miri64 commented Aug 15, 2016

OlegHahm commented Aug 16, 2016

miri64 commented Aug 30, 2016

OlegHahm commented Oct 30, 2016

kYc0o commented Nov 9, 2016 •

edited by miri64

Loading

kYc0o commented Nov 9, 2016

miri64 commented Dec 16, 2016

kYc0o commented Apr 26, 2017

kYc0o commented Jul 20, 2018

miri64 commented Jul 20, 2018

miri64 commented Jul 20, 2018

kYc0o commented Aug 14, 2018

miri64 commented Jan 29, 2019

obgm commented Mar 19, 2019

miri64 commented May 23, 2019

miri64 commented Sep 24, 2019

obgm commented Sep 24, 2019

miri64 commented Sep 24, 2019

miri64 commented Jul 8, 2020 •

edited

Loading

gnrc: nodes crashing with too small packet buffer #5748

gnrc: nodes crashing with too small packet buffer #5748

Comments

OlegHahm commented Aug 15, 2016

miri64 commented Aug 15, 2016

miri64 commented Aug 15, 2016

OlegHahm commented Aug 15, 2016

miri64 commented Aug 15, 2016

OlegHahm commented Aug 16, 2016

miri64 commented Aug 30, 2016

OlegHahm commented Oct 30, 2016

kYc0o commented Nov 9, 2016 • edited by miri64 Loading

kYc0o commented Nov 9, 2016

miri64 commented Dec 16, 2016

kYc0o commented Apr 26, 2017

kYc0o commented Jul 20, 2018

miri64 commented Jul 20, 2018

miri64 commented Jul 20, 2018

kYc0o commented Aug 14, 2018

miri64 commented Jan 29, 2019

obgm commented Mar 19, 2019

miri64 commented May 23, 2019

miri64 commented Sep 24, 2019

obgm commented Sep 24, 2019

miri64 commented Sep 24, 2019

miri64 commented Jul 8, 2020 • edited Loading

kYc0o commented Nov 9, 2016 •

edited by miri64

Loading

miri64 commented Jul 8, 2020 •

edited

Loading