====== Linux netfilter nf_conntrack_expect ======

I got a very weird issue with a **GNU/Linux firewall** used to do **NAT** for about 40 VoIP phones using the **SIP protocol**. Very often the phone calls broke unexpectedly, sometimes the call just drop, other times the voice stream stops.

===== Problem: all ports in use =====

Looking at the **syslog** I found a clear clue as to what the problem was:

<file>
nf_ct_sip: dropping packet: all ports in use for SDP media IN= OUT= SRC=a.a.a.a DST=b.b.b.b
           LEN=820 TOS=0x08 PREC=0x80 TTL=127 ID=23404 PROTO=UDP SPT=13182 DPT=5060 LEN=800
nf_ct_sip: dropping packet: cannot add expectation for voice IN= OUT= SRC=a.a.a.a DST=b.b.b.b
           LEN=820 TOS=0x08 PREC=0x80 TTL=127 ID=23404 PROTO=UDP SPT=13182 DPT=5060 LEN=800
nf_ct_sip: dropping packet: all ports in use for SDP media IN= OUT= SRC=a.a.a.a DST=b.b.b.b
           LEN=820 TOS=0x08 PREC=0x80 TTL=127 ID=23405 PROTO=UDP SPT=13182 DPT=5060 LEN=800
nf_ct_sip: dropping packet: cannot add expectation for voice IN= OUT= SRC=a.a.a.a DST=b.b.b.b
           LEN=820 TOS=0x08 PREC=0x80 TTL=127 ID=23405 PROTO=UDP SPT=13182 DPT=5060 LEN=800
</file>

So this is not a pure NAT problem, e.g. it is not a problem like a full NAT table. Here the problem is about the //expect table//, i.e. a table used by some specific helper modules to do connection tracking for specific network protocols. The specific connection tracking in my case is about **SIP protocol**, handled by kernel module **nf_ct_sip**.

The SIP protocol does not relay on normal TCP/IP sessions, so the association between the host behind NAT and the public server is not maintained by the normal NAT table. Instead, the kernel maintains the expect table, which can be inspected with:

<code>
cat /proc/net/nf_conntrack_expect
34 l3proto = 2 proto=17 src=b.b.b.b dst=a.a.a.a sport=0 dport=48968 PERMANENT sip/signalling
37 l3proto = 2 proto=17 src=b.b.b.b dst=a.a.a.a sport=0 dport=35460 PERMANENT sip/signalling
35 l3proto = 2 proto=17 src=b.b.b.b dst=a.a.a.a sport=0 dport=40949 PERMANENT sip/signalling
56 l3proto = 2 proto=17 src=b.b.b.b dst=a.a.a.a sport=0 dport=14161 PERMANENT sip/signalling
...
</code>

This should be the same information shown by:

<code>
conntrack -L expect
</code>

The //expect table// have a size limit, that can be inspected with:

<code>
cat /proc/sys/net/netfilter/nf_conntrack_expect_max
</code>

This is not the number of the //expect table// entries, but an unspecified size; I suspect that you have to **divide by 4** to obtain the hash table size. It seems that the default ''nf_conntrack_expect_max'' on GNU/Linux kernel 5.10 is a very low **256** (maybe 64 entries).

===== Solution =====

The solution is to enlarge the expect table. You can do it at runtime, after the **nf_conntrack module** is loaded, writing to the **/proc/** pseudo filesystem. Or you can pass the **expect_hashsize** parameter at module load time.

The last method is better, because you don't know when the kernel module is loaded during the boot process, so you risk to write the ''/proc/'' filesystem too early, when the module is still not present. So the best solution is to add a **/etc/modprobe.d/local.conf** file with this line:

<file>
options nf_conntrack expect_hashsize=2048
</file>

===== sysctl and systemd =====

You might think of a solution by adding a file **/etc/sysctl.d/local.conf** with this line:

<file>
# This does not work, because module nf_conntrack is not loaded yet.
net.netfilter.nf_conntrack_expect_max=8192
</file>

But the **systemd** unit **systemd-sysctl.service** may enforce that setting too early, when the kernel module is not loaded yet. The error message in syslog will be;

<code>
Couldn't write '8192' to 'net/netfilter/nf_conntrack_expect_max',
    ignoring: No such file or directory
</code>

The option works instead if, after the bootstrap, you run:

<code>
systemctl restart systemd-sysctl.service
</code>

===== Web references =====

  * **[[https://forums.thinkbroadband.com/dslrouter/4443347-conntrack-expectation-table-full-issues-nf-conntrack.html|conntrack expectation table full issues (nf_conntrack)]]**