Why can’t modprobe xt_HL work in init script?

Discussion in 'Tomato Firmware' started by Pess0g, Mar 13, 2018.

  1. Pess0g

    Pess0g Serious Server Member

    But the other scripts does well.After initialization, xt_HL can’t be found in lsmod.

    The router is in Gateway mod with WAN disabled.
  2. koitsu

    koitsu Network Guru Member

    You didn't list what firmware you're using (filename would be good). Below is for Toastman (tomato-RT-AC56U-9008.8Toastman-ARM-VPN-64K.trx).

    Use modprobe, not insmod.

    I haven't tested Scripts -> Init, but there's probably no reason it shouldn't work. Why don't you modprobe -s and log the result to syslog, that way logging info will show up in /var/log/messages? If the module can't be found, you will see it there.

    Manually it loads fine:

    root@gw:/tmp/home/root# find /lib/modules -iname "*xt_hl*"
    root@gw:/tmp/home/root# modprobe xt_HL
    root@gw:/tmp/home/root# lsmod | grep -i xt_hl
    xt_HL                   1396  0
    root@gw:/tmp/home/root# modprobe -r xt_HL
    root@gw:/tmp/home/root# modprobe xt_hl
    root@gw:/tmp/home/root# lsmod | grep -i xt_hl
    xt_hl                    890  0
    root@gw:/tmp/home/root# modprobe -r xt_hl
    root@gw:/tmp/home/root# lsmod | grep -i xt_hl
    ...although there seem to be two modules, both which claim to do the same thing. I'm not sure which is better/more recent/whatever:

    root@gw:/tmp/home/root# strings /lib/modules/ | egrep '^[a-z]+='
    description=Xtables: Hoplimit/TTL Limit field modification target
    author=Maciej Soltysiak <solt@dns.toxicfilms.tv>
    author=Harald Welte <laforge@netfilter.org>
    vermagic= SMP preempt mod_unload modversions ARMv7
    root@gw:/tmp/home/root# strings /lib/modules/ | egrep '^[a-z]+='
    description=Xtables: Hoplimit/TTL field match
    author=Maciej Soltysiak <solt@dns.toxicfilms.tv>
    vermagic= SMP preempt mod_unload modversions ARMv7
  3. Pess0g

    Pess0g Serious Server Member

    Shibby tomato-K26-1.28.RT-N5x-MIPSR2-132-Max

    I have said I used "modprobe".

    In "Scripts->Init", there is only "/jffs/init"
    In /jffs/init
    modprobe ipt_set
    modprobe ipt_REDIRECT
    modprobe ip_set_nethash
    modprobe -s xt_HL
    modprobe tun
    After restart, "lsmod | grep xt_HL"---nothing at all (checked "lsmod" as well) and iptables -t mangle -j TTL doesn't work, which needs module xt_HL.

    In log
    Jan  1 08:00:14 Router3 user.warn kernel: Algorithmics/MIPS FPU Emulator v1.5
    Jan  1 08:00:14 Router3 user.info kernel: JFFS2 version 2.2. (NAND) © 2001-2006 Red Hat, Inc.
    Jan  1 08:00:14 Router3 user.warn kernel: ip_set version 4 loaded
    Jan  1 08:00:15 Router3 user.info kernel: tun: Universal TUN/TAP device driver, 1.6
    Jan  1 08:00:15 Router3 user.info kernel: tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
    No information about module xt_HL logged.

    I can load it manually via "modprobe xt_HL" and iptables -t mangle -j TTL works.

    But this should be loaded at startup.
  4. koitsu

    koitsu Network Guru Member

    I don't know what to say/tell you. My gut feeling was that Scripts -> Init may be running too soon before the /jffs partition is mounted and available, but that surely can't be the case if ipt_set and tun are being loaded. There are way to properly design something running on filesystem mount, but in this case I don't think that's the problem.

    I do not know why this is happening, and I only know only two ways to partially debug/troubleshoot this. If neither of these turn up the root cause, then I literally have no idea of how to debug this further, because it would indicate it's something deep within the netfilter code and I don't know how to debug that (see last paragraph for further theories).

    Two things to try:

    1) In /jffs/init, do this insted:

    touch /tmp/log
    modprobe xt_HL >> /tmp/log 2>&1
    modprobe -s is supposed to send errors/etc to syslog (and I know for a fact it does), but I don't know if the entire netfilter library will end up sending data to syslog if there's a deeper problem. Some errors and C code end up doing things like opening stderr or stdout directly.

    2) I'm not sure if this will work for you because I use USB sticks and there are complications regarding timing and filesystem mounting that may be tricky to accomplish.

    Install Entware-ng and ensure there's an /opt bindable mount that actually points to wherever the Entware-ng installation is located. I don't know how to do this with JFFS so you are on your own here.

    Furthermore, right now something is wrong with their GitHub wiki so the link/docs for installation instructions redirects you to the main repo, I don't know what to do about this, it's a GitHub issue or something, I don't know.

    In short, you will need to make sure that /opt is pointing to your Entware-ng installation base directory and this needs to be done very very early on (usually people do this with /bin/mount -o bindable "/place/of/actual/opt/directory" /opt). You can't make a symlink because /opt is a directory as part of the ROM and cannot be removed, so the only way to work around it is to use an overlay/bindable mount. If this isn't set up/working, then Entware-ng will not function properly due to pathname problems.

    After that, change the modprobe xt_HL line to this:
    /opt/bin/strace -f -tt -s 2048 -o /tmp/log -- modprobe xt_HL
    Then paste the contents of /tmp/log onto pastebin and provide that link. I'm fairly certain there shouldn't be personal information in it.

    I know of no other way to debug this problem, given that it's occurring very early on in the startup process. This is one of the many complications with embedded devices. This may be one of those netfilter modules that needs to be compiled in statically, and not as a module, to alleviate odd issues/quirks. There have been some of those in the past.
  5. Pess0g

    Pess0g Serious Server Member

    I had done that before this thread was created( to syslog). But the result was nothing about xt_HL was logged.
    I don't use entware or similar platform and the router doesn't have a USB port.
  6. koitsu

    koitsu Network Guru Member

  7. lancethepants

    lancethepants Network Guru Member

    @koitsu I've uploaded those for you guys to try out.
  8. koitsu

    koitsu Network Guru Member

    @lancethepants Thank you so much!

    @Pess0g The instructions for this should be much simpler now. Assuming you have enough space in /jffs for the strace MIPS binary -- it's 904KBytes, and sorry, it's as small as it can possibly be:

    cd /jffs
    wget https://files.lancethepants.com/Binaries/strace/mipsel/strace%204.21/strace
    chmod 755 strace
    If this doesn't work -- specifically the wget failing, then it's probably because you're on older Shibby which doesn't work very well with HTTPS/SSL. So as an alternate, you can download it from my server (which is HTTP-only) and then decompress it. This would be the procedure for that:

    cd /tmp
    wget http://jdc.koitsu.org/strace.gz
    gzip -d strace.gz
    cp strace /jffs
    rm strace
    chmod 755 /jffs/strace
    After that, please see if the strace binary actually works by doing:

    /jffs/strace -V
    If it provides the strace version, then you're good. If not, please show the error you get. Assuming it works, change the modprobe xt_HL line in your script to this:

    /jffs/strace -f -tt -s 2048 -o /tmp/log -- modprobe xt_HL
    Then paste the contents of /tmp/log onto pastebin and provide that link.
  9. Pess0g

    Pess0g Serious Server Member

    @Pess0g The instructions for this should be much simpler now. Assuming you have enough space in /jffs for the strace MIPS binary -- it's 904KBytes, and sorry, it's as small as it can possibly be:

    What a shame there is only 108KB left.
  10. lancethepants

    lancethepants Network Guru Member

    @koitsu @Pess0g
    You can use upx to shrink down the binary, in most cases by a fraction.
    upx --ultra-brute ./binary.
    Strace for mipsel shrinks down to 232K, reported ratio of 26.21% it's original size. Upx is also on my site, or you can get it from their site.
  11. Pess0g

    Pess0g Serious Server Member

    Here is the log
  12. koitsu

    koitsu Network Guru Member

    Thank you. You can delete the strace binary at this point -- it was only needed for trying to troubleshoot this.

    Sadly, the output indicates the module is being read and loaded correctly, with init_module() returning 0 (success), and modprobe also returning/exiting with 0 (success).

    323   00:00:15.330652 open("/lib/modules/", O_RDONLY|O_LARGEFILE) = 3
    323   00:00:15.331490 read(3, "\177E", 2) = 2
    323   00:00:15.371151 _llseek(3, -2, [0], SEEK_CUR) = 0
    323   00:00:15.371776 fstat64(3, {st_mode=S_IFREG|0644, st_size=3884, ...}) = 0
    323   00:00:15.372635 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\1\0\10\0\1\0\0\0\0\0\0\0\ {snipping for brevity; this is the raw contents of the xt_HL.ko file}" ..., 4096) = 3884
    323   00:00:15.377772 read(3, "", 212)  = 0
    323   00:00:15.378333 close(3)          = 0
    323   00:00:15.378855 init_module(0x48b5a0, 3884, "") = 0
    323   00:00:15.384209 exit(0)           = ?
    323   00:00:15.385012 +++ exited with 0 +++
    I don't know what to say. This sounds almost like a weird/obscure bug in either:

    a) the module -- possibly it has other module dependencies which are missing from the actual code (you can tell netfilter modules what they depend on, and this one says it has none), therefore init_module() is successful but netfilter/kernel doesn't actually load it because of missing symbols due to dependent modules not being loaded in advance, I don't know
    b) netfilter -- possibly it should be throwing an error all the way back up to userland but it isn't

    As mentioned previously: in the past, we have seen cases with a couple different netfilter modules where if compiled as dynamically-loadable (CONFIG_NETFILTER_XT_xxx=m), the module simply "didn't work quite right" and behaved oddly. To solve this, we ended up compiling this module *statically* into the kernel (CONFIG_NETFILTER_XT_xxx=y). I don't recall the exact module where this was seen, but I keep thinking ipt_REDIRECT was one of them.

    In fact, come to think of it, most of the problematic ones are usually ones that require "uncommon" targets (ex. ipt_REDIRECT implements -j REDIRECT in iptables, xl_HT implements new options when used with -m ttl and -j TTL). xl_HT also only applies to the mangle table, which is dynamically loaded as a module (

    This in turn makes me think the problem is actually with netfilter; probably a bug that's since been solved, but we can't "safely" upgrade the netfilter code (and netfilter version differs between MIPS and ARM already).

    If the module was compiled in statically, you wouldn't need to use modprobe at all, and the functionality should (?) be there all the time. It's literally a one-line change in the Linux kernel config file config_base:


    However, now that I'm looking at config_base more, I also see this, which makes me wonder:

    # CONFIG_IP_NF_TARGET_TTL is not set
    Linux kernel is extremely bad about these naming conventions, so I will have to look up what the latter one does. The more I dig through this kernel config looking at "obscure" or uncommon netfilter or xtables things, the more I realise that a bunch of this might be broken.

    There's also the IPv6 aspect too, which complicates stuff: netfilter IPv6 modules are INTENTIONALLY built as modules, so that they are only loaded if IPv6 is enabled during runtime, or if another module requires them (for example, ip6table_mangle and ip6table_filter (these are for ip6tables) are loaded dynamically); xl_HL is one of those that supports IPv4 and IPv6. And that leads me to ask this:

    Do you have IPv6 enabled in TomatoUSB? If so: if you disable it, does modprobe xt_HL in Scripts -> Init start working?
  13. Sean B.

    Sean B. LI Guru Member

    .... Yikes.
  14. Pess0g

    Pess0g Serious Server Member

    Disabled IPv6 doesn't help
  15. koitsu

    koitsu Network Guru Member

    Thanks for testing. Yeah, sadly, I think this might be a case where since it's impossible to debug the problem, the workaround is to compile the module into the kernel statically so that it should always be available. As I said, it's literally a one-line change to a single file to do this.

    Note for tech readers: do not mistake xt_hl.c for xt_HL.c -- they are two different modules that look similar but do something very different. (Links in question go to Toastman-ARM7 branch because I'm lazy and don't feel like linking all the different branches):

    xl_hl (a.k.a. ipt_ttl for IPv4, ipt6_hl for IPv6) matches TTL field values. You use it like iptables -m ttl or ip6tables -m hl (verified by using -h with both of those).

    xt_HL (a.k.a. ipt_TTL for IPv4, ipt6_HL for IPv6) modifies TTL field values. This module creates a netfilter target called TTL (IPv4) or HL (IPv6). You use it like iptables -j TTL or ip6tables -j HL (verified by using -h with both of those). It also only works within the mangle table.

    hl/HL in IPv6 context refers to the Header Length field.

    Two remaining questions:

    1. Does it work if you move your modprobe xt_HL line into Scripts -> Firewall instead? Is there a reason it needs to be done so early in Scripts -> Init?

    2. What exact feature of xt_HL do you need / what for? I ask because I came across this thread on snbforums where RMerlin mentioned that if your ISP limits your TTL to 1, there's a new way (in stock Asus and Merlin firmware) to increase that to 64. I'm not sure if what Asus implemented is done using xt_HL or not, or if that situation applies to you, hence my question. Possibly this is something that could be implemented in TomatoUSB as well (a simple checkbox).
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice