Using ipset to bypass a VPN

Discussion in 'Tomato Firmware' started by Rob Meades, Jan 2, 2017.

  1. Rob Meades

    Rob Meades Network Newbie Member

    I'm attempting to use ipset to divert certain URLs around my VPN connection (the "Netflix" problem) with Tomato (Shibby 1.28) on an Asus RT-N66U router. The set of IP addresses I end up collecting looks convincing and marking the IP packets headed to those URLs appears to be working but I find that I am unable to browse to the URLs in the set at all (I can get to all other URLs). Can anyone see what step I've missed?

    Dnsmasq custom configuration (with _ replacing . to avoid URLs appearing in this post) is:
    Code:
    ipset=/netflix_com/nflxext_com/nflximg_com/nflxvideo_net/nflximg_net/netflix_edgekey_net/amazonaws_com/bbc_co_uk/BYPASS_VPN
    Init script is:
    Code:
    modprobe ipt_set
    ipset -N BYPASS_VPN iphash
    Firewall-up script is:
    Code:
    iptables -t mangle -D PREROUTING -i br0 -m set --set BYPASS_VPN dst -j MARK --set-mark 1
    iptables -t mangle -A PREROUTING -i br0 -m set --set BYPASS_VPN dst -j MARK --set-mark 1
    WAN-up script is:
    Code:
    ip rule del prio 100 fwmark 1 lookup 100 
    ip rule add prio 100 fwmark 1 lookup 100
    ip route flush table 100
    ip route add table 100 default via $(nvram get wan_gateway) dev $(nvram get wan_ifname)
    After a few attempts to get to one of the ipset URLs, iptables logging shows:
    Code:
    > iptables -t mangle -vnL PREROUTING 
    Chain PREROUTING (policy ACCEPT 108K packets, 71M bytes) 
     pkts bytes target     prot opt in     out     source               destination 
    29247 4436K DSCP       all  --  vlan2  *       0.0.0.0/0            0.0.0.0/0           DSCP set 0x00 
       15   760 MARK       all  --  br0    *       0.0.0.0/0            0.0.0.0/0           set BYPASS_VPN dst MARK set 0x1 
    ...so some stuff is definitely getting into the set.

    FYI, other debug items:
    Code:
    > ip rule show 
    0:      from all lookup local 
    100:    from all fwmark 0x1 lookup 100 
    32766:  from all lookup main 
    32767:  from all lookup default
    Code:
    > ip route show table 100 
    default dev vlan2  scope link
    Code:
    > ip route show 
    10.3.10.1 via 10.3.10.5 dev tun11 
    46.166.186.237 via 82.24.196.1 dev vlan2 
    10.3.10.5 dev tun11  proto kernel  scope link  src 10.3.10.6 
    82.24.196.1 dev vlan2  scope link 
    10.10.1.0/24 dev br0  proto kernel  scope link  src 10.10.1.1 
    82.24.196.0/22 dev vlan2  proto kernel  scope link  src 82.24.197.229 
    127.0.0.0/8 dev lo  scope link 
    0.0.0.0/1 via 10.3.10.5 dev tun11 
    128.0.0.0/1 via 10.3.10.5 dev tun11 
    default via 82.24.196.1 dev vlan2
    ...where 10.10.1.0/24 is my local LAN, 46.166.186.217 is my VPN IP address, 82.24.196.1 is the gateway address of my ISP, vlan2 is the WAN and tun11 is the VPN tunnel.
    Code:
    > ipset -L BYPASS_VPN
    Name: BYPASS_VPN 
    Type: iphash 
    References: 0 
    Header: hashsize: 1024 probes: 8 resize: 50 
    Members: 
    52.211.27.130 
    52.21.20.247 
    52.206.148.70 
    52.206.154.135 
    52.206.162.144 
    77.72.112.213 
    54.76.108.115 
    212.58.244.26 
    52.31.168.52 
    52.49.11.185 
    212.58.246.54 
    54.229.217.86 
    72.247.95.146 
    52.51.113.93 
    54.229.89.20 
    52.213.107.8 
    52.30.12.70 
    52.210.17.220 
    1.2.3.4 
    104.84.191.232 
    52.207.202.17 
    54.229.158.183 
    52.31.45.219 
    52.211.125.254 
    52.210.133.255 
    52.211.125.102 
    52.18.138.175 
    52.209.186.165 
    54.229.103.136 
    52.206.176.189 
    54.229.126.241 
    52.17.149.210 
    52.51.252.111 
    52.50.212.245 
    54.171.228.16 
    23.195.146.96 
    52.30.103.23 
    52.16.228.47 
    52.18.14.111 
    52.211.170.56 
    52.204.95.117 
    52.49.120.6 
    52.49.63.155 
    52.206.185.219
     
  2. Fab Five Freddy

    Fab Five Freddy Serious Server Member

    Shouldn't your table 100 be

    default via 82.24.196.1 dev vlan2
     
  3. Rob Meades

    Rob Meades Network Newbie Member

    Sorry, yes, I experimented with using the "via" directive and not using it when creating table 100, the ip route show table 100 output above happened to be when I wasn't using it. Agreed that:
    Code:
    ip route add table 100 default via $(nvram get wan_gateway) dev $(nvram get wan_ifname)
    ...does indeed produce:
    Code:
    default via 82.24.196.1 dev vlan2
    ...which, in fact, is what you see in the ip route show output.

    Either way the effect is the same unfortunately, I can't get to the things in the ipset URL list.
     
    Last edited: Jan 2, 2017
  4. grendelous

    grendelous Network Newbie Member

  5. Rob Meades

    Rob Meades Network Newbie Member

    Thanks, yes, I saw your post there but I wanted something that worked with URLs, whereas I think yours works with IP addresses, and so is likely to fail if those addresses ever change. Is that true? Mind you, mine is not working for anything at the moment...
     
  6. grendelous

    grendelous Network Newbie Member

    Mine does both, but I found that the domain names were not enough to fully cover everything Netflix needed.
     
    Rob Meades likes this.
  7. Rob Meades

    Rob Meades Network Newbie Member

    Oh, very good, I'll dig in and see how far I get.
     
  8. Rob Meades

    Rob Meades Network Newbie Member

    Well, I can browse to both Netflix and the BBC with your script, which is a definite improvement, but I still can't get Netflix to play anything, error M7111-1331-5059 is the order of the day it would seem. S'pose it only takes one different IP address to spoil the pot.
     
  9. Rob Meades

    Rob Meades Network Newbie Member

    Aaah! No, my router needs a slightly different syntax for iptables (--set rather than --match-set). It works!!!! Many thanks, very happy now. :)
     
  10. koitsu

    koitsu Network Guru Member

    Asus RT-N66U is MIPS (CPU architecture), on which Tomato uses Linux 2.6.22, using netfilter/iptables 1.3.8.

    ARM-based Tomato routers run Linux 2.6.34, using netfilter/iptables 1.4.x.

    There are many syntactical differences between the two versions of netfilter/iptables. Most of the crap you'll read online is for present-day Linux, which uses iptables 1.4.x.
     
  11. Rob Meades

    Rob Meades Network Newbie Member

    Yes, it can be a bit difficult determining the truth. I now have to route a VPN bridge around this tangle as well, so the truth is probably quite important to me. I'll go look for netfilter/iptables 1.3.8 man pages...
     
  12. Rob Meades

    Rob Meades Network Newbie Member

    There is a slightly odd thing here. This is all working for Netflix, which is very good, but if I have bbc.co.uk in the list of ipset URLs then I cannot browse to the BBC; HTTP never connects. When I do this after a fresh reboot ipset -L BYPASS_VPN shows that the only IP address in my bypass list is 1.2.3.4, which appears to be the BBC.

    If I take bbc.co.uk out of the ipset list I can browse to the BBC and, for reasons I don't understand, the BBC offers me the .co.uk pages, rather than the .com pages, despite my IP address not being in the UK according to whatismyipaddress (my VPN IP address emerges in The Netherlands). So it doesn't actually matter but it is a bit strange.

    FYI, if I take the Netflix URLs out of the ipset list then Netflix is definitely not playing ball, so ipset is certainly having an effect. Mysterious.
     
    Last edited: Jan 4, 2017
  13. Rob Meades

    Rob Meades Network Newbie Member

    There's another result thing that's confusing me with this script in place. I have added an IP address to the list, ipt 82.1.215.240, which I know to be pingable. If I print out the mangle table, it clearly includes that IP address:
    Code:
    iptables -t mangle -vnL PREROUTING
    ...
      0     0 MARK       all  --  br0    *       0.0.0.0/0            82.1.215.240        MARK set 0x88
    ...
    So packets sent to this IP address should be bypassing my VPN. Yet traceroute, run on the router itself, shows that packets to this destination are going via the VPN tunnel (10.28.10.1):
    Code:
    traceroute to 82.1.215.240 (82.1.215.240), 30 hops max, 38 byte packets
     1  10.28.10.1 (10.28.10.1)  34.736 ms  35.793 ms  35.044 ms
     2  46.166.188.254 (46.166.188.254)  35.191 ms  36.787 ms  36.308 ms
     3  185.107.116.22 (185.107.116.22)  36.795 ms  49.904 ms  39.888 ms
     4  nforce-globalswitch.telecity2.openpeering.nl (82.150.157.146)  36.468 ms  38.070 ms  48.843 ms
     5  telecity-ixr.openpeering.nl (217.170.0.244)  38.367 ms  39.155 ms  47.778 ms
     6  openpeering-10g.virginmedia.net (82.150.153.110)  38.797 ms  37.798 ms  38.162 ms
    ...and the counts in the mangle table against this IP address are not incrementing either. I know the script is having an effect on the world, because Netflix works and counts in the mangle table are increasing for some Netflix IP addresses, so why is traceroute getting this result? Could the mangle be ignoring ICMP or UDP packets for some reason (I think it should affect all packets by default)? I have tried adding -p 0, just in case, but it doesn't help.

    For completeness, here is table 200 and the ip rule list at the time:
    Code:
    ip route show table 200
    46.166.188.223 via 82.24.196.1 dev vlan2
    10.8.0.2 dev tun21  proto kernel  scope link  src 10.8.0.1
    82.24.196.1 dev vlan2  scope link
    10.28.10.5 dev tun11  proto kernel  scope link  src 10.28.10.6
    10.28.10.1 via 10.28.10.5 dev tun11
    10.10.0.0/24 via 10.8.0.2 dev tun21
    10.10.1.0/24 dev br0  proto kernel  scope link  src 10.10.1.1
    82.24.196.0/22 dev vlan2  proto kernel  scope link  src 82.24.197.229
    127.0.0.0/8 dev lo  scope link
    default via 82.24.196.1 dev vlan2
    
    ip rule list
    0:      from all lookup local
    32765:  from all fwmark 0x88 lookup 200
    32766:  from all lookup main
    32767:  from all lookup default
    EDIT: the above is true for traceroute run on the router. If I run traceroute from a PC on the LAN it looks correct. Does that imply that the mangling process does not work for packets that are created on the router itself? If so, this might explain why I'm having problems with routing packets created by OpenVPN.
     
    Last edited: Jan 10, 2017
  14. koitsu

    koitsu Network Guru Member

    Packets that are forwarded through the router are different than packets originating from the router. I've talked about this repeatedly over the years (in both threads you'll need to read my descriptions of chains, the rest is irrelevant):

    http://www.linksysinfo.org/index.php?threads/cant-save-admin-scripts.73072/#post-282662
    http://www.linksysinfo.org/index.php?threads/help-with-iptable-logging-for-a-newb.73138/#post-283433

    I don't know how mangle fits into the picture, but this seems to shed light on it: https://upload.wikimedia.org/wikipedia/commons/3/37/Netfilter-packet-flow.svg

    Pay very close attention to the "columns" (they're easy to miss): INPUT path, FORWARD path, and OUTPUT path. Ignore the stuff in blue for now (that's ebtables/link layer/layer 2), focus just on the green.

    So based on that diagram: your mangle table PREROUTING rule is matching an input interface of br0. I don't see how this is going to match packets originating from the router (for a couple reasons! :p). Table mangle, chain OUTPUT would be a more likely candidate for matching packets originating there (or maybe just the filter table OUTPUT chain -- again, see diagram).

    Remember: the "earlier on" you mess with packets (ex. raw vs. mangle vs. nat vs. filter), the more processing has to happen (because more packets are going to have to be analysed).
     
    Rob Meades likes this.
  15. Rob Meades

    Rob Meades Network Newbie Member

    Yes, yes, that's it (I realised as I went to sleep last night). The problem I have is that OpenVPN, running on the router, is creating packets that I need to be sent down a particular interface but OpenVPN cannot be told to do that, only to send to an IP address. Hmph.
     
  16. koitsu

    koitsu Network Guru Member

    This sounds more like an issue of IP routing (not iptables!) rules and metrics (or priorities). You can prioritise routes to certain networks (or IPs) to go through a particular interface. OpenVPN gets its own interface; it's either tunXX or tapXX, depending on your configuration/setup.

    I would suggest maybe ignoring the iptables bits and instead focusing on adding routes for certain IPs and see how Linux decides to route them, interface-wise. I believe ip route has a way to actually test this (at some level), as in "where would you route packet to dst addr x.x.x.x" and it shows you (but the ip command syntax is a little odd -- I spent some time on this months ago, and have since forgotten).

    I understand that it looks like the goal is to use iptables to "mark" packets with a particular number, which then IP routing rules (ip route) can then key off of ("fwmark") and make appropriate decisions.

    I'd suggest, for troubleshooting, removing ipset and iptables and "marks" from the picture. Just play with IP routes directly and see if you can get packets going out the right interface.
     
    Rob Meades likes this.
  17. Rob Meades

    Rob Meades Network Newbie Member

    That netfilter packet flow diagram is excellent, just what I was looking for to put things into context.

    I agree that this is a routing problem and I've tried removing ipset and "marks", as you suggest, but I've still not quite cracked it.

    Using traceroute -s I have verified that, with the PIA OpenVPN client running, when I ping Router B from 10.8.0.1 on Router A, the packets go out of vlan2 (so they are no longer going down the PIA OpenVPN tunnel) and arrive at Router B. However no packets ever get to the tun interface of Router B (10.8.0.2) while the PIA OpenVPN client is running. The VPN bridge is definitely up, I can see OpenVPN establish it in the router logs, and the moment I stop the PIA OpenVPN client the bridge immediately works, so it's not like it has to be re-established or some such. All the OpenVPN stats for the bridge on Router A now show non-zero entries, so OpenVPN thinks things are happening, but the OpenVPN stats for the bridge client on Router B always show zero for "TUN/TAP write bytes" and "TCP/UDP read bytes".

    Where can my OpenVPN packets be going...?
    Code:
           LAN         Shibby Tomato Router A                  Router B               Bridged LAN
                     ___________________________        _____________________            
                    |                           |      |                     |
                    |   OpenVPN Bridge Server   |      |    OpenVPN Client   |       10.10.0.0/24
                    |         10.8.0.1          |------|       10.8.0.2      |
                    |          tun21            |      |_____________________|
                    |                           | 
                    |                           |
                    |                           |                                Private Internet Access       The Internet
                    |                           |                             _____________________________                  
                    |       OpenVPN Client      |                            |                             |
      10.10.1.0/24  |         10.5.10.6         |--------------------------- | 10.5.10.5 -- 46.166.288.241 |         *
                    |           tun11           |                            |_____________________________|
                    |                           |
                    |                           |
                    |                           |                                          ISP                 The Internet
                    |                           |                             ______________________________                
                    |                           |                            |                              |
                    |           vlan2           |--------------------------- |         82.24.196.1          |        *
                    |                           |                            |______________________________|
                    |                           |
                    |___________________________|
     
  18. koitsu

    koitsu Network Guru Member

    Try using ip -s route get {addr} to see what interface things are flowing out of. It's possible that the problem isn't the outbound packets, but that the *return path* is different (i.e. response packets are coming back through the "wrong" interface); this is known as asymmetric routing.

    Using traceroute -s on the router itself (you didn't specify where you ran this from, so I have to assume the router), AFAIK, will not simulate packets going through the FORWARD chain.

    The easiest way to troubleshoot things like this is to start doing packet captures on both ends of a connection with tcpdump (you can't do it on just one, because you then only see half the picture). tcpdump -p -i {interface} -l -n with an expression that matches a specific kind of packet you're sending (ex. for ping just use "icmp" although traceroute will use that too for hop discovery but not for payload (it uses UDP for that), so traceroute -I is more useful). You may need to have several windows open with several tcpdumps running at once to see what packets are flowing. Remember: tcpdump (libpcap/bpf) will see packets regardless of firewall, i.e. it always sees everything flowing across the wire.

    It's also possible that some of the /proc entries for the relevant tunXX interfaces are lacking forwarding turned on. I'm simply not sure. Look around /proc/sys/net/ipv4/conf for this, and Google for info about that /proc tree.

    Reference for ip: http://linux-ip.net/html/tools-ip-route.html -- see examples D.2.2. and D.2.8. And do not miss sub-note [51] (I've been bit by this).
     
  19. Rob Meades

    Rob Meades Network Newbie Member

    ip -s route get 10.8.0.2, I think, shows that the router wants stuff to go down the correct tun:

    10.8.0.2 dev tun21 src 10.8.0.1

    ...but querying the cache returns no entry:

    ip -s route show cache 10.8.0.2

    ...so I've plainly not understood how to set the routing table correctly for the outgoing path yet. I've read the ip route pages a few times but haven't had a light-bulb moment, will continue reviewing. Forwarding seems to be on for tun21 so that should be OK.

    I would like to use tcpdump but my (recent) version of Shibby Tomato router doesn't come with it, so I'll go see if someone has a version somewhere that works on MIPS.

    Here's my route for 10.8.0.1 (which is just a copy of my main routing table with the 0.0.0.0/1 and 128.0.0.0/1 routes added by the PIA OpenVPN client removed) in case anyone can point out my idiocy:

    Code:
    ip rule list
    0:      from all lookup local
    32760:  from 10.8.0.1 lookup 300
    32766:  from all lookup main
    32767:  from all lookup default
    
    ip route show table 300
    46.166.190.190 via 82.24.196.1 dev vlan2
    10.8.0.2 dev tun21  proto kernel  scope link  src 10.8.0.1
    82.24.196.1 dev vlan2  scope link
    10.28.10.5 dev tun11  proto kernel  scope link  src 10.28.10.6
    10.28.10.1 via 10.28.10.5 dev tun11
    10.10.0.0/24 via 10.8.0.2 dev tun21
    10.10.1.0/24 dev br0  proto kernel  scope link  src 10.10.1.1
    82.24.196.0/22 dev vlan2  proto kernel  scope link  src 82.24.197.229
    127.0.0.0/8 dev lo  scope link
    default via 82.24.196.1 dev vlan2
     
    Last edited: Jan 13, 2017
  20. Rob Meades

    Rob Meades Network Newbie Member

    Update: I have tcpdump installed on Router A now and have monitored the OpenVPN bridge tun (tun21) and, at the same time, monitored packets leaving the vlan2 interface destined for the ISP address of Router B. I am running tracert 10.10.0.1 on a PC attached to the LAN of Router A, which sends packets over the VPN bridge.

    Whether the PIA OpenVPN client is up or not, I can see that these packets correctly enter tun21 but with the PIA OpenVPN client up they never get to the vlan2 interface. Also, though, with the PIA OpenVPN client up, the stats issued by the OpenVPN bridge server never increase.

    This implies that the packets are getting into tun21 but are not being read out of there by the OpenVPN bridge server if the OpenVPN client to PIA is up.

    Eh??? Is there a way that a routing problem could cause this?

    EDIT: that's not quite true, the stats issued by the OpenVPN bridge server do sometimes increase, but not at the time that something lands in tun21 (I think the increase is just down to the far end reestablishing the link because ping is failing).
     
    Last edited: Jan 15, 2017
  21. Rob Meades

    Rob Meades Network Newbie Member

    Finally, I have fixed this, having run tcpdump in loads of windows to understand what's going on.

    I had mistakenly though that packets from my OpenVPN bridge server would be coming from 10.8.0.1. In fact, of course, the traffic is going through the interface that the OpenVPN bridge server is using, tun21 in my case, and then out of the port that the OpenVPN bridge is attached to, 1194 in my case. So to grab packets from port 1194 and send them out of a specific interface, I needed to create a mangle to point to the new routing table. The mangle needed to be on the OUTPUT table since the packets are locally generated by the VPN, and look for things from the source port 1194.

    Hence a line of the form:

    iptables -t mangle -A OUTPUT -p udp --sport 1194 -j MARK --set-mark 0x88

    ...did the trick, having already set myself up a routing table and then added the rule:

    ip rule add fwmark 0x88 table 200

    Phew. At last.

    Rob
     
  22. hungluu

    hungluu Serious Server Member

    So Rod, can you paste the full script that's working for you to leave Netflix out of the VPN route? I've been reading this but can't seem to figure which one is currently working.
     
  23. Rob Meades

    Rob Meades Network Newbie Member

    Gimme a few days to get Amazon video working as well then I'll post the whole thing.
     
  24. Rob Meades

    Rob Meades Network Newbie Member

    OK, so I have taken the excellent work of grendelous (and those who went before) here:

    http://www.linksysinfo.org/index.ph...sp-and-two-vpn-connections.72248/#post-282992

    ...and modified it for my situation, specifically:
    • I'm situated in the UK but I run a VPN client (Private Internet Access) which emerges outside the UK ('cos I don't like the idea of T. May, her friends and misc. hackers reading my innermost),
    • I need Netflix, Amazon Instant Video and the BBC to be routed directly through my ISP instead as they have geographical restrictions,
    • I also run a VPN bridge, the server-side of which is running on this router, and that needs to not be fouled-up by the PIA VPN client either.
    After following the instructions at the link above, I found that I needed my ipset to be as follows:
    Code:
    ipset=/netflix.com/nflxext.com/nflximg.com/nflxvideo.net/amazonaws.com/nflximg.net/netflix.edgekey.net/amazon.com/media-amazon.com/bbci.co.uk/bbci.co.uk.edgekey.net/<url_of_other_end_of_vpn_bridge>/<url_of_dynamic_dns_client_if_you_have_one>/BYPASS_VPN
    ...where you will note that you need to fill in a few bits.

    Then, the firewall script purely to do the Netflix, Amazon Instant Video and BBC stuff became:
    Code:
    # http://www.linksysinfo.org/index.ph...en-regular-isp-and-two-vpn-connections.72248/
    # Put this in the Firewall portion of the script section in Administration
    
    #!/bin/sh
    #set -x # uncomment/comment to enable/disable debug mode
    
    (
    TID="200"
    FW_MARK="0x88"
    IPSET="BYPASS_VPN"
    
    # cleanup from prior execution
    (
    # stop split tunnel
    ip rule del fwmark $FW_MARK table $TID
    
    # delete firewall rules
    iptables -t mangle -F
    
    # delete ipset hash table
    ipset -F $IPSET
    ipset -X $IPSET
    
    # delete alternate routing table
    ip route flush table $TID
    
    # force routing system to recognize our changes
    ip route flush cache
    
    # enable reverse path filtering
    for i in /proc/sys/net/ipv4/conf/*/rp_filter; do echo 1 > $i; done
    
    sleep 3
    ) > /dev/null 2>&1
    
    # quit if neither OpenVPN client is active
    ! ip route show | egrep -qm1 'tun1[1-2]' && exit
    
    # copy main routing table (exclude all default gateway routes)
    ip route show | egrep -v '^default|^0.0.0.0/1|^128.0.0.0/1' \
    | while read route; do
    ip route add $route table $TID
    done
    
    # add WAN as default gateway
    ip route add default via $(nvram get wan_gateway) table $TID
    
    # force routing system to recognize our changes
    ip route flush cache
    
    # disable reverse path filtering on the interfaces of interest
    echo 0 > /proc/sys/net/ipv4/conf/br0/rp_filter
    echo 0 > /proc/sys/net/ipv4/conf/tun11/rp_filter
    echo 0 > /proc/sys/net/ipv4/conf/vlan2/rp_filter
    
    # load required netfilter module
    (modprobe xt_set || modprobe ipt_set || modprove ip_set_iphash) 2> /dev/null
    
    # IP block adder
    ipt() {
      iptables -t mangle -A PREROUTING -i br0 -d "$1" -j MARK --set-mark $FW_MARK
    }
    
    # Add Netflix IP blocks
    ipt 108.175.32.0/20
    ipt 108.175.32.0/24
    ipt 108.175.33.0/24
    ipt 108.175.34.0/24
    ipt 108.175.35.0/24
    ipt 108.175.38.0/24
    ipt 108.175.39.0/24
    ipt 108.175.40.0/24
    ipt 108.175.41.0/24
    ipt 108.175.42.0/24
    ipt 108.175.43.0/24
    ipt 108.175.44.0/24
    ipt 108.175.46.0/24
    ipt 108.175.47.0/24
    ipt 185.2.220.0/22
    ipt 185.2.222.0/24
    ipt 185.2.223.0/24
    ipt 185.9.188.0/22
    ipt 185.9.188.0/24
    ipt 192.173.112.0/20
    ipt 192.173.64.0/18
    ipt 192.173.64.0/20
    ipt 192.173.80.0/20
    ipt 192.173.96.0/20
    ipt 198.38.100.0/24
    ipt 198.38.101.0/24
    ipt 198.38.102.0/23
    ipt 198.38.102.0/24
    ipt 198.38.108.0/24
    ipt 198.38.109.0/24
    ipt 198.38.110.0/24
    ipt 198.38.111.0/24
    ipt 198.38.112.0/24
    ipt 198.38.113.0/24
    ipt 198.38.114.0/24
    ipt 198.38.115.0/24
    ipt 198.38.116.0/24
    ipt 198.38.117.0/24
    ipt 198.38.118.0/24
    ipt 198.38.119.0/24
    ipt 198.38.120.0/24
    ipt 198.38.121.0/24
    ipt 198.38.124.0/24
    ipt 198.38.125.0/24
    ipt 198.38.96.0/19
    ipt 198.38.96.0/24
    ipt 198.38.97.0/24
    ipt 198.38.98.0/24
    ipt 198.38.99.0/24
    ipt 198.45.48.0/20
    ipt 198.45.48.0/24
    ipt 198.45.49.0/24
    ipt 198.45.50.0/23
    ipt 198.45.50.0/24
    ipt 198.45.52.0/24
    ipt 198.45.53.0/24
    ipt 198.45.54.0/24
    ipt 198.45.55.0/24
    ipt 198.45.56.0/24
    ipt 198.45.57.0/24
    ipt 198.45.61.0/24
    ipt 198.45.62.0/24
    ipt 198.45.63.0/24
    ipt 208.75.79.0/24
    ipt 23.246.0.0/18
    ipt 23.246.12.0/24
    ipt 23.246.13.0/24
    ipt 23.246.14.0/24
    ipt 23.246.15.0/24
    ipt 23.246.16.0/24
    ipt 23.246.17.0/24
    ipt 23.246.18.0/24
    ipt 23.246.2.0/24
    ipt 23.246.20.0/24
    ipt 23.246.21.0/24
    ipt 23.246.22.0/24
    ipt 23.246.23.0/24
    ipt 23.246.24.0/24
    ipt 23.246.25.0/24
    ipt 23.246.26.0/24
    ipt 23.246.27.0/24
    ipt 23.246.28.0/22
    ipt 23.246.28.0/24
    ipt 23.246.29.0/24
    ipt 23.246.3.0/24
    ipt 23.246.30.0/24
    ipt 23.246.31.0/24
    ipt 23.246.36.0/24
    ipt 23.246.37.0/24
    ipt 23.246.38.0/24
    ipt 23.246.39.0/24
    ipt 23.246.4.0/24
    ipt 23.246.42.0/24
    ipt 23.246.44.0/24
    ipt 23.246.45.0/24
    ipt 23.246.46.0/24
    ipt 23.246.47.0/24
    ipt 23.246.48.0/24
    ipt 23.246.49.0/24
    ipt 23.246.5.0/24
    ipt 23.246.50.0/24
    ipt 23.246.51.0/24
    ipt 23.246.54.0/24
    ipt 23.246.55.0/24
    ipt 23.246.56.0/24
    ipt 23.246.57.0/24
    ipt 23.246.58.0/24
    ipt 23.246.59.0/24
    ipt 23.246.6.0/24
    ipt 23.246.7.0/24
    ipt 23.246.8.0/24
    ipt 23.246.9.0/24
    ipt 37.77.184.0/21
    ipt 37.77.186.0/24
    ipt 37.77.187.0/24
    ipt 37.77.188.0/24
    ipt 37.77.189.0/24
    ipt 37.77.190.0/24
    ipt 37.77.191.0/24
    ipt 45.57.0.0/17
    ipt 45.57.0.0/24
    ipt 45.57.1.0/24
    ipt 45.57.10.0/23
    ipt 45.57.10.0/24
    ipt 45.57.12.0/24
    ipt 45.57.13.0/24
    ipt 45.57.14.0/24
    ipt 45.57.15.0/24
    ipt 45.57.16.0/24
    ipt 45.57.17.0/24
    ipt 45.57.18.0/24
    ipt 45.57.19.0/24
    ipt 45.57.2.0/24
    ipt 45.57.20.0/24
    ipt 45.57.21.0/24
    ipt 45.57.22.0/24
    ipt 45.57.23.0/24
    ipt 45.57.3.0/24
    ipt 45.57.30.0/24
    ipt 45.57.31.0/24
    ipt 45.57.36.0/24
    ipt 45.57.37.0/24
    ipt 45.57.4.0/24
    ipt 45.57.5.0/24
    ipt 45.57.6.0/24
    ipt 45.57.7.0/24
    ipt 64.120.128.0/17
    ipt 66.197.128.0/17
    
    # Add what appear to be the Amazon Instant Video IP address ranges (obtained by running Wireshark while trying to play Amazon instant video)
    52.94.216.0/24
    54.239.33.0/24
    54.239.35.0/24
    54.239.38.0/24
    54.239.39.0/24
    54.230.196.0/24
    178.236.6.0/24
    176.32.109.0/24
    
    # Add BBC IP addresses (obtained by pinging bbc.co.uk lots and running Wireshark while trying to play in-line videos on the BBC website)
    ipt 212.58.244.0/24
    ipt 212.58.246.0/24
    ipt 178.79.208.0/24
    ipt 178.79.232.0/24
    ipt 23.23.175.175
    
    # create ipset hash table
    ipset -N $IPSET iphash -q
    ipset -F $IPSET
    
    # add firewall rule
    iptables -t mangle -A PREROUTING -m set --set $IPSET dst -j MARK --set-mark $FW_MARK
    
    # start split tunnel
    ip rule add fwmark $FW_MARK table $TID
    
    ) 2>&1 | logger -t "ovpn_split[$$]"
    Finally, to also have the VPN bridge (on port 1194) go around the PIA VPN client, underneath the line above where it says "iptables -t mangle -A PREROUTING -m set --set $IPSET dst -j MARK --set-mark $FW_MARK", I added another line that says:
    Code:
    # add firewall rule for packets generated locally by the house-to-house VPN bridge on port 1194
    iptables -t mangle -A OUTPUT -p udp --sport 1194 -j MARK --set-mark $FW_MARK
    I also added to the bottom of the above, to get the routing for the VPN bridge server to work:
    Code:
    iptables -I FORWARD -i br0 -o tun21 -j ACCEPT
    iptables -I FORWARD -i tun21 -o br0 -j ACCEPT
    iptables -I INPUT -i tun21 -j ACCEPT
    And I seem to have things working. Note that the list of IP addresses for these services may, of course, change. My method was simply:
    1. Run Wireshark on the Ethernet port of a PC.
    2. Try to do whichever thing is being blocked.
    3. Stop Wireshark and take a look at the IP addresses involved, adding any missing blocks to the list by SSHing into the router and adding them manually with the line iptables -t mangle -A PREROUTING -i br0 -d new.ip.address.range/here -j MARK --set-mark 0x88.
    4. Once I have established an updated list I add it to the list in the firewall script and save it.
    So if the script is not sufficient, please follow that procedure to make it sufficient.
     
    Last edited: Jan 21, 2017
  25. hungluu

    hungluu Serious Server Member

    Thanks Rob for the code! It works but there are repeating errors from the log. Below is the error:

    user.notice ovpn_split[2059]: iptables: Resource temporarily unavailable.
    user.notice ovpn_split[4725]: RTNETLINK answers: File exists
    unknown user.notice ovpn_split[4725]: iptables: Invalid argument. Run `dmesg' for more information.
     
  26. Rob Meades

    Rob Meades Network Newbie Member

    I suspect you are using an ARM-based router rather than a MIPS based one, see further up this thread where koitsu says:

    "Asus RT-N66U is MIPS (CPU architecture), on which Tomato uses Linux 2.6.22, using netfilter/iptables 1.3.8.

    ARM-based Tomato routers run Linux 2.6.34, using netfilter/iptables 1.4.x.

    There are many syntactical differences between the two versions of netfilter/iptables. Most of the crap you'll read online is for present-day Linux, which uses iptables 1.4.x."

    It might be that for your router you need to replace --set with --match-set. If you remove the # from the line that has the comment #set -x # uncomment/comment to enable/disable debug mode and then look at the log file you should be able to see exactly which line is upsetting it.

    I believe it is safe to ignore the RTNETLINK answers: File exists lines as that's just to do with some untidiness in the way that the VPN is started/restarted by the router.
     
  27. hungluu

    hungluu Serious Server Member

    You're correct I'm using ARM-based router. it's the RT-AC68U (Tmobile branded) and I flashed it with Tomato Shibby 1.28.0000 -138 K26ARM USB AIO-64K. I forgot to mention that I did read the thread and I did replace the --set with --match-set after pasting them into the Firewall script section. Now it stopped working and if it does, only Netflix on my macbook streams, not the Apple TV 4th gen. On the ATV4, I was able to browse Netflix just fine but it didn't load, nor it show any error saying it detects my behind a proxy or anything like that. I'm a total newbie with this. Been look for this for months but still haven't found the perfect solution for this.
     
  28. Rob Meades

    Rob Meades Network Newbie Member

    Don't worry, I was a newbie at this until a month or two ago. Have you tried SSHing into the router and running the iptables line on its own? You can then remove parameters until you find the one that is offending it.
     
  29. hungluu

    hungluu Serious Server Member

    Tried everything and it stopped working. Have no idea why. I'm just gonna stop trying :(. I'll just route 1 device (ATV4) through the VPN and just watch Netflix on others
     
  30. IANS325

    IANS325 New Member Member

    Hi Cant get this script to work, im in the UK - i have just a Modem ECHO HG612 - NETGEAR R7000, that setup on PPPOE with BT, Plus OpenVPN - IPVANISH, Can someone please help me

    192.168.1.1 0.0.0.0 255.255.255.255 UH 0 0 0 vlan2

    81.171.74.32 172.16.12.11 255.255.255.255 UGH 0 0 0 ppp0

    172.16.12.11 0.0.0.0 255.255.255.255 UH 0 0 0 ppp0

    192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 br0

    172.21.36.0 0.0.0.0 255.255.254.0 U 0 0 0 tun11

    127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 lo

    0.0.0.0 172.21.36.1 128.0.0.0 UG 0 0 0 tun11

    128.0.0.0 172.21.36.1 128.0.0.0 UG 0 0 0 tun11

    0.0.0.0 172.16.12.11 0.0.0.0 UG 0 0 0 ppp0
     
    Last edited: Mar 15, 2017
  31. hungluu

    hungluu Serious Server Member

    what's wrong with it?
     
  32. IANS325

    IANS325 New Member Member

    can't get netflix script to work, I'm connected to modem via r7000 in pppoe, and have openvpn working, not sure to make so that if type the netflix address, it goes straight via, my isp and the reset of the traffic goes through openvpn.


    All i have is hg612(modem) and r7000 (router)
     
  33. eibgrad

    eibgrad Network Guru Member

    Folks, as I said on the dd-wrt forum, and will repeat here, even the use of ipset is not a guarantee this will work. ipset assumes that all references used by Netflix are based on domain names. But what if that's not the case?! What if they use a domain name to get things started, then use explicit IPs to manage the remaining links?! Heck, if I was Netflix and wanted to thwart such efforts, that's exactly what I'd do. And I mix it up further by constantly changing those explicit IPs. Any company that's going to go through this much effort to prevent the use of VPNs/proxies is surely capable of coming up w/ obstacles like these.

    So I'm not the least bit surprised if this doesn't work, or only works sometimes. If Netflix is motivated enough, they can quite easily create one obstacle after another.

    Personally, I think the better solution is to use policy based routing. Unfortunately the policy based routing feature in the GUI sucks (lots of bugs). But using a quality policy based routing script, you could either make an exception for the device running Netflix so all its traffic was using the WAN, *or*, have the default gateway be the WAN and make exceptions to route other things over the VPN.

    As it happens, I've been working on a pair of policy based routing scripts. One I call "basic", the other "advanced". Each has advantages and disadvantages.

    Basic Policy Based Routing Script
    Advanced Policy Based Routing Script

    I had originally considered trying to fix all the bugs in the GUI w/ scripting, but it proved to be more hassle than it was worth. So I abandoned it in favor of my own scripts, where I have complete control and can make sure everything works correctly.

    Now realize, these are WIP (works in progress). I'm updating them ALL THE TIME! In fact, I just got to a point today that the advanced script works reasonably well on both MIPS and ARM processors (there are some differences when it comes to ipset, which the advanced script supports).

    I wasn't planning on releasing them at this time, but given I've being peppered lately for solutions to problems only addressable via policy based routing, and the fact I need some beta testers anyway to hammer these scripts, I figured it was time.

    Even w/ these scripts, you have to change your strategy. Rather than letting the default gateway change to the VPN and trying to make exceptions for Netflix over the WAN, maintain the WAN as the default gateway (by using the route-noexec directive) and be selective about what uses the VPN (which obviously shouldn't including Netflix). I think you're much more likely to have success under those conditions.
     
    Last edited: Mar 16, 2017
    mugs07 and Jason Meudt like this.
  34. Jason Meudt

    Jason Meudt Reformed Router Member

  35. eibgrad

    eibgrad Network Guru Member

    Admittedly there have been way too many scripts plastered over this forum wrt policy based routing. And I'm to blame as much as anyone else. And part of the reason is that everyone has different needs. And so rather than trying to take yet another old script and adapt it to the next situation, and perpetuating the current paradigm, this is my attempt to come up w/ scripts that will handle 99% of situations. I suppose we'll never have any script that covers EVERY possible situation, but I personally just don't want to deal anymore w/ all these various scripts (mine or anyone else's) written way back when. I want to point ppl to these scripts, get them working for 99% of users, then deal w/ the remaining 1% through minor modifications (or enhancements, if the changes would benefit others). But I can't get there if I don't have ppl to try them out and provide feedback. Otherwise, we're going to keep revisiting this topic for years to come, dealing w/ each situation on a case by case basis. I'm trying to put an end to all that. Perhaps even get these scripts pinned permanently to the forum!

    Case and point. My scripts now handle the routing for the route-noexec directive. What most ppl don't realize is that when you use route-nopull, this not only prevents the default gateway from being changed to the VPN, but it also blocks the VPN's DNS servers! And therefore you have the possibility of a DNS leak. The solution is to use route-noexec, which prevents the change in the default gateway, but without rejecting the VPN provider's DNS servers. But that introduces another problem. When you use route-noexec, OpenVPN requires the client to handle any other routes you specify in Custom Configuration.

    Code:
    route 121.34.5.9 255.255.255.255 net_gateway
    route 188.234.15.91 255.255.255.255 vpn_gateway
    But the GUI doesn't do this. So these routes get IGNORED!

    My scripts handle all this stuff.

    You can even (optionally) bind the VPN provider's DNS servers to the tunnel. So let's say you use route-noexec and the VPN provider pushes a google DNS server (8.8.4.4). That would normally be accessed over the WAN. But by binding 8.8.4.4 to the VPN, it prevents the ISP from snooping on your DNS queries. IOW, this extends the DNS leak prevention to more than just not using the ISP's DNS servers. The ISP can't even snoop your DNS queries, or worse, alter them.

    There are other features as well, such as built-in ipset support. And it's aware of the different syntax requirements for ipset since iptables 1.4.4.

    Even remote access is addressed. The script automatically enables remote access over the NON default gateway network interface without you having to implement any rules. The mere act of installing the script solves the problem. And therefore you don't even have to understand how it works to use it.

    That's what I mean by having scripts that we can rely on to cover as many of the policy based routing issues as possible, rather than dealing w/ each new case as if we had to reinvent the wheel.

    Again, these are WIP, so expect continuous updates for at least the next few weeks. And if I get feedback, I can fix things, and even add enhancements if ppl suggest good ideas.
     
    Last edited: Mar 20, 2017
  36. IANS325

    IANS325 New Member Member

    It would handy with this scripts to visual show some screen shots of where you are pulling the information from like go network screen write ip address, so us newbies could get better understanding
     
  37. hungluu

    hungluu Serious Server Member

    So if I use ipset directive in dnsmasq, do I need to change anything in the rules section? And any rules in the rule section is to bypass the VPN or to route it over VPN
     
  38. eibgrad

    eibgrad Network Guru Member

    No. It's all automatic. Any domains specified in dnsmasq for the ipset directive are rerouted according to the description below.

    Unlike the Routing Policy tab which only works one way (you use route-nopull (or route-noexec) to prevent the VPN from changing the default gateway from the WAN/ISP to the VPN, then use rules to be selective about what uses the VPN), the script is able to handle the default gateway being the WAN or VPN, then interpret the rules as exceptions to that default gateway.

    IOW, if the OpenVPN connection is configured to change the default gateway to the VPN, the rules are treated as exceptions to be rerouted to the WAN/ISP. OTOH, if the OpenVPN connection is configured to leave the default gateway as the WAN/ISP, the rules are treated as exceptions to be rerouted to the VPN.

    The goal here is to bury as much of the complexity for policy based routing into the bowels of the script (much like the GUI). By merely installing the script, it will fix a number of bugs found in the GUI's implementation of policy based routing. In some instances, you may not need any rules at all. The ones provided are just examples.
     
    Last edited: Mar 20, 2017
  39. cloneman

    cloneman Addicted to LI Member

    For those of us with entware, there is an iptables 1.4.x package available for MIPS.

    Question: What problems does it introduce to have 2 different copies of iptables installed? Will some internal processes use /sbin/iptables and others use /opt/sbin/iptables ? Will this corrupt the iptables file if the older one can't read the entries of the newer one?

    There's also the ipset-dns package, no idea if that can help anyone here or if the builtin support in dnsmasq is the same.
     
  40. hungluu

    hungluu Serious Server Member

    Tried it then got the error "could not execute external program".
     
  41. eibgrad

    eibgrad Network Guru Member

    Most likely you downloaded the file to a Windows file editor, made your changes, saved the file as a Windows file, then copied it over to the router. But Linux can't read Windows files! Windows and Linux use different EOL (end of line) chars. That's why it's important to always use a Linux compatible editor (notepad++, UltraEdit32, etc.) and save it on Windows as a Linux file before copying to the router. Or else copy the file directly to Linux and use the vi editor.

    A sure sign this is the problem is if you bring up the file on the router using the vi editor and see ^M at the end of every line. Or if you cat the file (cat <filename>), it’s doubled spaced. You can correct it on the router using the following command.

    Code:
    sed -i.bak 's/\r//' <file-name>
     
  42. hungluu

    hungluu Serious Server Member

    i'm practically new to this, but what does option -s and -m do? If I put this
    Code:
    add_rule -s 192.168.1.10
    Does it mean the device with the mentioned IP above will be routed outside the VPN (or through WAN)
    Also, if I want to route the specific port over the VPN for the specific IP, I'll just put this
    Code:
    add_rule -p tcp -s 192.168.1.10 --dports 32400
    Correct?
     
  43. eibgrad

    eibgrad Network Guru Member

    Much of this needs to be documented. But I wanted to get these scripts out and working asap.

    The rules in the advanced script are just firewall rules. So the syntax and meaning is consistent w/ iptables. The more you understand iptables, the more you'll understand the rules (and vice versa).

    The -m option is for loading netfilter modules/extensions. iptables natively understands only very basic things like -s for source ip, -d for destination ip, -p for protocol, etc. But other less used options have to be loaded using the -m option. That module/extension is then responsible for handling the option and reporting back to iptables if it did or did NOT match.

    You could even write your own modules/extensions and add them to iptables!

    For example, if I want to reference a MAC address, I have to reference the mac module/extension.

    -m mac --mac-source 00:11:22:33:44:55

    I can't just say "--mac-source 00:11:22:33:44:55" because iptables needs to know which module/extension is going to handle this option.

    Sometimes you also have to load the object libraries for these modules/extensions as well. They're not always loaded by default, to save space.

    How the rules are interpreted depend on how you configure the OpenVPN client.

    By default, all OpenVPN providers change the default gateway on the OpenVPN client to the VPN. And if you don't take steps to change that, then the rules will route those packets matching your specified criteria over the WAN.

    OTOH, if you *do* take steps to prevent the OpenVPN server from changing the default gateway to the VPN (e.g., adding the route-noexec directive to the custom configuration field, thus leaving the default gateway as the WAN/ISP), then the rules will route those packets matching your specified criteria over the VPN.

    IOW, the script is totally flexible. YOU always decide what will be the default gateway. The script then detects how you configured it and treats all your rules as exceptions that should use the other gateway.

    As far as the following rule specifically ...

    Code:
    add_rule -p tcp -s 192.168.1.10 --dports 32400
    You need to verify your syntax w/ the iptables and netfilter documentation. IIRC, it's --dport (singular), not --dports (multiple).

    Once corrected, it will reroute any packet meeting that criteria. I can say w/ certainty whether that will be the WAN or VPN since I don't know how you configured the default gateway. If we assume you did nothing to prevent the OpenVPN server from changing it to the VPN, then that rule will route that traffic over the WAN.
     
  44. hungluu

    hungluu Serious Server Member

    Sorry, I copied and pasted the sample code from your raw code but in my config I did put --dport, not --dports.
    So I got the script running and working as expected. However, when I use the
    Code:
    route-noexec
    , I noticed that the rules in the file will be routed through VPN, and if I did not use
    Code:
    route-noexec
    , whatever the rules are in the file will be routed through WAN. However, if I use
    Code:
    route-noexec
    , I set the ROUTE_DNS_THRU_VPN to 8.8.4.4, then I tested with ipleak.net and I saw that the IP i set in the rule is routed over VPN (which is correct), but then it leaks my ISP DNS, instead of showing the Google's DNS servers.

    Also, I wanted to open the port 32400 as I'm running a Plex server on my desktop, but this doesn't work
    Code:
    add_rule -p udp -s 192.168.1.199 --dport 32400
     
  45. hungluu

    hungluu Serious Server Member

    In regards to open and forward the port 32400 on my end, this is the code that I was given to put in the Administration/Scripts/Firewall
    Code:
    iptables -I FORWARD -i tun11 -p udp -d destIP --dport port -j ACCEPT
    iptables -I FORWARD -i tun11 -p tcp -d destIP --dport port -j ACCEPT
    iptables -t nat -I PREROUTING -i tun11 -p tcp --dport port -j DNAT --to-destination destIP
    iptables -t nat -I PREROUTING -i tun11 -p udp --dport port -j DNAT --to-destination destIP
    Now how can I convert this into your rule, @eibgrad, because seemse like the rule I added didn't work
     
  46. eibgrad

    eibgrad Network Guru Member

    Whether you port forward over the WAN (using the GUI) or port forward over the VPN (manually using those rules), it has nothing to do w/ the script. Port forwarding is port forwarding. The script is not about enabling/configuring port forwarding.

    What the script does is make it possible to access your port forwards over a given network interface (WAN or VPN depending on how you configured the OpenVPN client), even when that network interface is NOT the default gateway. It fixes that one narrow problem wrt port forwarding. And it does it by merely installing the script. No rules necessary.

    Now given the above, it should be obvious that if you configure remote access over the WAN and the WAN is the default gateway, OR, your configure remote access over the VPN and the VPN is the default gateway, you don't even need the script! At least not if that is the only reason you installed it (it solves other problems as well).

    IOW, port forwarding and remote access *only* becomes an issue when the incoming and outgoing network interfaces are different. When they're the same, there's no issue.

    When it comes to dns leaks, you can't use the typical dns leak testing websites for this purpose. Those utilities/tools assume the client is configured directly w/ the ISP's DNS server. But in the case of the router, your client is using the DNS server of the router (DNSMasq), at least if you're use the default config. Therefore, there's no way for these utilities/tools to know what DNS servers you're using. You have to take a different approach, which is discussed in detail in the following link.

    http://linksysinfo.org/index.php?threads/dns-leak.73296/
     
  47. hungluu

    hungluu Serious Server Member

    My bad there eibgrad, somehow the Plex server changed the port and it wasn't 32400 so that's why it didn't work. Now it did. Does your script work on Asuswrt-Merlin by any chance or it just works on Shibby v132?
     
  48. eibgrad

    eibgrad Network Guru Member

    I've tried to write it as generically as possible, so it *might* run on other platforms. But given these scripts are always running on routers w/ limited resources (memory, flash, cpu), you can't count on all the tools you might want or hope to have available to solve a problem. I'm extremely careful to not venture too far outside what I've come to expect is available and compatible within the dd-wrt and tomato platforms. In the case of the advanced script, I know tomato has long supported ipset, while dd-wrt has traditionally not. And so I have had to make adjustments. And who knows, maybe other firmware will expose other compatibility problems. Only testing will prove one way or the other. Since I'm not a user of Merlin, I have no clue.

    Another complication is how each platform implements OpenVPN. In the case of tomato, it doesn't use the OpenVPN scripting engine, so as a script developer, I'm free to use that scripting engine any way I see fit, without concern for the GUI. But that's not the case w/ dd-wrt. dd-wrt *does* use the scripting engine, so my scripts have to be written to take that into consideration. I actually had to build a "monitor" script for dd-wrt so I could detect when the OpenVPN client started, kill it, and restart it to handle my events in the OpenVPN scripting engine as well as dd-wrt's. It's a bit awkward, it's a bit messy, but it works.

    So I have my doubts when it comes to other platforms.

    As far as Shibby 132, I don't know exactly where the cut-off is for support since I run Shibby v132, and when I tried to run Shibby v138, I ran into the following bug.

    http://linksysinfo.org/index.php?th...ior-of-iproute2-in-shibby-arm-multiwan.72495/

    According to the latest update by the OP, that bug seems limited to v137 and v138. So maybe v133 through v136 might be fine. I don't know. I didn't test them all. That's why I need feedback so we can build a compatibility/incompatibility database. I only have the resources here to test a few routers.
     
    Last edited: Mar 29, 2017
  49. hungluu

    hungluu Serious Server Member

    I only have 1 router which is the one I currently run in my house so I'm not sure if I want to go through all that flashing process to flash it to Merlin and test it :(
     
  50. Jason Meudt

    Jason Meudt Reformed Router Member

    Copied over the script, added the openvpn config, and got a warning in the log...

    TomatoUSB daemon.warn openvpn[4253]: WARNING: Failed running command (--route-up): could not execute external program

    I set up the symbolic links and made the script +x...

    Running Tomato Firmware v1.28.9008 Toastman-ARM K26ARM USB VPN-64K
     
  51. eibgrad

    eibgrad Network Guru Member

    Another common problem; ppl download the script to a Windows editor, make their changes, save it as a Windows file, then upload it. But Linux can't read Windows files (the EOL chars are different). You need to use a Linux compatible editor (e.g., notepad++) and save it as a Linux file.

    The telltale sign this has happened is if you get double-spacing when you cat the file on the router, or bring it up in the vi editor and it has ^M at the end of every line.

    As a quick fix, you can convert the Windows file to Linux using the following.

    Code:
    sed -i.bak 's/\r//' <file-name>
     
    Jason Meudt likes this.
  52. Jason Meudt

    Jason Meudt Reformed Router Member

    Lol... Did that... I use notepad++ all the time. That WAS the issue however! Running the file through sed did the trick.

    As an aside, each port you wish to re-route in the rules section must be on it's own line. Merging them into one line does not work based on the logs/script useage... eg; 80,32400,8181 will not work as a single provided line (multiple ports...).
     
  53. eibgrad

    eibgrad Network Guru Member

    That's why I placed them on separate lines. The default option (--dport) only supports a single port (80), or range of ports (3000:3100). In order to support both on one line, you need to use the multiport extension.

    Code:
    add_rule -p tcp -s 192.168.1.112 -m multiport --dports 80,443,3000:3100
    Problem is, not all builds (dd-wrt or tomato) support multiport. And even when it does, depending on the iptables version, it may not necessarily support it in the mangle table! Some extensions will only work in the nat/filter table. You may also have to modprobe the relevant multiport module (xt_multiport or ipt_multiport).

    Because of all these complications, I purposely did NOT go beyond the most basic iptables rules to maximize compatibility. But if you want use more advanced options and fancier syntax, that's up to you. Just make sure you dump iptables afterwards and verify it worked. In fact, don't assume that *any* firewall rules worked w/o verification.

    As I've told others, policy based routing is complex. And there are a lot of small details you have to consider. There are usually good reasons I have things setup the way I do. A lot of the things that can go wrong have been quietly addressed in the script. And that's why you also have to be careful if you start messing w/ it beyond just adding/modifying rules. Even the order of operations can make a difference. In this last update, I had to relocate the execution of add_rules so it occurred before the routing tables were updated, lest domain names fail to resolve in those same rules.

    And there are more updates coming (recently found another more obscure problem).
     
  54. Jason Meudt

    Jason Meudt Reformed Router Member

    While I have not seen the need for any features or changes, is 0.1.7 from 09-apr-2017 still your most recent?

    https://pastebin.com/GMUbEtGj
     
  55. cfharp

    cfharp New Member Member

    Jason - I tried using one of your scripts (pastebin McjdEg8q -- too new, no links), and while it does seem to get Plex Remote Access working (this was my main goal), it doesn't seem to honor the ipset domains from Dnsmasq. When I check my IP address against canyouseeme and google, both come back with my VPN IP. But then I saw in one of your posts that you switched to eibgrad's script, so I thought I would look into that.

    And on that topic, can anyone help me get one of these new scripts set up? I see the instructions say to copy the script to jffs, but that's unfamiliar territory for me. Does that mean I don't use the Scripts/Firewall setting anymore? Any help would be much appreciated.

    Side notes: I'm running Tomato v1.28 by shibby (1.28.0000 MIPSR2-123 K26 USB AIO), PIA for VPN, and Plex Remote Access is my primary goal, but it would also be nice to have some of these other exceptions in place for testing IP/ports.
     
  56. Bunsen

    Bunsen Network Newbie Member

    Not sure if this is still an issue for you - also not sure if i can help - but I am using eibgrad's script that Jason mentioned above, so at least i can try.
    To answer your first question: "Yes" - eibgrad's script should be added to jffs, not in the Scripts/Firewall setting.

    There are several ways to get the script to your router but if you can ssh in, I think it makes the whole process easier. [as well as setting you up for additional "learning opportunities", or working around gui limitations you might encounter in the future] - Reply here if you still need help with this part, and I can help to spell it out.

    Second: Can you clarify what it means when you say: "it doesn't seem to honor the ipset domains from Dnsmasq"?
    Could you show the line(s) you're adding to the dnsmasq config?

    Finally - does anyone know what happened to eibgrad? His scripts, explanations, and networking knowledge are top-shelf, but I haven't seen any acct activity in any forum from him since April [I believe he was moving, but then never came back online].
     
  57. Jason Meudt

    Jason Meudt Reformed Router Member

    I am now running XWRT (Asuswrt-Merlin for the Netgear R7000) and it appears to be running ok...

    Glad to see you back eibgrad!
     
  58. Jason Meudt

    Jason Meudt Reformed Router Member


    @eibgrad I am now having some difficulties with XWRT and your script... It appears that none of the DNSMASQ entries, now placed within dnsmasq.conf.add since there is no custom configuration GUI entry in XWRT, are being routed outside of the VPN. While everything was working fine, the most recent update of the firmware may have done something...
     
  59. eibgrad

    eibgrad Network Guru Member

    I don't have an R7000, and I don't run Merlin. And even if I did, I'm not sure how well I could debug that particular firmware. I've never used it, so I don't know what kind of low level access it offers. Is it like dd-wrt or tomato?

    Only spare router I have that comes close would be a ASUS RT-AC68U. Would that work?
     
  60. eibgrad

    eibgrad Network Guru Member

    Ok, so here's where I stand.

    I grabbed one of my spare ASUS RT-AC68U routers and installed Merlin. Had a heck of time installing it until I decided to go back to the oldest available firmware (378.55).

    My only interest at the moment was getting the router configured generally, and the OpenVPN client specifically. I tried both the basic and advanced scripts, and everything seemed to run just fine. I didn't actually test it w/ a client, but I just wanted to make sure the scripts ran w/o errors, the data structures were built, etc. And when the OpenVPN client came down, all those data structures returned to normal.

    Frankly, when I first noticed that Merlin uses the route-up and route-pre-down directives for itself, I was a bit concerned, since many times the developer will use these for establishing firewall rules. But as I looked closer, I could see the only purposes was for implementing the GUI's own policy based routing. So my scripts simply overrode the GUI's.

    Now I tried the advanced script and the DNSMasq ipset functions. If you can't modify DNSMasq through the GUI, I'd be curious how you managed to change it. For now, I coped the original /etc/dnsmasq.conf to /etc/dnsmasq2.conf and added my ipset directives.

    etc/dnsmasq2.conf
    Code:
    pid-file=/var/run/dnsmasq.pid
    user=nobody
    bind-dynamic
    interface=br0
    interface=ppp1*
    no-dhcp-interface=ppp1*
    resolv-file=/tmp/resolv.conf
    servers-file=/tmp/resolv.dnsmasq
    no-poll
    no-negcache
    cache-size=1500
    min-port=4096
    dhcp-range=lan,192.168.1.2,192.168.1.254,255.255.255.0,86400s
    dhcp-option=lan,3,192.168.1.1
    dhcp-option=lan,252,"\n"
    dhcp-authoritative
    ipset=/ipchicken.com/netflix.com/ovpn_split
    ipset=/google.com/cnet.com/gov/ovpn_split
    Then I killed the current dnsmasq process and restarted dnsmasq w/ the new config file.

    Code:
    killall dnsmasq && sleep 3; dnsmasq --log-async -C /etc/dnsmasq2.conf
    And got the following error.

    Code:
    dnsmasq: recompile with HAVE_IPSET defined to enable ipset directives at line 17 of /etc/dnsmasq2.conf
    That tells me that ipset has NOT be compiled into DNSMasq. And so I don't know how this particular feature ever worked unless you're telling me specific Merlin builds have this ipset feature, while other do not.

    I did notice from the syslog that the script did load the ipset modules.

    Code:
    Feb 25 21:18:22 route-up[1768]: + configure_ipset
    Feb 25 21:18:22 route-up[1768]: + modprobe ip_set
    Feb 25 21:18:22 route-up[1768]: + modprobe ip_set_hash_ip
    Feb 25 21:18:22 kernel: ip_set version 4 loaded
    Feb 25 21:18:22 route-up[1768]: + modprobe ipt_set
    Feb 25 21:18:22 route-up[1768]: + MATCH_SET=--set
    So that's not the problem.
     
  61. Jason Meudt

    Jason Meudt Reformed Router Member

    I'm currently using 380.69_2...


    I added them through /jffs/configs/dnsmasq.conf.add

    https://github.com/RMerl/asuswrt-merlin/wiki/Custom-config-files


    Hmmm... I know it 'worked'... :)

    Looking through the versions, DNSMASQ was updated in 380.68_4, so I may have 'not noted' that the routes were failing...

    https://asuswrt.lostrealm.ca/changelog
     
  62. Sean B.

    Sean B. LI Guru Member

    dnsmasq -v will give you all compile time options used:

    Code:
    root@Storage:/tmp/home/root# dnsmasq -v
    Dnsmasq version 2.78  Copyright (c) 2000-2017 Simon Kelley
    Compile time options: IPv6 GNU-getopt no-RTC no-DBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset Tomato-helper auth DNSSEC loop-detect inotify
    
    This software comes with ABSOLUTELY NO WARRANTY.
    Dnsmasq is free software, and you are welcome to redistribute it
    under the terms of the GNU General Public License, version 2 or 3.
    root@Storage:/tmp/home/root#
     
    eibgrad likes this.
  63. eibgrad

    eibgrad Network Guru Member

    Code:
    admin@RT-AC68U-DEC8:/tmp/home/root# dnsmasq -v
    Dnsmasq version 2.73rc9  Copyright (c) 2000-2015 Simon Kelley
    Compile time options: IPv6 GNU-getopt no-RTC no-DBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP no-conntrack no-ipset no-auth no-DNSSEC no-loop-detect no-inotify
    
    This software comes with ABSOLUTELY NO WARRANTY.
    Dnsmasq is free software, and you are welcome to redistribute it
    under the terms of the GNU General Public License, version 2 or 3.
    admin@RT-AC68U-DEC8:/tmp/home/root#
    No ipset in my Merlin build 378.55. And I can't past this build no matter what I do. Neither using the GUI, nor tftp. The GUI always reports "File format or path is invalid". tftp fails as well, I assume for the same reason. Tried numerous builds from SourceForge and the mirror, nothing helps.

    What's the secret?

    P.S. It's been over year since the last update to that script, so I decided in the meantime to make some updates, including simplifying a few things, and a few other minor fixes. I'd like to test it out on Merlin w/ a working ipset feature, but again, I can't get anything installed except 378.55. I'm just sort of stuck at the moment.
     
    Last edited: Feb 27, 2018
  64. Monk E. Boy

    Monk E. Boy Network Guru Member

    Did you put FW_RT_AC68U_30043763626.trx onto the AC68 as the first flash?

    That version changes the rootfs partition from 32M to 64M. At that point you can go up to later firmwares which expect rootfs to be 64M.

    Pretty sure you can go back to that version now to do the same rootfs expansion.

    That's an ASUS OEM firmware, BTW.
     
    eibgrad likes this.
  65. eibgrad

    eibgrad Network Guru Member

    Thanks so much. That was it. I had read somewhere that installing v378.55 would expand the rootfs, and when it was the only firmware I was able to successfully install, I had just assume it was 64M. Then after you mentioned it, I went back and checked, and sure enough it was 32MB. Ugg.

    Anyway, finally have 384.3 on the router and can proceed. Again, thanks.
     
  66. eibgrad

    eibgrad Network Guru Member

    Good news. With the help of Monk E. Boy, I finally got the latest Merlin build installed (384.3). And I just tried my latest updated script and it seems to be working fine, including ipset. This is only a preliminary report. I need to do some more testing on Merlin, then on tomato (MIPS and ARM) before updating pastebin w/ the next release. Whether the old/current version has a problem, I don't know. But given all the problems I had getting Merlin updated, I decided to spend some time bringing the script up to date and that's the only one I tested.

    P.S. Is it just me, or does Merlin seem to mysteriously flush out /jffs from time to time? I can see on the Admin page that it says NOT to refresh jffs on reboot. But from time to time, I make my changes, then reboot, poof, they're gone. I've noticed too that sometimes when I Save and then Apply my certs and keys, they disappear as well on the next boot. All very annoying.
     
  67. eibgrad

    eibgrad Network Guru Member

    Just updated both the tomato basic and advanced pbr scripts on my PasteBin account.

    https://pastebin.com/u/eibgrad

    Did a bunch of testing w/ ARM and MIPS, seemed to work fine. Specifically tested w/ ASUS RT-AC68U using the latest Merlin build 384.3.

    I do a lot more checking to make sure the installed version of DNSMasq supports ipset, that the ipset modules are installed/installable, that I install the correct versions, detect which version of iptables is installed (how you configure ipset varies by version), etc. All these changes to ipset and iptables support of ipset over the years, combined w/ all these tomato variants, makes it tough to get right. But it should be significantly better than the previous release. And *maybe* that was part of the prior problem. Let's hope so anyway. Be sure to check the syslog if you have problems and look for error messages.

    Please report back, whether good or bad results.

    Btw, having now played w/ Merlin a bit, I couldn't find a lick of difference between it and tomato when it comes to these scripts. Only differences seem to lie outside the script, such as how you use /jffs/configs to modify DNSMasq.

    Thinking about creating a new Merlin-specific script if only to document those differences. Sort of hate doing that though since the code is identical in all other respects. Or may reference a second document for Merlin users. Haven't decided as yet.

    One other thing. I noticed Merlin writes to the syslog to JFFS! Did a little more research and found some ppl claiming ASUS has optimized that a bit by initially writing to /tmp, then periodically applying the differences to JFFS. But still sounds like a *really* bad idea given flash has a limited number of writes. I was pretty happy w/ Merlin (despite a few snags along the way) until I saw this issue w/ the syslog. That would be one good reason *I* would avoid, or if forced to, would certainly disable the syslog.
     
  68. koitsu

    koitsu Network Guru Member

    If someone was to make a separate thread discussing Merlin vs. TomatoUSB, I would be happy to paste my notes from the time I tried it (specifically RT-N66U_380.64_2.trx, which is quite old by now, but still). If I was to do that, know that my notes are shorthand and were written "as I encountered stuff", so I didn't do full digging, Googling, or anything else -- they're along the lines of "why is this happening?", "what is this daemon?", "why does X do Y?" "what is this behaviour/bug?". Then again, this isn't a Merlin support forum, but they are things as a Tomato user/dev that made me go "?!?!"
     
  69. Jason Meudt

    Jason Meudt Reformed Router Member

    Sorry for the late replay... Business trip.

    In any case, since I just got back, I pasted in your version and noted two warnings in my syslog:

    Code:
    Mar  2 17:22:30 openvpn[6026]: /jffs/route-pre-down tun11 1500 1558 10.29.10.6 10.29.10.5 init
    Mar  2 17:22:31 openvpn[6026]: WARNING: Failed running command (--up/--down): could not execute external program
    Mar  2 17:22:31 openvpn[6026]: Exiting due to fatal error
    
    and

    Code:
    Mar  2 17:22:55 openvpn[8092]: WARNING: Failed running command (--route-up): could not execute external program
    Mar  2 17:22:55 openvpn[8092]: Initialization Sequence Completed
    
     
  70. koitsu

    koitsu Network Guru Member

    Try doing chmod 755 on each of the route-up and route-down scripts.
     
  71. Jason Meudt

    Jason Meudt Reformed Router Member

    Those two are just symbolic links which point to the actual script, but they are/were set at 777 previously.

    Still the same thing even with an octal of 755...
     
  72. Sean B.

    Sean B. LI Guru Member

    Check for any spacing or CR/LF issues caused from copy/pasting the file.
     
  73. Jason Meudt

    Jason Meudt Reformed Router Member

    Nope... To be sure I ran it through sed:

    Code:
    sed -i.bak 's/\r//' <file-name>
    I still get the error on vpn startup:

    Code:
    openvpn[3201]: WARNING: Failed running command (--route-up): could not execute external program
    and on shut down:

    Code:
    openvpn[2832]: WARNING: Failed running command (--up/--down): could not execute external program
    Looking through the steps, the files in questions are symbolic links to another file... That file is an executable (+x) script file, so it looks good, but...
     
  74. Sean B.

    Sean B. LI Guru Member

    Is the process trying to run them under UID/GID root? Or does it drop privileges and run under UID/GID "nobody" ? If the latter, chown the links/files over to nobody and test.
     
  75. koitsu

    koitsu Network Guru Member

    If the scripts were 777'd already, then 755 will not fix/change anything other than removing group and world write from the files. What I was focused on was whether or not the scripts had the user/group/world execute bit set (which they do), since default umask (0022) tends to result in files that are 644 (no execute bit set).

    Do the scripts contain #!/bin/sh for their first line? If not, that may be the problem; the kernel actually reads the first line of a file to look for a hashbang line to determine if it's a script vs. an actual ELF binary.

    Other possibilties include:

    1. Permissions of every directory in the entire path. For example, if the script being called is /foo/bar/blat/up-script then one would need to check the permissions of all the path portions by doing: ls -ld / /foo /foo/bar /foo/bar/blat /foo/bar/blat/up-script

    2. Filesystem mount options, specifically ensuring the filesystem is not mounted noexec. You can check this by running the mount command.
     
  76. Jason Meudt

    Jason Meudt Reformed Router Member

    I am not sure... The permissions for the entire path are all the same along the entire path, but as far as who/how the vpn client is attempting to run them, I have no idea...
     
  77. Jason Meudt

    Jason Meudt Reformed Router Member

    Yes... The script has a #!/bin/sh as the first line....

    The permissions are the same along the entire path...

    The FS is not mounted noexec...

    I know that this entire process is somehow going to end in my own stupidity, but I'll be danged if I know!
     
  78. koitsu

    koitsu Network Guru Member

    There are a couple ways to find out what's going on. Give me a little bit to dig through code and I can probably find the answer. I doubt it's your own stupidity though. :)
     
  79. koitsu

    koitsu Network Guru Member

    I've reviewed the OpenVPN code in Toastman-ARM7 branch -- https://github.com/tomatofirmware/tomato/tree/Toastman-ARM7/release/src-rt-6.x.4708/router/openvpn -- but this is extremely tedious to go through. I actually did a write-up of it (it's hundreds upon hundreds of lines long) and felt that would be boring to go over unless you're a C programmer. Here are the code references, in order of execution:

    openvpn_run_script(): https://github.com/tomatofirmware/t....x.4708/router/openvpn/src/openvpn/misc.h#L96
    openvpn_execve_check(): https://github.com/tomatofirmware/t...x.4708/router/openvpn/src/openvpn/misc.c#L249
    openvpn_execve(): https://github.com/tomatofirmware/t...x.4708/router/openvpn/src/openvpn/misc.c#L282
    execve(): https://linux.die.net/man/2/execve
    waitpid(): https://linux.die.net/man/2/waitpid
    platform_system_ok(): https://github.com/tomatofirmware/t...08/router/openvpn/src/openvpn/platform.c#L207
    msg(): probably a vsprintf() equivalent, so I didn't look it up -- didn't need to
    system_error_message(): https://github.com/tomatofirmware/t...x.4708/router/openvpn/src/openvpn/misc.c#L219 (note line 240 here vs. line 309 in the same file / openvpn_execve()).

    I actually understand about 90% of the code, but I can't cover all runtime cases. I really need to see what's happening in real-time with regards to the result of execve() of the script itself.

    My gut feeling is that your up/down scripts are actually returning a non-zero exit code, and this is "trickling down" all the way back into OpenVPN (this is normal/correct!) to cause an error condition.

    Unless the scripts are deliberately calling exit 0 in them, then they will return the exit code of the last command run in them. If this is true, the solution is not to just add "exit 0" to the bottom of your scripts -- the solution is to find out what is causing a non-zero exit code within the scripts; usually that indicates a command failed (UNIX commands are supposed to exit with a zero code if things are OK, non-zero otherwise).

    Another possibility is that OpenVPN is running within some sort of chroot'd environment, in which case the pathnames to the scripts are within the chroot and not full/literal system paths. OpenVPN does have chroot support (either through config file, or deliberately running it with the --chroot dir flag). In this case, /jffs/something would be more like /chroot/dir/jffs/something.

    The next thing to do, for realtime debugging, would be to install Entware-ng and the strace package. This can shed light on what is happening at the syscall level (so I can see results of execve(), waitpid(), open(), etc.), but you have to run OpenVPN manually from the command-line (with all the correct flags/etc. that TomatoUSB would) under strace by doing something like: strace -f -s 256 -tt -o /tmp/strace.txt -- openvpn --flags ... and then provide me the content of /tmp/strace.txt. WARNING: strace.txt will be very large and WILL/COULD contain VERY sensitive information about your system, IPs, and parts of your private key! DO NOT paste or attach this file on the forum publicly!!! Send it to me in a PM. I have always respected people's private data; I treat things like this professionally.
     
  80. Jason Meudt

    Jason Meudt Reformed Router Member

    So... When all else fails, look to yourself for the answer/problem! I previously ran the file through SED and 'assumed' all went well...

    It didn't... To be sure (THIS time!), I ran it through DOS2UNIX and "voila!" we have usage!

    Your confidence, although loved, was misplaced... Tenacity wins the day however!


    To everyone who has helped, I thank you!
     
  81. Sean B.

    Sean B. LI Guru Member

    Ah, so it was spacing or CR/LF? Glad you found it.
     
    Jason Meudt likes this.
  82. eibgrad

    eibgrad Network Guru Member

    Sorry for not responding sooner, but I was tied up all weekend w/ business and relatives. I had ZERO time for anything else.

    This is a constant source of problems w/ providing scripts. The process is so error prone (this issue w/ DOS<->Linux conversions in particular), that I don't respond immediately anyway assuming the OP will eventually discover their own problems.

    In the future, you could just read the file directly into the router from PasteBin using the following script and avoid these headaches.

    Code:
    file="/jffs/tomato-ovpn-split-advanced.sh"
    rm -f $file
    curl -kLs https://pastebin.com/raw.php?i=GMUbEtGj | tr -d '\r' > $file
    chmod +x $file
    ln -sf $file /jffs/route-up
    ln -sf $file /jffs/route-pre-down
    At that point, you can either edit the file directly on the router w/ the vi editor (it pays to learn a few commands), or as a remote file using something like WinSCP (SCP or FTP) w/ its built-in editor. Far less chance of errors.

    The most error prone technique seems to be downloading it from PasteBin to Windows, editing it locally, then uploading.
     
    Last edited: Mar 4, 2018
  83. koitsu

    koitsu Network Guru Member

    Yep, that would explain the problem. The hashbang line was being interpreted by the kernel to try and run the command /bin/sh<0x0d> and there is no such file. The result would be exit code 127. Just like if you had done something like this:

    Code:
    root@gw:/tmp/home/root# /bin/this-doesnt-exist
    -sh: /bin/this-doesnt-exist: not found
    root@gw:/tmp/home/root# echo $?
    127
    
    I can confirm that Busybox sed does work properly with \r as a character, so I cannot explain why things for you previously did not work. They should have. That part of the mystery you will have to figure out yourself.

    General information: not all versions of sed/awk/etc. understand "escaped strings" (ex. carriage return (\r), tab (\t), newline (\n), etc.) -- it's bad to assume they do. Sometimes using these escaped characters requires additional command line flags to enable support for them; you really have to check the documentation. In some situations, you can use the octal equivalents, which is what I grew up with. Solaris, for example, tends to not support this. Anyway, confirmation of Busybox sed doing the right thing is here -- you can see carriage returns (byte 0x0d) have been removed in the resulting file:

    Code:
    root@gw:/tmp/home/root# printf "hello\r\nworld\r\n" > test.crlf
    root@gw:/tmp/home/root# xxd test.crlf
    00000000: 6865 6c6c 6f0d 0a77 6f72 6c64 0d0a       hello..world..
    root@gw:/tmp/home/root# cat test.crlf | sed 's/\r//' > test.new
    root@gw:/tmp/home/root# xxd test.new
    00000000: 6865 6c6c 6f0a 776f 726c 640a            hello.world.
    
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice