1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

My utilities web site revived

Discussion in 'Tomato Firmware' started by rhester72, Apr 10, 2010.

  1. rhester72

    rhester72 LI Guru Member

    I am finally in a position to (re)open my web site for Tomato lovers to sample my latest software research. Everything I have ever built has been made current and rebuilt, and I am continuing to add new packages at a somewhat brisk pace (spare time and interest notwithstanding ;).

    READ THE README!

    http://multics.minidns.net/blog/articles/tomato_utilities

    For the technically curious, this site is actually running on Hiawatha via my Asus RT-N16, compiled on an Ubuntu 64-bit VirtualBox VM I also use to compile custom firmware builds for said router. This stuff really works. Go figure. =)

    Rodney
    lancethepants and mstombs like this.
  2. Toastman

    Toastman Super Moderator Staff Member Member

    Some interesting stuff there. Kinda made me wonder how it would be possible to add this to the git repository. That made me remember your original comment on git when it was being discussed originally. Both methods have their uses!
  3. rhester72

    rhester72 LI Guru Member

    I honestly wouldn't mind at all if I could think of a good way to do so.

    The strong dependencies on libc (in fuller form than the image dynamic library) make it a challenge to do so without formally integrating all of them into the image, and it also makes rapid prototyping a challenge.

    On the other hand, it would certainly represent a lot less struggle for binaries with dynamic dependencies like lighttpd. :)

    Do you have any thoughts on how it could be done in a 'clean' fashion and still remain somewhat separate from the mainline (since my worst fear is turning Tomato mainline into effectively a 'mini-distribution' rather than a network-centric embedded appliance)?

    Rodney
  4. mstombs

    mstombs Network Guru Member

    Wow, thanks - that's a great collection - I challenge all to make a smaller contribution than mine (6.6kB binary)!

    Could I add darkstat to your wishlist? It uses libpcap which you already have with tcpdump.

    Should be straightforward for you to compile - we have it in Linux 2.4.17 for AR7 adsl routers in RouterTech firmware, but due to the Ti kernel mods and limited CPUs it does not capture and count all packets - it does maintain the list of sites visited which is still interesting. The N-16 with its more up to date kernel and extra ram may work well?
  5. Toastman

    Toastman Super Moderator Staff Member Member

    Mmm ! That makes me think, we have a tomato wishlist, which over the years has gotten so many weird requests in it, that everyone is ignoring it. There are obvious wishes that a lot of people need, which are commonly in router firmwares, and thus could be construed as useful rather than silly. Maybe a completely new thread with serious useful requirements could be beneficial. I might wade through it and see if there is any feature that sticks out.

    My own wish: a captive portal just to show an info screen and an "accept terms" checkbox or even a password, links to info/help page maybe.

    [ EDIT: Well, now we have NOCATSPLASH, I've found that it isn't actually very useful and ditched it after 3 days as being nothing more than a lot of hassle. Feedback indicated that almost nobody else ever uses it either :) ]

    Rodney, no, I can't really see a way to put it on git so that I, for example, could make use of it. But I'm very much a beginner, and I'm certainly not the best person to ask. TBH, for most smallish mods the diff is better for me. For continuous development cherrypicking from the git repo wins hands down though. I don't like github - it's like using a sledgehammer to crack a walnut. It creates so much extra work, leaving no time for development, not very nice for an individual hobbyist to get bogged down like that - so I deleted all my stuff from there long ago.
  6. rhester72

    rhester72 LI Guru Member

    Your wish is my command. Done. :) Pretty cool app, too!

    Rodney
  7. mstombs

    mstombs Network Guru Member

    Great, thanks - I'm surprised anything runs on K24 when compiled with the K26 toolchain, darkstat runs but errors - have yet to try on my Asus, but your compilation of pixelserv seems to run fine!

    There must be a good way to simply add apps to self-compiled Git images, the easiest I can imagine is for somewhere to drop in precompiled binaries so they end up in /usr/local/bin/. I have previously hacked timato/release/src/router/prebuilt to do this for specific things I wanted to try. The squashfs-lzma is very efficient and uses much less flash than jffs files
  8. rhester72

    rhester72 LI Guru Member

    darkstat-K24 in PRECOMPILED work any better on 2.4? Compiled with the latest K24 SDK (including a recompile of libpcap, then statically linked). If it works, I'll see if I can find a way to chain K24 and K26 trees simultaneously in PRECOMPILED.

    Rodney
  9. onehomelist

    onehomelist Serious Server Member

    I browsed through the packages and I want to try some of them. Can you please tell the procedure for adding the precompiled binaries to the router. I use Asus RT-N16.
  10. mstombs

    mstombs Network Guru Member

    I use winscp to copy binaries to temp ram disk then, run using telnet/ssh:-

    Code:
    Tomato v1.27.8745 ND Std
    
    
    BusyBox v1.14.4 (2010-04-05 09:58:16 EDT) built-in shell (ash)
    Enter 'help' for a list of built-in commands.
    
    root@unknown:/tmp/home/root# chmod +x  darkstat-K24
    root@unknown:/tmp/home/root# ./darkstat-K24 -i vlan1
    ./darkstat-K24: can't resolve symbol 'socketpair
    Same error as the K26 version, I assume something missing from my libs, or maybe user error in command line...
  11. onehomelist

    onehomelist Serious Server Member

    The darkstat site says that it serves reports over HTTP, and has an embedded web-server. As port 80 is used by tomato GUI, which port does darkstat use.
  12. rhester72

    rhester72 LI Guru Member

    667 by default.

    Let me do a little more research into the K24 issue - looks very much to be an (even more) incomplete libc implementation on that platform.

    I'm increasingly convinced that the benefits to whitelisting individual functions are not warranted - for the few routers where flash is at _that_ much of a premium, you can get away with micro, but we've evolved beyond that.

    Rant over. :)

    Rodney
  13. onehomelist

    onehomelist Serious Server Member

  14. onehomelist

    onehomelist Serious Server Member

    I get this error with dark stat on Asus RT - N16
    Code:
    root@unknown:/tmp/home/root# ./darkstat -i vlan1
      925: error: chdir("/var/empty") failed: No such file or directory
  15. onehomelist

    onehomelist Serious Server Member

    Finally it worked :) I used this command

    Code:
    ./darkstat --chroot /home/root -i vlan1
    it shows stats and host information but no graphs.
  16. mstombs

    mstombs Network Guru Member

    On an Asus RT-N16 the wan port is vlan2 (on a WRT54G-TM it is vlan1).
  17. onehomelist

    onehomelist Serious Server Member

    Yes mstombs. Now I get the graphs. Thanks. If I want to view my LAN users usage, which interface I should use.
  18. i1135t

    i1135t Network Guru Member

    Has anyone been able to compile a stand-alone version of darkstat for K24? I've been able to compile it for Ubuntu 9.10 and tested it on this platform, but haven't a clue on how to get it working properly for Tomato because of my limited compiling skills. If anyone is will to share, I'd be willing to test it. Thanks.
  19. mstombs

    mstombs Network Guru Member

    Lan ifname is usually br0 (eth0 the wired ports, eth1 wireless), but guess we should really be using

    $(nvram get lan_ifname) or $(nvram get wan_ifname) to work on a Tomato router

    You can run Rodney's K26 version on teddy_bear K24 kernel using, for example

    darkstat -i $(nvram get lan_ifname) --no-dns --chroot /home/root

    by default darkstat tries to create a second process that does reverse dns on all the IP addresses visited, it is this, using libresolv(?) that seems to be broken on this platform.

    You can view the darkstat stats on, for example

    http://192.168.1.1:667/

    But this version seems to have same problem as that in my adsl router fimware:-

    so numbers are not going to be quantitative
  20. rhester72

    rhester72 LI Guru Member

    I'm getting a very high percentage of dropped packets as well, though I'm honestly not sure why - my interrupt rate is well within norms, CPU utilization and 5-minute load are completely fine, actual packet rate is pretty light. Beats me. Still a pretty cool tool.

    K24 users: I can fix the problem at the expense of size/memory waste by fully statically linking. This is a horrible, horrible thing to do, but I don't think there's much choice. I've temporarily placed a statically-linked version up as "darkstat-static" and am looking into ways to create statics for everything (though it still seems God-awful to me!).

    I put a few fun things up today - hope someone finds them useful. Yes, they _can_ be used as scripting languages. Really. =)

    Rodney
  21. rhester72

    rhester72 LI Guru Member

    I also just put up darkstat-faster, which I am keenly interested to know if it drops packets to the same degree as the 'regular' compile. It is a static compile of libpcap and darkstat with the default optimizations (-pipe -O2) enabled. Please provide feedback if possible.

    Rodney
  22. mstombs

    mstombs Network Guru Member

    darkstat-faster works with dns on K24 (wrt54g-tm), don't have access to my N16 at the moment.

    It doesn't drop packets if you do a big download from Asus ftp @20kB/s, but drops about 50% on a fast 700kB/s download from Linksys. during the 'fast download' top on the wrt54 reports

    Code:
    Mem: 12900K used, 17720K free, 0K shrd, 1308K buff, 4540K cached
    CPU:   0% usr  28% sys   0% nic  71% idle   0% io   0% irq   0% sirq
    Load average: 0.00 0.01 0.00 2/25 3641
    so it looks to be idling - and this is the same behaviour I have seen before - so I think its a program issue, or the optimized kernel limiting what can be tracked.

    Lots of interesting info though, if you look at the router wan interface you can see how much comms from router dns services, and also the info on range of ports used.
  23. onehomelist

    onehomelist Serious Server Member

    I used darkstat-faster with the following command
    Code:
    ./darkstat-faster -i $(nvram get lan_ifname) --no-dns --chroot /home/root
    There are the stats
    Code:
    Running for 7 mins, 14 secs, since 2010-04-13 04:40:58 UTC+0000.
    Total 63,729,248 bytes, in 96,420 packets. (267,644 captured, 167,875 dropped)
  24. rhester72

    rhester72 LI Guru Member

    Looks like the drops aren't coming from load, then - it's got to be somewhere in kernel-land, because VPN over pcap doesn't seem to be affected by this (I'd like to think I'd notice 50% retransmissions!). Ugh.

    The good news is that K24 compatibility seems to be a 'simple' matter of full static linking...which is a disaster in terms of memory and disk usage, but it's either that or nothing. I am looking into a way of automatically producing "regular" and static binaries together during compiles for convenience/widest compatibility.

    Rodney
  25. i1135t

    i1135t Network Guru Member

    This is what I have so far and it appears the static version is not dropping packets as much on br0 interface.

    Code:
    ./darkstat-static --chroot /home/root/ -i $(nvram get lan_ifname)
    Code:
    Graphs (br0)
    
    Running for 16 mins, 40 secs, since 2010-04-13 13:17:37 UTC+0000.
    Total 2,543,304 bytes, in 11,220 packets. (11,446 captured, 125 dropped)
  26. rhester72

    rhester72 LI Guru Member

    It appears the drops have more to do with lack of interrupt coalescing than anything - you'll get more drops from a large number of small packets than you will a smaller number of large packets. That suggests some sort of interrupt flooding within the kernel which will be very tough to solve.

    Rodney
  27. rhester72

    rhester72 LI Guru Member

    For those who have been getting errors on accessing certain files, my apologies - there was a permissions problem in some of the original sources that has been corrected.

    Rodney
  28. rhester72

    rhester72 LI Guru Member

    I haven't updated the README yet, but _all_ binaries are now available in two flavors, regular and "-static". Whereas the regular binaries are targeted at Tomato K26, -static should work fine on _any_ router, including K24 and legacy Tomato (and probably DD-WRT and OpenWRT for that matter). If for some reason you find this not to be the case, please let me know.

    Rodney
  29. onehomelist

    onehomelist Serious Server Member

    I wish to have opentracker for tomato. It would be great to have tomato binaries, source-codes and SDKs distributed through bitotrrent using open tracker running on tomato itself.
  30. mstombs

    mstombs Network Guru Member

    Sorry to Hijack your thread Rodney, but I never managed to build darkstat for this platform and now you have...

    I believe I have found how to not drop packets, this is the new result of a 200MB download from linksys ftp

    Code:
    Running for 7 mins, 20 secs, since 2010-04-14 09:15:32 UTC+0000.
    Total 228,856,665 bytes, in 275,680 packets. (275,680 captured, 0 dropped)
    ...
    82.96.58.56  	a82-96-58-56.deploy.akamaitechnologies.com  	xxx	4,933,536  	223,060,857  	227,994,393  	1 min, 18 secs
    How? I found this

    http://staff.washington.edu/corey/gulp/

    which advises a big increase in "The default and maximum amount for the receive socket memory", so I

    Code:
    cat /proc/sys/net/core/rmem_max
    108544
    cat /proc/sys/net/core/rmem_default
    108544
    echo "4194304"> /proc/sys/net/core/rmem_max
    echo "4194304"> /proc/sys/net/core/rmem_default
    cat /proc/sys/net/core/rmem_max
    4194304
    cat /proc/sys/net/core/rmem_default
    4194304
    This is still on K24 on WRT54G-TM and darkstat-faster by the way, I wonder if this is useful elsewhere? No doubt the numbers can be tuned, but the -TM (and RT-N16) should both have plenty of ram.

    Note you need the -l lan for all LAN traffic to be included in the graphs

    Code:
    darkstat-faster -i br0 --chroot /home/root -l 192.168.1.0/255.255.255.0
  31. rhester72

    rhester72 LI Guru Member

    Setting rmem_default that high seems pretty risky - that's 4MB _per socket_, which will exhaust available RAM very quickly.

    Rodney
  32. rhester72

    rhester72 LI Guru Member

    Setting rmem_max by itself appears to be enough - checking with lsof, having rmem_max set to 512K results in four sockets with a read of 473K in my case (without having to force the default). Not very exhaustive testing yet, and I may have to increase rmem_max further, but you've clearly hit on the root cause. Good job!

    Rodney
  33. rhester72

    rhester72 LI Guru Member

    I've found a 256KB rmem_max good enough for my purposes here with no drops - but I am mightily confused about something.

    If I point darkstat at ppp0, the graph closely matches what I expect based on actual traffic loads - but I can't see which of my internal hosts are participating. If I instead point it at br0, I miss a lot because the internal hosts aren't counted. If I then add -l 172.16.0.0/255.255.255.0 (my internal subnet), I get more traffic, but the graphs don't seem to _at all_ correspond to actual traffic loads - for instance, when I crank up the VPN (on the same subnet using DHCP and bridging), I expect to see a dramatic rise in traffic on br0 (and I do on the Tomato graphs), but it doesn't seem to show in darkstat.

    Very, very confused.

    Rodney
  34. mstombs

    mstombs Network Guru Member

    The traffic stats and graphs here look fine now (normal traffic), but I recall issues with pppoe before, the TM seems fine the large receive buffer size, but the same size kills my modem!

    Code:
    Graphs (br0)
    
    Running for 10 hrs, 49 mins, 37 secs, since 2010-04-14 09:55:58 UTC+0000.
    Total 1,061,676,067 bytes, in 1,195,560 packets. (1,195,636 captured, 0 dropped)
  35. rhester72

    rhester72 LI Guru Member

    I still think it is safer just to set rmem_max and let the app set its own socket buffer size rather than forcing everything to be bigger than necessary.

    I found the "issue" with darkstat - I needed to use the source, Luke! ;)

    Code:
          if (dir_in == 1 && dir_out == 1)
             /* Traffic staying within the network isn't counted. */
             dir_in = dir_out = 0;
    Thus, the decision to keep pure LAN traffic out of the graphs was an intentional one - fair enough.

    Rodney
  36. mstombs

    mstombs Network Guru Member

  37. rhester72

    rhester72 LI Guru Member

    I've got a bit of a laundry list to work through at the moment (just finished up tor, now working on snort and nmap), but should be able to examine those in the next couple of days. Please don't let me forget!

    And yes, you know how I feel about NTP servers on soho routers, but I still have no objections to compiling the tools as a mental exercise. :)

    Rodney
  38. rhester72

    rhester72 LI Guru Member

    Having trouble with ntpd-static, but the regular binary is up. Didn't take that long. :) If you really need static, yell.

    Rodney
  39. mstombs

    mstombs Network Guru Member

    Wow what took you 6 mins? I seem to remember I didn't have a dev/random when I tried before - I also think you may need to change the default low privilege user in ntpd "NTPD_USER" would "nobody" work (as used by darkstat and dnsmasq).
  40. onehomelist

    onehomelist Serious Server Member

    My stats of darkstat-faster on Asus-RT-N16 after using mstombs code.

    Running for 3 hrs, 57 mins, 52 secs, since 2010-04-15 07:21:02 UTC+0000.
    Total 2,897,751,853 bytes, in 4,940,971 packets. (5,031,684 captured, 0 dropped)
  41. rhester72

    rhester72 LI Guru Member

    /dev/urandom appears to be sufficient without need of a recompile - I know it's present in K26, is it available in K24 as well?

    I've corrected the NTPD_USER to nobody but am keeping the default of using its home directory (/dev/null) - does that work?

    I'm not quite sure where/why it uses /dev/*random except for entropy, and I don't know exactly what it needs that entropy for.

    Rodney
  42. mstombs

    mstombs Network Guru Member

    The K24 does have random, but does need a static compiled version, there are other missing symbols.

    The fact it needs random suggests it is not appropriate for a router (too bloated). You only need entropy to seed secure comms don't you? And Firefox had a problem with entropy - it used to do on a sum on the contents of its cache, and as the cache gets fuller the sum took longer...

    There's an old msntp (source in debian), but I don't think its sntp server mode is compatable with XP.
  43. rhester72

    rhester72 LI Guru Member

    After two days of pretty exhaustive testing, a rmem_max value of 524288 appears to be sufficient to eliminate the drops with no need to change rmem_default (thus saving on memory wastage on lower-memory devices). Your mileage may vary. :)

    As for ntpd, I'm going to be digging a lot more deeply into it today, more out of curiousity about tearing it apart to see how it really works. I'm particularly interested in the need for entropy and understanding what's going on with the static build.

    After that, I'll take a look at the "Tomato-specific" server you mentioned before.

    Rodney
  44. rhester72

    rhester72 LI Guru Member

    Getting considerably closer.

    The entropy is needed for encrypted authentication, which is part of the spec. Switching away from OpenSSL's RC4 as an entropy source to the *BSD builtin helped a great deal with bloat.

    Static compilation _is_ required because adjtime() is missing in libc, so I've worked that issue out.

    A valid home directory and/or chroot jail is also required, /dev/null doesn't cut it - so I'm trying to work that out as non-intrusively as possible.

    So now with my test binary I've got it successfully talking to both the world and my network, but it doesn't yet trust *itself*. Sample output from ntpdate on a client machine:

    Code:
    172.16.0.1: Server dropped: Leap not in sync
    server 172.16.0.1, port 123
    stratum 3, precision -9, leap 11, trust 000
    I'm not sure if this is because of unacceptable drift on the server (meaning a router will _never_ work for this if the published spec is followed and you don't "cheat") or if I just haven't given it long enough to settle down/create a valid drift file yet. More on that soon.

    Rodney
  45. rhester72

    rhester72 LI Guru Member

    As luck would have it, 30 seconds after the above post, the server synced:

    Code:
    adjusting local clock by -0.054153s
    clock is now synced
    
    After that, the client was plenty happy:

    Code:
    server 172.16.0.1, port 123
    stratum 3, precision -9, leap 00, trust 000
    refid [172.16.0.1], delay 0.04587, dispersion 0.01968
    transmitted 4, in filter 4
    reference time:    cf72f04a.4cc67fff  Fri, Apr 16 2010 10:18:50.299
    originate timestamp: cf72f08d.426427ff  Fri, Apr 16 2010 10:19:57.259
    transmit timestamp:  cf72ed07.8396d091  Fri, Apr 16 2010 10:04:55.514
    filter delay:  0.04694  0.05032  0.05655  0.04587 
             0.00000  0.00000  0.00000  0.00000 
    filter offset: 901.7118 901.7119 901.7235 901.7360
             0.000000 0.000000 0.000000 0.000000
    delay 0.04587, dispersion 0.01968
    offset 901.736051
    
    16 Apr 10:04:55 ntpdate[23357]: step time server 172.16.0.1 offset 901.736051 sec
    
    (Obviously, this is from a client I've intentionally skewed the time on for testing.)

    So, it appears to work fine, and I take back what I said about slagging the router as a timesource. ;) If it meets the RFC, it's good enough for me.

    Will tidy up the binary a bit and move on.

    Rodney
  46. rhester72

    rhester72 LI Guru Member

    tomato-ntpd and tomato-ntpd-static (based on the referenced post for a Tomato-specific ntpd) have been compiled but are untested, feedback welcomed.

    Rodney
  47. rhester72

    rhester72 LI Guru Member

    Have now added M. Ring's modified ntpc and ntpc-static (which he symlinks as ntpstep), read the package-level README for more information.

    Rodney
  48. rhester72

    rhester72 LI Guru Member

    And from a time synchronization standpoint, things have now come full(y ridiculous) circle. ptpd now available. :)

    Rodney
  49. rhester72

    rhester72 LI Guru Member

    It's not yet clear to me whether -DWANT_COMPRESSION_GZIP *forces* gzipping or simply makes it available (I saw nothing in the example config suggesting it can be toggled), but for now it's been compiled with it turned on unless someone tells me it's not a good idea to do so (either because it really is forced and some clients can't deal with that or because the compression buries the router CPU under load).

    That having been said, binaries of current CVS are up.

    Rodney
  50. mstombs

    mstombs Network Guru Member

    So you now things it is a reasonable idea for the router to keep time for all devices on your LAN? If so I guess there is a way to pas this on to dhcp clients using dnsmasq...

    The source for msntp (pre 2000) says Linux adjtime is broken, guess that's now old info

    If you really want a challenge:- why is the Linux portable OpenNTPD so many versions behind the BSD version:-?
  51. onehomelist

    onehomelist Serious Server Member

  52. onehomelist

    onehomelist Serious Server Member

  53. rhester72

    rhester72 LI Guru Member

    re: time - Yes, I think the time can be maintained at sufficient resolution on most home router models to guarantee pretty close to 1-second accuracy, which is acceptable for most home uses. (At my job, we have to maintain 10 microseconds or better. Fun!)

    re: ntpd - Portable hasn't been updated in eons because, well, it does what it was intended to do, so there's no compelling reason to try to have it track full ntpd. That having been said, I'll take a shot at "real" ntpd as soon as I can get back to my machine (I hosed up my VPN and all outbound 'net services this morning doing a boneheaded remote configuration change. Oops.)

    re: wondershaper: It's just a script. Nothing to compile. Edit to taste. I called mine "tshaper" in homage to "wshaper", but literally haven't looked at it in years...posted an old(ish) version of it to the forum not that long ago.

    Rodney
  54. onehomelist

    onehomelist Serious Server Member

    I have 2 groups of users on my network. One via wireless (separate access points) and one via wired LAN. The wireless users flood the network by over-usage. So i want to divide the ISP supplied bandwidth with equal proportions for both groups. If its 4 Mbp/s then 2 Mbp/s each. I would create two separate networks by following this guide.
    Code:
    http://www.seiichiro0185.org/doku.php/blog:creating_a_seperate_guest_network_with_tomato
    Then I am planning to use wondershaper to rate-limit on one network (vlan2) with 2 Mbp/s bandwidth. I use RT-N16, so I'll create vlan3 as vlan2 is used by WAN. Would you say I could do it? Will I be able to force vlan3 interface down to 2Mbp/s up and down?
  55. rhester72

    rhester72 LI Guru Member

    Remarkably, the ntpd built into busybox works just as well as Portable OpenNTP and even adheres (closely enough) to the spec to reject NTP clients until it has settled down...which happens a _lot_ faster than Portable. I've switched. =)

    Rodney
  56. mstombs

    mstombs Network Guru Member

    Good to know - is it easy to drop your build of busybox 1.16.1 into the K26 Git source and just enable this? Probably doesn't add many kB?
  57. rhester72

    rhester72 LI Guru Member

    Should be - I didn't do anything to busybox, really =)

    Rodney
  58. rhester72

    rhester72 LI Guru Member

    I added sockstat 0.3 this morning - very useful for troubleshooting connections and getting to the bottom of what a PID is _really_ doing. Hope someone finds it handy.

    Rodney
  59. rhester72

    rhester72 LI Guru Member

    ...and hdparm 9.28, useful for USB testing.

    Be *VERY* careful with this. It _CAN_ destroy your data!!! YOU HAVE BEEN WARNED!

    Rodney
  60. onehomelist

    onehomelist Serious Server Member

    I tried to compile from the sources you'd modified for tomato (from your sites root folder). As you have not compressed and tarred, I found it bit difficult to download the files. Can you please provide them in tar or gz format. Thanks. I tried few of your precompiled binareis. I got so excited that I thought of compiling them on my own.
  61. rhester72

    rhester72 LI Guru Member

    Just pick up the original source from SOURCES, the tomato-<package> script from the appropriate directory, and any prerequisites (dependent libraries, the tomato-include script, etc.), or just read the tomato launch script for the package and it should be pretty obvious how I'm going about it.

    Rodney
  62. onehomelist

    onehomelist Serious Server Member

    Your utilities site is not available.
  63. mstombs

    mstombs Network Guru Member

    oh yes it is...
  64. onehomelist

    onehomelist Serious Server Member

    Your site is available now,Thanks rhester72. If moblock is complied for tomato, it'll be a powerful utility than any adblock scripts as it has adblock and spyware block lists as well. Even it'll provide an insight into how peerguardian block-lists can be effectively used to set rules for multiple clients on the network.

    http://sourceforge.net/projects/moblock-deb/develop
  65. rhester72

    rhester72 LI Guru Member

    Sorry for the lack of availability - the site is on my home DSL connection on the same router I also have to use for testing. :) From time to time it may go bump in the night, but for the most part it should be reasonably stable.

    I'll take a look at moblock as soon as I finish revamping the internal build process.

    Rodney
  66. rhester72

    rhester72 LI Guru Member

    moblock is now available. Only IPQ mode works right now, NFQ is busted and I'm not sure why (I've ruled a lot of things out, though).

    Rodney
  67. onehomelist

    onehomelist Serious Server Member

    Wow, great. I will surely implement it and report back to you the results.
  68. onehomelist

    onehomelist Serious Server Member

  69. onehomelist

    onehomelist Serious Server Member

    I tried it it gives this error

    Code:
    ./MoBlock-ipq.sh: line 17: basename: not found
    [: ipq: unknown operand
    [: nfq: unknown operand
    iptables v1.3.8: Unknown arg `-j'
    Try `iptables -h' or 'iptables --help' for more information.
    iptables v1.3.8: Unknown arg `-j'
    Try `iptables -h' or 'iptables --help' for more information.
    iptables v1.3.8: Unknown arg `-j'
    Try `iptables -h' or 'iptables --help' for more information.
    ./MoBlock-ipq.sh: line 87: ./moblock-: not found
    
    
    I edited the list source dir line like this

    Code:
    ./moblock-$MODE -p ./level1.txt ./moblock.log
    and I downloaded the list file extracted it and placed it in the same dir.
    then I gave this command
    Code:
    ./MoBlock-ipq.sh &
    It showed the above error
  70. rhester72

    rhester72 LI Guru Member

    I didn't realize that basename isn't compiled into stock busybox. Pick up my busybox compile, rename it to basename and place it somewhere in the search path. (Ideally, this should be included in stock, it's very common and very light on resources, it seems rather wasteful to use a big-boned busybox compile just for this...I'll work on modifying the script to have the same effect without needing basename.)

    Rodney
  71. rhester72

    rhester72 LI Guru Member

    Scripts updated to replace basename usage with something more Tomato-friendly, sorry for the miss.

    Rodney
  72. onehomelist

    onehomelist Serious Server Member

    It worked. But it gave a error that ip_queue.ko was not found. But it started populating the logfile. Nice work rhester72. looks like you are the first one to compile it for an embeded device. I will test it a little more and will report back to you the results. Thanks a lot.
  73. rhester72

    rhester72 LI Guru Member

    ip_queue.ko for K26 is also available on my site under lib/modules/2.6.22.19/kernel/net/ipv4/netfilter - the path needs to it needs to be explicitly set in the appropriate insmod line. This is very much required for proper operation. :)

    You can tell if it was loaded properly if it shows up in the list from lsmod after the script is running.

    Rodney
  74. rhester72

    rhester72 LI Guru Member

    I've made very significant changes to the launch scripts in hopes of making them a bit easier to manage/user-friendly. You should now only need to edit the paths (and optionally filenames) at the top of the script, the lines to change should (hopefully) be very obvious. Comments welcomed!

    Rodney
  75. onehomelist

    onehomelist Serious Server Member

    Thanks rhester72. It works absolutely well. No errors at all.
  76. i1135t

    i1135t Network Guru Member

    I get the "unable to load ip_queue.o module" error as well, but it reaches the end of the script and creates the pid and log file. I did modify the filter path and log path, however, I don't think it's working properly on K24. I am using the IPQ version. After executing the script, it does not show in memory and nothing populates in the log. Also, all of the new MoBlock IPTABLES that were supposed to be created, were not. Do I need the ip_queue.o file for K24 and if so, where can I find it?
  77. rhester72

    rhester72 LI Guru Member

    Yes, you absolutely need ip_queue.o for K24. I haven't compiled K24 in ages - anyone have that module handy for the K24 kernel?

    Rodney
  78. onehomelist

    onehomelist Serious Server Member

    Your suggestion for a captive portal on the other thread made me wonder about a light-weight captive portal. I found wifidog quite interesting. it doesn't have it's own dhcp or dns modules, and very few dependencies. if you could give a try to see if it can be compiled for tomato it will be a immense help. Sorry, I've asked more than what you can do.

    http://sourceforge.net/projects/wifidog/files/
    http://dev.wifidog.org/wiki/doc/install/gateway
    http://justuber.com/publicwifi:public_wireless_internet_access
    http://dev.wifidog.org/browser/trunk/wifidog/README.openwrt
  79. rhester72

    rhester72 LI Guru Member

    Compiled, utterly untested. :)

    Rodney
  80. onehomelist

    onehomelist Serious Server Member

    Nice. Exiting News. First captive portal for tomato. I will test and report back to you the results. Will it be okay if I put the conf file in the same dir?
  81. rhester72

    rhester72 LI Guru Member

    I have no idea, didn't read the docs. You're literally breaking new ground here. =)

    Rodney
  82. onehomelist

    onehomelist Serious Server Member

    The docs say that conf file must be placed in /etc dir manually, even under openwrt. I am going to test it pretty soon. I will let you know how it goes?
  83. onehomelist

    onehomelist Serious Server Member

    I placed wifidog.conf file in /etc

    I tried 'PRECOMPILED' version. I got this error
    Code:
    root@unknown:/tmp/home/root/wifidog# wifidog -f -d 7
    wifidog: can't load library 'libhttpd.so.0'
    Then I tried 'PRECOMPILED-static'. I got this error

    Code:
    root@unknown:/tmp/home/root/wifidog# wifidog -f -d 7
    [6][Thu Jun 10 16:23:45 2010][1148](conf.c:638) Reading configuration file '/usr/local/etc/wifidog.conf'
    [3][Thu Jun 10 16:23:45 2010][1148](conf.c:642) Could not open configuration file '/usr/local/etc/wifidog.conf', exiting...
    If the source is modified so as to make it look for conf file in same directory where binaries are, it might work.
  84. rhester72

    rhester72 LI Guru Member

    Code:
    Usage: wifidog [options]
    
      -c [filename] Use this config file
      -f            Run in foreground
      -d <level>    Debug level
      -s            Log to syslog
      -w <path>     Wdctl socket path
      -h            Print usage
      -v            Print version information
      -x pid        Used internally by WiFiDog when re-starting itself *DO NOT ISSUE THIS SWITCH MANUAlLY*
      -i <path>     Internal socket path used when re-starting self
    
    I think you're looking for the -c switch.

    Rodney
  85. rhester72

    rhester72 LI Guru Member

    Thanks to the wonders of pure-ftpd, my utilities site is now also available by anonymous FTP (ftp://multics.dynalias.com/tomato). This may make it a bit easier for some users to browse and should certainly improve upon the ability to snag entire directory trees where needed (though please don't try to mirror the whole thing just for kicks - this is my home DSL line!). Take what you need, but please don't abuse it.

    pure-ftpd for Tomato can be found there, of course. ;)

    Rodney
  86. ripat

    ripat Addicted to LI Member

    Hi Rodney,

    I would like to install lighttpd but I couldn't find a precompiled/static version of it. Coul'd you direct me to a step by step howto install it? If I download the precompiled (which I did) what dependencies should I also download and where should I place them?

    Thanks for the nice work you have done.

    JL
  87. rhester72

    rhester72 LI Guru Member

    lighty is on my utilities site, but is *NOT* available as a static compile (because it's impossible to do so, given the plugin system). You can check the library dependencies with "ldd".

    As for a step-by-step install, that would be all but impossible, because any web server solution like this is rather complex and setup is extremely dependent on what you're trying to achieve with it and the particulars of your environment (available storage and type, build type, paths, etc.). If you aren't already pretty comfortable with Linux command-line stuff, you may want to tap a local Linux geek for help. =)

    Rodney
  88. ripat

    ripat Addicted to LI Member

    Well, as I am the only linux "geek" in house you are the only guru I can ask. I will try to find some help on google and go throught the try and error dance. Can you at least tell me what I exactly have to download from your site to cross compile lighttpd on my debian box?

    Thanks.
  89. rhester72

    rhester72 LI Guru Member

    Cross-compiling is pretty straightforward - as long as you have the Tomato K26 SDK (see the first post in TB's huge thread) installed and working properly, you can take a look at the tomato-lighttpd script in the lighttpd directory on my site for clues on how I compile it. It isn't necessarily the "best" way because there is no "best" - but it works. =)

    If you don't need to self-compile, it's probably a lot simpler to take the precompile. You'll need (in addition to the lighttpd binaries and modules) libpcre, libssl, and libcrypto, all available in PRECOMPILED/lib, and make sure they are in your library search path (I default to /opt/usr/lib for RPATH, if you don't put them there it's fine, just make sure they are included in the LD_LIBRARY_PATH environment variable in your launcher script).

    Rodney
  90. ripat

    ripat Addicted to LI Member

    I think I'll have to give up as my jffs partition is not big enough to store all these files. The libcrypto.so.1 alone (1274K) fills up all my available space. Thanks for helping anyway.
  91. rhester72

    rhester72 LI Guru Member

    Ah, yes - there's no way to run lighty (or any web server, for that matter) off jffs - not that you'd want to. CIFS, however, works reasonably well.

    Rodney
  92. onehomelist

    onehomelist Serious Server Member

    I tried wifidog, and its actually a system where anyone can login and access internet. I thought of having something where admin will have the complete control. How about coovachilli, can it be compiled for tomato, as it has its own dns server, dnsmasq might have to be turned off.

    http://coova.org/CoovaChilli
  93. rhester72

    rhester72 LI Guru Member

    Compiled. I know very little about it, but it looks pretty complex, so don't expect much support with setup. ;) If different configure/compile options are required, let me know - I took the straight defaults.

    Rodney
  94. ewmailing

    ewmailing Serious Server Member

    Zeroconf (Avahi & Bonjour)

    Hi, I hope I understand the intent of this thread correctly.
    I am very interested in seeing Zeroconf on Tomato. In fact, I just wrote up a big long article on how to setup Avahi via Optware.

    http://playcontrol.net/ewing/jibberjabber/adventures-with-dd-wrt-part-4.html
    http://playcontrol.net/ewing/jibberjabber/adventures-with-dd-wrt-part-5.html
    http://playcontrol.net/ewing/jibberjabber/adventures-with-dd-wrt-adde.html

    One observation I made is that all the Avahi dependencies from Optware make it too big to be practical to fit as a standard feature of a stock firmware distribution. I am curious if the dbus dependency can be stripped down or removed and if linking to already available Tomato libraries would shrink the footprint enough that inclusion would be possible (from a technical standpoint).

    Would you be able to build this kind of package?

    Second, and maybe more interesting, would you be able to build Bonjour? Bonjour has a unique feature called Bonjour Sleep Proxy (described in my articles) which Avahi has not implemented yet (nor anybody else).

    The source code to Bonjour can be found here:
    http://developer.apple.com/opensource/
    (You may need to use wget because their SVN isn't working for me at the moment.)

    Thanks
  95. rhester72

    rhester72 LI Guru Member

    Well...not exactly. Think of what I provide as "poor man's Optware" - basic binaries requiring hand-install and configuration. It was useful to me (because I don't like the overhead of solutions like Optware), I hoped it would be useful to others.

    That having been said, it does show means by which various things *can* be built for Tomato for possible integration into base, and I've gotten Avahi compiled (static and dynamic) and available on my site (but completely untested).

    Static binaries are huge - avahi-daemon alone weighs in at nearly 600KB!

    Dynamic isn't any better - the perceived savings by linking to existing libraries are virtually non-existent because most of the required library dependencies are specific to Avahi itself:

    avahi-daemon: 59600
    libavahi-common.so: 72164
    libavahi-core.so: 279532
    libdaemon.so: 28118
    libexpat.so: 165021

    TOTAL: 604435

    So, it seems perfectly reasonable to run a static avahi-daemon on a suitably powerful router like the RT-N16, but very unlikely it would appear in even TB's "extra" builds...it's just too fat in terms of flash (and memory, in all likelihood). There isn't much more I can do to trim this down much further, certainly not the more than 400K more it would need to be shaved to even think about inclusion.

    I haven't tried to compile Bonjour itself yet, it's likely to be a can of worms, but I might look at it later today.

    Rodney
  96. rhester72

    rhester72 LI Guru Member

    mDNSResponder compiled and up on the utilities site. I can't get it to do much useful (no experience with it), but it appears to be working. It was indeed a bit of a challenge.

    Interestingly, the daemon itself appears to be about half the size of avahi-daemon.

    Rodney
  97. Aiko

    Aiko Serious Server Member

    If I can ask for a package to be compiled and included as a binary for Tomato, i'd request Mediainfo.

    My router is used to download torrents and has a HD with media in it. Mediainfo would come handy to check the profile of different files.

    I couldn't find any dependencies, and I also found a mipsel compiled version in this page

    http://tracker.netbsd.org/pub/NetBSD/NetBSD-current/pkgsrc/multimedia/mediainfo/README.html

    Just not sure if that would work with Tomato, or if this application is too much for a RT-N16 processor.

    Thanks!
  98. rhester72

    rhester72 LI Guru Member

    MediaInfo is doable, but a RT-N16 is pretty much required - the unstripped alpha binary alone weighs in at a whopping 5 *megs*!

    I'll clean up the compile process, get it stripped and static, and make it available on the site tomorrow.

    Rodney
  99. Amuro

    Amuro Networkin' Nut Member

    that seems interesting, i use mediainfo in my computer but never thought of running it in my router.

    it makes sense if you download a lot using the router as sorta a d/l station
  100. ewmailing

    ewmailing Serious Server Member

    Thanks for doing this. It's a little disappointing that Avahi is so big. As for expat, I'm sure that is all related to their XML config file support. I was hoping maybe something already used it in Tomato so the dependency would be mitigated. I suppose somebody more knowledgable will need to take a scalpel and cut out things that are not needed on a router to shrink the size, or we need to wait until 8MB or greater ROM sizes become standard.

    Anyway, I tested some of the Bonjour stuff. mdnsd (which is named mDNSResponder on Mac which confuses the heck out of me...what was Apple thinking here?) seems to work. I also tested mDNSResponder (which lets you advertise specific services like web, ssh, samba, etc). That seems to work too. I'll probably try mDNSProxyResponder at some point soon. (This is a new tool I've never seen before.)

    However, it seems that Bonjour Sleep Proxy is not enabled. I dug through the source code and I don't think they enable it in their non-Mac codebase. (And even in their Mac codebase, there are a lot of rules that seem to keep it disabled.) Long story short, I was wondering if you could try adding a line of code to the Posix implementation to enable Bonjour Sleep Proxy and rebuild it for me.

    In PosixDaemon.c, there is a function called Reconfigure().
    At the bottom, before mDNS_ConfigChanged(m), I want to add the line:
    mDNSCoreBeSleepProxyServer(m, 50, 25, 10, 70);

    I haven't tested any of this stuff. I'm making a lot of guesses based on the Mac code. The final function should look like below (I added comments too.)


    static void Reconfigure(mDNS *m)
    {
    mDNSAddr DynDNSIP;
    const mDNSAddr dummy = { mDNSAddrType_IPv4, { { { 1, 1, 1, 1 } } } };;
    mDNS_SetPrimaryInterfaceInfo(m, NULL, NULL, NULL);
    if (ParseDNSServers(m, uDNS_SERVERS_FILE) < 0)
    LogMsg("Unable to parse DNS server list. Unicast DNS-SD unavailable");
    ReadDDNSSettingsFromConfFile(m, CONFIG_FILE, &DynDNSHostname, &DynDNSZone, NULL);
    mDNSPlatformSourceAddrForDest(&DynDNSIP, &dummy);
    if (DynDNSHostname.c[0]) mDNS_AddDynDNSHostName(m, &DynDNSHostname, NULL, NULL);
    if (DynDNSIP.type) mDNS_SetPrimaryInterfaceInfo(m, &DynDNSIP, NULL, NULL);

    // Apple calls SetSPS(m) before mDNS_ConfigCahgned(m) in mDnSMacOS.c
    // SetSPS(m) wraps mDNSCoreBeSleepProxyServer. So this is my best guess.

    // 50 seems to be what Apple sets if NAT is running
    // SPMetricPortability = 35 for AppleTV (considered a low priority server)
    // SPMetricPortability = 25 is Xserve
    // SPMetricMarginalPower = 10 is Apple TV which seems to be a special always server (though low priority) case
    // SPMetricTotalPower = 73 for Apple TV for 20W. Each increment seems to be 5W.
    // So assume our DD-WRT/Tomato device is 5W, SPMetricMarginalPower = 70
    mDNSCoreBeSleepProxyServer(m, 50, 25, 10, 70);

    mDNS_ConfigChanged(m);

    }

    Thanks

Share This Page