Shibby: Memory allocation and Squid

Discussion in 'Tomato Firmware' started by stevepdx, Oct 5, 2012.

  1. stevepdx

    stevepdx Serious Server Member

    I am interested in adding squid and privoxy, but we're out of space on the RT-N66U. The issue we see is too much space is being allocated to JFFS. Is there a proper / more appropriate way to add squid to Shibby's build?
  2. koitsu

    koitsu Network Guru Member

    These consumer-grade routers do not have large amounts of RAM (the RT-N66U has 256MBytes; that's still incredibly tiny even for something like squid), and squid is known for being extremely memory-hungry. I talked at length about this in a different thread not too long ago, and this thread here is pretty much proof of what I predicted in the other. lancethepants had a good follow-up on the matter, but YMMV. squid != streaming videos.

    You're going to have to tune squid's configuration file to use significantly less resources -- the trade-off is performance (expect it to be crummy). "Less resources" means tuning its memory limits, as well as tuning its on-disk cache size.

    It's important to remember that disk space (outside of the actual flash filesystem) on these routers is effectively RAM -- for example if squid's cache is located in /tmp/squidcache, /tmp is RAM, so you've now got two memory contention problems to deal with (squid's in-memory cache and the on-disk cache which is also RAM).

    You can solve the latter problem by using a USB flash disk and store squid's on-disk cache on that. However, keep in mind that the USB I/O on these routers is not very good either -- if you end up doing craploads of writes to the on-disk cache (which is liable to happen if you have lots of people doing requests through squid simultaneously, or if squid's memory cache is small), then performance is going to suffer.

    Bottom line, like I eluded to in the other thread, is that squid should really be left for a dedicated PC or purchased enterprise appliance, and not for consumer-grade routers.
  3. stevepdx

    stevepdx Serious Server Member


    Your post appears to be warning off the use of squid, not on installation of squid. Performance will be a non-issue: transparent redirects only, no caching, low-throughput. The issue I've got is getting it installed into my router image, can you assist in that direction?
  4. koitsu

    koitsu Network Guru Member

    Try using Entware. You will need a USB flash drive or CIFS/SMB share, mounted as /opt, to provide necessary disk space (you will almost certainly not have enough using just the on-board flash).

    Entware does provide squid builds -- verification:

    root@gw:/tmp/home/root# opkg list | grep -i squid
    squid - 2.7.STABLE9-2 - Squid is a high-performance proxy caching server for web clients, caching software, Squid handles all requests in a single,
    squid-mod-basic-auth-getpwnam - 2.7.STABLE9-2 - getpwnam basic authentication helper
    squid-mod-basic-auth-ncsa - 2.7.STABLE9-2 - NCSA basic authentication helper
    squid-mod-basic-auth-smb - 2.7.STABLE9-2 - Samba basic authentication helper
    squid-mod-cachemgr - 2.7.STABLE9-2 - Web based proxy manager and reporting tool
    squid-mod-digest-auth-password - 2.7.STABLE9-2 - Password digest authentication helper
    squid-mod-external-acl-ip-user - 2.7.STABLE9-2 - IP user external ACL helper
    squid-mod-external-acl-unix-group - 2.7.STABLE9-2 - Unix group external ACL helper
    squid-mod-ntlm-auth-fakeauth - 2.7.STABLE9-2 - Fakeauth NTLM authentication helper
    squid-mod-ntlm-auth-smb-auth - 2.7.STABLE9-2 - Samba NTLM authentication helper
    For Entware-related questions (how to use it, bugs with packages, etc.), please:

    a) Post a question on the Entware Discussion Forum,
    b) Open a support ticket with the Entware folks to ask for assistance.
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice