1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Common Tomato Mod repository?

Discussion in 'Tomato Firmware' started by SgtPepperKSU, Nov 26, 2008.

  1. SgtPepperKSU

    SgtPepperKSU Network Guru Member

    This is a suggestion/question to those who already or want to make custom tomato builds. Those who don't have any interest in building from source can safely ignore this.

    I've recently moved my Tomato related sources to a local git repository and have found it to greatly aid in development. I have separate branches for each "module" like so:

    Code:
    linksys
    |
    tomato____________________________
    |         \      \         \      \
    tomato-ND  LZO    \         \      OpenSSL(still buggy, so haven't merged)
    \             \____OpenVPN   \
     \                        \___tomatovpn
      \                                    \
       \____________________________________tomatovpn-ND
    
    That way when an updated tomato, OpenVPN, or LZO version comes out, I just update the source in that branch, get any bugs worked out, and do a single git merge to get it into the tomatovpn branch. And, of course, it keeps track of what I've changed in the old versions, so I can cherry-pick them to the new version if appropriate.

    I got to thinking, and there seems to be some duplication of effort among the different mods. If we all developed in a common repository, then that wouldn't have to be. There could be an SD mod branch, an OpenVPN branch, a USB branch, a TCP Vegas branch, an OpenSSL branch, etc. Then, if you want to make a mod you just git merge whatever features you want. That way, if one of us fixes a bug, a git merge later and everyone reaps the benefit.

    Also, this would make it a lot easier for people to get the endless permutations of features that they ask for.

    What do you guys think? Would you guys be interested in something like this?
     
  2. Victek

    Victek Network Guru Member

    SgtPepperKSU,

    I did too, the problem is when the vanilla Tomato change, then all the tree needs to be rebuild. The idea was talked to Jon time ago, then we need a patch distribution to check diff with existing files in our mods.
     
  3. mstombs

    mstombs Network Guru Member

    The mlppp mod is also distributed as a patch file against stock Tomato so would be easy to add - to see what they have done to ppp code and GUI etc.

    Great story as to how "git" was named...

    http://en.wikipedia.org/wiki/Git_(software)
     
  4. SgtPepperKSU

    SgtPepperKSU Network Guru Member

    I'm not sure what you're saying the problem is here. When the vanilla Tomato source changes, it gets updated in the tomato branch. Then, any branch that wants the update does a git merge. If there are conflicts with our changes, they be indicated during the merge and you deal with them then. There's no need to have a patch distribution of vanilla Tomato (is that what you were suggesting?), as git is smart enough to notice only what has changed since the last commit when you update the source (even if you delete everything and unarchive the linksys+tomato sources).

    It seemed to work great with the Tomato 1.22 release for me.
     
  5. SgtPepperKSU

    SgtPepperKSU Network Guru Member

    Not only would it be easy to add to the repository, you could create the patch file without even cleaning the source directory as "git diff" only diffs tracked/committed files (by default). When it's time to do a release, one could just do a "git diff tomato mlppp" without having to deal with files that have changed due to the build, but that don't need to be distributed with the patch.
     
  6. occamsrazor

    occamsrazor Network Guru Member

    I am not a developer, just a user, but have a few thoughts on this.

    I have become a bit frustrated by the bewildering variety of different mods and builds for Tomato, and am a bit concerned at the fragmentation that has happened. I realise different people have different requirements, and so choice is a good thing, but it seems to me there is lots of great stuff that is going on in all sorts of different places and different threads, and there doesn't seem to be an efficient way for acknowledged advancements to be added to all branches.

    Anything that can add a bit more structure to the development of Tomato, enabling worthwhile additions to be added easily to all branches, can only be a good thing I think. For example I must have OpenVPN (thanks SgtPepperKSU and roadkill) but would like to see cool developments in other areas easily added to those openvpn builds. Also it seems the openvpn builds often take some time to catch up with the plain tomato updates.

    Anything that aids the various developers to more easily incorporate things into the different builds, and update them, seems a great idea to me.

    But like I say, I'm not a developer and don't pretend to understand the technical issues involved....
     
  7. ng12345

    ng12345 LI Guru Member

    I feel like this a great idea and am surprised that other mod developers (besides victeck) have not responded yet. I know he mentioned a possible problem, but as I am not a developer I do not know how big of a problem this is.

    It is unfortunate to see this thread bumped down so quickly.
     
  8. rhester72

    rhester72 Network Guru Member

    Honestly?

    a) Just not a major fan of git. *shrugs* I'm still old-school svn (thought I was going to say cvs, didn't you? ;).

    b) If I'm doing a serious mod, my intention isn't to further fracture Tomato but instead to get it in mainline. If it isn't appropriate for mainline, it's probably not appropriate for wide distribution.

    Just my $0.02.

    Rodney
     
  9. roadkill

    roadkill Super Moderator Staff Member Member

    I think it's a good idea whatever way you'll eventually use to accomplish it.
     
  10. SgtPepperKSU

    SgtPepperKSU Network Guru Member

    I don't think it is a problem at all. In fact, I think it demonstrates of the major benefits of using a common RCS.

    When a new upstream version comes out, just one person has to put it in the repository. Then it automatically enters it as a diff against the previous version, and anyone who merges only has to deal with the actual changes. If there are no conflicts (changes to the same lines of code that you've changed), then you're done.
    Have you used it recently? I never had before this, but decided to have an open mind and play with it. I've found the workflow to be seemless with what I want, and I'd now hate to go back to using CVS/SVN (which I admittedly haven't done for a number of years, so perhaps SVN has improved). I haven't found a single time it has gotten in my way (and I fully expected to going in).
    Fair enough.

    However, I'd argue the current situation with a slim Tomato and people making custom builds with added features is a great compromise (instead of living without or moving to an attempted all-inclusive firmware like dd-wrt). This way people get the great Tomato baseline, plus just the features they want.

    Just because a feature isn't applicable to all, or even most, Tomato users doesn't mean it isn't useful to many.

    And, I certainly wouldn't call it "fracture". Everyone is still testing (using) the Tomato code, and if there are any changes in the custom builds that are appropriate for mainline, I'd certainly hope the author would communicate it back to Jon.
     
  11. rhester72

    rhester72 Network Guru Member

    In that case, rather than deal with it as a "distro of the week" methodology, I'd rather take the "package"-based mods (things that add simple packages like snmpd, srelay, etc.) and simply place them in the release/src/router directory and control whether or not they are integrated into a particular build via the standard Tomato mechanisms. This won't work for everything (SD-MMC, USB, etc.), but would give us a common code base so that merely changing out a Makefile and a config file or two will result in a brand-new "distribution" with the desired mods. Better for developers, better for users, and best of all...we could *finally* move to a shared dynamic library model rather than static-linking libpthread for the umpteeth time. It would also encourage a common repository for porting efforts, since we seem to be suffering from a most serious case of "reinvent the wheel" syndrome.

    Maybe you have a point. :)

    Rodney
     
  12. humba

    humba Network Guru Member

    As an end user, I'd certainly like to see a lot more features incorporated into Tomato (many mods don't add new software so for me it's not so much bloating something but rather exposing the full potential to users who'd otherwise wouldn't use some functionality).. but seeing as the author of Tomato is very conservative in terms of what is included in the mainline, I think this is an excellent idea.. I'm certainly looking forward to combined mods. I need VPN but I'd prefer not to miss out on some of the other options that are floating around.
     
  13. SgtPepperKSU

    SgtPepperKSU Network Guru Member

    Without these "branches" (ie, having a common "trunk" codebase that configures in/out individual features), Jon (or somebody) would have to vet every new "package" and keep up with the updates. Forget one #ifdef FEATURE around a change, and it could break things for everyone else. And, as you say, it may not work with all types of mods.

    I keep separate branches for LZO, OpenVPN (merges LZO updates), VPNGUI (merges OpenVPN updates), and my release code (merges VPNGUI changes). Each is fully compilable on its own. That way if anyone else's branch wanted any of that without the rest, they can without any duplication of work.

    I've also developed the VPN GUI changes in such a way that simply saying "No" to OpenVPN during the router menuconfig (that already runs during vanilla tomato compilation) leaves all of my changes out of the build, so I definitely agree with what you're saying there, though.
    Couldn't agree more :)

    Just to clarify since it's sometimes hard to read tone in print: I'm really not try to do any preaching here. In fact, I have no investment in this idea since I'll continue to use an RCS and likely won't be including other non-mainstream features in my builds regardless of whether this comes to fruition. I just hate to see all of the duplication of effort and wanted to see if others also thought it would help (or more importantly if they thought it wouldn't). If so, I'd be more than happy to set up the mechanics of it.

    I'll probably go ahead and set it up, and if it falls into disuse: oh well. At the very least, I'll have a remote mirror of my code.
     
  14. Toastman

    Toastman Super Moderator Staff Member Member

    sgt

    I for one think it is a good idea if it can be made to work and maintained.
     
  15. SgtPepperKSU

    SgtPepperKSU Network Guru Member

    I pushed by branches yesterday, and they can be viewed here. You can clone, fetch, and/or pull using the URLs on that page.

    Those wishing to push changes to the repo should register a username and SSH public key. PM me with your username, and I'll give you push access.

    I strongly suggest that, instead of just pushing your source tree as is, you base a new branch off of the tomato branch and make only the needed changes there (not changes that were the result of building+cleaning or differences in unrelated places). That way, if people merge your branch, they only get what is actually needed. Further, if you could break it up into logical commits, it would help people to understand what was done (rather than just seeing a big, all-encompassing diff). Also, please try to limit it to one feature per branch (you can create a "release" branch that merges them all together - see my vpngui and tomatovpn branches for an example).

    For those not familiar with git, I'll post some use-case instructions sometime in the next couple of days. In the meantime, you can peruse the user manual and general tutorial.

    And, I'm definitely open for feedback. :smile:
     
  16. Victek

    Victek Network Guru Member

    :biggrin: Wonderful !!, I need to read the manual first ...
     
  17. SgtPepperKSU

    SgtPepperKSU Network Guru Member

    Okay, here's a few useful commands:

    To start, clone the repository:
    Code:
    git clone git://repo.or.cz/tomato.git <NewDirectoryName>
    If you plan to push code back to the repository, you should register a user name at repo.or.cz and use
    Code:
    git clone ssh://<username>@repo.or.cz/srv/git/tomato.git
    Now, all of the existing branches will be addressable with origin/<BranchName>.

    To just view a branch:
    Code:
    git checkout origin/<BranchName>
    To visualize the current branch (and ancestors):
    Code:
    gitk &
    To visualize all branches:
    Code:
    gitk --all &
    To update the origin branches:
    Code:
    git fetch
    To get a new branch from the remote repository:
    Code:
    git fetch origin <BranchName>
    If you want to make changes to that branch, you need to create a local version of it:
    Code:
    git checkout -b <BranchName> origin/<BranchName>
    If you want to make changes to a new branch, based on an existing branch:
    Code:
    git checkout -b <NewBranchName> origin/<BranchName>
    Now that you have a working branch, you can make changes and commit as appropriate:
    Code:
    <Make Changes>
    git add <Changed files>
    git commit
    <Make Changes>
    git add <Changed files>
    <Make Changes>
    git add <Changed files>
    git commit
    ...
    If you decide that the last commit(s) are bad and you haven't pushed them out to the central repo yet, you can
    Code:
    git reset --hard HEAD^
    to "erase" the last 1 commimt or
    Code:
    git reset --hard HEAD~4
    to "erase" the last 4 (obviously you can replace 4 with another number).

    If you decide that a previous commit (doesn't have to be the last one) was bad, but you have already pushed it out, then you can revert it with
    Code:
    git revert <commitid>
    where <commitid> is the id assigned to the commit you want to revert (you can get this from the "git log" command). This creates a new commit that undoes the changes in your old one.

    I strongly urge against using the "git rebase" commands unless you really know what you're doing, especially if you've already pushed the changes.

    If you want to merge another branch into yours:
    Code:
    git merge --no-ff <OtherBranchRef>
    Don't forget the --no-ff if merging, otherwise it may perform a "fast forward" instead of a proper merge and it won't be recorded in the history (it will just change the branch to be derived from the one your merging). The <OtherBranchRef> can be a tag name, a branch name, or a commit ref (long hexadecimal string).

    If you've reached a milestone (like a release), you can tag it:
    Code:
    git tag -s <TagName>
    The -s indicates to sign the tag (you'll need an OpenPGP key for that).

    Once you're ready to push a branch back up to the online repository:
    Code:
    git push origin <BranchName>

    If you have problems, I can often times be reached via XMPP (Jabber) at jabber@keithmoyer.com or email me at tomatovpn@keithmoyer.com.
     
  18. Victek

    Victek Network Guru Member

    I'm reading, thanks for the cookbook :)
     
  19. SgtPepperKSU

    SgtPepperKSU Network Guru Member

    For those interested in helping in updating OpenSSL, I've made my openssl branch available.

    I don't want to push it to the repo.or.cz repository until we have it figured out, so it's just on a local server (so be kind and don't repeatedly hammer it please :smile:, and uptime is not guaranteed).

    You can grab it with:
    Code:
    git remote add -t openssl SgtPepperKSU git://git.keithmoyer.com/tomato.git
    git fetch SgtPepperKSU
    If you run this in your repository that you've cloned from repo.or.cz (the one I previously posted), those commands will only download the few commits needed. If you run this in a new repository, it will download everything needed (including linksys and tomato code).

    The new branch will be part of your repo as the remote branch SgtPepperKSU/openssl (just like the origin/* branches). Just like before, to start coding on this branch:
    Code:
    git checkout -b openssl SgtPepperKSU/openssl
    I guess this also goes to show one of the strengths of git: it is decentralized. There is nothing special about the repo.or.cz repository; pushing/pulling between peers is seen just like pulling/pushing with a central repository, and commits are shared seemlessly.
     
  20. mstombs

    mstombs Network Guru Member

    Anyone give me a pointer to what I'm doing wrong?

    Code:
    user@Ubuntu:~/git$ git clone git://repo.or.cz/tomato.git tomato
    Initialized empty Git repository in /home/user/git/tomato/.git/
    remote: Counting objects: 60371, done.
    remote: Compressing objects: 100% (41839/41839), done.
    remote: Total 60371 (delta 16066), reused 60371 (delta 16066)
    Receiving objects: 100% (60371/60371), 191.70 MiB | 622 KiB/s, done.
    Resolving deltas: 100% (16066/16066), done.
    warning: remote HEAD refers to nonexistent ref, unable to checkout.
     
  21. SgtPepperKSU

    SgtPepperKSU Network Guru Member

    You've done nothing wrong. You just need to checkout a branch!
    Code:
    git checkout -b <branchname> origin/<branchname>
    Git doesn't know what branch you're going to try to checkout, so it tries to checkout "master", but we don't have a "master" branch. I've considered creating an empty "master" branch just to keep people from getting that warning.
     
  22. Toastman

    Toastman Super Moderator Staff Member Member

    Give Linux Mint a try...

    As there seems to be a number of references to the git on the forum recently, just a tip for those wishing to try it out.

    The most popular distro, Ubuntu, has a regular habit of breaking things that used to work in their latest releases. Ubuntu 9.10 (32 bit) has proved a nightmare for me, earlier code which I know was fine, would not compile consistently no matter what I did. In the end I dumped it and tried to install an earlier version, but I was plagued with installation problems. Ubuntu would not recognize two of my WD 1TB "Green" hard disks, and the ATI graphics drivers did not work correctly. The older versions 9.04 and 8.04 had both previously worked for me, on different hardware, but no joy ...

    I just spent 4 days trying many different distros, which were mostly about as useful as a chocolate teapot - except that you can at least eat the teapot.

    Finally, Linux Mint Helena installed correctly, recognized all of my hardware (which is all new and mainstream) - the online repositories all work properly, and everything you need to compile Tomato was readily available and installed without a hitch. It compiles both old and new code reliably with no weird problems. Bash shell is standard.

    It is my hope that this may save someone a lot of hassle !
     
  23. rhester72

    rhester72 Network Guru Member

    Interesting. I've (thus far) had no problem compiling under 9.10 Server.

    Rodney
     
  24. Toastman

    Toastman Super Moderator Staff Member Member

    9.10 compiled all of Fedor's code, and the original Linksys base, from the git. Anything in between suffered from several strange problems, many people offered suggestions as to what may be wrong but nothing helped. Victek's 1.25.8515.2 ND code (from a zipped tree on his website) would compile 2 weeks ago, but a week later suddenly refused to compile with the same error messages as I got from the git code. But - I had not done anything that I am aware of. Even re-installed a couple of times, with the same result.

    I had stuck with it because others said they had no problem, but in the end - Linux Mint to the rescue :tongue: Watching Victek's code fly through to the end felt rather like being let out of prison !
     
  25. ehunt123

    ehunt123 Networkin' Nut Member

    Is there a functional QEmu or VMWare image somewhere? I've had on and off success with a jailed *wrt environment for just making packages on Ubuntu, and they tend to be up-to-date with libraries. It would be much easier to just run a mipsel/whatever environment via qemu
     
  26. alien3456

    alien3456 Addicted to LI Member

    I've been wanting to do a frontend modification of Tomato to make the UI a little snappier and a bit more organized. Another topic mentioned altering the graphs and that got me thinking of all the different things I could improve. Which source do you think would be most logical to use as a base, in order for other mod branches to easily adopt my changes? I'll most likely not touch any of the core code, only the ASP/CSS or other web UI related files.
     
  27. SgtPepperKSU

    SgtPepperKSU Network Guru Member

    You should always use the least specific branch possible as a base, so that it can be used in as wide of a range of other branches as possible. So, using the "tomato" branch would be the way to go.
     

Share This Page