1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

I've created a command-line NVRAM cfg file parser

Discussion in 'Tomato Firmware' started by Charles Phillips, Mar 20, 2014.

  1. Charles Phillips

    Charles Phillips Reformed Router Member

    I built a tool using nodejs to parse, compare, and repack tomato .cfg files. Presenting:

    nvram-cfg-parser
    https://github.com/doublerebel/nvram-cfg-parser

    Changelog:
    1.0.0
    Support for ARM (tomato-arm) build .cfg format

    0.0.6 Standard tomato build .cfg format decoding, encoding, and diff

    What it does
    Decodes Tomato cfg files into JSON so they can be changed and compared against other backups. Encodes JSON-formatted Tomato cfg key/value pairs into the Tomato cfg format.

    Installation
    nvram-cfg-parser is available from npm, the standard package manager included with nodejs.

    Install from npm:
    # npm install -g nvram-cfg-parser

    Usage:

    decode
    nvramcfg decode <filename>

    Command example:

    $ nvramcfg decode tomato_v128_m943394.cfg

    Output example (keys in no particular order):
    {
    "0|31|||word text\n^begins-with.domain.\n.ends-with.net$\n^www.exact-domain.net$|0|exampl": "|1320|300|31|||word text\n^begins-with.domain.\n.ends-with.net$\n^www.exact-domain.net$|0|example",
    "wl_mac_deny": "",
    "wl_radius_port": "1812",
    "sb/1/ofdm2gpo": "0x44444444",
    "pptp_client_mru": "1450",
    "https_crt": "",
    "qos_reset": "1",
    ...


    decode to file
    $ nvramcfg decode tomato_v128_m943394.cfg > tomato_v128_m943394.json

    encode

    nvramcfg encode <format> <filename>

    Command example:

    $ nvramcfg encode tomato_v128_m943394-altered.json > tomato_v128_m943394-altered.cfg

    colorized json diff
    nvramcfg diff <filename1> <filename2>

    Command example:

    $ nvramcfg diff tomato_v128_m943394.json tomato_v128_m943394-altered.json

    How it works

    Original
    The tomato_vxxx_xxxxx.cfg files are gzipped utf-8 text with null characters bounding and separating the key=value pair sets. We unzip the file, strip the header and footer, and read the null-separated key=value pairs.

    ARM
    The tomato_vxxx_xxxxx.cfg files are not gzipped, but are an obfuscated version of the original utf-8-with-null-separators. We read the random value and de-obfuscate back to the original format.

    Programmatic usage in JavaScript
    Decode and encode are available from the NvramParser class. See comments in src/nvram-parser.iced, src/nvram-arm-parser.iced, for details.

    FAQ
    • How to tell if my router build is ARM format?
      The tomato builds for your router contain ARM in the filename. Example: tomato-RT-AC56U-AT-ARM-2.7-128-AIO-64K.trx
    MIT Licensed. Use at your own risk.

    ----------------

    Special thanks to threads inspiring this tool with their stories of long manual exporting, hex editing and cut and pasting:
    Edit .conf files, save to NVRAM how? http://tomatousb.org/forum/t-297838/
    Restoring config to the same OR another router. http://www.linksysinfo.org/index.ph...orial-and-discussion.28349/page-3#post-138676
    Read a tomato backup config file http://www.linksysinfo.org/index.php?threads/read-a-tomato-backup-config-file.33772/
    NVRAM Export can not handle wrap text? http://www.linksysinfo.org/index.php?threads/nvram-export-can-not-handle-wrap-text.68478/
    Backup/Restore http://www.linksysinfo.org/index.php?threads/backup-restore.23087/

    Recover DSL password from Tomato firmware? http://www.linksysinfo.org/index.php?threads/recover-dsl-password-from-tomato-firmware.31273/
    tomato config file http://www.linksysinfo.org/index.php?threads/tomato-config-file.33012/


    I wrote this in nodejs in hopes it would run completely cross-platform. So far this software has been tested on Linux and Mac. Windows should work the same. I have already decoded, diff'd, patched, encoded, and updated my router config using this software.

    In the future possibilities...
    Packaging this software as "normally" (without npm) downloadable, installable app on all platforms. (Because the Tomato cfg is a binary format, this javascript will not run in the browser. That would be a huge security risk anyway, passwords/keys are stored in plaintext.)

    Blacklists and whitelists of settings. This would enable porting configs between routers, or cleaning out passwords/keys before sending config for tech help review.

    Merging configurations. A command-line prompted old/new/blank for each option.

    This code is open-source and I also welcome any advice or corrections from users more familiar with Tomato software.

    Cheers,

    Charles
     
    Last edited: May 17, 2015
    Wolfgan, VoYaGeRTM, mmosoll and 3 others like this.
  2. lancethepants

    lancethepants Network Guru Member

    Very cool. Looks like a very helpful tool.
     
  3. Charles Phillips

    Charles Phillips Reformed Router Member

    Thanks lance!! Hopefully this tool will save many configuration-related headaches. First I would like to create the whitelist of portable features, so that configuration can be moved to another router without conflict.

    Based on some of the inspiration posts above, the settings could include:

    cifs*
    cstats_
    ddnsx[0_]
    dhcp*
    dhcp_
    dhcpd_
    dns_
    http_lanport
    http_passwd
    https_
    lan_hostname
    lan_hostname=
    lan_ipaddr
    lan_ipaddr=
    lan_proto
    log_
    macnames
    ntp_
    pppoe_passwd=
    pppoe_username=
    qos*
    router_name
    rrule*
    rstats_
    sch_
    script_
    sshd_
    tm_
    upnp_
    vpn_server1_
    vpn_server_eas
    wan_dns
    wan_domain
    wan_hostname
    wan_hostname=
    wan_proto
    wan_proto=
    web_css
    web_mx
    wl0_akm
    wl0_channel
    wl0_country
    wl0_crypto
    wl0_frameburst
    wl0_key
    wl0_mac
    wl0_mode
    wl0_nbw
    wl0_nctrlsb
    wl0_security_mode
    wl0_ssid
    wl0_txpwr
    wl0_wpa_psk
    wl_maclist
    wl_macmode



    Does anyone here keep a personal list of settings they always transfer? Any settings to avoid? If the whitelist can be tested to be rock-solid I can rewrite the config parser in ASP so that a config-migration can be done on the router itself.
     
  4. jnitis

    jnitis Reformed Router Member

    This had a ton of dependencies not mentioned in the installation instructions, gmake being one of them, node-gyp being another, etc.

    I'm stuck here now:

    Code:
    root@svn:~ # nvramcfg
    
    module.js:340
        throw err;
              ^
    Error: Cannot find module 'iced-coffee-script'
        at Function.Module._resolveFilename (module.js:338:15)
        at Function.Module._load (module.js:280:25)
        at Module.require (module.js:364:17)
        at require (module.js:380:17)
        at Object.<anonymous> (/usr/local/lib/node_modules/nvram-cfg-parser/nvram-parser.js:19:10)
        at Object.<anonymous> (/usr/local/lib/node_modules/nvram-cfg-parser/nvram-parser.js:198:4)
        at Module._compile (module.js:456:26)
        at Object.Module._extensions..js (module.js:474:10)
        at Module.load (module.js:356:32)
        at Function.Module._load (module.js:312:12)
    
    So then I try to..

    Code:
    root@svn:~ # npm install -g iced-coffee-script
    npm http GET https://registry.npmjs.org/iced-coffee-script
    npm http 304 https://registry.npmjs.org/iced-coffee-script
    npm http GET https://registry.npmjs.org/mkdirp
    npm http GET https://registry.npmjs.org/docco
    npm http GET https://registry.npmjs.org/iced-runtime
    npm http 304 https://registry.npmjs.org/mkdirp
    npm http 304 https://registry.npmjs.org/docco
    npm http 304 https://registry.npmjs.org/iced-runtime
    npm http GET https://registry.npmjs.org/fs-extra
    npm http GET https://registry.npmjs.org/marked
    npm http GET https://registry.npmjs.org/commander
    npm http GET https://registry.npmjs.org/underscore
    npm http GET https://registry.npmjs.org/highlight.js
    npm http 304 https://registry.npmjs.org/fs-extra
    npm http 304 https://registry.npmjs.org/marked
    npm http 304 https://registry.npmjs.org/commander
    npm http 304 https://registry.npmjs.org/highlight.js
    npm http 304 https://registry.npmjs.org/underscore
    npm http GET https://registry.npmjs.org/ncp
    npm http GET https://registry.npmjs.org/jsonfile
    npm http GET https://registry.npmjs.org/rimraf
    npm http 304 https://registry.npmjs.org/ncp
    npm http 304 https://registry.npmjs.org/rimraf
    npm http 304 https://registry.npmjs.org/jsonfile
    npm ERR! error rolling back Error: ENOTEMPTY, rmdir '/usr/local/lib/node_modules/iced-coffee-script/node_modules/docco/node_modules'
    npm ERR! error rolling back  docco@0.6.3 { [Error: ENOTEMPTY, rmdir '/usr/local/lib/node_modules/iced-coffee-script/node_modules/docco/node_modules']
    npm ERR! error rolling back   errno: 53,
    npm ERR! error rolling back   code: 'ENOTEMPTY',
    npm ERR! error rolling back   path: '/usr/local/lib/node_modules/iced-coffee-script/node_modules/docco/node_modules' }
    npm ERR! Error: ENOENT, lstat '/usr/local/lib/node_modules/iced-coffee-script/node_modules/docco/node_modules/highlight.js/docs/reference.rst'
    npm ERR! If you need help, you may report this log at:
    npm ERR!     <http://github.com/isaacs/npm/issues>
    npm ERR! or email it to:
    npm ERR!     <npm-@googlegroups.com>
    
    npm ERR! System FreeBSD 9.2-RELEASE-p9
    npm ERR! command "node" "/usr/local/bin/npm" "install" "-g" "iced-coffee-script"
    npm ERR! cwd /root
    npm ERR! node -v v0.10.12
    npm ERR! npm -v 1.2.28
    npm ERR! path /usr/local/lib/node_modules/iced-coffee-script/node_modules/docco/node_modules/highlight.js/docs/reference.rst
    npm ERR! fstream_path /usr/local/lib/node_modules/iced-coffee-script/node_modules/docco/node_modules/highlight.js/docs/reference.rst
    npm ERR! fstream_type File
    npm ERR! fstream_class FileWriter
    npm ERR! code ENOENT
    npm ERR! errno 34
    npm ERR! fstream_stack /usr/local/lib/node_modules/npm/node_modules/fstream/lib/writer.js:284:26
    npm ERR! fstream_stack Object.oncomplete (fs.js:107:15)
    npm ERR! Error: No compatible version found: mkdirp@'^0.5.0'
    npm ERR! Valid install targets:
    npm ERR! ["0.0.1","0.0.2","0.0.3","0.0.4","0.0.5","0.0.6","0.0.7","0.1.0","0.2.0","0.2.1","0.2.2","0.3.0","0.3.1","0.3.2","0.3.3","0.3.4","0.3.5","0.4.0","0.4.1","0.4.2","0.5.0"]
    npm ERR!     at installTargetsError (/usr/local/lib/node_modules/npm/lib/cache.js:719:10)
    npm ERR!     at /usr/local/lib/node_modules/npm/lib/cache.js:641:10
    npm ERR!     at RegClient.get_ (/usr/local/lib/node_modules/npm/node_modules/npm-registry-client/lib/get.js:101:14)
    npm ERR!     at RegClient.<anonymous> (/usr/local/lib/node_modules/npm/node_modules/npm-registry-client/lib/get.js:37:12)
    npm ERR!     at fs.js:266:14
    npm ERR!     at Object.oncomplete (fs.js:107:15)
    npm ERR! If you need help, you may report this log at:
    npm ERR!     <http://github.com/isaacs/npm/issues>
    npm ERR! or email it to:
    npm ERR!     <npm-@googlegroups.com>
    
    npm ERR! System FreeBSD 9.2-RELEASE-p9
    npm ERR! command "node" "/usr/local/bin/npm" "install" "-g" "iced-coffee-script"
    npm ERR! cwd /root
    npm ERR! node -v v0.10.12
    npm ERR! npm -v 1.2.28
    npm ERR!
    npm ERR! Additional logging details can be found in:
    npm ERR!     /root/npm-debug.log
    npm ERR! not ok code 0
    
    Any ideas?
     
  5. Charles Phillips

    Charles Phillips Reformed Router Member

    Just pushed version 1.0.0 to Github and npm. Original post is updated with new docs.

    There were a couple bugs in the previous version that may have resulted in one or two config lines being left out. But, the main change is that ARM versions of tomato are now supported! I was able to figure it out after poring over the C sources in tomato-arm.

    @jnitis apologies, the earlier version required iced-coffee-script package, this is no longer necessary. build-essentials and node-gyp requirements are probably due to the buffertools module which I use to manipulate the binary/hex buffers. I may be able to drop that requirement soon since node 0.12 is more widely available and has an updated Buffer module.

    HTH someone else out there! Cheers, Charles
     
    Goggy likes this.
  6. Charles Phillips

    Charles Phillips Reformed Router Member

    Here is a list of properties I successfully ported with this tool. Went from a Belkin Sharemax N300 (original, non-ARM) to an Asus RT-AC56U (ARM) router:

    block_wan
    ddnsx1
    ddnsx1_cache
    dhcp1_lease
    dhcp2_lease
    dhcp3_lease
    dhcp_num
    dhcp_start
    dhcpd_endip
    dhcpd_gwmode
    dhcpd_lmax
    dhcpd_startip
    dhcpd_static
    dns_intcpt
    http_enable
    http_passwd
    https_enable
    ipv6_accept_ra
    lan1_stp
    lan2_stp
    lan3_stp
    lan_hostname
    lan_ipaddr
    ntp_server
    portforward
    router_name
    smbd_autoshare
    smbd_cpage
    smbd_enable
    smbd_wgroup
    sshd_dsskey
    sshd_hostkey
    telnetd_eas
    tomatoanon_answer
    tomatoanon_enable
    ttb_css
    vpn_client1_addr
    vpn_client1_adns
    vpn_client1_cipher
    vpn_client1_cn
    vpn_client1_comp
    vpn_client1_custom
    vpn_client1_password
    vpn_client1_tlsremote
    vpn_client1_userauth
    vpn_client1_username
    vpn_client1_useronly
    vpn_client2_addr
    vpn_client2_adns
    vpn_client2_cipher
    vpn_client2_cn
    vpn_client2_comp
    vpn_client2_custom
    vpn_client2_hmac
    vpn_client2_password
    vpn_client2_port
    vpn_client2_tlsremote
    vpn_client2_userauth
    vpn_client2_username
    vpn_client2_useronly
    wan_hostname
    wan_lease
    watchdog
    web_css
    wl0_akm
    wl0_country_code
    wl0_lazywds
    wl0_mimo_preamble
    wl0_nctrlsb
    wl0_net_mode
    wl0_nmode
    wl0_nreqd
    wl0_plcphdr
    wl0_security_mode
    wl0_ssid
    wl0_wmf_bss_enable
    wl0_wpa_psk
    wl1_wme
    wl_country_code
    xtalfreq
    script_wanup


    I should cross-reference this with the list of properties above, I'm sure there are more but these covered most of my settings.

    Important note: ARM builds v128 and earlier can't restore a .cfg from the GUI. Instead the .cfg must be copied to the router (or on a USB key connected to the router) and restored from the command line:
    nvram restore </path/to/filename>
    nvram commit
    reboot


    I'm running AdvancedTomato which is still on v128. v129 tomato-arm from Shibby has the GUI .cfg bug fixed.
     
  7. ilium007

    ilium007 Reformed Router Member

    I get the following error when npm is trying to install nvram-cfg-parser:

    Code:
    MacBookAir:support ilium007$ sudo npm install -g nvram-cfg-parser
    
    > buffertools@2.0.1 install /usr/local/lib/node_modules/nvram-cfg-parser/node_modules/buffertools
    > node-gyp rebuild
    
      CXX(target) Release/obj.target/buffertools/buffertools.o
    ../buffertools.cc:36:10: error: unknown type name 'Arguments'; did you mean 'v8::internal::Arguments'?
                            const Arguments& args,
                                  ^~~~~~~~~
                                  v8::internal::Arguments
    /Users/ilium007/.node-gyp/0.12.5/deps/v8/include/v8.h:127:7: note: 'v8::internal::Arguments' declared here
    class Arguments;
          ^
    ../buffertools.cc:39:33: error: unknown type name 'Arguments'; did you mean 'v8::internal::Arguments'?
            Handle<Value> operator()(const Arguments& args) {
                                           ^~~~~~~~~
                                           v8::internal::Arguments
    /Users/ilium007/.node-gyp/0.12.5/deps/v8/include/v8.h:127:7: note: 'v8::internal::Arguments' declared here
    class Arguments;
          ^
    ../buffertools.cc:43:30: error: member access into incomplete type 'const v8::internal::Arguments'
                    Local<Object> target = args.This();
                                               ^
    /Users/ilium007/.node-gyp/0.12.5/deps/v8/include/v8.h:127:7: note: forward declaration of 'v8::internal::Arguments'
    class Arguments;
          ^
    ../buffertools.cc:46:38: error: type 'const v8::internal::Arguments' does not provide a subscript operator
                    } else if (Buffer::HasInstance(args[0])) {
                                                   ~~~~^~
    ../buffertools.cc:49:17: error: type 'const v8::internal::Arguments' does not provide a subscript operator
                            target = args[0]->ToObject();
                                     ~~~~^~
    ../buffertools.cc:51:55: error: no member named 'New' in 'v8::String'
                            return ThrowException(Exception::TypeError(String::New(
                                                                       ~~~~~~~~^
    ../buffertools.cc:55:16: error: no member named 'Close' in 'v8::HandleScope'
                    return scope.Close(static_cast<Derived*>(this)->apply(target, args, args_start));
                           ~~~~~ ^
    ../buffertools.cc:64:10: error: unknown type name 'Arguments'; did you mean 'v8::internal::Arguments'?
                            const Arguments& args,
                                  ^~~~~~~~~
                                  v8::internal::Arguments
    /Users/ilium007/.node-gyp/0.12.5/deps/v8/include/v8.h:127:7: note: 'v8::internal::Arguments' declared here
    class Arguments;
          ^
    ../buffertools.cc:67:33: error: unknown type name 'Arguments'; did you mean 'v8::internal::Arguments'?
            Handle<Value> operator()(const Arguments& args) {
                                           ^~~~~~~~~
                                           v8::internal::Arguments
    /Users/ilium007/.node-gyp/0.12.5/deps/v8/include/v8.h:127:7: note: 'v8::internal::Arguments' declared here
    class Arguments;
          ^
    ../buffertools.cc:71:30: error: member access into incomplete type 'const v8::internal::Arguments'
                    Local<Object> target = args.This();
                                               ^
    /Users/ilium007/.node-gyp/0.12.5/deps/v8/include/v8.h:127:7: note: forward declaration of 'v8::internal::Arguments'
    class Arguments;
          ^
    ../buffertools.cc:74:38: error: type 'const v8::internal::Arguments' does not provide a subscript operator
                    } else if (Buffer::HasInstance(args[0])) {
                                                   ~~~~^~
    ../buffertools.cc:77:17: error: type 'const v8::internal::Arguments' does not provide a subscript operator
                            target = args[0]->ToObject();
                                     ~~~~^~
    ../buffertools.cc:79:55: error: no member named 'New' in 'v8::String'
                            return ThrowException(Exception::TypeError(String::New(
                                                                       ~~~~~~~~^
    ../buffertools.cc:83:11: error: type 'const v8::internal::Arguments' does not provide a subscript operator
                    if (args[args_start]->IsString()) {
                        ~~~~^~~~~~~~~~~
    ../buffertools.cc:84:28: error: type 'const v8::internal::Arguments' does not provide a subscript operator
                            String::Utf8Value s(args[args_start]);
                                                ~~~~^~~~~~~~~~~
    ../buffertools.cc:85:17: error: no member named 'Close' in 'v8::HandleScope'
                            return scope.Close(static_cast<Derived*>(this)->apply(
                                   ~~~~~ ^
    ../buffertools.cc:93:31: error: type 'const v8::internal::Arguments' does not provide a subscript operator
                    if (Buffer::HasInstance(args[args_start])) {
                                            ~~~~^~~~~~~~~~~
    ../buffertools.cc:94:30: error: type 'const v8::internal::Arguments' does not provide a subscript operator
                            Local<Object> other = args[args_start]->ToObject();
                                                  ~~~~^~~~~~~~~~~
    ../buffertools.cc:95:17: error: no member named 'Close' in 'v8::HandleScope'
                            return scope.Close(static_cast<Derived*>(this)->apply(
                                   ~~~~~ ^
    fatal error: too many errors emitted, stopping now [-ferror-limit=]
    20 errors generated.
    make: *** [Release/obj.target/buffertools/buffertools.o] Error 1
    gyp ERR! build error
    gyp ERR! stack Error: `make` failed with exit code: 2
    gyp ERR! stack     at ChildProcess.onExit (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:269:23)
    gyp ERR! stack     at ChildProcess.emit (events.js:110:17)
    gyp ERR! stack     at Process.ChildProcess._handle.onexit (child_process.js:1074:12)
    gyp ERR! System Darwin 14.3.0
    gyp ERR! command "node" "/usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
    gyp ERR! cwd /usr/local/lib/node_modules/nvram-cfg-parser/node_modules/buffertools
    gyp ERR! node -v v0.12.5
    gyp ERR! node-gyp -v v2.0.1
    gyp ERR! not ok
    npm ERR! Darwin 14.3.0
    npm ERR! argv "node" "/usr/local/bin/npm" "install" "-g" "nvram-cfg-parser"
    npm ERR! node v0.12.5
    npm ERR! npm  v2.11.2
    npm ERR! code ELIFECYCLE
    
    npm ERR! buffertools@2.0.1 install: `node-gyp rebuild`
    npm ERR! Exit status 1
    npm ERR!
    npm ERR! Failed at the buffertools@2.0.1 install script 'node-gyp rebuild'.
    npm ERR! This is most likely a problem with the buffertools package,
    npm ERR! not with npm itself.
    npm ERR! Tell the author that this fails on your system:
    npm ERR!     node-gyp rebuild
    npm ERR! You can get their info via:
    npm ERR!     npm owner ls buffertools
    npm ERR! There is likely additional logging output above.
    
    npm ERR! Please include the following file with any support request:
    npm ERR!     /Users/ilium007/support/npm-debug.log
    MacBookAir:support ilium007$
    But I can install buffertools separately:

    Code:
    MacBookAir:support ilium007$ npm install buffertools
    \
    > buffertools@2.1.2 install /Users/ilium007/support/node_modules/buffertools
    > node-gyp rebuild
    
      CXX(target) Release/obj.target/buffertools/buffertools.o
      SOLINK_MODULE(target) Release/buffertools.node
    buffertools@2.1.2 node_modules/buffertools
    MacBookAir:support ilium007$
    
     
    Last edited: Jun 27, 2015
  8. ilium007

    ilium007 Reformed Router Member

    I got it working by changing the buffer tools dependancy in the package.json file. nvram-cfg-parser forked on github and modified. I have installed it below from my github repo.

    Code:
    MacBookAir:support ilium007$ npm install -g git+https://git@github.com/ilium007/nvram-cfg-parser.git
    
    > buffertools@2.1.2 install /usr/local/lib/node_modules/nvram-cfg-parser/node_modules/buffertools
    > node-gyp rebuild
    
      CXX(target) Release/obj.target/buffertools/buffertools.o
      SOLINK_MODULE(target) Release/buffertools.node
    /usr/local/bin/nvramcfg -> /usr/local/lib/node_modules/nvram-cfg-parser/nvramcfg.js
    nvram-cfg-parser@1.0.0 /usr/local/lib/node_modules/nvram-cfg-parser
    ├── is-gzip@1.0.0
    ├── iced-runtime@1.0.2
    ├── json-diff@0.3.1 (dreamopt@0.6.0, cli-color@0.1.7, difflib@0.2.4)
    └── buffertools@2.1.2
    MacBookAir:support ilium007$
     

Share This Page