1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

SCP won't run in startup script

Discussion in 'Tomato Firmware' started by bdf0506, Feb 25, 2013.

  1. bdf0506

    bdf0506 Serious Server Member

    I simply can't get an scp command to run in my startup script on tomato. My startup script simply adds the device it is connecting to into the known_hosts file, so that it won't prompt me to add the key to the known_hosts file, but the scp command will not complete.

    The scp line runs just fine if i run it in the command line by itself, but just won't run within my script. Thoughts?

    Code:
    # add Router to the known_hosts file
    echo "192.168.2.1 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgwCE4nJun26QsxsltBqWAhhDhoCsUBSY338oQWBMuGfNs8a6XR2c2/UZ80m4fV9myLKxMgE+ejoy8hqLaGJZC8OeOcBxiPS4Y5tQHurCemkGa4/jOb+RBfayFYhmtVN3K8TFXDAKmMLCv78023qZmsA8jC21Wno32sdfdseM9sAXT" > /tmp/home/root/.ssh/known_hosts
     
    scp -r -i /tmp/etc/dropbear/dropbear_rsa_host_key root@192.168.2.1:/mnt/sda1/Router/Utilities/* /tmp/home/root
    
     
  2. jerrm

    jerrm Network Guru Member

    Where do you have it? If in init without a wait condition, then the net, the drive, or both may not be available yet.
     
  3. bdf0506

    bdf0506 Serious Server Member

    This is in the init script. I know that the drive is available. I tried to stick in a sleep of 120 seconds before the scp tries to run, but still no luck. Is there some other wait condition that I should be sticking in the script? If I run the script from the command line once the device is up (/tmp/script_init.sh), it works fine, but I need this to run every time the device reboots without human intervention.

    Code:
    sleep 30
     
    # add Router to the known_hosts file
    echo "192.168.2.1 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgwCE4nJun26QsxsltBqWAhhDhoCsUBSY338oQWBMuGfNs8a6XR2c2/UZ80m4fV9myLKxMgE+ejoy8hqLaGJZC8OeOcBxiPS4Y5tQHurCemkGa4/jOb+RBfayFYhmtVN3K8TFXDAKmMLCv78jgvtZmsA8jC21Wno32fm8iqSKhSM9sAXT" > /tmp/home/root/.ssh/known_hosts
     
    sleep 180
     
    scp -r -i /tmp/etc/dropbear/dropbear_rsa_host_key root@192.168.2.1:/mnt/sda1/Router/Utilities/* /tmp/home/root
    
     
  4. koitsu

    koitsu Network Guru Member

    Regarding the echo: bummer about having to do this. Dropbear ssh has a flag called -y that auto-accepts any host key (without prompting); the scp source code is supposed to pass any flag you give -o on to ssh. However, that code has a bug:

    Code:
    326        while ((ch = getopt(argc, argv, "dfl:prtvBCc:i:P:q1246S:o:F:")) != -1)
    327                switch (ch) {
    ...
    336                case 'o':
    ...
    340                        addargs(&args, "-%c%s", ch, optarg);
    341                        break;
    
    This effectively passes the argument "-oy" on to ssh (it should be passing just -y), and then complains about "ignoring blah blah". Proof:

    Code:
    root@gw:/tmp/etc# scp -p shadow -oy jdc@192.168.1.51:
    WARNING: Ignoring unknown argument '-oy'
     
    Host '192.168.1.51' is not in the trusted hosts file.
    (fingerprint md5 XXX)
    Do you want to continue connecting? (y/n)
    
    I'm amazed nobody has caught this bug. I'm even looking at the latest source I just downloaded and it's still there. I'll contact the author and let him know; fixing this is super easy (one line of code).

    Regarding Scripts: there's really no easy way to debug these, I'm sorry to say. You usually have to redirect stdout and stderr in a command to a flag log file somewhere, e.g. you might have to do this:

    Code:
    touch /tmp/init.log
    scp -r -i /tmp/etc/dropbear/dropbear_rsa_host_key root@192.168.2.1:/mnt/sda1/Router/Utilities/* /tmp/home/root 1>>/tmp/init.log 2>&1
    echo "exit code: $?" >> /tmp/init.log
    
    And then go look at /tmp/init.log manually afterward to see what happened.

    Guesses:

    1. Specify the full path to scp, i.e. /usr/bin/scp; possibly the built-in default for $PATH does not have /usr/bin (I'm not sure).

    2. scp wants a pty/tty (for interaction), and isn't being given one since it's running from within a daemon. You could work around this using scp -oT (i.e. ssh -T which inhibits allocation of a tty), but this has the same bug as -oy as above. Cute eh. And don't try using -t natively -- that actually isn't documented in the scp usage syntax, but it actually means/does something -- the same goes for -d and -f. They're special; you'd have to see the source code in scp.c to understand.

    3. The use of an asterisk (*) is causing complications, e.g. the underlying /bin/sh is trying to expand it first and can't. You could try escaping it, but I would recommend putting your entire root@blah:/some/path/* into single quotes (not double). I've verified from the CLI that this does work.

    4. Try putting the entire script in Init -> Firewall instead. This gets executed after the iptables bits get added; possibly your above scp fails because there's no route due to iptables rules not being added yet, e.g. scp returns some cryptic error relating to the state of the firewall stack.

    You could solve all of these problems if you actually made use of a CIFS/SMB mount at run-time instead (it's possible to auto-run an entire script when a CIFS/SMB mount is made -- and I'm not talking about the Execute When Mounted GUI field, I'm talking about a mount.autorun file). In other words, the router (192.168.1.1) initiates a CIFS/SMB mount of something on 192.168.2.1, and once that's done, then runs a script called mount.autorun that's stored there, which proceeds to automatically populate your key and do an scp or cp or whatever else (all depends on what you want)). I know it sounds like a crummy workaround this is effectively what I do. This is my mount.autorun script which properly creates /opt on my TomatoUSB router (and handles situations like where you click Save in some spots causing multiple bind mounts to be made, which causes problems):

    Code:
    #!/bin/sh
    #
    # automount script for /cifs1
    #
    # When /cifs1 is mounted by TomatoUSB, this script will get run
    # automatically.  The reason we need this script rather than use
    # the "Execute When Mounted" features is that clicking "Save"
    # under Basic -> Network results in the "Executed When Mounted"
    # command being run mindlessly -- and we end up with two /opt
    # bindable mounts on the system simultaneously.
    #
    
    if /bin/grep -q /opt /proc/mounts
    then
      /bin/umount /opt
    fi
    
    /bin/mount -o bindable /cifs1/opt /opt
    
     
  5. bdf0506

    bdf0506 Serious Server Member

    Thanks for the long response, unfortunately, nothing you said worked. :(

    I put the output of the script to a file, and the issue is just that it just can't recognize that a key is already in the known_hosts file. But what I don't get is that I can run each command seperately from the command line and it works without any issues.

    When my script looks like this:
    Code:
    # add Router to the known_hosts file
    echo "192.168.2.1 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgwCE4nJun26QsxsltBqWAhhDhoCsUBSY338oQWBMuGfNs8a6XR2c2/UZ80m4fV9myLKxMgE+ejoy8hqLaGJZC8OeOcBxiPS4Y5tQHurCemkGa4/jOb+RBfayFYhmtVN3K8TFXDAKmMLCv78023qZmsA8jC21Wno32fm8iqSKhSM9sAXT" > /tmp/home/root/.ssh/known_hosts
     
    touch /tmp/init.log
    scp -r -i /tmp/etc/dropbear/dropbear_rsa_host_key root@192.168.2.1:/mnt/sda1/Router/Utilities/* /tmp/home/root 1>>/tmp/init.log 2>&1
    echo "exit code: $?" >> /tmp/init.log
    Output to the log file is this:
    Code:
    root@brad-wap:/tmp/home/root# cat /tmp/init.log 
    /usr/bin/dbclient: Warning: failed creating //.ssh: Read-only file system
     
    Host '192.168.2.1' is not in the trusted hosts file.
    (fingerprint md5 d4:af:ab:ec:9d:96:76:d6:9d:fd:6c:d1:ac:9e:ab:36)
    Do you want to continue connecting? (y/n) /usr/bin/dbclient: connection to root@192.168.2.1:22 exited: Didn't validate host key
    exit code: 1
    root@brad-wap:/tmp/home/root# 
     
    But if i start it with no script running at all, and I run the commands from the command line, it works fine.

    Code:
    root@brad-wap:/tmp/home/root# echo "192.168.2.1 ssh-rsa AAAAB3NzaC1yc2EAAAADAQAB
    AAAAgwCE4nJun26QsxsltBqWAhhDhoCsUBSY338oQWBMuGfNs8a6XR2c2/UZ80m4fV9myLKxMgE+ejoy
    8hqLaGJZC8OeOcBxiPS4Y5tQHurCemkGa4/jOb+RBfayFYhmtVN3K8TFXDAKmMLCv78023qZmsA8jC21
    Wno32fm8iqSKhSM9sAXT" > /tmp/home/root/.ssh/known_hosts
    root@brad-wap:/tmp/home/root# scp -r -i /tmp/etc/dropbear/dropbear_rsa_host_key
    root@192.168.2.1:/mnt/sda1/Router/Utilities/* /tmp/home/root
    root@brad-wap:/tmp/home/root#
    
    The one thing that I see that may cause concern is this, but I don't know if that's a big deal or not.... the .ssh folder already exists based on the NVRAM having an authorized key in the ssh folder.
    Code:
    /usr/bin/dbclient: Warning: failed creating //.ssh: Read-only file system
    Finally, to answer some of the questions from the last thread - I'm using the E1000 as simply a wireless access point, so the WAN never really comes up for the firewall script to apply. I have my router as an E3000, that copies over the applicable files from the attached usb drive to the root of the E3000. Then the WAP tries to copy the files from the router to the WAP with SCP, but that simply fails and requires me to do it manually. I've thought about trying a cron job or something, but it's really only a one time copy that I need to occur once the device boots up the first time.
     
  6. rhester72

    rhester72 Network Guru Member

    It looks like your script may be running before the root home directory has been built. I didn't review the entire thread, but have you tried doing this at WAN Up (so it's as late in the process as possible)?

    Rodney
     
  7. bdf0506

    bdf0506 Serious Server Member

    This is operating as a wireless bridged access point, so the WAN never comes up. I've tried giving it a sleep of 5 minutes, but it still won't run.
     
  8. rhester72

    rhester72 Network Guru Member

    Let's try to confirm the theory. Can you add "ls -l /tmp/home/root 2>&1" to the beginning of your script so we can see whether it's even there at the time you're running? I'd also consider adding "set" to the script underneath it so we can get a look at all of the environment variables at that time.

    Rodney
     
  9. bdf0506

    bdf0506 Serious Server Member

    Still nothing.

    I used this startup script:

    Code:
    # add Router to the known_hosts file
    echo "192.168.2.1 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgwCE4nJun26QsxsltBqWAhhDhoCsUBSY338oQWBMuGfNs8a6XR2c2/UZ80m4fV9myLKxMgE+ejoy8hqLaGJZC8OeOcBxiPS4Y5tQHurCemkGa4/jOb+RBfayFYhmtVN3K8TFXDAKmMLCv78023qZmsA8jC21Wno32fm8iqSKhSM9sAXT" > /tmp/home/root/.ssh/known_hosts
     
    touch /tmp/init.log
     
    echo "start" > /tmp/home/root/start
     
    echo "ls -l /tmp/home/root" 1>>/tmp/init.log 2>&1
    ls -l /tmp/home/root 1>>/tmp/init.log 2>&1
     
    echo "" 1>>/tmp/init.log 2>&1
     
    echo "ls -l /tmp/home/" 1>>/tmp/init.log 2>&1
    ls -l /tmp/home/ 1>>/tmp/init.log 2>&1
     
    echo "" 1>>/tmp/init.log 2>&1
     
    echo "ls -l /tmp/home/root/.ssh" 1>>/tmp/init.log 2>&1
    ls -l /tmp/home/root/.ssh 1>>/tmp/init.log 2>&1
     
    echo "" 1>>/tmp/init.log 2>&1
     
    echo "cat /tmp/home/root/.ssh/known_hosts" 1>>/tmp/init.log 2>&1
    cat /tmp/home/root/.ssh/known_hosts 1>>/tmp/init.log 2>&1
     
    echo "" 1>>/tmp/init.log 2>&1
     
    echo "scp -r -i /tmp/etc/dropbear/dropbear_rsa_host_key root@192.168.2.1:/mnt/sda1/Router/Utilities/* /tmp/home/root" 1>>/tmp/init.log 2>&1
    scp -r -i /tmp/etc/dropbear/dropbear_rsa_host_key root@192.168.2.1:/mnt/sda1/Router/Utilities/* /tmp/home/root 1>>/tmp/init.log 2>&1
     
    echo "" 1>>/tmp/init.log 2>&1
     
    echo "ls -l /tmp/home/root" 2>&1
    ls -l /tmp/home/root 2>&1
     
    echo "" 1>>/tmp/init.log 2>&1
     
    echo "ls -l /tmp/home/" 1>>/tmp/init.log 2>&1
    ls -l /tmp/home/ 1>>/tmp/init.log 2>&1
     
    echo "exit code: $?" >> /tmp/init.log
     
    echo "done" > /tmp/home/root/end
    output is this:

    Code:
    root@brad-wap:/tmp/home/root# cat /tmp/init.log
    ls -l /tmp/home/root
    -rw-r--r--    1 root    root            6 Feb 26 18:44 start
     
    ls -l /tmp/home/
    drwx------    3 root    root            80 Feb 26 18:44 root
     
    ls -l /tmp/home/root/.ssh
    -rwx------    1 root    root          394 Dec 31  1969 authorized_keys
    -rw-r--r--    1 root    root          225 Feb 26 18:44 known_hosts
     
    cat /tmp/home/root/.ssh/known_hosts
    192.168.2.1 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgwCE4nJun26QsxsltBqWAhhDhoCsUBSY338oQWBMuGfNs8a6XR2c2/UZ80m4fV9myLKxMgE+ejoy8hqLaGJZC8OeOcBxiPS4Y5tQHurCemkGa4/jOb+RBfayFYhmtVN3K8TFXDAKmMLCv78023qZmsA8jC21Wno32fm8iqSKhSM9sAXT
     
    scp -r -i /tmp/etc/dropbear/dropbear_rsa_host_key root@192.168.2.1:/mnt/sda1/Router/Utilities/* /tmp/home/root
    /usr/bin/dbclient: Warning: failed creating //.ssh: Read-only file system
     
    Host '192.168.2.1' is not in the trusted hosts file.
    (fingerprint md5 d4:af:ab:ec:9d:96:76:d6:9d:fd:6c:d1:ac:9e:ab:36)
    Do you want to continue connecting? (y/n) /usr/bin/dbclient: connection to root@192.168.2.1:22 exited: Didn't validate host key
     
     
    ls -l /tmp/home/
    drwx------    3 root    root            80 Feb 26 18:44 root
    exit code: 0
    root@brad-wap:/tmp/home/root# 
    I still think the issue has something to do with this, but don't know what it means or how to solve that.

    Code:
    /usr/bin/dbclient: Warning: failed creating //.ssh: Read-only file system
     
  10. jerrm

    jerrm Network Guru Member

    "failed creating //.ssh" probably tells you what you need to know.

    My bet is $HOME is not pointing to '/root' in the init script, but is instead set to '/'. Set the variable and see what happens.

    Add "set > /tmp/init.log" as the first line of the script to see.
     
  11. koitsu

    koitsu Network Guru Member

    This would indicate dropbear's dbclient is relying on the $HOME environment variable, or alternately using getpwent() (in which case setting $HOME does no good). I'll have to look at the code.

    BTW, for those wondering: dbclient is part of dropbear and is what contains important bits used with ssh. scp in dropbear is literally just a "weird wrapper" around ssh.
     
  12. koitsu

    koitsu Network Guru Member

    Checked the code -- it uses $HOME (you're lucky!) if it's set, otherwise if not, it falls back to getpwuid(), which per passwd(5) for root is /. Hence how the path //.ssh (with double slashes) is created. Here's the code:

    Code:
    (16:36:25 jdc@icarus) ~/tmp/dropbear-2012.55 $ grep "failed creating" *
    cli-kex.c:                              dropbear_log(LOG_INFO, "Warning: failed creating %s/.ssh: %s",
     
    (16:36:26 jdc@icarus) ~/tmp/dropbear-2012.55 $ vi cli-kex.c
    ...
    149 static FILE* open_known_hosts_file(int * readonly)
    150 {
    151        FILE * hostsfile = NULL;
    152        char * filename = NULL;
    153        char * homedir = NULL;
    154
    155        homedir = getenv("HOME");
    156
    157        if (!homedir) {
    158                struct passwd * pw = NULL;
    159                pw = getpwuid(getuid());
    160                if (pw) {
    161                        homedir = pw->pw_dir;
    162                }
    163        }
    164
    165        if (homedir) {
    166                unsigned int len;
    167                len = strlen(homedir);
    168                filename = m_malloc(len + 18); /* "/.ssh/known_hosts" and null-terminator*/
    169
    170                snprintf(filename, len+18, "%s/.ssh", homedir);
    171                /* Check that ~/.ssh exists - easiest way is just to mkdir */
    172                if (mkdir(filename, S_IRWXU) != 0) {
    173                        if (errno != EEXIST) {
    174                                dropbear_log(LOG_INFO, "Warning: failed creating %s/.ssh: %s",
    175                                                homedir, strerror(errno));
    176                                TRACE(("mkdir didn't work: %s", strerror(errno)))
    177                                goto out;
    178                        }
    179                }
    180
    181                snprintf(filename, len+18, "%s/.ssh/known_hosts", homedir);
    182                hostsfile = fopen(filename, "a+");
    
    Should be quite obvious what's going on there, and exactly how the error manifests itself. So yep, I'dsay add this to the top of your script:

    export HOME=/root

    And then give the script a try. The export part is important.
     
  13. bdf0506

    bdf0506 Serious Server Member

    that did it, thanks everyone! Such a simple little thing too.

    Final script that WORKS using scp:

    Code:
    sleep 30
     
    # add Router to the known_hosts file
    echo "192.168.2.1 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgwCE4nJun26QsxsltBqWAhhDhoCsUBSY338oQWBMuGfNs8a6XR2c2/UZ80m4fV9myLKxMgE+ejoy8hqLaGJZC8OeOcBxiPS4Y5tQHurCemkGa4/jOb+RBfayFYhmtVN3K8TFXDAKmMLCv78023qZmsA8jC21Wno32fm8iqSKhSM9sAXT" > /tmp/home/root/.ssh/known_hosts
     
    export HOME=/root
     
    scp -r -i /tmp/etc/dropbear/dropbear_rsa_host_key root@192.168.2.1:/mnt/sda1/Router/Utilities/* /tmp/home/root
    
    I would like to be able to prevent adding key manually to the known_hosts file, so hopefully one day a new release will address that. Thanks again!
     
  14. koitsu

    koitsu Network Guru Member

    I've sent an Email to the author of Dropbear, and included a patch, explaining the issue. Here's that mail + the patch. He has not responded at this time.

    Code:
    From: Jeremy Chadwick <jdc@koitsu.org>
    To: matt@ucc.asn.au
    Date: Sun, 24 Feb 2013 22:15:03 -0800
    Subject: Dropbear scp: -o broken (patch included)
     
    Matt,
     
    I found this while helping a Tomato/TomatoUSB user today:
     
    scp's -o flag is supposed to pass the optarg (and only that) on to ssh.
    However presently it treats it the same as -c, -i, or -F, which is
    incorrect.
     
    The test case I found: "scp -oy" gets turned into "ssh -oy" when it
    should get turned into "ssh -y".
     
    Below is a patch.  I don't know if there are any security implications
    by doing this.
     
    An alternate workaround would be to add a -y flag to scp.c (i.e. treated
    the same as the existing -[1246C] flags in scp.c).
     
    Also, I must say something because this really frustrated me when trying
    to find usage syntaxes and so on (particularly that there is no scp.1
    man page or equivalent):
     
    The scp.c usage() function should really be rewritten to look more like
    printhelp().  Furthermore, there are options in scp.c which aren't
    documented (such as -d, -f, and -t, which are passed on to a server
    command as arguments when calling ssh).
     
    If you agree but don't have the time let me know and I'll gladly provide
    a patch for both printhelp() (to clean up formatting/make it look
    prettier) and for usage().
     
    I'd also suggest changing the -o description from "ssh_option" to
    "ssh_flag"; when I saw "ssh_option" I assumed that it took the same kind
    of long option names as OpenSSH (e.g. -oStrictHostKeyChecking=no).  If I
    ended up cleaning up usage() I would simply add a line at the bottom
    that explain that -o just passes on whatever flag you give it to the ssh
    client.
     
    Thank you!
     
    --
    | Jeremy Chadwick                                  jdc@koitsu.org |
    | UNIX Systems Administrator                http://jdc.koitsu.org/ |
    | Mountain View, CA, US                                            |
    | Making life hard for others since 1977.            PGP 4BD6C0CB |
    
     

    Attached Files:

  15. koitsu

    koitsu Network Guru Member

    I've been hit up a couple times about the status of this (both in PM and in Email).

    I never received a response from the Dropbear author. The software has been actively maintained (commits as recent as a month ago, and a release made late last month).

    github (which is a read-only copy and is configured to not allow "Issues" / bug reporting) shows that said diff/fix has not been applied even in master:

    https://github.com/mkj/dropbear/blob/master/scp.c#L337

    So I don't know why my Email was ignored, but it looks like I'll have to send an Email to the dropbear mailing list and probably endure some kind of social chastising/argument/battle over this (I'm being pessimistic, but it's based on 20+ years of experience dealing with OSS and software devs and so on).
     

Share This Page