RT-N16 with Shibby - Python Script file operations problems

Discussion in 'Tomato Firmware' started by phil4200, Feb 21, 2014.

  1. phil4200

    phil4200 Reformed Router Member

    Here's the basic question, all of the details are below: How do I allow a python script at /opt/scripts/Server.py, running in the router on OPTWARE, to read/write/append files in an opt/scripts/ folder by itself, running as a router, NOT with someone telneted in as root.

    I've been following this forum for several months, while working on my own Asus RT-N16 project. I initially loaded vanilla TomatoUSB onto it, but upgraded recently to Shibby: tomato-K26USB-1.28.RT-N5x-MIPSR2-116-AIO.trx. The purpose of my router project is to act as a USB serial port (via either pl2303.ko or FTDI_sio.ko and usbserial.ko) which accepts GPS/NMEA sentences from a USB Serial device and then serves them to a TCP/IP connection via a webserver written in Python, running on OPTWARE. The python script is located in /opt/scripts/Server.py with /opt being a USB Thumbdrive. I have an init script in the Shibby GUI which sets up the mounting scheme (rather than doing it in NVRAM) as suggested by Shibby in other posts, it insmods the drivers, and it also calls the python script in the init script.

    Everything initially worked great, until I started to expand the python webserver. I added the ability to create a data file that gets re-written with every new sentence, for future expansion capability. I also set up logging in python, which reads from a log.conf file, and also writes to a log file, and the server also writes to a .csv file to record "track" data (the GPRMC sentence). All of these are currently located in /opt/scripts/. I also would like to have it read from a general config file for future user customization.

    Now, when the Server.py script is run via the init scripts at startup none of the files are read or written by the python script. However, when I remove the script from the init scripts, and start it manually by telnetting in as root user, it has NO problems, and reads and writes all the files I need it to. So, I'm thinking that this is a permission issue.

    I have done chmod -R 777 to the opt/scripts/ folder to give everything in it full Read, Write, Execute ability, with no change to my script's ability to write the files. It still could not. I have also done chown -R nobody:nobody to opt/scripts/ and then chmod 777 -R as well, and still did not change its behaviour. So, I changed it back to "root:root _rwxr_xr_x" as it was when I installed everything fresh. Is "nobody" the user/group when the router is running?

    This is driving me crazy! I've searched and searched trying to aviod posting such a seemingly noob-ish question! I'm really hoping this is possible, and an easy fix! Thank you for your help, and thank you to all who make these great builds possible! Other than this small issue, caused by my lack of knowledge, everything works great!
  2. jerrm

    jerrm Network Guru Member

    Normally, pretty much everything in tomato is run as root unless you intentionally change the user. Do your scripts do anything to change the user to "nobody" or something else?

    Are you sure it is running at all? Are you waiting for the drive to be mounted in init before attempting to execute the script?
  3. phil4200

    phil4200 Reformed Router Member

    jerrm, thank you for the response.

    There is nothing in my scripts which change the user to "nobody". I just did some searching and found that some webservers run programs as "nobody". It was a shot in the dard that didn't work, nothing more.

    Yes, I am sure the program is running. I have made try/except (error handling) statements at all the file operations in the python script, so my program will actually run at router startup, it just skips all file ops, which is not what I want. So, in that sense I believe I've narrowed it down to the file operations being the problem, which leads me to think that it's a permission issue.

    Also, to clarify, it is the exact same script that I can run from the telnet as "root", and all of those file operations happen flawlessly.

    Thanks again.
  4. phil4200

    phil4200 Reformed Router Member

    Unfortunately there is no way for me to see the exact errors that are occuring because, as far as I know, there is no way to get the normal console output (std.out) when the program is running within the router. I think I have narrowed it down pretty well though, based on the file operations not working when running as the router user.

    I initially did not have the file ops error handling, and the script would not run.

    So, the question remains, how to run a Python Script in OPTWARE on Shibby and allow the script to read/write/append files in it's directory.
  5. jerrm

    jerrm Network Guru Member

    If you are not changing the user, and running from init (or other GUI script blocks), then you are running (or at least starting) as root.

    Redirect output (or at least stderr) to logger and look in syslog.

    Log the environment and ls -l the folders in question to /tmp or syslog before and after executing the script.

    90% of these issues are the USB isn't mounted when trying to run from init, or some sort of environment issue discrepency. For example, the optware profile stuff hasn't been run for the init environment.
  6. phil4200

    phil4200 Reformed Router Member

    I will research how to output stderr (at a minimum) to logger. Remember that my logger will not write to a file, so I don't know exactly how I set it up to output to syslog. Where is the syslog located?

    How do I tell TomatoUSB or OPTWARE to "Log the environment"?

    My only problem with the possibility that the USB isn't mounted yet is that I don't see how the script could initialize in the first place if the USB isn't mounted. If I am able to start the Server.py script via the init script block, that should mean that the directory it's in is also available, should it not?

    Thanks for working through this with me. I would love to get this sorted out!
  7. koitsu

    koitsu Network Guru Member

    1. What is /opt? Don't say "a USB flash drive" (that's not entirely true). How are you making /opt? Provide exact commands you're using, and where/when you're using them (this is probably related to #3).

    2. What filesystem type are you using for your /tmp/mnt/{flashdrivename} ? It matters. Please provide output from both df -k and mount on the system.

    3. How are you automatically running scripts/etc. at start up? Do not be vague: explain exactly how you're accomplishing this (chances are you're doing it in a fashion that isn't reliable; jerrm and I for example disagree on how to go about accomplishing this, but I don't have any hard evidence at this point that this is the problem).

    Basically what I'm asking for is all the actual commands/scripts/etc. you're using. Your explanation of the issue is appreciated but you haven't really explained any of the above. :)

    P.S. -- Are you sure you're using Optware? If so, scrap it and use Entware ("scrap it" means format the USB stick, by the way -- don't think rm -fr is enough, as Optware apparently installs a bunch of dotfiles that screw with your environment that you normally wouldn't see without ls -a and people often forget/don't know about that). I wouldn't be surprised if this turned out to be some wonky problem relating to conflicting libraries. I trust absolutely nothing on Optware, while the Entware folks understand how to build software correctly.

    Entware does offer python, as well as many python eggs (ex. python curl, python openssl, etc.).

    root@gw:/tmp/home/root# opkg list | egrep "^python "
    python - 2.7.3-2 - Python is a dynamic object-oriented programming language that can be used for many kinds of software development. It offers strong support for integration with other languages and tools, comes with extensive standard libraries, and can be learned in a few days. Many Python programmers report substantial productivity gains and feel the language encourages the development of higher quality, more maintainable code. . This package contains the full Python install.
  8. phil4200

    phil4200 Reformed Router Member


    I will answer all of your questions with outputs and code when I am home from work tonight. I am not sitting in front of the system at this time. Thank you for your detailed questions!

    I can say with almost 100% certainty that I am using optware. I installed it via shibby20's instructions in thread:

    there are other commands in the init block, but I know I have this:
    mount /dev/sda2 /opt

    and then I installed optware using:

    I went with optware because a) it was resident in the Shibby build that I used, and b) because Shibby20 (who I am guessing is, or is somehow professionally related to "Shibby") said: "IMO Entware is not 100% compatible with Tomato." But, I also see your posts beneath that one, so basically, call it general laziness on my part that I went with Optware. I would like to avoid starting over if I can, because I don't have tons of free time, but I'll do what it takes to get this working. It's so close, and works so well when I log in from telnet and run it that I can't believe it's going to be that difficult.

    I'm pretty new to all of this, and I'm by no means a programmer, just a tinkerer, so the help is very appreciated!

    I am also going to try initializing the script using $ python2.7 Server.py > /tmp/logfile.txt
    To see if I can see what some of the errors are.
  9. jerrm

    jerrm Network Guru Member

    Cut and paste the below into init, note where I say to edit. Post the results.

    echo =================
    echo env
    echo =================
    echo =================
    echo mount
    echo =================
    echo =================
    echo df
    echo =================
    echo =================
    echo ls -l /opt/bin
    echo =================
    /bin/ls -l /opt/bin
    echo =================
    echo ls -l /opt/scripts
    echo =================
    /bin/ls -l /opt/scripts
    echo =================
    ####### This looks problematic if the script doesn't place itself into background.
    ####### Also, where do these files reside? are they in the path? do they depend
    ####### on an initialized optware environment?
    echo My Program
    echo =================
    python2.7 Server.py
    echo =================
    echo ps
    echo =================
    echo =================
    echo -----  END  -----
    echo =================
    ) 2>&1 | /usr/bin/logger -st "DEBUG" &> /tmp/init.log
    Last edited: Feb 21, 2014
  10. jerrm

    jerrm Network Guru Member

    We disagree? What are you wrong about?;)
    koitsu likes this.
  11. phil4200

    phil4200 Reformed Router Member

    I greatly appreciate the help... I've run into a few complications tonight. This may take me until tomorrow to execute! I am very excited to try this init script that jerrm provided, if only to learn a bit more about TomatoUSB and getting the most out of my router! Thanks!
  12. phil4200

    phil4200 Reformed Router Member

    So it was a long rainy weekend with the kids stuck in the house. I haven't got to most of the testing that you guys described, but I did want to pass along an update. On friday night, as I had a few spare minutes, I started working on what you both (jerrm and koitsu) had requested (and I will still provide it). I logged into my router via telnet and cd'd into the /tmp folder, and was surprised to find in there my data file (I'd never looked in there before)! It was being actively written to (my python script was running in the background). This made me recall a post I'd read on stackoverflow about file paths and python. At the time, I was just using relative file paths (simply referencing "NMEAdata.dat", "NMEAlogger.dat", etc), and for some reason this was ending up in the /tmp folder, and not in the script location of /opt/scripts. Obviously this was a problem. I changed the script to have absolute paths ("opt/scripts/NMEAdata.dat", etc) and voila, everything (almost) worked.

    I would still like to go through with the testing that you guys described because I'm wondering if my mounting scheme, or other things you asked about could affect the behaviour of the relative path references in my script. I'd also just like to have everything "correct", according to someone's standards who knows more about this than I do!

    The (almost) above refers to the one issue I am still having, and I am wondering if this is some kind of Tomato or optware limitation, or maybe a windows/linux difference, or maybe a python problem for another forum. In my python script, when I "serve" a text file after an HTML request, the file is read "f.read()", assigned to a variable as a string, and then sent to a socket via conn.send(). When I run this script on my win7 machine, the whole file is served and displayed on the browser, when the script is running off of Tomato only 20 kb of text are served to the browser. I've verified that the f.read() function is indeed reading the whole file (150 kb, for example), and my string variable that contains the 150 kb of text is holding the whole file, it just seems that the socket operations conn.send() in the Optware environment isn't sending more than 20 kb!

    Any ideas?

    I will get the other information to you soon. Thanks again.
  13. koitsu

    koitsu Network Guru Member

    Situation described in previous posts indicates that the Init script stuff has a current working directory of /tmp before your python script is started. You can verify this by adding the line /usr/bin/env > /tmp/environment before calling python and rebooting the router; I'm sure you'll see PWD=/tmp or something along those lines (you'd see the same if you ran /bin/pwd), along with output of the environment.

    There isn't anything strange about this to me. You should make your Init script use chdir, your python script use chdir() (or whatever the Python equivalent is -- I don't do snakes), or specify full paths to everything. Really, that's the proper solution. Honest. It's like this in any *IX.

    Can't help you with your Python performance thing and sending contents across a socket -- again, I don't speak Coiler.
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice