Login Main site Create account

25.04.2006 14:29

Uploading binary files in bash cgi script

Today I fixed the issue with my thinclients were I was unable to upload binary files (e.g. tar.gz) to my thinclient installation.

It's not very beautiful, but it works (I'm open for better solutions, so please send me e-mail if you know about a better solution to do this).
if [ "$REQUEST_METHOD" = "POST" ]; then
  cat >$TMPOUT

  # Get the line count
  LINES=$(wc -l $TMPOUT | cut -d ' ' -f 1)

  # Remove the first four lines
  tail -$((LINES - 4)) $TMPOUT >$TMPOUT.1

  # Remove the last line
  head -$((LINES - 5)) $TMPOUT.1 >$TMPOUT

  # Copy everything but the new last line to a temporary file
  head -$((LINES - 6)) $TMPOUT >$TMPOUT.1

  # Copy the new last line but remove trailing \r\n
  tail -1 $TMPOUT | perl -p -i -e 's/\r\n$//' >>$TMPOUT.1

This is necessary, because busybox's webserver (and maybe all others too) write four lines of content information at the beginning of the generated file and one line at the end as well as making a new line in the last line of the binary file causes the end of the last line to contain "\r\n" which is also subject to strip.

In my testings, I tried ten different binary files and always got the same md5sums after uploading, which means that this little shell script is likely to work.
Comments added earlier to http://tuxx-home.at/archives/2006/04/25/T14_29_04/index.html:
Guest on 2012-07-25 05:16:32 wrote:
Thanks for this. My only problem is that I'm running on an embedded system that doesn't include Perl, so I've replaced the final command with:
tail -1 $TMPOUT | tr -d '\r\n' >>$TMPOUT.1
Guest on 2012-08-10 17:56:00 wrote:
Unfortunately, the line
cat >$TMPOUT
does not ever complete on my machine. I mean that this command takes the input, redirects it to $TMPOUT and keeps waiting for something. I don't know how to make it read all the input, direct it to a file and exit. :(
Guest on 2012-09-09 18:54:51 wrote:
To Guest
Make sure the cgi script has permission to write to the directory
Guest on 2014-02-11 19:01:47 wrote:
Same script using sed :

if [ "$REQUEST_METHOD" = "POST" ]; then
cat >$TMPOUT

sed -e '1,4d;N;$!P;$!D;$d' $TMPOUT > TMPOUT.1
sed -e '1{$q;};$!{h;d;};x' $TMPOUT | perl -p -i -e 's/\r\n$//' >> TMPOUT.1

The first sed line replace the wc, first tail and first head. The second sed replace the second head and the second tail.
Guest on 2015-02-24 04:32:17 wrote:

Thank you for this page it helps me so much.
I did some mods to your code:
# get the sdtin
cat >/tmp/test

# get number of line
NBLIGNE=`wc -l ${TMPOUT} | awk '{print $1}'`

# Remove the first four lines
tail -n $((${NBLIGNE} - 3)) $TMPOUT > $TMPOUT.1

# Remove the last 5 lines
head -n $((${NBLIGNE} -8)) $TMPOUT.1 >$TMPOUT

Run great on FreeBSD 8.3 with shttpd. I tried png, gif, tar,gz and pdf without any error or problem.


Guest on 2015-02-24 21:12:28 wrote:

I finally mods my own mods to get
- rename the file with the original one
- binary upload or not ?

1- rename the file as the original name
- information:
Content-Disposition: form-data; name="file"; filename="foobar.jpg"
Content-Type: image/jpeg

-- get the header: ENTETE=`head -n 2 $TMPOUT`
-- retrieved the filename: NOMF=`echo ${ENTETE} | ${TR} '\n' ' ' | ${CUT} -d ";" -f 3 | ${AWK} '{print $1}' | ${CUT} -d "=" -f 2 | ${SED} -e 's:"::g' | ${TR} -d '\r'`

2- binary or ascii upload: use file (man file) with the --mime-encoding to get the encoding: binary or something else. if not binary I remove the '\r'

- file --mime-encoding /tmp/charlie.pdf
/tmp/charlie.pdf: binary
- file --mime-encoding /tmp/gravatar-l2fl2f.jpeg
/tmp/gravatar-l2fl2f.jpeg: binary
- file --mime-encoding /tmp/makefetchrecursi e.sh
/tmp/makefetchrecursive.sh: us-ascii

-- if [ ${#ENCODING} -eq 0 ] ; then
# text
# enleve \r + rename the file
${TR} -d '\r' < ${TMPOUT} > /tmp/${NOMF}
# binaire
# renomme le fichier avec son nom originel
${MV} ${MVPARAM} ${TMPOUT} /tmp/${NOMF} 2>&1 > /dev/null

Run on apache22 + FreeBSD 8.1 and I double check the file on the server: everything is perfect :)

If you need the whole code: yvesguerin [@] yahoo . CA


Guest on 2016-11-09 17:55:54 wrote:
Hi all,
On a Red Hat sed refuses to suppress \n character.
Hopefully, there is a "big" tool: truncate!

In place of all the lines after the following:
# Copy everything but the new last line to a temporary file

just do:
truncate -s -2 $TMPOUT.1

2 is for the size of CRLF. truncate reduce the size of $TMPOUT.1 with 2 characters.

I have lost 1 day for this :(
But now it is OK! I share it!
Guest on 2017-04-27 16:29:28 wrote:
If you're uploading executables (e.g. the firmware binary itself) it may be ok if the binary has just chopped of the 4 leading lines - the trailing stuff that httpd appends will not cause problems when being executed (at least it worked for me).
so it may be enough to just write

read X1
read X2
read X3
read X4

cat > /tmp/myfirmware
chmod a+x /tmp/myfirmware


Your comment (HTML tags will be stripped !!):

To verify You are not a bot, type down text from this image.

Your try: