bash: /bin/tar: Argument list too long

| 5 Comments | 1 TrackBack | Bookmark and Share

Anyone who has worked with large numbers of files has seen the error 'Argument list too long' returned by a command before, usually something like cp or rm. Eventually, they find xargs and all is well again; rather than globbing directly, you use find and xargs to feed the intended command:

~/tmp$ ls | wc -l
77029
~/tmp$ cp * /foo/bar/
bash: /bin/cp: Argument list too long
~/tmp$ find . -print0 | xargs -0 -I {} cp {} /foo/bar/

For those who don't know, that final command is running find, which simply prints a list of the files in the current directory. xargs then passes a not-too-long subset of that list to cp in batches, replacing the {} with said list. The -print0 tells find to terminate each filename with a null character rather than the default newline, so that weird filenames work; the corresponding option -0 must be used with xargs so that it looks for that syntax. Now, on to the problem of the day.

I know xargs, but for the problem I had, I didn't think it would help (more on that later). I wanted to tar up a whole bunch of files, too many for the command line to handle. The solution was simple, for tar provides an option to receive its list of files that makeup an archive from a file:

~/tmp$ find . -print > baz.lst
~/tmp$ tar -cf baz.tar --files-from baz.lst

Simple. There's probably a way to one-line that, but simply piping the output of find to tar with --files-from - didn't work, and I didn't spend any more time trying to figure it out. As I said previously, I didn't think xargs would help, but I pulled up the tar manpage and, using the -r option, you can append to an existing archive, which could work in conjunction with xargs just fine.

1 TrackBack

from Whoila Blog » Blog Archive » Tar large directory of files in UNIX on December 9, 2008 4:59 PM

TITLE: URL: http://www.whoila.com/blog/?p=611 IP: 208.97.183.7 BLOG NAME: Whoila Blog » Blog Archive » Tar large directory of files in UNIX DATE: 12/09/2008 04:59:01 PM Read More

5 Comments

Just wanted to say a big thank-you. I stupidly forgot to redirect a the output of a cron entry to /dev/null, and that script ran every 5 minutes, for months, before I sshed into the server and found that it took five minutes to get my bash prompt - that's how long it took to check for new mail. Your find and tar lines saved me a lot of time.

How about: $ find . -name '*' | xargs tar cf - > ../files.tar

What if I have a few directories and this adds up ?

Should i tar all 10 directories as 10 tar files, then tar them together, or should I run find 10 times and combine the results into 1 txt tmp file. ?

I should try them first… thanks

I think you will want to put them all into one tarball - this will probably help: http://www.apl.jhu.edu/Misc/Unix-info/tar/tar_28.html

You may also combine find & tar to create archives:

http://codesnippets.joyent.com/posts/show/1962

Leave a comment

About this Entry

This page contains a single entry by Drew Stephens published on February 14, 2007 6:25 AM.

Year 2000 Problem was the previous entry in this blog.

Senator Blackburn on The War in Iraq is the next entry in this blog.

Find recent content on the main index or look in the archives to find all content.

Powered by Movable Type 5.1