How to Move Large Number of Files in Linux

To move files from one directory to another, the ‘mv‘ command is used in Linux. This command is available in Linux by default and can be used to move files as well as directories.

The syntax to use to mv command is:

$ mv file1 file2 /tmp

If we want to move files of, say, the same extension (Eg. txt) or which are numbered in an incremental fashion (Eg. file_1, file_2, etc.), wildcards can be used to specify all those files instead of typing each name.

$ mv *.txt /tmp
$ mv file_* /tmp

Basically, ‘mv‘ can have any number of arguments, which are the files or directories to be moved. The last argument is the target directory where they are to be moved.

Problem with Moving Large Number of Files

Linux systems have a predefined limit on the maximum number of arguments that can be used with a single command. This limit varies from system to system based on the stack size.

Thus, if a very high number of files are being specified with the wildcard with ‘mv‘, Eg. over a hundred thousand files, it throws an error: “Argument list too long”.

For example, in the image below, the folder contains 253420 files. Thus, it gives the aforementioned error when we try to move them.

$ ls -l | wc -l
$ ls | head -1000

$ mv file_* /tmp

Solution: Use ‘Find’ Command

To solve this problem, we make use of the ‘find‘ command. The ‘find‘ command basically searches for files in directories based on different parameters.

We run ‘find‘ in the same directory as the files we want to move. Then we make use of the '-exec' parameter of ‘find‘ which will allow you to run any command over the output of find.

We run ‘mv‘ with '-exec' parameter.

$ find . -name "file_*" -exec mv  {} /tmp \;

Here, the first argument, '.' ((dot) current directory) is the directory where to find the files. The argument “-name” and the string after that specify the expression using which required files can be listed.

Then we specify the ‘mv‘ command after an argument '-exec'. The open brackets '{ }' are a placeholder for the argument which is to be used from the output of find.

You can confirm by running the following and verify that the files have been moved successfully to ‘/tmp’.

$ ls -l /tmp | wc -l

Related Read: How to Copy Large Number of Files in Linux

Conclusion

In this article, we have seen how to move a large number of files from one directory to another, and successfully bypass the ‘Argument list too long’ error in Linux.

If you have any feedback or questions, let us know in the comments below!