Is there any software for data management?

Question

Ok, so my workflow is around big files, these are sources from edutorial lectures, they weight a lot, one typical recording size is around 80 Gb.

I need to manage them, store and edit. My machine is 500 Gb SSD, 1 TB HDD and 25 TB network drive.

As I often need to edit many lectures, I copy them to my SSD or HDD for faster editing. Premiere can edit files directly from network drive, however, it has only 10 Mb/s access speed, so editing them is not very pleasant experience.

I have a great headache copying them around bothways. Everyday there are new recordings that need to be uploaded to the network drive, and there new lectures in queue for editing that are need to be copied locally. That would not be a problem, if not for the stupid Windows file copying software. Sometimes I accidentally close this dialog and then forget about what I was copying, sometimes it finds duplicates and messes me around.

So my question is if there any software that you use, that would allow setting directories for copying or moving bothways. Something like Google Drive would be ideal. It check automatically if the folder is uploaded, and if there is something missing, it uploads the rest.

I've taken a look on Robocopy (pre-installed Windows terminal tool), but I don't like with terminal interface that if there would be some action needed from me, I would not know. Also, I don't trust it working behind the curtains and doing something with my files. I would like something with GUI and status window, like Google Drive, again.

It would be nice, if it could:

• copy files from previously set queue
• pause or lower its speed on command for not interrupting me browsing this network drive
• check two directories for duplicates bit-by-bit
• scans specific directories (e.g. inserted media card) and copies them to the network storage with specific names and into the directory with today's date.
• be failsafe, that means if I turn off my machine, it would restore the queue and continue from the point it was broken
• have a nice GUI and a dialogs for viewing status and for resolving conflicts

Answer 1

While it doesn't cover all your criteria, I use a program called Total Commander for most of my file management needs and it works well for working with a very large number of files and huge file sizes.

Among it's features, it is able to do a directory synchronization that determines the differences between to folders, either by filesize/date or by content (though a content comparison across a network share is going to take a very long time).

It doesn't have the queue or resume functionality you want, but it handles duplicates well and is pretty easy to work with overall.

It does include a pause feature on the copies and lets you run multiple copies at the same time in the background.

It does include a variety of powerful file selection tools to let you choose particular sets of files that need to be copied.

It has a good UI and is relatively cheap. It is available as shareware to try for free and last I knew it was $35 for a lifetime license. My dad bought a copy when Windows 3.0 was still the current OS and hasn't paid again since. I got my license in 2002 or so.

Answer 2

For storage, get a real server chassis on ebay and run FreeNAS (on bare metal, not in a vm). Run a 1Gb or better network backbone. FreeNAS has dropins equivalent to gsuite (Google drive, docs,,sheets, etc). It may have some of the other facilities you are looking for, but at the very least will work with the other suggestions here.