NeoLoader Forum

How well does Neo handle upload areas containing large numbers/volumes of files?

In my case, that's ~37000 files and 11TB of data.

Unfortunately that amount of files will currently cause some problems, if it's possible at all. Once all those files are hashed (assuming you have a few GB of memory to keep all the hashes in the RAM), it will take at least 15 minutes to restart NeoLoader because loading a huge amount of files from the filelist-file from disk takes a very long time...

So right now NeoLoader isn't perfectly suited to handle that amount of data. But we definitely plan to work on it.
Some questions to better understand your situation and improve the file handling in future versions:
In what network do you plan to share your files? Torrent? ed2k? or Hosters? or all of them?
Do you know any client that can handle that amount of data without any problems? (including startup time)

Sorry for the long delay replying.

Ekliptor wrote:Unfortunately that amount of files will currently cause some problems, if it's possible at all. Once all those files are hashed (assuming you have a few GB of memory to keep all the hashes in the RAM), it will take at least 15 minutes to restart NeoLoader because loading a huge amount of files from the filelist-file from disk takes a very long time...

Most programs take quite a while to start up with this number of files on board

So right now NeoLoader isn't perfectly suited to handle that amount of data. But we definitely plan to work on it.
Some questions to better understand your situation and improve the file handling in future versions:
In what network do you plan to share your files? Torrent? ed2k? or Hosters? or all of them?
Do you know any client that can handle that amount of data without any problems? (including startup time)

Primarily ed2k/Kad, but "all" is a good long-term target.

I'm running on Linux, with the main system daemon-only.

Amule runs but has trouble with the numbers but takes about an hour to fully share everything.

Mldonkey runs fine and has the same problem as amule at startup. When adding files it can take 6-12 hours to notice and start advertising them - that's because the way it scans directories it can take several passes to pick up everything.

Both programs have memory footprints of about 2Gb and neither uses very much CPU (the system ~ 8 year old dual 2.1GHz 5140 Xeon server.)

hmm... I will release an up to date linux build this weekend.
So may by you can just test and tell us if it worked.

Cheers
David X.