query: large numbers of files?

Stoat
Site User
Posts: 2
Joined: Sun Feb 17, 2013 9:48 pm

query: large numbers of files?

Postby Stoat » Sun Feb 17, 2013 10:04 pm

How well does Neo handle upload areas containing large numbers/volumes of files?

In my case, that's ~37000 files and 11TB of data.

User avatar
Ekliptor
Site Admin
Posts: 37
Joined: Fri Feb 27, 2009 11:38 pm
Contact:

Postby Ekliptor » Mon Feb 18, 2013 8:06 am

Unfortunately that amount of files will currently cause some problems, if it's possible at all. Once all those files are hashed (assuming you have a few GB of memory to keep all the hashes in the RAM), it will take at least 15 minutes to restart NeoLoader because loading a huge amount of files from the filelist-file from disk takes a very long time...

So right now NeoLoader isn't perfectly suited to handle that amount of data. But we definitely plan to work on it.
Some questions to better understand your situation and improve the file handling in future versions:
In what network do you plan to share your files? Torrent? ed2k? or Hosters? or all of them?
Do you know any client that can handle that amount of data without any problems? (including startup time)

Stoat
Site User
Posts: 2
Joined: Sun Feb 17, 2013 9:48 pm

Postby Stoat » Fri Nov 01, 2013 1:35 am

Sorry for the long delay replying.

Ekliptor wrote:Unfortunately that amount of files will currently cause some problems, if it's possible at all. Once all those files are hashed (assuming you have a few GB of memory to keep all the hashes in the RAM), it will take at least 15 minutes to restart NeoLoader because loading a huge amount of files from the filelist-file from disk takes a very long time...


Most programs take quite a while to start up with this number of files on board

So right now NeoLoader isn't perfectly suited to handle that amount of data. But we definitely plan to work on it.
Some questions to better understand your situation and improve the file handling in future versions:
In what network do you plan to share your files? Torrent? ed2k? or Hosters? or all of them?
Do you know any client that can handle that amount of data without any problems? (including startup time)


Primarily ed2k/Kad, but "all" is a good long-term target.

I'm running on Linux, with the main system daemon-only.

Amule runs but has trouble with the numbers but takes about an hour to fully share everything.

Mldonkey runs fine and has the same problem as amule at startup. When adding files it can take 6-12 hours to notice and start advertising them - that's because the way it scans directories it can take several passes to pick up everything.

Both programs have memory footprints of about 2Gb and neither uses very much CPU (the system ~ 8 year old dual 2.1GHz 5140 Xeon server.)

User avatar
DavidXanatos
Site Admin
Posts: 769
Joined: Wed Jun 30, 2010 7:54 pm

Postby DavidXanatos » Fri Nov 01, 2013 9:06 am

hmm... I will release an up to date linux build this weekend.
So may by you can just test and tell us if it worked.

Cheers
David X.
Live free or die trying!


Return to “Support”

Who is online

Users browsing this forum: No registered users and 20 guests

cron
Fatal: Not able to open ./cache/data_global.php