logo
Apache Lounge
Webmasters

 

About Forum Index Downloads Search Register Log in RSS Twitter


Keep Server Online

If you find the Apache Lounge, the downloads and overall help useful, please express your satisfaction with a donation.

or

Bitcoin

A donation makes a contribution towards the costs, the time and effort that's going in this site and building.

Thank You! Steffen

Your donations will help to keep this site alive and well, and continuing building binaries. Apache Lounge is not sponsored.
Post new topic   Forum Index -> Coding & Scripting Corner View previous topic :: View next topic
Reply to topic   Topic: HTTP Protocol Core Functionality Doctor
Author
dejwa



Joined: 11 Jan 2023
Posts: 4

PostPosted: Wed 11 Jan '23 18:43    Post subject: HTTP Protocol Core Functionality Doctor Reply with quote

Hi guys, could someone help me please? I need a really advanced recommendation from HTTP protocol core functions expert.

I need to download the data as fast as possible from the slow server.. For e.g., around 1.000 requests (1kb each file). The issue is 50 more users could be downloading, they may also open large number of threads to the slow server. And I want this to be done as fast as possible.

I already have settings like number of download app servers, and number of threads per server implemented. But what will be the best settings to use? Or best way to make this?

Is it good to download data using 50 servers, and 5 threads each server, or maybe 2 servers and 200 threads.. Or something other?

Please help if you can, I will offer a good reward for the solution. I'm using a Python language, but I can implement a solution in any other language if required.

Target server is just a regular Apache2 server, with defaults.

Thank you very much for your help!
Back to top
James Blond
Moderator


Joined: 19 Jan 2006
Posts: 7169
Location: Germany, Next to Hamburg

PostPosted: Thu 12 Jan '23 13:38    Post subject: Reply with quote

Hi Dejwa,
there are several projects doing this

https://github.com/axel-download-accelerator/axel
https://github.com/maxogden/nugget
https://github.com/lavv17/lftp
https://github.com/huydx/hget
https://github.com/Code-Hex/pget
https://github.com/aria2/aria2
Back to top
dejwa



Joined: 11 Jan 2023
Posts: 4

PostPosted: Thu 12 Jan '23 14:27    Post subject: Reply with quote

Hi, thank you very much for your reply.

I've been using ARIA2 for a couple of years, I will take a look to the other apps.

The issue is my file size is only 1kb, so many of ARIA2 features will be useless. Like downloading single file using multiple threads.

Do you know maybe how the slow web server will act when there is 100 of users downloading using multiple threads? Please note server will be really slow in this scenario, who will have the advantage?

The user downloading using 50 servers (50 IP addresses) and small number of threads (like 5 ), or user downloading using 1 server (1 IP address) but high number of threads (like 200 or 400)

I think my answer relies in the core functionality of the web server software.

Thank you for your help,
Best
Back to top
dejwa



Joined: 11 Jan 2023
Posts: 4

PostPosted: Thu 12 Jan '23 14:30    Post subject: Reply with quote

One more question please...

Do you know the Apache2 webserver will act when I open 200 or 400 threads to it? At the moment when there are 100 of users downloading, and the server is busy - speed is really slow.

Will it disconnect majority of my threads, and lower it down to for e.g., 10?

If this is the scenario, how it will act when I open 5 threads from 50 servers (50 different IP addresses) to it?

Thank you one more time.
Back to top
James Blond
Moderator


Joined: 19 Jan 2006
Posts: 7169
Location: Germany, Next to Hamburg

PostPosted: Thu 12 Jan '23 17:08    Post subject: Reply with quote

Apache can handle a lot of concurrent requests. Especially static files. if you are under 10k concurrent downloads you are usually fine.

I wonder why you can't combine all those small files into one large compressed file and do a single download.
Back to top
dejwa



Joined: 11 Jan 2023
Posts: 4

PostPosted: Thu 12 Jan '23 20:49    Post subject: Reply with quote

Naah, I will try to make it work.

Thank you very much for your help and enjoy
Back to top


Reply to topic   Topic: HTTP Protocol Core Functionality Doctor View previous topic :: View next topic
Post new topic   Forum Index -> Coding & Scripting Corner