logo
Apache Lounge
Webmasters

 

About Forum Index Downloads Search Register Log in RSS X


Keep Server Online

If you find the Apache Lounge, the downloads and overall help useful, please express your satisfaction with a donation.

or

Bitcoin

A donation makes a contribution towards the costs, the time and effort that's going in this site and building.

Thank You! Steffen

Your donations will help to keep this site alive and well, and continuing building binaries. Apache Lounge is not sponsored.
Post new topic   Forum Index -> Third-party Modules View previous topic :: View next topic
Reply to topic   Topic: mod_xsendfile with fastcgi/cgi ?
Author
jabaltie



Joined: 02 Aug 2006
Posts: 12
Location: Brazil

PostPosted: Wed 02 Aug '06 16:11    Post subject: mod_xsendfile with fastcgi/cgi ? Reply with quote

Mr Steffen kindly provided us with a Windows version for mod_xsendfile, that has been originally published at

http://celebnamer.celebworld.ws/stuff/mod_xsendfile/

I installed the module, declared it's load by using the directive below :

LoadModule xsendfile_module modules/mod_xsendfile.so

And also turned it on, by setting

XSendFile on

However, it's not working. Some points :

- My scripts are issuing the http header as described. That I can see from FireFox using it's Livehttpheaders extension.

- I'm issuing that header with a file path unix-style (/ instead of \)

- That file exists, of course.

- When I use Lighthttpd, which also has a similar directive (it was the inspiration for Apache's one), it works perfectly.


What could be wrong with Apache ?


I guess I'm putting it's directives in a wrong place. I'm also using FastCGI, so I dont know if it's compatible or not.


Just in case you want to see my configuration file, get it from here, please :

http://www.des.online.unimep.br/au/pub/apache2.conf

Thanks in advance 4 your support ![/list]
Back to top
James Blond
Moderator


Joined: 19 Jan 2006
Posts: 7360
Location: Germany, Next to Hamburg

PostPosted: Wed 02 Aug '06 16:24    Post subject: Reply with quote

How does your PHP script look? Tried headers_sent() ?
The version is 0.8 may be still some errors in the code

Which version of Apache do you use? I see that x-sendfile uses apr_* it has changed in Apache 2.1 against 2.0!

For that contanct the author Nils Maier, MaierMan(%at%)web(%dot%)de
Back to top
jabaltie



Joined: 02 Aug 2006
Posts: 12
Location: Brazil

PostPosted: Wed 02 Aug '06 16:41    Post subject: Reply with quote

James Blond wrote:
How does your PHP script look? Tried headers_sent() ?
The version is 0.8 may be still some errors in the code


I dont use PHP. I have a FastCGI server of my own.

However, I can see from the browser side that the http header was issued indeed.

Quote:

Which version of Apache do you use? I see that x-sendfile uses apr_* it has changed in Apache 2.1 against 2.0!

For that contanct the author Nils Maier, MaierMan(%at%)web(%dot%)de


I'm using 2.0.58.
Back to top
MaierMan



Joined: 02 Aug 2006
Posts: 7

PostPosted: Thu 03 Aug '06 4:55    Post subject: hi from the author. Reply with quote

I just got mod_xsendfile working with fastcgi/cgi.
Was a bit tricky (and the code at least "feels" somewhat dirty).
Why in hell do those cgi-provided headers go into err_headers_out instead of headers_out... Argh. Took some time to figure it out.

Will prepare a new release and instructions the next few hours/days Wink

Nils

PS. Thanks Steffen for pointing me at here. Wink
Back to top
jabaltie



Joined: 02 Aug 2006
Posts: 12
Location: Brazil

PostPosted: Thu 03 Aug '06 14:37    Post subject: Re: hi from the author. Reply with quote

MaierMan wrote:


Will prepare a new release and instructions the next few hours/days Wink

Nils

PS. Thanks Steffen for pointing me at here. Wink


Thank you so much Nils.

I also have a suggestion of enhancement for your module :

Consider this situation : a script runs some sort of command line tool to generate a huge graphical file. That file is generated on a RAMDRIVE.

Thanks to your extension, the script may instruct the http server to send that file, preventing the script to re-read it again right away. An enormous time saver, I think.

So far, so good.

Now, my suggestion is : I'd like to have one more option for X-SENDFILE : an instruction for the http server to REMOVE that file as soon it is delivered to to the browser. Because then we would clean-up the RAMDRIVE automatically (intrinsically).

It could be another module, say

X-SENDFILE-AND-REMOVE-IT-AFTERWARDS

As for the complexity of doing it, I really dont know. I dont know the internals of Apache. But I have done a similar thing using an API of the http server I'm working on : WebSite PRO from Oreilly (discontinued product).

The problem, I think, is to "buble-up" that order of removing the file back inside to the kernel of the http server. Cause removing should be done after the deliver. I guess this is far away from your current code. But, again, this is a guess.

Maybe it's on the tip of your fingers ...

Let me know what do you think of this idea, OK ?

BTW, I'm anxiously waiting for the newest version of mod_xsendfile for Windows (Steffen ?) !
Back to top
MaierMan



Joined: 02 Aug 2006
Posts: 7

PostPosted: Fri 04 Aug '06 10:02    Post subject: Reply with quote

Ok, here is a test-release of the new version.
Zip includes the source, the slighty updated manual and a windows binary (d'oh, I compiled with SSE, but what CPU nowadays does not support SSE :p)
Not really "official" yet. But kinda RC. Wink

http://celebnamer.celebworld.ws/stuff/mod_xsendfile/mod_xsendfile-0.9.bin.zip

Usage hint:
You'll have to enable XSendFile for the actual fastcgi handler and not for any locations/scripts it handles.
Something like this
Code:
<Location /php-fcgi/php-cgi.exe>
  Options +ExecCGI
  SetHandler fastcgi-script
  XSendFile On
</Location>



I hope to add a second filter later so that I can grab the original request path so that no subrequest is required when used with cgi/fcgi.
But I had no time to look into this yet.
But it's planned, as I will propably use the module in a apache2-worker + php-fcgi environment.

(Rant: Can somebody tell those "professional" php "application" developers that there is no point to implement serving static files from within php with fpassthru/fread and without even thinking of cache-management. Beside the fact that this will simply increase memory usage and will make nice stuff like sendfile() useless... :p)


I won't implement your REMOVE-FILE sugguestion:

  • I have no direct gains from this, and not enough spare time and dedication for this.
  • nor is this filter IMO the right place to implement this
  • nor is it easy to do. Your "bubble" statement is perfectly valid. The filter will replace the contents, and then pass to the next filter in chain, like the transcoding filters and the final network pump. You'd have to do "nasty" tricks that break the filter stack by keeping the filter alive to remove that file afterwards.
  • (nor do I really think this is from a security perspective; stealing files is something bad, but removing them :p)
Back to top
Steffen
Moderator


Joined: 15 Oct 2005
Posts: 3093
Location: Hilversum, NL, EU

PostPosted: Fri 04 Aug '06 13:12    Post subject: Reply with quote

The version 0.9 does not build against Apache 2.2, get an error at the APR_BRIGADE_FOREACH block.

APR_BRIGADE_FOREACH is deprecated.

Steffen
Back to top
jabaltie



Joined: 02 Aug 2006
Posts: 12
Location: Brazil

PostPosted: Fri 04 Aug '06 14:10    Post subject: Reply with quote

MaierMan wrote:
Usage hint:
You'll have to enable XSendFile for the actual fastcgi handler and not for any locations/scripts it handles.
Something like this
Code:
<Location /php-fcgi/php-cgi.exe>
  Options +ExecCGI
  SetHandler fastcgi-script
  XSendFile On
</Location>



I really need some help to configure this. I'm newbie with Apache and dont understand it's configuration file. Think it's too complicated cause I dont know the concepts.

I used WebSite PRO from OReilly and it had all of a Configuration Panel (Windows Dialogs). Easy, very well organized.

Now I have to understand this configuration.

Anyway, the problem is that I want to add X-SENDFILE for all requests that are processed through my FastCGI external server.

This is the piece of the config file :

FastCgiExternalServer c:\apache2\htdocs\api\gtwebsiteapi.wsa -host localhost:101 -appConnTimeout 120 -idle-timeout 300

CheckSpelling on

<Location /api/gtwebsiteapi.wsa>
XSendFile On
</Files>

The whole config file can be get from

www.des.online.unimep.br/au/pub/apache3.conf

When I leave the configuration like that, it will issue a 404. If I remove the location part and keep the XSendFile On, it will process but will not interpret the xsendfile http header issued by the backend.

What do I need to do ?


Quote:

I hope to add a second filter later so that I can grab the original request path so that no subrequest is required when used with cgi/fcgi.
But I had no time to look into this yet.
But it's planned, as I will propably use the module in a apache2-worker + php-fcgi environment.


I hope you DO NOT prevent the requests to pass through the script !

In my case, the script will access a mainframe database in order to see if the user has rights (or not) to see a JPG file which is stored on a folder. As a matter of fact , the mainframe response will tell if the user has rights or not and, also , the location (path) of the JPG file.

So, I REALLY need that ALL the requests GO THROUGH the script.

You CANNOT bufferize or cache it anyhow.

Got it ?

Quote:

(Rant: Can somebody tell those "professional" php "application" developers that there is no point to implement serving static files from within php with fpassthru/fread and without even thinking of cache-management. Beside the fact that this will simply increase memory usage and will make nice stuff like sendfile() useless... :p)


I understand perfectly. What is a static hit should be left as simple as that. But sometimes we need a processing before, mostly to check rights.

Quote:

I won't implement your REMOVE-FILE sugguestion:

Crying or Very sad Crying or Very sad Crying or Very sad
Quote:


  • I have no direct gains from this, and not enough spare time and dedication for this.
  • nor is this filter IMO the right place to implement this
  • nor is it easy to do. Your "bubble" statement is perfectly valid. The filter will replace the contents, and then pass to the next filter in chain, like the transcoding filters and the final network pump. You'd have to do "nasty" tricks that break the filter stack by keeping the filter alive to remove that file afterwards.
  • (nor do I really think this is from a security perspective; stealing files is something bad, but removing them :p)


OK. Understood your points. I'm trying to make Lighty's author to do it for me but, he has not even read my email yet !

If I dont get it working inside the http server, I'll build a queue of my own. But it is too costy and will have a calculated delay to remove the files. Very likely it will remove the files 5 minutes after creation. I guess this is time enough for it to be delivered to the browser.

Better, of course, it would be to have the http server removing it ASAP.

Anyway, thank you so much !

Hope I can make at least my configuration work.

Thanks again Smile
Back to top
MaierMan



Joined: 02 Aug 2006
Posts: 7

PostPosted: Sat 05 Aug '06 5:02    Post subject: Reply with quote

Try turning on |XsendFileAllowAbove on| as well.
Otherwise only files below the request-path (in your case c:\apache2\htdocs\api\) will be allowed to be send out.

Some general notes:
In your particular case it makes more sense to send those files directly from the DB without "buffering" them to the disk first, as they are "one-time" sends anyway if I understood you correctly...
X-Sendfile (or lightys equivalant) are designed to handle files that are stored on disk anyway or for kinda "perisistent" cache files.


To your remarks about the lighty and it's author:
I already noticed that you must be the same guy that is positng to the lighty list as well.
Just a hint: You're asking for not-so-general-prupose features and configuration advice, but you (usually) won't get premium support or custom development for free. This kind of work is usually paid work Wink
Back to top
Steffen
Moderator


Joined: 15 Oct 2005
Posts: 3093
Location: Hilversum, NL, EU

PostPosted: Sat 05 Aug '06 11:43    Post subject: Reply with quote

Steffen wrote:
The version 0.9 does not build against Apache 2.2, get an error at the APR_BRIGADE_FOREACH block.

APR_BRIGADE_FOREACH is deprecated.


Nils, you overlooked this post ?


Steffen
Back to top
MaierMan



Joined: 02 Aug 2006
Posts: 7

PostPosted: Sat 05 Aug '06 15:12    Post subject: Reply with quote

Steffen wrote:
Steffen wrote:
The version 0.9 does not build against Apache 2.2, get an error at the APR_BRIGADE_FOREACH block.

APR_BRIGADE_FOREACH is deprecated.


Nils, you overlooked this post ?


Steffen

yep, I did. (I actually overlooked it twice. First your post and then I didn't meant to keep it in the code.)
The code will get some final cleanup/review anyway.
The 0.9.bin.zip from above was posted for testing pruposes only. Wink
Back to top
jabaltie



Joined: 02 Aug 2006
Posts: 12
Location: Brazil

PostPosted: Sat 05 Aug '06 15:33    Post subject: Reply with quote

I logged on this Saturday only to see if there were some answer. And I see there is !

Thank you Mr Miers !

MaierMan wrote:
Try turning on |XsendFileAllowAbove on| as well.
Otherwise only files below the request-path (in your case c:\apache2\htdocs\api\) will be allowed to be send out.


I tried. But it doesnt work Embarassed

Again, here's the configuration part :

Code:

LoadModule fastcgi_module     modules/mod_fastcgi.dll
LoadModule xsendfile_module modules/mod_xsendfile.so

FastCgiExternalServer c:\apache2\htdocs\api\gtwebsiteapi.wsa -host localhost:101 -appConnTimeout 120 -idle-timeout 300

XsendFileAllowAbove on


All I want is to have everything that is processed through my FastCGI server to interpret X-SENDFILE. Just that.

Notice that the script IS issuing the header cause I can see it on FireFox. Also, it doesnt matter if I use normal windows slashes (\) or (/). It doesnt work with none of them. Firefox shows me an empty (blank) page.

When it works for you, which are the headers ? My guess is that maybe I'm issuing the headers on the wrong order (or adding headers that will disable your module). Can you tell me in your case which are the headers received by your browser ? Can you paste them here ?

Here's a screen shot of my headers :

http://www.des.online.unimep.br/au/pub/headers.jpg

Quote:


Some general notes:
In your particular case it makes more sense to send those files directly from the DB without "buffering" them to the disk first, as they are "one-time" sends anyway if I understood you correctly...
X-Sendfile (or lightys equivalant) are designed to handle files that are stored on disk anyway or for kinda "perisistent" cache files.


They are NOT on a DataBase. They're simply files which are stored on disk. Each student/employee/teacher has its own folder (and subfolders).

On this context, X-SENDFILE makes all the sense cause the script will check if the request has the rights or not to see those files (if who's viewing is the owner of the document or say, the boss wants to see his employee document for instance). Then, the script simply points the http server to send that file which is already on disk.

Again, there is no database. Files are spread amongst zillions of folders on a SCSI disk.

Simple and effective.

But I also use the same resource for processed files. My application is external and, it generates the response files on a RAMDRIVE. Then, I use X-SENDFILE to simply instruct the http server to send those already generated files. Why is it like that ? Because my application has a kind of a cache too. If it sees that it doesnt need to re-process the request, it simply talks back the file path (and FastCGI server will in its turn also talk back the file path via your module).

Further, I plan to do the http server itself to check if the cache is valid or not, thus saving it to have to launch the script over and over again, just to see if the cache is valid.

Lighty has something like that. It can prevent scripts from being triggered again. It is something regarded to CML (cache markup language) and cache module.

But this is LATER. Now I must replace my current http server and EVERYTHING should work as it is today. I shall not make this migration more complicated than what it is already.


Quote:
To your remarks about the lighty and it's author:
I already noticed that you must be the same guy that is positng to the lighty list as well.
Just a hint: You're asking for not-so-general-prupose features and configuration advice, but you (usually) won't get premium support or custom development for free. This kind of work is usually paid work Wink


Yes, I'm an over-talkative member of their Forum.

Apache is working more properly , although I do prefer Lighty.

But Lighty has had many problems :

- When using an external FastCGI server, it has some problems regarded to large form submissions, say when you upload files.
- Also, you cannot use PATH INFO cause when you do that, it wont find the script (404).

But I'm still around Lighty.

As for the file removal, I have built a secondary thread on my FastCGI server that will run after the files to be removed. But I think this is more complicated and more costy than having the http server by removing it itself.

I'm taking too much from your time here but please, I do need your help to configure Apache !

This is the only missing part on my FastCGI server. That is, in order to put my kit (Apache/FastCGI server) into PROduction site, I only need to solve the X-SENDFILE mysthery !

Can you help me ?

Here's how my configuration file right now :

http://www.des.online.unimep.br/au/pub/apache4.conf
Back to top
MaierMan



Joined: 02 Aug 2006
Posts: 7

PostPosted: Sat 05 Aug '06 16:01    Post subject: Reply with quote

Jose,
where exactly do you enable XSendFile in your configuration?
I guess nowhere (if you didn't put it in some .htaccess file).

You need to do |XSendFile on| AND |XSendFileAllowAbove on| in your particular case.
Former enables XSendFile and latter allows sending of files from everywhere not just below the original request folder.
But latter does not imply former.

It's like most stuff in apache.
No RewriteRule without RewriteEngine Wink

Header order is not important at all.
The header name (X-SENDFILE) is not even case sensitive.


PS: Not that this is important, or that I care a lot about it, but in fact my family name is spelled Maier.
Back to top
jabaltie



Joined: 02 Aug 2006
Posts: 12
Location: Brazil

PostPosted: Sat 05 Aug '06 16:13    Post subject: Reply with quote

Thanks again !

Thought only me would work on Saturdays...

MaierMan wrote:
Jose,
where exactly do you enable XSendFile in your configuration?
I guess nowhere (if you didn't put it in some .htaccess file).


I've added right after the LoadModules declaration.

Quote:

You need to do |XSendFile on| AND |XSendFileAllowAbove on| in your particular case.
Former enables XSendFile and latter allows sending of files from everywhere not just below the original request folder.
But latter does not imply former.


Let's add both then....

Perfect ! Now it's working !!!!!!!!!!!!!!

Quote:

It's like most stuff in apache.
No RewriteRule without RewriteEngine Wink

Header order is not important at all.
The header name (X-SENDFILE) is not even case sensitive.


PS: Not that this is important, or that I care a lot about it, but in fact my family name is spelled Maier.


Sorry !

Now, one more thing : you dont care about headers but I do !

I can see from the browser perspective that X-SENDFILE will be there.

This is very, very risky here.

This will reveal the original file path. For cache generated files is not a problem, however, for those other official documents, it is a big vulnerability.

For instance, student's grades report are stored like that. As well as teachers and employees copies of their pay-checks (holleriths).

So, I NEVER, EVER let end users know the original file path. One could see the other documents. Of course those folders are not windows shareable but it is not good for people out there to know where they are.

And believe-me, students are very, very curious. I can tell N stories of what they've done here. Some very creative and bizarre artistic works with teachers photos on PhotoShop for instance. Teachers photos are stored on those folders , simply like JPG files too.

So, is there a way to make it be swallowed by the http server ?

Thanks again !
Admin note:
Please open new topic for new questions
Back to top


Reply to topic   Topic: mod_xsendfile with fastcgi/cgi ? View previous topic :: View next topic
Post new topic   Forum Index -> Third-party Modules