logo
Apache Lounge
Webmasters

 

About Forum Index Downloads Search Register Log in RSS X


Keep Server Online

If you find the Apache Lounge, the downloads and overall help useful, please express your satisfaction with a donation.

or

Bitcoin

A donation makes a contribution towards the costs, the time and effort that's going in this site and building.

Thank You! Steffen

Your donations will help to keep this site alive and well, and continuing building binaries. Apache Lounge is not sponsored.
Post new topic   Forum Index -> Apache View previous topic :: View next topic
Reply to topic   Topic: Apache 2.4.43 / Windows 10 / Crash of libhttpd.dll Page 1, 2  Next
Author
mdrmdr



Joined: 05 Apr 2020
Posts: 16

PostPosted: Sun 05 Apr '20 10:21    Post subject: Apache 2.4.43 / Windows 10 / Crash of libhttpd.dll Reply with quote

Hi,

since a while I use Apache x64 2.4.41 for the staging of my website. It runs on latest Windows 10 (all updates) with PHP 7.4.4 and Perl 5.30.2. It's configured for TLS 1.3 only with "SSLProtocol -all TLSv1.3" and HTTP/2.

Yesterday I updated to Apache x64 2.4.43. I copied the whole "conf" directory from .41 to .43. Most of my site still works. But *all* https calls to Perl URLs now fail. Regardless of direct or using PHP cURL. http works fine. PHP/https also works fine. Perl is configured as "ScriptInterpreterSource registry-strict" using .pl and .cgi settings in the registry.

There are *no* error messages at all in the browser. Also *no* error messages when I instruct cURL to write a verbose logfile. Windows event log shows an app crash of httpd.exe (in detail libhttpd.dll). Exception is "0xc0000005" (access violation) at "Error offset: 0x00000000000266f9".

Where do I start to look for errors? Why fails a fully working .41 environment in .43? If required, I'll upload (parts of) my config files.

Thanks for any help
mdrmdr
Back to top
glsmith
Moderator


Joined: 16 Oct 2007
Posts: 2268
Location: Sun Diego, USA

PostPosted: Sun 05 Apr '20 22:17    Post subject: Reply with quote

Odd.

Try adding to httpd.conf
ScriptLog logs/script-error.log

See if script-error.log hints at something.
Back to top
mdrmdr



Joined: 05 Apr 2020
Posts: 16

PostPosted: Mon 06 Apr '20 0:19    Post subject: Reply with quote

The ScriptLog file does not even get created.

I now monitored system activity with Sysinternals "process monitor". Immediately after reading the path to perl.exe from the .cgi registry value, it starts "C:\Windows\system32\WerFault.exe" because httpd.exe has crashed. This does not happen with .41.

Here's an excerpt of the output filtered for httpd.exe:

Code:
23:56:13,8866572   httpd.exe   12404   RegQueryValue   HKCR\.cgi\Shell\ExecCGI\Command\(Default)   SUCCESS   Type: REG_SZ, Length: 60, Data: X:\Apps\Perl\bin\perl.exe -WT
23:56:13,8866719   httpd.exe   12404   RegCloseKey   HKCR\.cgi   SUCCESS   
23:56:13,8867205   httpd.exe   12404   QueryNameInformationFile   X:\Apps\Apache24\bin\httpd.exe   SUCCESS   Name: \Apps\Apache24\bin\httpd.exe
23:56:13,9697247   httpd.exe   12404   Process Create   C:\Windows\system32\WerFault.exe   SUCCESS   PID: 6652, Command line: C:\Windows\system32\WerFault.exe -u -p 12404 -s 2184
Back to top
mdrmdr



Joined: 05 Apr 2020
Posts: 16

PostPosted: Mon 06 Apr '20 0:29    Post subject: Reply with quote

A call to httpd.exe from the commandline with "httpd.exe -X -e debug" just exits the programm without any further message once I call a https perl script in the browser. Guess this is the same httpd.exe hard crash as if I run it via Apache Monitor.

I'm currently giving up and switching back to .41 which works fine.
Back to top
mdrmdr



Joined: 05 Apr 2020
Posts: 16

PostPosted: Wed 08 Apr '20 9:42    Post subject: Reply with quote

I now asked the same question in the Apache House forum:
https://forum.apachehaus.com/index.php?topic=1614.0.

Once I (hopefully) get an solution, I'll post details here as well...
Back to top
admin
Site Admin


Joined: 15 Oct 2005
Posts: 677

PostPosted: Wed 08 Apr '20 10:31    Post subject: Reply with quote

What you can try:

Replace libhttpd.dll with the one from 41, and see if it errors.

Replace a used module and mentioned in the change log 42/43 one by one with a 41 one, and see if it errors.
Back to top
mdrmdr



Joined: 05 Apr 2020
Posts: 16

PostPosted: Wed 08 Apr '20 11:59    Post subject: Reply with quote

Tried that already but did it again rigth now.
No change. httpd.exe still crashes. Excerpt from the event viewer:

Code:
Name of the faulty application: httpd.exe, version: 2.4.43.0, timestamp: 0x5e834d57
Name of the faulty module: libhttpd.dll, version: 2.4.41.0, timestamp: 0x5d4d8749
Exception code: 0xc0000005
Error offset: 0x0000000000000015d9
ID of the faulty process: 0x2028
Back to top
Jan-E



Joined: 09 Mar 2012
Posts: 1248
Location: Amsterdam, NL, EU

PostPosted: Wed 08 Apr '20 15:36    Post subject: Reply with quote

The problem might be a change in an Apache module.

Try disabling as much of the changed modules as possible or replace them by the 2.4.41 versions. Start with replacing mod_cgi with the 2.4.41 version and/or disable mod_http2. If you are using mod_md: disable it as well.

See the changelog: https://www.apachelounge.com/Changelog-2.4.html
Back to top
glsmith
Moderator


Joined: 16 Oct 2007
Posts: 2268
Location: Sun Diego, USA

PostPosted: Wed 08 Apr '20 21:17    Post subject: Reply with quote

I have a problem believing it is a problem with mod_cgi because I use that all day long and all my cgi scripts work just fine. I use old school #!shebang however and have perl in the system path so I only have to use #!perl. ScriptInterpreterSource is in core, not in mod_cgi.

Comment out ScriptInterpreterSource (defaults to "Script") and add the shebang to the script. Does it then work?

If it does then something's been borked in the core.
Or, did you switch to perl64? It installs into c:\program files\perl64 (unless you tell it different instead of c:\program files\perl and you would need to modify the registry key/s you've set up.
Back to top
mdrmdr



Joined: 05 Apr 2020
Posts: 16

PostPosted: Wed 08 Apr '20 22:43    Post subject: Reply with quote

The replace of the modules uncovered the evil program Smile

It's mod_http2. Either when I disable http2 in httpd.conf or when I use the .41 module, everything works again.
So one of the changes from .41 to .43 must cause the crash in libhttpd.dll when I call a Perl script and http/2 is active.

Commenting out ScriptInterpreterSource did by the way not help...

And just for complete information: I'm using the latest x64 Strawberry Perl 5.30.2.1. But after my finding, this should not matter anymore.

What to do next? Can you possibly re-create the error yourself now? Should I test with a special debug version of mod_http2?
Back to top
glsmith
Moderator


Joined: 16 Oct 2007
Posts: 2268
Location: Sun Diego, USA

PostPosted: Thu 09 Apr '20 4:14    Post subject: Reply with quote

Looking into a few files I see trace logging of 1, 2 & 4 so;

LogLevel warn http2:trace4

However, if you have a debugger go ahead and use it as it will find the where it's crashing and show the road taken the bits took to get there.

I'll try to duplicate. I do not use ScriptInterpreterSource but I do use TLS/1.3 and http2.

Just curious, what happens if you allow TLS/1.2?
Back to top
mdrmdr



Joined: 05 Apr 2020
Posts: 16

PostPosted: Thu 09 Apr '20 9:15    Post subject: Reply with quote

Lowering from TLS1.3 to TLS1.2 does not help. I do not have a debugger to run httpd.exe.
I added the LogLevel statement and the result was:

Code:
[Thu Apr 09 09:07:14.618745 2020] [http2:debug] [pid 8820:tid 608] mod_http2.c(111): AH03089: initializing post config dry run
[Thu Apr 09 09:07:14.642767 2020] [http2:info] [pid 8820:tid 608] AH03090: mod_http2 (v1.15.8, feats=, nghttp2 1.40.0), initializing...
[Thu Apr 09 09:07:14.642767 2020] [http2:trace1] [pid 8820:tid 608] h2_h2.c(447): h2_h2, child_init
[Thu Apr 09 09:07:14.642767 2020] [http2:trace1] [pid 8820:tid 608] h2_switch.c(44): h2_switch init
[Thu Apr 09 09:07:14.651775 2020] [mpm_winnt:notice] [pid 8820:tid 608] AH00455: Apache/2.4.43 (Win64) OpenSSL/1.1.1f PHP/7.4.4 configured -- resuming normal operations
[Thu Apr 09 09:07:14.651775 2020] [mpm_winnt:notice] [pid 8820:tid 608] AH00456: Apache Lounge VS16 Server built: Mar 31 2020 16:01:02
[Thu Apr 09 09:07:14.651775 2020] [core:notice] [pid 8820:tid 608] AH00094: Command line: 'X:\\Apps\\Apache24\\bin\\httpd.exe -d X:/Apps/Apache24'
[Thu Apr 09 09:07:14.653777 2020] [mpm_winnt:notice] [pid 8820:tid 608] AH00418: Parent: Created child process 10560
[Thu Apr 09 09:07:14.918017 2020] [http2:debug] [pid 10560:tid 668] mod_http2.c(111): AH03089: initializing post config dry run
[Thu Apr 09 09:07:14.937034 2020] [http2:info] [pid 10560:tid 668] AH03090: mod_http2 (v1.15.8, feats=, nghttp2 1.40.0), initializing...
[Thu Apr 09 09:07:14.937034 2020] [http2:trace1] [pid 10560:tid 668] h2_h2.c(447): h2_h2, child_init
[Thu Apr 09 09:07:14.937034 2020] [http2:trace1] [pid 10560:tid 668] h2_switch.c(44): h2_switch init
[Thu Apr 09 09:07:14.945041 2020] [http2:trace3] [pid 10560:tid 668] h2_conn.c(133): h2_workers: min=64 max=96, mthrpchild=64, idle_secs=600
[Thu Apr 09 09:07:14.945041 2020] [http2:trace2] [pid 10560:tid 668] h2_workers.c(102): h2_workers: new thread for slot 95
<start of workers deleted>
[Thu Apr 09 09:07:14.948044 2020] [http2:trace2] [pid 10560:tid 668] h2_workers.c(102): h2_workers: new thread for slot 0
[Thu Apr 09 09:07:14.949046 2020] [mpm_winnt:notice] [pid 10560:tid 668] AH00354: Child: Starting 64 worker threads.
<crash below>
[Thu Apr 09 09:07:48.725754 2020] [mpm_winnt:notice] [pid 8820:tid 608] AH00428: Parent: child process 10560 exited with status 3221225477 -- Restarting.
[Thu Apr 09 09:07:48.750777 2020] [http2:info] [pid 8820:tid 608] AH03090: mod_http2 (v1.15.8, feats=, nghttp2 1.40.0), initializing...
<following lines identical to above after "initializing...">


The error code 3221225477 is 0xC0000005 in hex, which on Windows is:
#define STATUS_ACCESS_VIOLATION ((NTSTATUS)0xC0000005L)
[read this on stackoverflow]
Back to top
admin
Site Admin


Joined: 15 Oct 2005
Posts: 677

PostPosted: Thu 09 Apr '20 10:14    Post subject: Reply with quote

Informed the dev at the dev-list http://apache-http-server.18135.x6.nabble.com/Apache-HTTP-Server-Dev-f4771363.html
Back to top
mdrmdr



Joined: 05 Apr 2020
Posts: 16

PostPosted: Thu 09 Apr '20 10:42    Post subject: Reply with quote

Great, thanks! I'll check here and at the mailing list regularily...
Back to top
Steffen
Moderator


Joined: 15 Oct 2005
Posts: 3049
Location: Hilversum, NL, EU

PostPosted: Sat 11 Apr '20 12:21    Post subject: Reply with quote

Stefan wrote at dev-list to try the revisions between .41.and .43 :

r1861247
r1864126
r1872230


So here the first Win64 one : Removed
when not ok then build the next


Last edited by Steffen on Sun 12 Apr '20 20:34; edited 1 time in total
Back to top
Steffen
Moderator


Joined: 15 Oct 2005
Posts: 3049
Location: Hilversum, NL, EU

PostPosted: Sat 11 Apr '20 13:08    Post subject: Reply with quote

Just figured out that r1861247 and r1864126 are already in .41 so that should work at you. You can check to be sure.

At the dev-list Stefan says: if we do not get a minimal configuration where you can reproduce.

So. Maybe it helps that you post at dev-list more info of your configuration and you Perl script.


ps.
I do not know why you put things in the registry. I run all kind of scripts just only using the #!shebang line.
Back to top
mdrmdr



Joined: 05 Apr 2020
Posts: 16

PostPosted: Sat 11 Apr '20 13:50    Post subject: Reply with quote

Steffen wrote:
Just figured out that r1861247 and r1864126 are already in .41 so that should work at you. You can check to be sure.

Hm, I only can use .41 (which works) and .43 (which fails). I cannot compile the releases inbetween myself, as I do not have a development environment on my Windows system.

Steffen wrote:
At the dev-list Stefan says: if we do not get a minimal configuration where you can reproduce.

What is this? An excerpt of httpd config files? Which one? All?

Steffen wrote:
So. Maybe it helps that you post at dev-list more info of your configuration and you Perl script.

The Perl script is just a 3-liner direct called in the browser:

Code:
#!/usr/bin/perl
print "Content-Type: text/html\n\n";
print substr($^V,1);


Steffen wrote:
ps.
I do not know why you put things in the registry. I run all kind of scripts just only using the #!shebang line.

Because pathes on my prod system (hosted & unix) and my dev system (@home & W10) are different... And, with the correct shebang and registry .cgi disabled, it crashes as well...
Back to top
Steffen
Moderator


Joined: 15 Oct 2005
Posts: 3049
Location: Hilversum, NL, EU

PostPosted: Sat 11 Apr '20 14:21    Post subject: Reply with quote

Just trying to help.

You wrote: Hm, I only can...
You missed my post above that I already build one for you ? Download link to http2-1861247 in above post.

You wrote: What is this? ...
Ask Stefan what info he wants more.

Better to discuss with Stefan at the dev-list, he is the author.
Back to top
mdrmdr



Joined: 05 Apr 2020
Posts: 16

PostPosted: Sat 11 Apr '20 15:18    Post subject: Reply with quote

I'm terribly sorry. Yes, I've overseen the rar archive.

Just tested it. And ... IT WORKS! Smile

Are you posting this result to the mailing list or should I and ask Stefan what he needs for analysis?

Thanks a lot for your help!

Edit:
Just sent it to the mailing.list...
Back to top
Steffen
Moderator


Joined: 15 Oct 2005
Posts: 3049
Location: Hilversum, NL, EU

PostPosted: Sun 12 Apr '20 15:43    Post subject: Reply with quote

You can report:

By Stefan Advised revisions

r1861247 - works
r1864126 - works
r1872230 - works

Test some more Revisions:

r1873368 - works
r1874286 - works
r1874347 - works
r1874909 (=2.4.43) - fails!
Back to top


Reply to topic   Topic: Apache 2.4.43 / Windows 10 / Crash of libhttpd.dll View previous topic :: View next topic
Post new topic   Forum Index -> Apache Page 1, 2  Next