| Author |
|
James Blond
Moderator
Joined: 19 Jan 2006
Posts: 7389
Location: Germany, Next to Hamburg
|
 Posted: Fri 31 Jan '25 11:57 Post subject: |
 |
|
For me, it starts every time
--- edit ---
log file at https://pastebin.com/s6VUvsKc
Last edited by James Blond on Fri 31 Jan '25 12:05; edited 1 time in total |
|
| Back to top |
|
James Blond
Moderator
Joined: 19 Jan 2006
Posts: 7389
Location: Germany, Next to Hamburg
|
| Posted: Fri 31 Jan '25 12:01 Post subject: |
|
|
| The 0xc0000005 is an access violation. I wonder about that. Do you run that with an unprivileged user? |
|
| Back to top |
|
jmweb
Joined: 08 Jun 2017
Posts: 19
Location: USA, Charlotte
|
| Posted: Fri 31 Jan '25 13:02 Post subject: |
|
|
| James Blond wrote: | | The 0xc0000005 is an access violation. I wonder about that. Do you run that with an unprivileged user? |
I tried with an elevated command prompt and httpd still failed to start. |
|
| Back to top |
|
tangent
Moderator
Joined: 16 Aug 2020
Posts: 362
Location: UK
|
| Posted: Fri 31 Jan '25 19:25 Post subject: |
|
|
I've tried your minimal test configuration too. It works without error for me in both Windows 10 and 11 x64 VMs.
Out of interest, though probably not relevant, I also tried an old Windows 7 x64 VM. This did crash with the same sort of event viewer error relating to libcrypto-3-x64.dll, namely an access exception 0xc0000005.
| James Blond wrote: | | The 0xc0000005 is an access violation. I wonder about that. |
So do I; most of such errors are "use after free", or null pointer exceptions, but is this a standalone issue with the later OpenSSL, or something Apache is causing when using a function in the OpenSSL DLL?
I enabled process dumps on the Windows 7 VM, restarted Apache, and opened the dump file it tirggered on the Windows 10 VM using WinDbg, to see if there was anything of interest. Without debug symbols, this is the only clue of interest; a reference to RC4_options, but not how we got here.
| Code: | EXCEPTION_RECORD: (.exr -1)
ExceptionAddress: 000007fee43606c0 (libcrypto_3_x64!RC4_options+0x0000000000139170)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 0000000000000000
Parameter[1]: 000007fee484dbe0
Attempt to read from address 000007fee484dbe0 |
I'm not sure if you're able to reproduce a similar result on your setup. |
|
| Back to top |
|
jmweb
Joined: 08 Jun 2017
Posts: 19
Location: USA, Charlotte
|
| Posted: Sat 01 Feb '25 8:01 Post subject: |
|
|
Since it is now confirmed the binaries do not run on Win 7, what are the chances that 2.4.63 can be distributed with a version of OpenSSL that is compatible with the OS's that are stated on the downloads page?
| Quote: |
Runs on: 7 SP1, Vista SP2, 8/8.1, 10, 11 Server 2008 SP2 / R2 SP1, Server 2012 / R2, Server 2016/2019/2022.
|
|
|
| Back to top |
|
Steffen
Moderator
Joined: 15 Oct 2005
Posts: 3112
Location: Hilversum, NL, EU
|
|
| Back to top |
|
tangent
Moderator
Joined: 16 Aug 2020
Posts: 362
Location: UK
|
| Posted: Sat 01 Feb '25 14:22 Post subject: |
|
|
| Done |
|
| Back to top |
|
Steffen
Moderator
Joined: 15 Oct 2005
Posts: 3112
Location: Hilversum, NL, EU
|
|
| Back to top |
|
tangent
Moderator
Joined: 16 Aug 2020
Posts: 362
Location: UK
|
| Posted: Sun 02 Feb '25 20:58 Post subject: |
|
|
Out of interest, I built Apache 2.4.63 with OpenSSL 3.3.2, and using the above minimal test configuration, this doesn't crash on Windows 7.
Is it worth considering offering this combination as an alternative download, until this OpenSSL 3.4.0 issue can be resolved? |
|
| Back to top |
|
Steffen
Moderator
Joined: 15 Oct 2005
Posts: 3112
Location: Hilversum, NL, EU
|
| Posted: Sun 02 Feb '25 21:42 Post subject: |
|
|
It can still be that Win7 has an issue.
I think also that the best we can do is replacing the 3.4.0 version with OpenSSL 3.3.2 |
|
| Back to top |
|
jmweb
Joined: 08 Jun 2017
Posts: 19
Location: USA, Charlotte
|
| Posted: Mon 03 Feb '25 7:28 Post subject: |
|
|
| tangent wrote: | Out of interest, I built Apache 2.4.63 with OpenSSL 3.3.2, and using the above minimal test configuration, this doesn't crash on Windows 7.
Is it worth considering offering this combination as an alternative download, until this OpenSSL 3.4.0 issue can be resolved? |
I would very much appreciate this. I do not like my development and production environments being out of sync with staging. |
|
| Back to top |
|
Steffen
Moderator
Joined: 15 Oct 2005
Posts: 3112
Location: Hilversum, NL, EU
|
| Posted: Mon 03 Feb '25 11:23 Post subject: |
|
|
Please try :
Removed, download page update with 3.3.2
Last edited by Steffen on Fri 07 Feb '25 11:46; edited 1 time in total |
|
| Back to top |
|
timj
Joined: 30 Jan 2025
Posts: 7
Location: UK
|
| Posted: Mon 03 Feb '25 13:51 Post subject: |
|
|
Thank you Steffen (and others that have looked at this). So far, after ~6 restarts, this hasn't crashed. Due to the intermittent nature of this issue I'm not confident enough to say it's resolved yet.
I spent most of Friday trying to figure this out but didn't make any progress:
- I wrote a simple script to start/stop the Apache service 100 times (using httpd -k start|stop). I ran this multiple times throughout the day with various config changes so probably started the service >1000 times in total. There were no crashes.
- jmweb's minimal config test also did not cause any crashes (using the same script).
- I still saw intermittent crashes (with the same Windows event logged) at startup. Indeed, this morning when I started my laptop the service crashed.
So I still can't reliably reproduce the crash. |
|
| Back to top |
|
admin
Site Admin
Joined: 15 Oct 2005
Posts: 700
|
| Posted: Mon 03 Feb '25 14:28 Post subject: |
|
|
Thanks.
Maybe you can test with 2.4.62 which has OpenSSL 3.1.7 |
|
| Back to top |
|
timj
Joined: 30 Jan 2025
Posts: 7
Location: UK
|
| Posted: Mon 03 Feb '25 16:18 Post subject: |
|
|
| admin wrote: | Thanks.
Maybe you can test with 2.4.62 which has OpenSSL 3.1.7 |
My servers (Win 2k16/2k19) are running the httpd-2.4.62-240904-win64-VS17 build. They all reboot weekly as part of their maintenance cycle.
My laptop has the same config (for dev/test) so I think the previous build is stable. |
|
| Back to top |
|
jmweb
Joined: 08 Jun 2017
Posts: 19
Location: USA, Charlotte
|
| Posted: Mon 03 Feb '25 19:48 Post subject: |
|
|
| This works. httpd starts without crashing! |
|
| Back to top |
|
tangent
Moderator
Joined: 16 Aug 2020
Posts: 362
Location: UK
|
| Posted: Tue 04 Feb '25 22:00 Post subject: |
|
|
I'm frustrated by this problem, and have spent some time looking into it, by way of comparing what happens with different combinations of Apache and OpenSSL versions. By compiling Apache 2.4.63 and OpenSSL
3.4.0 with debug enabled, using the PDB files so produced, I've managed to get some form of meaningful stack trace from the Windows 7 minidump file, viz:
| Code: | 0:000> k
# Child-SP RetAddr Call Site
00 00000000`0020e4d8 000007fe`fd6b1430 ntdll!ZwWaitForMultipleObjects+0xa
01 00000000`0020e4e0 00000000`77202ce3 KERNELBASE!WaitForMultipleObjectsEx+0xe8
02 00000000`0020e5e0 00000000`77279105 kernel32!WaitForMultipleObjectsExImplementation+0xb3
03 00000000`0020e670 00000000`77279287 kernel32!WerpReportFaultInternal+0x215
04 00000000`0020e710 00000000`772792df kernel32!WerpReportFault+0x77
05 00000000`0020e740 00000000`772794fc kernel32!BasepReportFault+0x1f
06 00000000`0020e770 00000000`77493398 kernel32!UnhandledExceptionFilter+0x1fc
07 00000000`0020e850 00000000`774185c8 ntdll! ?? ::FNODOBFM::`string'+0x2365
08 00000000`0020e880 00000000`77429d2d ntdll!_C_specific_handler+0x8c
09 00000000`0020e8f0 00000000`774191cf ntdll!RtlpExecuteHandlerForException+0xd
0a 00000000`0020e920 00000000`77451248 ntdll!RtlDispatchException+0x45a
0b 00000000`0020f000 000007fe`f1602070 ntdll!KiUserExceptionDispatch+0x2e
0c 00000000`0020f718 000007fe`f166a6fa libcrypto_3_x64!err_string_data_cmp [C:\Development\apache24\src\openssl-3.4.0\crypto\err\err.c @ 182]
0d 00000000`0020f720 000007fe`f166a244 libcrypto_3_x64!getrn+0x8a [C:\Development\apache24\src\openssl-3.4.0\crypto\lhash\lhash.c @ 347]
0e 00000000`0020f750 000007fe`f1601427 libcrypto_3_x64!OPENSSL_LH_retrieve+0x24 [C:\Development\apache24\src\openssl-3.4.0\crypto\lhash\lhash.c @ 183]
0f (Inline Function) --------`-------- libcrypto_3_x64!int_err_get_item+0x32 [C:\Development\apache24\src\openssl-3.4.0\crypto\err\err.c @ 194]
10 00000000`0020f780 000007fe`f1e40c37 libcrypto_3_x64!ERR_reason_error_string+0x77 [C:\Development\apache24\src\openssl-3.4.0\crypto\err\err.c @ 633]
11 00000000`0020f7c0 000007fe`f1e40e32 libssl_3_x64!ossl_err_load_SSL_strings+0x17 [C:\Development\apache24\src\openssl-3.4.0\ssl\ssl_err.c @ 627]
12 (Inline Function) --------`-------- libssl_3_x64!ossl_init_load_ssl_strings+0x5 [C:\Development\apache24\src\openssl-3.4.0\ssl\ssl_init.c @ 51]
13 00000000`0020f7f0 000007fe`f1690ba5 libssl_3_x64!ossl_init_load_ssl_strings_ossl_+0x12 [C:\Development\apache24\src\openssl-3.4.0\ssl\ssl_init.c @ 43]
14 00000000`0020f820 000007fe`f1e40d71 libcrypto_3_x64!CRYPTO_THREAD_run_once+0x55 [C:\Development\apache24\src\openssl-3.4.0\crypto\threads_win.c @ 557]
15 00000000`0020f850 000007fe`f1e42abc libssl_3_x64!OPENSSL_init_ssl+0xf1 [C:\Development\apache24\src\openssl-3.4.0\ssl\ssl_init.c @ 104]
16 00000000`0020f880 00000000`7146bd13 libssl_3_x64!SSL_CTX_new_ex+0x7c [C:\Development\apache24\src\openssl-3.4.0\ssl\ssl_lib.c @ 3870]
17 00000000`0020f8d0 00000000`7146d102 mod_ssl!ssl_init_ctx_protocol+0x273 [C:\Development\apache24\src\httpd-2.4.63\modules\ssl\ssl_engine_init.c @ 694]
18 00000000`0020f990 00000000`7146f55a mod_ssl!ssl_init_ctx+0x32 [C:\Development\apache24\src\httpd-2.4.63\modules\ssl\ssl_engine_init.c @ 1269]
19 00000000`0020f9d0 00000000`7146ac1d mod_ssl!ssl_init_server_ctx+0x29a [C:\Development\apache24\src\httpd-2.4.63\modules\ssl\ssl_engine_init.c @ 2028]
1a 00000000`0020fa90 00000000`7146a7ac mod_ssl!ssl_init_ConfigureServer+0xdd [C:\Development\apache24\src\httpd-2.4.63\modules\ssl\ssl_engine_init.c @ 2130]
1b 00000000`0020faf0 00000000`712eefd5 mod_ssl!ssl_init_Module+0x6dc [C:\Development\apache24\src\httpd-2.4.63\modules\ssl\ssl_engine_init.c @ 406]
1c 00000000`0020fbc0 00000001`3f153bd0 libhttpd!ap_run_post_config+0x85 [C:\Development\apache24\src\httpd-2.4.63\server\config.c @ 102]
1d 00000000`0020fc00 00000001`3f155db9 httpd!main+0x10a0 [C:\Development\apache24\src\httpd-2.4.63\server\main.c @ 831]
1e 00000000`0020fdb0 00000001`3f155c62 httpd!invoke_main+0x39 [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl @ 79]
1f 00000000`0020fe00 00000001`3f155b1e httpd!__scrt_common_main_seh+0x132 [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl @ 288]
20 00000000`0020fe70 00000001`3f155e4e httpd!__scrt_common_main+0xe [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl @ 331]
21 00000000`0020fea0 00000000`771f652d httpd!mainCRTStartup+0xe [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_main.cpp @ 17]
22 00000000`0020fed0 00000000`7742c541 kernel32!BaseThreadInitThunk+0xd
23 00000000`0020ff00 00000000`00000000 ntdll!RtlUserThreadStart+0x1d |
I also added SSL trace logging to jmweb's minimal httpd configuration, to see how far Apache got before it died, hoping this would help with the problem analysis, viz:
| Code: | [Tue Feb 04 11:26:39.088474 2025] [ssl:info] [pid 5040:tid 120] AH01883: Init: Initialized OpenSSL library
[Tue Feb 04 11:26:39.089586 2025] [ssl:trace2] [pid 5040:tid 120] ssl_engine_rand.c(125): Init: Seeding PRNG with 0 bytes of entropy
[Tue Feb 04 11:26:39.090699 2025] [ssl:debug] [pid 5040:tid 120] ssl_engine_init.c(364): AH01886: OpenSSL has FIPS mode disabled
[Tue Feb 04 11:26:39.090699 2025] [ssl:info] [pid 5040:tid 120] AH01887: Init: Initializing (virtual) servers for SSL
[Tue Feb 04 11:26:39.090699 2025] [ssl:info] [pid 5040:tid 120] AH01914: Configuring server 192.168.56.60:443 for SSL protocol
[Tue Feb 04 11:26:39.090699 2025] [ssl:trace3] [pid 5040:tid 120] ssl_engine_init.c(648): Creating new SSL context (protocols: TLSv1, TLSv1.1, TLSv1.2, TLSv1.3)
[Tue Feb 04 11:26:39.091811 2025] [ssl:trace1] [pid 5040:tid 120] ssl_engine_init.c(1042): Configuring permitted SSL ciphers [HIGH:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4:!SSLv2:!SSLv3:!aNULL:!eNULL:!EXP]
[Tue Feb 04 11:26:39.091811 2025] [ssl:debug] [pid 5040:tid 120] ssl_engine_init.c(536): AH01893: Configuring TLS extension handling
[Tue Feb 04 11:26:39.092924 2025] [ssl:trace3] [pid 5040:tid 120] ssl_util_ssl.c(436): [192.168.56.60:443] modssl_X509_match_name: expecting name '192.168.56.60', NOT matched by ID 'localhost'
[Tue Feb 04 11:26:39.092924 2025] [ssl:debug] [pid 5040:tid 120] ssl_util_ssl.c(451): AH02412: [192.168.56.60:443] Cert does not match for name '192.168.56.60' [subject: CN=localhost,OU=IT,O=JM Web Services\\, Inc,L=Charlotte,ST=North Carolina,C=US / issuer: CN=localhost,OU=IT,O=JM Web Services\\, Inc,L=Charlotte,ST=North Carolina,C=US / serial: 0648570645604E1BBC7977C0E5C2D60D64217216 / notbefore: Jan 31 07:32:17 2025 GMT / notafter: Jan 31 07:32:17 2027 GMT]
[Tue Feb 04 11:26:39.092924 2025] [ssl:warn] [pid 5040:tid 120] AH01909: 192.168.56.60:443:0 server certificate does NOT include an ID which matches the server name
[Tue Feb 04 11:26:39.092924 2025] [ssl:info] [pid 5040:tid 120] AH02568: Certificate and private key 192.168.56.60:443:0 configured from C:/Apache24/test/ssl/localhost.crt and C:/Apache24/test/ssl/localhost.key
[Tue Feb 04 11:26:39.095149 2025] [ssl:info] [pid 5040:tid 120] AH01876: mod_ssl/2.4.63 compiled against Server: Apache/2.4.63, Library: OpenSSL/3.4.0 AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 192.168.56.60. Set the 'ServerName' directive globally to suppress this message
[Tue Feb 04 11:26:39.104049 2025] [ssl:info] [pid 5040:tid 120] AH01883: Init: Initialized OpenSSL library
[Tue Feb 04 11:26:39.104049 2025] [ssl:trace2] [pid 5040:tid 120] ssl_engine_rand.c(125): Init: Seeding PRNG with 0 bytes of entropy
[Tue Feb 04 11:26:39.104049 2025] [ssl:debug] [pid 5040:tid 120] ssl_engine_init.c(364): AH01886: OpenSSL has FIPS mode disabled
[Tue Feb 04 11:26:39.104049 2025] [ssl:warn] [pid 5040:tid 120] AH01873: Init: Session Cache is not configured [hint: SSLSessionCache]
[Tue Feb 04 11:26:39.104049 2025] [ssl:info] [pid 5040:tid 120] AH01887: Init: Initializing (virtual) servers for SSL
[Tue Feb 04 11:26:39.104049 2025] [ssl:info] [pid 5040:tid 120] AH01914: Configuring server 192.168.56.60:443 for SSL protocol
[Tue Feb 04 11:26:39.104049 2025] [ssl:trace3] [pid 5040:tid 120] ssl_engine_init.c(648): Creating new SSL context (protocols: TLSv1, TLSv1.1,
TLSv1.2, TLSv1.3) |
I'm no expert looking through the modules and code referenced in the stack trace to know what's going on, but what I have noticed is Apache seems to request a new SSL context twice for the same virtual server.
The OpenSSL 3.4.0 library triggers the exception error during the second request to create this SSL context.
By comparison, when using OpenSSL 3.3.2, the library call returns and Apache continues with its configuration and functionality as normal.
So why does the Apache ssl_module call ssl_engine_init.c twice to create an SSL context, and equally important why does OpenSSL 3.4.0 appear to crash with such a request (albeit ostensibly on Windows 7)?
The relevant chunk of OpenSSL code appears to be ssl\ssl_init.c, and this has notably changed between OpenSSL 3.3.2 and 3.4.0. Specifically, the function ssl_library_stop() called in the event of an error has been removed. This was defined in ssl\ssl_ciph.c and that has also changed significantly between the two releases. I've no idea if this is relevant, but it just struck me as being notable.
So are we caught between two stools here? Is this a defect within the updated OpenSSL 3.4.0 code, or a fault with Apache for appearing to request a duplicate SSL context?
I shall add this detail to the Bugzilla post. |
|
| Back to top |
|
jmweb
Joined: 08 Jun 2017
Posts: 19
Location: USA, Charlotte
|
| Posted: Tue 04 Feb '25 23:09 Post subject: |
|
|
| Great work! |
|
| Back to top |
|
admin
Site Admin
Joined: 15 Oct 2005
Posts: 700
|
|
| Back to top |
|
jmweb
Joined: 08 Jun 2017
Posts: 19
Location: USA, Charlotte
|
| Posted: Wed 05 Feb '25 21:14 Post subject: |
|
|
Based on the discussion, it appears it is a difficult bug to track down. It also appears to be 3.4.0 specific. If I could vote, I would vote to redistribute httpd with an openssl version that is more stable. |
|
| Back to top |
|
