logo
Apache Lounge
Webmasters

 

About Forum Index Downloads Search Register Log in RSS X


Keep Server Online

If you find the Apache Lounge, the downloads and overall help useful, please express your satisfaction with a donation.

or

Bitcoin

A donation makes a contribution towards the costs, the time and effort that's going in this site and building.

Thank You! Steffen

Your donations will help to keep this site alive and well, and continuing building binaries. Apache Lounge is not sponsored.
Post new topic   Forum Index -> Other Software View previous topic :: View next topic
Reply to topic   Topic: Jboss log size is around 10GB and application stopped workig
Author
Shrinidhi0409



Joined: 24 Sep 2021
Posts: 18
Location: India

PostPosted: Fri 07 Apr '23 8:16    Post subject: Jboss log size is around 10GB and application stopped workig Reply with quote

Hi,

Our application uses RHEL 8.5 and Jboss version is 4.2.1 GA version. we got heap.dump few months back due to some network glitches. After that we did not removed heap.dump file and re-started servers which was working fine till now. After that, application suddenly stopped working because of No disk space error. When we analyzed the disc space, Logs were occupying almost 12 GB (including 10 Gb of Jboss.log and heap.dump of 2 GB) which is huge. When we saw jboss.log which is getting updated with repeated logs as below.


WARN [JDBCSupport] SQLException caught, SQLState null code:0- assuming deadlock detected, try:18
org.jboss.util.NestedSQLException: Could not enlist in transaction on entering meta-aware object!; - nested throwable: (javax.transaction.SystemException: java.lang.Throwable: Unabled to enlist resource, see the previous warnings. tx=TransactionImple < ac, BasicAction: 6a288574:63f3:63d001d8:7c1fff status: ActionStatus.ABORT_ONLY >); - nested throwable: (org.jboss.resource.JBossResourceException: Could not enlist in transaction on entering meta-aware object!; - nested throwable: (javax.transaction.SystemException: java.lang.Throwable: Unabled to enlist resource, see the previous warnings. tx=TransactionImple < ac, BasicAction: 6a288574:63f3:63d001d8:7c1fff status: ActionStatus.ABORT_ONLY >))
at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:94)
at org.jboss.messaging.core.impl.JDBCSupport$JDBCTxRunner.execute(JDBCSupport.java:383)
at org.jboss.messaging.core.impl.JDBCSupport$JDBCTxRunner.executeWithRetry(JDBCSupport.java:407)
at org.jboss.messaging.core.impl.JDBCPersistenceManager.reapUnreferencedMessages(JDBCPersistenceManager.java:2357)
at org.jboss.messaging.core.impl.JDBCPersistenceManager.access$1000(JDBCPersistenceManager.java:7Cool
at org.jboss.messaging.core.impl.JDBCPersistenceManager$Reaper.run(JDBCPersistenceManager.java:2381)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
Caused by: org.jboss.resource.JBossResourceException: Could not enlist in transaction on entering meta-aware object!; - nested throwable: (javax.transaction.SystemException: java.lang.Throwable: Unabled to enlist resource, see the previous warnings. tx=TransactionImple < ac, BasicAction: 6a288574:63f3:63d001d8:7c1fff status: ActionStatus.ABORT_ONLY >)
at org.jboss.resource.connectionmanager.TxConnectionManager.managedConnectionReconnected(TxConnectionManager.java:343)
at org.jboss.resource.connectionmanager.BaseConnectionManager2.reconnectManagedConnection(BaseConnectionManager2.java:518)
at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:399)
at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:842)
at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:8Cool
... 7 more
Caused by: javax.transaction.SystemException: java.lang.Throwable: Unabled to enlist resource, see the previous warnings. tx=TransactionImple < ac, BasicAction: 6a288574:63f3:63d001d8:7c1fff status: ActionStatus.ABORT_ONLY >
at org.jboss.resource.conne..nManager.java:744)
at org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener.enlist(TxConnectionManager.java:577)
at org.jboss.resource.connectionmanager.TxConnectionManager.managedConnectionReconnected(TxConnectionManager.java:337)
... 11 more
05:51:09,339 WARN [JDBCSupport] Trying again after a pause
05:51:09,531 WARN [loggerI18N] [com.arjuna.ats.internal.jta.transaction.arjunacore.enliststarterror] [com.arjuna.ats.internal.jta.transaction.arjunacore.enliststarterror] TransactionImple.enlistResource - XAResource.start returned: XAException.XAER_PROTO for < 131075, 31, 29, 1-6a288574:63f3:63d001d8:7c20036a288574:63f3:63d001d8:7c2006^
ERROR [STDERR] org.jboss.resource.connectionmanager.JBossLocalXAException: Trying to start a new tx when old is not complete! old: < 131075, 31, 29, 1-6a288574:63f3:63d001d8:7bf8476a288574:63f3:63d001d8:7bf84a.... @^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ >, new < 131075, 31, 29, 1-6a288574:63f3:63d001d8:7c20036a288574:63f3:63d001d8:7c2006..@^@^@^@^@^@^@^@^@^@^@^@ >, flags 0
05:51:09,531 ERROR [STDERR] at org.jboss.resource.connectionmanager.TxConnectionManager$LocalXAResource.start(TxConnectionManager.java:886)
05:51:09,531 ERROR [STDERR] at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.enlistResource(TransactionImple.java:774)
05:51:09,531 ERROR [STDERR] at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.enlistResource(TransactionImple.java:497)
05:51:09,531 ERROR [STDERR] at org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener$TransactionSynchronization.enlist(TxConnectionManager.java:773)
05:51:09,531 ERROR [STDERR] at org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener.enlist(TxConnectionManager.java:564)
05:51:09,531 ERROR [STDERR] at org.jboss.resource.connectionmanager.TxConnectionManager.managedConnectionReconnected(TxConnectionManager.java:337)
05:51:09,531 ERROR [STDERR] at org.jboss.resource.connectionmanager.BaseConnectionManager2.reconnectManagedConnection(BaseConnectionManager2.java:518)
05:51:09,531 ERROR [STDERR] at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:399)
05:51:09,531 ERROR [STDERR] at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:842)
05:51:09,531 ERROR [STDERR] at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:8Cool
05:51:09,531 ERROR [STDERR] at org.jboss.messaging.core.impl.JDBCSupport$JDBCTxRunner.execute(JDBCSupport.java:383)
05:51:09,531 ERROR [STDERR] at org.jboss.messaging.core.impl.JDBCSupport$JDBCTxRunner.executeWithRetry(JDBCSupport.java:407)
05:51:09,531 ERROR [STDERR] at org.jboss.messaging.core.impl.JDBCPersistenceManager.reapUnreferencedMessages(JDBCPersistenceManager.java:2357)
05:51:09,531 ERROR [STDERR] at org.jboss.messaging.core.impl.JDBCPersistenceManager.access$1000(JDBCPersistenceManager.java:7Cool
05:51:09,531 ERROR [STDERR] at org.jboss.messaging.core.impl.JDBCPersistenceManager$Reaper.run(JDBCPersistenceManager.java:2381)
05:51:09,531 ERROR [STDERR] at java.util.TimerThread.mainLoop(Timer.java:555)
05:51:09,531 ERROR [STDERR] at java.util.TimerThread.run(Timer.java:505)




ERROR [JDBCSupport] Retried 25 times, now giving up
05:51:06,096 ERROR [JDBCPersistenceManager] Failed to reap
java.lang.IllegalStateException: Failed to excecute transaction
at org.jboss.messaging.core.impl.JDBCSupport$JDBCTxRunner.executeWithRetry(JDBCSupport.java:423)
at org.jboss.messaging.core.impl.JDBCPersistenceManager.reapUnreferencedMessages(JDBCPersistenceManager.java:2357)
at org.jboss.messaging.core.impl.JDBCPersistenceManager.access$1000(JDBCPersistenceManager.java:7Cool
at org.jboss.messaging.core.impl.JDBCPersistenceManager$Reaper.run(JDBCPersistenceManager.java:2381)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)

Can you please let us know about,

1. Why this error occured?
2. In what scenario it occurs??
3. What caused jboss.log to update with repeated logs and accumulating so much space?


Regards,
Shrinidhi S
Back to top
admin
Site Admin


Joined: 15 Oct 2005
Posts: 677

PostPosted: Fri 07 Apr '23 8:46    Post subject: Reply with quote

Not related to Apache HTTPD. Hopefully someone can help.
Back to top
tangent
Moderator


Joined: 16 Aug 2020
Posts: 305
Location: UK

PostPosted: Fri 07 Apr '23 20:44    Post subject: Reply with quote

I used to support a legacy JBoss 4.2.3 GA application behind Apache, which generated large log files, as well as the occasional heap dump file.

Re your questions, there are clues in your log file segment.

Quote:
1. Why this error occured?
2. In what scenario it occurs??
3. What caused jboss.log to update with repeated logs and accumulating so much space?

I'd suggest your 'network glitch' caused blocked threads due to multiple requests for a database resource that was previously locked. The Java database code evidently wasn't written to recover from this situation, by using some sort of timeout mechanism to release locks and roll back incomplete transactions.

If you still have an application team who support the Java code, you could ask them to review that part of the application, specifically by analysing one of your heap dump files.

Re your storage space problem, I used to use the cronolog utility (still readily available) to age and rotate JBoss logs (without restarting JBoss), which could then be reviewed and aged by a suitable cron job task.

You'll need to edit the JBoss startup run.sh script with suitable changes to pipe the Java output into cronolog, e.g.

Code:
CRONOLOG="/usr/local/bin/cronolog"
CRONOLOG_OPTS="--hardlink jboss_server.log archivelogs/jboss_server.%Y-%m-%d"

"$JAVA" $JAVA_OPTS \
    -Djava.endorsed.dirs="$JBOSS_ENDORSED_DIRS" \
    -classpath "$JBOSS_CLASSPATH" \
    org.jboss.Main "$@" 2>&1 | $CRONOLOG $CRONOLOG_OPTS &
Back to top


Reply to topic   Topic: Jboss log size is around 10GB and application stopped workig View previous topic :: View next topic
Post new topic   Forum Index -> Other Software