[Sequoia] Bug ?

[BESSON-DEBLON, Pierre (SOGETI HIGH TECH)]

Hello,

We use sequoia with 2 controllers on RAIDb1, sequoia is 2.10.9 with jgroups.

We experienced this weird behavior :

# on controller 1 (c1) backend deactivated, on c2 backend activated
c2.log -> 29 Apr 2009 12:56:12,400 | sequoia.controller.RequestManager.db | Database backend db2 is now enabled

# c1 receive a read request and forward to c2
c1.log ->
29 Apr 2009 12:56:16,271 | sequoia.controller.virtualdatabase.request.db | S 844424930131983 0 SELECT ...
29 Apr 2009 12:56:16,273 | sequoia.controller.RequestManager.db |Sending request SELECT ... to Member c2

c2.log ->
29 Apr 2009 12:56:16,353 | sequoia.controller.virtualdatabase.db | S 844424930131983 0 SELECT ...
29 Apr 2009 12:56:16,355 | sequoia.controller.virtualdatabase.db | 172.20.1.81:65209: Scheduling distributedRequest 844424930131983 from Member c1
29 Apr 2009 12:56:16,355 | sequoia.controller.distributedvirtualdatabase.request.db | S 844424930131983 0 SELECT ...
29 Apr 2009 12:56:16,356 | sequoia.controller.virtualdatabase.db | handleMessageMultiThreaded (class org.continuent.sequoia.controller.virtualdatabase.protocol.ExecRemoteStatementExecuteQuery): S 844424930131983 0 SELECT ...
29 Apr 2009 12:56:16,356 | sequoia.controller.virtualdatabase.db | 172.20.1.81:65209: Executing distributedRequest 844424930131983 from Member c1
29 Apr 2009 12:56:16,356 | sequoia.controller.RequestManager.db | Database schema is dirty, refreshing it
29 Apr 2009 12:56:16,359 | sequoia.controller.RequestManager.db | Scheduling read Request (844424930131983): SELECT ...
29 Apr 2009 12:56:16,359 | sequoia.controller.RequestManager.db | Load balancing read request (844424930131983): SELECT ...

and right after in c2.log file
29 Apr 2009 12:56:21,268 | sequoia.controller.virtualdatabase.VirtualDatabaseWorkerThread.db | StatementExecuteQuery command
29 Apr 2009 12:56:21,268 | sequoia.controller.virtualdatabase.request.db | S 562949953430692 0 SELECT ...

c1.log
29 Apr 2009 12:56:21,233 | sequoia.controller.RequestManager.db | Request SELECT ... completed.
29 Apr 2009 12:56:21,234 | sequoia.controller.RequestManager.db | An error occured while executing remote select request 844424930131983
org.continuent.sequoia.common.exceptions.NoMoreBackendException
at org.continuent.sequoia.controller.requestmanager.distributed.RAIDb1DistributedRequestManager.execRemoteStatementExecuteQuery(Unknown Source)
at org.continuent.sequoia.controller.requestmanager.distributed.DistributedRequestManager.statementExecuteQuery(Unknown Source)
at org.continuent.sequoia.controller.virtualdatabase.VirtualDatabase.statementExecuteQuery(Unknown Source)
at org.continuent.sequoia.controller.virtualdatabase.VirtualDatabaseWorkerThread.statementExecuteQuery(Unknown Source)
at org.continuent.sequoia.controller.virtualdatabase.VirtualDatabaseWorkerThread.run(Unknown Source)
29 Apr 2009 12:56:21,234 | sequoia.controller.virtualdatabase.db | Request '844424930131983' failed (null)
29 Apr 2009 12:56:21,234 | sequoia.controller.virtualdatabase.VirtualDatabaseWorkerThread.db | Error during command execution (null)


So, How can request fail with NoMoreBackendException despite db2 on c2 is enabled ?
Is it normal to have no more log in c2.log to indicate Exception ?, is it possible that request is hanging on c2 and a timeout raise the exception ?


Thanks for any help.


Pierre Besson-Deblon


The information in this e-mail is confidential. The contents may not be disclosed or used by anyone other then the addressee. Access to this e-mail by anyone else is unauthorised.
If you are not the intended recipient, please notify Airbus immediately and delete this e-mail.
Airbus cannot accept any responsibility for the accuracy or completeness of this e-mail as it has been sent over public networks. If you have any concerns over the content of this message or its Accuracy or Integrity, please contact Airbus immediately.
All outgoing e-mails from Airbus are checked using regularly updated virus scanning software but you should take whatever measures you deem to be appropriate to ensure that this message and any attachments are virus free.

[BESSON-DEBLON, Pierre (SOGETI HIGH TECH)]

continue ... I made an edit at the end of previous message


-----Message d'origine-----
De : sequoia-bounces-RyYwo1q5J+pm4DMUGMHt/jIQnlzKTOB+***@public.gmane.org
[mailto:sequoia-bounces-RyYwo1q5J+pm4DMUGMHt/jIQnlzKTOB+***@public.gmane.org]De la part de
BESSON-DEBLON, Pierre (SOGETI HIGH TECH)
Envoyé : 18 May 2009 19:32
À : Sequoia general mailing list
Objet : [Sequoia] Bug ?


Hello,

We use sequoia with 2 controllers on RAIDb1, sequoia is 2.10.9 with jgroups.

We experienced this weird behavior :

# on controller 1 (c1) backend deactivated, on c2 backend activated
c2.log -> 29 Apr 2009 12:56:12,400 | sequoia.controller.RequestManager.db | Database backend db2 is now enabled

# c1 receive a read request and forward to c2
c1.log ->
29 Apr 2009 12:56:16,271 | sequoia.controller.virtualdatabase.request.db | S 844424930131983 0 SELECT ...
29 Apr 2009 12:56:16,273 | sequoia.controller.RequestManager.db |Sending request SELECT ... to Member c2

c2.log ->
29 Apr 2009 12:56:16,353 | sequoia.controller.virtualdatabase.db | S 844424930131983 0 SELECT ...
29 Apr 2009 12:56:16,355 | sequoia.controller.virtualdatabase.db | 172.20.1.81:65209: Scheduling distributedRequest 844424930131983 from Member c1
29 Apr 2009 12:56:16,355 | sequoia.controller.distributedvirtualdatabase.request.db | S 844424930131983 0 SELECT ...
29 Apr 2009 12:56:16,356 | sequoia.controller.virtualdatabase.db | handleMessageMultiThreaded (class org.continuent.sequoia.controller.virtualdatabase.protocol.ExecRemoteStatementExecuteQuery): S 844424930131983 0 SELECT ...
29 Apr 2009 12:56:16,356 | sequoia.controller.virtualdatabase.db | 172.20.1.81:65209: Executing distributedRequest 844424930131983 from Member c1
29 Apr 2009 12:56:16,356 | sequoia.controller.RequestManager.db | Database schema is dirty, refreshing it
29 Apr 2009 12:56:16,359 | sequoia.controller.RequestManager.db | Scheduling read Request (844424930131983): SELECT ...
29 Apr 2009 12:56:16,359 | sequoia.controller.RequestManager.db | Load balancing read request (844424930131983): SELECT ...

and right after in c2.log file
29 Apr 2009 12:56:21,268 | sequoia.controller.virtualdatabase.VirtualDatabaseWorkerThread.db | StatementExecuteQuery command
29 Apr 2009 12:56:21,268 | sequoia.controller.virtualdatabase.request.db | S 562949953430692 0 SELECT ...

c1.log
29 Apr 2009 12:56:21,233 | sequoia.controller.RequestManager.db | Request SELECT ... completed.
29 Apr 2009 12:56:21,234 | sequoia.controller.RequestManager.db | An error occured while executing remote select request 844424930131983
org.continuent.sequoia.common.exceptions.NoMoreBackendException
at org.continuent.sequoia.controller.requestmanager.distributed.RAIDb1DistributedRequestManager.execRemoteStatementExecuteQuery(Unknown Source)
at org.continuent.sequoia.controller.requestmanager.distributed.DistributedRequestManager.statementExecuteQuery(Unknown Source)
at org.continuent.sequoia.controller.virtualdatabase.VirtualDatabase.statementExecuteQuery(Unknown Source)
at org.continuent.sequoia.controller.virtualdatabase.VirtualDatabaseWorkerThread.statementExecuteQuery(Unknown Source)
at org.continuent.sequoia.controller.virtualdatabase.VirtualDatabaseWorkerThread.run(Unknown Source)
29 Apr 2009 12:56:21,234 | sequoia.controller.virtualdatabase.db | Request '844424930131983' failed (null)
29 Apr 2009 12:56:21,234 | sequoia.controller.virtualdatabase.VirtualDatabaseWorkerThread.db | Error during command execution (null)


So, How can request fail with NoMoreBackendException despite db2 on c2 is enabled ?
Is it normal to have no more log in c2.log to indicate Exception ?, is it possible that request is hanging on c2 and a timeout raise the exception ?

edit >> found this in c2.log
29 Apr 2009 12:56:24,553 | sequoia.controller.virtualdatabase.VirtualDatabaseWorkerThread.db | Runtime error during command execution (null)
java.lang.StackOverflowError
29 Apr 2009 12:56:24,554 | sequoia.controller.virtualdatabase.VirtualDatabaseWorkerThread.db | Reset command
I think it could help


Thanks for any help.


Pierre Besson-Deblon


The information in this e-mail is confidential. The contents may not be disclosed or used by anyone other then the addressee. Access to this e-mail by anyone else is unauthorised.
If you are not the intended recipient, please notify Airbus immediately and delete this e-mail.
Airbus cannot accept any responsibility for the accuracy or completeness of this e-mail as it has been sent over public networks. If you have any concerns over the content of this message or its Accuracy or Integrity, please contact Airbus immediately.
All outgoing e-mails from Airbus are checked using regularly updated virus scanning software but you should take whatever measures you deem to be appropriate to ensure that this message and any attachments are virus free.

[Emmanuel Cecchet]

Hi Pierre,

Note that the mailing list has now moved to
sequoiadb-discuss-5NWGOfrQmneRv+***@public.gmane.org
The java.lang.StackOverflowError error probably comes from an infinite
retry loop somewhere (hard to tell where since the exception is null).
Could you reproduce the issue with Sequoia 2.10.10? I don't have the
source code of 2.10.9 readily available and a number of fixes (as well
as a JGroups update) have been performed in 2.10.10.
Does the problem happen systematically?

Thanks for your feedback,
Emmanuel

Post by BESSON-DEBLON, Pierre (SOGETI HIGH TECH)
continue ... I made an edit at the end of previous message
-----Message d'origine-----
BESSON-DEBLON, Pierre (SOGETI HIGH TECH)
Envoyé : 18 May 2009 19:32
À : Sequoia general mailing list
Objet : [Sequoia] Bug ?
Hello,
We use sequoia with 2 controllers on RAIDb1, sequoia is 2.10.9 with jgroups.
# on controller 1 (c1) backend deactivated, on c2 backend activated
c2.log -> 29 Apr 2009 12:56:12,400 | sequoia.controller.RequestManager.db | Database backend db2 is now enabled
# c1 receive a read request and forward to c2
c1.log ->
29 Apr 2009 12:56:16,271 | sequoia.controller.virtualdatabase.request.db | S 844424930131983 0 SELECT ...
29 Apr 2009 12:56:16,273 | sequoia.controller.RequestManager.db |Sending request SELECT ... to Member c2
c2.log ->
29 Apr 2009 12:56:16,353 | sequoia.controller.virtualdatabase.db | S 844424930131983 0 SELECT ...
29 Apr 2009 12:56:16,355 | sequoia.controller.virtualdatabase.db | 172.20.1.81:65209: Scheduling distributedRequest 844424930131983 from Member c1
29 Apr 2009 12:56:16,355 | sequoia.controller.distributedvirtualdatabase.request.db | S 844424930131983 0 SELECT ...
29 Apr 2009 12:56:16,356 | sequoia.controller.virtualdatabase.db | handleMessageMultiThreaded (class org.continuent.sequoia.controller.virtualdatabase.protocol.ExecRemoteStatementExecuteQuery): S 844424930131983 0 SELECT ...
29 Apr 2009 12:56:16,356 | sequoia.controller.virtualdatabase.db | 172.20.1.81:65209: Executing distributedRequest 844424930131983 from Member c1
29 Apr 2009 12:56:16,356 | sequoia.controller.RequestManager.db | Database schema is dirty, refreshing it
29 Apr 2009 12:56:16,359 | sequoia.controller.RequestManager.db | Scheduling read Request (844424930131983): SELECT ...
29 Apr 2009 12:56:16,359 | sequoia.controller.RequestManager.db | Load balancing read request (844424930131983): SELECT ...
and right after in c2.log file
29 Apr 2009 12:56:21,268 | sequoia.controller.virtualdatabase.VirtualDatabaseWorkerThread.db | StatementExecuteQuery command
29 Apr 2009 12:56:21,268 | sequoia.controller.virtualdatabase.request.db | S 562949953430692 0 SELECT ...
c1.log
29 Apr 2009 12:56:21,233 | sequoia.controller.RequestManager.db | Request SELECT ... completed.
29 Apr 2009 12:56:21,234 | sequoia.controller.RequestManager.db | An error occured while executing remote select request 844424930131983
org.continuent.sequoia.common.exceptions.NoMoreBackendException
at org.continuent.sequoia.controller.requestmanager.distributed.RAIDb1DistributedRequestManager.execRemoteStatementExecuteQuery(Unknown Source)
at org.continuent.sequoia.controller.requestmanager.distributed.DistributedRequestManager.statementExecuteQuery(Unknown Source)
at org.continuent.sequoia.controller.virtualdatabase.VirtualDatabase.statementExecuteQuery(Unknown Source)
at org.continuent.sequoia.controller.virtualdatabase.VirtualDatabaseWorkerThread.statementExecuteQuery(Unknown Source)
at org.continuent.sequoia.controller.virtualdatabase.VirtualDatabaseWorkerThread.run(Unknown Source)
29 Apr 2009 12:56:21,234 | sequoia.controller.virtualdatabase.db | Request '844424930131983' failed (null)
29 Apr 2009 12:56:21,234 | sequoia.controller.virtualdatabase.VirtualDatabaseWorkerThread.db | Error during command execution (null)
So, How can request fail with NoMoreBackendException despite db2 on c2 is enabled ?
Is it normal to have no more log in c2.log to indicate Exception ?, is it possible that request is hanging on c2 and a timeout raise the exception ?
edit >> found this in c2.log
29 Apr 2009 12:56:24,553 | sequoia.controller.virtualdatabase.VirtualDatabaseWorkerThread.db | Runtime error during command execution (null)
java.lang.StackOverflowError
29 Apr 2009 12:56:24,554 | sequoia.controller.virtualdatabase.VirtualDatabaseWorkerThread.db | Reset command
I think it could help
Thanks for any help.
Pierre Besson-Deblon
The information in this e-mail is confidential. The contents may not be disclosed or used by anyone other then the addressee. Access to this e-mail by anyone else is unauthorised.
If you are not the intended recipient, please notify Airbus immediately and delete this e-mail.
Airbus cannot accept any responsibility for the accuracy or completeness of this e-mail as it has been sent over public networks. If you have any concerns over the content of this message or its Accuracy or Integrity, please contact Airbus immediately.
All outgoing e-mails from Airbus are checked using regularly updated virus scanning software but you should take whatever measures you deem to be appropriate to ensure that this message and any attachments are virus free.
_______________________________________________
Sequoia mailing list
http://forge.continuent.org/mailman/listinfo/sequoia

...