11 Replies Latest reply on Jun 19, 2013 11:40 AM by ataylor

Both severs from an ex live-backup configuration continuously log warnings after the live-backup config is undone

crytek Jun 19, 2013 9:37 AM

Conditions:

Two Jboss AS 7.2.0 servers with HQ 2.3.0 CR1 inside cluster configuration.

Setup a live-backup configuration, using replication:

(Live-Backup config - CLI commands)

Live :

/subsystem=messaging/hornetq-server=default/:write-attribute(name=shared-store, value=false)

/subsystem=messaging/hornetq-server=default/:write-attribute(name=allow-failback, value=true)

/subsystem=messaging/hornetq-server=default/:write-attribute(name=failover-on-shutdown, value=true)

/subsystem=messaging/hornetq-server=default/:write-attribute(name=check-for-live-server, value=true)

/subsystem=messaging/hornetq-server=default/:write-attribute(name=backup-group-name, value=bk_gr_name)

Backup:

/subsystem=messaging/hornetq-server=default/:write-attribute(name=backup, value=true)

/subsystem=messaging/hornetq-server=default/:write-attribute(name=shared-store, value=false)

/subsystem=messaging/hornetq-server=default/:write-attribute(name=allow-failback, value=true)

/subsystem=messaging/hornetq-server=default/:write-attribute(name=check-for-live-server, value=true)

/subsystem=messaging/hornetq-server=default/:write-attribute(name=failover-on-shutdown, value=true)

/subsystem=messaging/hornetq-server=default/:write-attribute(name=backup-group-name, value=bk_gr_name)

Play a little bit with this configuration: stop LIVE, BACKUP becomes active. Things go as expected.

Undo the live backup configuration:

LIVE:

/subsystem=messaging/hornetq-server=default/:undefine-attribute(name=shared-store)

/subsystem=messaging/hornetq-server=default/:undefine-attribute(name=allow-failback)

/subsystem=messaging/hornetq-server=default/:undefine-attribute(name=failover-on-shutdown)

/subsystem=messaging/hornetq-server=default/:undefine-attribute(name=check-for-live-server)

/subsystem=messaging/hornetq-server=default/:undefine-attribute(name=backup-group-name)

BACKUP:

/subsystem=messaging/hornetq-server=default/:undefine-attribute(name=backup)

/subsystem=messaging/hornetq-server=default/:undefine-attribute(name=shared-store)

/subsystem=messaging/hornetq-server=default/:undefine-attribute(name=allow-failback)

/subsystem=messaging/hornetq-server=default/:undefine-attribute(name=check-for-live-server)

/subsystem=messaging/hornetq-server=default/:undefine-attribute(name=failover-on-shutdown)

/subsystem=messaging/hornetq-server=default/:undefine-attribute(name=backup-group-name)

Restart both servers. After restart, both servers continuously log the same warning:

WARN [org.hornetq.core.client] (hornetq-discovery-group-thread-dg-group1) HQ212050: There are more than one servers on the network broadcasting the same node id. You will see this message exactly once (per node) if a node is restarted, in which case it can be safely ignored. But if it is logged continuously it means you really do have more than one node on the same network active concurrently with the same node id. This could occur if you have a backup node active at the same time as its live node. nodeID=79d6e606-d831-11e2-9ce6-d9222ff73a66

Only after removing "data" folder from the live server and restart it , the warning doesn't appear anymore.

Any ideas ? Maybe I am doing something wrong ?

live.log.zip 4.6 KB
backup.log.zip 3.5 KB

1. Re: Both severs from an ex live-backup configuration continuously log warnings after the live-backup config is undone

ataylor Jun 19, 2013 5:41 AM (in response to crytek)

make sue both servers are using different Journals
Actions
2. Re: Both severs from an ex live-backup configuration continuously log warnings after the live-backup config is undone

crytek Jun 19, 2013 5:48 AM (in response to ataylor)

The servers are implicitly using different journals because they are on different physical machines. They do not share the same journal data folder.
Actions
3. Re: Both severs from an ex live-backup configuration continuously log warnings after the live-backup config is undone

ataylor Jun 19, 2013 7:20 AM (in response to crytek)

when you start both servers the backup should come up as a backup, if you are seeing this message then both servers are live, could you post your logs please?
Actions
4. Re: Both severs from an ex live-backup configuration continuously log warnings after the live-backup config is undone

crytek Jun 19, 2013 9:40 AM (in response to ataylor)

I've attached the server logs from live and backup , when both servers are removed from the live-backup configuration, and restarted ... the las warnings are continuously logged. I am wondering if someone managed to reproduce this scenario or maybe the problem is on my side.
Actions
5. Re: Both severs from an ex live-backup configuration continuously log warnings after the live-backup config is undone

ataylor Jun 19, 2013 9:44 AM (in response to crytek)

The backup server is being started as a live server not a backup, see:

2013-06-19 15:02:24,886 INFO [org.hornetq.core.server] (MSC service thread 1-2) HQ221001: live server is starting with configuration HornetQ Configuration (clustered=true,backup=false,sharedStore=true,journalDirectory=/opt/ui/etc/slot1/standalone/data/messagingjournal,bindingsDirectory=/opt/ui/etc/slot1/standalone/data/messagingbindings,largeMessagesDirectory=/opt/ui/etc/slot1/standalone/data/messaginglargemessages,pagingDirectory=/opt/ui/etc/slot1/standalone/data/messagingpaging)

can you check your config
Actions
6. Re: Both severs from an ex live-backup configuration continuously log warnings after the live-backup config is undone

crytek Jun 19, 2013 10:47 AM (in response to ataylor)

I can see now, however, if I am running the CLI command for viewing the HQ config, I cannot see where the problem is. I believe everything should look as before the backup was configured.
The fact is that I am undoing the live-backup configuration, so of course the "backup" server should not behave as a backup anymore. The two servers should just be in cluster mode as before, and nothing more. The only difference is that they are continuously logging warnings and only if I am removinf the "data" directory from jboss AS , the warning is not logged anymore.

standalone@vm-fslevoaca-4933.development.lan:10099 /] /subsystem=messaging/hornetq-server=default/:read-resource
{
    "outcome" => "success",
    "result" => {
        "acceptor" => undefined,
        "allow-failback" => true,
        "async-connection-execution-enabled" => true,
        "backup" => false,
        "backup-group-name" => undefined,
        "bridge" => undefined,
        "check-for-live-server" => false,
        "cluster-password" => expression "${jboss.default.multicast.address:230.0.0.4}",
        "cluster-user" => "hornetQClusterUser",
        "clustered" => false,
        "connection-ttl-override" => -1L,
        "connector" => undefined,
        "connector-service" => undefined,
        "create-bindings-dir" => true,
        "create-journal-dir" => true,
        "divert" => undefined,
        "failback-delay" => 5000L,
        "failover-on-shutdown" => false,
        "grouping-handler" => undefined,
        "id-cache-size" => 15000,
        "jms-topic" => undefined,
        "jmx-domain" => "org.hornetq",
        "jmx-management-enabled" => true,
        "journal-buffer-size" => undefined,
        "journal-buffer-timeout" => undefined,
        "journal-compact-min-files" => 10,
        "journal-compact-percentage" => 30,
        "journal-file-size" => 102400L,
        "journal-max-io" => undefined,
        "journal-min-files" => 2,
        "journal-sync-non-transactional" => true,
        "journal-sync-transactional" => true,
        "journal-type" => "ASYNCIO",
        "log-journal-write-rate" => false,
        "management-address" => "jms.queue.hornetq.management",
        "management-notification-address" => "hornetq.notifications",
        "memory-measure-interval" => -1L,
        "memory-warning-threshold" => 25,
        "message-counter-enabled" => false,
        "message-counter-max-day-history" => 10,
        "message-counter-sample-period" => 10000L,
        "message-expiry-scan-period" => 30000L,
        "message-expiry-thread-priority" => 3,
        "page-max-concurrent-io" => 5,
        "perf-blast-pages" => -1,
        "persist-delivery-count-before-delivery" => false,
        "persist-id-cache" => true,
        "persistence-enabled" => true,
        "queue" => undefined,
        "remoting-incoming-interceptors" => undefined,
        "remoting-interceptors" => undefined,
        "remoting-outgoing-interceptors" => undefined,
        "replication-clustername" => undefined,
        "run-sync-speed-test" => false,
        "scheduled-thread-pool-max-size" => 5,
        "security-domain" => "other",
        "security-enabled" => true,
        "security-invalidation-interval" => 10000L,
        "server-dump-interval" => -1L,
        "shared-store" => true,
        "thread-pool-max-size" => 30,
        "transaction-timeout" => 300000L,
        "transaction-timeout-scan-period" => 1000L,
        "wild-card-routing-enabled" => true,
        "address-setting" => {"#" => undefined},
        "broadcast-group" => {"bg-group1" => undefined},
        "cluster-connection" => {"my-cluster" => undefined},
        "connection-factory" => {
            "InVmConnectionFactory" => undefined,
            "InVmXAConnectionFactory" => undefined,
            "NettyConnectionFactory" => undefined,
            "NettyXAConnectionFactory" => undefined
        },
        "core-address" => {
            "jms.queue.DLQ" => undefined,
            "jms.queue.testQueue1" => undefined,
            "sf.my-cluster.8206ce64-d8cf-11e2-9d03-67134ae63711" => undefined,
            "jms.queue.resultQueue" => undefined,
            "jms.queue.liveBackupQ" => undefined,
            "jms.queue.PsiCallbackQueue" => undefined,
            "jms.queue.ExpiryQueue" => undefined
        },
        "discovery-group" => {"dg-group1" => undefined},
        "in-vm-acceptor" => {"in-vm" => undefined},
        "in-vm-connector" => {"in-vm" => undefined},
        "jms-queue" => {
            "DLQ" => undefined,
            "testQueue1" => undefined,
            "ExpiryQueue" => undefined
        },
        "path" => {
            "journal-directory" => undefined,
            "paging-directory" => undefined,
            "bindings-directory" => undefined,
            "large-messages-directory" => undefined
        },
        "pooled-connection-factory" => {"hornetq-ra" => undefined},
        "remote-acceptor" => {
            "netty" => undefined,
            "netty-throughput" => undefined
        },
        "remote-connector" => {
            "netty" => undefined,
            "netty-throughput" => undefined
        },
        "runtime-queue" => {
            "jms.queue.testQueue1" => undefined,
            "sf.my-cluster.8206ce64-d8cf-11e2-9d03-67134ae63711" => undefined,
            "jms.queue.DLQ" => undefined,
            "jms.queue.liveBackupQ" => undefined,
            "jms.queue.resultQueue" => undefined,
            "jms.queue.ExpiryQueue" => undefined,
            "jms.queue.PsiCallbackQueue" => undefined
        },
        "security-setting" => {"#" => undefined}
    }
Actions
7. Re: Both severs from an ex live-backup configuration continuously log warnings after the live-backup config is undone

ataylor Jun 19, 2013 10:43 AM (in response to crytek)

"backup" => false this should be true for the backup
Actions
8. Re: Both severs from an ex live-backup configuration continuously log warnings after the live-backup config is undone

crytek Jun 19, 2013 11:05 AM (in response to ataylor)

Yes, but that's the idea. I don't want to keep the backup anymore. I want to remove the live-backup configuration and to use the two servers as normal clustered servers as before.
Actions
9. Re: Both severs from an ex live-backup configuration continuously log warnings after the live-backup config is undone

ataylor Jun 19, 2013 11:25 AM (in response to crytek)

once a backup server is a backup its always a backup (until it fails over), thats how it is configured. since it is replicating the other live server it has the same journal, which is used when you delete the backup and re create it as live. you will need to restart the whole server completely after deleting the data directory.

If you want the journal deleted when you delete the server then you will have to raise this with the AS7 guys
Actions
10. Re: Both severs from an ex live-backup configuration continuously log warnings after the live-backup config is undone

crytek Jun 19, 2013 11:31 AM (in response to ataylor)

I understand. Thanks for the help. So this is the expected behaviour afterall.
Actions
11. Re: Both severs from an ex live-backup configuration continuously log warnings after the live-backup config is undone

ataylor Jun 19, 2013 11:40 AM (in response to crytek)

yes, this is the expected behaviour, but it doesnt mean the AS7 guys cant add some code to delete the journal, raise a jira if you wish
Actions

Go to original post