We help IT Professionals succeed at work.

JBoss Clustering on Linux IP_MULTICAST_IF  error

* JBoss 4.0.5
* Fedora Core 6  (Linux)
* Java 1.6.1_01  (also tried 1.5.1_01) JDK
* Starting JBoss as a Linux service with a jboss link under  'etc/rc.d/init.d'
   starting parameters  ::  run.sh -c default -b 0.0.0.0 -g DefaultPartition
-    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    

Hello everyone:
- I am trying to cluster JBoss across two Linux boxes.
- I Have installed the JEMS installer of JBoss 4.0.5 with EJB3 clustered option selected.  (cluster-service.xml is present under "server/default/deploy"  directory.
- I will only try to start configuring the 2nd server 'after' I get this server up and running:

I keep getting the following error:
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
08:49:04,042 ERROR [JChannel] exception: java.lang.Exception: exception caused by UDP.start(): java.net.SocketException: bad argument for IP_MULTICAST_IF: address not bound to any interface
08:49:04,045 WARN  [ServiceController] Problem starting service jboss.cache:service=TomcatClusteringCache
ChannelException: java.lang.Exception: exception caused by UDP.start(): java.net.SocketException: bad argument for IP_MULTICAST_IF: address not bound to any interface
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

> I've read articles about trying to disable IP_MULTICAST_IF to prevent application chatter over network
> Someone in my shop suggested 'IP Multicast' is a network-policy level blocking

I need some help to get past this error.

I Will include my full   cluster-config.xml in a following comment (to keep this posting clean.)

Thanks
Comment
Watch Question

Author

Commented:
CLUSTER-SERVICE.XML
-    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    

<?xml version="1.0" encoding="UTF-8"?>

<!-- ===================================================================== -->
<!--                                                                       -->
<!--  Sample Clustering Service Configuration                              -->
<!--                                                                       -->
<!-- ===================================================================== -->

<server>

   <!-- ==================================================================== -->
   <!-- Cluster Partition: defines cluster                                   -->
   <!-- ==================================================================== -->


   <mbean code="org.jboss.ha.framework.server.ClusterPartition"
      name="jboss:service=${jboss.partition.name:DefaultPartition}">
         
      <!-- Name of the partition being built -->
      <attribute name="PartitionName">${jboss.partition.name:DefaultPartition}</attribute>

      <!-- The address used to determine the node name -->
      <attribute name="NodeAddress">${jboss.bind.address}</attribute>

      <!-- Determine if deadlock detection is enabled -->
      <attribute name="DeadlockDetection">False</attribute>
     
      <!-- Max time (in ms) to wait for state transfer to complete. Increase for large states -->
      <attribute name="StateTransferTimeout">30000</attribute>

      <!-- The JGroups protocol configuration -->
      <attribute name="PartitionConfig">
         <!--
         The default UDP stack:
         - If you have a multihomed machine, set the UDP protocol's bind_addr attribute to the
         appropriate NIC IP address, e.g bind_addr="192.168.0.2".
         - On Windows machines, because of the media sense feature being broken with multicast
         (even after disabling media sense) set the UDP protocol's loopback attribute to true
         -->
         <Config>
            <!--
            <UDP mcast_addr="${jboss.partition.udpGroup:228.1.2.3}" mcast_port="45566"
               ip_ttl="${jgroups.mcast.ip_ttl:8}" ip_mcast="true"
            -->
            <UDP mcast_addr="${jboss.partition.udpGroup:10.46.5.32}" mcast_port="45566"
               ip_ttl="${jgroups.mcast.ip_ttl:8}" ip_mcast="true"
               mcast_recv_buf_size="2000000" mcast_send_buf_size="640000"
               ucast_recv_buf_size="2000000" ucast_send_buf_size="640000"
               loopback="false"/>
            <PING timeout="2000" num_initial_members="3"
               up_thread="true" down_thread="true"/>
            <MERGE2 min_interval="10000" max_interval="20000"/>
            <FD_SOCK down_thread="false" up_thread="false"/>
            <FD shun="true" up_thread="true" down_thread="true"
               timeout="10000" max_tries="5"/>
            <VERIFY_SUSPECT timeout="3000" num_msgs="3"
               up_thread="true" down_thread="true"/>
            <pbcast.NAKACK gc_lag="50" retransmit_timeout="300,600,1200,2400,4800"
               max_xmit_size="8192"
               up_thread="true" down_thread="true"/>
            <UNICAST timeout="300,600,1200,2400,4800" window_size="100" min_threshold="10"
               down_thread="true"/>
            <pbcast.STABLE desired_avg_gossip="20000" max_bytes="400000"
               up_thread="true" down_thread="true"/>
            <FRAG frag_size="8192"
               down_thread="true" up_thread="true"/>
            <pbcast.GMS join_timeout="5000" join_retry_timeout="2000"
               shun="true" print_local_addr="true"/>
            <pbcast.STATE_TRANSFER up_thread="true" down_thread="true"/>
         </Config>

         <!-- Alternate TCP stack: customize it for your environment, change bind_addr and initial_hosts -->
         <!--
         <Config>
            <TCP bind_addr="thishost" start_port="7800" loopback="true"
                 recv_buf_size="2000000" send_buf_size="640000"
                 tcp_nodelay="true" up_thread="false" down_thread="false"/>
            <TCPPING initial_hosts="thishost[7800],otherhost[7800]" port_range="3" timeout="3500"
               num_initial_members="3" up_thread="false" down_thread="false"/>
            <MERGE2 min_interval="5000" max_interval="10000"
               up_thread="false" down_thread="false"/>
            <FD_SOCK down_thread="false" up_thread="false"/>
            <FD shun="true" up_thread="false" down_thread="false"
               timeout="10000" max_tries="5"/>
            <VERIFY_SUSPECT timeout="1500" down_thread="false" up_thread="false" />
            <pbcast.NAKACK up_thread="false" down_thread="false" gc_lag="100"
               retransmit_timeout="300,600,1200,2400,4800"/>
            <pbcast.STABLE desired_avg_gossip="20000" max_bytes="400000"
               down_thread="false" up_thread="false" />
            <pbcast.GMS join_timeout="5000" join_retry_timeout="2000" shun="true"
               print_local_addr="true"  up_thread="false" down_thread="false"/>
            <FC max_credits="2000000" down_thread="false" up_thread="false"
                 min_threshold="0.10"/>
            <FRAG2 frag_size="60000" down_thread="false" up_thread="true"/>
            <pbcast.STATE_TRANSFER up_thread="false" down_thread="false"/>
         </Config>
         -->
      </attribute>
      <depends>jboss:service=Naming</depends>
   </mbean>

   <!-- ==================================================================== -->
   <!-- HA Session State Service for SFSB                                    -->
   <!-- ==================================================================== -->

   <mbean code="org.jboss.ha.hasessionstate.server.HASessionStateService"
      name="jboss:service=HASessionState">
      <depends>jboss:service=Naming</depends>
      <!-- We now inject the partition into the HAJNDI service instead
           of requiring that the partition name be passed -->
      <depends optional-attribute-name="ClusterPartition"
         proxy-type="attribute">jboss:service=${jboss.partition.name:DefaultPartition}</depends>
      <!-- JNDI name under which the service is bound -->
      <attribute name="JndiName">/HASessionState/Default</attribute>
      <!-- Max delay before cleaning unreclaimed state.
           Defaults to 30*60*1000 => 30 minutes -->
      <attribute name="BeanCleaningDelay">0</attribute>
   </mbean>

   <!-- ==================================================================== -->
   <!-- HA JNDI                                                              -->
   <!-- ==================================================================== -->

   <mbean code="org.jboss.ha.jndi.HANamingService"
      name="jboss:service=HAJNDI">
      <!-- We now inject the partition into the HAJNDI service instead
           of requiring that the partition name be passed -->
      <depends optional-attribute-name="ClusterPartition"
         proxy-type="attribute">jboss:service=${jboss.partition.name:DefaultPartition}</depends>
      <!-- Bind address of bootstrap and HA-JNDI RMI endpoints -->
      <attribute name="BindAddress">${jboss.bind.address}</attribute>
      <!-- Port on which the HA-JNDI stub is made available -->
      <attribute name="Port">1100</attribute>
      <!-- RmiPort to be used by the HA-JNDI service once bound. 0 => auto. -->
      <attribute name="RmiPort">1101</attribute>
      <!-- Accept backlog of the bootstrap socket -->
      <attribute name="Backlog">50</attribute>
      <!-- The thread pool service used to control the bootstrap and
      auto discovery lookups -->
      <depends optional-attribute-name="LookupPool"
         proxy-type="attribute">jboss.system:service=ThreadPool</depends>

      <!-- A flag to disable the auto discovery via multicast -->
      <attribute name="DiscoveryDisabled">false</attribute>
      <!-- Set the auto-discovery bootstrap multicast bind address. If not
      specified and a BindAddress is specified, the BindAddress will be used. -->
      <attribute name="AutoDiscoveryBindAddress">${jboss.bind.address}</attribute>
      <!-- Multicast Address and group port used for auto-discovery -->
   <!--attribute name="AutoDiscoveryAddress">${jboss.partition.udpGroup:10.46.5.31}</attribute -->
   <!--attribute name="AutoDiscoveryAddress">${jboss.partition.udpGroup:230.0.0.4}</attribute -->
      <attribute name="AutoDiscoveryAddress">${jboss.partition.udpGroup:230.0.0.4}</attribute>
      <attribute name="AutoDiscoveryGroup">1102</attribute>
      <!-- The TTL (time-to-live) for autodiscovery IP multicast packets -->
      <attribute name="AutoDiscoveryTTL">16</attribute>
      <!-- The load balancing policy for HA-JNDI -->
      <attribute name="LoadBalancePolicy">org.jboss.ha.framework.interfaces.RoundRobin</attribute>
     
      <!-- Client socket factory to be used for client-server
           RMI invocations during JNDI queries
      <attribute name="ClientSocketFactory">custom</attribute>
      -->
      <!-- Server socket factory to be used for client-server
           RMI invocations during JNDI queries
      <attribute name="ServerSocketFactory">custom</attribute>
      -->
   </mbean>

   <mbean code="org.jboss.invocation.jrmp.server.JRMPInvokerHA"
      name="jboss:service=invoker,type=jrmpha">
      <attribute name="ServerAddress">${jboss.bind.address}</attribute>
      <attribute name="RMIObjectPort">4447</attribute>
      <!--
      <attribute name="RMIClientSocketFactory">custom</attribute>
      <attribute name="RMIServerSocketFactory">custom</attribute>
      -->
      <depends>jboss:service=Naming</depends>
   </mbean>

   <!-- the JRMPInvokerHA creates a thread per request.  This implementation uses a pool of threads -->
   <mbean code="org.jboss.invocation.pooled.server.PooledInvokerHA"
      name="jboss:service=invoker,type=pooledha">
      <attribute name="NumAcceptThreads">1</attribute>
      <attribute name="MaxPoolSize">300</attribute>
      <attribute name="ClientMaxPoolSize">300</attribute>
      <attribute name="SocketTimeout">60000</attribute>
      <attribute name="ServerBindAddress">${jboss.bind.address}</attribute>
      <attribute name="ServerBindPort">4446</attribute>
      <attribute name="ClientConnectAddress">${jboss.bind.address}</attribute>
      <attribute name="ClientConnectPort">0</attribute>
      <attribute name="EnableTcpNoDelay">false</attribute>
      <depends optional-attribute-name="TransactionManagerService">jboss:service=TransactionManager</depends>
      <depends>jboss:service=Naming</depends>
   </mbean>

   <!-- ==================================================================== -->

   <!-- ==================================================================== -->
   <!-- Distributed cache invalidation                                       -->
   <!-- ==================================================================== -->

   <mbean code="org.jboss.cache.invalidation.bridges.JGCacheInvalidationBridge"
      name="jboss.cache:service=InvalidationBridge,type=JavaGroups">
      <!-- We now inject the partition into the HAJNDI service instead
           of requiring that the partition name be passed -->
      <depends optional-attribute-name="ClusterPartition"
         proxy-type="attribute">jboss:service=${jboss.partition.name:DefaultPartition}</depends>
      <depends>jboss.cache:service=InvalidationManager</depends>
      <attribute name="InvalidationManager">jboss.cache:service=InvalidationManager</attribute>
      <attribute name="BridgeName">DefaultJGBridge</attribute>
   </mbean>


   <mbean code="org.jboss.ejb3.cache.tree.PassivationTreeCache"
       name="jboss.cache:service=EJB3SFSBClusteredCache">
     
        <attribute name="IsolationLevel">READ_UNCOMMITTED</attribute>
        <attribute name="CacheMode">REPL_SYNC</attribute>
        <attribute name="ClusterName">SFSB-Cache</attribute>
        <attribute name="ClusterConfig">
            ... ...
        </attribute>

        <!--  Number of milliseconds to wait until all responses for a
              synchronous call have been received.
        -->
        <attribute name="SyncReplTimeout">10000</attribute>

        <!--  Max number of milliseconds to wait for a lock acquisition -->
        <attribute name="LockAcquisitionTimeout">15000</attribute>

        <!--  Name of the eviction policy class. -->
        <attribute name="EvictionPolicyClass">
            org.jboss.ejb3.cache.tree.StatefulEvictionPolicy
        </attribute>

        <!--  Specific eviction policy configurations. This is LRU -->
        <attribute name="EvictionPolicyConfig">
            <config>
                <attribute name="wakeUpIntervalSeconds">1</attribute>
                <name>statefulClustered</name>
                <region name="/_default_">
                    <attribute name="maxNodes">1000000</attribute>
                    <attribute name="timeToIdleSeconds">300</attribute>
                </region>
            </config>
        </attribute>

        <attribute name="CacheLoaderFetchPersistentState">false</attribute>
        <attribute name="CacheLoaderFetchTransientState">true</attribute>
        <attribute name="FetchStateOnStartup">true</attribute>
        <attribute name="CacheLoaderClass">
            org.jboss.ejb3.cache.tree.StatefulCacheLoader
        </attribute>
        <attribute name="CacheLoaderConfig">
            location=statefulClustered
        </attribute>
   </mbean>

</server>
There is an issue in the Linux kernel that causes ipv6 to not work correctly with JDK 1.5 and lower. As a result, jbossas does not start by default on Linux unless ipv4 is forced. To that end, run.sh needs updating.

Something like:

   if [ "`uname`" = "Linux" ]; then
       JAVA_OPTS="$JAVA_OPTS -Djava.net.preferIPv4Stack=true"
   fi
fi

So that Linux systems use ipv4
See http://jira.jboss.com/jira/browse/JBAS-4332 for more details

Author

Commented:
Thanks for the reply, ut I did disable IPV6 and it had no effect.
I posted this same question on JBOSS.org and got the reply to not bind to IP 0.0.0.0 but to a specfic IP.

So I am not running the current Startup command in my init.d script:
default -b 0.0.0.0 -Dbind.address=10.46.5.32 -Djava.net.preferIPv4Stack=true

Another issue has popped up with UDP.start() . . .
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
10:04:11,981 ERROR [JChannel] exception: java.lang.Exception: exception caused by UDP.start(): java.lang.Exception: UDP.createSockets(): cannot list on any port in range 0-1
10:04:11,984 WARN  [ServiceController] Problem starting service jboss.cache:service=TomcatClusteringCache
ChannelException: java.lang.Exception: exception caused by UDP.start(): java.lang.Exception: UDP.createSockets(): cannot list on any port in range 0-1
      at org.jgroups.JChannel.connect(JChannel.java:328)
      at org.jboss.cache.TreeCache.startService(TreeCache.java:1424)
      at org.jboss.cache.aop.PojoCache.startService(PojoCache.java:94)
      at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:289)
      at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:245)
      at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)

Thanks
actually you should set -b 10.46.5.32 and remove -Dbind.address property.
Because JBOSS uses -b parametger to bind to specific address

Author

Commented:
When I change the run options to -b 10.46.blah.blah  I get the following error:
--> java.lang.Exception: Port 8083 already in use.

My command line statement:
--> /opt/jboss-4.0.5.GA/bin/run.sh -c default -b 10.46.5.32 -Djava.net.preferIPv4Stack=true

- - - - - - - - - -

09:23:56,430 INFO  [WebService] Using RMI server codebase: http://10.46.5.32:8083/
09:23:56,455 INFO  [Log4jService$URLWatchTimerTask] Configuring from URL: resource:log4j.xml
09:23:56,660 WARN  [ServiceController] Problem starting service jboss:service=WebService
java.lang.Exception: Port 8083 already in use.
      at org.jboss.web.WebServer.start(WebServer.java:235)
      at org.jboss.web.WebService.startService(WebService.java:337)
      at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:289)
      at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:245)


But when I start  with parameters  >>>   -b 0.0.0.0 -Dbind.address=10.46.5.32  >>> the port in use error goes away, but I'm back to the original error:

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
09:31:15,854 ERROR [JChannel] exception: java.lang.Exception: exception caused by UDP.start(): java.lang.Exception: UDP.createSockets(): cannot list on any port in range 0-1
09:31:15,857 WARN  [ServiceController] Problem starting service jboss.cache:service=TomcatClusteringCache
ChannelException: java.lang.Exception: exception caused by UDP.start(): java.lang.Exception: UDP.createSockets(): cannot list on any port in range 0-1
      at org.jgroups.JChannel.connect(JChannel.java:328)
      at org.jboss.cache.TreeCache.startService(TreeCache.java:1424)
      at org.jboss.cache.aop.PojoCache.startService(PojoCache.java:94)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Thoughts?
actually -D... parameters should be added to run.conf - they are not set correctly then you provide them on command line.
So in run.conf file there is a line:
JAVA_OPTS="-Xms128m -Xmx512m -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000"
Just add before closing quote -Djava.net.preferIPv4Stack=true
So you line will be:
JAVA_OPTS="-Xms128m -Xmx512m -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djava.net.preferIPv4Stack=true"