JBoss Clustering on Linux IP_MULTICAST_IF error

* JBoss 4.0.5
* Fedora Core 6  (Linux)
* Java 1.6.1_01  (also tried 1.5.1_01) JDK
* Starting JBoss as a Linux service with a jboss link under  'etc/rc.d/init.d'
   starting parameters  ::  run.sh -c default -b 0.0.0.0 -g DefaultPartition
-    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    

Hello everyone:
- I am trying to cluster JBoss across two Linux boxes.
- I Have installed the JEMS installer of JBoss 4.0.5 with EJB3 clustered option selected.  (cluster-service.xml is present under "server/default/deploy"  directory.
- I will only try to start configuring the 2nd server 'after' I get this server up and running:

I keep getting the following error:
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
08:49:04,042 ERROR [JChannel] exception: java.lang.Exception: exception caused by UDP.start(): java.net.SocketException: bad argument for IP_MULTICAST_IF: address not bound to any interface
08:49:04,045 WARN  [ServiceController] Problem starting service jboss.cache:service=TomcatClusteringCache
ChannelException: java.lang.Exception: exception caused by UDP.start(): java.net.SocketException: bad argument for IP_MULTICAST_IF: address not bound to any interface
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

> I've read articles about trying to disable IP_MULTICAST_IF to prevent application chatter over network
> Someone in my shop suggested 'IP Multicast' is a network-policy level blocking

I need some help to get past this error.

I Will include my full   cluster-config.xml in a following comment (to keep this posting clean.)

Thanks
fshtankAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

fshtankAuthor Commented:
CLUSTER-SERVICE.XML
-    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    -    

<?xml version="1.0" encoding="UTF-8"?>

<!-- ===================================================================== -->
<!--                                                                       -->
<!--  Sample Clustering Service Configuration                              -->
<!--                                                                       -->
<!-- ===================================================================== -->

<server>

   <!-- ==================================================================== -->
   <!-- Cluster Partition: defines cluster                                   -->
   <!-- ==================================================================== -->


   <mbean code="org.jboss.ha.framework.server.ClusterPartition"
      name="jboss:service=${jboss.partition.name:DefaultPartition}">
         
      <!-- Name of the partition being built -->
      <attribute name="PartitionName">${jboss.partition.name:DefaultPartition}</attribute>

      <!-- The address used to determine the node name -->
      <attribute name="NodeAddress">${jboss.bind.address}</attribute>

      <!-- Determine if deadlock detection is enabled -->
      <attribute name="DeadlockDetection">False</attribute>
     
      <!-- Max time (in ms) to wait for state transfer to complete. Increase for large states -->
      <attribute name="StateTransferTimeout">30000</attribute>

      <!-- The JGroups protocol configuration -->
      <attribute name="PartitionConfig">
         <!--
         The default UDP stack:
         - If you have a multihomed machine, set the UDP protocol's bind_addr attribute to the
         appropriate NIC IP address, e.g bind_addr="192.168.0.2".
         - On Windows machines, because of the media sense feature being broken with multicast
         (even after disabling media sense) set the UDP protocol's loopback attribute to true
         -->
         <Config>
            <!--
            <UDP mcast_addr="${jboss.partition.udpGroup:228.1.2.3}" mcast_port="45566"
               ip_ttl="${jgroups.mcast.ip_ttl:8}" ip_mcast="true"
            -->
            <UDP mcast_addr="${jboss.partition.udpGroup:10.46.5.32}" mcast_port="45566"
               ip_ttl="${jgroups.mcast.ip_ttl:8}" ip_mcast="true"
               mcast_recv_buf_size="2000000" mcast_send_buf_size="640000"
               ucast_recv_buf_size="2000000" ucast_send_buf_size="640000"
               loopback="false"/>
            <PING timeout="2000" num_initial_members="3"
               up_thread="true" down_thread="true"/>
            <MERGE2 min_interval="10000" max_interval="20000"/>
            <FD_SOCK down_thread="false" up_thread="false"/>
            <FD shun="true" up_thread="true" down_thread="true"
               timeout="10000" max_tries="5"/>
            <VERIFY_SUSPECT timeout="3000" num_msgs="3"
               up_thread="true" down_thread="true"/>
            <pbcast.NAKACK gc_lag="50" retransmit_timeout="300,600,1200,2400,4800"
               max_xmit_size="8192"
               up_thread="true" down_thread="true"/>
            <UNICAST timeout="300,600,1200,2400,4800" window_size="100" min_threshold="10"
               down_thread="true"/>
            <pbcast.STABLE desired_avg_gossip="20000" max_bytes="400000"
               up_thread="true" down_thread="true"/>
            <FRAG frag_size="8192"
               down_thread="true" up_thread="true"/>
            <pbcast.GMS join_timeout="5000" join_retry_timeout="2000"
               shun="true" print_local_addr="true"/>
            <pbcast.STATE_TRANSFER up_thread="true" down_thread="true"/>
         </Config>

         <!-- Alternate TCP stack: customize it for your environment, change bind_addr and initial_hosts -->
         <!--
         <Config>
            <TCP bind_addr="thishost" start_port="7800" loopback="true"
                 recv_buf_size="2000000" send_buf_size="640000"
                 tcp_nodelay="true" up_thread="false" down_thread="false"/>
            <TCPPING initial_hosts="thishost[7800],otherhost[7800]" port_range="3" timeout="3500"
               num_initial_members="3" up_thread="false" down_thread="false"/>
            <MERGE2 min_interval="5000" max_interval="10000"
               up_thread="false" down_thread="false"/>
            <FD_SOCK down_thread="false" up_thread="false"/>
            <FD shun="true" up_thread="false" down_thread="false"
               timeout="10000" max_tries="5"/>
            <VERIFY_SUSPECT timeout="1500" down_thread="false" up_thread="false" />
            <pbcast.NAKACK up_thread="false" down_thread="false" gc_lag="100"
               retransmit_timeout="300,600,1200,2400,4800"/>
            <pbcast.STABLE desired_avg_gossip="20000" max_bytes="400000"
               down_thread="false" up_thread="false" />
            <pbcast.GMS join_timeout="5000" join_retry_timeout="2000" shun="true"
               print_local_addr="true"  up_thread="false" down_thread="false"/>
            <FC max_credits="2000000" down_thread="false" up_thread="false"
                 min_threshold="0.10"/>
            <FRAG2 frag_size="60000" down_thread="false" up_thread="true"/>
            <pbcast.STATE_TRANSFER up_thread="false" down_thread="false"/>
         </Config>
         -->
      </attribute>
      <depends>jboss:service=Naming</depends>
   </mbean>

   <!-- ==================================================================== -->
   <!-- HA Session State Service for SFSB                                    -->
   <!-- ==================================================================== -->

   <mbean code="org.jboss.ha.hasessionstate.server.HASessionStateService"
      name="jboss:service=HASessionState">
      <depends>jboss:service=Naming</depends>
      <!-- We now inject the partition into the HAJNDI service instead
           of requiring that the partition name be passed -->
      <depends optional-attribute-name="ClusterPartition"
         proxy-type="attribute">jboss:service=${jboss.partition.name:DefaultPartition}</depends>
      <!-- JNDI name under which the service is bound -->
      <attribute name="JndiName">/HASessionState/Default</attribute>
      <!-- Max delay before cleaning unreclaimed state.
           Defaults to 30*60*1000 => 30 minutes -->
      <attribute name="BeanCleaningDelay">0</attribute>
   </mbean>

   <!-- ==================================================================== -->
   <!-- HA JNDI                                                              -->
   <!-- ==================================================================== -->

   <mbean code="org.jboss.ha.jndi.HANamingService"
      name="jboss:service=HAJNDI">
      <!-- We now inject the partition into the HAJNDI service instead
           of requiring that the partition name be passed -->
      <depends optional-attribute-name="ClusterPartition"
         proxy-type="attribute">jboss:service=${jboss.partition.name:DefaultPartition}</depends>
      <!-- Bind address of bootstrap and HA-JNDI RMI endpoints -->
      <attribute name="BindAddress">${jboss.bind.address}</attribute>
      <!-- Port on which the HA-JNDI stub is made available -->
      <attribute name="Port">1100</attribute>
      <!-- RmiPort to be used by the HA-JNDI service once bound. 0 => auto. -->
      <attribute name="RmiPort">1101</attribute>
      <!-- Accept backlog of the bootstrap socket -->
      <attribute name="Backlog">50</attribute>
      <!-- The thread pool service used to control the bootstrap and
      auto discovery lookups -->
      <depends optional-attribute-name="LookupPool"
         proxy-type="attribute">jboss.system:service=ThreadPool</depends>

      <!-- A flag to disable the auto discovery via multicast -->
      <attribute name="DiscoveryDisabled">false</attribute>
      <!-- Set the auto-discovery bootstrap multicast bind address. If not
      specified and a BindAddress is specified, the BindAddress will be used. -->
      <attribute name="AutoDiscoveryBindAddress">${jboss.bind.address}</attribute>
      <!-- Multicast Address and group port used for auto-discovery -->
   <!--attribute name="AutoDiscoveryAddress">${jboss.partition.udpGroup:10.46.5.31}</attribute -->
   <!--attribute name="AutoDiscoveryAddress">${jboss.partition.udpGroup:230.0.0.4}</attribute -->
      <attribute name="AutoDiscoveryAddress">${jboss.partition.udpGroup:230.0.0.4}</attribute>
      <attribute name="AutoDiscoveryGroup">1102</attribute>
      <!-- The TTL (time-to-live) for autodiscovery IP multicast packets -->
      <attribute name="AutoDiscoveryTTL">16</attribute>
      <!-- The load balancing policy for HA-JNDI -->
      <attribute name="LoadBalancePolicy">org.jboss.ha.framework.interfaces.RoundRobin</attribute>
     
      <!-- Client socket factory to be used for client-server
           RMI invocations during JNDI queries
      <attribute name="ClientSocketFactory">custom</attribute>
      -->
      <!-- Server socket factory to be used for client-server
           RMI invocations during JNDI queries
      <attribute name="ServerSocketFactory">custom</attribute>
      -->
   </mbean>

   <mbean code="org.jboss.invocation.jrmp.server.JRMPInvokerHA"
      name="jboss:service=invoker,type=jrmpha">
      <attribute name="ServerAddress">${jboss.bind.address}</attribute>
      <attribute name="RMIObjectPort">4447</attribute>
      <!--
      <attribute name="RMIClientSocketFactory">custom</attribute>
      <attribute name="RMIServerSocketFactory">custom</attribute>
      -->
      <depends>jboss:service=Naming</depends>
   </mbean>

   <!-- the JRMPInvokerHA creates a thread per request.  This implementation uses a pool of threads -->
   <mbean code="org.jboss.invocation.pooled.server.PooledInvokerHA"
      name="jboss:service=invoker,type=pooledha">
      <attribute name="NumAcceptThreads">1</attribute>
      <attribute name="MaxPoolSize">300</attribute>
      <attribute name="ClientMaxPoolSize">300</attribute>
      <attribute name="SocketTimeout">60000</attribute>
      <attribute name="ServerBindAddress">${jboss.bind.address}</attribute>
      <attribute name="ServerBindPort">4446</attribute>
      <attribute name="ClientConnectAddress">${jboss.bind.address}</attribute>
      <attribute name="ClientConnectPort">0</attribute>
      <attribute name="EnableTcpNoDelay">false</attribute>
      <depends optional-attribute-name="TransactionManagerService">jboss:service=TransactionManager</depends>
      <depends>jboss:service=Naming</depends>
   </mbean>

   <!-- ==================================================================== -->

   <!-- ==================================================================== -->
   <!-- Distributed cache invalidation                                       -->
   <!-- ==================================================================== -->

   <mbean code="org.jboss.cache.invalidation.bridges.JGCacheInvalidationBridge"
      name="jboss.cache:service=InvalidationBridge,type=JavaGroups">
      <!-- We now inject the partition into the HAJNDI service instead
           of requiring that the partition name be passed -->
      <depends optional-attribute-name="ClusterPartition"
         proxy-type="attribute">jboss:service=${jboss.partition.name:DefaultPartition}</depends>
      <depends>jboss.cache:service=InvalidationManager</depends>
      <attribute name="InvalidationManager">jboss.cache:service=InvalidationManager</attribute>
      <attribute name="BridgeName">DefaultJGBridge</attribute>
   </mbean>


   <mbean code="org.jboss.ejb3.cache.tree.PassivationTreeCache"
       name="jboss.cache:service=EJB3SFSBClusteredCache">
     
        <attribute name="IsolationLevel">READ_UNCOMMITTED</attribute>
        <attribute name="CacheMode">REPL_SYNC</attribute>
        <attribute name="ClusterName">SFSB-Cache</attribute>
        <attribute name="ClusterConfig">
            ... ...
        </attribute>

        <!--  Number of milliseconds to wait until all responses for a
              synchronous call have been received.
        -->
        <attribute name="SyncReplTimeout">10000</attribute>

        <!--  Max number of milliseconds to wait for a lock acquisition -->
        <attribute name="LockAcquisitionTimeout">15000</attribute>

        <!--  Name of the eviction policy class. -->
        <attribute name="EvictionPolicyClass">
            org.jboss.ejb3.cache.tree.StatefulEvictionPolicy
        </attribute>

        <!--  Specific eviction policy configurations. This is LRU -->
        <attribute name="EvictionPolicyConfig">
            <config>
                <attribute name="wakeUpIntervalSeconds">1</attribute>
                <name>statefulClustered</name>
                <region name="/_default_">
                    <attribute name="maxNodes">1000000</attribute>
                    <attribute name="timeToIdleSeconds">300</attribute>
                </region>
            </config>
        </attribute>

        <attribute name="CacheLoaderFetchPersistentState">false</attribute>
        <attribute name="CacheLoaderFetchTransientState">true</attribute>
        <attribute name="FetchStateOnStartup">true</attribute>
        <attribute name="CacheLoaderClass">
            org.jboss.ejb3.cache.tree.StatefulCacheLoader
        </attribute>
        <attribute name="CacheLoaderConfig">
            location=statefulClustered
        </attribute>
   </mbean>

</server>
ramazanyichCommented:
There is an issue in the Linux kernel that causes ipv6 to not work correctly with JDK 1.5 and lower. As a result, jbossas does not start by default on Linux unless ipv4 is forced. To that end, run.sh needs updating.

Something like:

   if [ "`uname`" = "Linux" ]; then
       JAVA_OPTS="$JAVA_OPTS -Djava.net.preferIPv4Stack=true"
   fi
fi

So that Linux systems use ipv4
See http://jira.jboss.com/jira/browse/JBAS-4332 for more details
fshtankAuthor Commented:
Thanks for the reply, ut I did disable IPV6 and it had no effect.
I posted this same question on JBOSS.org and got the reply to not bind to IP 0.0.0.0 but to a specfic IP.

So I am not running the current Startup command in my init.d script:
default -b 0.0.0.0 -Dbind.address=10.46.5.32 -Djava.net.preferIPv4Stack=true

Another issue has popped up with UDP.start() . . .
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
10:04:11,981 ERROR [JChannel] exception: java.lang.Exception: exception caused by UDP.start(): java.lang.Exception: UDP.createSockets(): cannot list on any port in range 0-1
10:04:11,984 WARN  [ServiceController] Problem starting service jboss.cache:service=TomcatClusteringCache
ChannelException: java.lang.Exception: exception caused by UDP.start(): java.lang.Exception: UDP.createSockets(): cannot list on any port in range 0-1
      at org.jgroups.JChannel.connect(JChannel.java:328)
      at org.jboss.cache.TreeCache.startService(TreeCache.java:1424)
      at org.jboss.cache.aop.PojoCache.startService(PojoCache.java:94)
      at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:289)
      at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:245)
      at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)

Thanks
Get Blueprints for Increased Customer Retention

The IT Service Excellence Tool Kit has best practices to keep your clients happy and business booming. Inside, you’ll find everything you need to increase client satisfaction and retention, become more competitive, and increase your overall success.

ramazanyichCommented:
actually you should set -b 10.46.5.32 and remove -Dbind.address property.
Because JBOSS uses -b parametger to bind to specific address
fshtankAuthor Commented:
When I change the run options to -b 10.46.blah.blah  I get the following error:
--> java.lang.Exception: Port 8083 already in use.

My command line statement:
--> /opt/jboss-4.0.5.GA/bin/run.sh -c default -b 10.46.5.32 -Djava.net.preferIPv4Stack=true

- - - - - - - - - -

09:23:56,430 INFO  [WebService] Using RMI server codebase: http://10.46.5.32:8083/
09:23:56,455 INFO  [Log4jService$URLWatchTimerTask] Configuring from URL: resource:log4j.xml
09:23:56,660 WARN  [ServiceController] Problem starting service jboss:service=WebService
java.lang.Exception: Port 8083 already in use.
      at org.jboss.web.WebServer.start(WebServer.java:235)
      at org.jboss.web.WebService.startService(WebService.java:337)
      at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:289)
      at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:245)


But when I start  with parameters  >>>   -b 0.0.0.0 -Dbind.address=10.46.5.32  >>> the port in use error goes away, but I'm back to the original error:

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
09:31:15,854 ERROR [JChannel] exception: java.lang.Exception: exception caused by UDP.start(): java.lang.Exception: UDP.createSockets(): cannot list on any port in range 0-1
09:31:15,857 WARN  [ServiceController] Problem starting service jboss.cache:service=TomcatClusteringCache
ChannelException: java.lang.Exception: exception caused by UDP.start(): java.lang.Exception: UDP.createSockets(): cannot list on any port in range 0-1
      at org.jgroups.JChannel.connect(JChannel.java:328)
      at org.jboss.cache.TreeCache.startService(TreeCache.java:1424)
      at org.jboss.cache.aop.PojoCache.startService(PojoCache.java:94)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Thoughts?
ramazanyichCommented:
actually -D... parameters should be added to run.conf - they are not set correctly then you provide them on command line.
So in run.conf file there is a line:
JAVA_OPTS="-Xms128m -Xmx512m -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000"
Just add before closing quote -Djava.net.preferIPv4Stack=true
So you line will be:
JAVA_OPTS="-Xms128m -Xmx512m -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djava.net.preferIPv4Stack=true"

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Linux

From novice to tech pro — start learning today.