WebSphere Application Server - DNS, you can't fool it - or can you ?

I saw this: -

[3/12/17 19:55:21:158 UTC] 00000001 LogAdapter    E   DCSV9403E: Received an illegal configuration argument. Parameter MulticastInterface, value: 9.20.65.171. Exception is java.lang.Exception: Network I
nterface 9.20.65.171 was not found in local machine network interface list. Make sure that the NetworkInterface property is properly configured!
at com.ibm.rmm.mtl.transmitter.Config.<init>(Config.java:238)

while attempting to start a WebSphere Application Serve (WAS) Deployment Manager ( as part of an IBM BPM Advanced build ).

I wanted to drill into the issue, and see what hostname the offending IP address ( 9.20.65.171 ) was aliased to.

I tried/failed to use dig as it wasn't installed on my RHEL 7.3 VM.

So I installed bind-utils: -

yum install -y bind-utils

and then used dig as follows: -

dig 9.20.65.171

; <<>> DiG 9.9.4-RedHat-9.9.4-38.el7_3.2 <<>> 9.20.65.171
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 39457
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;9.20.65.171. IN A

;; AUTHORITY SECTION:
. 10800 IN SOA a.root-servers.net. nstld.verisign-grs.com. 2017031201 1800 900 604800 86400

;; Query time: 20 msec
;; SERVER: 9.20.136.11#53(9.20.136.11)
;; WHEN: Sun Mar 12 21:07:38 UTC 2017
;; MSG SIZE  rcvd: 115

and host : -

host 9.20.65.171

171.65.20.9.in-addr.arpa domain name pointer hyc1-vm2417.hursley.ibm.com.

I also checked /etc/hosts which contained: -

# Your system has configured 'manage_etc_hosts' as True.

# As a result, if you wish for changes to this file to persist
# then you will need to either
# a.) make changes to the master file in /etc/cloud/templates/hosts.redhat.tmpl
# b.) change or remove the value of 'manage_etc_hosts' in
#     /etc/cloud/cloud.cfg or cloud-config from user-data
# 
# The following lines are desirable for IPv4 capable hosts
127.0.0.1 bpm857.novalocal bpm857 bpm857.uk.ibm.com
127.0.0.1 localhost.localdomain localhost
127.0.0.1 localhost4.localdomain4 localhost4

# The following lines are desirable for IPv6 capable hosts
::1 bpm857.novalocal bpm857
::1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6

9.20.65.75 oracle.uk.ibm.com oracle
9.20.65.140 windows2012.uk.ibm.com windows2012
9.20.65.171 bpm857.uk.ibm.com bpm857

I revisited an old post of mine: -

More on Java and Canonical Hostnames

and resurrected an old Java class: -

hostStuff.java

import java.net.InetAddress;
import java.net.UnknownHostException;

public class hostStuff
{
public static void main(String[] args)
{

try
{
InetAddress address = InetAddress.getLocalHost();
System.out.println("My IP address ( via InetAddress.getLocalHost() ) is " + address.toString());
System.out.println("My hostname ( via InetAddress.getHostName() ) is " + address.getHostName());
System.out.println("My hostname ( via InetAddress.getCanonicalHostname() ) is  " + address.getCanonicalHostName());
}
catch (UnknownHostException e)
{
       System.out.println("I'm sorry. I don't know my own name.");
}
}
}

which, when run, returned: -

My IP address ( via InetAddress.getLocalHost() ) is bpm857.uk.ibm.com/127.0.0.1
My hostname ( via InetAddress.getHostName() ) is bpm857.uk.ibm.com
My hostname ( via InetAddress.getCanonicalHostname() ) is  bpm857.novalocal

I suspect that WAS is executing InetAddress.getCanonicalHostname() under the covers, which is the real hostname of the box - bpm857.novalocal - as dictated by OpenStack, whereas I'm trying to spoof WAS to use a different name.

So I "hacked" /etc/hosts : -

# Your system has configured 'manage_etc_hosts' as True.
# As a result, if you wish for changes to this file to persist
# then you will need to either
# a.) make changes to the master file in /etc/cloud/templates/hosts.redhat.tmpl
# b.) change or remove the value of 'manage_etc_hosts' in
#     /etc/cloud/cloud.cfg or cloud-config from user-data
# 
# The following lines are desirable for IPv4 capable hosts
127.0.0.1 bpm857.uk.ibm.com bpm857
127.0.0.1 localhost.localdomain localhost
127.0.0.1 localhost4.localdomain4 localhost4

# The following lines are desirable for IPv6 capable hosts
::1 bpm857.novalocal bpm857
::1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6

9.20.65.75 oracle.uk.ibm.com oracle
9.20.65.140 windows2012.uk.ibm.com windows2012
9.20.65.171 bpm857.uk.ibm.com bpm857

effectively removing reference to bpm857.novalocal which my class confirmed: -

My IP address ( via InetAddress.getLocalHost() ) is bpm857.uk.ibm.com/127.0.0.1
My hostname ( via InetAddress.getHostName() ) is bpm857.uk.ibm.com
My hostname ( via InetAddress.getCanonicalHostname() ) is  bpm857.uk.ibm.com

I tried to start the Deployment Manager, but that again failed with the same exception as before.

I further "hacked" /etc/hosts : -

# Your system has configured 'manage_etc_hosts' as True.
# As a result, if you wish for changes to this file to persist
# then you will need to either
# a.) make changes to the master file in /etc/cloud/templates/hosts.redhat.tmpl
# b.) change or remove the value of 'manage_etc_hosts' in
#     /etc/cloud/cloud.cfg or cloud-config from user-data
# 
# The following lines are desirable for IPv4 capable hosts
127.0.0.1 localhost.localdomain localhost
127.0.0.1 localhost4.localdomain4 localhost4

# The following lines are desirable for IPv6 capable hosts
::1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6

9.20.65.75 oracle.uk.ibm.com oracle
9.20.65.140 windows2012.uk.ibm.com windows2012
9.20.65.171 bpm857.uk.ibm.com bpm857

but to no avail.

In the end, I bowed to the inevitable and stuck with the host / domain that OpenStack was allocating, especially given that my VM is going to be connecting to other VMs on the same OpenStack box, as well as to the outside world.

Therefore, I reverted my WAS configuration to use the bpm857.novalocal host/domain name internally, and continued to reference it externally as bpm857.uk.ibm.com.

Thankfully, when I rebooted the VM, the /etc/hosts file was reverted back to its original state, 

Now, when I run my hostStuff class, I get: -

My IP address ( via InetAddress.getLocalHost() ) is bpm857.novalocal/127.0.0.1
My hostname ( via InetAddress.getHostName() ) is bpm857.novalocal
My hostname ( via InetAddress.getCanonicalHostname() ) is  bpm857.novalocal

and I've reverted my response file accordingly: -

cat /tmp/ResponseFiles/Advanced-PS-ThreeClusters-Oracle.properties |grep -i hostname

bpm.de.psProcessCenterHostname=
bpm.dmgr.hostname=bpm857.novalocal
bpm.de.node.1.hostname=bpm857.novalocal
bpm.de.db.1.hostname=oracle.novalocal
bpm.de.db.2.hostname=oracle.novalocal
bpm.de.db.3.hostname=oracle.novalocal
bpm.de.db.4.hostname=oracle.novalocal

So we're back in the game ….

The moral of the story ? Ensure that you use host/domain names that are valid for your environment, networking with your network folks, as required.

Also, never use localhost for WAS hostnames :-)