Cannot connect to remote WildFly management console.

[george.labuschagne]

Hi all

I cannot connect to a remote running WildFly management console. I can however open the console if logged in locally or view the management web site.

Here is my startup parameters:

./standalone.sh -b=0.0.0.0 -bmanagement=0.0.0.0 --debug

I also tried with:

./standalone.sh -b=192.168.0.80 -bmanagement=192.168.0.80 --debug

The last lines of my log says:

13:17:37,393 INFO  [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0212: Resuming server
13:17:37,397 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0060: Http management interface listening on http://192.168.0.80:9990/management
13:17:37,398 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0051: Admin console listening on http://192.168.0.80:9990
13:17:37,398 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: WildFly Full 14.0.0.Final (WildFly Core 6.0.1.Final) started in 48453ms - Started 1784 of 1957 services (331 services are lazy, passive or on-demand)

The command I use to attempt to connect from a local WildFly 14 - same as server - bin directory on my local machine is:

./jboss-cli.sh --connect --controller=remote+http://192.168.0.80:9990

All required ports are opened on the server (else the web site would not work either).

I get a request to authenticate against:

./jboss-cli.sh jboss-cli.sh --connect --controller=remote+http://192.168.0.80:9990
Picked up _JAVA_OPTIONS: -Xms512m -Xmx2g
Authenticating against security realm: ManagementRealm
Username: myuser
Password: 
Failed to connect to the controller: The controller is not available at 192.168.0.80:9990: java.net.ConnectException: WFLYPRT0023: Could not connect to remote+http://192.168.0.80:9990. The connection timed out: WFLYPRT0023: Could not connect to remote+http://192.168.0.80:9990. The connection timed out

The error I get is:

Failed to connect to the controller: The controller is not available at 192.168.0.80:9990: java.net.ConnectException: WFLYPRT0023: Could not connect to remote+http://192.168.0.80:9990. The connection timed out: WFLYPRT0023: Could not connect to remote+http://192.168.0.80:9990. The connection timed out

I need to be able to connect to this remote management console for remote debugging via IntelliJ.

This is nmap reports, first one from the server running WildFly, second from devbox trying to connect to WildFly:

Host is up (0.0021s latency).
Not shown: 990 closed ports
PORT     STATE SERVICE
22/tcp   open  ssh
25/tcp   open  smtp
80/tcp   open  http
111/tcp  open  rpcbind
443/tcp  open  https
873/tcp  open  rsync
3306/tcp open  mysql
5432/tcp open  postgresql
8080/tcp open  http-proxy
8443/tcp open  https-alt


Nmap done: 1 IP address (1 host up) scanned in 0.13 seconds

Host is up (0.0040s latency).
Not shown: 994 filtered ports
PORT     STATE  SERVICE
22/tcp   open   ssh
80/tcp   open   http
873/tcp  open   rsync
8009/tcp closed ajp13
8080/tcp open   http-proxy
8443/tcp open   https-alt


Nmap done: 1 IP address (1 host up) scanned in 16.17 seconds

This also appears in the server log when trying to remotely connect:

WARN  [org.jboss.as.domain.management.security] (default I/O-6) WFLYDM0113: Generated self signed certificate at /home/edsadmin/devbuild/wildfly-14.0.0.Final/standalone/configuration/application.keystore. Please note that self signed certificates are not secure, and should only be used for testing purposes. Do not use this self signed certificate in production.
SHA-1 fingerprint of the generated key is 37:a5:dd:3a:1b:0e:1b:ba:48:89:23:be:e6:d6:21:14:61:85:18:90
SHA-256 fingerprint of the generated key is 47:a2:5b:e4:73:d0:54:62:90:8b:a2:0f:e1:03:f2:c7:02:c6:fe:04:6f:ab:fe:85:31:35:99:31:11:b4:3c:95

OS is CentOS in both cases.

Any ideas please?

[claudio4j]

Can you try the status with netcat ? NMAP requires you to inform the 9990 port.

$ nc -zv -w2 192.168.25.220 9990
Ncat: Version 7.60 ( https://nmap.org/ncat )
Ncat: Connected to 192.168.25.220:9990.
Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.


$ sudo nmap -sT 192.168.25.220 -p9990
Starting Nmap 7.60 ( https://nmap.org ) at 2018-09-06 12:05 -03
Nmap scan report for uranus (192.168.25.220)
Host is up (0.000046s latency).
PORT     STATE SERVICE
9990/tcp open  osm-appsrvr

[mchoma]

When connecting locally JBOSS_LOCAL authentication is used. When connecting remotely you have to have user myuser in ManagementRealm. You can add that by bin/add-user.sh for example. I don't see 9990 port open in your list.

[george.labuschagne]

Good day Claudio claudio4j

Thanks for the response.

The output from the two commands ran on the client are:

[georgel@devbox bin]$ nc -zv -w2 192.168.0.80 9990
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to 192.168.0.80:9990.
Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.

and

[georgel@devbox bin]$ sudo nmap -sT 192.168.25.220 -p9990
[sudo] password for georgel:


Starting Nmap 6.40 ( http://nmap.org ) at 2018-09-07 07:06 SAST
Note: Host seems down. If it is really up, but blocking our ping probes, try -Pn
Nmap done: 1 IP address (0 hosts up) scanned in 0.46 seconds
[georgel@devbox bin]$

If I use the -Pn parameter nmap is able to connect, is this causing the timeout perhaps? If so how to resolve?

[georgel@devbox bin]$ sudo nmap -sT -Pn 192.168.25.220 -p9990
[sudo] password for georgel: 


Starting Nmap 6.40 ( http://nmap.org ) at 2018-09-07 07:12 SAST
Nmap scan report for 192.168.25.220
Host is up.
PORT     STATE    SERVICE
9990/tcp filtered osm-appsrvr


Nmap done: 1 IP address (1 host up) scanned in 13.07 seconds
[georgel@devbox bin]$ 

So it seems netcat is able to connect to the port but nmap can't ping the port. I can ping the machine however:

[georgel@devbox bin]$ ping 192.168.0.80 -c 5
PING 192.168.0.80 (192.168.0.80) 56(84) bytes of data.
64 bytes from 192.168.0.80: icmp_seq=1 ttl=63 time=2.03 ms
64 bytes from 192.168.0.80: icmp_seq=2 ttl=64 time=1.68 ms
64 bytes from 192.168.0.80: icmp_seq=3 ttl=64 time=1.71 ms
64 bytes from 192.168.0.80: icmp_seq=4 ttl=64 time=4.27 ms
64 bytes from 192.168.0.80: icmp_seq=5 ttl=64 time=1.70 ms


--- 192.168.0.80 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4009ms
rtt min/avg/max/mdev = 1.686/2.282/4.274/1.006 ms
[georgel@devbox bin]$ 

netstat on the server shows that the server is listening:

[edsadmin@bugzilla ~]$ netstat -at
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 0.0.0.0:smtp            0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:pcsync-https    0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:osm-appsrvr     0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:rsync           0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:mysql           0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:sunrpc          0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:webcache        0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:ssh             0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:postgres        0.0.0.0:*               LISTEN     
tcp        0      0 bugzilla.headoffi:39116 webgateway.hea:webcache TIME_WAIT  
tcp        0      0 bugzilla.headoffice:ssh 192.168.41.111:49466    ESTABLISHED
tcp        0      0 localhost:postgres      localhost:46840         ESTABLISHED
tcp        0      0 bugzilla.headoffice:ssh 192.168.41.111:49472    ESTABLISHED
tcp        0      0 localhost:46840         localhost:postgres      ESTABLISHED
tcp6       0      0 [::]:https              [::]:*                  LISTEN     
tcp6       0      0 [::]:rsync              [::]:*                  LISTEN     
tcp6       0      0 [::]:sunrpc             [::]:*                  LISTEN     
tcp6       0      0 [::]:http               [::]:*                  LISTEN     
tcp6       0      0 [::]:ssh                [::]:*                  LISTEN     
tcp6       0      0 [::]:postgres           [::]:*                  LISTEN   

[george.labuschagne]

Good day Martin mchoma

I already have a user with a similar username on the server and client, both as ManagementRealm users.

I created on both now as well a user literately called 'myuser', added it to the ManagementRealm and to the 'public' group on both machine.

However the timeout after I enter the password still persist:

[georgel@devbox bin]$ ./add-user.sh
Picked up _JAVA_OPTIONS: -Xms512m -Xmx2g


What type of user do you wish to add?
a) Management User (mgmt-users.properties)
b) Application User (application-users.properties)
(a): a


Enter the details of the new user to add.
Using realm 'ManagementRealm' as discovered from the existing property files.
Username : myuser
Password recommendations are listed below. To modify these restrictions edit the add-user.properties configuration file.
- The password should be different from the username
- The password should not be one of the following restricted values {root, admin, administrator}
- The password should contain at least 8 characters, 1 alphabetic character(s), 1 digit(s), 1 non-alphanumeric symbol(s)
Password :
Re-enter Password :
What groups do you want this user to belong to? (Please enter a comma separated list, or leave blank for none)[  ]: public
About to add user 'myuser' for realm 'ManagementRealm'
Is this correct yes/no? yes
Added user 'myuser' to file '/home/georgel/opt/wildfly-14.0.0.Final/standalone/configuration/mgmt-users.properties'
Added user 'myuser' to file '/home/georgel/opt/wildfly-14.0.0.Final/domain/configuration/mgmt-users.properties'
Added user 'myuser' with groups public to file '/home/georgel/opt/wildfly-14.0.0.Final/standalone/configuration/mgmt-groups.properties'
Added user 'myuser' with groups public to file '/home/georgel/opt/wildfly-14.0.0.Final/domain/configuration/mgmt-groups.properties'
Is this new user going to be used for one AS process to connect to another AS process?
e.g. for a slave host controller connecting to the master or for a Remoting connection for server to server EJB calls.
yes/no? no

and then

[georgel@devbox bin]$ ./jboss-cli.sh --connect --controller=192.168.0.80:9990
Picked up _JAVA_OPTIONS: -Xms512m -Xmx2g
Authenticating against security realm: ManagementRealm
Username: myuser
Password:
Failed to connect to the controller: The controller is not available at 192.168.0.80:9990: java.net.ConnectException: WFLYPRT0023: Could not connect to remote+http://192.168.0.80:9990. The connection timed out: WFLYPRT0023: Could not connect to remote+http://192.168.0.80:9990. The connection timed out
[georgel@devbox bin]$

[george.labuschagne]

Some additional info

When no username is supplied the following appears:

[georgel@devbox bin]$ ./jboss-cli.sh --connect --controller=192.168.0.80:9990
Authenticating against security realm: ManagementRealm
Username: 
Failed to connect to the controller: Unable to authenticate against controller at 192.168.0.80:9990: Authentication failed: all available authentication mechanisms failed:
   DIGEST-MD5: javax.security.sasl.SaslException: ELY05053: Callback handler failed for unknown reason [Caused by javax.security.sasl.SaslException: No username supplied.]
[georgel@devbox bin]$ 

When an incorrect password is supplied:

georgel@devbox bin]$ ./jboss-cli.sh --connect --controller=192.168.0.80:9990
Authenticating against security realm: ManagementRealm
Username: edsadmin
Password: 
Failed to connect to the controller: Unable to authenticate against controller at 192.168.0.80:9990: Authentication failed: all available authentication mechanisms failed:
   DIGEST-MD5: javax.security.sasl.SaslException: DIGEST-MD5: Server rejected authentication
[georgel@devbox bin]$ 

[yadavprem]

hi is --controller=192.168.0.80:9990  is a domain controller ?

[george.labuschagne]

Good day Prem yadavprem

No it is not. It is another standalone installation in a testing environment that I want to connect to for debugging purposes via IntelliJ

[mayerw01]

This seems to be an issue with your firewall.

Did you check this also with Wireshark?

[george.labuschagne]

Good day Wolfgang mayerw01

Thanks for the reply.

I disabled the firewalld and SELINUX on both machines:

[root@devbox ~]# systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
  Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
  Active: inactive (dead)
    Docs: man:firewalld(1)
[root@devbox ~]#

and

[root@devbox ~]# sestatus 
SELinux status:                 disabled
[root@devbox ~]#

still get the timeout after authentication.

[mayerw01]

Did you check this also with a network analyzer like Wireshark?

[claudio4j]

Enable security logging on server as

<logger category="org.wildfly.security" use-parent-handlers="true">
    <level name="ALL"/>
</logger>

And do a tail -f on server.log

Also, related to nmap, the port state is filtered, which means nmap cannot determine whether the port is open because packet filtering prevents its probes from reaching the port. You should double check if there are firewall rules on 9990 port.

Port Scanning Basics | Nmap Network Scanning

[george.labuschagne]

Good day claudio4j

I completely disabled firewalld and SELINUX on both machines.

I still get the timeout after attempting to authenticate.

Below is the output from the log as requested. I definitely can reach the server based on this log file, but something then goes wrong with the authentication it seems.

[edsadmin@bugzilla log]$ tail -f on server.log
tail: cannot open ‘on’ for reading: No such file or directory
==> server.log <==
2018-09-12 08:02:53,629 INFO  [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0010: Deployed "postgresql-42.2.5.jar" (runtime-name : "postgresql-42.2.5.jar")
2018-09-12 08:02:53,635 INFO  [org.jboss.as.server] (ServerService Thread Pool -- 42) WFLYSRV0010: Deployed "edsnext.war" (runtime-name : "edsnext.war")
2018-09-12 08:02:53,819 INFO  [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0212: Resuming server
2018-09-12 08:02:53,822 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0060: Http management interface listening on http://0.0.0.0:9990/management
2018-09-12 08:02:53,823 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0051: Admin console listening on http://0.0.0.0:9990
2018-09-12 08:02:53,823 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: WildFly Full 14.0.0.Final (WildFly Core 6.0.1.Final) started in 29326ms - Started 1782 of 1956 services (332 services are lazy, passive or on-demand)
2018-09-12 08:04:34,509 TRACE [org.wildfly.security] (management I/O-2) Handling MechanismInformationCallback type='SASL' name='DIGEST-MD5' host-name='bugzilla.headoffice.megchem.co.za' protocol='remote'
2018-09-12 08:04:34,511 TRACE [org.wildfly.security] (management I/O-2) Handling AvailableRealmsCallback: realms = [ManagementRealm]
2018-09-12 08:04:34,520 TRACE [org.wildfly.security] (management I/O-2) Creating SaslServer [org.wildfly.security.sasl.digest.DigestSaslServer@574f31a4] for mechanism [DIGEST-MD5] and protocol [remote]
2018-09-12 08:04:34,523 TRACE [org.wildfly.security] (management I/O-2) Created SaslServer [org.wildfly.security.sasl.util.SecurityIdentitySaslServerFactory$1@27646241->org.wildfly.security.sasl.util.AuthenticationTimeoutSaslServerFactory$DelegatingTimeoutSaslServer@2c4151b3->org.wildfly.security.sasl.util.AuthenticationCompleteCallbackSaslServerFactory$1@58b7e66d->org.wildfly.security.sasl.digest.DigestSaslServer@574f31a4] for mechanism [DIGEST-MD5]
2018-09-12 08:05:07,204 TRACE [org.wildfly.security] (management task-1) Handling RealmCallback: selected = [ManagementRealm]
2018-09-12 08:05:07,205 TRACE [org.wildfly.security] (management task-1) Handling NameCallback: authenticationName = edsadmin
2018-09-12 08:05:07,210 TRACE [org.wildfly.security] (management task-1) Principal assigning: [edsadmin], pre-realm rewritten: [edsadmin@ManagementRealm], realm name: [DIGEST], post-realm rewritten: [edsadmin@ManagementRealm], realm rewritten: [edsadmin@ManagementRealm]
2018-09-12 08:05:07,228 TRACE [org.wildfly.security] (management task-1) Handling CredentialCallback: obtained credential for correct realm "ManagementRealm"
2018-09-12 08:05:07,228 TRACE [org.wildfly.security] (management task-1) Handling CredentialCallback: obtained credential: org.wildfly.security.credential.PasswordCredential@bebfb5b9
2018-09-12 08:05:07,230 TRACE [org.wildfly.security] (management task-1) Role mapping: principal [edsadmin@ManagementRealm] -> decoded roles [] -> realm mapped roles [] -> domain mapped roles []
2018-09-12 08:05:07,231 TRACE [org.wildfly.security] (management task-1) Authorizing principal edsadmin.
2018-09-12 08:05:07,232 TRACE [org.wildfly.security] (management task-1) Authorizing against the following attributes: [] => []
2018-09-12 08:05:07,233 TRACE [org.wildfly.security] (management task-1) Permission mapping: identity [edsadmin@ManagementRealm] with roles [] implies ("org.wildfly.security.auth.permission.LoginPermission" "") = true
2018-09-12 08:05:07,234 TRACE [org.wildfly.security] (management task-1) Authorization succeed
2018-09-12 08:05:07,236 TRACE [org.wildfly.security] (management task-1) RunAs authorization succeed - the same identity
2018-09-12 08:05:07,236 TRACE [org.wildfly.security] (management task-1) Handling AuthorizeCallback: authenticationID = edsadmin  authorizationID = edsadmin  authorized = true
2018-09-12 08:05:07,237 TRACE [org.wildfly.security] (management task-1) Handling AuthenticationCompleteCallback: succeed
2018-09-12 08:05:07,237 TRACE [org.wildfly.security] (management task-1) Handling SecurityIdentityCallback: identity = SecurityIdentity{principal=edsadmin@ManagementRealm, securityDomain=org.wildfly.security.auth.server.SecurityDomain@17b8d842, authorizationIdentity=EMPTY, realmInfo=RealmInfo{name='DIGEST', securityRealm=org.jboss.as.domain.management.security.SecurityRealmService$SharedStateSecurityRealm@20c1cb10}, creationTime=2018-09-12T06:05:07.230Z}

[yadavprem]

did you entered the base64 (password encoded) into host.xml of the machine from which you want to connect

[dlofthouse]

The example being discussed here is just using standalone.xml so the host.xml is not used in this situation.