Note that this authentication method is no longer supported since NoMachine v. 6: https://www.nomachine.com/FR01O03307
The way NoMachine login works is by using a special user account named 'nx', created on the server host during the software installation, and whose shell, /etc/NX/nxserver, is executed any time a remote user connects by SSH and NoMachine login.
The nx account is needed to setup the initial stage of the client-server connection and uses a RSA keypair to authenticate. This is all done before the real user login happens. The RSA key is NOT used for making the user to log into the system. The private and public part of the RSA key pair is provided by the installation of the NoMachine client and server software.
Using the RSA key pair forces the SSH server to execute the nxserver shell and prevents any possibility for the special user 'nx' to login on the server host outside NoMachine. As the 'nx' user is required to properly function, the username for 'nx' cannot be changed.
Once that the initial stage is completed and the 'nx' user has logged-in through the usual host authentication and SSL key negotiation mechanisms offered by the Transport Level Security built in SSH, a secure encrypted channel is established. This encrypted channel is then used for the authentication of the real user.
Hence nobody can login to the system via NoMachine and with only knowing the NoMachine RSA key. He can only start the initial 'handshake' phase and is then stucked. There is no way for the 'nx' user to gain administrative privileges ('root' privileges on Linux and Mac) other than by exploiting some other bug that is present in the OS or in a existing program that is allowed to run with administrative privileges. Additionally, no operation is allowed by the NoMachine server until the real user has authenticated.
By comparison, this use of SSH is in practice the same that HTTPS servers do with the Secure Socket Layer specification to provide secure access to Web content.
For a further level of control over the access via NoMachine, administrators can replace the default RSA key-pairs with their own SSH keys. To do that it's necessary to generate a custom SSH key pair by means of the 'nxserver --keygen' command and distribute the private part of the key pair to all clients that have to be granted access to the server host. In case of multiple NoMachine servers, it's possible to set-up different RSA key-pairs for each server. Detailed instructions for v. 5 are available here: https://www.nomachine.com/DT09M00103#6
Once the initial authentication stage between client and server is completed, it's requested that the real user authenticates:
(i) either against the authentication system by providing his system username and password .
This offers seamless integration with the authentication subsystem (e.g. PAM,LDAP and AD) configured on the machine.
To verify the password, NoMachine will submit the user's credentials to the authentication subsystem. If it guarantees the access, then the NoMachine server will allow the user to enter the session.
(ii) or against the NoMachine password database.
In this case, the server verifies the user's password againts its pasword db. NoMachine password can be different to system password. This offers a separation between system users and NoMachine users, as only users with a system account and that have been added to the NoMachine system will be able to connect by NoMachine. The user has to be added to the NoMachine password db by means of the 'nxserver --useradd' command.
Besides the basic configuration outlined here, NoMachine offers a number of methods for ensuring that the system administrator maintains full control on the set of users that are allowed to connect. This happens by manipulating two distinct databases: the NoMachine passwords database, see point (ii) above and the users database. When this last one is enabled, users that are authenticated by system passwords might be required to exist in the NoMachine users database, so that it is possible to guarantee access to only a restricted set of system accounts.
Authentication via SSH and NoMachine login relies heavily on SSH functionalities to make it possible to run contemporary desktops and arbitrary network applications, across the Internet, in a secured and controlled fashion. NoMachine leverages the same logging and auditing facilities of SSH, as well the secure access to critical authentication records that LDAP,PAM and AD provide together with SSH.
1) Connections by SSH + NoMachine login method is the only way supported by NX 3.5.0
2) The RSA key pair has been introduced since v. 5.1.22, previous versions use a DSA key. See also: https://www.nomachine.com/FR04N03093
3) On Linux and Mac, connections by SSH relies on the system SSHD. On Windows, they rely on nxsshd, the SSH server based on the OpenSSH Port to Win32 made by NoMachine and included in the NoMachine server package.
4) In case of multi-node environments the user's account must exist on the server host and on the node(s) the user should be granted to access. Username must be the same on all hosts, password can be different.