Since v. 4, NoMachine implements its own protocol for secure communication over the network. The products targeting commercial use, so both of the Enterprise Server and Terminal Server families, additionally support the SSH protocol out of the box. All products use the NX protocol as default. There are multiple reasons for using our own protocol rather than SSH.
The first is performance. Tunneling over SSH means that our packets have to traverse at least 1 additional process before coming to the destination (at least the SSHD process, if we don’t run a separate SSH client). This is an additional process for each machine traversed, so in a multi-node server there are at least two. Then with SSH we have processes communicating through pipes (like multiple separate commands piped in a shell), so we are adding a further encryption stage at each hop.
With NX we can simply hand over the SSL context from one process to the next (as Apache does) and relay connections by only running encryption end-to-end. We can’t provide the details, but we were in a situation where a display packet, to come to the client, had to traverse 12 processes and be encrypted 3 times. Not so with the NX protocol.
Additionally, when using the NX protocol, audio and video can use UDP. Not that we couldn’t use a UDP side-channel with SSH, but it would have been hard to explain to managers in a company that, yes, we use SSH for the connection but then most data is not going through SSH. There are additional tiny details, like the efficiency of the crypto key used, that adds to the speed, or the fact that SSHD is a single-threaded process while with NX everything is multi-threaded and can run on platforms, like iOS, where multi-process is not an option.
A second reason is that, using SSH, we can’t simply support a number of features we need in NX, like keeping a NX users separate from the system users, supporting guest connections and redirecting users to different machines without having to create system accounts. Unless recurring to workarounds. That is what we did with the use of the “NoMachine login”. These workarounds are perfectly in the spirit of SSH but were judged “questionable” by some, simply because they by-passed PAM and let NoMachine create users and check passwords on its own. With the NX protocol these “questionable” uses of SSHD are gone.
A third reason is Windows. We ported the OpenSSH client and server to native Windows and released it on the same licensing terms of OpenSSH. This was done to offer the same set of features on all platforms, but it's not in our plans to develop further SSH for Windows.
A fourth reason is supportability. It’s hard to support something you have no control over how it is used or configured. SSH is an extremely powerful and configurable tool. When users install NoMachine and NoMachine doesn’t work because the SSH client or server are configured to do something special that we could not foresee, users tend to blame NoMachine. This is well within their rights to do so, but now having two distinct protocol options, enterprise users can still use SSH if they want, but at least we will know if a problem they report is due to SSH or NoMachine.