RSA keys generated with openssh version 7.8p1-1 or later introduces a new format: the private key has "OPENSSH" written in the first line instead than "RSA".
This new format has to be supported by NoMachine when using key-based authentication, both in case of connections by NX and SSH protocol. This last case require updates to the libssh2 library shipped by NoMachine packages (see https://www.nomachine.com/FR05Q03833).
As a temporary solution, it's possible to generate SSH keys in the old format by using openssh 7.8p1-1or later with appropriate switches.
For example on Linux:
ssh-keygen -m PEM -t rsa -b 4096
(PEM specifies the old format)
Ref. https://github.com/duplicati/duplicati/issues/3360