From version 6.6.8, you don't need to do anything. NoMachine supports H.264 GPU-accelerated (hardware) and software encoding/decoding out-of-the-box.
Please consult the changelog for details about version 6.6.8: https://www.nomachine.com/SU04Q00203
See also https://www.nomachine.com/AR04Q01022
All packages for Windows, Linux, Mac and Raspberry include the AVC/H.264 software codec. When the hardware encoding or decoding is not available, NoMachine falls back to software encoding/decoding. No further actions are necessary to enable H.264 support in NoMachine sessions.
For web sessions (from v. 6), H.264 is used when the browser supports WebRTC/HTML5 and H.264 and the support for WebRTC is enabled on the NoMachine Server.
At the moment, H.264 hardware encoding is not available when virtual desktops are running in X11 vector graphics mode (default) and it's not supported in Linux custom sessions. The related Feature Request is https://www.nomachine.com/FR11O03550.
The X11 Vector Graphics mode is a proprietary method which uses a combinations of different techniques to encode each element of the screen in the most convenient way. For example, a video is encoded by H.264 software encoding, static images are encoded by JPEG and textual elements of GUIs are encoded by X graphics primitives by using the NX compression algorithm that is an optimization of the X11 protocol. All these methods are applied at the same time.
NoMachine and H.264 hardware acceleration (GPU)
NoMachine supports H.264 hardware acceleration for these types of sessions:
(i) Connections to the remote physical desktop.
(ii) Linux virtual desktops without activating the X11 vector graphics mode.
(iii) Web sessions with support for WebRTC enabled
GPU encoding (server side)
Currently available on Windows and Linux, NoMachine supports H.264 hardware encoding provided by graphics cards (GPUs) with Nvidia Kepler microarchitecture onward. H.264 HW encoding is not available on Mac computers.
From v. 6, also Intel Quick Sync video cards on Windows and Linux are supported. On Linux, it's necessary to apply a manual configuration as explained here: https://www.nomachine.com/AR09O00938.
Support for hardware H.264 encoder on AMD cards will be soon available (https://www.nomachine.com/FR01P03578).
GPU decoding (client side)
NoMachine software on the user's computer (i.e. NoMachine used as a client or Enterprise Client) supports the H.264 hardware decoding on Windows and Mac hosts with hardware accelerated video cards (GPUs).
We plan to extend support for H.264 HW decoding also to iOS and Android tablets (https://www.nomachine.com/FR12M02992 and https://www.nomachine.com/FR12M02991).
NoMachine and H.264 software acceleration (MPEG-4 AVC libraries)
H.264 software acceleration is used when the server or client doesn't have the necessary hardware requirements to use HW acceleration.
NoMachine supports H.264 software acceleration for these types of sessions:
(i) Connections to the remote physical desktop.
(ii) Linux virtual desktops without activating X11 vector graphics mode.
(iii) Linux virtual desktops in X11 vector graphics mode.
(iv) Multimedia content in custom sessions.
In the case of virtual desktops in X11 vector graphics mode and custom sessions, video streaming techniques, and therefore H.264 SW encoding, are applied only to multimedia contents.
The X11 vector graphics mode is not available for web sessions.
For older NoMachine versions
1) All server products distributed under a customer's subscription (e.g., NoMachine Workstation, Terminal Server and so on) include the MPEG-LA H.264 software encoding/decoding libraries. H.264 software encoding is therefore fully supported. No further action is necessary.
2) NoMachine server packages for evaluation up to v. 6.6.8 didn't include the H.264 encoder. We recommend to upgrade the installation.
3) The NoMachine free version up to v. 6.6.8 was not including the H.264 encoder. If you're still running and older version, please upgrade your installation or refer to instructions for compiling and installing the libx264 library: https://www.nomachine.com/AR10K00695.
4) NoMachine Enterprise Client packages previous than v. 6.6.8 didn't include the H.264 libraries for software encoding. If you're still running and older version, please upgrade your installation or refer to the following instructions to install FFMPEG: https://www.nomachine.com/AR10K00696.
5) The NoMachine AVC Pack (obsolete) was an add-on software for the NoMachine (free) and NoMachine Enterprise Client packages for providing the necessary libraries for H.264 software encoding/decoding. With the release of 6.6.8, the AVC Pack is no longer necessary on either the connecting client or the remote host.