Private Nameservers in WHM/cPanel
Here are the steps:
Step 1. Configuring server
Login to whm: https://YourIP:2087
Find the navigation section on the left called Basic cPanel & WHM Setup, under Server Configuration. This is the first item on the left navigation bar.
Find Nameservers section at the bottom. If you see ns1.domain.com and ns2.domain.com you can skip this step. If you don't see those name servers, specify them and press "Assign IP address"
You can also do this from command line (you must do it only from command line if you want to force cPanel/WHM using only 1 ip address for nameservers):
Remove cache file:
Before you make any changes to ns file you can make a backup:
Open ns file:
Edit the file so that it contain your IP adresses and name servers like below:
Step 2. Register your name servers
Login to your domain control panel (at your registrar) and create child name servers ns1 and ns2. Exact steps may vary depending on your registrar. Here is an example:
ns1.domain.com 18.104.22.168 [your VPS IP N1]
ns2.domain.com 22.214.171.124 [your VPS IP N2]
As described above, your VPS need to have 2 IP addresses to make nameservers work.
NOTE : WHM needs 2 IPs for setting up name servers but you can bypass that with editing /var/cpanel/nameserverips.yaml /as described before/ if you are familiar with ssh. Using two IPs for nameservers is recommended in general.
Now specify that new name servers for the domain.
Full propagation may take some time (up to 24 hours).
Step 3. DNS Zones
Find the navigation section on the left called Edit DNS Zone, under DNS Functions.
Select your domain and press "Edit".
Here you can add and edit dns records.
If you can't see your domain in dns zones, you can add it in the navigation section on the left called Add a DNS Zone, under DNS Functions.