Integrating CentOS 7.5 BareMetal Server with NSX-T 2.5

By | 20. December 2019

This post demonstrates how to integrate a bare-metal server on OS-Level into NSX-T overlay networking and security. As there was no bare-metal system in my demo environment available I created a CentOS 7.5 VM on a non-NSX enabled ESXi Server. Topology shown below

After successful installation it should look like this

Pre-Requisites:

  1. Have NSX-T up and running
  2. Read the NSX-T Bare-Metal Integration Documentation https://docs.vmware.com/en/VMware-NSX-T-Data-Center/2.5/installation/GUID-1B199883-EAED-45B2-B920-722EF4A54B33.html
  3. prepare a CentOS 7.5 system (bare metal or VM), integrate 2 NIC [referenced here as ens192 and ens224]
  4. one NIC [ens192] needs a configured IP address [10.10.100.45/24] and should be connected to your management network
  5. setup a ssh server on your CentOS 7.5. It should start on boot. Password based SSH root login must be enabled
  6. the second NIC needs to be connected to your NSX overlay transport network [VLAN10]. Don’t configure IP or anything else yet.
  7. have a Management System ready which is allowed to ssh to your CentOS system. git & ansible need to be installed

If you do this in a CentOS VM don’t forget to change the VDS Port Group Security settings in vCenter for the overlay interface [VLAN10]. Enable “Promiscous mode” and “Forged transmits”. NSX will later create virtual interfaces. If not changed the ESXi will drop packages from these interfaces.

Information needed:

ValueExample (in my demo)
NSX Manager API User admin
NSX Manager API Password VMware1!VMware!
NSX Manager API thumbprint 66f[…cut…]f507
NSX Manager IP10.10.100.11
NSX TEP Pooltep-pool
NSX Logical Segment the CentOS Server connects to LS_TEST
Application Interface IP Address for this segment 10.10.110.45/24

You can get the NSX Manager API certificate when you ssh as admin into your NSX Manager IP and run

nsxmgr1> get certificate api thumbprint
66f[...cut...]f507

OS Setup:

SSH into your CentOS system and login as root.

It can be useful to disable network-manager on the second interface. So add “NM_CONTROLLED=no” to /etc/sysconfig/network-scripts/ifcfg-ens224 and restart the interface.

ifconfig should show both interfaces, just one is configured with management IP

[root@bm-centos-mgmt ~]# ifconfig
ens192: flags=4163 mtu 1500
inet 10.10.100.45 netmask 255.255.255.0 broadcast 10.10.100.255
inet6 fe80::250:56ff:feb3:6f9 prefixlen 64 scopeid 0x20
ether 00:50:56:b3:06:f9 txqueuelen 1000 (Ethernet)
RX packets 5028 bytes 7566950 (7.2 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 3625 bytes 280022 (273.4 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

ens224: flags=4163 mtu 1500
ether 00:50:56:b3:0e:bc txqueuelen 1000 (Ethernet)
RX packets 3 bytes 180 (180.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73 mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10
loop txqueuelen 1000 (Local Loopback)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

NSX-T Documentation lists packages you should have installed. Additionally I recommend to add “vim-common” as it might be needed to run the documented command to extract the ssh thumbprint. I recommend to put all needed packages into a textfile and install them:

[root@bm-centos-mgmt ~]# echo "tcpdumpboost-filesystem PyYAML boost-iostreams boost-chrono python-mako python-netaddr python-six gperftools-libs libunwind libelf-dev snappy boost-date-time c-ares redhat-lsb-core wget net-tools yum-utils lsof python-gevent libev python-greenlet libvirt-libs vim-common" > packagelist.txt

[root@bm-centos-mgmt ~]# yum -y install $(cat packagelist.txt)

Optional: get the SSH SHA256 thumbprint.

[root@bm-centos-mgmt ~]# awk '{print $2}' /etc/ssh/ssh_host_rsa_key.pub | base64 -d | sha256sum -b | sed 's/ .*$//' | xxd -r -p | base64
DUA[...cut...]WcQ=

Now you can prepare your CentOS system as NSX-T Transport Node. Login to NSX Manager GUI, goto System -> Fabric -> Nodes and add a standalone Host Transport Node. I’ve also pre-created a new uplink profile shown in the last picture.

Config Screen
Config Screen

Uplink profile for my setup

Uplink Profile

It can take several minutes to complete.

Now you can login to your Management System.

Clone the VMware Bare Metal Git Repository

vmware@mgr:~$ git clone https://github.com/vmware/bare-metal-server-integration-with-nsxt.git

change into directory

vmware@mgr:~$ cd bare-metal-server-integration-with-nsxt/bms-ansible-nsx/

edit the following parts of the “hosts” file.

[TransportNodes:vars]

ansible_ssh_user=root
ansible_ssh_pass=VMware1!

[servers_static]

server1 ansible_ssh_host=bm-centos-mgmt.corp.local static_ip=10.10.110.45 netmask=255.255.255.0 ls_name=LS_TEST

[NSX]

nsxmanager ip=10.10.100.11 username=admin password=VMware1!VMware1! thumbprint=66f[...cut...]f507

Now you can run the ansible playbook

vmware@mgr:~/bare-metal-server-integration-with-nsxt/bms-ansible-nsx$ ansible-playbook -i hosts static_config.yml

Should finish with something similar like that

PLAY RECAP *
server1 : ok=34 changed=21 unreachable=0 failed=0

ifconfig now shows the VTEP Interface [10.10.10.18], the application interface [10.10.110.45]. Overlay networking now should work, so you should be able to connect to VMs residing in NSX Logical Segments.

[root@bm-centos-mgmt ~]# ifconfig
ens192: flags=4163 mtu 1500
inet 10.10.100.45 netmask 255.255.255.0 broadcast 10.10.100.255
inet6 fe80::250:56ff:feb3:6f9 prefixlen 64 scopeid 0x20
ether 00:50:56:b3:06:f9 txqueuelen 1000 (Ethernet)
RX packets 14159 bytes 62060667 (59.1 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 15443 bytes 2379513 (2.2 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

ens224: flags=4163 mtu 1500
ether 00:50:56:b3:0e:bc txqueuelen 1000 (Ethernet)
RX packets 2938 bytes 291354 (284.5 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1920 bytes 260447 (254.3 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

genev_sys_6081: flags=4163 mtu 65470
inet6 fe80::505d:acff:fef9:9231 prefixlen 64 scopeid 0x20
ether 52:5d:ac:f9:92:31 txqueuelen 1000 (Ethernet)
RX packets 1400 bytes 72800 (71.0 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1491 bytes 77548 (75.7 KiB)
TX errors 0 dropped 8 overruns 0 carrier 0 collisions 0

hyperbus: flags=4163 mtu 1500
inet6 fe80::acf4:3cff:fe75:6d7 prefixlen 64 scopeid 0x20
ether ae:f4:3c:75:06:d7 txqueuelen 1000 (Ethernet)
RX packets 8 bytes 648 (648.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 8 bytes 648 (648.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73 mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10
loop txqueuelen 1000 (Local Loopback)
RX packets 184269 bytes 47619581 (45.4 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 184269 bytes 47619581 (45.4 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

nsx-eth: flags=4163 mtu 1500
inet 10.10.110.45 netmask 255.255.255.0 broadcast 10.10.110.255
inet6 fe80::5855:e7ff:fe5a:9d19 prefixlen 64 scopeid 0x20
ether 5a:55:e7:5a:9d:19 txqueuelen 1000 (Ethernet)
RX packets 8 bytes 648 (648.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 8 bytes 648 (648.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

nsx-eth-peer: flags=4163 mtu 1500
inet6 fe80::5497:61ff:fe34:a1ce prefixlen 64 scopeid 0x20
ether 56:97:61:34:a1:ce txqueuelen 1000 (Ethernet)
RX packets 8 bytes 648 (648.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 8 bytes 648 (648.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

nsx-vtep0.0: flags=4163 mtu 1500
inet 10.10.10.18 netmask 255.255.255.0 broadcast 10.10.10.255
inet6 fe80::a088:7bff:fed1:a43c prefixlen 64 scopeid 0x20
ether a2:88:7b:d1:a4:3c txqueuelen 1000 (Ethernet)
RX packets 2840 bytes 243974 (238.2 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1517 bytes 170578 (166.5 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

print

4 thoughts on “Integrating CentOS 7.5 BareMetal Server with NSX-T 2.5

  1. FABIO MOLINA ALVARO

    Hello, thanks for this help.

    Do you know how to add persistent route pointing to this new interface that is created (nsx-eth)?

    Reply
    1. Daniel Paul Post author

      Hi, thanks for your comment. When editing the “hosts” file from the “VMware Bare Metal Server Integration repository” you can add a “routing_table” parameter.
      Example:

      [servers static]
      server1 ansible_ssh_host=bm-centos-mgmt.corp.local static_ip=10.10.110.45 netmask=255.255.255.0 ls_name=LS_TEST routing_table=”-net 172.16.11.0/24 gw 10.10.110.1″

      Hope this helps?
      Best,
      Daniel

      Reply
  2. zhanlin

    Hi Daniel,

    With your setup, did you try l3 connectivity of 10.10.110.45? like vm resides in another segment, or even outside of T0?

    Reply
    1. Daniel Paul Post author

      Hi, if L3 connectivity for the overlay segment (LS_TEST) is configured there should be no problem for the bare metal system to use L3 connectivity like any other (VM) Workload on the segment. But keep in mind to modify the routing table to use the gateway provided by overlay network. Just check the first comment on my blog post how to use “route_table” to configure L3 overlay routing during setup.

      Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.