RunNSX

So recently at a customer we had the dreaded Not Ready Status under Networking and Security, Installation.  Clicking on Resolve does not fix the issue.  While this can be several things the most common issue is Update Manager not being available, closely followed by time.  Yep, time not being in-sync between vCenter, NSX Manager, SSO, and the ESXi hosts. However getting the details in the logs will give you a narrowed focus.  Whenever you have VIB installation issues always check the host logs and the vCenter logs. ESXi Logs /var/log/esxupdate.log vCenter Logs /storage/log/vmware/vpx/eam.log on Linux vCenter ProgramData/VMware/VMware VirtualCenter/Logs/ on Windows vCenter To see detailed steps to resolve installation errors see the following VMware KB https://kb.vmware.com/kb/2075600  

Almost daily, we read about a prolific rise in breaches and expansion of the security threat landscape. With each attack demanding new approaches and concepts when applying network security, highly virtualized or cloud based environments are often a critical focal point. Virtualization today is more than simply optimizing physical hardware by running multiple virtual machines.  Many organizations have seen increased benefits in disaster recovery, faster time to market, greater automation and ease of migration to the public cloud.  The latest organization and capability to take advantage of the platform afforded by virtualization is security.  Our own organization has recognized this as well – in Pat Gelsinger’s key note at RSA he spoke about how leveraging the virtualization layer enables a new ubiquitous insertion point for innovative security services and solutions. At my current client, a multi-national major airline carrier of 3.9 billion in revenue, VMware is heavily engaged in helping them achieve the first step down the path of Software Defined Security. The first step, of course, being Micro-Segmentation. The key to an effective implementation of micro-segmentation is a thorough understanding of how applications communicate from system to system over the network. However, I have come to the realization that often organizations are blissfully and wildly out of sync on what their overall application landscape consists of. Not only from security controls and flow perspective, but even understanding what the various dependencies, interactions and data classifications are that comprise of business critical applications. For larger enterprises, the grueling task of reverse engineering what these application profiles are might seem arduous or even torturous. Applications consuming the network is considered a bitwise plumping operation and it’s ultimately more about moving those bits than analyzing or gaining meaningful visibility along the way. To amplify the problem, meeting compliance requirements in a converged infrastructure requires pervasive placement of network segmentation controls at multiple convergence points such as Data Centers, Branch Office or even Kiosk terminals. How can an organization hope to achieve and maintain a specific security posture such as PCI-DSS when this lack of metadata exists? By introducing micro segmentation into the picture, we suddenly can introduce new micro-granularity security controls at the datacenter level. Applying consistent security policy an be as simple as assigning tags to workloads or other virtual constructs such as logical switches, port groups or VM naming conventions. The scenario above is exactly the challenge I’m addressing addressing at my current customer.  We are implementing software defined networking and micro-segmentation NSX-based solution to drive a reality shift in security design within the Data Center. The benefits of this solution for the customer – why this is worth the effort – is a significant increase in security protections with a decrease in security management costs.  This also sets the stage for moving from organizations placing hardware at artificially-created choke points to a distributed security model. This Software Defined Security model is one that will enable enterprises and service providers to leverage central management to implement pervasive security quickly and effectively. The end state for the customer will be to achieve PCI-DSS compliance with a pronounced decrease in auditing costs combined with establishing a platform for greater protection.  As we proceed down this path, watch this space for more updates: Tools and technologies to support this change, people and process evolutions to take advantage of new capability, and realized business benefits as the organization matures. Thanks for reading! – Adam Wysockyj is a Senior Consultant at VMware within the Networking and Security Professional Services (PSO) team. The Networking and Security team delivers solutions for VMware customers in the areas of Platform Security (Virtual Security Assessments / Architecture), Hybrid Cloud Security, Network Security, and Network Virtualization.  Adam has spent over a decade building expertise in networking, security, and virtualization in order to design, engineer, and execute optimal solutions for VMware’s strategic customers across multiple verticals such as large financials, retail, and travel.  His expertise with VMware solutions include NSX, vSphere, SRM, and supporting technologies such as Palo Alto Networks, Check Point, and anti-malware solutions.  Adam excels in bringing combinations of products and solutions together to aid customers in developing 21st century network and security architecture that reduce the cost of security while significantly improving protections.  Adam holds many industry certifications such as VCP-DCV,  VCIX-NV, CCNA (R&S/DC/Security), MCSE, MCITP.

http://vmware.jobs/new-york-ny/sr-consultant-networking-and-security/483D9488DD474ADDAD69A765687688E1/job/

We have a typical design with Nexus 7K Border-Leaf and a pair of NSX Edge Gateways enabled for Equal-Cost Multi-Path, and finally the Logical Router which is also enabled for ECMP. The issue I needed to resolve recently at a customer was to remove 0.0.0.0/0 from the OSPF routing table of the Border Leaf Cisco Nexus 7706’s.  This was being redistributed improperly by the NSX Edge Services Gateway (ESG). You see even if we do not enable default originate we still get the 0.0.0.0/0 because we told the ESG’s to redistribute static and connected.  In the NSX Edge when we select redistribute static the default route is advertised to the N7Ks and the DLR.   Note: Redistribute Static is required to properly enable ECMP as we need all Edge devices to provide a default route to the Distributed Logical Router (LDR). We attempted several methods before we found the golden ticket.  table-map

Table Map A table map is a unique feature of NX-OS that allows the network administrator to filter routes or selectively modify the distance of the routes before the routes are sent to routing information base (RIB). The table map uses the route map to select routes based on a wide variety of parameters: metrics, level, type, next hop, outgoing interface, etc.

We used the following set of commands to setup filtering:

! ip prefix-list nsx-routes seq 10 permit 10.0.100.0/24 ip prefix-list nsx-routes seq 20 permit 10.0.101.0/24 ip prefix-list nsx-routes seq 30 permit 10.0.102.0/24 ip prefix-list nsx-routes seq 40 permit 10.0.99.240/28 ! route-map ospf-in permit 10 match ip address prefix-list nsx-routes ! router ospf 1 table-map ospf-in filter

Now we check the on the Nexus and voila we have it!

Came across this in my current engagement.  Was happening on one of the servers in the Payload Cluster, which was a fresh install of 6.0.0, 2494585 After upgrading to or installing ESXi 6.0.x and ESXi 6.0 Update 1, you may experience these symptoms:

• ESXi 6.0 host network connectivity is lost randomly
• ESXi enters into a non-responding state and becomes unmanageable until reboot
• Network connectivity does not return until the ESXi 6.0 host is rebooted
• After rebooting, the issue is temporarily resolved for a period of time, but occurs again after a random interval unless the workaround (script) is implemented

In the /var/log/vmkernel.log file, there is evidence of transmit timeouts.  This is often logged by the NETDEV WATCHDOG service in ESXi. You may see entries similar to:

YYYY-MM-DDT21:35:34.007Z cpu0:33245)WARNING: LinNet: netdev_watchdog:3678: NETDEV WATCHDOG: vmnic0: transmit timed out YYYY-MM-DDT21:35:34.007Z cpu0:33245)WARNING: at vmkdrivers/src_92/vmklinux_92/vmware/linux_net.c:3707/netdev_watchdog() (inside vmklinux)

This issue can impact multiple network adapter types across multiple hardware vendors The exact logging that occurs during a transmit timeout may vary from card to card. Note: This issue does not affect ESXi 5.5 and earlier releases, only ESXi 6.0 is affected. This issue is resolved in ESXi 6.0 Update 1a, available at VMware Downloads. For more information, see the VMware ESXi 6.0 Update 1a Release Notes. ESXi 6.0 network connectivity is lost with NETDEV WATCHDOG timeouts in the vmkernel.log (2124669)