Nexus 7k ISSU code update issues

So as part of that datacenter migration I'm working on, I need to get all my Nexus 7000's on the same code revision. First because they are a few years out of date, but more importantly because newer code has the ability to have dissimilar VLANs on each side. I fly out to the first DC and since it's a DR site with a change window for the whole time I'm there I start preparing for the first code upgrade. As always, start with getting a backup of the running config (both on and off chassis), as well as an overview of routing protocol neighbors, routing tables, etc to validate against in case of issues.

NTG7010-CORE#copy run start

[########################################]  

100%Configuration update aborted: request was aborted

What? No it wasn't. Try again, same result. Not how to start a "simple" upgrade. Start looking into it and it starts to look like a bug. We find a few similar situations, like Geeky Nick's issue. While not 100% identical there were a lot of similarities, and essentially it amounted to systems with a long uptime fill some files to the point no more space remains (I can't seem to find the bug ID). The "workaround" is to reboot the device to clear it. I reboot the standby supervisor thinking if I can reboot that one, do a switchover, and reload the previous active supervisor then I can work around the issue. Unfortunately, the standby then started into a bootloop. This was something Geeky Nick's blog post had as well. We couldn't get the boot image on the standby (it wouldn't recognize USB, or TFTP on my laptop), so I grabbed the show run and saved it a few different places (tftp worked, as did writing to a USB stick), issued the reload command, and hoped it worked.

Thankfully it did. No config lost and could save to bootflash. Yay! From there the upgrade went as planned, with a minor exception that I'll talk about in a later post. When doing an ISSU, bugs from previous versions can carry over through ISSU, even if the bug is fixed in the release you're on. To resolve it we had to reload the modules and it resolved the problem.

Anyconnect VPN tied to AD groups

We have been using Cisco's old IPSec remote access VPN client at my company since before I started. A few months back we upgraded our ASA to the new -X platform with Anyconnect support, but I had shelved the project for a while due to other higher priority projects on the books. I've had the time to be able to work on this lately though, and we are now in the initial testing phases and things are looking pretty good. Only issue is the automatic installer from our public website for it seems to be flaky (something to look into), which I'm sure will create a ton of headaches from end users. But that's a post for another day.

We decided to tie access permissions to AD user groups so we can just handle everything via AD groups through RADIUS (2008 R2 NPS). For ease of user access I decided to tie the 4 accounts together in a single connection profile. Each AD group would get a different permission set (i.e. RDP only, or file sharing if you're on company equipment, or for myself and tech groups SSH access) which will mean that when a user connects to login they can leave the default drop down, use their window credentials, the ASA ships the login auth request to the RADIUS server, which sends back the users auth status plus what AD group they are in, which the ASA then uses to assign users to different filter groups. Here's how I did it via ASDM followed with CLI code.

First things first, the environment:
ASA 5500-X platform running ASA 9.2(2)4/ASDM 7.2(2)
2008 R2 NPS server group (ASA has AAA group pointing to the server group)
Anyconnect connection profile built

Step 1
Create the group policies that will define access permissions in Remote Access VPN -> Network (Client) Access -> Group Policies and click "Add" naming it whatever works for you. For me GroupPolicy_VPNRDP. On the General tab expand out "More Options" and deselect inherit Filter, then "Manage" to create a new filter

In here create a new ACL, mine is named "VPN_RDP_Filter" (I like names that I can easily map through a config file if you hadn't noticed) and throw in the ACL's to allow traffic you want this group to have. Keep in mind you need to grant access to 80/443 traffic if you want them to be able to get to the internet and intranet websites. And it might be nice to allow DNS udp/53 (also Netbios) so they don't have to memorize the global internet IP addresses. Port 3389 is RDP traffic.

Repeat this for all filter groups you want to make. Now one other note here, you will want to create a dummy group to black hole traffic/deny access for users who don't fall into one of your AD groups. I chose to null route them to 1.1.1.1 and deny it. So if a user isn't in an AD group they won't get anywhere.

Step 2
 We need to build an LDAP Attribute map to pull in AD user groups. Under Remote Access VPN->AAA/Local Users tab go to AAA Server Groups and expand the LDAP attribute map at the bottom of the window (I am doing this after building it so my config is already in there) and click Add

We are using the LDAP attribute "memberOf" and tying it to the Cisco Attribute "Group-Policy" (this is not in older code versions and used "IETF-Radius-Class" previously which has a slightly different config) and add it.

Then we need to create the actual mappings of the LDAP attribute "memberOf" (what AD/RADIUS uses for AD group membership) to the Cisco attribute value (what Cisco ASA's use be able to tie a group policy to the AD group). Click "Add" and you'll enter the AD group value. In my case I'm using the following:

CN=VPNRDP,OU=Service Accounts,DC=company,DC=net

The other 3 groups are the same just different CN's. Then I enter the Cisco Attribute Value of my group policy name, in this case GroupPolicy_VPNRDP.

Rinse and repeat for all the groups you want

Step 3
Now is the easy part. Assign the null route group policy to your Connection Profile.

Now go move yourself, or a test account, to a group and test. This is the time consuming part as you need to validate that things are working as expected for all groups. Alternatively you can just push it to your end users and see what they complain about. You know, if you prefer to live adventurously.

Note - RDP access does allow for file transfer over RDP since it's tunneled in the RDP ports. Now this is limiting in that they only can do it to systems they RDP to, but it is still a security risk that you may not be comfortable with depending on your environment/company. I am looking into ways to filter this but I might just take the easy way out and only allow RDP to end user subnets (it'll bother some people I'm sure but if they need more access we can move them to another group).

Update 4/5: I completely forgot to post up the CLI config last night. Here it is though:

Filters:
access-list VPN_RDP_Filter extended permit udp x.x.x.x 255.255.255.0 any eq domain
access-list VPN_RDP_Filter extended permit udp x.x.x.x 255.255.255.0 any eq netbios-ns
access-list VPN_RDP_Filter extended permit udp x.x.x.x 255.255.255.0 any eq netbios-dgm
access-list VPN_RDP_Filter extended permit tcp x.x.x.x 255.255.255.0 any eq 137
access-list VPN_RDP_Filter extended permit tcp x.x.x.x 255.255.255.0 any eq netbios-ssn inactive
access-list VPN_RDP_Filter extended permit tcp x.x.x.x 255.255.255.0 any eq 3389
access-list VPN_RDP_Filter extended permit tcp x.x.x.x 255.255.255.0 any eq www
access-list VPN_RDP_Filter extended permit tcp x.x.x.x 255.255.255.0 any eq https
access-list VPN_RDP_Filter extended permit icmp x.x.x.x 255.255.255.0 any inactive
access-list dummy_split standard permit host 1.1.1.1
access-list dummy_filter extended deny ip x.x.x.x 255.255.255.0 host 1.1.1.1

LDAP attribute map
aaa-server LDAP (Internal) host y.y.y.y
  ldap-attribute-map RADIUS_security_groups
ldap attribute-map RADIUS_security_groups
  map-name  memberOf Group-Policy
  map-value memberOf "CN=VPNRDP,OU=Service Accounts,DC=(company),DC=net" GroupPolicy_VPNRDP

Group policies
group-policy Dummy_GP internal
group-policy Dummy_GP attributes
 wins-server value (server 1) (server 2)
 dns-server value (server 1) (server 2)
 vpn-filter value dummy_filter
 split-tunnel-network-list value dummy_split
group-policy GroupPolicy_VPNRDP internal
group-policy GroupPolicy_VPNRDP attributes
 wins-server value (server 1) (server 2)
 dns-server value (server 1) (server 2)
 vpn-filter value VPN_RDP_Filter

Apply to tunnel group
tunnel-group Remote general-attributes
 authentication-server-group LDAP
 default-group-policy Dummy_GP