WSUS, 3rd Party Update and Local Publishing to a DFS or UNC

I'm not going to get elaborate with this post.  Mostly because I don't post often.  However this is one of those "Woohoo!" moment when you get something working despite everyone else saying it does not work or there is no workaround.

I'm referring to a "bug" in the WSUS Publishing API when your WSUS Content folder resides on a UNC or DFS path.

http://blogs.msdn.com/b/minfangl/archive/2014/01/29/system-center-update-publisher-2011-troubleshoot-series-3-publish-fail-when-scup-2011-is-remote-and-wsus-is-using-unc-as-content-folder.aspx

In this scenario your WSUS servers have a registry key:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Update Services\Server\Setup\ContentDir

This points to the UNC path were all your content is published to.  The problem is that this information is not available if your using SCUP on a remote system or your 3rd party updating tool is on a different server from your WSUS server.

Honestly after beating my head on a wall for a few days I just exported the whole key:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Update Services\Server\Setup

Then imported that onto the system where your running SCUP.  I expect the same would work for other 3rd party patching tools like from SolarWinds or others.

http://www.solarwinds.com/patch-manager.aspx

I'm guessing that the WSUS Console expects these registry keys to be available.  All the 3rd party patching just called it a bug.  I bet if you ran Process Monitor you could identify the specific registry keys that are needed rather than exporting all the sub-keys.

http://technet.microsoft.com/en-us/sysinternals/bb896645.aspx

Cloud & Hosted Security

SO it’s a blast watching patch 61 of 98 installing at 2AM…

Something my troubled brain has pondered is why companies seem to be migrating services to cloud servers in droves.  How can you trust anything if you don’t manage the physical hardware?

Whoever manages that physical hardware is holding all the keys.  No matter how “secure” you make the VM from network attacks, use duper strong pass phrases and keep the system patches and even install HIPS its easily bypassed by the provider.

This article about how a collection of 3 letter agencies were able to take down "The Silk Road" a website that operated in the Tor network (just to be clear I am not upset they took the website down but this is not a post about morality).  Anyway this re-started my brain pondering the security of cloud VMs and hosted servers.

http://arstechnica.com/tech-policy/2013/10/how-the-feds-took-down-the-dread-pirate-roberts/

The interesting part in the article was “the government asked for an image of the server's hard drives”.  This was accomplished without turning off or shutting down the VM.  It happened without the owner ever knowing something happened.

It’s a security issue that’s been raised before.

http://searchvmware.techtarget.com/news/1378347/How-to-steal-a-virtual-machine-and-its-data-in-3-easy-steps

So what can we do to mitigate this type of security issue?

Nothing.  Seriously, nothing.

For example how about installing PGP WDE/TrueCrypt on the VM and encrypt the entire VM contents?  Sound like that would fix everything right?  I mean the virtual drive is encrypted and so if they capture that then it’s still encrypted.  Great!  Oh well the problem is "they" have physical access to the host so the operator can simply snapshot the entire VM and dump the VM Memory to a file.  Now can you guess where the encryption keys are stored while the VM is running?  Yep in memory.  So now they just search the memory for where PGP WDE/TrueCrypt stores the keys.  Well shoot we didn’t even have to hack into the console redirection code of the host operating system to log console redirection when the VM owner had to manually enter the PGP WDE/TrueCrypt key to get the VM to boot.

So not even the coveted whole disk encryption can save your VM from curious eyes that might be working for a cloud provider. 

Now this could have also been done on any physical server with raid 1 drives.  They would just have to pull a drive out claiming it had failed and needed to be replace.  Once pulled they could simply  pop that drive into another system and capture the image.  For physical servers there is some additional security such as adding some type of whole disk encryption on the physical server would prevent the drive from being pulled and used to capture the system image.  However it would still be possible to extract the encryption keys from the physical servers memory the process is much more involved and would require some noticeable downtime while the system memory was dumped.

There is also the TPM (Trusted Platform Module) chip on some motherboards that provides a secure location to store the encryption keys between rebooting.  This eliminates the requirement to enter any pass phrase to being booting the system.  The problem here is that the manufacture creates the RSA keys used by the TPM chip and they cannot be re-generated (to my knowledge).  So every TPM chip produced is intentionally compromised.  Yah very “trusted”.  This is similar to the issue with centralized Certification Authorities.  So even [insert trusted root CA here] SSL certificates are intentionally compromised but that’s another rant of mine.

If you cannot trust the physical network, physical security and host the VM is being executed on there is no amount of “security” you can patch on to make it secure. 

So where does that leave us?  Lockup your servers in a dark, cold, tower of a room and do not let anyone in.

Ok back to watching servers patch.

J

Document Active Directory with GPO Reports

Accuratly documenting your network is one of the most important things you can do.  Not only does it make your job easier but it also helps make everyone elses job just a little easier.  Knowing how something should be can really help in getting it fixed faster.

We have had a big push to more acuratly document our network.  Being a big fan of GPOs we have quite a few of them.  Around 160 of them actually.  So I wanted to document them but without having to look through each one and write every setting down.

Luckily the Group Policy MMC has a neat little report you can export.  Whats even neater is you can call that up with PowerShell and export it without using the GUI.  So you can export every setting in all your GPOs with a single script.

We like to keep records of what changes so each time you run this it puts the reports into a dated folder.

Here is that script.

Import-Module GroupPolicy $year = get-date -uformat "%Y" $month = get-date -uformat "%m" $day = get-date -uformat "%d" New-Item -ItemType directory -Path C:\GPOReports\$year\$month\$day -ErrorAction silentlyContinue $GPOs = Get-GPO -All $pattern = "[{0}]" -f ([Regex]::Escape([String][System.IO.Path]::GetInvalidFileNameChars())) foreach($GPO in $GPOs){  $filename = $GPO.DisplayName  $filename = [Regex]::Replace($filename, $pattern, ' ')  $path = 'C:\GPOReports\' + $year + '\' +$month + '\' + $day + '\' + $filename + '.htm'  Get-GPOReport -Guid $GPO.Id -ReportType HTML -Path $path }

Spreading the word about Solarwinds

Everyone knows monitoring your network is important and there are so many tools out there to get the job done it can be overwhelming.  I have tried several and have found SolarWinds to be my favorite.  I have been using it since around 2007 and love the active support and community.

SolarWinds has a whole set of tools for whatever your needs are from standard network monitoring to configuration management and security event management.  The great thing about all their products is they offer some of the highest quality support I have ever dealt with.  Their support is responsive and they really know the products they support.

If you’re looking at getting something for network management checkout the solutions from SolarWinds.

CentOS & RHEL 6.3 missing libnetfilter_queue

I am no Linux expert so it bugs me when I have to go hunting for packages to meet my requirements. In this case I was trying to setup a new CentOS 6.3 development VM for my project OpenNOP. To build OpenNOP it requires libnetfilter_queue & libnetfilter_queue-devel. These are very popular packages now and most distro have them out the door. Debian, Fedora, openSuSE are the ones I know of to include these packages. Now that would be fine if these were burried away down in some "extra" or "contrib" repo but no they just dont seem to exist. Redhat has not packaged them for RHEL thus CentOS does not have them either.

Well to finally end my frustration I was able to find a repo that contains these packages for me and from a trusted source. ClearOS the gateway appliance based on CentOS has built packages for libnetfilter_queue and libnetfilter_queue-devel.

Just create a new repo and enable them to install the packages. Then disable them when finished. They offer almost all the same packages as the RHEL/CentOS repos so we want to be very careful when using these. Be even more careful when trying to update them.

nano /etc/yum.repos.d/ClearOS-Base.repo # CentOS-Base.repo # # The mirror system uses the connecting IP address of the client and the # update status of each mirror to pick mirrors that are updated to and # geographically close to the client. You should use this for CentOS updates # unless you are manually picking other mirrors. # # If the mirrorlist= does not work for you, as a fall back you can try the # remarked out baseurl= line instead. # # # # Added Custom Repo for libnetfilter_queue # #base - packages by ClearOS [clearos] name=ClearOS-$releasever - Base #mirrorlist=http://mirror.clearfoundation.com/?release=$releasever&arch=$basearch&rep$ #baseurl=http://mirror.centos.org/centos/$releasever/contrib/$basearch/ baseurl=http://mirror.clearfoundation.com/clearos/community/$releasever/os/$basearch/ gpgcheck=0 enabled=0 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6 # # Added Custom Repo for libnetfilter_queue # #addons - packages by ClearOS [addons] name=ClearOS-$releasever - Addons #mirrorlist=http://mirror.clearfoundation.com/?release=$releasever&arch=$basearch&rep$ #baseurl=http://mirror.centos.org/centos/$releasever/contrib/$basearch/ baseurl=http://mirror.clearfoundation.com/clearos/community/$releasever/addons/$basearch/ gpgcheck=0 enabled=0 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6

Grading the quality of your source code

While reviewing the source code for OBS over on GitHub. I stumbled upon a very neat little tool called Code Climate. It reviews your code and provides you with real-time feedback on it's quality. Currently Code Climate only supports Ruby and while that helps me very little the concept is still pretty great. I asked them if they had plans to include other languages for their service. They answered "Yes" but did not have any plans to include C or a link to SVN at this time. If you’re into Ruby and Git it’s a pretty neat tool. I hope they will soon include other languages and help people make better code.

WSUS Wont download updates

Over the years of working with WSUS I have had some clients that just refuse to download and install updates. Eventually manually clearing out the BITS download cache and trying to resync they will eventually work. So I made a little script to do that for me. net stop wuauserv net stop bits REG DELETE "HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate\Auto Update" /v NextDetectionTime /f del %windir%\SoftwareDistribution\*.* /q/s net start bits net start wuauserv wuauclt /detectnow

New Core Switches

I got some new toys at work and am in the process of getting the first pair installed. We still need to figure out how to free some rack space to place the second pair at our other location. Here is a photo of the first pair of 6506Es we got installed. There is one more vertical cable organizer to install on the left side but the install went pretty smooth. Considering we had to pull out two 23" racks and install the two 19" racks without turning some equipment off. Thats not something I want to do every week.

Here are the specs of the four switches we got:
Cisco Catalyst 6506E
SUP-720-3B
WS-X6748-GE-TX + DFC
WS-CAC-2500W
WS-CAC-2500W

Throttle WSUS durring business hours

Deploying a WSUS server in a corporate network is pretty standard practice but it can negatively impact users when the PCs have to download updates from a remote WSUS server over a WAN connection.

The problem is that WSUS does not throttle the amount of bandwidth available to download the updates it will just use as much of your WAN circuit as possible while the PCs download the required updates.

One solution would be to enable throttling of the WSUS traffic on your routers during business hours. This can be complex to setup if it’s even supported by your routers. An easier option is to enable IIS throttling on your WSUS site in IIS during business hours and disable IIS throttling after closing.

To do this you need to use the script adsutil.vbs in "C:\Inetpub\AdminScripts\" and create a scheduled task to enable/disable the throttle.

1. You need to find the site identifier for your WSUS Administration site. Open the IIS Manager click on "Web Sites". Look at the "Identifier" column of your WSUS Administration site. This is your sites id.

2. Create a new scheduled task to enable the throttle. Enter the site id for your WSUS Administration site from step 1. Pick a time when you want to start the throttle. I picked 6:00AM because I want to make sure it’s enabled before anyone shows up to work. The property to enable the throttle is "MaxBandwidth" and is in bps. If you want to limit WSUS to 512K you must enter "524288" (512 * 1024) the IIS Manager will list this in Kbps.
C:\WINDOWS\system32\cscript.exe "C:\Inetpub\AdminScripts\adsutil.vbs" SET /w3svc/<siteid>/MaxBandwidth 524288

3. Create a second task to disable the throttle. To disable the throttle we set the value to its max not to 0. I scheduled this task to run at 8:00PM after everyone has left.
C:\WINDOWS\system32\cscript.exe "C:\Inetpub\AdminScripts\adsutil.vbs" SET /w3svc/<siteid>/MaxBandwidth 4294967295

In the end you have two scheduled task that look somewhat like this. One that will enable throttling on your WSUS site to prevent it from saturating your WAN during the day and a second to disable it so the PCs can get their updates as fast as possible in the evening.

SCVMM: Mount Shared ISO

I have been using SCVMM R2 for a while but one thing I never got working correctly is the ability to mount an ISO directly from the library share rather than copy the ISO to the VM. There are plenty of guides to help you get that setup. Like here:
http://blogs.technet.com/b/dutchpts/archive/2009/02/09/hyper-v-and-scvmm-2008-mounting-iso-s-from-a-network-share.aspx?wa=wsignin1.0

It never worked for me until I found this other little blog entry by Timothy. http://www.avianwaves.com/Blog/default.aspx?id=187&t=SCVMM-Mounting-Shared-ISOs-From-The-Lib

Basically if your SCVMM library is on the same server as the Hyper-V host you just need to add the NT AUTHORITY\Network Service user account to both your library Share and NTFS permissions.

iPad

So I got an Apple iPad Tablet (32GB, Wi-Fi) from work and have been playing with it. Its a pretty neat toy. I have been looking around for apps and have found a few that I like so far.

Kindle
iBooks
ABC Player
Epicurious
Wikipanion
Pandora
Dungeon Hunter HD
Marvel
Netfilx
SSH Terminal
Nightstand
RDesktop
Calculator XL

I loaded the sample of Understanding the Linux Kernel, Third Edition and tried reading a few chapters. It was actually pretty easy on the eyes after lowering the brightness and tilting the screen a little so the lcd was not shining directly into my eyes. It’s still not even close to as easy to read as a kindle but makes a great alternative multiuse device.

Droid X

We recently got new phones for work and most of us got Android based phones either the Droid 2 or the Droid X. I opted for a Droid X due to the larger screen size and HD camera. The slide out keyboard was tempting but I had a Blackberry Storm so am accustomed to the on screen keyboard already.

Overall I am very happy with the phone. I have found a few quirky issues that bug me a little. Like when you search your corporate directory and click on the contacts phone number to place the call it does nothing until you use the "back" button to exit the corporate search at that point the phone makes the call. Not a huge problem just annoying.

I was extremely worried about dropping or scratching the phone so went ahead and got some accessories to protect the phone. First was the Carbon fiber case from Verizon. While I was waiting for that to arrive I was also looking for screen protectors. I ended up getting the ZAGG invisibleSHIELD for Motorola Droid X (Screen). Its not the easiest screen protector to put on but it does a great job once you finally get it on.

OpenNOP Open Source Network Accelerator

So I have been sporadically working on my pet project OpenNOP which stands for (Open source Network Optimization Platform). Several months ago I re-branded it from its former name of PacketSqueezer as it was to very similar to another open source project that I had been testing called TrafficSqueezer.

Originally I was using PacketSqueezer as an example on how similar results could be achieved without having to write an entire custom kernel. Something I really didn't like was having to compile a custom kernel to use TrafficSqueezer. Some of us just want to install and go.

I am really not a programmer so its mostly just me fumbling around. So if you are a programmer with some networking skills please spare a few cycles and check out the code. Its all uploaded at http://www.sourceforge.net/projects/opennop. I could use the code review and if you want to work on this just let me know. I would love to see it mature and become the open source alternative to "insert expensive commercial WAN acceleration solution here".

CCNP Routing & Switching Certification Library

I received my CCNP Routing & Switching Certification Library today and have to say that I am very excited about getting into it. CCNP has been something of a dream goal of mine for the last several years so actually starting to study for the exams is invigorating.

Passed IINS Exam 640-533 today!

Well I took the CCNA Security exam today and past. It was down to the wire as my CCNA cert would have expire Aug 2nd.

I scored a 955 on the exam after studying hard the last three weeks. I studied mostly using the CCNA Security Exam Cram by Eric Stewart. He said I should get his book in the forums over at www.dslreports.com so I did. The last week I used Boson ExSim-Max to get use to the test taking process again.

Have to say pretty happy I passed. Was not looking forward to taking the CCNA exam again if I failed.

Open Source WAN Acceleration

I have been doing research on WAN acceleration for a while now. Unfortunately due to the cost I have been unable to get it installed in the network that I manage. So several months ago I started working on my own WAN accelerator as an open source project.

I recently finished a pre-alpha, and am moving forward with its development. I have setup a portal for it at http://www.opennop.org. The pre-alpha is entirely contained in a kernel module, and is limited to compression. The next release I am working on is a combination kernel module and user space service that does the optimizations. It also currently capable of compression of the application data within the TCP segment, but is also multi-threaded to increase performance.

The next feature I will be adding is block based data reduction. This will be a difficult feature to implement, but is one of the most beneficial in terms of increasing WAN performance.

Fun with a Smart-UPS 1400RMNET.

I got an old UPS its a Smart-UPS 1400RMNET. Its the type without a battery tray just an array of batteries that must be slid into the unit, and then connected.

This unit uses 4 batteries, but I didn't happen to have the wiring diagram so I was just connecting them how I thought they might have been connected. I thought they were wired in two parallel series giving the battery array a total of 24v, but longer run time. The unit made all kinds of horrid noise when I turned it on. So I tried a few different ways trying to get it to work.



I eventually found a picture of what the battery array should look like, and was able to make a wiring diagram for myself. I also found that the battery array should be 48v not 24v.

Intercepting IP packets using Netfilter hooks

Netfilter hooks are very useful when you wish to intercept, and modify IP packets inside a kernel module.

One great thing about using Netfilter hooks over say dev_add_pack() is that the bad L2 frames, and bad IP packets get filtered out earlier in the network stack so you only deal with good packets.

Now the whole point of the Netfilter hook is to customize what the kernel does with the packets received by the system so here is an example hook function. It simply does some checks to make sure the packet is a TCP packet, and then prints the IP, and port info if it is. You may also want to check for other traffic types such as UDP, or ICMP you should be able to figure that out.

Somewhere in your function you must tell the system what to do with the packet that is being processed. You have several options with what you do with the packet. They are.

• NF_DROP Discard the packet.
• NF_ACCEPT Keep the packet.
• NF_STOLEN Forget about the packet.
• NF_QUEUE Queue packet for userspace.
• NF_REPEAT Call this hook function again.

My example hook function.

static unsigned int
myhook_func(unsigned int hooknum,
struct sk_buff **skb,
const struct net_device *in,
const struct net_device *out,
int (*okfn)(struct sk_buff *)){
struct iphdr *iph = NULL;
struct tcphdr *tcph = NULL;

if ((skb != NULL) &&
((*skb)->pkt_type == PACKET_HOST) &&
((*skb)->protocol == htons(ETH_P_IP))){
iph = ip_hdr((*skb)); // access ip header.

/*
* yaple: Process only TCP segments.
*/

if ((iph->protocol == IPPROTO_TCP) ){
tcph = (struct tcphdr *)(skb_network_header((*skb)) + ip_hdrlen((*skb))); // access tcp header.
printk(KERN_ALERT "INFO: Source IP Address: %u.\n",iph->saddr);
printk(KERN_ALERT "INFO: Destination IP Address: %u.\n",iph->daddr);
printk(KERN_ALERT "INFO: Source Port: %u.\n",tcph->source);
printk(KERN_ALERT "INFO: Destination Port: %u.\n",tcph->dest);

}
}
return NF_ACCEPT; // Tells the system to accept the packet, and process the next one.
}



One thing to note about accessing the IP header, and TCP header fields. The data within most of them will be in a in a reversed order than the system is use to working with. To swap the "endianness" between host format, and network format use these function ntohs(), htons(), ntohs(), and ntohl().

So now that you have your hook function lets setup the hook. Its actually quite simple here is an example.

static struct nf_hook_ops my_hook = { // This is a Netfilter hook.
.hook = myhook_func, // Function that executes when a packet hits this hook.
.hooknum = NF_IP_FORWARD, // For routed traffic only.
.pf = PF_INET, // Only for IP packets.
.priority = NF_IP_PRI_FIRST, // My hook executes first.
};



You can hook into five different points as the packets pass through the system. Good documentation can be found here about each of them. http://www.iptables.org/documentation/HOWTO/netfilter-hacking-HOWTO-3.html

• NF_IP_PRE_ROUTING
• NF_IP_LOCAL_IN
• NF_IP_FORWARD
• NF_IP_POST_ROUTING
• NF_IP_LOCAL_OUT

Now in your module initialization function you need to register your hook.

nf_register_hook(&my_hook);



You also need to unregister your hook in your module exit function.

nf_unregister_hook(&my_hook);


Network Access Protection Unseen Issues

Anyone that has implemented NAP knows its not that difficult. Its much easier than it used to be to get IAS setup to support 802.1x network authentication, and it does health checks on the systems to make sure they are configured correctly. Now that sounds great, but its not so great when those health checks cause 50% of your desktops to spike 100% CPU usage for 3-7 minuets. That makes your users very unhappy.

Our users have never been happy with how slow the WAN is. Things got more vocal after we implemented NAP, and according to the network monitoring software the bandwidth was fine. Also every time we checked the PCs they would be running fine.

Well after a while someone finally noticed that the slowdown was happening right when the Offline Health Checks were running, and they always ran around the same time on these effected systems. So we watched one, and sure enough svchost.exe spiked to 100% for around 5 minuets.

Ok thats not helpful at all because tons of stuff runs under svchost.exe. So I download process explorer and launched it. We left it running on one of the effected PCs until it happened again about 60 minuets later.

So now we know what svchost.exe process was causing the issue, but sure enough this particular svchost.exe had all kinds of stuff running in it so we opened the "Threads" tab of this particular svchost.exe process, and waited for it again.

Finally identified the particular thread, and its associated file as wuaueng.dll. Well thats just the Windows Update service. So we checked its version 7.1.x.x, and compared it to a system that works fine, and it was 7.2.x.x oh look at that.

So I found this update http://support.microsoft.com/kb/949104, and after installing it on all the "slow" PCs they work fine.

Windows 7 x64 XP Mode

Well I finally had the need to try out Windows 7 feature called "XP Mode". I upgraded my laptop to Windows 7 x64 that I had been running Windows XP on for developing with Visual Studio 2005. Well VS2005 does not like Windows 7 so I installed XP Mode that allows you to run software in a Windows XP virtual machine. The cool part is that its totally seamless so it looks just like a normal app running on your system.