Workstation Patching

Overview

All supported desktop and laptop workstations supported by IT Services are configured to automatically install critical application and operating system (OS) updates. The exception to the rule are machines that cannot be automatically restarted because of their function (e.g., systems connected to a piece of lab equipment, Call Center workstations). These machines need to be identified so that they are not accidentally restarted. IT Services should continue to utilize BigFix for pushing out non-critical OS and application-specific patches.

Current State

  • Computers that are covered by SLA (Service Level Agreement) and were built/configured by IT Services have OS updates set to automatic (but the client has the ability to change those settings). They also have BigFix installed on them. Those that weren't built/configured by IT Services have various configurations related to critical OS updates; if they are self-registered through SNSR (Stanford Network Self-Registration), they have BigFix installed on them.
  • Computers that are covered by SLA in at least one area of IT Services have non-critical OS patches and application-specific patches pushed on a regular basis (usually weekly).
  • Once a month, Microsoft and Apple security patches are pushed site-wide to all end points with BigFix installed. All Microsoft and Apple security patches are added to a cumulative multi-action group each month to ensure all endpoints receive the current month's patches, as well as any prior patches.
  • Adobe Flash security updates are currently being pushed site-wide.
  • Adobe Reader patches are currently being tested, and smaller groups are being targeted at this time to ensure there is no negative client impact. The site administrators are working with Stanford's ISO (Information Security Office) and BigFix console operators to resolve any issues. Increasingly, Adobe applications are being targeted by malware and compromising machines. The ISO is adamant that IT Services makes a push to keep Adobe Reader updated and secured.
  • Computers not covered by SLA have unknown OS update settings and may or may not have non-critical and application patches applied on a regular basis, either remotely or manually. They may or may not have BigFix installed on them, so patch management through BigFix may or may not be occurring.

Workstation patching is done on a monthly basis. The schedule is as follows:

  • On the second Tuesday of the month, Microsoft patches are reviewed by BigFix site admins.
  • On Tuesday/Wednesday: Microsoft and Apple security patches are discussed with ISO.
  • On Wednesday: An email is sent out to bigfix-ops@lists.stanford.edu to inform the BigFix community which patches will be deployed site-wide. Big Fix console operators are encouraged to start testing and replying back to the list if there are any issues with the current month's patches.
  • On Thursday/Friday: Local console operators continue patching their test machines and, in some cases, their whole department's computers. The BigFix site admins watch the BigFix ops mailing list for any feedback from console operators and address any issues with the current month's patching.
  • On Friday at 3 p.m.: Microsoft/Apple security patches are deployed site-wide silently. All patches are deployed in the background with no user intervention. Patching an individual machine takes approximately 15 minutes, assuming the machine is on the Stanford Network and is unlocked and that the BigFix client port is accessible. The silent push is designed to not disrupt the end user if they are working on their machines. The assumption is that users will power down or reboot their machines for the weekend to complete the patch process.
  • On the third Tuesday of the month: x64 and foreign language patches are deployed site-wide. The vendor generally doesn’t have these updates available until this point in the month.
  • On the third Friday of the month: Endpoints still in a pending reboot state from the previous week's patching are reviewed and a "prompt to reboot" message is sent to users at 3 p.m. There is no forced reboot and the end user may dismiss the reboot notification for three hours, after which time the reboot prompt will stay in the foreground.

Vision

IT Services needs to be more proactive about patching client workstations and verifying that client machines are up-to-date in terms of patches.Although PCs currently require more effort than Macs do, the process(es) should work for both platforms. However, there cannot be a one-size-fits-all method for patching. Scheduling of patches and of restarts needs to take into account the client's business needs, so that the patches avoid having a negative impact on productivity. The priority for patching should be:

  1. New client machines should have automatic updates enabled.
  2. Regular monitoring of client machines to ensure they're up-to-date should begin.
  3. Client machines that do not not have the most current patches should be patched and then have automatic updates enabled.
  4. Client machines that need to have patches scheduled (instead of automatically installed because of the client's business needs) should be patched.

Currently BigFix supports Windows, Apple, and various flavors of UNIX; mobile clients are on the roadmaps but are not currently supported. While clients who pay IT Services to support their workstations get the direct benefits of workstation patching, the processes should be available to the greater university so that other users can benefit from them, instead of having to reinvent the wheel.

Roadmap

  • IT Services needs to discuss the importance of having workstations patched regularly with each of its clients, and the results of this discussion need to be included in each client SLA. The discussion should determine:
    • the best timeframe for doing patching.
    • whether workstations can be automatically restarted: If so, when; if not, then determine options to present to users for manually restarting.
    • any workstations that shouldn't be patched and/or automatically restarted and determine a process for getting these machines patched.
  • Those groups that are paid to do desktop support need to develop a process for ensuring that client workstations get patched, and then need to proactively follow that process (machines that can't have automatic updates and/or restarts should be included in this process).
  • Groups paid to perform desktop support need to come up with a solution or template for updating end points for clients. The technical process of patching a machine is not different from client to client; the variable is the time when a client indicates patching will cause the least amount of disruption. It would streamline the process if desktop support services has patch offerings or a set maintenance window with their departments. Working with departments to reach a common time should be a reasonable request and add some predictability of when patching occurs.
    • Sample schedule 1: Desktop Support should look into which updates are important to their end users with the exception of the site-wide pushes. Since site-wide pushes occur at a set date and time, console operators can schedule work within the third week of the month to deploy updates silently, if possible ,and let the third Friday reboot prompt take care of the any needed restarts.
    • Sample schedule 2: Client machines can be patched from 12 a.m. to 6 a.m. BigFix has the ability to schedule patching during the normal business day and have the actual patching execute at 12 a.m., with the option of only patching between 12 a.m. to 6 a.m., and have the action expire at 6 a.m. to ensure no further machines are patched.
  • IT Services CRC (Computer Resource Consulting) should be given the ability to manage all machines in IT Services and not be bound to which groups/departments are willing to pay for support. IT Services should be consumers of all and as many of the products or services that are offered to the university. IT Services employees are likely to have the most diverse set of technical skills of any group at Stanford, and would give the most critical feedback on how to improve its services.