Linux System Administration Service Description

University IT will:

• Respond to monitoring alerts and client-reported problems.  During non-business hours, support will be provided when either the hardware, operating system, or infrastructure software is unavailable or the ability to use these resources is severely degraded.   Please note, should after-hours support be required due to a change that was performed by the application owner without proper planning or notification, emergency support fees may apply.
• Troubleshoot and resolve system-related problems
• Monitor vendor resources for any required operating system patches or upgrades
• Monitor vendor resources for any required hardware upgrades
• Monitor for file system intrusion (intrusion detection)
• Manage hardware warranties
• Coordinate hardware upgrades with hardware support vendors as needed
• Monitor security advisories for operating system and infrastructure software, and take appropriate actions to safeguard resources
• Implement security patches as needed
• System account management
• Root e-mail review
• Per client request, restore files that have been backed up.
• Document and submit change management requests for proper approval as required.  Change Management is required for any change that may impact end-users.
• Install security patches, upgrade software packages, and update system configuration to meet UNIX group best practices (patching every four months, or more frequently as needed).
• Firmware upgrades as required
• Maintain operating system and supported software documentation
• System-level housekeeping activities to ensure systems are operating at optimal levels
• Backup management
• Requesting firewall configuration

Support requests that fall outside of normal operations will be reviewed on a case-by-case basis and may incur additional costs.  Only vendor-supported versions of operating system or application software will be maintained.  Clients should work in conjunction with the systems administration team to plan for upgrades when vendors announce that supported versions of software are being removed from support services. 

It is expected that all application and system changes that may impact services will adhere to standard Change Management policies as specified in the Service Agreement.  Clients should submit any non-impacting or non-urgent system requests via HelpSU.   Exceptions to address emergency situations should be handled via notification of the IT Operations Center.  HelpSU and notification procedures are specified in the SLA. 

Non-urgent and low severity HelpSU tickets will be assigned within 8 business hours. The client can expect an acknowledgement via email that states either:

• the request has been assigned and additional information is required in order to complete the request
• the request has been assigned and a target timeframe as to when the request might be completed
• the request has been assigned and completed

Completed requests will be indicated via a ticket status of “resolved”.  Routine support requests are typically resolved within 24 hours of assignment.  Clients will receive email notification when a request has been resolved.

All systems managed by Technology Consulting are built and patched from automated build systems maintained by Technology Consulting.  Existing systems that enter Unix systems coverage will be rebuilt from the Operating System level to ensure consistency and meet Technology Consulting best practices.  Technology Consulting recommends that systems be rebuilt to migrate between major versions of any supported Operating System.  Upgrades in-place are possible, but require an extended outage window (over 24 hours) to deal with issues that may arise as part of the operating system upgrade. 

Linux systems managed by Technology Consulting are managed by a centralized server automation infrastructure, called “puppet”.  Systems Administrators create a complete model of the system in puppet that includes package lists, configuration files, and user management.  The state of a running system is compared to this model every half hour by the puppet daemon.  Any configuration that doesn’t match this model will be reverted automatically.  This means that changes to all files managed by Puppet will need to be coordinated through Technology Consulting.

ssh and remctl will be set up on all systems.  All remote connections will use Kerberos for authentication.  Root access via ssh is disabled for security reasons.  Where superuser access by a client is required, sudo with password is the preferred method, with all commands issued as root (or via sudo) being logged on the local system.  Where group accounts are required, the only supported mechanism for authentication is using a Kerberos .k5login file with user accounts listed individually for revocation and audit purposes.  Any use of ssh host keys or other public key authentication must be included as a variance to the Service Description.  Host-based access or shared passwords are not recommended and must also listed as a variance.

Technology Consulting will install any package that is part of the Operating System distribution.  Alternate versions, if they can be easily incorporated into the package management infrastructure, may also be installed, but a variance is required.  Installation of any package that isn’t part of the Operating System distribution is a variance, and additional charges may be required for maintenance of that package.  Linux systems are set up in accordance with the Linux File Hierarchy Standard.

Technology Consulting will install and configure application software per special arrangement with client. 

AFS is installed on all systems managed by Technology Consulting.  Home directories for all users are stored in AFS by default. 

University IT systems group manages various types of logs via log rotation.  System-level log files are collected and filtered to notify Technology Consulting.  Application-level log files are rotated locally for applications installed by Technology Consulting, and client logs may be rotated or filtered by arrangement.  Time and materials fees may apply for any custom development.

Technology Consulting will manage all accounts present on the system.  However, any changes to accounts from the client’s organization must be coordinated by the client with the UNIX group.

Technology Consulting uses the University IT Change Management Process for all changes that may affect the service provided to the client or the University community. Tools and configuration used internally by UNIX Systems staff for administrative or maintenance actions on a system may be installed and updated outside of the change management process provided that those tools are only used interactively by UNIX Systems staff and not part of any automated or continuous management or monitoring of the system, or are unrelated to any services running on the system and not intended for use by clients, only by UNIX Systems staff.

Examples include:

• remctl backend for forcing a Puppet run
• Configuration for remctl ACL files to install for that backend
• root bash shell configuration

but not:

• remctl backend for Nagios monitoring
• Script used to generate new iptables rules after rule updates
• Apache management scripts run by the client

Periodically, Technology Consulting will update software on the system, usually as part of a new minor release of the operating system.  These changes will be coordinate with the clients via the change management.

Periodically, Technology Consulting will release new systems configuration via Puppet.  In some cases, these changes need to be applied to all systems managed by the UNIX systems group simultaneously. 

For Standard and Non-Standard clients, work on the system that is controlled under the change management process can be confined to maintenance windows by arrangement with the client.  Technology Consulting maintenance windows are Tuesday 5-8 PM, Thursday mornings 5-7 AM, and Weekend mornings 5-8 AM.  Any work performed during this time will be coordinated via the change management process.