MSL Home > For Librarians > Montana Shared Catalog > Members > Incident Display
Incident Display
Summary Statistics
| Start Date/Time: | 4/22/2006 00:00 | |||||||||
| Todays Date/Time: | 5/21/2012 03:44 | |||||||||
| Total Elapsed Time: | 6Y, 0M, 29D, 3H, 44 Min. | 100.00% | ||||||||
| Total Up Time: | 6Y, 0M, 13D, 3H, 4 Min. | 99.28% | ||||||||
| Total Incident Down Time: | 16D, 40 Min. | 0.72% | ||||||||
Incident Detail
| Inc. # | Date Down | Time Down | Date Up | Time Up | Per Incident Down Time | Cause of Down Time | Scheduled? | Description | Symptoms | Comment |
| 56 | 12/3/2011 | 13:00 | 12/3/2011 | 15:55 | 2H, 55 Min. | State Network | No | State network outage in the Helena area prevented access to the system. | MSC library staff and patrons unable to connect to the MSC integrated library system. | ITSD not forthcoming with cause of the outage. | 55 | 10/8/2011 | 08:45 | 10/8/2011 | 14:10 | 5H, 25 Min. | MSL Network | No | Scheduled change over to new SAN was not supposed to affect the MSC Production server. However, during the cabling switch-over, it was discovered that the MSC production server was not properly re-connected to the network. | Production server (Workflows and OPAC) not accessible to the users until re-connected to the network. Directors Station and Test Server down due to change to new SAN and awaiting reconfiguration. | Directors Station and Test Server back online 1 week later after being re-connected to the old SAN. Future re-configuration will be necessary to connect them to the new SAN. | 54 | 9/10/2011 | 08:15 | 9/10/2011 | 13:25 | 5H, 10 Min. | MSC PRD Server | No | A scheduled, 4-hour building power outage resulted in the the MSC servers to go offline after UPS lost power. The state network also went down when the ITSD switch room UPS lost power. Unbeknownst to us, neither system was connected to the buildings emergency generator. | All MSC systems were down until power and network were restored and servers re-booted. | Lesson learned. MSC servers and UPS are now connected to the building's emergency generator. Work request has been submitted to ITSD to do the same for their network switch room. | 53 | 2/22/2011 | 21:30 | 2/23/2011 | 02:15 | 4H, 45 Min. | MSC PRD Server | Yes | Migrate Symphony ILS to new production server | All systems down until migration complete | Migration completed. | 52 | 1/1/2011 | 01:00 | 1/1/2011 | 09:00 | 8H, 0 Min. | MSC PRD Server | No | Oracle database not restarting | Failed Communication error | Nightly HIR failed to halt Oracle database completely thus system restart failure until database stopped and restarted manually. | 51 | 12/23/2010 | 01:00 | 12/23/2010 | 09:00 | 8H, 0 Min. | MSC PRD Server | No | Oracle database no restarting | Failed Communication error | Nightly HIR failed to halt Oracle database completely thus system restart failure until database stopped and restarted manually. | 50 | 12/10/2010 | 02:00 | 12/10/2010 | 08:15 | 6H, 15 Min. | N/A | No | Oracle database not restarting | Failed Communication error. | Nightly routine to copy-over production to test resulted in faulty Oracle shutdown. | 49 | 12/6/2010 | 08:15 | 12/6/2010 | 13:00 | 4H, 45 Min. | MSC PRD Server | No | Oracle database crash | Failed communication error when attempting to access Workflows and online catalog. | Incorrect permissions settings after the 3.3.1 upgrade caused the crash. | 48 | 12/5/2010 | 09:15 | 12/5/2010 | 12:30 | 3H, 15 Min. | MSC PRD Server | Yes | Upgrade production server to Symphony 3.3.1 pc 1 | Production server down during upgrade | Upgrade appears to have completed successfully | 47 | 11/2/2010 | 08:45 | 11/2/2010 | 15:05 | 6H, 20 Min. | MSC PRD Server | Yes | AIX upgrade from v5.2 to v5.3 | Production system Workflows and eLibrary) down | Upgrade completed successfully. | 46 | 5/31/2010 | 11:00 | 5/31/2010 | 12:00 | 1H, 0 Min. | MSL Network | Yes | System down while changing server IP addresses in MSL in switch to state VLAN | All client and web access to MSC servers not available during down time. | IP addresses changed successfully and all systems back online. | 45 | 4/24/2010 | 19:10 | 4/25/2010 | 13:30 | 18H, 20 Min. | MSC PRD Server | Yes | Authority records maintenance and catalog re-indexing | Workflows and e-Library down. | Annual authority maintenance completed successfully | 44 | 4/17/2010 | 08:45 | 4/17/2010 | 14:30 | 5H, 45 Min. | MSC PRD Server | No | access permissions on a system file prevented users from using the Workflows client. | Unable to login to Workflows Client | Workflows access enabled after adjusting access permissions on the system file. | 43 | 10/18/2009 | 08:00 | 10/18/2009 | 15:00 | 7H, 0 Min. | State Network | No | State network down for scheduled maintenance originally scheduled for an hour but problems prevented them from bringing the system back online. MSC hardware and software was not affected. | Users had unreliable and sporadic access the MSC servers. | Lingering effects of the outage persisted into Monday and definitively resolved Monday afternoon. | 42 | 8/2/2009 | 09:00 | 8/2/2009 | 12:30 | 3H, 30 Min. | MSL Network | Yes | Network upgrade work in the MSL server room. | MSC production, development and Directors Station servers not accessible via the internet. | Upgrade work completed successfully. | 41 | 7/8/2009 | 08:30 | 7/8/2009 | 09:15 | 0H, 45 Min. | MSC PRD Server | Yes | Symphony 3.2.1 patch cluster 5 update. | Production server down. | Update completed successfully. | 40 | 5/16/2009 | 20:10 | 5/17/2009 | 10:00 | 13H, 50 Min. | MSC PRD Server | Yes | Reload and reindex of changed bib records from authority maintenace | Workflows and e-Library down | changed bibs and reindexing completed successfully. | 39 | 5/2/2009 | 19:40 | 5/3/2009 | 14:30 | 18H, 50 Min. | MSC PRD Server | Yes | Authority records maintenance and catalog re-indexing | Workflows and e-Library down. | Updated authority records and changed bib records re-indexed successfully. | 38 | 5/1/2009 | 20:15 | 5/2/2009 | 07:45 | 11H, 30 Min. | MSC PRD Server | Yes | Authority records update and load | Workflows and e-Library down | Authority records updated and loaded successfully. | 37 | 4/23/2009 | 02:00 | 4/23/2009 | 07:30 | 5H, 30 Min. | MSC PRD Server | No | Fix indexing problem and installing Patch Cluster 4 | Production server down. | Indexing problem fixed and patch cluster 4 successfully installed. | 36 | 3/31/2009 | 08:45 | 3/31/2009 | 09:15 | 0H, 30 Min. | MSC PRD Server | Yes | Symphony 3.2.1 Patch Cluster 3 update | Production system down | Update completed successfully | 35 | 1/31/2009 | 09:11 | 1/31/2009 | 09:29 | 0H, 18 Min. | MSC PRD Server | Yes | Production server reboot | Production system down | Server rebooted as suggested by SirsiDynix client care. | 34 | 1/19/2009 | 08:55 | 1/19/2009 | 13:35 | 4H, 40 Min. | MSC PRD Server | Yes | Scheduled software upgrade to Symphony 3.2.1 | Production server down during upgrade | Upgrade completed successfully | 33 | 1/12/2009 | 08:30 | 1/12/2009 | 09:45 | 1H, 15 Min. | MSC PRD Server | Yes | Scheduled Oracle database upgrade | Production server down during the upgrade | Upgrade completed successfully | 32 | 10/21/2008 | 10:45 | 10/21/2008 | 12:30 | 1H, 45 Min. | State Network | No | ITSD experienced a network outage. This outage was caused when their distribution switch’s processors experienced extreme high utilization. ITSD is still investigating the root cause of this issue | Workflows and iBlink not available. MSC staff unable to send/receive email. | 31 | 9/8/2008 | 18:00 | 9/8/2008 | 20:30 | 2H, 30 Min. | MSC PRD Server | No | Large reports file in /tmp directory reduced available disk space until no longer useable. Service restored when file was deleted and performed HIR. | Server response times slowed and no access to reports section of Workflows. Search errors in iBlink. | 30 | 8/17/2008 | 06:00 | 8/17/2008 | 16:00 | 10H, 0 Min. | State Network | No | Originally an ITSD scheduled 5 minute network outage for maintenance resulted in total network outage for most of the day. | Workflows and iBlink not available | 29 | 7/11/2008 | 22:00 | 7/12/2008 | 05:00 | 7H, 0 Min. | MSC PRD Server | Yes | Reindexing required after removal of Ronan holdlings from the catalog. | Workflows and iBlink not available. | 28 | 6/26/2008 | 07:25 | 6/26/2008 | 07:45 | 0H, 20 Min. | MSC PRD Server | Yes | GL 3.1 patch update | Workflows and iBlink not available | This was an update to fix a big introduced in patch 5e. | 27 | 5/18/2008 | 12:30 | 5/18/2008 | 14:30 | 2H, 0 Min. | MSC PRD Server | No | Large reports file in /tmp directory reduced available to disk space until no longer useable. Service restored when file was deleted and performed HIR. | Server response times slowed and no access to reports section of Workflows. Search errors in iBlink. | 26 | 4/29/2008 | 12:30 | 4/29/2008 | 15:15 | 2H, 45 Min. | MSC PRD Server | No | Production server powered off. Cause unknown. Had to do a cold reboot of server and then bring back up Workflows and the Apache web server. | Workflows and iBlink not available | Suspect someone unplugged the server or the servers power strip and then plugged it back in. | 25 | 4/25/2008 | 21:00 | 4/27/2008 | 16:00 | 1D, 19H, 0 Min. | MSC PRD Server | Yes | Scheduled Down Time for yearly load of Authority records and changed Bib records | Access to Workflows and production iBistro unavailable. Access to test instance of iBistro available for patron searching. | 24 | 4/12/2008 | 08:06 | 4/12/2008 | 08:20 | 0H, 14 Min. | MSC PRD Server | Yes | GL 3.1 Patch 5e upgrade | Workflows and iBlink not available during upgrade. | 23 | 4/4/2008 | 13:15 | 4/4/2008 | 14:30 | 1H, 15 Min. | MSC PRD Server | No | Production server slowed to a halt. Required server reboot to restore system. Cause unknown. | Workflows and iBlink not available. | 22 | 9/13/2007 | 08:07 | 9/13/2007 | 08:25 | 0H, 18 Min. | MSC PRD Server | Yes | GL 3.1 Patch 5d upgrade | Workflows and iBlink not available during upgrade. | 21 | 8/9/2007 | 08:30 | 8/9/2007 | 09:15 | 0H, 45 Min. | MSC PRD Server | Yes | Symphony 3.2.1 Patch Cluster 5 update. | Production system down. | Update completed successfully. | 20 | 7/19/2007 | 08:00 | 7/19/2007 | 15:00 | 7H, 0 Min. | MSC PRD Server | Yes | GL3.1 Upgrade | 19 | 7/10/2007 | 08:30 | 7/11/2007 | 15:20 | 1D, 6H, 50 Min. | MSC PRD Server | Yes | Oracle Upgrade. | Workflows and iBistro down for this scheduled upgrade. | 18 | 5/15/2007 | 12:00 | 5/15/2007 | 14:00 | 2H, 0 Min. | MSC PRD Server | Yes | This was a scheduled reboot of the server. Two administrative reports were beginning to show a slowdown in completing weekly or daily and a reboot may help. In the course of rebooting, there were unexpected difficulties when users attempted to connect following the reboot. Issue was resolved by a second reboot of the server. | System down: no access to Workflows or to iBistro. | 17 | 4/21/2007 | 07:45 | 4/21/2007 | 19:55 | 12H, 10 Min. | Power | Yes | Scheduled downtime at entire Montana State Library building, to install new electrical panel in preparation to connect new backup generator. | No power during installation time at MSL: no access to Workflows client or to iBistro, both production and development. | 16 | 12/17/2006 | 11:00 | 12/18/2006 | 10:30 | 23H, 30 Min. | MSC PRD Server | Yes | Upgrade to GL3.0 platform | Workflows and iBistro unavailable. Standalone/Offline and test iBistro used during scheduled down time | 15 | 12/15/2006 | 15:45 | 12/15/2006 | 17:30 | 1H, 45 Min. | MSC PRD Server | No | MSL Storage Area Network failed. MSC Production and MSC Development servers lost connection with the SAN | Workflows and iBistro unavailable | 14 | 11/4/2006 | 10:15 | 11/4/2006 | 12:15 | 2H, 0 Min. | MSC PRD Server | No | Power outage in Helena. State down. Back up power to MSC server eventually failed. Had to do a cold reboot of server and then bring back up Workflows and the Apache web server. | MSC libraries could not connect to WF or iBistro. | 13 | 10/3/2006 | 09:50 | 10/3/2006 | 11:40 | 1H, 50 Min. | N/A | No | SIP2 server generating "ghost sessions" since the 10/01 AIX upgrade. These sessions numbered in the thousands and caused Oracle errors and brought down the system. Decision was made to halt the SIP2 server until Sirsi can deliver a fix for our Unicorn version. | No access to Workflows or to iBlink. | 12 | 10/1/2006 | 09:15 | 10/1/2006 | 11:00 | 1H, 45 Min. | MSC PRD Server | Yes | Upgrade to production server's AIX operating system, to AIX 5.2. Server and system down during upgrade. | Workflows and iBlink not available during upgrade. | 11 | 9/13/2006 | 15:40 | 9/13/2006 | 15:50 | 0H, 10 Min. | State Network | No | State Internet unavailable | No network connection | Quickly resolved by State | 10 | 8/28/2006 | 09:00 | 8/28/2006 | 11:25 | 2H, 25 Min. | State Network | No | State of Montana experienced network problems that appeared to primarily internal network issues. | Some agency libraries lost connectivity to the system. The MLN site was unavailable both inside and outside of the network. | 9 | 8/24/2006 | 08:45 | 8/24/2006 | 09:15 | 0H, 30 Min. | MSC PRD Server | Yes | Due to extreme slowness reported by multiple MSC libraries over 48 hours, a reboot of the server was done. The reboot appeared to resolve the issue. | System unavailable during the reboot process. | 8 | 8/17/2006 | 17:00 | 8/17/2006 | 17:40 | 0H, 40 Min. | State Network | No | Due to State power supply problems, the State Network was brought down incrementally to prevent a hard shutdown. Email was initially affected around 3:30 p.m. and network access to the system was lost at 5:00. Network access was restored around 5:40 p.m. | No Internet access | The phone tree was initiated because email was unavailable. | 7 | 8/16/2006 | 06:30 | 8/16/2006 | 06:45 | 0H, 15 Min. | MSL Network | Yes | MSL Network switch maintenance | Except for minimal network downtime during early morning hours which caused some regularly running reports to be delayed, this incident did not disrupt the system. | 6 | 7/27/2006 | 13:00 | 7/27/2006 | 15:10 | 2H, 10 Min. | State Network | No | Power failure in Helena. Affected entire state government campus and parts of Helena. Cause of power failure: squirrel. | Interrupted all access to state servers. No access to MSC servers or to MSC email. | 5 | 7/12/2006 | 07:15 | 7/12/2006 | 08:15 | 1H, 0 Min. | MSC PRD Server | Yes | Mike Price upgrading AIX operating system. First step in process | 4 | 6/9/2006 | 20:00 | 6/11/2006 | 15:00 | 1D, 19H, 0 Min. | MSC PRD Server | Yes | Scheduled Down Time for yearly load of Authority records and changed Bib records | Access to Workflows and production iBistro unavailable. Access to test instance of iBistro available for patron searching. | 3 | 5/28/2006 | 02:00 | 5/28/2006 | 09:00 | 7H, 0 Min. | State Network | No | Power outage within state's network. Part of state network access unavailable | No access to MSC servers, inside and outside of state's network. | 2 | 5/22/2006 | 02:00 | 5/23/2006 | 02:00 | 1D, 0H, 0 Min. | State Network | No | State of Montana network failure. | No access to MSC servers, inside and outside of state's network. | 1 | 12/9/2001 | 08:30 | 12/9/2001 | 09:45 | 1H, 15 Min. | N/A | Yes | Production system, Oracle database upgrade. | Production system down | Upgrade completed normally. |