What were the issues?
SCADA System Infrastructure Operational Review Checklist
Name
First Name
Last Name
Date
-
Month
-
Day
Year
Date Picker Icon
Time
1
2
3
4
5
6
7
8
9
10
11
12
:
Hour
00
10
20
30
40
50
Minutes
AM
PM
AM/PM Option
pfSense Appliance
CPU Utilzation
Memory Usage
CARP Status Master?
WAN1@114
LAN@1
WAN2@115
Are these spare parts available for install (not including test dev)?
Hirschman MAR1040 Router
Hirschman Eagle30 Firewall
Hirschman RS40
Hirschman RS30
Hirschman RS20
Sierra Wireless RV50x
SFP Module – FAST
SPF Module – MM
SPF Module – SM
SPF Module – SM Simplex
MM Patch Cord SC-LC
SM Patch Cord SC-LC
MM Patch Cord LC-LC
24 Volt DC Power Supply
1756-ControlNet Card
1756-Ethernet Card
1756-L73 Processor
1769-L32 Processor
Spare Configured NUC/Monitor
Other
Are the following interfaces up?
WAN1
LAN
VPN1
WAN2
SYNC
Network Traffic Greater than 1 Meg?
WAN1
LAN
VPN1
WAN2
Configuration change is last week?
Yes
No
Backup saved to owncloud?
Yes
No
Production Server Health
Are the following servers healthy? (Cooling, Power, Storage, CPU, Memory, System)
DMZ-S1
DMZ-S2
DMZ-S3
DMZ-S4
DMZ-S5
P001-S1
P001-S2
P002-S1
P002-S2
P003-S1
P003-S2
P004-S1
P004-S2
P005-S1
P005-S2
P006-S1
P006-S2
DMZ-VMWare
TestDev1
TestDev2
Unhealthy Server
None
DMZ-S1
DMZ-S2
DMZ-S3
DMZ-S4
DMZ-S5
P001-S1
P001-S2
P002-S1
P002-S2
P003-S1
P003-S2
P004-S1
P004-S2
P005-S1
P005-S2
P006-S1
P006-S2
DMZ-VMWare
TestDev1
TestDev2
Who was notified?
No one
SCADA Supervisor
SCADA Advisor
SCADA Manager
Unhealthy Server
none
pfsense primary
pfsense secondary
What were the issues?
Who was notified?
SCADA Advisor
SCADA Manager
Are the servers alert free?
DMZ-S1
DMZ-S2
DMZ-S3
DMZ-S4
DMZ-S5
P001-S1
P001-S2
P002-S1
P002-S2
P003-S1
P003-S2
P004-S1
P004-S2
P005-S1
P005-S2
P006-S1
P006-S2
DMZ-VMWare
TestDev1
TestDev2
Unhealthy Server
None
DMZ-S1
DMZ-S2
DMZ-S3
DMZ-S4
DMZ-S5
P001-S1
P001-S2
P002-S1
P002-S2
P003-S1
P003-S2
P004-S1
P004-S2
P005-S1
P005-S2
P006-S1
P006-S2
DMZ-VMWare
TestDev1
TestDev2
What were the issues?
Who was notified?
SCADA Advisor
SCADA Manager
All Storage space in green health
Yes
No
Orphaned snapshots
Yes
No
VM snapshots related to non-existent backups
Yes
No
Alarms?
Yes
No
Unhealthy Server
None
DMZ-S1
DMZ-S2
DMZ-S3
DMZ-S4
DMZ-S5
P001-S1
P001-S2
P002-S1
P002-S2
P003-S1
P003-S2
P004-S1
P004-S2
P005-S1
P005-S2
P006-S1
P006-S2
DMZ-VMWare
TestDev1
TestDev2
What were the issues?
Who was notified?
SCADA Advisor
SCADA Manager
Number of Unhandled Host Problems
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
0 is , 20 is
Monthly Availability Report Generated and Stored on Owncloud?
Yes
No
Number of Unhandled Service Problems
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
0 is , 20 is
What were the issues?
Who was notified?
SCADA Advisor
SCADA Manager
Are the following network rings healthy?
WAN
Lakeview WTP
Lorne Park WTP
GEBooth WWTP
Clarkson WWTP
Hanlan
Beckett Sproule
Streetsville
No security warnings for any switch/router?
WAN
Lakeview WTP
Lorne Park WTP
GEBooth WWTP
Clarkson WWTP
Transmission
North Peel Water
North Peel Waste
Remote Flow/Pressure
Remote Valve
Monthly - All ports locked down?
WAN
Lakeview WTP
Lorne Park WTP
GEBooth WWTP
Clarkson WWTP
Transmission
North Peel Water
North Peel Waste
Remote Flow/Pressure
Remote Valve
All switch backups saved to owncloud?
Yes
No
What were the issues?
Who was notified?
SCADA Advisor
SCADA Manager
Assetcenter
PLC Backup Health
Failures
Changes
Success
Overall
Lakeview
Lorne Park
Clarkson
GEBooth
North Peel Water
North Peel Waste
Manual Backup Completed?
Not required
Yes
No
What were the issues?
Who was notified?
SCADA Advisor
SCADA Manager
Change reports stored to Owncloud?
Yes
No
Mail Queue Empty?
Yes
No
Type a question
Less Than 0.5 MB
Less Than 2 MB
Less Than 10 MB
mh.*@scada.peelregion.ca
wh.*@scada.peelregion.ca
*.ignition@scada.peelregion.ca
All Others
Mail boxes emptied?
Yes
No
What were the issues?
Who was notified?
SCADA Advisor
SCADA Manager
Remote support login functional
Yes
No
What were the issues?
Who was notified?
SCADA Advisor
SCADA Manager
DMZ Terminal
Remote SCADA Login Functional?
Yes
No
North Water Ignition
North Waste Ignition
What were the issues?
Who was notified?
SCADA Advisor
SCADA Manager
Review the health of the Ingition Server
Yes
No
All Databases Connected
Store and Forward Empty
All EAM Agents Connected
Gateway Network Healthy
Scheduled Reports Functional
What were the issues?
Who was notified?
SCADA Advisor
SCADA Manager
FT Historian
C: Drive has 6GB free space?
Yes
No
E: Drive has 250 GB free space?
Yes
No
Interface Nodes Up and Available?
Yes
No
Lakeview
Lorne Park
GEBooth
Clarkson
Transmission
North Peel
Are there any stale tags?
Yes
No
What were the issues?
Who was notified?
SCADA Advisor
SCADA Manager
C: Drive has 6GB free space?
Yes
No
E: Drive has 250 GB free space?
Yes
No
What were the issues?
Who was notified?
SCADA Advisor
SCADA Manager
System Performance
Yes
No
CPU usage creep
Network creep
Disk usage creep
System Performance
Yes
No
CPU usage creep
Network creep
Disk usage creep
What is the health of the ignition servers?
P001
P002
P003
P004
P005
P006
P007
Redunancy Available
Yes
No
Yes
No
Yes
No
Yes
No
Yes
No
Yes
No
Yes
No
All Databases Connected
Yes
No
Yes
No
Yes
No
Yes
No
Yes
No
Yes
No
Yes
No
Store and Forward Empty
Yes
No
Yes
No
Yes
No
Yes
No
Yes
No
Yes
No
Yes
No
All EAM Agents Connected
Yes
No
Yes
No
Yes
No
Yes
No
Yes
No
Yes
No
Yes
No
Gateway Network Healthy
Yes
No
Yes
No
Yes
No
Yes
No
Yes
No
Yes
No
Yes
No
Scheduled Reports Functional
Yes
No
Yes
No
Yes
No
Yes
No
Yes
No
Yes
No
Yes
No
No CPU Usage Creep
Yes
No
Yes
No
Yes
No
Yes
No
Yes
No
Yes
No
Yes
No
No Network Usage Creep
Yes
No
Yes
No
Yes
No
Yes
No
Yes
No
Yes
No
Yes
No
Backup Stored on OwnCloud
Yes
No
Yes
No
Yes
No
Yes
No
Yes
No
Yes
No
Yes
No
What were the issues?
Who was notified?
SCADA Advisor
SCADA Manager
Monthly Failover Test Successful?
P001
P002
P003
P004
P005
P006
P007
Monthly Log Review - No Errors?
P001
P002
P003
P004
P005
P006
P007
What is the health of the wonderware system?
P001
P002
P003
P004
P006
LGH Storage Space Available
Yes
No
Yes
No
Yes
No
Yes
No
Yes
No
Alarm Manger is running
Yes
No
Yes
No
Yes
No
Yes
No
Yes
No
InTouch Application Running
Yes
No
Yes
No
Yes
No
Yes
No
Yes
No
Historian Storage Space >250 GB?
Yes
No
Yes
No
Yes
No
Yes
No
Yes
No
What were the issues?
Who was notified?
SCADA Advisor
SCADA Manager
Test/Dev VMs Match Versions for Productions VMs
Yes
No
Monthly - XOA VM cloned to P006-S2
Not required
Yes
No
What were the issues?
Who was notified?
SCADA Advisor
SCADA Manager
Clarkson Weekly Backup completed?
On Primary
On Secondary
Backup On Primary
Backup On Secondary
P001-AMS
Yes
No
Yes
No
Yes
No
Yes
No
P001-FTIN
Yes
No
Yes
No
Yes
No
Yes
No
P001-IOServer Liquids
Yes
No
Yes
No
Yes
No
Yes
No
P001-IOServer Solids
Yes
No
Yes
No
Yes
No
Yes
No
P001-IgnitionP
Yes
No
Yes
No
Yes
No
Yes
No
P001-TerminalP
Yes
No
Yes
No
Yes
No
Yes
No
P001-Waste Poll
Yes
No
Yes
No
Yes
No
Yes
No
P001-SQL
Yes
No
Yes
No
Yes
No
Yes
No
P001-Historian
Yes
No
Yes
No
Yes
No
Yes
No
P001-IgnitionS
Yes
No
Yes
No
Yes
No
Yes
No
P001-IOServer Liquids
Yes
No
Yes
No
Yes
No
Yes
No
P001-IOServer Solids
Yes
No
Yes
No
Yes
No
Yes
No
P001-TerminalS
Yes
No
Yes
No
Yes
No
Yes
No
What were the issues?
Who was notified?
SCADA Advisor
SCADA Manager
GEBooth Weekly Backup Completed?
On Primary
On Secondary
Backup On Primary
Backup On Secondary
P002-AMS
Yes
No
Yes
No
Yes
No
Yes
No
P002-FTIN
Yes
No
Yes
No
Yes
No
Yes
No
P002-IOServer Liquids
Yes
No
Yes
No
Yes
No
Yes
No
P002-IOServer Solids
Yes
No
Yes
No
Yes
No
Yes
No
P002-IgnitionP
Yes
No
Yes
No
Yes
No
Yes
No
P002-Terminal
Yes
No
Yes
No
Yes
No
Yes
No
P002-Waste Poll
Yes
No
Yes
No
Yes
No
Yes
No
P002-SQL
Yes
No
Yes
No
Yes
No
Yes
No
P002-Historian
Yes
No
Yes
No
Yes
No
Yes
No
P002-IgnitionS
Yes
No
Yes
No
Yes
No
Yes
No
P002-IOServer Liquids
Yes
No
Yes
No
Yes
No
Yes
No
P002-IOServer Solids
Yes
No
Yes
No
Yes
No
Yes
No
P002-Terminal
Yes
No
Yes
No
Yes
No
Yes
No
What were the issues?
Who was notified?
SCADA Advisor
SCADA Manager
Lorne Park Weekly Backup Completed?
On Primary
On Secondary
Backup On Primary
Backup On Secondary
P003-AMS
Yes
No
Yes
No
Yes
No
Yes
No
P003-FTIN
Yes
No
Yes
No
Yes
No
Yes
No
P003-IOServer Primary
Yes
No
Yes
No
Yes
No
Yes
No
P003-IOServer Secondary
Yes
No
Yes
No
Yes
No
Yes
No
P003-IgnitionP
Yes
No
Yes
No
Yes
No
Yes
No
P003-IgnitionS
Yes
No
Yes
No
Yes
No
Yes
No
P003-Terminal Primary
Yes
No
Yes
No
Yes
No
Yes
No
P003-Terminal Primary
Yes
No
Yes
No
Yes
No
Yes
No
P003-Water Poll
Yes
No
Yes
No
Yes
No
Yes
No
P003-SQL
Yes
No
Yes
No
Yes
No
Yes
No
P003-GE Insite
Yes
No
Yes
No
Yes
No
Yes
No
P003-Historian
Yes
No
Yes
No
Yes
No
Yes
No
What were the issues?
Who was notified?
SCADA Advisor
SCADA Manager
Lakeview Weekly Backup Completed?
On Primary
On Secondary
Backup On Primary
Backup On Secondary
P004-AMS
Yes
No
Yes
No
Yes
No
Yes
No
P004-FTIN
Yes
No
Yes
No
Yes
No
Yes
No
P004-IOServer Treatment
Yes
No
Yes
No
Yes
No
Yes
No
P004-IOServer Transmission
Yes
No
Yes
No
Yes
No
Yes
No
P004-IOServer Treatment
Yes
No
Yes
No
Yes
No
Yes
No
P004-IOServer Transmission
Yes
No
Yes
No
Yes
No
Yes
No
P004-IgnitionP
Yes
No
Yes
No
Yes
No
Yes
No
P004-IgnitionS
Yes
No
Yes
No
Yes
No
Yes
No
P004-Terminal Treatment
Yes
No
Yes
No
Yes
No
Yes
No
P004-Terminal Treatment
Yes
No
Yes
No
Yes
No
Yes
No
P004-Conv Poll
Yes
No
Yes
No
Yes
No
Yes
No
P004-SQL
Yes
No
Yes
No
Yes
No
Yes
No
P004-Historian
Yes
No
Yes
No
Yes
No
Yes
No
P004-EMS RDP
Yes
No
Yes
No
Yes
No
Yes
No
P004 GE Insite OBMI
Yes
No
Yes
No
Yes
No
Yes
No
P004 GE Insite OBMII
Yes
No
Yes
No
Yes
No
Yes
No
P004-Terminal Transmission
Yes
No
Yes
No
Yes
No
Yes
No
P004-Terminal Transmission
Yes
No
Yes
No
Yes
No
Yes
No
P004-OBMI Poll
Yes
No
Yes
No
Yes
No
Yes
No
P004-OBMII Poll
Yes
No
Yes
No
Yes
No
Yes
No
P004-Transmission Poll
Yes
No
Yes
No
Yes
No
Yes
No
What were the issues?
Who was notified?
SCADA Advisor
SCADA Manager
North Peel
On Primary
On Secondary
Backup On Primary
Backup On Secondary
P005-AMS
Yes
No
Yes
No
Yes
No
Yes
No
P005-IgnitionP
Yes
No
Yes
No
Yes
No
Yes
No
P005-SQL
Yes
No
Yes
No
Yes
No
Yes
No
P005 FTIN
Yes
No
Yes
No
Yes
No
Yes
No
P005-Historian
Yes
No
Yes
No
Yes
No
Yes
No
P005-Terminal
Yes
No
Yes
No
Yes
No
Yes
No
P006-FTIN
Yes
No
Yes
No
Yes
No
Yes
No
P05-IOServer Waste
Yes
No
Yes
No
Yes
No
Yes
No
P005-Waste Poll
Yes
No
Yes
No
Yes
No
Yes
No
What were the issues?
Who was notified?
SCADA Advisor
SCADA Manager
Tullamore Backup
On Primary
On Secondary
Backup On Primary
Backup On Secondary
P005-IgnitionS
Yes
No
Yes
No
Yes
No
Yes
No
P004-IOServer Transmission
Yes
No
Yes
No
Yes
No
Yes
No
P006-IgnitionP
Yes
No
Yes
No
Yes
No
Yes
No
P007-IgnitionP
Yes
No
Yes
No
Yes
No
Yes
No
What were the issues?
Who was notified?
SCADA Advisor
SCADA Manager
Monthly - Copy Random Backup VM from Each Production Area and check functionality in Test/Dev
Clarkson
GEBooth
Lorne Park
Lakeview
Transmission
North Peel
DMZ-S1 Backup
Completed Successfully?
DomainPrimary
Yes
No
ESXi-ENG
Yes
No
HiVision
Yes
No
Nagios
Yes
No
OwnCloud-8
Yes
No
Printer
Yes
No
RD Gateway
Yes
No
WSUS
Yes
No
FT-AssentAgent03
Yes
No
FT-AssentAgent01
Yes
No
What were the issues?
Who was notified?
SCADA Advisor
SCADA Manager
DMZ-S2 Backup
Completed Successfully
AMS Server
Yes
No
Cont-Ignition
Yes
No
DMZ-Terminal
Yes
No
FT Activation
Yes
No
FT AssetCentre Server
Yes
No
FT AssetCentre Agent 4
Yes
No
FT AssetCentre Agent 2
Yes
No
FT Directory
Yes
No
FT Studio5000
Yes
No
MailServer
Yes
No
Mantis
Yes
No
NodeRed
Yes
No
PIUFL
Yes
No
PIVision
Yes
No
ERIS
Yes
No
What were the issues?
Who was notified?
SCADA Advisor
SCADA Manager
DMZ-S3 Backup
Completed Successfully?
Datawarehouse
Yes
No
FT VantagePoint
Yes
No
FT Historian
Yes
No
What were the issues?
Who was notified?
SCADA Advisor
SCADA Manager
DMZ-S4 Backup
Completed Successfully?
EMS Primary
Yes
No
EMS Secondary
Yes
No
What were the issues?
Who was notified?
SCADA Advisor
SCADA Manager
MMAT No Errors from Alerts?
Clarkson
GEBooth
Lakeview
Lorne Park
Transmisison
North Peel
What were the issues?
Who was notified?
SCADA Advisor
SCADA Manager
Review HiVision Alerts
No Spanning Tree Changes
No Unauthorized Access
No Switch Configuration Changes
No Ports Flapping
What were the issues?
Who was notified?
SCADA Advisor
SCADA Manager
MotionEye Server Room Access Alerts
Clarkson - No Unidentifiable Entry
GEBooth - No Unidentifiable Entry
Lakeview Basement - No Unidentifiable Entry
Lakeview 2nd Floor - No Unidentifiable Entry
Lorne Park - No Unidentifiable Entry
Tullamore - No Unidentifiable Entry
What were the issues?
Who was notified?
SCADA Advisor
SCADA Manager
Submit
I hereby acknowledge that this report has been completed in full and with accuracy to the best of my abilities.
Should be Empty: