Skip to content

Tips and Tricks for monitoring Log Files with System Center Operations Manager

Last week i have been working on monitoring a  business critical application. This application does not write events to the Windows event log but only in it’s own log files. With System Center Operations Manager you have several options on monitoring these log files. In this post some tips and tricks on how to do log file monitoring with System Center Operations Manager. There are some important considerations that must be taken into account when monitoring log files with System Center Operations Manager. The next lines are taken from the Microsoft support site: (http://support2.microsoft.com/kb/2691973)

When monitoring a log file, Operations Manager remembers the last line read within the file (a ‘high water mark’). It will not re-read data before this point unless the file is deleted and recreated, or renamed and recreated, which will reset the high water mark. If a logfile is deleted and recreated with the same name within the same minute, the high water mark will not be reset, and log entries will be ignored until the high water mark is exceeded. An implication of this is that log files that are cleared periodically without being renamed and recreated, or deleted and recreated, will not have entries in them processed until the high water mark from before the log is cleared is exceeded. Operations Manager cannot monitor ‘circular log files’ (i.e. log files that get to a certain size or line count, then start writing the newest entries at the beginning of the log) for the same reason. The log file must be deleted or renamed and then recreated, or the application configured to write to a new log once the current log is filled. Example:

  • 100 lines are written to logfile.txt
  • logfile.txt is cleared of all entries
  • log entries are written to logfile.txt (position 0 of the file)
  • None of the new entries will be processed until line 101 is written

Each line of a log file must end with a new line (0x0A0x0A hex sequence) before it will be read and processed by Operations Manager. If a rule or monitor is configured to match a pattern for log file names (e.g. using the ? or * wildcard characters), it is important that only ONE log that matches the pattern is written. If multiple logs that match the pattern are being written to, the high water mark is reset to the beginning of the file with each write to a different file. The result is that all previous log entries will be reprocessed. Example:

  • The log file name pattern is generic_csv??.txt
  • The current log is generic_csv01.txt and writes happen to this log.
  • A new log, generic_csv02.txt, is created. Writes occur to this log.
  • When the next line is written to generic_csv01.txt, the Operations Manager will read from the beginning of generic_csv.txt, not from the last point that was read from generic_csv01.txt. Lines previously processed will be processed again, possibly resulting in alerts or other actions (depending on the rule configuration).

Another consideration is that when the log file you configured does not exist you won’t get an alert. When monitoring log files you have again the choice to use a rule or monitor. If you want it to affect the health status of your object you use a monitor. In all other cases a rule. When using a monitor, you have preferably have a healthy log entry so you can automaticly have the object turn healthy again. Read more…

Exchange 2013 Management Pack Updated

Yesterday Microsoft released a new version for the Exhange 2013 management pack. The new version is 15.0.652.19.
Some cool and informational dashboards and extra reports have been added:

Exchange2013 Dashboards

More information on the management pack can be found on the Operations Manager Engineering Blog here.

Download link for the management pack: http://www.microsoft.com/en-us/download/details.aspx?id=39039

Let’s make it manageable!

System Center Operations Manager Updated Management Packs

The Operations Manager product team recently updated some management packs:

Windows Server DNS version: 7.1.10259.0
Windows Server DHCP version: 6.0.7230.0
MSMQ 5.0, 6.0, 6.3 version: 7.1.10109.0
Windows Server Cluster version: 6.0.7230.0
Active Directory Domain Services version: 6.0.8293.0
Windows Server Core Operating System version: 6.0.7230.0

More information on the changes can be found in the original post on the System Center: Operations Manager Engineering blog.

Tools for those who work with Operations Manager

As an Operations Manager specialist I use quite some tools for my daily tasks. In this post I would like to share some of those tools with a small motivation.

Tool

Description

Operations Manager versions

Operations Manager 2012 Sizing Helper Tool

The OpsMgr 2012 Sizing Helper is an interactive document designed to assist you with planning & sizing deployments of System Center 2012 Operations Manager. It helps you plan the correct amount of infrastructure needed for a new OpsMgr 2012 deployment, removing the uncertainties in making IT hardware purchases and optimizes cost. A typical recommendation will include minimum hardware specification for each server role, topology diagram and storage requirement.

2012/2012 R2

PowerShell ISE

With OpsMgr PowerShell is also a must. The Windows PowerShell Integrated Scripting Environment (ISE) is a host application for Windows PowerShell. In Windows PowerShell ISE, you can run commands and write, test, and debug scripts in a single Windows-based graphic user interface with multiline editing, tab completion, syntax coloring, selective execution, context-sensitive help, and support for right-to-left languages. You can use menu items and keyboard shortcuts to perform many of the same tasks that you would perform in the Windows PowerShell console. For example, when you debug a script in the Windows PowerShell ISE, to set a line breakpoint in a script, right-click the line of code, and then click Toggle Breakpoint.

Not OpsMgr specific

Notepad++

XML editing is something that I do on a regular basis. This because XML is the language the management packs are built in. If I do some small adjustments I like to do this in a pleasant way. Notepad++ comes in very handy..

Not OpsMgr specific

MPViewer 2.2.3

This tool made by Boris Yanushpolsky is one I almost use every day. It gives a very good view on what’s is in a management pack. It support .MP,.MPS and XML files. A handy feature is that the MP content can be export into readable content.

2012/2012 R2   (The 2007/2007R2 version 1.7 can be downloaded here.)

OverrideExplorer 3.7

Also a tool made by Boris Yanushpolsky. This tool gives a quick overview on the configured overrides in the management group. You can see the target and destination management pack. You even have the option to move the overrides to another management pack. There has been some errors reported on the moving option so be carefull with this.

2012/2012 R2   (The 2007/2007R2 version can be downloaded here.)

Silect MP Authoring Tool

This is a free tool for building simple management packs.

2012/2012 R2

System Center Operations Manager 2007 R2 Authoring Resource Kit

There are many tools available in this resource kit. Please visit the site for more information on this: http://www.microsoft.com/en-us/download/details.aspx?id=18222

2007/2007 R2

Visio 2010 Add-in for Operations Manager 2012

With this Visio add-in you can create simple but effective dashboards. Use existing Visio drawing and attach them to OpsMgr objects to display the health state. Unfortunately only for available for Visio 2010

2012/2012 R2

MPSeal & SN.exe

MP seal is a tool for sealing management packs. SN.exe is a tool for creating a key that is needed for sealing the management pack. Most authoring tools can seal mp’s too.   A good example on how to use these two tools is written by Jonathan Almquist.

2012/2012 R2

Dwdatarp.exe

Data Warehouse Data Retention Policy. With this tool you can change the retention time for the Data Warehouse data in an easy way. Just start the tool form a command line and make the required changes. More info on how to use the tool can be found on the Operations Manager Team Blog site.

2007/2007 R2/2012/2012 R2

OpsMgr 2012 – Useful SQL Queries

A post with some Operations Manager 2012 SQL queries i use from time to time:

Get manually installed agents:

select bme.DisplayName from MT_HealthService mths
INNER JOIN BaseManagedEntity bme on bme.BaseManagedEntityId = mths.BaseManagedEntityId
where IsManuallyInstalled = 1

Make all agents remotely manageable again:

UPDATE MT_HealthService
SET IsManuallyInstalled=0
WHERE IsManuallyInstalled=1

Get Management packs waiting for synchronisation with the Data Warehouse:

SELECT
ManagementPackId, MPFriendlyName,MPName, mp.MPVersionDependentId, MPLastModified, MPKeyToken, ContentReadable
FROM ManagementPack mp
WHERE MPVersionDependentId

NOT IN
(SELECT mpv.ManagementPackVersionDependentGuid
FROM OperationsMAnagerDW.dbo.ManagementPackVersion mpv
JOIN OperationsMAnagerDW.dbo.ManagementGroupManagementPackVersion mgmpv
ON (mpv.ManagementPackVersionRowId = mgmpv.ManagementPackVersionRowId)
WHERE (mgmpv.LatestVersionInd > 0))

AND NOT EXISTS
(SELECT * FROM ManagementPackReferences mpr
JOIN ManagementPack mpv
ON (mpr.ManagementPackIdSource = mpv.ManagementPackId)
WHERE (mpr.ManagementPackIdReffedBy = mp.ManagementPackId)
AND (mpv.MPVersionDependentId NOT IN
(SELECT mpv.ManagementPackVersionDependentGuid
FROM OperationsMAnagerDW.dbo.ManagementPackVersion mpv
JOIN OperationsMAnagerDW.dbo.ManagementGroupManagementPackVersion mgmpv
ON (mpv.ManagementPackVersionRowId = mgmpv.ManagementPackVersionRowId)
WHERE (mgmpv.LatestVersionInd > 0))))

OpsMgr Agents fail heartbeat on 2012 R2 Domain Controllers

A quick post,

Just ran into a problem where i found agent on Server 2012 R2 failing to heartbeat. When checking the Microsoft Monitoring Service was running correctly. Even after removing the health cache the problem keeps occuring.
I ran into the an article from Kevin Holman who has done some investigation on this.

The solution is installing the Update Rollup for Server 2012 R2: http://support.microsoft.com/kb/2919394

Enjoy!

 

Operations Manager 2012 – Useful PowerShell Commands and Scripts

A collection for some PowerShell commands and scripts i use(d) on a regular basis:

Install Prerequisites 2012/2012r2 management server and/or console

$dwnld = 'C:\SCOM2012R2Prereqs'
if (!(Test-Path -path $dwnld))
 {
 New-Item $dwnld -type directory
 }
$object = New-Object Net.WebClient
$RPTurl = 'http://download.microsoft.com/download/F/B/7/FB728406-A1EE-4AB5-9C56-74EB8BDDF2FF/ReportViewer.msi'
$object.DownloadFile($RPTurl, "$dwnld\ReportViewer.msi")
 
$RPTurl = 'http://go.microsoft.com/fwlink/?LinkID=239644&clcid=0x409'
$object.DownloadFile($RPTurl, "$dwnld\SQLSysClrTypes.msi")
 
Start-Process -FilePath "$dwnld\SQLSysClrTypes.msi" -ArgumentList '/q' -Wait
Start-Process -FilePath "$dwnld\ReportViewer.msi" -ArgumentList '/q' -Wait

Install Prerequisites 2012/2012r2 web console

This script assumes the source files for the .NET installation are on S:\Software\Tools\sxs.

Import-Module ServerManager
Add-WindowsFeature NET-Framework-Core,Web-Static-Content,Web-Default-Doc,Web-Dir-Browsing,Web-Http-Errors,Web-Http-Logging,Web-Request-Monitor,Web-Filtering,Web-Stat-Compression,Web-Mgmt-Console,Web-Metabase,Web-Asp-Net,Web-Windows-Auth,Web-ASP,Web-CGI -source S:\Software\Tools\sxs -Restart
Add-WindowsFeature Web-Asp-Net45,AS-HTTP-Activation

Back-up Unsealed Management packs

Import-Module Operationsmanager

Get-SCOMManagementPack | where {$_.Sealed -eq $false} | export-SCOMmanagementpack -path C:\MPBackups

Reset Monitors for closed alerts

# $Alertname=@();
# $State=@();
# $Displayname=@();

# Import Operations Manager Module and create Connection
Import-Module OperationsManager
New-SCOMManagementGroupConnection <managementgroup name>

$alerts=get-scomalert -Criteria "Severity!=0 AND IsMonitorAlert=1 AND ResolutionState=255"| where {$_.LastModified -ge ((get-date).AddDays(-7)).ToUniversalTime()}
if ($alerts -is [object])
{
foreach ($alert in $alerts)
{
$monitoringobject = Get-SCOMClassinstance -id $alert.MonitoringObjectId
# Reset Monitor
If (($monitoringobject.HealthState -eq "Error") -or ($monitoringobject.HealthState -eq "Warning"))
{
$monitoringobject.ResetMonitoringState()
$State+=$monitoringobject.HealthState
$Displayname+=$monitoringobject.displayname
$Alertname+=$alert.Name
}
}
}

Display primairy and failover servers for all Gateway servers:

#Display Primary and Failover Management Servers for all Gateway Servers
$GWs = Get-SCOMManagementServer | where {$_.IsGateway -eq $true}
$GWs | sort | foreach {
Write-Host "";
"Gateway MS :: " + $_.Name;
"--Primary MS :: " + ($_.GetPrimaryManagementServer()).ComputerName;
$failoverServers = $_.getFailoverManagementServers();
foreach ($managementServer in $failoverServers) {
"--Failover MS :: " + ($managementServer.ComputerName);
}
}
Write-Host "";
</span>

Configure failover for Gateway Servers:

This script can be used for configuring primairy and failover management server for a gateway. The script is run on the management server.

$primaryMS = Get-SCOMManagementServer | where {$_.Name –eq '<primairy ms>'}
$failoverMS = Get-SCOMManagementServer | where {$_.Name –eq '<secundairy ms'}
$gatewayMS = Get-SCOMManagementServer | where {$_.IsGateway -eq $true}
Set-SCOMParentManagementServer -GatewayServer: $gatewayMS -PrimaryServer: $primaryMS
Set-SCOMParentManagementServer -GatewayServer: $gatewayMS -FailoverServer: $failoverMS
</span>

Count all closed alerts created by monitors in the last X days:

$targetdate = (get-date).AddDays(-1)

(Get-SCOMAlert -criteria 'ResolutionState = "255" AND IsMonitorAlert = "True"'| Where-Object {$_.LastModified -gt $targetdate }).count

Disable a specific monitor:

$MP = Get-SCOMManagementPack -displayname "<override management pack>" | where {$_.Sealed -eq $False}
$Class = Get-SCOMClass -DisplayName "<class name>"
$Monitor = Get-SCOMMonitor -DisplayName "<monitor name>"
Disable-SCOMMonitor -Class $Class -ManagementPack $MP -Monitor $Monitor</pre>
$Monitor = Get-SCOMMonitor -DisplayName "<monitor name>"
Disable-SCOMMonitor -Class $Class -ManagementPack $MP -Monitor $Monitor

Enable a specific monitor:

$MP = Get-SCOMManagementPack -displayname "<override management pack>" | where {$_.Sealed -eq $False}
$Class = Get-SCOMClass -DisplayName "<class name>"
$Monitor = Get-SCOMMonitor -DisplayName "<monitor name>"
Enable-SCOMMonitor -Class $Class -ManagementPack $MP -Monitor $Monitor</pre>
$Monitor = Get-SCOMMonitor -DisplayName "<monitor name>"
Enable-SCOMMonitor -Class $Class -ManagementPack $MP -Monitor $Monitor

Turn on Agent proxy for all agents where it is disabled:

get-SCOMagent | where {$_.ProxyingEnabled -match "False"} | Enable-SCOMAgentProxy

Turn on Agent proxy by default for all new agents:

add-pssnapin "Microsoft.EnterpriseManagement.OperationsManager.Client";
new-managementGroupConnection -ConnectionString:scomserver.domain.com;
set-location "OperationsManagerMonitoring::"; 
Set-DefaultSetting -Name HealthService\ProxyingEnabled -Value True

System Center 2012 R2 Rollup 1 Available

Microsoft puts a lot of effort in the System Center suite and therefore tries to fix issues as soon as possible. The Rollup 1 for 2012 R2 is already released. For Operations Manager quite some issues are fixed:

http://support.microsoft.com/kb/2904678/en-us

Let’s Fix it!

Monitors and Rules in Operations Manager – Learn it!!!

I feel I need to bring this subject under the attention again. Why?

Many, many, MANY, many customers do not quite (or not even at all 😦 ) understand how important it is to know the difference between rules and monitors in Operations Manager. This needs to be basic knowledge when you start working with this fantastic monitoring solution.

The key difference is: Monitors affect the health state of a managed object, rules don’t.

Closing an alert generated by a monitor can create as what i like call a “blind Spot”. A monitor only fires an alert when the state changes to a warning and/or critical state. When the monitor is in that unhealthy state it might turn healthy again in the next interval. If it doesn’t and you closed the alert, you never see an alert again!! (unless a miracle happens).

Read more…

A PowerShell Gem I Will Use More Often

Until a few weeks ago I was struggling with exporting and sorting data that I get from PowerShell commands (running against Operations Manager :-)). I could not think of an efficient way to accomplish this. I used the export-csv CmdLet to do an export and after that open the CSV file and sort the data in Excel.

Of course there are many more ways to sort data but I really like the Out-GridView Cmdlet I “discovered” a few weeks ago.
This one is available in PowerShell 2.0. The Out-GridView cmdlet is automatically installed when you install PowerShell 2.0; however, the cmdlet won’t do anything unless you also have .NET Framework 3.0 installed.

OutGridView

My life gets easier every day!!