Windows and SQL Server metrics data capturing

Here are step by step instructions on setting up a process to collect Windows and SQL Server related metrics data. Most of the automation scripts are written in PowerShell. To be honest, given a choice, I'd rather write my tools in either Python or Perl. However, on Windows systems, those languages require a separate install, whereas PowerShell is present already. That, plus the tight integration between PowerShell and Windows (and other Microsoft server products), makes starting with PowerShell pretty compelling. I may present the same tools written in Python, Perl, and/or other languages if/when I develop them.

Setting this up is pretty easy. Here is a quick overview before we delve into the details:

• We will need to create a SQL Server database, JiMetrics, to store metrics data;
• The collection process relies on SQL Server Agent;
• The domain account SQL Server Agent jobs run under must have:
  • Administrative access to all server hosts you intend to collect metrics data from
  • sysadmin access to all SQL Server instances you intend to collect metrics data from
  • dbo access to the JiMetrics database.
• Manual steps needed are minimal. If you create steps in the order that is presented in this page, you will be fine. Each PowerShell script corresponds to one SQL Server Agent job. Inside the PowerShell script, it knows which stored procedure to call, which Hosts are involved, what metrics to gather, and which table to put that metrics into.
• Once you've started collecting for a while, you will certainly be able to get a lot out of this information. There are very few tables involved so the database is pretty easy to query against. I suggest you follow the database diagram for help.

The whole process is pretty lightweight, and I suggest you start doing this as soon as possible. Over time, you will find the information gathered very valuable and you'd be happy with the fact that you start the collection sooner rather than later.

Please follow this link to create the JiMetrics database for metrics data storage. Come back to this page and continue the steps below when the database is ready;

Enter Windows server hosts whose metrics you are interested in gathering. You can do this manually by running SQL statements like this:

USE JiMetrics;
GO
INSERT INTO Windows.Host (HostName) VALUES ('WindowsHost1');
GO

Or, if you have a list of servers in a file, it is easy to import that into the Host table.

Use the domain account mentioned above to create a SQL Server credential, use that credential to create a SQL Server Agent PowerShell proxy account afterwards. Please use this proxy account to run ALL SQL Server Agent jobs' steps that you will create below.

Please follow this link to create a SQL Server job for updating server host info. These include things like domain, OS Version Number, hardware model, hardware vendor, memory size in GB, CPU type, CPU core count, if the host is a VM, etc. The domain account that this job runs under needs to in the local administrator's group on the host whose metrics it is trying to get.

Manually run the job after it is created. It should succeed if the proxy account has proper access defined above. Run the T-SQL code below to check the results:

USE JiMetrics;
GO
SELECT * FROM Windows.Host;
GO

Please follow this link to create a SQL Server job for gathering server host disk information. Manually run the job after its creation. Run the T-SQL code below to check the results:

USE JiMetrics;
GO
SELECT * FROM Windows.Storage;
GO

Please follow this link to create a SQL Server job for discovering installed SQL Server instance(s) on server hosts in the Host table. Manually run the job after its creation. Run the T-SQL code below to check the results:

USE JiMetrics;
GO
SELECT * FROM Windows.Instance;
GO

After SQL Server instances are discovered in all active servers in the Windows.Host table, it is now time to find out if the instances are running and their edition level and versions. Please follow this link to create a SQL Server job. Manually run the job after its creation. Run the T-SQL code below to check the results:

USE JiMetrics;
GO
SELECT * FROM Windows.Instance;
GO

Now that we have names of all currently running SQL Server instances, it is time to gather their configuration information, such as its min and max memory usage boundaries, instance level parallelism, CLR usage, et cetera. Please follow this link to create a SQL Server job to gather that data. Manually run the job after its creation. Run the T-SQL code below to check the results:

USE JiMetrics;
GO
SELECT * FROM Windows.InstanceConfig;
GO

To get a good handle on the performance characteristics and metrics, it is important to gather and store performance counters. Please follow this link to create a SQL Server job to gather that data. Manually run the job after its creation. Run the T-SQL code below to check the results:

USE JiMetrics;
GO
SELECT * FROM Windows.InstanceDmvPerfCounter;
GO

After server hosts, hosts' disk information, and SQL Server instance information have been gathered, we drill down one more level to database files. These include database name, its associated files, file size, free size, max size information, and growth property (percentage growth or not). Please follow this link. Manually run the job after its creation. Run the T-SQL code below to check the results:

USE JiMetrics;
GO
SELECT * FROM Windows.DbFileStats;
GO

We drill down one more level to metrics about database tables. These include database name, schema name, table name, total row count, data size, and index size. Please follow this link. Manually run the job after its creation. Run the T-SQL code below to check the results:

USE JiMetrics;
GO
SELECT * FROM Windows.TableStats;
GO