Process Memory Utilization / Iron Scripter Challlenge (20 Jul 2021)

I've heard of the Iron Scripter Challenge, but never really looked into it. This week I decided to take the plunge and I'm glad I did. Basically, the Iron Scripter Challenge is a series (I believe monthly) of scripting challenges in PowerShell. If your like me and you love challenges, or learn by doing, I highly recommend it. Here is a link to the challenge I worked on this week https://ironscripter.us/a-memory-reporting-challenge/ 

The overall goal was to collect memory data on running processes by process name. For example, if there are 20 instances of Chrome process running, combine the memory data for all of them. I modified this requirement a little and included the path with the name because I know malicious process can use legitimate names running in incorrect locations. For example, what if one of those 20 Chrome processes was executed from C:\Windows\Temp? My Spidey sense would be tingling. 

Here's the basic data I needed to collect: 

1. Process name  
2. Total number of processes 
3. Total WorkingSet 
4. Percentage of in-use memory

While it would be easy to write this in a verbose script with variables and foreach loops to solve this, I thought a good challenge for me was to write this in one command with many pipes. Additionally, I wanted to make this as efficient as possible with that limitation.

Process objects returned from the Get-Process, Group-Object, and Measure-Command cmdlets provide all the data we need. I'll break down each here with the corresponding requirement:

1. Process name  (Get-Process ProcessName property)
2. Total number of processes (Use Group-Object Count property)
3. Total WorkingSet (Use Measure-Object Sum property on the WorkingSet of the group)
4. Percentage of in-use memory (Divide the WorkingSet for the group by the WorkingSet sum for all properties)

1   Get-Process |

2       Group-Object Name, path |

3           Select-Object @{

4                   n="ProcName"

5                   e={($_.name -split ", ")[0]}

6               },

7               @{

8                   n="Path"

9                   e={($_.name -split ", ")[-1]}

10              },

11              @{

12                  n="TotalProcs"

13                  e={$_.Count}

14              }, 

15              @{

16                  n="WorkingSetTotal"

17                  e={($_.Group.WorkingSet | Measure-Object -Sum).sum}

18              },

19              @{

20                  n="Percentage"

21                  e={($_.Group.WorkingSet | Measure-Object -Sum).sum / 

22                  ((Get-Process).workingset | Measure-Object -Sum).Sum 

23                  }

24              }

I used calculated properties for most of the requirements. The Group-Object command will give me a Name, Count, and Group property.  Name will be a string in the format of "Name, Path". Depending on the user's permissions when running the command, some process paths may be null. By using the -split operator with the regular expression of ", " I account for these instances. In those cases the Path value will be the same as the name because in an array with 1 element, item [0] and [-1] are the same. The Count property will be the total number of processes for that group.  The group property will be all the process objects belonging to that group, so $_.group will be an array of those objects.

After this first go around, I wasn't quite satisfied with the Percentage calculation even though it worked just fine. I didn't like the fact I was running Get-Process again for each group when I already ran it. This is where Tee-Object comes. By using Tee-Object, I can assign the original process objects collected in the first command on the pipeline to a variable then call it later down the pipe. Here's the updated version.

1   Get-Process |

2       Tee-Object -Variable Procs |

3           Group-Object Name, path |

4               Select-Object @{

5                  n="ProcName"

6                  e={($_.name -split ", ")[0]}

7               },

8               @{

9                  n="Path"

10                 e={($_.name -split ", ")[-1]}

11              },

12              @{

13                 n="TotalProcs"

14                 e={$_.Count}

15              }, 

16              @{

17                 n="WorkingSetTotal"

18                 e={($_.Group.WorkingSet | Measure-Object -Sum).sum}

19              },

20              @{

21                 n="Percentage"

22                 e={($_.Group.WorkingSet | Measure-Object -Sum).sum / 

23                    ($Procs.workingset | Measure-Object -Sum).Sum 

24                    }

25              }

Being satisfied with this, I moved on to the "bonus" requirements. Here they are:

1. Create a PowerShell function around your code
2. Support querying a remote computer
3. Provide sorted results
4. Provide formatted output
5. Services run in a process so create a command to report on the same memory usage for a given service. It is OK if the service shares a process. Although you might want to indicate that.

I sat out to tackle the first four.  For the first requirement, I decided to create 2 functions a "Get" and a "Show".  I did this because a couple requirements were formatting and sorting which are not appropriate for a "Get" function, so I split them up into 2. Both functions will work the same way in fact I just call the "Get" function from the "Show" function to get the data before formatting.

Let's break down the "Get" function first.  Here is the top of the function with the param block.

1   function Get-ProcessMemory {

2       [CmdletBinding(DefaultParameterSetName = 'None')]

3       [OutputType([PSCustomObject])]

4   

5       param (

6           # Remote computer or collection of remote computers

7           [Parameter(Mandatory = $true, ValueFromPipeline = $true, ParameterSetName = 'CN')]

8           [string[]]

9           $ComputerName,

10  

11          # PSCredential for remote computer(s)

12          [Parameter(ParameterSetName = 'CN')]

13          [pscredential]

14          $Credential,

15  

16          # PSSession for remote connection

17          [Parameter(Mandatory = $true, ParameterSetName = 'Session')]

18          [System.Management.Automation.Runspaces.PSSession]

19          $Session

20      )

21         

22  

I intentionally used parameter names that match some of the Invoke-Command parameters. I did this because I intend to use that cmdlet to satisfy the second bonus requirement using the $PSBoundParameters automatic variable. I also have 3 parameter sets, even though you only see 2 explicitly named in the parameters. The third one is default "None" which will happen when no parameters passed (technically it would be the default if the parameter set couldn't be determined). What I'm trying to allow here is the ability to just call the function on the local machine without passing any arguments, but include them for remoting if needed.

On to the process block:

23      process {

24          $Command = @'

25              Get-Process |

26                  Tee-Object -Variable Procs |

27                      Group-Object Name, path |

28                          Select-Object @{

29                              n="ProcName"

30                              e={($_.name -split ", ")[0]}

31                          },

32                          @{

33                              n="Path"

34                              e={($_.name -split ", ")[-1]}

35                          },

36                          @{

37                              n="TotalProcs"

38                              e={$_.Count}

39                          }, 

40                          @{

41                              n="WorkingSetTotal"

42                              e={($_.Group.WorkingSet | Measure-Object -Sum).sum}

43                          },

44                          @{

45                              n="Percentage"

46                              e={($_.Group.WorkingSet | Measure-Object -Sum).sum / 

47                                  ($Procs.workingset | Measure-Object -Sum).Sum 

48                              }

49                          }

50  '@ #Command here-string to invoke

51  

52  

I created a Here-String containing my original command. I did this so I can use it for both Invoke-Expression for the local machine or Invoke-Command for remote machines.  On to the next/last portion.

53          if ($PSCmdlet.ParameterSetName -eq "None") {

54              Invoke-Expression -Command $Command

55          } #if ParameterSetName is None (local machine)

56          

57          else {

58              $InvokeCommandArgs = $PSBoundParameters #works because I use the same parameter names

59              $InvokeCommandArgs.ScriptBlock = [scriptblock]::Create($Command)

60              Invoke-Command @InvokeCommandArgs

61          } #else - ParameterSetName is NOT None (remoting)   

62  

63      } #Process Script Block for Get-ProcessMemory Function 

64  

65  } #Get-ProcessMemory Function Definition

66  

In this last section I determine if it is going to be ran on the local machine (ParameterSet -eq "None") or remote machine. For local machine I just used the Invoke-Expression cmdlet to run my command. I could have converted it to a script block and called the Invoke method, but I like to use native cmdlets when I can.  

For remote machines, I take advantage of using standard names for my parameters so I can just use the $PSBoundParameters automatic variable to splat Invoke-Command.  I did convert $Command to a scriptblock to make this call.

The "Show" function has the exact same parameters so I'll skip that and go straight to the process block.

100     process {

101         

102         Get-ProcessMemory @PSBoundParameters |

103             Sort-Object WorkingSetTotal -Descending |

104                 Format-Table -Property @{

105                     n="Name"

106                     e={$_.ProcName}

107                     w=35

108                 },

109                 @{

110                     n="Path"

111                     e={$_.Path}

112                     w=30

113                 },

114                 @{

115                     n="Procesess"

116                     e={$_.TotalProcs}

117                     w=10

118                 },

119                 @{

120                     n="Usage/MB"

121                     e={$_.WorkingSetTotal / 1MB}

122                     f="N2"

123                     w=15

124                 },

125                 @{

126                     n="Percentage"

127                     e={$_.Percentage}

128                     f="P2"

129                     w=15

130                 }

131         

132     } #Process Script Block for Show-ProcessMemory Function

133 

Since the parameters were the same I just called my "Get" function splatting the $PSBoundParameters variable. I decided to sort it by memory usage so the highest usage is at the top. Then, I put all the data neatly in a table using format specifiers for the numbers and specified the width as well. If you are unfamiliar with this take a look at about_calculated_properties. This is about the only place I use shorthand. "n" is short for "name", "e" is short for "expression", "f" is short for "formatstring" and "w" is short for "width".

Finally I bundled it up in a module, exporting both functions. All the code is available for download in my github repository here https://github.com/ralphmwr/MemoryUse

Here is a screen shot of the Show-ProcessMemory function in action:

Looking forward to the next Iron Scripter Challenge.

Please comment if you have questions or suggestions.

Running Process Analysis - Part 1 Data Collection

Comparing artifacts to a baseline is one of the simplest ways to find indicators of compromise.  Basically, you have a known good baseline of data used to compare with host data.  If there are differences, then you might have an indicator of compromise to hunt down.  This basic analysis method can be used for many different host artifacts.  In this article, I will focus on running processes.

When evaluating processes, first think about what properties you might want to compare.  A simple comparison might just be the name of the process, but malware can be much stealthier than just running under its name like "evil_process".  One of the ways malware can disguise itself is by using common names but running from incorrect locations.  In this case the path of the process executable would be a good property to check.  What about an instance where the malware is actually infected a legitimate process.  I'm not talking about process hollowing.  I'll save that discussion for a later post.  In a simpler context, the malware could just overwrite the legitimate executable.  In this case we could analyze the hash of the executable.

So far we have decided to look at the name, path and hash of the running process objects.  How could we get this data using a PowerShell script?  Fortunately, PowerShell has a built-in cmdlet that will retrieve process objects for each running process on a host machine.  You could find this cmdlet using a search.  This search takes advantage of PowerShell's consistent naming convention and superior documentation.  These attributes combine to make PowerShell a very discoverable language.  Here is how we could search for a cmdlet to retrieve running processes.

To make sure the cmdlet does what you expect, you can review the documentation.  PowerShell's documentation is top notch and will not only give you a terrific description of the cmdlet, but descriptions of all of its parameters along with several examples.



Next, we need to test and see if the cmdlet provides all the properties we want to analyze.  From the previous discussion, what we want is name, executable path and a hash of the executable.  To discover all the properties of objects returned from our cmdlet, simply pipe the results to Get-Member.  This command will perform what's called "introspection" and display the object types along with their associated properties and methods.



Looking at the list of properties returned, we can see 2 of the 3 properties we need are contained in these process objects.  The only one not contained is the file hash of the executable.  That's ok, we can add that one in using a different cmdlet.  How could we find a cmdlet that will return a hash of file?  You guessed it with Get-Command.



Let's test this cmdlet out and find out what type of object it returns along with its properties.  We'll just run it against a file on our computer and pipe the results to Get-Member.



Looking at the properties, it appears the "hash" property contains the data we need for our script.

Now lets start putting this together.  We need to run Get-Process but only need 2 of the automatic object properties.  We also need a third custom property that will use Get-FileHash to generate the property value.  We can use the Select-Object cmdlet to select object properties along with creating our own custom properties.  We can pipe objects to Select-Object to enumerate the properties.  We'll start with the 2 automatic properties.



Now we need to add the custom property.  To add a custom property, simply add a hashtable to the list of properties.  Hashtables in PowerShell can be created with a special operator @{}.  Inside the curly braces place the key/value pairs.  This syntax is <key>=<value>.  For custom properties we need two keys: name and expression.  These can be abbreviated with n and e.  Name will be a string that is the name of the new property and expression will be a script block that returns the value for the new custom property.  So it looks like this:



Note the $_ is used to reference the current object in the pipeline.  Now we have the command all built out we can view the data right on the screen, save the results to variable or export the results to a file for analysis later.

In the next post we will add to the functionality of this command by exporting the data and comparing it to a baseline.

Why PowerShell

While there are some great tools out there for cyber threat hunting, most require some type of agent installed on the hosts or a centralized server to function effectively.  Unfortunately, host and sever configuration for many networks, especially those on DoD or other government systems is very controlled and the agents required for those tools simply can't be installed.  In those instances, cyber threat hunters must "live off the land" when collecting host data across networks.  This means we must use tools already present on hosts/servers to hunt for indicators of compromise.

Fortunately, most of the networks I deal with and many other networks are already administered using PowerShell.  This means that not only is PowerShell installed on most of the host machines, but the network is already configured to allow PowerShell commands to be remotely executed.  Since PowerShell rides on the .NET framework, there are many providers available to expose data critical to cyber threat hunting.  Additionally, PowerShell has the ability to leverage WMI so even if standard .NET classes or PowerShell's thousands of built-in cmdlets/functions don't expose the data you need, WMI probably does.

PowerShell was built to help administrators get back on the command line and automate tasks for management of network systems.  It is very well documented and supported.  Unfortunately, much of what I find online in the way of PowerShell support/examples revolve around system administration tasks and not threat hunting.  There are some unique challenges a threat hunter might have when leveraging PowerShell that a system administrator may not have.  For example, since a threat hunter is likely to be limited to whatever is installed on host systems, he or she may have to use older cmdlets or WMI if an older version of PowerShell is installed.  I have seen many cases where someone creates an amazing PowerShell script for threat hunting data collection but it doesn't function on all or some of the hosts in a network because it relies on functionality not available in older versions.  Threat hunters have to be flexible and not just rely on scripts that can't be customized for specific situations.  Additionally, threat hunters may have to work around network limitations.  Recently, I was on a network where the WinRM service was disabled on several machines.  While WinRM is necessary for standard PowerShell remoting, there is more than one way to skin a cat.  In this case we leveraged RPC using specific PowerShell cmdlets in Windows PowerShell v5.1.

This post was intended to just give you a taste of why PowerShell is an important tool for cyber threat hunting.  In future posts, I will show real-world examples and hypotheticals to illustrate how PowerShell can be leveraged for threat hunting.