Inspect VMware ESXi log with local llama3.1 AI.

In this example I would like to demonstrate the use of the llama 3.1 AI Large Language Model. In this example a locally running llama 3.1 will inspect a VMware ESXi vmkernel log.

I am using an Apple MacBook Air M2 with 16GB of RAM. If you are using a Windows or Linux system the procedure might vary a little bit.

So the first thing to do is to download/install ollama from https://ollama.com. It is shockingly simple. You don’t need to create an account or register your email address. Simply download and install. It couldn’t be easier. Therefore I will not go further into detail about the installation of ollama.

I assume you have Microsoft Powershell and VMware PowerCLI running on your machine. If not, I quick google search should lead you to the steps needed to install PowerCLI, and if needed Powershell (newer Windows systems should have Powershell installed by default)

On a Mac I have to open a terminal window and start powershell by typing pwsh.

mikey@Mac-Air ~ % pwsh
PowerShell 7.4.4
PS /Users/mikey>         

Now we connect to VMware vCenter server by typing the following (vcenter.lab being DNS the name of the vCenter Server. )

connect-viserver vcenter.lab

You will be prompted for username and password for the vCenter Server.

Let’s get a vmhost object using the Get-VMHost command and ESXi name and save it in the $vmhost variable.

$vmhost = Get-VMHost myesxi1.lab

With the cmdlet Get-LogType you can gather the available log types from the ESXi host.

Get-LogType -VMHost $vmhost             

Key                  Summary
---                  -------
fdm                  
hostd                Server log in 'plain' format
vmkernel             Server log in 'plain' format
vpxa                 vCenter Server agent log in 'plain' format

Now we are using the cmdlet Get-Log with the required parameter -key and optional parameter -vmhost and save the result in the variable $vmkernellog

$vmkernellog = Get-Log -Key vmkernel -VMHost $vmhost 

The output is not a simple string but of type VMware.VimAutomation.ViCore.Types.V1.LogDescriptor

To get the log, simply use $vmkernellog.Entries.

Now we use ollama to ask llama3.1 questions about the log file. The command for this will be ollama run and the LLM which in this case is llama3.1. llama3.1 is the default 8B model. To use the 70B model the command would be ollama run llama3.1:70b and for the 405B model it the command would be ollama run llama3.1:405b. However your computer most likely will not be powerful enough to run the 70B or 405B model. Ollama doesn’t have access to your local file system and therefore you cannot direct it to a file. But you can parse the string, which in our case is $vmkernellog.Entries and then ask your question about it. I have just stopped a vm called photon1 and I am going to ask llama3.1 about it. My question here is “vm photon1 failed, can you determine the reason?”

ollama run llama3.1 $vmkernellog.Entries vm photon1 failed, can you determine the reason? ResourceUnavailable: Program 'ollama' failed to run: An error occurred trying to start process '/usr/local/bin/ollama' with working directory '/Users/mikey'. Argument list too longAt line:1 char:1
+ ollama run llama3.1 $vmkernellog.Entries vm photon1 failed, can you d …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~.

As we see, we will get an error message back. The reason for this is that llama3.1 can “only” handle 128K tokens. So we need to shorten the log first. The cmdlet Get-Log has a parameter -NumLines. So would could specify, let’s say 500 lines. However it would return the oldest 500 lines and I want the latest 500 lines. For that we have to filter the log after the fact. I use select -last 500 to do that and save the result in the $mylog variable.

$mylog = $vmkernellog.Entries | select -last 500

Now I can run ollama again using $mylog

ollama run llama3.1 $mylog vm photon1 failed, can you determine the reason?               
The provided log snippet appears to be from an ESXi host, and it reports issues related to a Photon OS virtual machine (vm). I'll highlight some key points that could help 
in understanding why "Photon1" might have failed:

### Key Points:

* **Port unblocking**: The first message mentions an event where a port (`0x400000a`) was disabled and then disconnected. This doesn't seem directly related to the failure 
of "Photon1".
* **VSCSI Events**: There are several events related to Virtual SCSI (VSCSI) devices, particularly one with `GID:8211` and associated with a handle `9018516493574163`. The 
event mentions destroying a device for world `2099787`, which might be relevant depending on the context of VSCSI usage in your environment.
* **NMP and Device Errors**: There are several messages indicating issues with the NMP (Network Multipathing) daemon, including an error status from dev 
"mpx.vmhba32:C0:T0:L0" with sense data 0x5 0x24 0x0. These errors suggest problems with a device or path in your storage configuration.
* **HPP and Unknown Check Condition**: The log mentions HPP (Host Physical Port) related issues, including an unknown check condition for dev "mpx.vmhba32:C0:T0:L0". This 
might be linked to the NMP errors.

### To Determine the Reason of Photon1 Failure:

Given these points, it seems the most likely cause of "Photon1" failing is related to storage or path issues, as hinted at by the NMP and VSCSI logs. Here are some steps 
you can take:

1. **Check Storage Configuration**: Review your ESXi host's storage configuration for any errors or inconsistencies that might be causing these symptoms.
2. **Verify Connectivity**: Ensure all paths (physical or virtual) to the devices mentioned in the logs (`mpx.vmhba32:C0:T0:L0`) are functioning correctly.
3. **Check VM Settings and Resources**: Make sure "Photon1" is configured to use a valid storage path or controller, with sufficient resources allocated.

By investigating these areas, you should be able to identify why "Photon1" failed on your ESXi host.

Ok, the result is not very convincing, but you get the idea. Of course you are not limited to ESXi log files but pretty much any data represented as string. To set this up only takes minutes. I was really impressed how easy it was to setup ollama and to get the logs from ESXi is also quick and easy. It also runs locally on your machine, you don’t need to worry about your ESXi log data ending up on the internet.

I would love to hear your feedback if you have use this method to have other data reviewed by llama and if you get any results that are valuable.