VCAP5-DCA Objective 6.1 – Configure, manage and analyse vSphere log files

Objectives

  • Generate vCenter Server and ESXi log bundles
  • Use esxcli system syslog to configure centralized logging on ESXi hosts
  • Test centralized logging configuration
  • Analyze log entries to obtain configuration information
  • Analyze log entries to identify and resolve issues
  • Install and configure VMware syslog Collector and ESXi Dump Collector

Generate vCenter Server and ESXi log bundles

Official Documentation:
vCenter Server Host Management Guide, Chapter 8, “System Log Files”, page 91.

Summary:

vCenter Server

To generate vCenter Server log bundles. There are a few ways to get started, but depending on your location in the vSphere Client) options can vary. The best starting points:

  • Menu, Administration, Export System Logs
  • Home, System Logs, the button  “Export System Logs”

From here, you can select where logging should be gathered from.

Optional you can include information from the vCenter Server and your vSphere Client.

Figure 1

On the next page, you can specify which system logs will be included, Performance data is optional.

Figure 2

After you have specified the download location, the collecting will start.

Figure 3

Adjust the Logging Level in vCenter, go to menu: Administration, vCenter Server Settings, Logging Options. Default is “Information”.

Figure 4

Another way to generate vCenter Server Logbundles, is directly from the vCenter Server. RDP to the vCenter Server, from the Start Menu, select the option. In fact this runs a CLI script vc-support.wsf. The difference between the two options is an extra switch.

Figure 5

Some information on Logfiles.

ESXi server

While connected to an ESXI server with the vSphere Client, you can export logs:

  • Home, System Logs, the button  “Export System Logs”
  • From the menu: File, Export, Export System Logs (make sure you select the ESXi host!)

The rest of the process is nearly identical.

  • VMware KB “Location of ESXi 5.0 log files
  • Compared to vSphere 4.x, the number of log files has been increased. The KB presents a nice overview. To highlight a few:
  • /var/log/auth.log: ESXi Shell authentication success and failure.
  • /var/log/hostd.log: Host management service logs, including virtual machine and host Task and Events, communication with the vSphere Client and vCenter Server vpxa agent, and SDK connections.
  • /var/log/shell.log: ESXi Shell usage logs, including enable/disable and every command entered.
  • /var/log/syslog.log: Management service initialization, watchdogs, scheduled tasks and DCUI use.
  • /var/log/vmkernel.log: Core VMkernel logs, including device discovery, storage and networking device and driver events, and virtual machine startup.
  • /var/log/vmkwarning.log: A summary of Warning and Alert log messages excerpted from the VMkernel logs.
  • /var/log/vmksummary.log: A summary of ESXi host startup and shutdown, and an hourly heartbeat with uptime, number of virtual machines running, and service resource consumption.
  • /var/log/vpxa.log: vCenter Server vpxa agent logs, including communication with vCenter Server and the Host Management hostd agent.
  • /var/log/fdm.log: vSphere High Availability logs, produced by the fdm service.

While directly logged on to an ESXI server (SSH session), you can collect log files, using the following command:

# vm-support

To see available options, use:

# vm-support –help

Other references:

  • A

 

Use esxcli system syslog to configure centralized logging on ESXi hosts

Official Documentation:
VMware KB “Configuring syslog on ESXi 5.0

Summary:
This topic is part of a more complex subject. Because an ESXi host loses all log files after a reboot. Or worse after a crash, configuring centralized logging is highly recommended.

Configuring centralized logging comes in two parts:

  • Install and configure a server for collecting the log files. There are many ways, you can set up an Linux Syslog server, or a Windows based product, like Kiwi Syslog server. My personal favourite in vSphere 4.x was the vilogger in the vMA. This option has gone in vSphere 5.x. VMware has introduced the “Network Syslog Collector” as part of vCenter Server.
  • You need to configure your ESXi hosts and direct them to the Syslog server

Although, I prefer a Linux solution, I have installed the “Network Syslog Collector” for the purpose of this Study; see under “Other References”.

Most important step during installation imho is the location of the Log files.

Figure 6

Good reading on configuring the ESXi host, the actual topic is VMware KB “Configuring syslog on ESXi 5.0”. This KB describes the five configurable options and three ways to configure, including the esxcli option

The five configurable options:

  • logDir; A location on a local or remote datastore and path where logs are saved to.
  • logHost; A remote server where logs are sent using the syslog protocol.
  • logDirUnique, A boolean option which controls whether a host-specific directory is created within the configured logDir.
  • defaultRotate, The maximum number of log files to keep locally on the ESXi host in the configured logDir. Default=8
  • defaultSize, The maximum size, in kilobytes, of each local log file before it is rotated. Default=1024 KB.

Note: the last two options do not affect remote syslog server retention.

The most important esxcli commands for configuring syslog. To view current config:

# esxcli system syslog config get

To set an option, we configure our remote syslog server, IP address 192.168.100.105 on TCP port 514.

# esxcli system syslog config set --loghost=’tcp://192.168.100.105:514’

After making configuration changes, load the new configuration using the command:

# esxcli system syslog reload

Note: In case you cannot remember the correct format specifying the log host, a trick, in the vSphere Client, go to the advanced settings

Figure 7

Note: after applying these changes, unfortunately, the log files will not be received on your configured Syslog server. This is because of the Firewall settings of the ESXi host. By default, outgoing Syslog traffic is disabled.

To configure the firewall, in the vSphere Client,

  • select the ESXi host, go to Configuration, Software, Security Profile, Open the Firewall Properties

Figure 8

  • Place a tick at syslog.
  • Button  “Firewall…” open the Firewall Settings window.

Figure 9

  • Change to “Only allow connections from the following networks:” and add the IP address of the Syslog server.
  • You can also use esxcli to open outbound traffic via the ESXi Firewall use the following commands:
    # esxcli network firewall ruleset set --ruleset-id=syslog --enabled=true
    
    # esxcli network firewall refresh
    
  • When everything has been configured correctly, ESXi hosts being logged show up in vCenter.

Figure 10

  • Unfortunately, you cannot browse the log files from here. You will have to browse the folder where the logs are stored. In my case, RDP to the server and browse the folder E:\SYSLOG_DATA.

Other references:

  • Alternatives for a Syslog server from Virtually Ghetto, here.
  • Install and Configure “Network Syslog Collector” from mwpreston.net  post.

 

Test centralized logging configuration

Official Documentation:

Summary:
When everything has been configured correctly, log files should show up in the Syslog server.

In my case, using the “Network Syslog Collector“, the actual log files can be retrieved.

Figure 11

Notice that:

  • A folder has been created for every ESXi host, identified by the management IP address;
  • In each folder a single file, named syslog.log, containing entries from the Hostd.log and the Vpxa.log

In case, logging does not show up, try the following:

  • Check the configuration of the ESXi host, especially the syntax of the loghost;
  • Check the configuration of the ESXi firewall, outgoing syslog allowed;
  • On the ESXi host, try restarting the Managent Agent. From the DCUI or
    # /sbin/services.sh restart
  • On the Syslog server, also check the firewall settings, is incoming traffic allowed?
  • Try to connect to the Syslo server using the telnet command, e.g.:
    > telnet <IP Syslog server> 514
  • In case you use the “Network Syslog Collector”, review the settings

Other references:

  • A

 

Analyze log entries to obtain configuration information

Official Documentation:

Summary:
Not much official documentation on this topic.

In the first topic, I referenced to VMware KB with an overview on products and log file locations.

I encourage you to log on to an ESXi host, cd into /var/log, and have a look at the logfiles available. Use the commands: more or vi to browse the log files.

Imho, the following logs at least contain some information on the configuration of an ESXi host. For those familiar to Unix and Linux OS, a very useful log file in case of startup and configuration issues is dmesg. ESXi has a few of that kind of logs:

  • /var/log/syslog.log
  • /var/log/vmkernel.log
  • /var/log/vmkwarning.log, contains a summary of warnings and alert log messages from the vmkernel.log

TIP: you can use the grep command to search for specific terms, e.g.:

# grep disk vmkernel.log

For those familiar with vi, once opened the log file, you can use the ‘/’ and ‘?’ to quickly search.

Other references:

 

Analyze log entries to identify and resolve issues

Official Documentation:

Summary:
See also previous topic. While investigating an issue, it is a good idea to analyze log files, like the hostd.log or vmkernel.log for specific messages. Those messages can help you finding a VMware KB that can solve your issue or contacting a colleague or VMware Support.

Other references:

  • A

 

Install and configure VMware Syslog Collector and ESXi Dump Collector

Official Documentation:
vSphere Installation and Setup Guide, Chapter 12 “After You Install vCenter Server”, Section “Install vSphere ESXi Dump Collector”, page 212. Also section “Install vSphere Syslog Collector”, page 213

Summary:
VMware Syslog Collector

A part of the configuration of the Syslog Collector has been discussed in the topic “Use esxcli system syslog to configure centralized logging on ESXi hosts“.

  • The Syslog Collector can be installed on the vCenter Server or on a separate server that has a network connection to the vCenter Server.
  • The Syslog Collector does not support IPv6.
  • The product is on the same media as the vCenter Server
  • The installation is pretty straightforward. During the installation you can adjust parameters, like;
    • Location where to install
    • Location for the Syslog Repository
    • Max. size of the repository
    • Max.number of log rotations to keep
    • Protocols and Ports to be used and whether secure connections (SSL) should be used

Configuartion of the ESXI hosts has been discussed.

ESXi Dump Collector

You can configure ESXi to dump the vmkernel memory to a network server, rather than to a disk, when the system has encountered a critical failure. Install vSphere ESXi Dump Collector to collect such memory dumps over the network.

In the vCenter Appliance, the ESXi Dump Collector is enabled by default. This section applies to Windows based environments.

  • The ESXi Dump Collector can be installed on the vCenter Server or on a separate server that has a network connection to the vCenter Server.
  • The ESXi Dump Collector does not support IPv6.
  • The product is on the same media as the vCenter Server
  • The installation is pretty straightforward. During the installation you can adjust parameters, like;
    • Location where to install
    • Server Port to be used, default is 6500.

Figure 12

The configuration of the ESXi hosts is outlined in the vSphere Installation and Setup Guide, Chapter 5 “Installing ESXi Using vSphere Auto Deploy”, Section “Configure ESXi Dump Collector with esxcli”, page 87.

One remarkable note from the documentation:

If you configure an ESXi system that is running inside a virtual machine that is using a vSphere standard switch, you must choose a VMkernel port that is in promiscuous mode. ESXi Dump Collector is not supported on vSphere distributed switches.

In this example, vmk0 is VMkernel NIC for management; 192.168.100.105 is the vCenter Server with ESXi Dump Collector installed.

# esxcli system coredump network set --interface-name=vmk0 --server-ipv4=192.168.100.105 --server-port=6500

# esxcli system coredump network set --enable=true

# esxcli system coredump network get
Enabled: true
Host VNic: vmk0
Network Server IP: 192.168.100.105
Network Server Port: 6500

After finishing, two questions remained?

The answer to both questions is in this discussion in the VMware Communities, thank you very much MattBr.

Figure 13 – Why you should always test you configuration…

Other references:

Setting up the ESXi 5.0 Dump Collector, from the VMware Blogs.

Advertisements

3 Responses to VCAP5-DCA Objective 6.1 – Configure, manage and analyse vSphere log files

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: