Usually when an issue arises in Eucalyptus, you can find information that points to the nature of the problem either in the Eucalyptus log files or in the system log files.
By default, the Eucalyptus log files are stored in /var/log/eucalyptus/ on each machine that hosts a Eucalyptus component. If Eucalyptus is installed somewhere other than the filesystem root (/), log files are stored in $EUCALYPTUS/var/log/eucalyptus/.
Here are the relevant logs for each component:
These components also include specialized developer log files. These are not relevant to troubleshooting a production system, and are not affected by any log level settings. These logs include the following:
You might also find helpful information about the nature of an issue in the system logs. In particular, the following logs may be relevant:
For the cluster controller and the node controller, log level is configured using the LOGLEVEL parameter in eucalyptus.conf. This parameter will be picked up dynamically when the value is changed in the config file, without requiring a restart of the component.
For all other components, the log level can be configured by passing an appropriate --log-level argument in the init script. It can also be dynamically changed using euca-modify-property to set an appropriate value for cloud.euca_log_level. This takes precedence over the value specified in the init script.
Valid log levels are as follows, from most to least verbose:
If no value is specified, the default INFO is used.
Eucalyptus logs now have a standard format, which varies slightly per log level.
For log levels FATAL, ERROR, WARN and INFO:
YYYY-MM-DD HH:MM:SS LEVEL | message
For log levels DEBUG and TRACE:
YYYY-MM-DD HH:MM:SS LEVEL PROCESS:THREAD loggingMethodOrClass | message
For log level EXTREME and ALL:
YYYY-MM-DD HH:MM:SS LEVEL PROCESS:THREAD loggingMethodOrClass FILENAME:LineNumber | message
Eucalyptus includes fault logs for easy identification of conditions outside of Eucalyptus's control that may cause it to fail. These messages are logged per component, and each fault is logged only once per component, in /var/log/eucalyptus/[component]-fault.log. The messages include a suggested resolution, and can be customized. Where they have been translated, Eucalyptus will use the system-configured LOCALE variable to serve appropriate messages.
Fault messages are based on XML-formatted templates, stored in a per-locale directory structure, with one file per fault message, and one file storing common strings. Default templates are shipped with Eucalyptus. These are stored in /usr/share/eucalyptus/faults/ as follows:
/usr/share/eucalyptus/faults/en_US/0001.xml ... /usr/share/eucalyptus/faults/en_US/1234.xml /usr/share/eucalyptus/faults/en_US/common.xml
Localized messages are located in a per-locale directory under /usr/share/eucalyptus/faults/. If localized messages are available matching the system LOCALE, Eucalyptus will use those messages. If no LOCALE is set, Eucalyptus defaults to en_US.
Set the system LOCALE in /etc/sysconfig/i18n as follows:
To use your own customized messages, copy the message files to the appropriate directory under /etc/eucalyptus/faults/ and edit them. Do not change the filenames. To test the fault template, run euca-generate-fault, providing the component name, fault ID, and any relevant parameters along with their values.
euca-generate-fault -c component_name fault_id [param] [value]
euca-generate-fault -c broker 1008 daemon ntpd
The test fault should be logged in the appropriate component fault log (in this case, /var/log/eucalyptus/broker-fault.log
Eucalyptus uses customized messages where they are available, preferring a non-localized custom message over a localized default message. Localized messages should be in a per-locale directory under /etc/eucalyptus/faults/, with a directory name that matches the system LOCALE. If no LOCALE is set, Eucalyptus defaults to en_US.