FEATURE PREVIEW: The multipathing feature is
not yet complete, and may change or be removed from future releases.
It is included in this release so that users can try it out and
The Eucalyptus EMC VNX Multipathing feature requires the following
to function properly:
Properly installed and configured Linux Device Mapper
Multipathing software on both the Storage Controller and Node
Correctly configured iSCSI path system property and related
STORAGE_INTERFACES parameters in the “eucalyptus.conf”
configuration file for both SC and NC.
Prerequisites for Troubleshooting Typical Multipathing
Before you start diagnosing the problems with
multipathing, make sure you set the proper logging level
on both SC and NC machines, so that you can get detailed
failure logs. To do that:
Set the “cloud.euca_log_level” system property to
Uncomment the “LOGLEVEL=DEBUG” entry in the
“eucalyptus.conf” file on the NC, and then restart
the NC service
General Troubleshooting Techniques for Multipathing
The following are general tips to help diagnose
Make sure you turn on the DEBUG log level for both
SC and NC so that you can get detailed information
from the logs.
Eucalyptus calls some external Perl scripts to
perform the actual iSCSI connect/disconnect
operations. These scripts are:
The STDERR output of these scripts is logged; you
can add debug code to print information to
STDERR to see what happens during connection or
The iscsiadm open-iscsi
initiator command line tool can help you get the
current status of all the iSCSI connections in the
system. For example:
iscsiadm -m session -P 3
Use the multipath command line tool to see
multipathing status. For example:
multipath -ll -v 3
Cannot attach volumes
This can occur for a number of reasons. To diagnose this,
try some of the following:
Make sure you can attach a volume without using
Check your SAN-related system properties to see if
you have set the correct values.
Use a single path for the NC; for example, set
“PARTITION.storage.ncpaths” to something like
“192.168.25.182”. If you specify an iface in your
path, like “iface0:192.168.25.182”, also make sure
you have “iface0” defined with “STORAGE_INTERFACES”
in “eucalyptus.conf” configuration file on the NC.
If you have no problem attaching a volume with a
single path, the failure may be due to the incorrect
state of the Linux device mapper multipathing tool.
Check if the “multipathd” service is running on the
NC hosts and if “/etc/multipath.conf” is installed
and configured properly (for example, copy the
example configuration provided by Eucalyptus).
Remember to set “user_friendly_names” to “yes” in
“/etc/multipath.conf”. You can try restarting
“multipathd” and/or reloading “/etc/multipath.conf”
if you changed it previously. Run “multipath -ll” on
NC host and see if it returns reasonable output
without any error.
Check that the “PARTITION.storage.ncpaths”
configuration file entries are correct. A typo can
cause volume attach failures.
Make sure that the networking configuration is
correct for the NC hosts. If you set the paths
without specific ifaces, check to see if you can
connect to each IP in the path using default network
interface; otherwise, check each path’s connectivity
using a specific network interface.
Check network connectivity with all of the
Check the “nc.log” log file for the string
“connect_iscsitarget”. Examine the return results,
especially the “stderr” output.
Not all paths are connected
Sometimes when you run “multipath -ll” on NC hosts after
attaching a volume, you find that the multipath device
does not have all of the paths connected. In this case,
the problem could be due to one of the following:
There is a mistake in the paths in one of the
“PARTITION.storage.ncpaths” entries. If one of the
paths specified in the system property is wrong,
then it is possible that the specific path can not
be connected. Make sure you have all the paths
The missing paths are not valid networking paths,
or have networking issues. For example, when you
ignore the iface part of a path, are you sure that
the destination of the path (the IP part of the
path) can be connected via the default network
interface? Or if you specified the iface, are you
sure you defined the iface in the “eucalyptus.conf”
configuration file, and that the destination can be
connected with the specified network interface?
If the paths specified are all valid, but some of
them do not have connectivity, try to ping each of
the specified paths on the NC hosts to check for
connectivity. If there are connectivity issues,
contact your network administrator.
The Eucalyptus Storage Controller needs to attach a
volume on the machine it runs so it can upload to Walrus
during an EC2 snapshot call. To help ensure maximum
reliability for snapshotting, you should use
multipathing for the SC host; this is configured with
the “PARTITION.storage.scpaths” system property. When
multipathing is enabled for the SC, if you see a
snapshot failure, it may be caused by multipathing.
Techniques for diagnosing SC multipathing failures is
similar to those used for NC multipathing failures. In
the case of SC multipathing failures, the logs are in
“/var/log/eucalyptus/cloud-*.log”, not “nc.log”, since
the iSCSI connect scripts are invoked by Java code.