Isilon - Statistiken

Cluster Übersicht

# isi statistics pstat
___________________________NFS Operations Per Second___________________________
null               0.00/s  getattr        10547.28/s  setattr           33.42/s
lookup           541.25/s  access          1940.87/s  readlink           0.00/s
read              73.72/s  write            158.96/s  create            10.34/s
mkdir              1.08/s  symlink            0.00/s  mknod              0.00/s
remove             4.09/s  rmdir              0.00/s  rename             6.25/s
link               0.00/s  readdir            3.44/s  readdirplus      166.48/s
statfs             1.08/s  fsinfo             0.00/s  pathconf           0.00/s
commit            82.27/s  noop               0.00/s
TOTAL          13570.53/s
 
___CPU Utilization___                                     _____OneFS Stats_____
user             0.5%                                     In          3.05 MB/s
system          19.3%                                     Out         1.27 MB/s
idle            78.3%                                     Total       4.32 MB/s
 
____Network Input____        ____Network Output___        _______Disk I/O______
MB/s             2.51        MB/s             2.78        Disk     2667.00 iops
Pkt/s        10475.00        Pkt/s        10475.60        Read       11.46 MB/s
Errors/s         0.00        Errors/s         0.00        Write       2.22 MB/s

Node Übersicht

# isi statistics system --nodes --oprates
 Node   CPU  CIFS   FTP  HTTP ISCSI    NFS Total  NetIn NetOut DiskIn DiskOut
  LNN %Used Ops/s Ops/s Ops/s Ops/s  Ops/s Ops/s    B/s    B/s    B/s     B/s
  All 31.8*  2.2*  0.0*  0.0*  0.0*   13K*   13K  3.3M*  8.9M*  2.2M*    12M*
    4  27.4   0.0   0.0   0.0   0.0   5.9K  5.9K   1.2M   4.3M   588K    3.6M
    5  45.9   0.0   0.0   0.0   0.0   3.5K  3.5K   1.2M   2.9M   803K    4.7M
    6  22.2   2.2   0.0   0.0   0.0   3.5K  3.5K   927K   1.7M   802K    3.9M
 
# isi statistics system --nodes
 Node   CPU   CIFS   FTP  HTTP ISCSI    NFS Total  NetIn NetOut DiskIn DiskOut
  LNN %Used    B/s   B/s   B/s   B/s    B/s   B/s    B/s    B/s    B/s     B/s
  All 31.8* 426.1*  0.0*  0.0*  0.0*  9.1M*  9.1M  3.3M*  8.9M*  2.2M*    12M*
    4  24.5    0.0   0.0   0.0   0.0   3.8M  3.8M   1.5M   4.2M   1.8M    3.6M
    5  33.1    0.0   0.0   0.0   0.0   2.7M  2.7M   1.1M   2.3M   803K    4.7M
    6  16.7  426.1   0.0   0.0   0.0   2.7M  2.7M   1.2M   1.8M   802K    3.9M

Clients / Pfade

Client Stats

# isi statistics client --totalby=RemoteAddr --nodes=all --orderby=Ops
      RemoteAddr        Ops         In        Out    TimeAvg       Node      Proto            Class   UserName        LocalName       RemoteName
                        N/s        B/s        B/s         us
    10.100.10.10       2.0K       307K       237K      300.9          *          *                *          *                *                *
    10.100.10.20       2.0K       305K       237K      293.2          *          *                *          *                *                *
    10.100.10.30      814.0       107K        95K       55.9          *          *                *          *                *                *
    10.100.10.40      639.6       136K        98K      137.7          *          *                *          *                *                *
    10.100.10.50      518.5        82K        98K       72.3          *          *                *          *                *                *
...

Pfad IOPS

# isi statistics heat --nodes=all --pathdepth=3 --totalby=Path
       Ops  Node      Event           Class Path
       N/s
      1.6K     *          *               * /ifs/data/Customers/share1001/
     993.2     *          *               * /ifs/data/Customers/share1002/
     848.4     *          *               * /ifs/data/Customers/share1003/
     723.5     *          *               * /ifs/data/Customers/share1004/
     668.6     *          *               * /ifs/data/Customers/share1005/
...

Operationen

# isi statistics protocol --totalby=Class --nodes=all
   Ops    In   Out TimeAvg Node Proto           Class Op
   N/s   B/s   B/s      us
  17.9  3.2K  4.9K  4464.5    *     *          create  *
  15.0  2.1K  2.2K 12101.9    *     *          delete  *
   0.7   0.0   0.0   295.8    *     *      file_state  *
  9.5K  1.2M  1.5M  2642.2    *     *  namespace_read  *
  63.3   11K   11K  3654.0    *     * namespace_write  *
  87.0   13K   14K   733.8    *     *           other  *
  65.4  9.1K  1.1M 19858.2    *     *            read  *
 160.9  781K   26K  1784.8    *     *           write  *

Hardware

Disk IOPS pro Platte/Node

# isi statistics query -n all --stats=node.disk.xfers.rate.0,node.disk.xfers.rate.1,node.disk.xfers.rate.2,node.disk.xfers.rate.3,node.disk.xfers.rate.4,node.disk.xfers.rate.5,node.disk.xfers.rate.6,node.disk.xfers.rate.7,node.disk.xfers.rate.8,node.disk.xfers.rate.9,node.disk.xfers.rate.10,node.disk.xfers.rate.11
  NodeID node.disk.xfers.rate.0 node.disk.xfers.rate.1 node.disk.xfers.rate.2 node.disk.xfers.rate.3 node.disk.xfers.rate.4 node.disk.xfers.rate.5 node.disk.xfers.rate.6 node.disk.xfers.rate.7 node.disk.xfers.rate.8 node.disk.xfers.rate.9 node.disk.xfers.rate.10 node.disk.xfers.rate.11
       4                   73.8                   79.0                   79.2                   80.8                   75.0                  101.8                   83.2                   73.8                   81.4                   80.0                    73.0                    75.0
       5                   97.2                   80.4                   88.6                   80.6                   90.8                  108.4                  113.6                   96.8                   85.2                   94.6                    74.0                    93.4
       6                   73.4                   77.0                   86.6                   74.8                   73.4                   67.8                   76.0                   71.6                   75.6                   69.6                   109.6                    66.4
 average                   81.5                   78.8                   84.8                   78.7                   79.7                   92.7                   90.9                   80.7                   80.7                   81.4                    85.5                    78.3

Disk IOPS pro Node

# isi statistics query -n all --stats=node.disk.xfers.rate.sum,node.disk.bytes.in.rate.sum,node.disk.bytes.out.rate.sum
  NodeID node.disk.xfers.rate.sum node.disk.bytes.in.rate.sum node.disk.bytes.out.rate.sum
       4                   1181.0                   3084902.4                    4338073.6
       5                   1437.2                   3077222.4                    5448908.8
       6                    981.2                   3675238.4                    3612979.2
 average                   1199.8                   3279121.1                    4466653.9

Disk Queue

# isi_for_array -s 'sysctl -aN hw.iosched.|grep total_inqueue|xargs sysctl|xargs|sed -e "s/hw.iosched.//g" -e "s/.total_inqueue//g"'
isi-san-sxb-4: ad0: 0 da1: 0 da2: 0 da3: 0 da4: 0 da5: 2 da6: 0 da7: 0 da8: 0 da9: 0 da10: 1 da11: 0 da12: 0
isi-san-sxb-5: ad0: 0 da1: 0 da2: 0 da3: 0 da4: 0 da5: 0 da6: 0 da7: 0 da8: 0 da9: 0 da10: 0 da11: 0 da12: 0
isi-san-sxb-6: ad0: 0 da1: 0 da2: 0 da3: 0 da4: 0 da5: 0 da7: 0 da8: 0 da9: 0 da10: 0 da11: 0 da12: 0 da6: 1

Ethernet Bandbreite

# isi statistics query -n all --stats=node.net.ext.bytes.in.rate,node.net.ext.bytes.out.rate
  NodeID node.net.ext.bytes.in.rate node.net.ext.bytes.out.rate
       4                  1699933.4                    969937.8
       5                  8814418.6                   1081392.4
       6                   771120.0                   1375492.9
 average                  3761824.0                   1142274.4

Infiniband Bandbreite

# isi statistics query -n all --stats=node.net.int.bytes.in.rate,node.net.int.bytes.out.rate
  NodeID node.net.int.bytes.in.rate node.net.int.bytes.out.rate
       4                    98977.6                     17560.0
       5                  1979862.8                   8824409.4
       6                  3528272.4                   3910889.8
 average                  1869037.6                   4250953.1

Sonstiges

Statistik Parameter

Namen der Statistikwerte. Diese sind vorallem für “isi statistics history” und “isi statistics query” erforderlich.

# isi statistics list stats
cluster.cpu.count
cluster.cpu.idle.avg
cluster.cpu.idle.max
cluster.cpu.intr.avg
cluster.cpu.intr.max
cluster.cpu.nice.avg
cluster.cpu.nice.max
cluster.cpu.sys.avg
cluster.cpu.sys.max
...

Statistik History

Gibt die Werte für die einzelnen Statistik-Parameter für die Vergangenheit zurück. Zur Auswertung sollte man sich die weiteren Parameter des Befehles anschauen.

# isi statistics history --stats=node.net.iface.packets.in.rate.0
1326882436                    4022.89160156
1326882556                    3698.16259766
1326882676                    3991.22509766
1326882796                    3127.85009766
1326882916                    3589.77514648
1326883156                    3292.33325195
...