Red Hat ENTERPRISE LINUX 5.4 - SYSTEMTAP BEGINNERS GUIDE Informations techniques

SUSE LinuxEnterprise Serverwww.suse.com11 SP3October 16, 2014 System Analysis and Tuning Guide

x System Analysis and Tuning GuidePartIV, “Resource Management” (page113)Learn how to set up a tailor-made system fitting exactly the server'

Kernel Probes 87Kernel Probes6Kernel probes are a set of tools to collect Linux kernel debugging and performanceinformation. Developers and system

88 System Analysis and Tuning Guide6.1 Supported ArchitecturesKernel probes are fully implemented on the following architectures:• i386• x86_64 (AMD

Kernel Probes 896.2.2 JprobeJprobe is implemented through the kprobe mechanism. It is inserted on a function'sentry point and allows direct ac

90 System Analysis and Tuning Guideregister_jprobe()Inserts a break-point in the specified address. The address has to be the address ofthe first in

Kernel Probes 916.4.1 How to List Registered KernelProbesThe list of all currently registered kprobes is in the /sys/kernel/debug/kprobes/list fil

92 System Analysis and Tuning Guide• Thorough but more technically oriented information about kernel probes is in/usr/src/linux/Documentation/kprobe

Perfmon2—Hardware-Based Performance Monitoring 93Perfmon2—Hardware-Based PerformanceMonitoring7Perfmon2 is a standardized, generic interface to acc

94 System Analysis and Tuning GuideFigure7.1: Architecture of perfmon2-pfmonGenericperfmonArchitecture specificperfmonPMUCPU HardwareLinux Kernel

Perfmon2—Hardware-Based Performance Monitoring 951306604 CPU_OP_CYCLES_ALLThe following command gives the count of a specific function and the proc

About This Guide xiDeployment Guide (↑Deployment Guide)Shows how to install single or multiple systems and how to exploit the productinherent capab

96 System Analysis and Tuning GuideModel Processor9000/9100 (Montecito, Montvale)and GenericAMD X86 Opteron (K8, fam 10h)Intel X86 Intel P6 (Pentium

Perfmon2—Hardware-Based Performance Monitoring 97ALAT_CAPACITY_MISS_ALLALAT_CAPACITY_MISS_FPALAT_CAPACITY_MISS_INTBACK_END_BUBBLE_ALLBACK_END_BUBBL

98 System Analysis and Tuning Guidepfmon -v --system-wide...selected CPUs (2 CPU in set, 2 CPUs online): CPU0 CPU12 Delimit your session. The follow

Perfmon2—Hardware-Based Performance Monitoring 99pfmon --cpu-list=0-1 --system-wide -u -e CPU_OP_CYCLES_ALL,IA64_INST_RETIRED -- ls -l /dev/nullcrw

100 System Analysis and Tuning GuideThe data is located under /sys/kernel/debug/perfmon/ and organizedper CPU. Each CPU contains a set of metrics, a

Perfmon2—Hardware-Based Performance Monitoring 101File Descriptionis called (used for timeout-based setswitching)handle_work_count Number of timesp

102 System Analysis and Tuning GuideFile Descriptionreset_pmds_count Number of timespfm_reset_pmds() is calledset_switch_count Number of event set s

Perfmon2—Hardware-Based Performance Monitoring 103Chapter8, OProfile—System-Wide Profiler (page105)Consult this chapter for other performance opt

OProfile—System-Wide Profiler 105OProfile—System-WideProfiler8OProfile is a profiler for dynamic program analysis. It investigates the behaviour of

xii System Analysis and Tuning Guideage KVM with libvirt or QEMU. The guide also contains detailed informationabout requirements, limitations, and s

106 System Analysis and Tuning Guide8.2 Installation and RequirementsIn order to make use of OProfile, install the oprofile package. OProfile works

OProfile—System-Wide Profiler 107Applications usually do not need to profile the Kernel, so better use the --no-vm-linux option to reduce the amoun

108 System Analysis and Tuning GuideopreportOverflow stats not availableCPU: CPU with timer interrupt, speed 0 MHz (estimated)Profiling through time

OProfile—System-Wide Profiler 109 ---------- 0x4f: No unit mask BR_MISS_PRED_RETIRED: (counter: all)) number of mispredicted b

110 System Analysis and Tuning Guide8.6 Generating ReportsBefore generating a report, make sure OProfile has dumped your data to the /var/lib/oprofi

OProfile—System-Wide Profiler 111/usr/share/doc/packages/oprofile/oprofile.htmlContains the OProfile manual.http://developer.intel.com/Architecture

PartIV.ResourceManagement

General System Resource Management 115General System ResourceManagement9Tuning the system is not only about optimizing the kernel or getting the mo

About This Guide xiiiBugs and Enhancement RequestsFor services and support options available for your product, refer to http://www.suse.com/support

116 System Analysis and Tuning Guide• using separate disks for the operating system, the data, and the log files• placing a mail server's spool

General System Resource Management 117with network, no X). You will still be able to start graphical applications from remoteor use the startx comm

118 System Analysis and Tuning GuidesmbfsServices needed to mount SMB/CIFS file systems from a Windows server.splash / splash_earlyShows the splash

General System Resource Management 119To turn off access time updates, mount the file system with the noatime option. Todo so, either edit /etc/fst

Kernel Control Groups 121Kernel Control Groups10Kernel Control Groups (abbreviated known as “cgroups”) are a kernel feature that al-lows aggregatin

122 System Analysis and Tuning Guide• Every task running in the system is in exactly one of the cgroups in the hierarchy.10.2 ScenarioSee the follow

Kernel Control Groups 123Figure10.1: Resource PlanningMemoryProfessors (50%)Students (30%)System (20%)Disk I/OProfessors (50%)Students (30%)Syst

124 System Analysis and Tuning GuideWeb browsers such as Firefox will be part of the Web network class, while the NFSdaemons such as (k)nfsd will be

Kernel Control Groups 125# Create a child cgroup:mkdir /freezer/0# Put a task into this cgroup:echo $task_pid > /freezer/0/tasks# Freeze it:echo

xiv System Analysis and Tuning Guide• #amd64 em64t ipf: This paragraph is only relevant for the architectures amd64,em64t, and ipf. The arrows mark

126 System Analysis and Tuning Guide• Anonymous and file cache.• No limits for kernel memory.• Maybe in another subsystem if needed.For more informa

Kernel Control Groups 127Network Traffic (Resource Control)With cgroup_tc, a network traffic controller is available. It can be usedto manage traff

128 System Analysis and Tuning Guide10.4.2 Checking the EnvironmentThe kernel shipped with SUSE Linux Enterprise Server supports cgroups. There is n

Kernel Control Groups 129mount -t cgroup -ocpuset cpuset /sys/fs/cgroup/cpusetcd /sys/fs/cgroup/cpusetmkdir Charliecd Charlie# List of CPUs in this

130 System Analysis and Tuning Guidecd /sys/fs/cgroup/cpuset/cgroupmkdir prioritycd prioritycat cpu.shares2 Understanding cpu.shares:• 1024 is the d

Kernel Control Groups 131• /usr/src/linux/Documentation/cgroups/resource_counter.txt• For Linux Containers (LXC) based on cgroups, see Virtualizati

Power Management 133Power Management11Power management aims at reducing operating costs for energy and cooling systemswhile at the same time keepin

134 System Analysis and Tuning Guide11.1.1 C-States (Processor OperatingStates)Modern processors have several power saving modes called C-states. Th

Power Management 135Mode Definitionsor maintains all software-visiblestates, but may take longer to wake upthrough interrupts.C3 Stops all CPU inte

PartI.Basics

136 System Analysis and Tuning GuideC-states and P-states can vary independently of one another.11.1.3 T-States (Processor ThrottlingStates)T-states

Power Management 13711.2 The Linux Kernel CPUfreqInfrastructureProcessor performance states (P-states) and processor operating states (C-states) ar

138 System Analysis and Tuning GuideHowever, using this governor often does not lead to the expected power savingsas the highest savings can usually

Power Management 139/sys/devices/system/cpu/. If you list the contents of this directory, you willfind a cpu{0..x} subdirectory for each processor,

140 System Analysis and Tuning GuideMonitoring Power Consumption with powerTOP (page145)powerTOP combines various sources of information (analysis

Power Management 141 available frequency steps: 3.40 GHz, 2.80 GHz available cpufreq governors: conservative, userspace, powersave, ondemand, perfo

142 System Analysis and Tuning Guidecpupower -c 4 frequency-info (versus cpufreq-info -c 4)cpupower lets you also specify a list of CPUs with -c. Fo

Power Management 14311.3.2.2 Viewing and Modifying Kernel IdleStatistics with cpupowerThe idle-info subcommand shows the statistics of the cpuidle

144 System Analysis and Tuning GuideThe monitor subcommand allows you to execute performance benchmarks and tocompare Kernel statistics with hardwar

Power Management 145maximum CPU frequency the governor may select or to create a new governor. Withthe -c option, you can also specify for which of

146 System Analysis and Tuning Guide[...]Suggestion: Enable SATA ALPM link power management via:echo min_power > /sys/class/scsi_host/host0/lin

Power Management 14711.4.1 Tuning Options for P-StatesThe CPUfreq subsystem offers several tuning options for P-states: You can switch be-tween the

148 System Analysis and Tuning Guidecd /sys/devices/system/cpu/cpu0/cpufreq/conservative/2 Show the current value of ignore_nice_load with:cat ignor

Power Management 149Procedure11.2: Scheduling Processes on Cores1 Become root on a command line.2 To view the current value of sched_mc_power_sa

150 System Analysis and Tuning Guide3 Edit the settings in /etc/pm-profiler/testprofile/config and savethe file. You can also remove variables that

Power Management 151In case of a CPU upgrade, make sure to upgrade your BIOS, too. The BIOS needsto know the new CPU and its valid frequencies step

152 System Analysis and Tuning Guidebrary/l-cpufreq-3/?ca=dgr-lnxw03ReduceLXPWR-P1dth-LX&S_TACT=105AGX59&S_CMP=grlnxw03• The LessWatts.org p

PartV.Kernel Tuning

Installing Multiple Kernel Versions 155Installing Multiple KernelVersions12SUSE Linux Enterprise Server supports the parallel installation of multi

General Notes on System Tuning 3General Notes on SystemTuning1This manual discusses how to find the reasons for performance problems and pro-vides

156 System Analysis and Tuning Guidefer to Section“The File /etc/sysconfig/bootloader” (Chapter10, TheBoot Loader GRUB, ↑Administration Guide).12.

Installing Multiple Kernel Versions 1572.6.32.12-0.7 : keep the kernel with the specified version numberlatest: keep the kernel with the highes

158 System Analysis and Tuning GuideFigure12.1: The YaST Software Manager - Multiversion View3 Select a package and open its Version tab in the b

Installing Multiple Kernel Versions 159S | Name | Type | Version | Arch | Repository --+----------------+---------

Tuning I/O Performance 161Tuning I/O Performance13I/O scheduling controls how input/output operations will be submitted to storage.SUSE Linux Enter

162 System Analysis and Tuning Guideecho SCHEDULER > /sys/block/DEVICE/queue/schedulerwhere SCHEDULER is one of cfq, noop, or deadline and DEVICE

Tuning I/O Performance 163/sys/block/<device>/queue/iosched/quantumThis option limits the maximum number of requests that are being processed

164 System Analysis and Tuning Guideseveral parallel readers from a SAN and for databases (especially when using “TCQ”disks). The DEADLINE scheduler

Tuning the Task Scheduler 165Tuning the Task Scheduler14Modern operating systems, such as SUSE® Linux Enterprise Server, normally runmany different

4 System Analysis and Tuning Guidescription. If you plan to tune your Web server for faster delivery of static pages, forexample, it makes a differe

166 System Analysis and Tuning Guide14.1.1 PreemptionThe theory behind task scheduling is very simple. If there are runnable processes ina system, a

Tuning the Task Scheduler 167One approach is to classify a process either I/O-bound or processor-bound.I/O-boundI/O stands for Input/Output devices

168 System Analysis and Tuning GuideThe scheduler calculates the timeslices dynamically. However, to determine the ap-propriate timeslice is a compl

Tuning the Task Scheduler 169As a result, CFS brings more optimized scheduling for both servers and desktops.14.4.1 How CFS WorksCFS tries to guara

170 System Analysis and Tuning Guide14.4.3 Kernel Configuration OptionsBasic aspects of the task scheduler behavior can be set through the kernel co

Tuning the Task Scheduler 171SCHED_BATCHScheduling policy designed for CPU-intensive tasks.SCHED_IDLEScheduling policy intended for very low priori

172 System Analysis and Tuning Guidesaturn.example.com:~ # chrt -b -p 0 16244saturn.example.com:~ # chrt -p 16244pid 16244's current scheduling

Tuning the Task Scheduler 173A list of the most important task scheduler sysctl tuning variables (located at /proc/sys/kernel/) with a short descri

174 System Analysis and Tuning GuideThis value also specifies the maximum amount of time during which a sleepingtask is considered to be running for

Tuning the Task Scheduler 175SCHED_OTHER policy, they will all be run on the same processor. The de-fault value is 32. Increasing this value gives

General Notes on System Tuning 5analysis, the Linux kernel offers means to perform such analysis. See PartIII, “KernelMonitoring” (page69) for co

176 System Analysis and Tuning Guiderunnable tasks: task PID tree-key switches prio exec-runtime sum-exec sum-sleep-----------------------

Tuning the Task Scheduler 177• General information on Linux task scheduling is described in Inside the Lin-ux scheduler [http://www.ibm.com/develop

Tuning the Memory Management Subsystem 179Tuning the MemoryManagement Subsystem15In order to understand and tune the memory management behavior of

180 System Analysis and Tuning GuideFinally, the workload itself should be examined and tuned as well. If an application isallowed to run more proce

Tuning the Memory Management Subsystem 181as inode tables, allocation bitmaps, and so forth. Buffercache can be reclaimed simi-larly to pagecache.1

182 System Analysis and Tuning Guide15.1.7.2 Directory Entry CacheThis is an in-memory cache of the directory entries in the system. These contain a

Tuning the Memory Management Subsystem 18315.2.3 Memory Controller (MemoryCgroups)If the memory cgroups feature is not needed, it can be switched o

184 System Analysis and Tuning Guide/proc/sys/vm/vfs_cache_pressureThis variable controls the tendency of the kernel to reclaim the memory which isu

Tuning the Memory Management Subsystem 185Similar percentage value as above. When this is exceeded, applications that wantto write to the pagecache

System Analysis and Tuning GuideCopyright © 2006–2014 SUSE LLC and contributors. All rights reserved.Permission is granted to copy, distribute and/or

186 System Analysis and Tuning Guideti-socket servers are NUMA machines. NUMA is a secondary concern to managingswapping and caches in terms of perf

Tuning the Network 187Tuning the Network16The network subsystem is rather complex and its tuning highly depends on the systemuse scenario and also

188 System Analysis and Tuning GuideThe special files in the /proc file system can modify the size and behavior of kernelsocket buffers; for general

Tuning the Network 189/proc/sys/net/ipv4/tcp_sackSelect acknowledgments (SACKS).Use sysctl to read or write variables of the /proc file system. sys

190 System Analysis and Tuning GuideWhen the kernel queue becomes full, all new packets are dropped, causing exist-ing connections to fail. The &apo

PartVI.HandlingSystem Dumps

Tracing Tools 193Tracing Tools17SUSE Linux Enterprise Server comes with a number of tools that help you obtain use-ful information about your syste

194 System Analysis and Tuning GuideTo run a new command and start tracing its system calls, enter the command to bemonitored as you normally do, an

Tracing Tools 195The -e option understands several sub-options and arguments. For example, to traceall attempts to open or write to a particular fi

PartII.System Monitoring

196 System Analysis and Tuning Guidebrk(0) = 0x69e000mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMO

Tracing Tools 19717.2 Tracing Library Calls withltraceltrace traces dynamic library calls of a process. It is used in a similar way tostrace, and m

198 System Analysis and Tuning Guideclock_gettime(1, 0x7fff4b5c34d0, 0, 0, 0) = 0clock_gettime(1, 0x7fff4b5c34c0, 0xffffffffff60018

Tracing Tools 199• ppc64• System z17.3.3 General InformationThe main advantage of Valgrind is that it works with existing compiled executables.You

200 System Analysis and Tuning GuidemassifA heap profiler. Heap is an area of memory used for dynamic memory allocation.This tool helps you tune you

Tracing Tools 201grind, it also detects errors in associated libraries (like C, X11, or Gtk libraries). Be-cause you probably do not need these err

202 System Analysis and Tuning Guide1. By default, Valgrind sends its messages to the file descriptor 2, which is the stan-dard error output. You ca

Tracing Tools 203sages. In case of a duplicate error, it is recorded but no message is shown. This mech-anism prevents you from being overwhelmed b

kexec and kdump 205kexec and kdump18kexec is a tool to boot to another kernel from the currently running one. You can per-form faster system reboot

206 System Analysis and Tuning Guide• Booting without GRUB or LILO configurationWhen the system boots a kernel with kexec, it skips the boot loader

kexec and kdump 207The capture kernel is loaded to the reserved area and waits for the kernel to crash.Then kdump tries to invoke the capture kerne

208 System Analysis and Tuning Guide6 Remount the root file system in read-only mode:mount -o remount,ro /7 Initiate the reboot of the kernel that y

kexec and kdump 20918.6 Basic kdump ConfigurationYou can use kdump to save kernel dumps. If the kernel crashes, it is useful to copythe memory imag

210 System Analysis and Tuning Guide18.6.1 Manual kdump Configurationkdump reads its configuration from the /etc/sysconfig/kdump file. To makesure t

kexec and kdump 2112 Enable kdump init script:chkconfig boot.kdump on3 You can edit the options in /etc/sysconfig/kdump. Reading the commentswill h

212 System Analysis and Tuning GuideDo not reset the computer because kdump always needs some time to com-plete its task.18.6.2 YaST ConfigurationIn

kexec and kdump 213In the Dump Target window, select the type of the dump target and the URL whereyou want to save the dump. If you selected a netw

214 System Analysis and Tuning GuideThe first parameter represents the kernel image. The second parameter is the dumpfile captured by kdump. You can

kexec and kdump 215If you decide to analyze the dump on another machine, you must check both the ar-chitecture of the computer and the files necess

System Monitoring Utilities 9System Monitoring Utilities2There are number of programs, tools, and utilities which you can use to examine thestatus

216 System Analysis and Tuning GuideThis program has absolutely no warranty. Enter "help warranty" for details.GNU gdb 6.1Copyright 2004

kexec and kdump 217 RIP: 00007fa958991f60 RSP: 00007fff61330390 RFLAGS: 00010246 RAX: 0000000000000001 RBX: ffffffff8020bfbb RCX: 0000000

218 System Analysis and Tuning GuideYou can change the directory for the kernel dumps with the KDUMP_SAVEDIRoption. Keep in mind that the size of ke

kexec and kdump 219• For more details on kdump specific to SUSE Linux, see http://ftp.suse.com/pub/people/tiwai/kdump-training/kdump-training.pdf .

GNU LicensesAThis appendix contains the GNU Free Documentation License version 1.2.GNU Free Documentation LicenseCopyright (C) 2000, 2001, 2002 Free S

222 System Analysis and Tuning GuideThe "Cover Texts" are certain short passages of text that are listed, as Front-Cover Texts or Back-Cov

GNU Licenses 223A.Use in the Title Page (and on the covers, if any) a title distinct from that of the Document, and from those of previous versions

224 System Analysis and Tuning Guide6. COLLECTIONS OF DOCUMENTSYou may make a collection consisting of the Document and other documents released und

GNU Licenses 225 with the Invariant Sections being LIST THEIR TITLES, with the Front-Cover Texts being LIST, and with the Back-Cover Texts bein

10 System Analysis and Tuning Guideof the system at a glance. Use these tools first in order to get an overview and find outwhich part of the system

System Monitoring Utilities 11bShows the number of processes waiting for a resource other than a CPU. A highnumber in this column may indicate an I

12 System Analysis and Tuning GuideinInterrupts per second. A high value indicates a high I/O level (network and/ordisk).csNumber of context switche

System Monitoring Utilities 13the fly or query existing reports gathered by the system activity data collector (sadc).sar and sadc both gather all

14 System Analysis and Tuning Guideand count. If filename, interval and count are not specified, sar attempts to generatea report from /var/log/sa/s

System Monitoring Utilities 1516:12:32 259320 1796356 87.39 20808 72660 2229080 62.0616:12:42 381096 1674580 81.46 21

ContentsAbout This Guide ix1 Available Documentation ... x2 Feedback ...

16 System Analysis and Tuning Guide16:28:51 scd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0016:28:51 DEV tps rd_sec/s w

System Monitoring Utilities 172.2 System Information2.2.1 Device Load Information: iostatiostat monitors the system device loading. It generates re

18 System Analysis and Tuning GuideWith the -P option, you can specify the number of processors to be reported (notethat 0 is the first processor).

System Monitoring Utilities 192.2.4 Kernel Ring Buffer: dmesgThe Linux kernel keeps certain messages in a ring buffer. To view these messages,enter

20 System Analysis and Tuning Guideuseful. However, the list of all files can be combined with search functions to gener-ate useful lists. For examp

System Monitoring Utilities 21UEVENT[1138806687] add@/devices/pci0000:00/0000:00:1d.7/usb4/4-2/4-2.2UEVENT[1138806687] add@/devices/pci0000:00/0000

22 System Analysis and Tuning Guide0x00000000 83984391 tux 666 282464 20x00000000 84738056 root 644 151552 2

System Monitoring Utilities 23 3 0 [events/0] 4 0 [khelper] 5 0 [kthread] 11 0 [kblockd/0] 12 0 [kacpid] 472

24 System Analysis and Tuning Guide | `-hald-addon-stor |-kded |-kdeinit-+-kdesu---su---kdesu_stub---yast2---y2controlcenter |

System Monitoring Utilities 25 1746 root 15 0 0 0 0 S 0.0 0.0 0:00.00 w1_control 1752 root 15 0 0 0 0 S 0.0

2.8 Files and File Systems ... 402.9 User Information ...

26 System Analysis and Tuning GuideOutput for the sys_list window under LPAR:12:30:48 | CPU-T: IFL(18) CP(3) UN(3) ?=helpsystem #cpu cpu m

System Monitoring Utilities 278 IFL 0.00 0.00 | |9 IFL 0.00 0.00 |

28 System Analysis and Tuning Guide2.3.7 Modify a process' niceness: niceand reniceThe kernel determines which processes require more CPU time

System Monitoring Utilities 29Swap: 2104472 0 2104472The options -b, -k, -m, -g show the output in bytes, KB, MB, or GB, respectiv

30 System Analysis and Tuning GuideHugePages_Rsvd: 0HugePages_Surp: 0Hugepagesize: 2048 kBDirectMap4k: 2689024 kBDirectMap2M

System Monitoring Utilities 31MappedMemory claimed with the mmap system callSlabKernel data structure cacheSReclaimableReclaimable slab caches (ino

32 System Analysis and Tuning Guide UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:68562268 errors:0 dropped:460981

System Monitoring Utilities 332.5.3 Show the Network Status: netstatnetstat shows network connections, routing tables (-r), interfaces (-i), masque

34 System Analysis and Tuning Guide TCPAbortOnLinger: 0 TCPAbortFailed: 0 TCPMemoryPressures: 02.5.4 Interactive Network Monitor: iptrafThe

System Monitoring Utilities 35Mon Mar 23 10:08:02 2010; UDP; eth0; 107 bytes; from 192.168.1.192:33157 to \ 239.255.255.253:427Mon Mar 23 10:08:02

6 Kernel Probes 876.1 Supported Architectures ... 886.2 Types of Kernel Probes

36 System Analysis and Tuning Guide 10: 0 XT-PIC uhci_hcd:usb3 11: 71772 XT-PIC uhci_hcd:usb2, eth0 12: 101150

System Monitoring Utilities 37-rw-r--r-- 1 tux users 0 2007-07-16 17:04 oom_adj-r--r--r-- 1 tux users 0 2007-07-16 17:04 oom_scorelrwxrwxrwx 1 tux

38 System Analysis and Tuning Guidesw irq: 1:15:35.25 0.4% swap in : 69idle : 9d 16:07:56.79 73.8% swap out: 209uptime:

System Monitoring Utilities 392.7 Hardware Information2.7.1 PCI Resources: lspciNOTE: Accessing PCI configuration.Most operating systems require ro

40 System Analysis and Tuning Guide Capabilities: [dc] Power Management version 2 Capabilities: [e4] PCI-X non-bridge device Kernel driver in use

System Monitoring Utilities 41tux@mercury:~> file /usr/share/man/man1/file.1.gz/usr/share/man/man1/file.1.gz: gzip compressed data, from Unix, m

42 System Analysis and Tuning Guidetux@mercury:~> du -sh /opt192M /opt2.8.3 Additional Information about ELFBinariesRead the content of binari

System Monitoring Utilities 43 File: "/etc/profile" ID: d4fb76e70b4d1746 Namelen: 255 Type: ext2/ext3Block size: 4096 Funda

44 System Analysis and Tuning Guide2.10.1 Time Measurement with timeDetermine the time spent by commands with the time utility. This utility is avai

System Monitoring Utilities 45As mentioned above, RRDtool is designed to work with data that change in time. Theideal case is a sensor which repeat

9.3 File Systems and Disk Access ... 11810 Kernel Control Groups 12110.1 Technical Overv

46 System Analysis and Tuning Guide sleep $INTERVAL echo "rrdtool update free_mem.rrd $DATE:$FREEMEM"donePoints to Notice• The time

System Monitoring Utilities 47rrdtool create free_mem.rrd --start 1272974834 --step=4 \DS:memory:GAUGE:600:U:U RRA:AVERAGE:0.5:1:24Points to Notice

48 System Analysis and Tuning Guide2.11.2.4 Viewing Measured ValuesWe have already measured the values, created the database, and stored the measure

System Monitoring Utilities 49Points to Notice• free_mem.png is the filename of the graph to be created.• --start and --end limit the time range wi

50 System Analysis and Tuning GuideApart form RRDtool's man page (man 1 rrdtool) which gives you only ba-sic information, you should have a loo

Monitoring with Nagios 51Monitoring with Nagios3Nagios is a stable, scalable and extensible enterprise-class network and system moni-toring tool wh

52 System Analysis and Tuning GuideFor further information on how to install packages see:• Section“Using Zypper” (Chapter6, Managing Software wit

Monitoring with Nagios 533.3.1 Object Definition FilesIn addition to those configuration files Nagios comes with very flexible and highlycustomizab

54 System Analysis and Tuning Guidewith the max_check_attempts directive. All configuration flags beginning withnotification handle how Nagios shoul

Monitoring with Nagios 55the person who is contacted on a failure of a service. Usually this is the responsibleadministrator. use inherits configur

14 Tuning the Task Scheduler 16514.1 Introduction ... 16514.2 Pro

56 System Analysis and Tuning Guide max_check_attempts 10 contact_groups admins notification_interval 60 notification_options d,u,r}

Monitoring with Nagios 573.4.2 Monitoring Remote Host-Resourceswith NagiosThis section explains how to monitor remote host resources with Nagios.Pr

58 System Analysis and Tuning Guide5 Execute rcnagios restart and rcnsca restart.Proceed as follows on the client you want to monitor:Procedure3.3:

Monitoring with Nagios 59(Return code of 127 is out of bounds - plugin may bemissing)Make sure that you have installed nagios-plugins.E-mail notifi

Analyzing and Managing System Log Files 61Analyzing and ManagingSystem Log Files4System log file analysis is one of the most important tasks when a

62 System Analysis and Tuning GuideauditLogs from the audit framework. See Part“The Linux AuditFramework” (↑Security Guide) for details.boot.msgLog

Analyzing and Managing System Log Files 63mail*Mail server (postfix, sendmail) logs.messagesThis is the default place where all Kernel and system l

64 System Analysis and Tuning GuideXorg.0.logX startup log file. Refer to this in case you have problems starting X. Copies fromprevious X starts ar

Analyzing and Managing System Log Files 65logrotate is usually run as a daily cron job. It does not modify any log files morethan once a day unless

18.2 Required Packages ... 20618.3 kexec Internals ...

66 System Analysis and Tuning Guide4.4 Monitoring Log Files withlogwatchlogwatch is a customizable, pluggable log-monitoring script. It parses syste

Analyzing and Managing System Log Files 67ignore.confFilter for all lines that should globally be ignored by logwatch.services/*.confThe service di

PartIII.Kernel Monitoring

SystemTap—Filtering and Analyzing System Data 71SystemTap—Filtering andAnalyzing System Data5SystemTap provides a command line interface and a scri

72 System Analysis and Tuning Guide5.1.1 SystemTap ScriptsSystemTap usage is based on SystemTap scripts (*.stp). They tell SystemTap whichtype of in

SystemTap—Filtering and Analyzing System Data 735.1.3 Commands and PrivilegesThe main commands associated with SystemTap are stap and staprun. To e

74 System Analysis and Tuning Guide/usr/share/systemtap/tapset/Holds the standard library of tapsets./usr/share/doc/packages/systemtap/examplesHolds

SystemTap—Filtering and Analyzing System Data 75• kernel-*-devel• kernel-source-*• gccTo get access to the man pages and to a helpful collection of

About This GuideSUSE Linux Enterprise Server is used for a broad range of usage scenarios in enter-prise and scientific data centers. SUSE has ensured

76 System Analysis and Tuning GuideIn case any error messages appear during the test, check the output for hints about anymissing packages and make

SystemTap—Filtering and Analyzing System Data 77 exit ()}Start of the probe.Event begin (the start of the SystemTap session).Start of the handler

78 System Analysis and Tuning Guide• Synchronous events: Occur when any process executes an instruction at a particu-lar location in kernel code. Th

SystemTap—Filtering and Analyzing System Data 79by a name. They take any number of string or numeric arguments (by value) and mayreturn a single st

80 System Analysis and Tuning Guidetid()ID of the current thread.pid()Process ID of the current thread.uid()ID of the current user.cpu()Current CPU

SystemTap—Filtering and Analyzing System Data 81For more information about supported SystemTap functions, refer to the stap-funcs man page.5.3.3.2

82 System Analysis and Tuning Guidehardware platform). With the global statement it is possible to use the variablescount_jiffies and count_ms also

SystemTap—Filtering and Analyzing System Data 83!=: Is not equal to>=: Is greater than or equal to<=: Is less than or equal to5.4 Examp

84 System Analysis and Tuning Guide• IP address from which the TCP connection originated (IP_SOUCE)To run the script, executestap/usr/share/doc/pac

SystemTap—Filtering and Analyzing System Data 855.6 For More InformationThis chapter only provides a short SystemTap overview. Refer to the followi