DevOps

Sunday, 27 October 2013

Scenario Based Questions & Answers

Scenario :1
If the Unix server is not accessible(Mean to say not able to ssh/telnet the server) then what are things you will check it in first shot.

Answer:
1.Try to ping the server to check whether server is responding or not.If not responds,then login to console for further troubleshooting.
2. Login to hardware console(ILOM or XSCF) and try to access the OS from there(console -d 0 ).If server is up ,then login to server and check the IP address flags(ifconfig -a) and  router connectivity. 
3.If the address flag is "up; running" and router connectivity also fine ,then check the ssh/telnet services.(Ex: svcs -xv)
4.If everything is fine ,then check the system load.("w" or "uptime")
5.You will get fork errors in OS console if the system is running out resource.In these cases,you may not able to access the system and you need to free up some memory and reduce CPU usage.

 Scenario :2
Application team is requesting you to reboot the Solaris global zone.Will you reboot it immediately  ? will you wait for stakeholders approval ?

Answer:
1.You should not reboot the server without proper stake holders approvals.You need to involve incident management team and SDM prior to rebooting the server. 
2.You need to snooze the monitoring alerts from monitoring system to stop false alerts being generated while rebooting .
3.If any cluster service is running ,please make sure you are halting the cluster properly.
4.Last but not least ,you should take all the configuration backup prior to the system rebooting.
5.Its better to have root disk details (eeprom) prior to rebooting it.

 Scenario :3
Application team is complaining about system performance issue.What are the things you check in the first shot ?

Answer:
1.Run vmstat command to identify the system bottle neck. 
2.Run iowait command to determine the disk I/O issue.
3.Check the NIC card status (dladm show-dev).If any link is down,then we need to involve network team to check it .
4.Perform hardware check .(fmadm faulty)
5.Get the list of high memory & CPU consuming process list and inform the process owner to check it.
6.If you are having many local zones on that global,identify which local zones consuming more resources. (prstat -Z) and fix the issue on that local zone first.

Scenario:4
System is totally hung. How do you reboot the server after getting proper approval. ?

Answer:
1.If the server is hung,you should not perform hard reboot in first shot.you need to force the system to panic.
2.You need to raise support case with oracle to find the root cause using the generated crashdump .

Scenario:5
In some situations,system will not respond properly. (Ex:Unable to kill the process,Unable to umount the filesystem,local zone went to shutting_down state and never halted)

Answer:
1.Most of the Unix admins will be faced such a situation in their experience. In these cases   
you need to generate the live crashdump using "savecore -L" command and upload to oracle support to find the root cause. In most of the cases you need to end up with rebooting the system to fix these kind of issues.

Scenario:6
What are the challenges you have faced in Unix Administration ?

Answer:
You can impress the interviewer by saying couple of tough challenge which you have faced in the past.Here are the some of the challenges for you.
1.Recovered vxfs filesystem with help of Symantec.
2.Recovered the Solaris 10 using ZFS snapshot. 
3.Recovered the destroyed zpool
4.Fixed the Strange issue with ifconfig
5.Find the root cause for high kernel usage using Dtrace.
6.Plumbed New IP address on the local zone.
7.Completed successful migration from UFS to ZFS.
8.Liveupgrade issues with local zone sitting on top of ZPOOL.
9.Patched Solaris global zone with ZFS and  Non-global zone's root FS with VXFS

Scenario:7
For an example,If you are keep on getting incidents from one server.Those may be false alerts or there is an issue within server which needs UnixAdmin attention.What are the process you will follow to fix the repetitive alerts ?

Answer:
1.In these kind of situations ,you need to involve problem management team to fix these repetitive alerts.Problem management team will involve multiple teams if required to fix this issue.

How To Add New Disk To The Hardware Raid Controller Without Rebooting The Server In Linux


First find out your RAID hardware as we will need appropriate OS utility to manage the samefrom the same vendor. In our case it was lsi which can be managed via Megacli utility.
[root@server 8.02.21_Linux_MegaCLI]# lspci | grep -i raid
07:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 2108 [Liberator] (rev 05)
Lets first install the utilty :To do this we need to install the Megacli utility on our server Centos 6.2.

Step:1. Install the below two packages.(downlaod from http://www.lsi.com/downloads/Public/MegaRAID%20Common%20Files/8.02.21_MegaCLI.zip) .

#rpm -ivh Lib_Utils-1.00-09.noarch.rpm
#rpm -ivh MegaCli-8.02.21-1.noarch.rpm

Step:2. By default the utility is placed at location as below..

[root@server 8.02.21_Linux_MegaCLI]# cd /opt/MegaRAID/MegaCli/
[root@server MegaCli]# lsinstall.log MegaCli64 MegaSAS.log

Step:3. Run the below command before adding the brand new disk drive.[root@server MegaCli]# /opt/MegaRAID/MegaCli/MegaCli64 -PDList -aAll | grep -e “Enclosure Device ID” -e SlotEnclosure Device ID: 252Slot Number: 0Enclosure Device ID: 252Slot Number: 1

Step:4. Run the command again after injecting the new HDD in drive slot.[root@server MegaCli]# /opt/MegaRAID/MegaCli/MegaCli64 -PDList -aAll | grep -e “Enclosure Device ID” -e SlotEnclosure Device ID: 252Slot Number: 0Enclosure Device ID: 252Slot Number: 1Enclosure Device ID: 252Slot Number: 3

Step:5. Now you have the following information the Enclosure Device ID: 252 and Slot Number: 3 for the newly added drive.

 Step:6. Once added the new drive will be in foreign state. Give the below command to clear the foreign status.[root@server MegaCli]#/opt/MegaRAID/MegaCli/MegaCli64 -CfgForeign -Clear -aALL

Step:7. Add the new drive to the desired RAID level by running below command.#/opt/MegaRAID/MegaCli/MegaCli64 -CfgLdAdd -r0 [252:3] -a0Options:-r1 = RAID1252 = Enclosure Device ID2 and 3 = Slot Number-a0 = AdapterIn our case :-r0 = RAID0252 = Enclosure Device ID3 = Slot Number-a0 = Adapter

Step:8. tail the /var/log/messages for the new disksd 0:2:1:0: [sdb] Mode Sense: 1f 00 00 08sd 0:2:1:0: [sdb] Write cache: enabled, read cache: enabled, doesn’t support DPO or FUAsdb: sdb1 sdb2sd 0:2:1:0: [sdb] Attached SCSI disk

Now you can use the new disk as any other normal drive.

linux commands


linux commands
Command Description
apropos whatis Show commands pertinent to string. See also threadsafe
man -t ascii | ps2pdf - > ascii.pdf make a pdf of a manual page
  which command Show full path name of command
  time command See how long a command takes
time cat Start stopwatch. Ctrl-d to stop. See also sw
dir navigation
cd - Go to previous directory
cd Go to $HOME directory
  (cd dir && command) Go to dir, execute command and return to current dir
pushd . Put current dir on stack so you can popd back to it
file searching
alias l='ls -l --color=auto' quick dir listing
ls -lrt List files by date. See also newest and find_mm_yyyy
ls /usr/bin | pr -T9 -W$COLUMNS Print in 9 columns to width of terminal
  find -name '*.[ch]' | xargs grep -E 'expr' Search 'expr' in this dir and below. See also findrepo
  find -type f -print0 | xargs -r0 grep -F 'example' Search all regular files for 'example' in this dir and below
  find -maxdepth 1 -type f | xargs grep -F 'example' Search all regular files for 'example' in this dir
  find -maxdepth 1 -type d | while read dir; do echo $dir; echo cmd2; done Process each item with multiple commands (in while loop)
find -type f ! -perm -444 Find files not readable by all (useful for web site)
find -type d ! -perm -111 Find dirs not accessible by all (useful for web site)
locate -r 'file[^/]*\.txt' Search cached index for names. This re is like glob *file*.txt
look reference Quickly search (sorted) dictionary for prefix
grep --color reference /usr/share/dict/words Highlight occurances of regular expression in dictionary
archives and compression
  gpg -c file Encrypt file
  gpg file.gpg Decrypt file
  tar -c dir/ | bzip2 > dir.tar.bz2 Make compressed archive of dir/
  bzip2 -dc dir.tar.bz2 | tar -x Extract archive (use gzip instead of bzip2 for tar.gz files)
  tar -c dir/ | gzip | gpg -c | ssh user@remote 'dd of=dir.tar.gz.gpg' Make encrypted archive of dir/ on remote machine
  find dir/ -name '*.txt' | tar -c --files-from=- | bzip2 > dir_txt.tar.bz2 Make archive of subset of dir/ and below
  find dir/ -name '*.txt' | xargs cp -a --target-directory=dir_txt/ --parents Make copy of subset of dir/ and below
  ( tar -c /dir/to/copy ) | ( cd /where/to/ && tar -x -p ) Copy (with permissions) copy/ dir to /where/to/ dir
  ( cd /dir/to/copy && tar -c . ) | ( cd /where/to/ && tar -x -p ) Copy (with permissions) contents of copy/ dir to /where/to/
  ( tar -c /dir/to/copy ) | ssh -C user@remote 'cd /where/to/ && tar -x -p' Copy (with permissions) copy/ dir to remote:/where/to/ dir
  dd bs=1M if=/dev/sda | gzip | ssh user@remote 'dd of=sda.gz' Backup harddisk to remote machine
rsync (Network efficient file copier: Use the --dry-run option for testing)
  rsync -P rsync://rsync.server.com/path/to/file file Only get diffs. Do multiple times for troublesome downloads
  rsync --bwlimit=1000 fromfile tofile Locally copy with rate limit. It's like nice for I/O
  rsync -az -e ssh --delete ~/public_html/ remote.com:'~/public_html' Mirror web site (using compression and encryption)
  rsync -auz -e ssh remote:/dir/ . && rsync -auz -e ssh . remote:/dir/ Synchronize current directory with remote one
ssh (Secure SHell)
  ssh $USER@$HOST command Run command on $HOST as $USER (default command=shell)
ssh -f -Y $USER@$HOSTNAME xeyes Run GUI command on $HOSTNAME as $USER
  scp -p -r $USER@$HOST: file dir/ Copy with permissions to $USER's home directory on $HOST
  scp -c arcfour $USER@$LANHOST: bigfile Use faster crypto for local LAN. This might saturate GigE
  ssh -g -L 8080:localhost:80 root@$HOST Forward connections to $HOSTNAME:8080 out to $HOST:80
  ssh -R 1434:imap:143 root@$HOST Forward connections from $HOST:1434 in to imap:143
  ssh-copy-id $USER@$HOST Install public key for $USER@$HOST for password-less log in
wget (multi purpose download tool)
(cd dir/ && wget -nd -pHEKk http://www.pixelbeat.org/cmdline.html) Store local browsable version of a page to the current dir
  wget -c http://www.example.com/large.file Continue downloading a partially downloaded file
  wget -r -nd -np -l1 -A '*.jpg' http://www.example.com/dir/ Download a set of files to the current directory
  wget ftp://remote/file[1-9].iso/ FTP supports globbing directly
wget -q -O- http://www.pixelbeat.org/timeline.html | grep 'a href' | head Process output directly
  echo 'wget url' | at 01:00 Download url at 1AM to current dir
  wget --limit-rate=20k url Do a low priority download (limit to 20KB/s in this case)
  wget -nv --spider --force-html -i bookmarks.html Check links in a file
  wget --mirror http://www.example.com/ Efficiently update a local copy of a site (handy from cron)
networking (Note ifconfig, route, mii-tool, nslookup commands are obsolete)
  ethtool eth0 Show status of ethernet interface eth0
  ethtool --change eth0 autoneg off speed 100 duplex full Manually set ethernet interface speed
  iwconfig eth1 Show status of wireless interface eth1
  iwconfig eth1 rate 1Mb/s fixed Manually set wireless interface speed
iwlist scan List wireless networks in range
ip link show List network interfaces
  ip link set dev eth0 name wan Rename interface eth0 to wan
  ip link set dev eth0 up Bring interface eth0 up (or down)
ip addr show List addresses for interfaces
  ip addr add 1.2.3.4/24 brd + dev eth0 Add (or del) ip and mask (255.255.255.0)
ip route show List routing table
  ip route add default via 1.2.3.254 Set default gateway to 1.2.3.254
host pixelbeat.org Lookup DNS ip address for name or vice versa
hostname -i Lookup local ip address (equivalent to host `hostname`)
whois pixelbeat.org Lookup whois info for hostname or ip address
netstat -tupl List internet services on a system
netstat -tup List active connections to/from system
windows networking (Note samba is the package that provides all this windows specific networking support)
smbtree Find windows machines. See also findsmb
  nmblookup -A 1.2.3.4 Find the windows (netbios) name associated with ip address
  smbclient -L windows_box List shares on windows machine or samba server
  mount -t smbfs -o fmask=666,guest //windows_box/share /mnt/share Mount a windows share
  echo 'message' | smbclient -M windows_box Send popup to windows machine (off by default in XP sp2)
text manipulation (Note sed uses stdin and stdout. Newer versions support inplace editing with the -i option)
  sed 's/string1/string2/g' Replace string1 with string2
  sed 's/\(.*\)1/\12/g' Modify anystring1 to anystring2
  sed '/ *#/d; /^ *$/d' Remove comments and blank lines
  sed ':a; /\\$/N; s/\\\n//; ta' Concatenate lines with trailing \
  sed 's/[ \t]*$//' Remove trailing spaces from lines
  sed 's/\([`"$\]\)/\\\1/g' Escape shell metacharacters active within double quotes
seq 10 | sed "s/^/      /; s/ *\(.\{7,\}\)/\1/" Right align numbers
  sed -n '1000{p;q}' Print 1000th line
  sed -n '10,20p;20q' Print lines 10 to 20
  sed -n 's/.*<title>\(.*\)<\/title>.*/\1/ip;T;q' Extract title from HTML web page
  sed -i 42d ~/.ssh/known_hosts Delete a particular line
  sort -t. -k1,1n -k2,2n -k3,3n -k4,4n Sort IPV4 ip addresses
echo 'Test' | tr '[:lower:]' '[:upper:]' Case conversion
tr -dc '[:print:]' < /dev/urandom Filter non printable characters
tr -s '[:blank:]' '\t' </proc/diskstats | cut -f4 cut fields separated by blanks
history | wc -l Count lines
set operations (Note you can export LANG=C for speed. Also these assume no duplicate lines within a file)
  sort file1 file2 | uniq Union of unsorted files
  sort file1 file2 | uniq -d Intersection of unsorted files
  sort file1 file1 file2 | uniq -u Difference of unsorted files
  sort file1 file2 | uniq -u Symmetric Difference of unsorted files
  join -t'\0' -a1 -a2 file1 file2 Union of sorted files
  join -t'\0' file1 file2 Intersection of sorted files
  join -t'\0' -v2 file1 file2 Difference of sorted files
  join -t'\0' -v1 -v2 file1 file2 Symmetric Difference of sorted files
math
echo '(1 + sqrt(5))/2' | bc -l Quick math (Calculate φ). See also bc
seq -f '4/%g' 1 2 99999 | paste -sd-+ | bc -l Calculate π the unix way
echo 'pad=20; min=64; (100*10^6)/((pad+min)*8)' | bc More complex (int) e.g. This shows max FastE packet rate
echo 'pad=20; min=64; print (100E6)/((pad+min)*8)' | python Python handles scientific notation
echo 'pad=20; plot [64:1518] (100*10**6)/((pad+x)*8)' | gnuplot -persist Plot FastE packet rate vs packet size
echo 'obase=16; ibase=10; 64206' | bc Base conversion (decimal to hexadecimal)
echo $((0x2dec)) Base conversion (hex to dec) ((shell arithmetic expansion))
units -t '100m/9.58s' 'miles/hour' Unit conversion (metric to imperial)
units -t '500GB' 'GiB' Unit conversion (SI to IEC prefixes)
units -t '1 googol' Definition lookup
seq 100 | (tr '\n' +; echo 0) | bc Add a column of numbers. See also add and funcpy
calendar
cal -3 Display a calendar
cal 9 1752 Display a calendar for a particular month year
date -d fri What date is it this friday. See also day
[ $(date -d '12:00 +1 day' +%d) = '01' ] || exit exit a script unless it's the last day of the month
date --date='25 Dec' +%A What day does xmas fall on, this year
date --date='@2147483647' Convert seconds since the epoch (1970-01-01 UTC) to date
TZ='America/Los_Angeles' date What time is it on west coast of US (use tzselect to find TZ)
date --date='TZ="America/Los_Angeles" 09:00 next Fri' What's the local time for 9AM next Friday on west coast US
locales
printf "%'d\n" 1234 Print number with thousands grouping appropriate to locale
BLOCK_SIZE=\'1 ls -l Use locale thousands grouping in ls. See also l
echo "I live in `locale territory`" Extract info from locale database
LANG=en_IE.utf8 locale int_prefix Lookup locale info for specific country. See also ccodes
locale -kc $(locale | sed -n 's/\(LC_.\{4,\}\)=.*/\1/p') | less List fields available in locale database
recode (Obsoletes iconv, dos2unix, unix2dos)
recode -l | less Show available conversions (aliases on each line)
  recode windows-1252.. file_to_change.txt Windows "ansi" to local charset (auto does CRLF conversion)
  recode utf-8/CRLF.. file_to_change.txt Windows utf8 to local charset
  recode iso-8859-15..utf8 file_to_change.txt Latin9 (western europe) to utf8
  recode ../b64 < file.txt > file.b64 Base64 encode
  recode /qp.. < file.qp > file.txt Quoted printable decode
  recode ..HTML < file.txt > file.html Text to HTML
recode -lf windows-1252 | grep euro Lookup table of characters
echo -n 0x80 | recode latin-9/x1..dump Show what a code represents in latin-9 charmap
echo -n 0x20AC | recode ucs-2/x2..latin-9/x Show latin-9 encoding
echo -n 0x20AC | recode ucs-2/x2..utf-8/x Show utf-8 encoding
CDs
  gzip < /dev/cdrom > cdrom.iso.gz Save copy of data cdrom
  mkisofs -V LABEL -r dir | gzip > cdrom.iso.gz Create cdrom image from contents of dir
  mount -o loop cdrom.iso /mnt/dir Mount the cdrom image at /mnt/dir (read only)
  cdrecord -v dev=/dev/cdrom blank=fast Clear a CDRW
  gzip -dc cdrom.iso.gz | cdrecord -v dev=/dev/cdrom - Burn cdrom image (use dev=ATAPI -scanbus to confirm dev)
  cdparanoia -B Rip audio tracks from CD to wav files in current dir
  cdrecord -v dev=/dev/cdrom -audio -pad *.wav Make audio CD from all wavs in current dir (see also cdrdao)
  oggenc --tracknum='track' track.cdda.wav -o 'track.ogg' Make ogg file from wav file
disk space (See also FSlint)
ls -lSr Show files by size, biggest last
du -s * | sort -k1,1rn | head Show top disk users in current dir. See also dutop
du -hs /home/* | sort -k1,1h Sort paths by easy to interpret disk usage
df -h Show free space on mounted filesystems
df -i Show free inodes on mounted filesystems
fdisk -l Show disks partitions sizes and types (run as root)
rpm -q -a --qf '%10{SIZE}\t%{NAME}\n' | sort -k1,1n List all packages by installed size (Bytes) on rpm distros
dpkg-query -W -f='${Installed-Size;10}\t${Package}\n' | sort -k1,1n List all packages by installed size (KBytes) on deb distros
dd bs=1 seek=2TB if=/dev/null of=ext3.test Create a large test file (taking no space). See also truncate
> file truncate data of file or create an empty file
monitoring/debugging
tail -f /var/log/messages Monitor messages in a log file
strace -c ls >/dev/null Summarise/profile system calls made by command
strace -f -e open ls >/dev/null List system calls made by command
strace -f -e trace=write -e write=1,2 ls >/dev/null Monitor what's written to stdout and stderr
ltrace -f -e getenv ls >/dev/null List library calls made by command
lsof -p $$ List paths that process id has open
lsof ~ List processes that have specified path open
tcpdump not port 22 Show network traffic except ssh. See also tcpdump_not_me
ps -e -o pid,args --forest List processes in a hierarchy
ps -e -o pcpu,cpu,nice,state,cputime,args --sort pcpu | sed '/^ 0.0 /d' List processes by % cpu usage
ps -e -orss=,args= | sort -b -k1,1n | pr -TW$COLUMNS List processes by mem (KB) usage. See also ps_mem.py
ps -C firefox-bin -L -o pid,tid,pcpu,state List all threads for a particular process
ps -p 1,$$ -o etime= List elapsed wall time for particular process IDs
last reboot Show system reboot history
free -m Show amount of (remaining) RAM (-m displays in MB)
watch -n.1 'cat /proc/interrupts' Watch changeable data continuously
udevadm monitor Monitor udev events to help configure rules
system information (see also sysinfo) ('#' means root access is required)
uname -a Show kernel version and system architecture
head -n1 /etc/issue Show name and version of distribution
cat /proc/partitions Show all partitions registered on the system
grep MemTotal /proc/meminfo Show RAM total seen by the system
grep "model name" /proc/cpuinfo Show CPU(s) info
lspci -tv Show PCI info
lsusb -tv Show USB info
mount | column -t List mounted filesystems on the system (and align output)
grep -F capacity: /proc/acpi/battery/BAT0/info Show state of cells in laptop battery
# dmidecode -q | less Display SMBIOS/DMI information
# smartctl -A /dev/sda | grep Power_On_Hours How long has this disk (system) been powered on in total
# hdparm -i /dev/sda Show info about disk sda
# hdparm -tT /dev/sda Do a read speed test on disk sda
# badblocks -s /dev/sda Test for unreadable blocks on disk sda
interactive (see also linux keyboard shortcuts)
readline Line editor used by bash, python, bc, gnuplot, ...
screen Virtual terminals with detach capability, ...
mc Powerful file manager that can browse rpm, tar, ftp, ssh, ...
gnuplot Interactive/scriptable graphing
links Web browser
xdg-open . open a file or url with the registered desktop application

Thursday, 24 October 2013

Bad Sector on Hard Disk


Modern disk map bad sector to good sector to avoid any sort of error due to bad sector. If a large number of sector mapped to another sectors that means disk is going to fail
 smartctl command is very helpful command to diagnose disk related problems

To instruct for selftest , we can issue following command
 root# smartctl -l selftest /dev/sda

To get attributes list you can use following command
 root#smartctl -A /dev/sda

The output of above command is self explanatory and help you to recognize if there is any problem on hard disk.

Linux Performance Monitoring and Tuning Introduction


To identify system bottlenecks and come up with solutions to fix it, you should understand how various components of Linux works. For example, how the kernel gives preference to one Linux process over others using nice values, how I/O interrupts are handled, how the memory management works, how the Linux file system works, how the network layer is implemented in Linux, etc.,
Please note that understanding how various components (or subsystems) works is not the same as knowing what command to execute to get certain output. For example, you might know that “uptime” or “top” command gives the “load average”. But, if you don’t know what it means, and how the CPU (or process) subsystem works, you might not be able to understand it properly. Understanding the subsystems is an on-going task, which you’ll be constantly learning all the time.
On a very high level, following are the four subsystems that needs to be monitored.
  • CPU
  • Memory
  • I/O
  • Network

1. CPU

You should understand the four critical performance metrics for CPU — context switch, run queue, cpu utilization, and load average.

Context Switch

  • When CPU switches from one process (or thread) to another, it is called as context switch.
  • When a process switch happens, kernel stores the current state of the CPU (of a process or thread) in the memory.
  • Kernel also retrieves the previously stored state (of a process or thread) from the memory and puts it in the CPU.
  • Context switching is very essential for multitasking of the CPU.
  • However, a higher level of context switching can cause performance issues.

Run Queue

  • Run queue indicates the total number of active processes in the current queue for CPU.
  • When CPU is ready to execute a process, it picks it up from the run queue based on the priority of the process.
  • Please note that processes that are in sleep state, or i/o wait state are not in the run queue.
  • So, a higher number of processes in the run queue can cause performance issues.

Cpu Utilization

  • This indicates how much of the CPU is currently getting used.
  • This is fairly straight forward, and you can view the CPU utilization from the top command.
  • 100% CPU utilization means the system is fully loaded.
  • So, a higher %age of CPU utilization will cause performance issues.

Load Average

  • This indicates the average CPU load over a specific time period.
  • On Linux, load average is displayed for the last 1 minute, 5 minutes, and 15 minutes. This is helpful to see whether the overall load on the system is going up or down.
  • For example, a load average of “0.75 1.70 2.10″ indicates that the load on the system is coming down. 0.75 is the load average in the last 1 minute. 1.70 is the load average in the last 5 minutes. 2.10 is the load average in the last 15 minutes.
  • Please note that this load average is calculated by combining both the total number of process in the queue, and the total number of processes in the uninterruptable task status.

2. Network

  • A good understanding of TCP/IP concepts is helpful while analyzing any network issues. We’ll discuss more about this in future articles.
  • For network interfaces, you should monitor total number of packets (and bytes) received/sent through the interface, number of packets dropped, etc.,

3. I/O

  • I/O wait is the amount of time CPU is waiting for I/O. If you see consistent high i/o wait on you system, it indicates a problem in the disk subsystem.
  • You should also monitor reads/second, and writes/second. This is measured in blocks. i.e number of blocks read/write per second. These are also referred as bi and bo (block in and block out).
  • tps indicates total transactions per seconds, which is sum of rtps (read transactions per second) and wtps (write transactions per seconds).

4. Memory

  • As you know, RAM is your physical memory. If you have 4GB RAM installed on your system, you have 4GB of physical memory.
  • Virtual memory = Swap space available on the disk + Physical memory. The virtual memory contains both user space and kernel space.
  • Using either 32-bit or 64-bit system makes a big difference in determining how much memory a process can utilize.
  • On a 32-bit system a process can only access a maximum of 4GB virtual memory. On a 64-bit system there is no such limitation.
  • The unused RAM will be used as file system cache by the kernel.
  • The Linux system will swap when it needs more memory. i.e when it needs more memory than the physical memory. When it swaps, it writes the least used memory pages from the physical memory to the swap space on the disk.
  • Lot of swapping can cause performance issues, as the disk is much slower than the physical memory, and it takes time to swap the memory pages from RAM to disk.
All of the above 4 subsystems are interrelated. Just because you see a high reads/second, or writes/second, or I/O wait doesn’t mean the issue is there with the I/O sub-system. It also depends on what the application is doing. In most cases, the performance issue might be caused by the application that is running on the Linux system.
Remember the 80/20 rule — 80% of the performance improvement comes from tuning the application, and the rest 20% comes from tuning the infrastructure components.
There are various tools available to monitor Linux system performance. For example: top, free, ps, iostat, vmstat, mpstat, sar, tcpump, netstat, iozone, etc., We’ll be discussing more about these tools and how to use them in the upcoming articles in this series.
Following is the 4 step approach to identify and solve a performance issue.
  • Step 1 – Understand (and reproduce) the problem: Half of the problem is solved when you clearly understand what the problem is. Before trying to solve the performance issue, first work on clearly defining the problem. The more time you spend on understanding and defining the problem will give you enough details to look for the answers in the right place. If possible, try to reproduce the problem, or at least simulate a situation that you think closely resembles the problem. This will later help you to validate the solution you come up to fix the performance issue.
  • Step 2 – Monitor and collect data: After defining the problem clearly, monitor the system and try to collect as much data as possible on various subsystems. Based on this data, come up list of potential issues.
  • Step 3 – Eliminate and narrow down issues: After having a list of potential issues, dive into each one of them and eliminate any non issues. Narrow it down further to see whether it is an application issue, or an infrastructure issue. Drill down further and narrow it down to a specific component. For example, if it is an infrastructure issue, narrow it down and identify the subsystem that is causing the issue. If it is an I/O subsystem issue, narrow it down to a specific partition, or raid group, or LUN, or disk. Basically, keep drilling down until you put your finger on the root cause of the issue.
  • Step 4 – One change at a time: Once you’ve narrowed down to a small list of potential issues, don’t try to make multiple changes at one time. If you make multiple changes, you wouldn’t know which one fixed the original issue. Multiple changes at one time might also cause new issues, which you’ll be chasing after instead of fixing the original issue. So, make one change at a time, and see if it fixes the original problem.

What is the difference between Telnet and SSH?


ssh is a secured shell, where telnet is not a secured
one.when you ssh to trasnfer data between a system, the data
will be send in the encrypted form, where the hacker cannot
encode or decode it. While you telnet,the data send between
the system is alphabetical format(ASCII), where every one
can understand. More over as per network security, telnet
and ftp are prohibited. Always, trust SSL based data transfer.

Telnet ->
Its just getting (Telenet) a connection to the server.
Its not more secure. Anybody can use it.
It can be easly hacked.It can be easily read by anybody in
that network

SSH -> secured shocket shell
Its more secure than Telnet .
This has an encrption and decrption of the data /usr/pwd
None can hack this. It is the good way to transfer the data

Linux change the speed and duplex settings of an Ethernet card.


Q. How do I change the speed, duplex on for my Ethernet card?
A. Under Linux use mii-tool or ethtool package which allows a Linux sys admin to modify/change and view the negotiated speed of network interface card (NIC) i.e. it is useful for forcing specific Ethernet speed and duplex settings.
Depending on which type of Ethernet card is installed on the system you need to use either mii-tool or ethtool. I recommend installing both and use one of the tool, which will work with your card.

Task: Install mii-tool and ethtool tools

If you are using Debian Linux you can install both of these package with following command:# apt-get install ethtool net-toolsIf you are using Red Hat Enterprise Linux you can install both of these package with following command:# up2date ethtool net-toolsIf you are using Fedora Core Linux you can install both of these package with following command:# yum install ethtool net-tools

Task: Get speed and other information for eth0

Type following command as root user:
# ethtool eth0Output: 
Settings for eth0:
     Supported ports: [ TP MII ]
     Supported link modes:   10baseT/Half 10baseT/Full
                             100baseT/Half 100baseT/Full
     Supports auto-negotiation: Yes
     Advertised link modes:  10baseT/Half 10baseT/Full
                             100baseT/Half 100baseT/Full
     Advertised auto-negotiation: Yes
     Speed: 100Mb/s
     Duplex: Full
     Port: MII
     PHYAD: 32
     Transceiver: internal
     Auto-negotiation: on
     Supports Wake-on: pumbg
     Wake-on: d
     Current message level: 0x00000007 (7)
     Link detected: yes
Or use mii-tool command as follows:# mii-tool eth0Output: 
eth0: negotiated 100baseTx-FD flow-control, link ok

Task: Change the speed and duplex settings

Setup eth0 negotiated speed with mii-tool
Disable autonegotiation, and force the MII to either 100baseTx-FD, 100baseTx-HD, 10baseT-FD, or 10baseT-HD:# mii-tool -F 100baseTx-HD
# mii-tool -F 10baseT-HD
Setup eth0 negotiated speed with ethtool# ethtool -s eth0 speed 100 duplex full
# ethtool -s eth0 speed 10 duplex half
To make these settings permanent you need to create a shell script and call from /etc/rc.local (Red Hat) or if you are using Debian create a script into the directory /etc/init.d/directory and run update-rc.d command to update the script.
Read man page of mii-tool and ethtool for more information.

Monitor Your System Performance on linux


Monitor Your System Performance

[root@server ~]# uptime
 13:14:02 up  1:30,  2 users,  load average: 0.04, 0.26, 0.27


[root@server ~]# ps -eo %u | sort | uniq -c | sort -rn
     56 root
     22 nagios
      8 apache
      1 xfs
      1 RUSER
      1 rpc
      1 ntp
      1 named
      1 mysql
      1 ldap


[root@server ~]# sar -c          
 ## Process/sec
Linux 2.6.18-164.el5 (server.scratch.com)       08/24/2011

11:50:01 AM    proc/s
12:00:01 PM      6.87
12:10:01 PM      6.90
12:20:01 PM      7.10
12:30:01 PM      6.77
12:40:01 PM      6.99
12:50:01 PM      6.85
01:00:01 PM      6.83
01:10:01 PM      6.92
Average:         6.90

[root@server ~]# sar -b  

Data r/wr ps
Linux 2.6.18-164.el5 (server.scratch.com)       08/24/2011

11:50:01 AM       tps      rtps      wtps   bread/s   bwrtn/s
12:00:01 PM     24.81      0.18     24.63      6.37    376.94
12:10:01 PM     26.14      0.33     25.81     10.68    388.21
12:20:01 PM     57.68     28.05     29.63    620.21    466.02
12:30:01 PM     24.16      0.13     24.03      1.92    369.00
12:40:01 PM     24.55      0.10     24.44      1.70    374.73
12:50:01 PM     25.07      0.14     24.93      4.31    380.10
01:00:01 PM     24.95      0.58     24.37      6.20    377.88
01:10:01 PM     24.57      0.01     24.56      0.04    376.16
Average:        28.98      3.68     25.30     81.17    388.59

[root@server ~]# iostat -k
Linux 2.6.18-164.el5 (server.scratch.com)       08/24/2011

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.40    0.00    3.84    0.44    0.00   95.32

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda               6.60        32.24        65.92     177987     363873
sda1              0.10         0.21         0.00       1163          5
sda2              6.50        32.01        65.92     176684     363868
dm-0             20.19        31.54        65.91     174109     363844
dm-1              0.02         0.08         0.00        440          0
dm-2              0.06         0.25         0.00       1397         20
dm-3              0.03         0.11         0.00        613          4

[root@server ~]#






# cat /proc/stat
cpu  1864 0 16482 416939 2375 34 527 0
cpu0 1864 0 16482 416939 2375 34 527 0
intr 2416023 2368439 12 0 2 2 0 5 0 1 0 0 0 116 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 29989 0 0 0 0 0 0 0 37 0 0 0 0 0 0 0 17399 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
ctxt 258727
btime 1314166387
processes 32539
procs_running 1
procs_blocked 0

# cat /proc/meminfo
MemTotal:       218872 kB
MemFree:         36680 kB
Buffers:         40416 kB
Cached:          79080 kB
SwapCached:          0 kB
Active:         116728 kB
Inactive:        47320 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:       218872 kB
LowFree:         36680 kB
SwapTotal:      614384 kB
SwapFree:       614384 kB
Dirty:             460 kBWriteback:           0 kB
AnonPages:       44556 kB
Mapped:          15936 kB
Slab:            11628 kB
PageTables:       2320 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:    723820 kB
Committed_AS:   376900 kB
VmallocTotal:   802808 kB
VmallocUsed:      3564 kB
VmallocChunk:   798832 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     4096 kB

[root@server ~]# cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 23
model name      : Pentium(R) Dual-Core  CPU      E5300  @ 2.60GHz
stepping        : 10
cpu MHz         : 2592.870
cache size      : 2048 KB
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss constant_tsc up pni
bogomips        : 5185.74


[root@server ~]# cat /proc/buddyinfo
Node 0, zone      DMA     37     33     31     29     19     12      9      2      0      0      0
Node 0, zone   Normal   2217    861    467    317     71     17      6      1      1      0      0
[root@server ~]#

[root@server ~]# cat /proc/execdomains
0-0     Linux                   [kernel]

[root@server ~]# cat /proc/filesystems
nodev   sysfs
nodev   rootfs
nodev   bdev
nodev   proc
nodev   cpuset
nodev   binfmt_misc
nodev   debugfs
nodev   securityfs
nodev   sockfs
nodev   usbfs
nodev   pipefs
nodev   anon_inodefs
nodev   futexfs
nodev   tmpfs
nodev   inotifyfs
nodev   eventpollfs
nodev   devpts
        ext2
nodev   ramfs
nodev   hugetlbfs
        iso9660
nodev   mqueue
        ext3

[root@server ~]# cat /proc/interrupts
           CPU0
  0:    2539099    IO-APIC-edge  timer
  1:         12    IO-APIC-edge  i8042
  6:          5    IO-APIC-edge  floppy
  7:          0    IO-APIC-edge  parport0
  8:          1    IO-APIC-edge  rtc
  9:          0   IO-APIC-level  acpi
 12:        116    IO-APIC-edge  i8042
 15:         21    IO-APIC-edge  ide1
 51:      31376   IO-APIC-level  ehci_hcd:usb1, ioc0
 59:         37   IO-APIC-level  uhci_hcd:usb2
 67:      19497   IO-APIC-level  eth0
 75:          0   IO-APIC-level  Ensoniq AudioPCI
NMI:          0
LOC:    4700127
ERR:          0
MIS:          0

[root@server ~]# cat /proc/iomem

[root@server ~]# cat /proc/loadavg
0.33 0.36 0.29 1/92 2916

[root@server ~]# cat /proc/locks
1: POSIX  ADVISORY  READ  2689 fd:00:320039 4 4
2: POSIX  ADVISORY  WRITE 2732 fd:00:931460 0 EOF
3: POSIX  ADVISORY  READ  2689 fd:00:320021 4 4
4: POSIX  ADVISORY  WRITE 2689 fd:00:288615 0 0
5: POSIX  ADVISORY  READ  2686 fd:00:320027 4 4
6: POSIX  ADVISORY  READ  2686 fd:00:320026 4 4
7: POSIX  ADVISORY  READ  2686 fd:00:320025 4 4
8: POSIX  ADVISORY  READ  2686 fd:00:320024 4 4
9: POSIX  ADVISORY  READ  2686 fd:00:320021 4 4
10: POSIX  ADVISORY  WRITE 2686 fd:00:288614 0 0
11: FLOCK  ADVISORY  WRITE 2641 fd:00:288138 0 EOF
12: POSIX  ADVISORY  WRITE 2592 fd:00:288509 0 EOF
13: POSIX  ADVISORY  WRITE 2592 fd:00:288508 0 EOF
14: POSIX  ADVISORY  WRITE 2592 fd:00:288507 0 EOF
15: POSIX  ADVISORY  WRITE 2457 fd:00:288680 1024 2047

[root@server ~]# cat /proc/net/tcp
  sl  local_address rem_address   st tx_queue rx_queue tr tm->when retrnsmt   uid  timeout inode
   0: 00000000:0185 00000000:0000 0A 00000000:00000000 00:00000000 00000000     0        0 7642 1 c8adbb40 3000 0 0 2 -1
   1: 0100007F:0CEA 00000000:0000 0A 00000000:00000000 00:00000000 00000000    27        0 7887 1 c8adb240 3000 0 0 2 -1
   2: 00000000:008B 00000000:0000 0A 00000000:00000000 00:00000000 00000000     0        0 8138 1 c8ada4c0 3000 0 0 2 -1
   3: 00000000:006F 00000000:0000 0A 00000000:00000000 00:00000000 00000000     0        0 7482 1 c9720040 3000 0 0 2 -1
   4: 00000000:0015 00000000:0000 0A 00000000:00000000 00:00000000 00000000     0        0 7778 1 c8adb6c0 3000 0 0 2 -1
   5: 0708A8C0:0035 00000000:0000 0A 00000000:00000000 00:00000000 00000000    25        0 7399 1 c9720940 3000 0 0 2 -1
   6: 0808A8C0:0035 00000000:0000 0A 00000000:00000000 00:00000000 00000000    25        0 7397 1 c9720dc0 3000 0 0 2 -1
   7: 0908A8C0:0035 00000000:0000 0A 00000000:00000000 00:00000000 00000000    25        0 7395 1 c9721240 3000 0 0 2 -1
   8: 0A08A8C0:0035 00000000:0000 0A 00000000:00000000 00:00000000 00000000    25        0 7393 1 c97216c0 3000 0 0 2 -1
   9: 0100007F:0035 00000000:0000 0A 00000000:00000000 00:00000000 00000000    25        0 7391 1 c9721b40 3000 0 0 2 -1
  10: 0100007F:03B9 00000000:0000 0A 00000000:00000000 00:00000000 00000000    25        0 7407 1 c97204c0 3000 0 0 2 -1
  11: 00000000:01BD 00000000:0000 0A 00000000:00000000 00:00000000 00000000     0        0 8137 1 c8ada940 3000 0 0 2 -1
  12: 0908A8C0:D26D 3208A8C0:0016 02 00000001:00000000 01:000000DF 00000000   502        0 143213 3 c28d9b40 3000 0 0 2 -1
[root@server ~]#
[root@server ~]# cat /proc/net/udp
  sl  local_address rem_address   st tx_queue rx_queue tr tm->when retrnsmt   uid  timeout inode
   9: 0A08A8C0:0089 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 8108 2 c4db8d00
   9: 0908A8C0:0089 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 8106 2 c4db32c0
   9: 0808A8C0:0089 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 8104 2 c4db36c0
   9: 0708A8C0:0089 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 8102 2 c4db3ac0
   9: 00000000:0089 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 8099 2 c8a04080
  10: 0A08A8C0:008A 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 8109 2 c4db8b00
  10: 0908A8C0:008A 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 8107 2 c4db30c0
  10: 0808A8C0:008A 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 8105 2 c4db34c0
  10: 0708A8C0:008A 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 8103 2 c4db38c0
  10: 00000000:008A 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 8100 2 c4db3cc0
  53: 00000000:0035 00000000:0000 07 00000000:00000000 00:00000000 00000000    25        0 7403 2 c96e9c40
  53: 0708A8C0:0035 00000000:0000 07 00000000:00000000 00:00000000 00000000    25        0 7398 2 c96e9240
  53: 0808A8C0:0035 00000000:0000 07 00000000:00000000 00:00000000 00000000    25        0 7396 2 c96e9440
  53: 0908A8C0:0035 00000000:0000 07 00000000:00000000 00:00000000 00000000    25        0 7394 2 c96e9640
  53: 0A08A8C0:0035 00000000:0000 07 00000000:00000000 00:00000000 00000000    25        0 7392 2 c96e9840
  53: 0100007F:0035 00000000:0000 07 00000000:00000000 00:00000000 00000000    25        0 7390 2 c96e9a40
 111: 00000000:006F 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 7476 2 c96e9040
 123: 0708A8C0:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 7770 2 c8a04280
 123: 0808A8C0:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 7769 2 c8a04480
 123: 0908A8C0:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 7768 2 c8a04680
 123: 0A08A8C0:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 7767 2 c8a04880
 123: 0100007F:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 7766 2 c8a04a80
 123: 00000000:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 7762 2 c8a04c80

[root@server ~]# cat /proc/net/protocols
protocol  size sockets  memory press maxhdr  slab module     cl co di ac io in de sh ss gs se re sp bi br ha uh gp em
RAWv6      648     -1      -1   NI       0   yes  ipv6        y  y  y  n  y  y  y  n  y  y  y  y  n  y  y  y  y  n  n
UDPv6      620     -1       0   NI       0   yes  ipv6        y  y  y  n  y  n  y  n  y  y  y  y  n  n  y  y  y  y  n
TCPv6     1248     19       1   no     224   yes  ipv6        y  y  y  y  y  y  y  y  y  y  y  y  n  n  y  y  y  y  y
PACKET     444     -1      -1   NI       0   no   kernel      n  n  n  n  n  n  n  n  n  n  n  n  n  n  n  n  n  n  n
UNIX       416     -1      -1   NI       0   yes  kernel      n  n  n  n  n  n  n  n  n  n  n  n  n  n  n  n  n  n  n
RAW        500     -1      -1   NI       0   yes  kernel      y  y  y  n  y  y  n  n  y  y  y  y  n  y  y  y  y  n  n
UDP        508     -1       0   NI       0   yes  kernel      y  y  y  n  y  n  y  n  y  y  y  y  y  n  y  y  y  y  n
TCP       1136     19       1   no     224   yes  kernel      y  y  y  y  y  y  y  y  y  y  y  y  n  n  y  y  y  y  y
NETLINK    404     -1      -1   NI       0   no   kernel      n  n  n  n  n  n  n  n  n  n  n  n  n  n  n  n  n  n  n

[root@server ~]# cat /proc/partitions
major minor  #blocks  name

   8     0   31457280 sda
   8     1     104391 sda1
   8     2   20860402 sda2
 253     0    7168000 dm-0
 253     1     512000 dm-1
 253     2    3891200 dm-2
 253     3    2048000 dm-3

[root@server ~]# cat /proc/schedstat
version 12
timestamp 2436227
cpu0 4336 4336 7 4343 7528 307075 58464 139144 139144 192048 1771799 248611
domain0 00000001 661081 661081 0 0 0 0 0 661081 360 360 0 0 0 0 0 360 58464 58464 0 0 0 0 0 58464 0 0 0 0 0 0 0 0 0 0 0 0
[root@server ~]#


[root@server ~]# cat /proc/stat
cpu  2085 0 18967 484888 2403 42 607 0
cpu0 2085 0 18967 484888 2403 42 607 0
intr 2808623 2749553 12 0 2 2 0 5 0 1 0 0 0 116 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 33013 0 0 0 0 0 0 0 37 0 0 0 0 0 0 0 25861 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
ctxt 302029
btime 1314166387
processes 37438
procs_running 1
procs_blocked 0


[root@server ~]# cat /proc/swaps
Filename                                Type            Size    Used    Priority
/dev/mapper/MYVOL-SWAPer                partition       511992  0       -1
/extraswap                              file            102392  0       -2
[root@server ~]#


[root@server ~]# cat /proc/scsi/scsi
Attached devices:
Host: scsi0 Channel: 00 Id: 00 Lun: 00
  Vendor: VMware,  Model: VMware Virtual S Rev: 1.0
  Type:   Direct-Access                    ANSI SCSI revision: 02
[root@server ~]#

[root@server ~]# cat /proc/tty/drivers
/dev/tty             /dev/tty        5       0 system:/dev/tty
/dev/console         /dev/console    5       1 system:console
/dev/ptmx            /dev/ptmx       5       2 system
/dev/vc/0            /dev/vc/0       4       0 system:vtmaster
serial               /dev/ttyS       4 64-95 serial
pty_slave            /dev/pts      136 0-1048575 pty:slave
pty_master           /dev/ptm      128 0-1048575 pty:master
unknown              /dev/tty        4 1-63 console

[root@server ~]# cat /proc/uptime
5253.31 4967.57
[root@server ~]#

[root@server ~]# cat /proc/version
Linux version 2.6.18-164.el5 (mockbuild@builder16.centos.org) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-46)) #1 SMP Thu Sep 3 03:33:56 EDT 2009
[root@server ~]# cat /proc/vmstat
nr_anon_pages 9791
nr_mapped 3984
nr_file_pages 30072
nr_slab 3002
nr_page_table_pages 521
nr_dirty 107
nr_writeback 0
nr_unstable 0
nr_bounce 0
pgpgin 183179
pgpgout 345817
pswpin 0
pswpout 0
pgalloc_dma 8166
pgalloc_dma32 0
pgalloc_normal 3542647
pgalloc_high 0
pgfree 3561159
pgactivate 41504
pgdeactivate 13669
pgfault 12248866
pgmajfault 885
pgrefill_dma 1954
pgrefill_dma32 0
pgrefill_normal 22667
pgrefill_high 0
pgsteal_dma 1177
pgsteal_dma32 0
pgsteal_normal 13810
pgsteal_high 0
pgscan_kswapd_dma 576
pgscan_kswapd_dma32 0
pgscan_kswapd_normal 7328
pgscan_kswapd_high 0
pgscan_direct_dma 608
pgscan_direct_dma32 0
pgscan_direct_normal 7744
pgscan_direct_high 0
pginodesteal 213
slabs_scanned 23552
kswapd_steal 7342
kswapd_inodesteal 289
pageoutrun 198
allocstall 142
pgrotated 0

[root@server ~]#
[root@server ~]# vmstat -m


## Check NC status
[root@server ~]# mii-tool eth0