====== Memroy Toubleshooting Guide ====== ===== kmeminfo ===== * memory usage infomation * **kmeminfo -u** 02:48:18 root@test1:/tmp$ ./kmeminfo -u tool: kmeminfo 7.02 unix: /stand/vmunix 11.23 64bit IA64 on "test1" core: /dev/kmem live link: Mon May 15 15:36:06 JST 2006 boot: Mon Oct 22 10:25:05 2007 time: Thu Oct 25 02:48:27 2007 nbpg: 4096 bytes ---------------------------------------------------------------------- Summary of processes memory usage: List sorted by physical size, in pages/bytes: virtual physical swap pid ppid pages / bytes pages / bytes pages / bytes command 4770 4769 366536 1.4g 167591 654.7m 166488 650.3m java 4746 4745 357763 1.4g 153809 600.8m 156841 612.7m java .... ... ... 27071 2744 18 72.0k 16 64.0k 19 76.0k ia64_corehw physical swap pages / bytes pages / bytes Total: 645284 2.5g 682292 2.6g ===== kmtune ===== * query, set, or reset system parameters root@soafdb2 /var/opt/perf$ kmtune | grep dbc kmtune is a wrapper script which exists for compatibility reasons only. The underlying command used is 'kctune'. New or modified scripts or procedures should use kctune directly. About to execute: kctune dbc_max_pct 4 4 Immed dbc_min_pct 4 4 Immed ===== swapinfo ===== root@soafdb2 /var/opt/perf$ swapinfo -ta Kb Kb Kb PCT START/ Kb TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME dev 16384000 0 16384000 0% 0 - 1 /dev/vg00/lvol2 dev 25378816 0 25378816 0% 0 - 1 /dev/vg00/lvol5 reserve - 10441640 -10441640 total 41762816 10441640 31321176 25% - 0 - ===== glance ===== ==== Memory Sorting ==== - glance의 glogal 화면은 일반적으로 cpu 사용량이 많은 프로세스로 sorting되어 있습니다. - sorting 순서를 변경하시려면 'o'를 치시고, 1을 선택하시면 아래과 같이 나옵니다. GlancePlus C.04.50.00 20:28:18 soafdb1 ia64 Current Avg High -------------------------------------------------------------------------------- CPU Util SSUU | 6% 7% 8% Disk Util F F | 11% 9% 11% Mem Util S SU UBB | 67% 67% 67% Networkil R R | 26% 26% 26% -------------------------------------------------------------------------------- INTERESTING PROCESS THRESHOLD OPTIONS Display processes with resource usage: Current Thresholds: CPU Utilization > (1.0 % ) Disk I/O Rate > (1.0 IOs/sec) Resident Set Size > (20 Mbytes ) Virtual Set Size > (500 Mbytes ) User name = (all ) Program name = (all ) TTY path name = (all ) Use match logic (and/or) : (or ) Sort key (name/cpu/disk/rss): (cpu ) <-- ""여기"" GlancePlus started or last reset: 08/06/2007 20:28:12 - 위에서 그냥 enter 치면서 내려오다가 Sort Key 부분을 **rss** 라고 치고 Enter 치면 확인 물어보고, y 선택하면 됩니다. ===== MeasureWare ===== - MeasureWare 를 통해 /var/opt/perf/datafiles 에 저장해 놓은 datafile로 부터 \\ 원하는 resource 사용량을 extract 를 이용하여 추출할 수 있다. - pv를 실행시켜서 화면으로 midaemon이 저장한 내용을 보실수 있고, extract 명령을 통해 excel로 변환할 수 있습니다. ==== extract (Report) ==== - MeasureWare 데몬이 돌아가고 있는지 확인한다. - /var/opt/perf/datafiles 디렉토리에 log*로 시작하는 파일이 최근변경된 것인지 확인한다. root@soafdb2 /var/opt/perf/datafiles$ date Mon Aug 6 20:37:40 KST 2007 root@soafdb2 /var/opt/perf/datafiles$ ls -al log* -rw-r--r-- 1 root root 16460528 Aug 6 20:35 logappl -rw-r--r-- 1 root root 17854008 Aug 6 20:35 logdev -rw-r--r-- 1 root root 16662640 Aug 6 20:35 logglob -rw-r--r-- 1 root root 10104 Aug 6 17:23 logindx -rw-r--r-- 1 root root 27547224 Aug 6 20:37 logproc -rw-r--r-- 1 root root 9834768 Aug 6 20:35 logtran - 최근에 변경된 파일이면, MeasureWare 가 돌아가고 있는 것이다. - 만약 변경된 일시가 오래된 것이라면 MeasureWare를 restart 시켜준다. root@soafdb2 /var/opt/perf/datafiles$ /opt/perf/bin/mwa restart all - report 형식을 만들 config 파일을 카피한다. \\ reptall은 원본을 유지한다. root@soafdb2 /var/opt/perf$ cp /var/opt/perf/reptall /var/opt/perf/rept.hpk - 원하는 정보를 출력할 수 있게 config 파일을 수정한다. \\ 맨앞 *는 주석이다. root@soafdb2 /var/opt/perf$ vi rept.hpk ... **................................ Global Record Identification Metrics * BLANK * RECORD_TYPE DATE <--- DATE 항목 추출 TIME <--- TIME 항목 추출 * YEAR ... **................................ Global CPU Metrics * GBL_ACTIVE_CPU GBL_CPU_TOTAL_UTIL <--- CPU 사용량 추출 * GBL_CPU_TOTAL_TIME ... **................................ Global Disk Metrics ... * GBL_DISK_HISTOGRAM GBL_DISK_UTIL_PEAK <--- DISK 사용량 추출 * GBL_DISK_TIME_PEAK ... **................................ Global Memory Metrics GBL_MEM_UTIL <--- MEMORY 사용량 추출 * GBL_MEM_USER_UTIL GBL_SWAP_SPACE_UTIL <--- SWAP SPACE 사용량 추출 * GBL_MEM_PAGE_REQUEST_RATE ... - extract 명령을 통해 data를 추출하는 스크립트를 만든다. \\ report filename : rept.out \\ duration : 2007-07-31 ~ today \\ date format : 'mm/dd/yy' root@soafdb2 /var/opt/perf$ cat extract.hpk if [ -e rept.out ] ; then echo "delete rept.out" rm -f rept.out fi # date format "mm/dd/yy" extract -xp -v -p -r /var/opt/perf/rept.hpk -f /var/opt/perf/rept.out -b 07/31/07 -e today echo "##################################################" echo " date : 2007-07-31 ~ today " echo " logfile : rept.out " echo "##################################################" - extract 옵션 -gapkdzcntuyGADZNTUY Selects types of data to extract/export g = global detail a = application detail p = process detail k = process (killed records only) d = disk device detail z = logical volume detail c = configuration detail n = netif detail t = transaction detail (Not in Tru64 UNIX) u = CPU detail y = filesystem detail G = global summary (Export only) A = application summary (Export only) D = disk device summary (Export only) Z = logical volume summary (Export only) N = netif summary (Export only) T = transaction summary (Export only) (Not in Tru64 UNIX) U = CPU summary (Export only) - 결과파일을 확인한다. root@soafdb2 /var/opt/perf$ vi rept.out # vi /tmp/report.txt ======================================================== | | | Peak |Memory| | Date |Time |CPU % |Disk %| % |Swap %| 02/07/02|10:16| 38.36| 24.89| 73.89| 33.00| 02/07/02|10:20| 14.55| 2.39| 78.42| 37.00| 02/07/02|10:25| 7.74| 1.83| 79.00| 37.00| ... ======================================================== - 위의 예는 DATE, TIME, CPU, DISK, MEMORY, SWAP 항목에 대해서만 data 를 추출한 것입니다. \\ 다른 항목은 /var/opt/perf/reptall 파일을 참조하십시오. ==== 다른설정예 ==== - rept.hpk를 수정한다. root@soafdb2 /var/opt/perf$ cat rept.hpk ... REPORT "OVPA Export !DATE !TIME Logfile: !LOGFILE !COLLECTOR !SYSTEM_ID" FORMAT ASCII HEADINGS ON SEPARATOR="|" SUMMARY=60 MISSING=0 DATA TYPE GLOBAL DATA TYPE APPLICATION DATA TYPE PROCESS DATE TIME YEAR PROC_PROC_NAME PROC_USER_NAME PROC_CPU_TOTAL_UTIL PROC_MEM_RES DATA TYPE TRANSACTION DATA TYPE DISK_SPACE DATA TYPE DISK DATA TYPE LVOLUME DATA TYPE NETIF DATA TYPE CPU DATA TYPE FILESYSTEM DATA TYPE CONFIGURATION DATA TYPE GLOBAL_SUMMARY DATA TYPE APPLICATION_SUMMARY DATA TYPE TRANSACTION_SUMMARY DATA TYPE DISK_SUMMARY DATA TYPE LVOLUME_SUMMARY DATA TYPE NETIF_SUMMARY DATA TYPE CPU_SUMMARY DATA TYPE FILESYSTEM_SUMMARY ... - 시간,프로세스,사용자,CPU사용률,MEM사용률이 나온다. root@soafdb2 /var/opt/perf$ cat rept.out | | | Process | User | | Res | Date |Time |Year| Name | Name | CPU % | Mem | 08/06/2007|17:23|2007|vxfsd |root | 5.08| 27008| 08/06/2007|17:23|2007|ora_lgwr_SOAF2 |oracle | 0.02| 43008| 08/06/2007|17:23|2007|coda |root | 0.00| 24728| 08/06/2007|17:23|2007|oracleSOAF2 |oracle | 10.06| 31292| ...