SBK1972_uk
asked on
Swap / Paging / System Performance
All,
I need advice / help.
Now, I have a ecom system, webserver - applicationserver - oracleserver model, that uses oracle 8.1.5 for the database layer, installed on a E4500 server, using sol2.6. The ecom application, install on a E420R, is Intershop's Enfinity V1.1, and has been bastardized using poor bespoke code that results in multiple oracle connections that are never closed, causing the databases servers system resouces to be munched up.
The E4500 has 8 CPU's (400), 4Gb of physical RAM, swap -l :-
swapfile dev swaplo blocks free
/dev/vx/dsk/swapvol 158,5 16 4094704 2841552
/newswap/swapfile - 16 4095984 2846080
/u09/newswap2/swapfile2 - 16 4095984 2844160
However, my system pages out like a devil. Ive recently rebooted, about one hour ago, and the vmstat 5 looks like this :-
procs memory page disk faults cpu
r b w swap free re mf pi po fr de sr s0 s1 s2 s1 in sy cs us sy id
0 0 0 263176 704936 45 1026 253 520 6698 0 2525 38 38 2 38 3328 37602 10132 19
13 68
0 1 0 5664600 77600 44 1159 302 0 9233 0 3364 55 54 1 55 2763 45128 11788 13 14
73
0 3 0 5664680 85720 65 1065 243 0 9043 0 2837 53 50 1 52 3651 58058 14068 34 15 51
0 3 0 5664272 80056 169 1282 489 1 7438 0 3149 59 60 4 59 4398 68807 16417 34 17 49
0 2 0 5664704 80584 71 1340 540 0 7683 0 3190 35 37 5 36 4195 77320 16109 33 20 47
0 0 0 5664728 92048 3 1119 684 0 6876 0 2423 32 32 2 32 4556 99437 18438 27 22 51
0 1 0 5664520 85816 4 1295 472 0 8540 0 2971 65 63 1 65 3515 53485 12999 19 17 64
0 1 0 5664504 76904 5 1344 270 1 9064 0 17702 64 61 0 64 3162 56088 13021 18 17 65
0 0 0 5663136 75120 4 1575 252 0 10529 0 4283 66 66 1 66 2813 46791 10852 13 15 72
0 1 0 5664504 68976 10 1170 486 612 11920 0 3891 76 74 0 76 3073 42042 10835 12 15 73
root @ >: uptime
9:07am up 1:30, 2 users, load average: 3.10, 3.07, 3.15
Any ideas ? I cant say Ive done much tunning before and so seek your advice / suggestions. Everything considered ! ;-)
SBK
I need advice / help.
Now, I have a ecom system, webserver - applicationserver - oracleserver model, that uses oracle 8.1.5 for the database layer, installed on a E4500 server, using sol2.6. The ecom application, install on a E420R, is Intershop's Enfinity V1.1, and has been bastardized using poor bespoke code that results in multiple oracle connections that are never closed, causing the databases servers system resouces to be munched up.
The E4500 has 8 CPU's (400), 4Gb of physical RAM, swap -l :-
swapfile dev swaplo blocks free
/dev/vx/dsk/swapvol 158,5 16 4094704 2841552
/newswap/swapfile - 16 4095984 2846080
/u09/newswap2/swapfile2 - 16 4095984 2844160
However, my system pages out like a devil. Ive recently rebooted, about one hour ago, and the vmstat 5 looks like this :-
procs memory page disk faults cpu
r b w swap free re mf pi po fr de sr s0 s1 s2 s1 in sy cs us sy id
0 0 0 263176 704936 45 1026 253 520 6698 0 2525 38 38 2 38 3328 37602 10132 19
13 68
0 1 0 5664600 77600 44 1159 302 0 9233 0 3364 55 54 1 55 2763 45128 11788 13 14
73
0 3 0 5664680 85720 65 1065 243 0 9043 0 2837 53 50 1 52 3651 58058 14068 34 15 51
0 3 0 5664272 80056 169 1282 489 1 7438 0 3149 59 60 4 59 4398 68807 16417 34 17 49
0 2 0 5664704 80584 71 1340 540 0 7683 0 3190 35 37 5 36 4195 77320 16109 33 20 47
0 0 0 5664728 92048 3 1119 684 0 6876 0 2423 32 32 2 32 4556 99437 18438 27 22 51
0 1 0 5664520 85816 4 1295 472 0 8540 0 2971 65 63 1 65 3515 53485 12999 19 17 64
0 1 0 5664504 76904 5 1344 270 1 9064 0 17702 64 61 0 64 3162 56088 13021 18 17 65
0 0 0 5663136 75120 4 1575 252 0 10529 0 4283 66 66 1 66 2813 46791 10852 13 15 72
0 1 0 5664504 68976 10 1170 486 612 11920 0 3891 76 74 0 76 3073 42042 10835 12 15 73
root @ >: uptime
9:07am up 1:30, 2 users, load average: 3.10, 3.07, 3.15
Any ideas ? I cant say Ive done much tunning before and so seek your advice / suggestions. Everything considered ! ;-)
SBK
1. make parallel proces in Oracle
(7 or 6)
2. make the importand table load in the memory
3. If you have sort by make temp in the memory
4. table from index on the separate disk
5. system is the most use --> separate disk
(7 or 6)
2. make the importand table load in the memory
3. If you have sort by make temp in the memory
4. table from index on the separate disk
5. system is the most use --> separate disk
ASKER
Please dont suggest up grading oracle / Enfinity, costs mean I have to tune. Also, its patched up to the latest sol2.6 level.
SBK
SBK
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Hello again.
Yuzh, many thanks for those links. Made interesting reading. Tuning the kernal is a fiddly thing. The number of parameters you have to change, how they are related is amazing.
Little bit of an update on my problem though, the reason why my system was slow was due to an external web using `attacking` my site. he/she was pumping hundreds of requests on my site, via some sort of web stress software, cuasing my system to over load.
Nope, my ecom app doesnt cater well for Denial of service requests, first generation ecom model, although we are so to upgrade to WebSphere.
Anyway, any suggests, tricks, software, I can use to monitor any attacks, find out who's doing it, etc etc ??
Obviously I have firewalls, but as these attacks were genuine port 80 request, they wouldnt be reject/ dropped and arent being displayed in the reject logs of the FW software. The way I had to do it was to stop external traffic entering my site, clear the NES logs, they allow external traffic back in. I then sifted though the NES logs working out the top ten number of IP address. Turns out it was a 217.33 network addres aka BTopenworld.
Now this process took ages, anyone suggest any tips ??
SBK
Yuzh, many thanks for those links. Made interesting reading. Tuning the kernal is a fiddly thing. The number of parameters you have to change, how they are related is amazing.
Little bit of an update on my problem though, the reason why my system was slow was due to an external web using `attacking` my site. he/she was pumping hundreds of requests on my site, via some sort of web stress software, cuasing my system to over load.
Nope, my ecom app doesnt cater well for Denial of service requests, first generation ecom model, although we are so to upgrade to WebSphere.
Anyway, any suggests, tricks, software, I can use to monitor any attacks, find out who's doing it, etc etc ??
Obviously I have firewalls, but as these attacks were genuine port 80 request, they wouldnt be reject/ dropped and arent being displayed in the reject logs of the FW software. The way I had to do it was to stop external traffic entering my site, clear the NES logs, they allow external traffic back in. I then sifted though the NES logs working out the top ten number of IP address. Turns out it was a 217.33 network addres aka BTopenworld.
Now this process took ages, anyone suggest any tips ??
SBK
Hi SBK1972_uk,
I don't have experienced with E-Com type of web server. I have experiences with Online Teching Webserver (30k Uni students, students could be somewhere overseas) and Web Servers for Research. I would like to know the best solutions for your case.
Apart from the firwall you have. You can also do the followings, keep the OS patchs level up-to-date AND:
1. Disable the ability to execute code from the stack.
vi /etc/system file , add:
* Disable the ability to execute code from the stack
* This will actively prevent many buffer overflows
set noexec_user_stack=1
* This will report buffer overflows
set noexec_user_stack_log=1
2. Disable IP forwardind, I add (appen) the following to /etc/rc2.d/S69inet file:
### Set kernel parameters for /dev/ip
# Turn off IP forwarding
ndd -set /dev/ip ip_forwarding 0
ndd -set /dev/ip ip_respond_to_echo_broadca
ndd -set /dev/ip ip_forward_src_routed 0
ndd -set /dev/ip ip_forward_directed_broadc
ndd -set /dev/ip ip_ignore_redirect 1
ndd -set /dev/ip ip_send_redirects 0
# Disable respond to ICMP netmask requests, need to enable for boot server !!!
ndd -set /dev/ip ip_respond_to_address_mask
# For boxes outsite the firewall, disable respond to ICMP broadcast echo
# requests (ping). default is 1 (true)
ndd -set /dev/ip ip_respond_to_echo_broadca
# disable respond to ICMP timestamp requests, for boxes outside firewall only
ndd -set /dev/ip ip_respond_to_timestamp 0
ndd -set /dev/ip ip_respond_to_timestamp_br
# The machine will drop packets coming through one interface that are destined for
# another interface, prevent host spoofing
ndd -set /dev/ip ip_strict_dst_multihoming 1
# End of script
3. Tell the machine that not to forward packages
touch /etc/notrouter
chmod 600 /etc/notrouter
4. Disable all the unwanted server in /etc/inetd.conf
5. /etc/ftpuser and /etc/passwd , make some of the system user have no login shell ...etc.
For network monitoring, you can use "Nmap", it is free, you can download it from:
http://sunfreeware.com/
or get someting like:
ethereal
snort
tcpdump
tcpflow
You can read more UNIX security in:
http://www.stokely.com/unix.sysadm.resources/security.html
http://www.itworld.com/nl/unix_sec/
http://www.insecure.org/tools.html
http://www.sans.org/resources/idfaq/
Hope that I do not wast your time. and I would like to learn some tips from you. (I hnew you have a lot of experiences).
Cheers!
yuzh
I don't have experienced with E-Com type of web server. I have experiences with Online Teching Webserver (30k Uni students, students could be somewhere overseas) and Web Servers for Research. I would like to know the best solutions for your case.
Apart from the firwall you have. You can also do the followings, keep the OS patchs level up-to-date AND:
1. Disable the ability to execute code from the stack.
vi /etc/system file , add:
* Disable the ability to execute code from the stack
* This will actively prevent many buffer overflows
set noexec_user_stack=1
* This will report buffer overflows
set noexec_user_stack_log=1
2. Disable IP forwardind, I add (appen) the following to /etc/rc2.d/S69inet file:
### Set kernel parameters for /dev/ip
# Turn off IP forwarding
ndd -set /dev/ip ip_forwarding 0
ndd -set /dev/ip ip_respond_to_echo_broadca
ndd -set /dev/ip ip_forward_src_routed 0
ndd -set /dev/ip ip_forward_directed_broadc
ndd -set /dev/ip ip_ignore_redirect 1
ndd -set /dev/ip ip_send_redirects 0
# Disable respond to ICMP netmask requests, need to enable for boot server !!!
ndd -set /dev/ip ip_respond_to_address_mask
# For boxes outsite the firewall, disable respond to ICMP broadcast echo
# requests (ping). default is 1 (true)
ndd -set /dev/ip ip_respond_to_echo_broadca
# disable respond to ICMP timestamp requests, for boxes outside firewall only
ndd -set /dev/ip ip_respond_to_timestamp 0
ndd -set /dev/ip ip_respond_to_timestamp_br
# The machine will drop packets coming through one interface that are destined for
# another interface, prevent host spoofing
ndd -set /dev/ip ip_strict_dst_multihoming 1
# End of script
3. Tell the machine that not to forward packages
touch /etc/notrouter
chmod 600 /etc/notrouter
4. Disable all the unwanted server in /etc/inetd.conf
5. /etc/ftpuser and /etc/passwd , make some of the system user have no login shell ...etc.
For network monitoring, you can use "Nmap", it is free, you can download it from:
http://sunfreeware.com/
or get someting like:
ethereal
snort
tcpdump
tcpflow
You can read more UNIX security in:
http://www.stokely.com/unix.sysadm.resources/security.html
http://www.itworld.com/nl/unix_sec/
http://www.insecure.org/tools.html
http://www.sans.org/resources/idfaq/
Hope that I do not wast your time. and I would like to learn some tips from you. (I hnew you have a lot of experiences).
Cheers!
yuzh
Hi,
Have you enabled priority paging in solaris (2.6 and 2.7)
1. To enable priority paging, set the following in /etc/system:
set priority_paging=1
2. upgrade to oracle 8.1.7 (if possible)
3. Your scan rate is too way high!!!
you need to add more memory.
jundpogi
Have you enabled priority paging in solaris (2.6 and 2.7)
1. To enable priority paging, set the following in /etc/system:
set priority_paging=1
2. upgrade to oracle 8.1.7 (if possible)
3. Your scan rate is too way high!!!
you need to add more memory.
jundpogi
ASKER
Hello gents,
Firstly, many thanks for all your replys. Ive followed all the links, read 90% of the documentation you've suggested. The out come is that Ive now got sar running, Ive added several performance tunes that sun recommends, including the priority page /etc/system addage. One thing there, you must have a certain patch for that if your running sol26 and you must have you permissoning sorted out, or it sees all files as a binary and doesnt swap iut out.
Its like all performance issues, trial and error. Im pretty much stuck by what the application Im running demands, I cant upgrade the app, which only works on sol26 and oracle 8.1.5
Anyway, I have already got 4Gb of ram, and 3 gb of swap (mutlple mkfile + swap -a ), yes that much ! hahahah But the poor application doenst clear up after it makes odbc connections which uses system resources, blah blah blah.
I have noted pretty much all your adivce in my little Unix book of knowledge, well a spread sheet now with mutliple worksheets for different apps. Always use this on different clients sites.
Many thanks all.
SBK
Firstly, many thanks for all your replys. Ive followed all the links, read 90% of the documentation you've suggested. The out come is that Ive now got sar running, Ive added several performance tunes that sun recommends, including the priority page /etc/system addage. One thing there, you must have a certain patch for that if your running sol26 and you must have you permissoning sorted out, or it sees all files as a binary and doesnt swap iut out.
Its like all performance issues, trial and error. Im pretty much stuck by what the application Im running demands, I cant upgrade the app, which only works on sol26 and oracle 8.1.5
Anyway, I have already got 4Gb of ram, and 3 gb of swap (mutlple mkfile + swap -a ), yes that much ! hahahah But the poor application doenst clear up after it makes odbc connections which uses system resources, blah blah blah.
I have noted pretty much all your adivce in my little Unix book of knowledge, well a spread sheet now with mutliple worksheets for different apps. Always use this on different clients sites.
Many thanks all.
SBK
ASKER
SBK