?
Solved

Swap / Paging / System Performance

Posted on 2003-03-10
8
Medium Priority
?
325 Views
Last Modified: 2013-12-27
All,

I need advice / help.

Now, I have a ecom system, webserver - applicationserver - oracleserver model,  that uses oracle 8.1.5 for the database layer, installed on a E4500 server, using sol2.6.  The ecom application, install on a E420R, is Intershop's Enfinity V1.1, and has been bastardized using poor bespoke code that results in multiple oracle connections that are never closed, causing the databases servers system resouces to be munched up.
 

The E4500 has 8 CPU's (400), 4Gb of physical RAM, swap -l :-

swapfile                    dev   swaplo blocks   free
/dev/vx/dsk/swapvol         158,5 16     4094704 2841552
/newswap/swapfile     -           16     4095984 2846080
/u09/newswap2/swapfile2  -        16     4095984 2844160


However, my system pages out like a devil. Ive recently rebooted, about one hour ago, and the vmstat 5 looks like this :-

procs     memory            page            disk          faults      cpu
r b w   swap  free  re  mf pi po fr de sr s0 s1 s2 s1   in   sy   cs us sy id
0 0 0 263176 704936 45 1026 253 520 6698 0 2525 38 38 2 38 3328 37602 10132 19
13 68
0 1 0 5664600 77600 44 1159 302 0 9233 0 3364 55 54 1 55 2763 45128 11788 13 14
 73
0 3 0 5664680 85720 65 1065 243 0 9043 0 2837 53 50 1 52 3651 58058 14068 34 15 51
0 3 0 5664272 80056 169 1282 489 1 7438 0 3149 59 60 4 59 4398 68807 16417 34 17 49
0 2 0 5664704 80584 71 1340 540 0 7683 0 3190 35 37 5 36 4195 77320 16109 33 20 47
0 0 0 5664728 92048  3 1119 684 0 6876 0 2423 32 32 2 32 4556 99437 18438 27 22 51
0 1 0 5664520 85816  4 1295 472 0 8540 0 2971 65 63 1 65 3515 53485 12999 19 17 64
0 1 0 5664504 76904  5 1344 270 1 9064 0 17702 64 61 0 64 3162 56088 13021 18 17 65
0 0 0 5663136 75120  4 1575 252 0 10529 0 4283 66 66 1 66 2813 46791 10852 13 15 72
0 1 0 5664504 68976 10 1170 486 612 11920 0 3891 76 74 0 76 3073 42042 10835 12 15 73

root @ >: uptime
 9:07am  up  1:30,  2 users,  load average: 3.10, 3.07, 3.15


Any ideas ? I cant say Ive done much tunning before and so seek your advice / suggestions.  Everything considered ! ;-)

SBK


 
0
Comment
Question by:SBK1972_uk
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
8 Comments
 
LVL 1

Author Comment

by:SBK1972_uk
ID: 8101570
Please dont suggest up grading oracle / Enfinity, costs mean I have to tune.  Also, its patched up to the latest sol2.6 level.

SBK
0
 
LVL 4

Expert Comment

by:Otetelisanu
ID: 8101610
1. make parallel proces in Oracle
   (7 or 6)
2. make the importand table  load in the memory
3. If you have sort by make temp in the memory
4. table from index on the separate disk
5. system is the most use -->  separate disk
0
 
LVL 1

Author Comment

by:SBK1972_uk
ID: 8103612
Please dont suggest up grading oracle / Enfinity, costs mean I have to tune.  Also, its patched up to the latest sol2.6 level.

SBK
0
What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

 
LVL 38

Accepted Solution

by:
yuzh earned 375 total points
ID: 8108965
0
 
LVL 1

Author Comment

by:SBK1972_uk
ID: 8118104
Hello again.

Yuzh, many thanks for those links. Made interesting reading.  Tuning the kernal is a fiddly thing. The number of parameters you have to change, how they are related is amazing.

Little bit of an update on my problem though, the reason why my system was slow was due to an external web using `attacking` my site.  he/she was pumping hundreds of requests on my site, via some sort of web stress software, cuasing my system to over load.

Nope, my ecom app doesnt cater well for Denial of service requests, first generation ecom model, although we are so to upgrade to WebSphere.

Anyway, any suggests, tricks, software, I can use to monitor any attacks, find out who's doing it, etc etc ??

Obviously I have firewalls, but as these attacks were genuine port 80 request, they wouldnt be reject/ dropped and arent being displayed in the reject logs of the FW software.  The way I had to do it was to stop external traffic entering my site, clear the NES logs, they allow external traffic back in.  I then sifted though the NES logs working out the top ten number of IP address. Turns out it was a 217.33 network addres aka BTopenworld.

Now this process took ages, anyone suggest any tips ??

SBK
0
 
LVL 38

Expert Comment

by:yuzh
ID: 8134446
Hi SBK1972_uk,

   I don't have experienced with E-Com type of web server. I have experiences with Online Teching Webserver (30k Uni students, students could be somewhere overseas) and Web Servers for Research. I would like to know the best solutions for your case.

   Apart from the firwall you have. You can also do the followings, keep the OS patchs level up-to-date AND:

   1. Disable the ability to execute code from the stack.
       vi /etc/system file , add:

       * Disable the ability to execute code from the stack
       * This will actively prevent many buffer overflows
       set noexec_user_stack=1

       * This will report buffer overflows
       set noexec_user_stack_log=1

   2. Disable IP forwardind, I add (appen) the following to /etc/rc2.d/S69inet file:

### Set kernel parameters for /dev/ip
# Turn  off IP forwarding
        ndd -set /dev/ip ip_forwarding 0          
        ndd -set /dev/ip ip_respond_to_echo_broadcast 0
        ndd -set /dev/ip ip_forward_src_routed 0
        ndd -set /dev/ip ip_forward_directed_broadcasts 0
        ndd -set /dev/ip ip_ignore_redirect 1

        ndd -set /dev/ip ip_send_redirects 0          

# Disable respond to ICMP netmask requests, need to enable for boot server !!!
        ndd -set /dev/ip ip_respond_to_address_mask_broadcast 0

# For boxes outsite the firewall, disable respond to ICMP broadcast echo
# requests (ping). default is 1 (true)
        ndd -set /dev/ip ip_respond_to_echo_broadcast 0

# disable respond to ICMP timestamp requests, for boxes outside firewall only
        ndd -set /dev/ip ip_respond_to_timestamp 0
        ndd -set /dev/ip ip_respond_to_timestamp_broadcast 0

# The machine will drop packets coming through one interface that are destined for
# another interface, prevent host spoofing
        ndd -set /dev/ip ip_strict_dst_multihoming 1

# End of script

3. Tell the machine that not to forward packages
     touch /etc/notrouter
     chmod 600 /etc/notrouter

4. Disable all the unwanted server in /etc/inetd.conf

5. /etc/ftpuser and /etc/passwd , make some of the system user have no login shell ...etc.

For network monitoring, you can use "Nmap", it is free, you can download it from:
http://sunfreeware.com/

or get someting like:
ethereal
snort
tcpdump
tcpflow

You can read more UNIX security in:
http://www.stokely.com/unix.sysadm.resources/security.html
http://www.itworld.com/nl/unix_sec/
http://www.insecure.org/tools.html
http://www.sans.org/resources/idfaq/

Hope that I do not wast your time. and I would like to learn some tips from you. (I hnew you have a lot of experiences).

Cheers!

yuzh








       
0
 

Expert Comment

by:jhundpogi
ID: 8150772
Hi,

Have you enabled priority paging in solaris (2.6 and 2.7)

1. To enable priority paging, set the following in /etc/system:

 set priority_paging=1

2. upgrade to oracle 8.1.7 (if possible)

3. Your scan rate is too way high!!!
you need to add more memory.

jundpogi
0
 
LVL 1

Author Comment

by:SBK1972_uk
ID: 8150844
Hello gents,

Firstly, many thanks for all your replys.  Ive followed all the links, read 90% of the documentation you've suggested.  The out come is that Ive now got sar running, Ive added several performance tunes that sun recommends, including the priority page /etc/system addage.  One thing there, you must have a certain patch for that if your running sol26 and you must have you permissoning sorted out, or it sees all files as a binary and doesnt swap iut out.

Its like all performance issues, trial and error. Im pretty much stuck by what the application Im running demands, I cant upgrade the app, which only works on sol26 and oracle 8.1.5

Anyway, I have already got 4Gb of ram, and 3 gb of swap (mutlple mkfile + swap -a ), yes that much ! hahahah But the poor application doenst clear up after it makes odbc connections which uses system resources, blah blah blah.

I have noted pretty much all your adivce in my little Unix book of knowledge, well a spread sheet now with mutliple worksheets for different apps.  Always use this on different clients sites.

Many thanks all.

SBK
0

Featured Post

Enroll in August's Course of the Month

August's CompTIA IT Fundamentals course includes 19 hours of basic computer principle modules and prepares you for the certification exam. It's free for Premium Members, Team Accounts, and Qualified Experts!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Attention: This article will no longer be maintained. If you have any questions, please feel free to mail me. jgh@FreeBSD.org Please see http://www.freebsd.org/doc/en_US.ISO8859-1/articles/freebsd-update-server/ for the updated article. It is avail…
FreeBSD on EC2 FreeBSD (https://www.freebsd.org) is a robust Unix-like operating system that has been around for many years. FreeBSD is available on Amazon EC2 through Amazon Machine Images (AMIs) provided by FreeBSD developer and security office…
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…
This video shows how to set up a shell script to accept a positional parameter when called, pass that to a SQL script, accept the output from the statement back and then manipulate it in the Shell.
Suggested Courses

764 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question